JP3667614B2 - Spoken dialogue method and system - Google Patents

Description

[0001]
[Industrial application fields]
The present invention relates to a voice dialogue system that uses various input / output means including voice recognition and voice synthesis.
[0002]
[Prior art]
In recent years, it has become possible to perform human-computer interaction in various forms by inputting, outputting, and processing multimedia such as characters, sounds, graphics, and images.
[0003]
In particular, workstations and personal computers that can handle multimedia have recently been developed due to dramatic improvements in memory capacity and computer power, and various applications have been developed. It has not yet reached the point of organically combining various media with just putting in and out the media.
[0004]
On the other hand, linguistic data including characters has become common instead of conventional numerical data, and black and white image data has been expanded to handle colorization, graphics, animation, three-dimensional graphics, and even moving images. . As for speech and audio signals, speech recognition and speech synthesis functions are being researched and developed in addition to simple speech signal level input / output. The current situation is limited to limited fields.
[0005]
That is, as described above, characters, texts, voices, graphic data, and the like continue to be developed from conventional input / output processing (recording-reproduction) to various media understanding and generation functions. In other words, understanding and generation of media such as audio and graphics for the purpose of handling the content, structure, and semantic content of the media from the surface processing of each media, and making human-computer interaction more natural and comfortable. The construction of a dialogue system that uses the Internet is being studied.
[0006]
As for speech recognition, it has progressed from isolated word recognition to continuous word recognition and continuous speech recognition, and is also being developed in a task-oriented direction for practical use. In such application situations, it is more important to understand the speech utterance content than to recognize the face of the speech as a spoken dialogue system. For example, a speech understanding system using knowledge of application fields based on keyword spotting is also available. Has been studied. On the other hand, with regard to speech synthesis, for example, the present inventors have conducted research on speech synthesis systems for dialogue that emphasizes intonation from conventional text-to-speech systems. Application is expected.
[0007]
However, the understanding and generation of media such as voice is different from simple data input / output, and information loss and errors are inevitable when converting media. That is, speech understanding is a process of extracting speech utterance content and speaker's intention from speech pattern data with a large amount of information, and speech recognition errors and ambiguity occur in the process of compressing information. Therefore, in order to deal with the above-mentioned speech recognition imperfections such as recognition errors and ambiguities, the voice dialogue system needs to ask the user appropriate questions and confirmations from the system side and proceed with the dialogue smoothly by dialogue control. .
[0008]
By the way, when performing some kind of dialogue with the user from the dialogue system side, it is important as an easy-to-use human interface to cover the imperfection of voice recognition and accurately convey the status of the computer. However, in the conventional spoken dialogue system, there are many cases in which text synthesis is performed simply by reading a sentence as a voice response. Or, there is no voice response, and all responses from the computer are displayed as text on the screen, or a system that displays graphic data, video, icons, and numerical values is common, and the burden on vision is heavy. It was.
[0009]
As described above, the various interactive systems described above have been developed recently. However, studies on the use of various media in response from the system side to deal with incomplete speech recognition have been sufficiently conducted so far. It was not done, and it was a big problem of speech recognition technology. In other words, voice recognition is unstable, weak against noise and unwanted words, and it is difficult to efficiently convey the user's intentions by voice, so it can only be used for voice media such as telephones. Application was limited to strong scenes.
[0010]
[Problems to be solved by the invention]
As described above, the conventional speech dialogue system using speech recognition and synthesis technology is a combination of speech recognition, speech synthesis, and screen display technologies that are separately developed. Have not been fully considered. In other words, there are recognition errors and ambiguity in the speech recognition function, the speech synthesis function is less clear than human speech, and the ability to transmit intentions and emotions is insufficient due to insufficient control of intonation, There is a fundamental problem of lack of naturalness. In addition, the current technology is not sufficient to generate an appropriate response using the speech recognition result on the system side. On the other hand, it can be expected that the transmission capability will be improved by displaying the response in combination with voice, but it is possible to display a two-dimensional planar and three-dimensional spatial screen for instantaneous and continuous time-series voice responses. It is an unresolved issue how to use and control the timing of both. Another important issue is what should be displayed as a spoken dialogue system using other media.
[0011]
The present invention has been made in view of the above circumstances, and an object of the present invention is to provide a voice dialogue system that can efficiently and accurately perform voice dialogue between the system and a user, and enables remarkable improvement in usability. To do.
[0012]
[Means for Solving the Problems]
The present invention provides a voice understanding unit that receives a voice input and understands the meaning content of the input voice, a dialogue management unit that performs a semantic determination of a response content based on an understanding result by the voice understanding unit, and a dialogue management unit The response generation means for generating the voice response output and the screen display output based on the response content determined in (1), and the output means for outputting the voice response output and the screen display output generated by the response generation means.
[0013]
The dialogue management means displays the person image information about the person image of the speaker who makes a voice response based on the understanding result of the voice understanding means, the response text information of the spoken sentence corresponding to the voice response, and the understanding contents related to the voice response contents. Each visualization information to be visualized is output as a response content.
[0014]
The response generation means generates a screen display output of at least one of the action and facial expression of the person image based on the person image information of the speaker who performs the voice response output from the dialogue management means.
[0015]
Further, the response generation means generates a screen display output of at least one of the action and facial expression of the person image based on the person image information of the speaker who performs the voice response output from the dialogue management means, and the sound corresponding to each screen display The voice response output having the emotion or strength is generated.
[0016]
Furthermore, it has human state detecting means for detecting a human state relating to the movement of a person, and the dialog management means makes a meaningful determination of the response content based on the detection result of the human state detecting means.
[0017]
An icon indicating whether voice input is possible is displayed.
[0018]
The present invention also provides: Speech understanding means for receiving a speech input, understanding the meaning content of the input speech, and scoring the degree of certainty of the meaning content with respect to the understood meaning content; Dialog management means for selecting an input semantic expression having the highest degree of certainty and generating an output semantic expression according to the semantic content and the score of the selected input semantic expression, and a system response according to the output semantic expression Response output means for outputting an output, wherein the dialog management means is a state transition between a user state in which voice input is given to the voice understanding means and a system state in which a system response output is output from the response output means Manage user interaction with the system by controlling It is characterized by that.
[0019]
Furthermore, the present invention provides speech understanding means for receiving speech input and understanding the semantic content of the input speech by detecting a keyword in the speech input, and depending on the state of dialogue between the system and the user, A dialogue management unit that restricts keywords in speech input detected by the speech understanding unit in advance, and a response output unit that outputs a system response output based on an understanding result by the speech understanding unit, To do.
[0020]
[Action]
As a result, the present invention uses a screen display of a response as a response output from the system side to the user in addition to the voice recognition and the voice response when performing a dialogue between the user and the system. At this time, by displaying the person image corresponding to the voice speaker from the system side, the image of the speaker becomes representative of the function of the system, and the user speaks toward the person on the screen. In addition, it is possible to grasp the progress of dialogue and the reliability of speech recognition by the movement and facial expression of the person's mouth on the screen.
[0021]
On the other hand, regarding the response contents from the system, in addition to displaying a character string representing the response sentence, objects (for example, items and things such as products and concepts) and numbers are displayed in figures etc. You can also tell the user quickly. Furthermore, speech recognition is incomplete with frequent misrecognition and ambiguity. When understanding the content of speech, incorrect information that is not intended by the user may be transmitted to the computer. In addition, by using visual response displays in parallel, it is possible to greatly improve the efficiency of voice conversation, and to improve naturalness and ease of use. In addition, since the synthesized speech is lower than natural speech, it is extremely useful to improve the dialogue by using the facial expression of the speaker, the response sentence, and the visualization of the response content.
[0022]
【Example】
Hereinafter, an embodiment of the present invention will be described with reference to the drawings.
[0023]
FIG. 1 shows a schematic configuration of a system to which a screen display as a voice interactive system is added.
[0024]
The speech dialogue system includes a speech understanding unit 11 that understands the meaning content of input speech, a dialogue management unit 12 that makes a semantic determination of response contents based on the understanding result in the speech understanding unit 11, and a dialogue management unit 12. A response generation output unit 13 that generates a voice response output and a screen display output based on the determined response content, a screen display output unit 14 that outputs a screen display generated by the response generation output unit 13, and a voice that outputs a voice response The output unit 15 is configured.
[0025]
The speech understanding unit 11 extracts the meaning content by understanding the input speech uttered by the user, instead of recognizing the character face of the speech, that is, recognizing a word or sentence. Then, an input semantic expression representing the understood semantic content is generated and sent to the dialogue management unit 12.
[0026]
The dialog management unit 12 makes a semantic decision on the response content for the input semantic representation of the input speech using information on the history of the dialog, the current dialog state, knowledge of the progress of the dialog, and knowledge of the application field. The response content information of the utterance corresponding to the response is output to the response generation / output unit 13.
[0027]
Furthermore, the dialogue management unit 12 processes spoken words including omissions and instructional pronouns, enabling natural dialogue as well as improving speech understanding performance and reducing the amount of processing. Further, the dialogue management unit 12 displays the person image information of the speaker who is displayed and output on the display 14 and makes a voice response, and the visualization information which is the information for visualizing the understanding content related to the content of the voice response. Output to.
[0028]
In addition, the output semantic expression generated by the dialogue management unit 12 is sent to the speech understanding unit 11, and the keyword and syntactic semantic rules of the next utterance are narrowed down from the output semantic expression to improve the speech understanding performance of the next utterance. It becomes possible.
[0029]
The response generation / output unit 13 outputs a response sentence generated based on the response content information input from the dialogue management unit 12 from the speaker 15 with synthesized speech, and an action and a facial expression are determined based on the person image information and the response sentence. The content visualization information generated based on the visualization information, which is information for visually displaying the image of the person who makes the voice response on the display 14 and visualizing the content understood by the system through the conversation so far Is visually displayed on the display 14 and a response is presented to the user in a multimodal manner using a plurality of media. That is, by presenting audio information and visual information together to the user, the human interface of the voice interaction system is improved, and natural conversation is possible.
[0030]
Further, information indicating that a response is currently being output is sent from the response generation unit 13 to the dialogue management unit 12. The dialogue management unit 12 can improve the voice understanding performance by sending the above information to the voice understanding unit 11 and controlling the timing of the start / end detection processing of the input voice and the keyword detection processing, for example.
[0031]
Next, each part of the above-described spoken dialogue system will be described in more detail, assuming an order task in fast food as an application.
[0032]
First, the voice understanding unit 11 will be described. As described above, the role of the voice understanding unit 11 is to understand the meaning of the voice and the intention and situation of the speaker, rather than recognizing the character face as in text input or a voice word processor. It is an object.
[0033]
In this case, for ticket vending machines targeting unspecified users, aircraft and train seat reservation systems, bank cash drawers, etc., voice differences between speakers, unnecessary words, differences in spoken language, noise effects Even if speech recognition technology is actually applied, sufficient recognition performance may not be expected, and it is particularly difficult to recognize a spoken sentence with high accuracy. For this, for example, literature (Hiroyuki Tsuboi, Hideki Hashimoto, Yoichi Takebayashi: “Analysis of keyword lattices for continuous speech understanding” is a method to understand the utterance content by analyzing candidate keyword sequences from continuously spoken speech. Proceedings of the Acoustical Society of Japan, 1-5-11, pp. 21-22, 1991-10), and if this method is used, the user's speech is not limited as much as possible in limited applications. In addition, free speech can be understood at high speed.
[0034]
FIG. 2 shows a schematic configuration of the voice understanding unit 11 using the keyword described above.
[0035]
In this case, the voice understanding unit 11 includes a keyword detection unit 21 and a syntax and semantic analysis unit 22. The keyword detection unit 21 includes a speech analysis unit 21a and a keyword spotting processing unit 21b. The syntax and semantic analysis unit 22 includes a sentence start end determination unit 22a, a sentence candidate analysis unit 22b, a sentence end determination unit 22c, and a sentence candidate table 22d. It is constituted by.
[0036]
In the keyword detecting unit 21, the input speech is passed through a low-pass filter (LPF) by the speech analyzing unit 21a and A / D converted at a sampling frequency of 12 kHz and a quantization bit of 12 bits to be converted into a digital signal, followed by spectrum analysis, and further After using FFT, smoothing is performed in the frequency domain, logarithmic conversion is performed, and a voice analysis result is output every 8 ms from a 16-channel bandpass filter (BPF). Keyword spotting processing is performed on this output. Is executed. The keyword spotting process in this case is, for example, a document (Kanazawa, Tsuboi, Takebayashi: “Word detection from continuous speech including unnecessary words”, IEICE Speech Study Group, sp91-22, pp. 33-39, 1991. It can be carried out by the method disclosed in -6).
[0037]
As a result, the keyword detection unit 21 extracts a keyword candidate series (lattice) from the continuous input speech. Fig. 3 shows an example of a candidate sequence of keywords extracted from the continuous input voice "Please give three hamburgers, potatoes and coffee" when applied to the exchange of orders at a fast food store by voice dialogue. Is shown.
[0038]
Note that the speech analysis and keyword detection processing described above can be found in other documents ("Development of speech recognition system using high-speed DSP board", Acoustical Society of Japan Proceedings, 3-5-12, 1991-3). Real-time processing is also possible by using a DSP board.
[0039]
Next, the syntactic and semantic analysis unit 22 analyzes the syntactic and semantic analysis of the keyword candidate series detected in this way, and obtains an input semantic expression for speech input as shown in FIG.
[0040]
Here, the application is limited to the fast food task, and the input semantic expression in the frame format is composed of an ACT frame in which the input utterance indicates the type of order processing and an ordered item frame in which the order contents are expressed. The ACT frame expresses semantic information about the order processing such as “order”, “addition”, “deletion”, “replacement”, etc., while the order item frame includes the name, size and number of slots. The contents of the ordered items can be expressed.
[0041]
That is, the keyword lattice obtained by the keyword detection unit 21 is sent to the syntax and semantic analysis unit 22. The syntax and semantic analysis unit 22 includes a sentence start end determination unit 22a, a sentence candidate processing unit 22b, and a sentence end determination unit 22c, and has a sentence candidate table 22d. The syntax and semantic analysis unit 22 processes each word in the keyword lattice from left to right.
[0042]
The sentence start end determination unit 22a determines whether or not the currently processed word can be the start end of the sentence based on the syntactic and semantic constraints. If it can be the beginning of a sentence, the word is registered in the sentence candidate table 22d as a new partial sentence candidate.
[0043]
The sentence candidate analysis unit 22b determines whether or not the word and each partial sentence candidate in the sentence candidate table 22d can be connected from the syntactic semantic time limit. If it can be connected, the partial sentence candidate is copied, the input word is connected to it, and it is registered in the sentence candidate table 22d.
[0044]
The sentence end determination unit 22c determines whether or not the partial sentence candidate processed immediately before by the sentence candidate analysis unit 22b can be syntactically and semantically established as a sentence. It is output as the output of the syntax and semantic analysis unit 22.
[0045]
The output sentence candidate is subjected to semantic analysis at the same time as syntactic analysis, so this means the input semantic expression as it is. The above processing is performed in a pipeline with respect to the input. Thus, a plurality of input semantic expressions for the input voice can be obtained.
[0046]
In the fast food task in this case, the keywords shown in FIG. 5 are used as keywords, but different utterances may have the same meaning depending on the situation of dialogue. In other words, in speech understanding based on keywords, “one” and “one” have the same semantic expression, and “please” and “please” may also have the same semantic expression. This is different from the input speech expression. This is the difference between speech recognition and speech understanding. In the spoken dialogue system handled in the present invention, task-dependent speech understanding processing using knowledge in the application field is required.
[0047]
Next, the dialogue management unit 12 will be described. In the spoken dialogue system according to the present invention, as shown in FIG. 1, the input meaning expression output from the voice understanding unit 11 is sent to the dialogue management unit 12, and knowledge of dialogue, knowledge of application fields, dialogue history and status information are sent. Is used to determine the response content semantically, generate an output semantic expression that is response content information for confirmation response, and output it to the response generation output unit 13. As shown in FIG. 6, the output semantic expression uses a frame format expression as in the case of the input semantic expression.
[0048]
In the present embodiment, the input semantic expression shown in FIG. 4 is used as the content expression of one utterance of the input speech, but the contents of the order so far are stored as the contents understood by the system from the start of the dialogue. An order table is prepared separately as shown in FIG. In addition, changes in the order table accompanying the progress of the dialogue are prepared as a dialogue history in the form of an order table (old order table) at the time of the previous question response as shown in the example of FIG. Furthermore, the dialog management unit 12 holds dialog status information indicating the status of the dialog. This dialogue status information includes information such as the current dialogue state, the next transition state, the number of dialogue repetitions, the certainty factor, the emphasis item, the dialogue history, and the like. It is used as information.
[0049]
The order table is rewritten based on the ACT information of the input semantic expression and the order content, and the format is a table of only the order content obtained by removing the ACT information from the input semantic expression. In other words, this order table reflects the contents understood in the dialog since the start of the dialog. The old order table has the same configuration as the order table, holds an order table in a question response at the time of the previous dialog, and records the status of the order table as dialog history information.
[0050]
In this way, the dialogue management unit 12 responds using the knowledge of the dialogue progress method and application field based on the semantic representation of the input speech (input semantic representation), the history information of the dialogue (old order table), and the state of the dialogue system. Response content information (output semantic expression) representing the output content is generated. In other words, the input semantic expression and the order table are referred to, processing depending on the state (state number) of the system at that time is performed, and the output semantic expression expressing the response generation contents including the response generation contents and the response ACT is generated. Like to do. As described above, the output semantic expression in this case uses a frame format expression in the same manner as the input semantic expression. Furthermore, the dialog status information is generated based on the dialog history information (old order table) and the status of the dialog system so that the response generation output unit 13 can refer to the person image information on the response screen display.
[0051]
FIG. 9 shows an example of state transition inside the dialogue management unit.
[0052]
In this example, the dialog is managed and progressed by expressing the state transition based on the dialog progress method and the knowledge of the application field. The dialogue management unit 12 is roughly divided into two states: the user 72 and the system 71.
[0053]
Here, the role of the state of the user 72 is to transition to the state of the system 71 according to the input semantic expression of the user's utterance, while the role of the state of the system 71 depends on the understood utterance content, The contents of the order table are changed, the output semantic expression of the response is output, the flow of the dialog is advanced, and the state of the user 72 is changed. Thus, by having the internal state of the system divided in two, various exchanges between the user and the system can be expressed, and a flexible dialogue can be performed.
[0054]
The dialog status information used at this time indicates the status name of the dialog management being processed and the number of partial dialog repetitions. The status name of the ongoing dialog, the name of the next transition, and the same topic are the same. The number of times a partial dialogue such as a question is repeated is sequentially recorded so that it can be easily referred to. In order to convey the state of the system in a natural and easy-to-understand manner, It is used for response generation output by using it as human image information such as movement, emotion of voice response, and emphasis.
[0055]
Now, in FIG. 9, when the dialog management unit 12 detects the presence of the user (customer), the dialog starts from the initial state S0 of the system 71, generates an output semantic expression related to greetings and order requests, and responds. It is sent to the generation output unit 13 and transitions to the initial state U0 of the user 72. Furthermore, dialog history information (old order table) is initialized, and dialog status information is generated based on the transition of the system state from state S0 to state U0. This dialog situation information can be referred to by the response generation / output unit 13 as person image information on the response screen display.
[0056]
Then, the response generation unit 13 generates a voice response, a person image, a text, and content visualization information while referring to the system state, the conversation history information, and the order table based on the output semantic expression.
[0057]
At this time, in the initial state U0 of the user 72, when the input semantic expression ACT information of the next utterance is "order", the dialog progress state SP of the system 71 and the user's dialog progress state UP are displayed in a general order flow. Transition to the transition between.
[0058]
On the other hand, when the ACT information of the input semantic expression is other than an order, the user's utterance is regarded as an unexpected user's utterance, and the system 71 transits to the dialog correction state S10.
[0059]
If the system 71 transitions to the dialog correction state S10 of the system 71, the system 71 uses the input meaning expression, the order table and its history information, and the voice input received from the user 72 has unexpected contents, Inform the user 72 of the fact that he / she could not hear it with an appropriate response according to the situation, or output an output meaning expression for confirming the details of each item in detail, and transition to the user's 72 dialog progress state UP. become. Then, the response generation unit 13 generates a voice response, a person image, a text, and content visualization information while referring to the system state, dialog history information, and order table based on the output semantic expression.
[0060]
In this way, the exchange between the system 71 and the user 72 continues, the utterance at the user 72 and the response at the system 71 are performed, and the state transitions, but when the user 72 finishes the target order, That is, when the user 72 utters an affirmative response to the confirmation response for all orders in the system 71, the process moves to the end state S9 of the system 71 and the dialog is terminated.
[0061]
FIG. 10 shows a flowchart of the user status process.
[0062]
In this case, first, a plurality of input semantic expressions are read (step S81), an abbreviated expression is inferred (step S82), and a score (scoring) regarding the probability of each input semantic expression is performed (step S83). Next, the input semantic expression having the highest score is selected from the input semantic expression candidates (step S84), the speech act is determined (step S85), and the system state is changed based on the speech act (step S85). S86).
[0063]
On the other hand, FIG. 11 shows a flowchart of processing of the system state.
[0064]
In this case, the contents of the order table are first changed based on the input semantic expression (step S91), an output semantic expression is generated in consideration of the state of the system at that time (step S92), and the response content is output (step S91). S93), the state changes to the user state (step S94). The response generation unit 13 generates a voice response, a person image, a text, and content visualization information based on the generated output semantic expression.
[0065]
As described above, in the system according to the present embodiment, when a message is received from the other party separately for the user and the system, it is possible to perform processing in consideration of various knowledge, situations, and message contents. The processing that seems to be possible becomes possible.
[0066]
Next, FIG. 12 shows a specific example of dialogue processing in the system of this embodiment.
[0067]
In this case, in the system, assuming that the order table shown in FIG. 12B and the output semantic expression shown in FIG. 12A in the previous state are presented to the user's utterance, FIG. As shown in (c), a confirmation message “Your order is 1 hamburger, 2 coffees, 4 colas” is generated, voice response based on this, confirmation text, items in order table The picture and the number of images are presented to the user in a multimodal manner using audio media and visual media.
[0068]
On the other hand, if the user inputs a voice message “Please add one cola” as shown in FIG. 12C, the voice understanding unit 11 shown in FIG. Is detected, the keyword candidate series (keyword lattice) is analyzed (parsing), and the input semantic expression candidate 1 and the input semantic expression candidate 2 shown in FIG. 12D and FIG. Get.
[0069]
Each candidate here has a score (score) D about the probability, the input semantic expression candidate 1 has a score of D1, and the input semantic expression candidate 2 has a score of D2, and they are arranged in the order of the scores.
[0070]
In this case, in the input semantic expression candidate 1 of the score D1, the ACT information is added, the product name is cola, the size is indefinite, the number is 1, and in the input semantic expression candidate 2 of the score D2, the ACT information is deleted. The product name is potato, the size is large, and the number is 2.
[0071]
Then, the process in the user's state is executed as shown in FIG.
[0072]
First, for input semantic expression candidate 1, referring to the fact that the cola size of the previous output semantic expression was large, the abbreviated expression is supplemented by inferring that the current additional order of cola is also large. (Step S111). The input semantic expression candidate 2 is not particularly omitted, so this inference is not performed (step S113).
[0073]
Next, the validity is checked. That is, the contents of the order table are compared with the input semantic expression candidates to check for inconsistencies (steps S112 and 114).
[0074]
In this example, for the input semantic expression candidate 2, since the utterance ACT of the input semantic expression is “deleted” and the product name is large, the order table has no potato, so the score D2 is D2 ′ = D2 × α ( The process of reducing by the process of α <1.0) is performed.
[0075]
Next, the input semantic expression candidate scores are compared (step S115). In this case, since D2 ′ <D1 ′, candidate 1 is selected. On the other hand, the utterance ACT of the input from the user is determined to be “addition” (step S116), and a transition is made to the state SA of the system that confirms the addition (step S117). Here, the state SA of the system for confirming the addition is rewritten so that one large cola is added to the order table based on the input semantic expression.
[0076]
In this case, the status processing in the system is executed as shown in FIG.
[0077]
That is, in this state, the system side selects a confirmation response to the user from the response ACT list shown in FIG. In this example, since the ACT of the input semantic expression is added, the fourth addition confirmation is selected for the response ACT, and a response output (response sentence) is determined and output using these pieces of information.
[0078]
First, one large cola is added to the order table (step S121). Then, since the utterance ACT of the input semantic expression shown in FIG. 15A is added, the response ACT is selected as addition confirmation, and the output semantic expression shown in FIG. 15C is obtained from these information (step S122). ). Next, a response output (response sentence) is determined and output (step S123). The response sentence in this case is determined based on the example of the system response shown in FIG. 16 and is output, for example, “Check. Add one large cola.” Then, the process proceeds to step S124, and the state transitions to the state UA of the user who has received the response ACT of the addition confirmation, and the process in the state of the user is performed.
[0079]
The dialogue management unit 12 sends the user's utterance speed and the likelihood of each keyword received from the voice understanding unit 11 to the response generation output unit 13 together with the output semantic expression obtained as described above.
[0080]
Here, the voice rate of the user is obtained as follows. That is, the keyword spotting processing unit 21b of the voice understanding unit 11 in FIG. 2 detects the obtained keyword, its start / end, and the user's utterance speed obtained based on the keyword. Further, the likelihood of each keyword of the voice understanding result is output to the dialogue management unit 12 together with the input semantic expression. If the start and end of the word obtained from the keyword spotting processing unit 21b and the word are known, the user's utterance speed can be obtained as shown in FIG. That is, when three keywords “hamburger”, “potato”, and “please” are detected from the user's utterance as the start t1 and the end t2, the start t3 and the end t4, and the start t5 and the end t6, respectively, the number of mora of these keywords Is 6, 3 and 4, so the average utterance speed of the user is
[Expression 1]
{6 / (t2-t1) + 3 / (t4-t3) + 4 / (t6-t5)} / 3
It can be calculated as follows.
[0081]
The voice rate and likelihood of the user obtained in this way are input to the dialogue management unit 12 together with the input semantic expression.
[0082]
The dialog management unit 12 adds the user's utterance speed and keyword likelihood input from the speech understanding unit 11 to the response generation output unit 13 in the output semantic expression generated based on the processing described in FIGS. input. An example of the output meaning expression at this time is shown in FIG.
[0083]
Next, the response generation output unit 13 will be described. In the response generation output unit 13 according to the present invention, the output meaning expression which is the response content information, the person image information composed of the conversation situation information and the conversation history information, and the visualization for visualizing the contents understood by the system so far in an easy-to-understand manner Based on the information, voice response, person image, text, and content visualization information are generated and output. Here, the voice response, the person image, and the text are generated based on the output semantic expression and the person image information, with facial expressions, emotions, and emphasis so that the contents to be conveyed can be presented in an easy-to-understand manner in consideration of the conversation status. The contents visualization information is for displaying the contents of the dialog understood by the system so that the progress of the dialog can be easily understood, and is generated and output based on the visualization information output from the dialog management unit 12. Is.
[0084]
FIG. 19 shows an example of the configuration of the response generation / output unit 13. The response generation unit 13 includes a response sentence generation unit 131, a human image expression determination unit 132, a human image generation unit 133, an emotion / emphasis determination unit 134, a voice response generation unit 135, a content visualization information generation unit 136, and an output integration control unit 137. Composed.
[0085]
The response generation / output unit 13 receives the output semantic expression and the person image information from the dialogue management unit 12, and generates a response sentence and its structure information in the response sentence generation unit 131. Based on the generated response sentence and the person image information from the dialogue management unit 12, the person image expression determination unit 132 determines the expression of the person image to be voice-responsive on the screen display, and generates the person image of the determined expression. Generated by the unit 133 and output to the output integration control unit 137. Also, based on the generated response sentence and sentence structure information and the person image information from the dialogue management unit 12, the emotion / emphasis determination unit 134 determines the emotional expression and emphasized part of the voice response, and the voice having emotion and emphasis. A response is generated by the voice response generation unit 135 and output to the output integration control unit 137. Further, the generated response sentence is output to the output integration control unit 137 as text information. In addition, in order to visualize and display the understanding content related to the response content, the response generation output unit 13 receives the visualization information output from the dialogue management unit 12, and the content visualization information generation unit 136 generates the content visualization information. The data is output to the output integration control unit 137.
[0086]
The output integrated control unit 137 receives a human figure having a facial expression, a voice response having emotion and emphasis, text information, and content visualization information from each unit, and controls the screen display output unit 14 and the voice output unit while controlling the temporal presentation order. The response contents are integrated and displayed for the user.
[0087]
Next, the operation of each part of the response generation output unit 13 will be described with reference to FIG.
[0088]
First, information processed by each unit of the response generation / output unit 13 will be described.
[0089]
The output semantic representation has the same frame format as the input semantic representation as shown in FIG. 6, and the ACT information indicates an action in response.
[0090]
The person image information is the expression of the person image that responds by voice and the emotion / emphasis information of the voice response displayed on the screen display output unit 14, and has a structure as shown in FIG. The system state number and the user state number indicate the number of each state when the output state expression is generated by transitioning from the system state to the user state in the dialog processing of the dialog management unit 12. SP1 and UP3 shown in FIG. 20 respectively indicate one state of the system-side state set SP of the dialog state transition of FIG. 19 and one state of the user-side state set SU. The number of repetitions is the number of times when the same question is repeatedly repeated in the dialogue or the same content is repeatedly checked. The emphasis item indicates an item in the output semantic expression that needs to be particularly confirmed. The certainty factor indicates the certainty factor of the corresponding content based on the output semantic expression, and is a score D obtained when the dialogue management unit 12 interprets the input semantic expression based on the likelihood of the input semantic expression. The response sentence generation unit 131 generates a response sentence and its sentence structure from the output semantic expression generated by the dialogue management unit 12. For sentence generation, there are generally known ones that use rewrite rules, one that uses hole filling, one that synthesizes from a tree structure, and one that combines from a semantic structure.
[0091]
For generating the output response sentence, for example, as shown in FIG. 21, a response sentence type with a hole for embedding an item, size and number is prepared for each ACT information, and the sentence structure is prepared, and according to the flowchart shown in FIG. It can be realized by filling the space based on the output semantic expression. That is, first, a variable n indicating the number of repetitions is set to 0 in step S141, and then the number of items in the output semantic expression is set to a variable M in step S142. In the case of the output semantic representation of FIG. Next, in step 143, the product name, size, and quantity for one order item are embedded in the response sentence. Next, while repeatedly adding the variable n in step S144, the process is repeated until embedding is completed in step S145. When the output semantic expression of Fig. 22 (b) is embedded in the response sentence type of Fig. 22 (c), as shown in Fig. 22 (d), "Confirm. There is one large cola and three small potatoes. ”Is obtained.
[0092]
The person image expression determination unit 132 determines the expression of the person image from the sentence generated by the response sentence generation unit 131 and the person image information input from the dialogue management unit 12. An example of the person image facial expression determination unit 132 is shown in FIG. The system state number, the user state number, the number of repetitions, and the certainty factor are obtained from the person image information, and the person image and the expression for each value are expressed in the form of a table in advance. For example, if the degree of certainty is high in the confirmation when the number of repetitions is one, confirmation is performed with an ordinary expression, and if the degree of certainty is low, confirmation is performed with a confused expression.
[0093]
The person image generation unit 133 generates an image to be displayed on the screen from the person image and expression information output from the person image expression determination unit 132. At this time, control for changing the display time and the person image is performed. For example, multiple images are prepared when using still images, and continuous operations when using moving images, so that the movement of the mouth when a human image responds by voice or the operation of bowing when greeting is generated. A video of human figure and facial expression is specified. In addition, when using computer graphics, an image of a specified operation is generated.
[0094]
The emotion / emphasis determination unit 134 determines voice emphasis and emotion to respond from the person image information. An example of the emotion / emphasis determination unit 134 is shown in FIG. The expression format is the same as that of the human face expression determination unit 132. From the system state number, the user state number, the number of repetitions, and the certainty level, the human image and voice response emotions for each value are expressed in a table format in advance. It is a thing. For example, when the degree of certainty is high when the number of repetitions is one, the confirmation is performed with a normal voice, and when the degree of certainty is low, the confirmation is performed with a confusing voice. In addition, there is an emphasis item in the person image information in order to emphasize when confirming and convey the confirmation contents to the user in an easy-to-understand manner. In this case, items to be confirmed when the response contents are generated as output semantic expressions in the dialog management unit 12 are determined. The emotion / emphasis determination unit 134 extracts items to be emphasized in the response sentence from the output semantic expression and transmits them to the next voice response generation unit 135.
[0095]
The voice response generation unit 135 performs voice synthesis based on outputs from the response sentence generation unit 131 and the emotion / emphasis determination unit 134. A conventional recording editing type can be used as a speech generation method, but this embodiment is characterized by a response with emphasis and emotion, and in order to realize it by controlling the speech generation unit, speech rule synthesis is used. desirable.
[0096]
An example of the configuration of the voice response generation unit 135 is shown in FIG. The voice response generator 135 includes a phoneme processor 151, a prosody processor 152, a control parameter generator 153, and a voice waveform generator 154.
[0097]
Here, the phoneme processing unit 151 and the prosody processing unit 152 each perform phonological processing based on the phrase (phrase) to be emphasized and the type of emotion input from the emotion / emphasis determination unit 134 and the generated response sentence and its structure. Prosody processing is performed, and a time series of control parameters used in the speech waveform generation unit 154 is output from the control parameter generation unit 153 to the speech waveform generation unit 154.
[0098]
Based on the response sentence generated by the response sentence generation unit 131 and its sentence structure, the phonological processing unit 151 determines the reading of the output response sentence according to generally well-known phonological rules such as nasalization, devoicing, and rendaku. Output a symbol string.
[0099]
The prosodic processing unit 152 determines the prosodic components such as the fundamental frequency pattern, power, duration, and pose position based on the response sentence and its structure, the word information to be emphasized, and the type of emotion.
[0100]
In particular, as shown in the model of FIG. 26, the basic frequency pattern generation is performed by analyzing the amount of accent components and phrase components of each response sentence when not emphasized in advance indicated by a dotted line and when emphasized indicated by a solid line. In addition, it can be realized by using the component in a word or phrase during synthesis. Also, sentence types such as plain texts, question sentences, and command sentences may be classified, and accent and phrase rules may be created for each sentence type. For example, from the literature (Hirose, Fujisaki, Kawai "Continuous Speech Synthesis System-Especially Prosodic Features Synthesis-", Acoustical Society of Japan Speech Research Material S85-43 (1985)), from the accent type of words and sentence breaks The rule of the plain text can be determined from the position of the word and the modification relationship.
[0101]
Prosody with emotion is described in the literature (K. Sheahan, Y. Yamashita, Y. Takebayashi, “Synthsis of Nonverbal Expressions for Human-Computer Interaction”, Acoustical Society of Japan 2-4-6 (1990.3)). As stated, it is mainly controlled by the rate of change of the fundamental frequency and the dynamic range, utterance time length, and energy. Therefore, as shown in FIG. 27, in the case of pleasure, the accent is increased by 1.2 times, the utterance time is increased by 0.9 times, and the energy is increased by 2 dB compared to the case without emotion. Reduces the accent by 0.9 times, utterance time by 1.1 times, and energy by 2 dB. This makes it possible to synthesize a voice that is clearly spoken word-by-word when accompanied by joy, and generates a slightly slower synthesized sound with little inflection when accompanied by sadness.
[0102]
The control of the fundamental frequency is not limited to that used in FIG. 41, but there are a method using linear approximation and a method of expressing the fundamental frequency pattern at a high and low sound level, not limited to those described here, but the gist of the invention. Various methods may be used without departing from the above.
[0103]
The control parameter generation unit 153 determines control parameters used in the speech waveform generation unit 154 based on the phoneme symbols and prosodic symbols from the phoneme processing unit 151 and the prosody processing unit 152. Since the control parameter generation unit 153 also controls the utterance speed, it is possible to synthesize voice in accordance with the user's utterance speed, and to proceed with the conversation at the pace of the user's utterance.
[0104]
Therefore, the utterance time length obtained by the control parameter generation unit is output to the output integrated control unit 137 in order to synchronize the mouth movement of the person image and the voice response.
[0105]
In the response generation / output unit 13, the response sentence is generated by the response sentence generation unit 131, the emotion / emphasis determination unit 134, and the voice response generation unit 135 as described above. Reference to reflect the sentence length. This is because when a conversation with a good tempo is performed, it is better to respond shortly, and when speaking slowly for reasons such as the user being confused, it is better to respond without carefully omitting. For example, if the speaking rate is faster than 9 mora per second, this can be realized by selecting a short response sentence pattern.
[0106]
In addition, the likelihood of each keyword given from the dialogue management unit 12 is used, for example, to properly use “Is it?” At the end of a sentence in a confirmation scene. In other words, “?” Is used when the average likelihood of a keyword is lower than, for example, the setting range value 0.5, or the likelihood of any keyword is very low, and “I” is used when the likelihood is high. As a result, in addition to other response outputs, the level of understanding of the computer can be understood from the response sentence, and the user can easily interact.
[0107]
In addition, it may be possible to change the sentence pattern after determining the sentence pattern, without having “Is it?” As a sentence pattern table. Also, other words may be used, such as using information on whether or not the response is polite, such as “Is it?
[0108]
The voice waveform generation unit 154 uses rule synthesis by a formant synthesizer as shown in FIG. 29, for example. For example, a voice can be synthesized by updating a synthesis parameter every 8 msec with a sampling frequency of 12 kHz and using a sound source obtained by applying an impulse to a low-pass filter. However, it is possible to use generally known configurations of the synthesizer, the type of sound source, the sampling frequency, and the like.
[0109]
Note that in the speech waveform generation unit 154 including the formant synthesizer shown in FIG. 29, the control parameters input from the control parameter synthesizer 169 are the impulse generator 161, the noise generator 162, the low-pass filters 163A and 163B, the amplitude. This is distributed to the controller 167, the high-pass filter 168, and the resonators 166A and 166B, respectively.
[0110]
The visualization information is information such as contents transmitted to the system during the conversation, contents understood by the system, system status, etc. Based on this visualization information, the contents visualization information generation unit 136 generates the contents visualization information and the user By visually presenting it to the system, it becomes possible for the system and the user to share the state of the system and the contents of understanding, and the dialogue can be naturally and easily understood.
[0111]
In this embodiment, the order table is used as visualization information. In the order table, all items, sizes, and quantities already ordered by the user are recorded, and the order contents at each point of the dialog can be confirmed. In this way, for example, when placing an order with many items, each item, size, and number are visually displayed rather than being heard continuously in time only by voice response to convey the contents of the order in parallel. Is possible. The content visualization information generation unit 136 generates an image based on the visualization content information. As the image generation method here, the method described in the person image generation unit 133 can be used. That is, control for changing the display time and the display image is performed. When a still image is used, a plurality of images are prepared, and when a moving image is used, a display image of a continuous operation is designated. In addition, when using computer graphics, an image of a specified operation is generated.
[0112]
The output integration control unit 137 includes image information of a human image having a facial expression that is output from the human image generation unit 133, signal information of voice response having emotion and emphasis that is output from the voice response generation unit 135, and a character string of a response sentence. Is received from each unit and is output to the screen display output unit 14 and the voice output unit 15 while controlling the temporal presentation order, and is output to the user. Integrate and present response content.
[0113]
What is important here is that it is not necessary to present each output individually, but it is necessary to present them while considering the temporal relationship of the individual output information. For example, it is necessary to synchronize the voice output with the control of the voice response output when the person image moves in response to the voice response, and the voice output with the voice output when greeting while bowing. It is also important to control the order in which each output is presented.
[0114]
30, 31, 32, and 33 show examples of time control of the output presentation order. FIG. 30 shows the control of the scene of the first greeting. Since there is no order yet, the contents visualization information is not displayed, but the text information of the greeting is displayed at the time of t0, and at the same time, the person image is “welcome”. While moving the mouth in response to the voice response “please order”, prompt the order. The content of the utterance and the person image screen are synchronized in this way, and all the text information is displayed at time t0 so that it can be easily understood in advance.
[0115]
FIG. 31 shows a scene in which one hamburger and one cola have already been ordered, and text information of the response confirmation content “Order is one hamburger and one coffee” is displayed until time t0. Next, the content visualization information is newly updated at t0 when the voice response and the human image start, and three hamburgers, two coffees, and one cola are displayed. Further, the person image is controlled so as to move the mouth from t0 to t3 in accordance with the voice utterance.
[0116]
The time control shown in this example is determined based on the length of the voice response. That is, in FIG. 30, the duration from t0 to t1 is determined by “welcome” and the duration from t1 to t2 is determined by “please order”. Each duration time is determined by the voice response generation unit 135, and the voice response signal and its duration time are sent to the output integrated control unit 137 and used for time control. In addition to the above, it is also possible to perform time control on the basis of the content visualization information to be presented and the display time of the human image.
[0117]
FIG. 32 shows the control of the confirmation of all orders after receiving the first order. The items to be confirmed are three items, two hamburgers, one cheeseburger, and three coffees. In FIG. 32, the text information “The order is two hamburgers, one cheeseburger, and three coffees” is displayed at the time of t0, and the voice response and the human figure corresponding to it are displayed. Start mouth movement. The contents visualization information is not displayed until the voice "Order is", but at the time t1 when the order contents are uttered, two hamburgers, one cheeseburger, and three coffees are displayed as the contents visualization information. Like that. The person image is controlled so as to move its mouth from t0 to t4 in accordance with the voice.
[0118]
Here, the response sentence for confirming all orders is generated by the response sentence generating unit 131. However, when the number of items to be confirmed increases, the generated response sentence becomes longer and the length of the voice response becomes longer. However, in the example of FIG. 32, the user can understand the response contents of the system or the state of the system and the understanding contents without listening to the voice response by the contents visualization information displayed at the time t1. Information output from t1 after displaying the content visualization information to t4 when the voice response ends is a redundant response for the user.
[0119]
Therefore, in this embodiment, as shown in FIG. 33, when confirming all orders and there are three or more items to be confirmed, the output presentation order is changed, and the immediately preceding response text is displayed at the time t0. One end is deleted, and two hamburgers, one cheeseburger, and three coffees are displayed as the contents visualization information. Next, at the time t1 when the processing for displaying the content visualization information is finished, a response sentence text “Are you sure?” Is displayed and a human image and voice response are started. The time control shown in this example is performed by the output integration control 137 based on the ACT information and the number of items of the output semantic expression generated by the dialogue management unit 12, and the response sentence is generated by the response sentence generation unit 131. The
[0120]
This is not limited to the confirmation of all orders. When other confirmations are made, if there are a lot of response confirmation items or if they are complicated and difficult to understand, after the visual response is output first, It is also possible to efficiently perform dialogue in a short time by performing a voice response that is abbreviated using a pronoun or the like.
[0121]
It should be noted that, instead of the number of items to be confirmed, the change in response output as described above may be controlled using an index indicating the length of another voice response, for example, the number of words or the number of mora in the voice response. .
[0122]
Further, the output integration control unit 137 controls the display location of each image display information. For example, a person image can be controlled and displayed on the screen of the image output device 14 in the upper left, content visualization information on the right, and text information on the lower left. This display position can be changed based on the control of the output integrated control unit 137.
[0123]
As described above, the present invention is characterized in that the dialogue is advanced by using both voice input and output and screen display. Here, an actual example of the screen display in the present invention will be specifically described.
[0124]
First, FIG. 34 shows an initial screen. When the customer is not in the store or does not come close, only a sentence such as “Welcome to” is displayed on the screen, and no voice response is output.
[0125]
Here, when the user (customer) approaches the system (counter, drive-through window, etc.), for example, when the user is detected by sensor information such as a floor mat with a pressure sensor or an image of a surveillance camera, FIG. As shown in Fig. 9, the system displays a mixed sentence of kanji and kana, "Please come and place an order." And displays a cheerful clerk on the screen and outputs a voice response (state S0 in FIG. 9). ).
[0126]
At this time, it is important for the user's detection to take into account the movement and position of the person, and in particular, to execute the above-described processing when a stop is detected and to start a voice conversation with the user with peace of mind. In particular, the smile of the clerk has the effect of relaxing the customer, and it is also desirable to synthesize a bright voice. These technologies have already been developed, and recorded synthesized sounds and natural sounds can also be used.
[0127]
From this state, it is assumed that the user makes an order with a quick mouth while looking at the screen, “Eh, please give me two hamburgers and two coffees. Then, in the system, the user's pronunciation in the state U0 in FIG. 9 is processed, but there is a part that cannot be heard at present, and if no result is obtained from the voice understanding unit 11 shown in FIG. Then, it corresponds to rejection.
[0128]
In this case, as shown in FIG. 36, the system displays a mixed sentence of kana and kana, saying “I could not hear it clearly. respond. In this state, the system side cannot hear the user's order at all, and there is no order table at that time (empty), so no screen display regarding the order is made. The salesclerk's facial expression generation is output as a response sentence. In this case, the process shifts from the user state U0 in FIG. 9 to the reject dialogue correction state S10, where the response and facial expression are determined.
[0129]
Next, if the user who receives the response from this system places an order saying “Please have 2 hamburgers and 2 coffees” in a clearer way than the previous time, the speech understanding process and the dialog process described above will be executed. After generating the input semantic expression and the order table, the output semantic expression is determined. Then, when the output meaning response ACT becomes “confirm all”, the screen display and the voice response shown in FIG. 37 are used together as the next response.
[0130]
In this case, the system displays a mixed sentence such as “Your order is two hamburgers and two coffees.” And displays the face of the clerk on the screen and outputs a voice response. Become. At this time, the facial expression of the clerk and the emotion of the voice response are determined in consideration of the sentence and the state as described above, and an ordinary facial expression and voice response are output here. In addition, the contents of the order table are displayed on the screen together with the response text, and the user can confirm in a short time whether or not the item is the item requested by the user.
[0131]
In this case, the product may be displayed on the screen in a state in which the product is arranged in the order of the number without displaying the number. Here, numerical information is important, so an area having the same height as an article such as a hamburger is provided to display the numeral. In other words, information about the number (number) is important, so that it can be conveyed naturally to the user. As for the display size of numbers, it is also effective to display the information by changing the size because it can convey roughly information by size, and also output using text information or color information together By doing so, it becomes possible to convey a realistic image to the user more quickly than a voice response or a text response, thereby realizing a faster confirmation dialogue. On the other hand, for a person image related to a store clerk, a picture with a small amount of information that conveys a point to be transmitted is more effective than a real expression. Further, the above image display can naturally be performed using three-dimensional graphics.
[0132]
Now, suppose that the user has confirmed the order from the system side and pronounced the user with a little hesitation saying, “Well, that ’s OK.” Then, in the system, since no result is obtained from the voice understanding unit 11, the dialogue management unit 12 responds to the rejection. In this case, as shown in FIG. 38, the system displays a mixed sentence of kanji and kana as "I'm sorry. Please input again." And displays a clerk on the screen and makes a voice response. The message in this case is shorter than the case of FIG. 36 described above, and the response sentence is determined using the state and the history information of the dialogue so as to convey the voice dialogue briefly. In addition, as for the facial expression of the store clerk, what seems to be sorry is output corresponding to the response sentence.
[0133]
The point of this screen display is that the contents of the order currently understood by the system are displayed in the right area. Since the display of the order item displays the order table as it is, it is possible to make up for a temporary defect of the voice. In other words, confirmation for addition, replacement, or deletion is performed for a partial local order by voice response or response sentence, but the effect of displaying the accumulated order as a result of the progress of the dialogue and the order item as a result of confirmation is great. .
[0134]
Such a display can be easily realized by the processing in the dialog management unit 12 described above. Further, a visual display can be used for partial confirmation, and the entire order can be displayed continuously in another display area. Furthermore, in order to temporarily hide the display of all ordered items and concentrate the user's consciousness on the partial confirmation, a partial confirmation screen display can be performed. In other words, it is effective to use a display method that combines the advantages of both, depending on the situation.
[0135]
After this, when the user utters “So, it ’s okay” with a clear voice, the system understands this as affirmative and transitions to S9 in FIG. 9, and the system “Thank you” as shown in FIG. A kana-kana mixed sentence is displayed, and a clerk with a bowed head is displayed on the screen, and a voice response is made to end the dialogue.
[0136]
Response sentence generation, smile generation, and bow gesture generation at this time are also performed by the same processing as described above. The total amount can be displayed with various responses (voice, screen display).
[0137]
In the case of the confirmation in FIG. 38 described above, it is also effective to display the screen so as to guide the answer “Yes” or “No” as shown in FIG. In this case, you can use information on the number of times you have heard and corrected, and the system will respond according to the situation, such as “I ’m sorry. Is your order two hamburgers or two coffees? There is an advantage that can be done.
[0138]
FIG. 41 briefly summarizes the processing procedure in such an embodiment.
[0139]
In this case, when the floor mat 220 detects the user, the keyword detection unit 21 detects the keyword with respect to the voice input from the user, obtains the word candidate series 222, and then the syntactic and semantic analysis unit 22 performs voice understanding based on the keyword. The input semantic expression 224 is obtained. Further, dialogue processing by dialogue and application field knowledge is performed in the dialogue control unit 12 to obtain an output semantic expression 226, which is given to the response generation unit 13, and voice response output and screen display by rule synthesis here A multimodal consisting of output responds.
[0140]
Although the above description has been given by way of an example of fast food ordering, it can be applied to a seat reservation system using an information service, multimedia, a workstation, and a communication network.
[0141]
Next, another embodiment of the present invention will be described with reference to FIG.
[0142]
FIG. 42 shows a voice dialogue system according to the present invention in which a function for detecting a human movement state is incorporated. In this case, human state detection is a function that is necessary for the system to automatically start and end the dialogue, and to naturally advance the dialogue by understanding the user's state and reaction at the start and end of the dialogue. Is possible. As a human state detection method, it is conceivable to perform processing by processing light, ultrasonic waves, infrared pressure, etc. Here, an example using a floor mat capable of detecting one adult will be described.
[0143]
In FIG. 42, a human state detection unit 233 is provided in addition to the voice input unit 231, the voice understanding unit 232, the dialogue management unit 234, the response generation unit 235, the display 236, and the speaker 237 similar to those described in FIG. 1. Yes.
[0144]
In this case, as shown in FIG. 43, the human state detection unit 233 displays the human state detection meaning expression 1 when a person is on the mat, and the human state detection meaning expression 2 when the person gets off the mat. Are output, and the dialog management unit 234 is notified of these outputs.
[0145]
In addition to the human state detection semantic expression from the human state detection unit 233, the dialogue management unit 234 also captures the input semantic expression from the voice understanding unit 232 in the same manner as in the above-described embodiment, and stores dialogue knowledge and dialogue history information. Used to generate output semantic representation for confirmation response.
[0146]
In this case, when the dialogue management unit 234 receives the input semantic representation from the speech understanding unit 232 and the human state detection semantic representation from the human state detection unit 233, each semantic representation is processed in order according to the state of dialogue, It is possible to proceed with the dialogue by understanding the user's state and various reactions.
[0147]
Thus, when the user gets on the mat, the human state detection semantic expression 1 is output from the human state detection unit 233 and sent to the dialogue management unit 234. Then, the dialogue management unit 234 sends the greeting output meaning expression 1 to the response generation output unit 235, and “please come and place an order” is output from the display 236 and the speaker 237 as the response output.
[0148]
Next, when the user inputs “two hamburgers and two coffees”, the input semantic expression 1 is output from the voice understanding unit 232 and sent to the dialogue management unit 234. As a result, the dialogue management unit 234 refers to the input semantic expression and the contents of the order table, outputs the output semantic expression 2, and outputs a response “Hamburger 2 to coffee 2” through the response generation output unit 235. Become so.
[0149]
In this case, usually, as shown in FIG. 44, the dialogue proceeds such as “Two hamburgers, two coffees,” “Yes,” “Thank you.” However, the user leaves the mat on the way. FIG. 45 shows a case where it has been closed.
[0150]
That is, after the response “output hamburger 2 and coffee 2” is output, the human state detection semantic expression 2 is output from the human state detection unit 233 and input to the dialogue management unit 234. Become so. In this case, since the user has left without confirming the utterance content, the dialog management unit 234 does not register the order content and sends a natural response “Thank you for using” in the output semantic expression 4. It comes to output.
[0151]
In this way, by combining the human state detection unit 233 with the dialogue management unit 234, it becomes possible to understand the user's state and reaction, and the dialogue can proceed naturally.
[0152]
In this embodiment, the mat is used for detecting the human state, but the present invention is not limited to this, and other methods such as a surveillance camera may be used.
[0153]
Next, such processing will be described with reference to the flowchart of FIG.
[0154]
In this case, the system has states # 0, # 1, # 2, and # 3 in the dialog management 234, and the initial state is the state # 0 (step S281). In the state # 0, it is confirmed whether the human state ACT of the human state detection meaning expression is “human presence” (step S282). If there is a person, the state is set to # 1 and a response is made by the output semantic expression of the greeting. It is generated and output (step S283).
[0155]
Next, in the state # 1, when the utterance ACT of the input meaning expression is an order from the voice understanding unit 232 (steps S284 and S285), the output meaning expression for confirming the order contents is sent based on the dialog knowledge and a response is output. To come. If the utterance ACT is yes (step S287), the state is set to # 2, and a response is output with an output semantic expression corresponding to the utterance act yes (step S288). If the utterance ACT is NO (step S289), an output meaning expression for reconfirmation of the order contents is transmitted and a response is output. Furthermore, when it is confirmed that the human state ACT of the human state detection meaning expression is “no person” (step S291), the state is set to # 3.
[0156]
In state # 2, “thank you” is output as thank you 1 (steps S293 and S294), and in state # 3, “thank you for using” is output as thank you (steps S293 and S294). S295, S296).
[0157]
Next, another embodiment of the present invention will be described with reference to FIG.
[0158]
This embodiment will be described in detail with respect to the voice input / output and the human state detection in the voice understanding unit 11 and the response generation / output unit 13 described in FIG.
[0159]
In this case, as shown in FIG. 47, the entire voice dialogue system includes a calculation unit 291, a memory unit 292, a storage unit 293, a storage unit interface 2931, a communication unit 294, a communication unit interface 2941, an A / D unit 295, a mat unit 296, An arithmetic processing unit 297, a D / A unit 298, and a display unit 299 are included.
[0160]
Here, the A / D unit 295 includes a microphone 2951, a filter amplification unit 2952, an A / D conversion unit 2953, and an A / D conversion unit interface 2954. The filter amplification unit 2952 has a high-frequency cutoff filter function for amplification and A / D conversion of the input from the microphone 2951. The cutoff frequency of the filter here is determined by the sampling frequency of the A / D conversion. However, for example, when sampling at 12 kHz, the high frequency component is cut off at 5.4 kHz. Also, the A / D conversion unit 2953 digitizes the amplified input sound, for example, at 16 kHz or 12 kHz, temporarily stores it in the A / D conversion unit interface 2954, and transfers it to the memory unit 292 under the control of the calculation unit 291. I am doing so.
[0161]
The mat unit 296 includes a mat 2961, a mat control unit 2962, and a mat control unit interface 2963. The mat control unit 2962 detects the presence / absence of a person on the mat 2961, and the result is transmitted through the mat control unit interface 2963. I try to forward it.
[0162]
The arithmetic processing unit 297 includes a high-speed arithmetic processing unit 2971 and a high-speed arithmetic processing unit interface 2972. The high-speed calculation processing unit 2971 is used for processing necessary for a large amount of calculation such as voice understanding processing, response generation processing, and human state detection processing by image processing. In this case, such processing needs to be performed simultaneously, and a plurality of high-speed arithmetic processing units 2971 can be used simultaneously. In each calculation process, the input data is transferred from the memory unit 292 to the high-speed calculation processing unit 2971 under the control of the calculation unit 291, and the result is transferred to the memory unit 292 after the processing is completed.
[0163]
The D / A unit 298 includes a D / A conversion unit interface 2981, a D / A conversion unit 2982, a filter amplification unit 2983, and a speaker 2984. The digital data stored in the memory unit 292 is converted to D under the control of the calculation unit 291. The data is transferred to the D / A conversion unit 2982 through the / A conversion unit interface 2981, converted to analog data at a constant period, for example, 12 kHz, and output to the speaker 2984 through the filter amplification unit 2983. In this case, the D / A conversion unit 2982 has a data temporary storage unit, and the arithmetic unit 291 can also perform other processes by performing data transfer from the memory unit 292 at high speed.
[0164]
The display unit 299 includes a display control unit interface 2991, a display control unit 2992, and a display 2993, and displays information such as changes in images, characters, graphics, moving image information, color and brightness, and density information under the control of the calculation unit 291. The information is displayed on the display 2993 by the control unit 2992.
[0165]
The communication unit 294 communicates control information data with external computers, information processors, service processing devices, and the like, and each data is exchanged through the communication unit interface 2941 under the control of the arithmetic unit 291.
[0166]
The storage unit 293 stores data, control information, programs, intermediate information, and the like necessary for voice understanding, dialogue management, and response generation under the control of the calculation unit 291.
[0167]
The calculation unit 291 uses information of each unit stored in the memory unit 292, an execution program, and a program therefor to use the A / D unit 295, the mat unit 296, the calculation processing unit 297, the D / A unit 298, the communication unit 294, The storage unit 293 is controlled.
[0168]
Here, the program executed by the arithmetic unit 291 is processed in the multitask format by performing processing in the voice understanding unit 11, the dialogue management unit 12, and the response generation output unit 13 described in FIG. For this purpose, task switching is performed sequentially at regular intervals, but when it is necessary to prioritize processing, such as when processing of each unit or input / output is completed, the processing is prioritized by interruption.
[0169]
In the above description, the A / D unit 295 and the D / A unit 298 can be individually operated. Thus, since voice input and synthesized sound output can be handled simultaneously and separately, voice input is possible even during synthesized sound output, and detection and recognition of the input voice can be performed by the synthesized sound cancellation.
[0170]
However, as the configuration of the A / D unit 295 and the D / A unit 298, a common A / D and D / A conversion unit interface 301 may be used as shown in FIG. ), A common A / D, D / A conversion unit interface 302, A / D, D / A conversion filter unit 303, and amplification unit 304 may be used.
[0171]
However, in such a configuration, data cannot be exchanged in both directions at the same time, and is limited to either input or output. Therefore, it is possible to output a synthesized sound simultaneously with the reception of the voice input during the synthesized sound output. Can not.
[0172]
In this case, when there is no means for knowing the voice input acceptance state, the user waits for a response for an unaccepted utterance or the first half of the utterance is not input. Therefore, by displaying the voice input / output permission status as an image, the computer side can inform the user of the voice input / output permission status. In particular, by combining the image display and the character display, for example, as shown in FIG. 49, it is possible to speak by “Clips” and “SPEAK”, and by the “Clips” and “LISTEN” sealed as shown in FIG. Each state that cannot be spoken can be displayed.
[0173]
In this way, the status and status of each input / output device can be communicated by another output device, enabling a more natural and easy-to-understand conversation. Furthermore, not only the status of each input / output device, but also the image display and text display when there is a need for the user to listen carefully because it is important, or when voice input is not desired in dialog management. Attention can be urged by the combination and further changes in color, brightness and density.
[0174]
The present invention is not limited to the above-described embodiments, and can be appropriately modified and implemented without changing the gist.
[0175]
【The invention's effect】
As described above, according to the present invention, when outputting a response from the system side to the user, in addition to the voice response, the person (or his face) has a sense of familiarity with the display system and at the same time is synchronized with the voice response. The mouth is moved, the attention point of the user is determined, and the usability is improved. In addition, the voice response sentence is also displayed in the form of text data on the same screen to cover the low quality of the voice response, and the user can receive the response sentence faster than the speech rate of voice. In addition, the response contents are visualized on the same screen, and the meaning and contents of important messages that should be transmitted in a form suitable for various applications are synchronized with the person-displayed voice response and voice response text. By outputting in this way, it is possible to realize high-speed message transmission from the system side to the user as seen by the user.
[0176]
As described above, since various types of responses are output from the system side to the user in parallel on the same screen, the user can select appropriate individual responses according to the situation, or two or three types of modes. Response data can be received at the same time, and the effect of utilizing the characteristics of each medium can be obtained, the degree of freedom for the user is increased, and an easy-to-use multimodal human interface can be realized.
[0177]
As a result, it is possible to quickly and efficiently cover the error confirmation and incompleteness based on ambiguity of the voice confirmation unit of the voice dialogue system, which was a problem in the past, and the user's intention by the progress of the dialogue. Easy to understand.
[0178]
In addition, there is a human condition detection means such as a mat or a camera on the input side, so that it can not only make a synthesized sound but also make the person's face on the display screen brighter and smile as well as detecting the user. In addition to improving the timing, it is possible to realize a user-friendly voice interface that can be used with peace of mind without being surprised by the user. Furthermore, when applied to a multimodal speech dialogue system, a user-friendly natural system can be obtained and the operability of the computer can be significantly improved.
[0179]
In addition, by adding a voice response cancel function, the user can view the screen display result even during voice response, and can always interrupt (Interrupt), enabling speedy voice interaction and voice recognition performance. Even if it is low, it can be covered with dialogue exchange, and the efficiency of intent transmission and data input can be greatly improved.
[0180]
In summary, in the present invention, in the speech dialogue system having speech recognition, speech synthesis, and dialogue management functions, the response content is visualized in parallel with the speech response that is time-series information when the system responds to the user. In particular, users can respond from various points of view at the same time because they can display facial expressions and gestures, display compensatory items (objects) such as items, sizes and types, and output response sentences in parallel. You will be able to receive, increase the degree of freedom, select the correct information as needed, and it is effective in improving familiarity, efficiency, comfort, reducing eye and ear fatigue, etc. .
[Brief description of the drawings]
FIG. 1 is a diagram showing a schematic configuration of an embodiment of the present invention.
FIG. 2 is a diagram showing a detailed configuration of a voice understanding unit.
FIG. 3 is a diagram for explaining a keyword candidate series.
FIG. 4 is a diagram illustrating an example of an input semantic expression.
FIG. 5 is a view showing the content of a keyword.
FIG. 6 is a diagram showing an example of output semantic expression.
FIG. 7 is a diagram showing an example of an order table.
FIG. 8 is a diagram showing an example of an old order table.
FIG. 9 is a diagram showing an example of state transition in the dialogue management unit.
FIG. 10 is a flowchart for explaining user state processing;
FIG. 11 is a flowchart for explaining system state processing;
FIG. 12 is a diagram showing a specific example of dialogue processing.
13 is a diagram for explaining user state processing in the dialogue processing shown in FIG. 12; FIG.
FIG. 14 is a diagram for explaining system state processing in dialogue processing;
15 is a diagram showing a specific example of the dialogue processing in FIG.
FIG. 16 is a diagram showing an example of an output response sentence from the system.
FIG. 17 is a diagram for explaining how to obtain a user's voice rate;
FIG. 18 is a diagram illustrating an example of an output of a dialogue management unit.
FIG. 19 is a diagram showing a detailed configuration of a response generation output unit.
FIG. 20 is a diagram showing an example of person image information.
FIG. 21 is a diagram showing an example of a response sentence pattern.
FIG. 22 is a flowchart and a specific example of a response sentence generation in a response sentence generation unit.
FIG. 23 is a diagram illustrating an example of a human figure expression determination unit.
FIG. 24 is a diagram showing an example of an emotion / emphasis determination unit.
FIG. 25 is a diagram showing a detailed configuration of a voice response generation unit.
FIG. 26 is a diagram showing an example of a basic frequency pattern model.
FIG. 27 is a diagram showing a change in the fundamental frequency pattern in the case of a response with pleasure.
FIG. 28 is a diagram showing a change in the basic frequency pattern in the case of a response accompanied by sadness.
FIG. 29 is a diagram showing an example of a specific configuration of a speech waveform generation unit.
FIG. 30 is a timing chart showing an example of time control of the output presentation order;
FIG. 31 is a timing chart showing another example of time control of the output presentation order.
FIG. 32 is a timing chart showing another example of time control of the output presentation order.
FIG. 33 is a timing chart showing another example of time control of the output presentation order.
FIG. 34 is a diagram showing a display example on the display screen.
FIG. 35 is a diagram showing a display example on the display screen.
FIG. 36 is a diagram showing a display example on the display screen.
FIG. 37 is a diagram showing a display example on the display screen.
FIG. 38 is a diagram showing a display example on the display screen.
FIG. 39 is a diagram showing a display example on the display screen.
FIG. 40 is a diagram showing a display example on the display screen.
FIG. 41 is a diagram briefly showing the procedure of dialogue processing.
FIG. 42 is a diagram showing a schematic configuration of another embodiment of the present invention.
FIG. 43 is a diagram for explaining a human state detection unit;
44 is a view for explaining the operation of the other embodiment shown in FIG. 42;
45 is a view for explaining the operation of the other embodiment shown in FIG. 42;
FIG. 46 is a flowchart for explaining the operation of the other embodiment shown in FIG. 42;
FIG. 47 is a diagram showing a schematic configuration of another embodiment of the present invention.
48 is a diagram showing an example in which a part of the other embodiment shown in FIG. 47 is modified.
FIG. 49 is a diagram showing a display example on the display screen.
FIG. 50 is a diagram showing a display example on the display screen.
[Explanation of symbols]
11,232 Speech Understanding Department
12,234 Dialogue management department
13,235 Response generator
14,236 display
15,237 Speaker
21 Keyword detector
21a Speech analysis unit
21b Keyword spotting processor
22 Syntax and semantic analysis part
22a sentence start end determination unit
22b sentence candidate analysis part
22c sentence end judgment part
22d sentence candidate table
231 Voice input unit
233 Human condition detector
291 Calculation unit
292 Memory part
293 Storage Department
294 Communication Department
295 A / D section
296 mat part
297 Arithmetic processing part
298 D / A section
299 display

Claims (2)

Speech understanding means for receiving speech input, understanding the meaning content of the input speech, and scoring the degree of likelihood of the meaning content for each of the understood meaning content;
Dialog management means for selecting an input semantic expression having the highest degree of certainty among the input semantic expressions and generating an output semantic expression according to the semantic content of the selected input semantic expression and the score;
Response output means for outputting a system response output according to the output semantic expression,
The dialogue management means controls a state transition between a user state in which a voice input is given to the voice understanding means and a system state in which a system response output is outputted from the response output means. Spoken dialogue system characterized by managing A speech understanding step in which speech input is given, the meaning content of the input speech is understood, and each of the understood meaning content is scored with respect to the degree of likelihood of the meaning content;
A dialog management step of selecting an input semantic expression having the highest degree of certainty among the input semantic expressions and generating an output semantic expression according to the semantic content of the selected input semantic expression and the score;
A response output step of outputting a system response output according to the output semantic expression,
The dialog management step controls a state transition between a user state in which a voice input is given to the voice understanding step and a system state in which a system response output is output from the response output step. A voice dialogue method characterized by managing dialogue.

Images