JP2001142484A - Method for voice conversation and system therefor - Google Patents

Abstract

PROBLEM TO BE SOLVED: To jointly use a screen display for voice response as response output from the system side to a user in the case of a voice conversation between the user and the system. SOLUTION: The system is provided with an understanding means 11 which is given an input and understands semantic contents of the input, a conversation management means 12 which makes semantic decision of response contents on the basis of the understood result of the understanding means, a response generation means 13 which generates a voice response output and a picture display output on the basis of response contents decided by the conversation management means, and output means 14 and 15 which output the voice response output and the picture display output generated by the response generation means.

Description

DETAILED DESCRIPTION OF THE INVENTION

[0001]

BACKGROUND OF THE INVENTION 1. Field of the Invention The present invention relates to a speech dialogue system using various input / output means including speech recognition and speech synthesis.

[0002]

2. Description of the Related Art In recent years, by inputting, outputting, and processing multimedia such as characters, sounds, figures, and images,
Human-Computer Interac
) can be performed in various forms.

In particular, workstations and personal computers capable of handling multimedia have recently been developed due to the dramatic improvements in memory capacity and computer power, and various applications have been developed. However, it is merely a matter of taking various media in and out, and has not yet reached the point where organically fuses various media.

On the other hand, linguistic data including characters has become commonplace in place of conventional numerical data, and black-and-white image data has been expanded to handle colorization, graphics, animation, three-dimensional graphics, and even moving images. Is coming. As for voices and audio signals, voice recognition and voice synthesis functions are being researched and developed in addition to mere input / output of voice signal levels. At present, it is limited to limited fields.

That is, as described above, with respect to characters, texts, voices, graphic data, and the like, the conventional input / output processing (recording / reproducing) has been developed to understand and generate various media. In other words, from the surface processing of each media, it handles the contents, structure, and semantic contents of the media, and understands and generates media such as voice and graphics for the purpose of making human-computer conversations more natural and comfortable. The construction of a dialogue system that uses is being studied.

[0006] However, speech recognition has evolved from isolated word recognition to continuous word recognition and continuous speech recognition.
d) Development is underway. In such application scenes, it is more important for speech dialogue systems to understand the utterance content of speech than to recognize the character surface of speech.For example, a speech understanding system that uses knowledge of the application field based on keyword spotting is also used. Has been studied. On the other hand, conventional speech-to-speech conversion (text-to-speec
h) Research on a speech synthesis system for dialogues that emphasizes intonation from the system has been conducted by, for example, the present inventors, and applications to speech dialogues are expected.

[0007] However, understanding and generating media such as audio are different from mere data input / output, and in the case of media conversion, information loss and errors are inevitable. That is, speech understanding is a process of extracting the content of speech utterance and the intention of the speaker from speech pattern data having a large amount of information, and speech recognition errors and ambiguity occur in the process of compressing information. Therefore, in order to cope with incomplete speech recognition such as the recognition error and ambiguity described above, it is necessary for the spoken dialogue system to appropriately ask and confirm the user from the system side and smoothly proceed the dialogue by the dialogue control. .

[0008] In the case where the dialogue system performs some type of dialogue with the user, it is important to cover imperfect speech recognition and to accurately convey the status of the computer as an easy-to-use human interface. However, in a conventional voice interaction system, text synthesis for simply reading a sentence as a voice response is often performed as a voice response. Or, there is no voice response, and all responses from the computer display a response sentence on the screen as text, or a system that displays graphic data, video, icons and numerical values, which is a heavy burden on vision. Had become.

As described above, in recent years, various interactive systems described above have been developed. However, studies on the use of various media in responses from the system side in order to deal with incomplete speech recognition have been conducted. This has not been done enough, and has become a major problem in speech recognition technology. In other words, speech recognition is unstable, weak against noise and unnecessary words, and it is difficult to convey the user's intention efficiently by voice. Application was limited to strong scenes.

[0010]

As described above, in the conventional speech dialogue system utilizing the speech recognition and synthesis techniques, the speech recognition, speech synthesis and screen display techniques developed separately are simply combined. And it has not been fully considered from the perspective of speech dialogue. In other words, the speech recognition function has recognition errors and ambiguity, the speech synthesis function is less intelligible than human utterance, and the ability to convey intentions and emotions is insufficient due to insufficient control of intonation, There is a fundamental problem of lack of naturalness. In addition, it is not enough with the current technology to generate an appropriate response using the speech recognition result on the system side. On the other hand, the transmission ability can be expected to be improved by displaying the image in combination with the voice, but the two-dimensional and three-dimensional spatial screen display can be used for the instantaneous continuous and time-series voice response. It is an unsolved problem how to utilize the two and control the timing of both. Another important issue is what to display as a spoken dialogue system using other media.

SUMMARY OF THE INVENTION The present invention has been made in view of the above circumstances, and provides a voice dialogue system capable of efficiently and accurately performing voice dialogue between a system and a user, and enabling a remarkable improvement in usability. With the goal.

[0012]

SUMMARY OF THE INVENTION According to the present invention, there is provided a voice comprehension means for receiving a voice input and comprehending the meaning content of the inputted voice, and a semantic determination of a response content based on an understanding result of the voice comprehension means. Generating means for generating a voice response output and a screen display output based on the response content determined by the dialog management means, and outputting a voice response output and a screen display output generated by the response generating means Means.

The dialogue management means is associated with person image information relating to a person image of a speaker who makes a voice response based on the understanding result of the voice understanding means, response content text information of a voice sentence corresponding to the voice response, and content of the voice response. Visualization information for visualizing the understanding content is output as response content.

The response generating means generates at least one of a motion and a facial expression of a human image on the screen based on the human image information of the speaker making the voice response output from the dialog managing means.

The response generating means generates at least one screen display output of the motion and expression of the person image based on the person image information of the speaker performing the voice response output from the dialog management means, and outputs the image to each screen display. A voice response output having a corresponding voice emotion or strength is generated.

Further, there is provided a human state detecting means for detecting a human state relating to the movement of the person, and the dialog management means makes a semantic determination of the response content based on the detection result of the human state detecting means. I have.

An icon indicating whether or not voice input is possible can be displayed.

Further, the present invention provides a voice comprehension means for receiving a voice input and understanding the meaning of the input voice, and a response output means for outputting a system response output based on the result of the understanding by the voice comprehension means. And managing the interaction between the system and the user by controlling a state transition between a user state in which a voice input is given to the voice understanding means and a system state in which a system response output is output from the response output means. And a dialogue management means.

Further, the present invention provides a speech understanding means which is provided with a speech input and understands the meaning of the inputted speech by detecting a keyword in the speech input, according to a state of a dialog between the system and the user. A dialogue management means for preliminarily restricting a keyword in a voice input detected by the voice understanding means, and a response output means for outputting a system response output based on a result of the understanding by the voice understanding means. It is characterized by.

[0020]

As a result, according to the present invention, when a dialogue between the user and the system is performed, in addition to the voice recognition and the voice response, the screen display of the response is used as the response output from the system to the user. Become. At this time, by displaying a person image corresponding to the speaker of the voice from the system side, the image of the speaker becomes representative of the function of the system,
The user can naturally speak toward the person on the screen, and can grasp the progress of the dialogue and the reliability of speech recognition based on the movement and expression of the mouth of the person on the screen.

On the other hand, regarding the contents of the response from the system, a character string representing the response sentence is displayed, and the object (for example, a product or a concept such as a product or a concept) or the number is displayed by a graphic or the like. The response content can be quickly transmitted to the user. Furthermore, speech recognition is an incomplete one in which erroneous recognition and ambiguity frequently occur. When understanding the contents of speech, erroneous information not intended by the user may be transmitted to the computer side. In addition, by using visual response displays in parallel, it is possible to greatly improve the efficiency of voice dialogue, and to improve naturalness and usability. Also, since the synthesized speech is lower than natural speech,
Use of the visualization of the speaker's expression, response sentence, and response content is extremely useful for improving dialogue.

[0022]

An embodiment of the present invention will be described below with reference to the drawings.

FIG. 1 shows a schematic configuration of a system including a screen display as a voice interactive system.

The speech dialogue system includes a speech understanding unit 11 for understanding the meaning of the input speech, a dialogue management unit 12 for making a semantic determination of the response content based on the result of the understanding by the speech understanding unit 11, and a dialogue management. A response generation output unit 13 that generates a voice response output and a screen display output based on the response content determined by the unit 12, a screen display output unit 14 that outputs the screen display generated by the response generation output unit 13, and a voice response. It comprises an audio output unit 15 for outputting.

The voice comprehension section 11 recognizes the character surface of the voice,
That is, instead of recognizing a word or a sentence, the input voice spoken by the user is understood and the meaning is extracted. Then, an input semantic expression representing the understood semantic content is generated, and the dialog managing unit 12
Send to

The dialog management unit 12 determines the meaning of the response to the input semantic expression of the input voice by using the information on the history of the dialog and the current state of the dialog, the method of proceeding the dialog, and the knowledge of the application field. Then, the response content information of the utterance corresponding to the voice response is output to the response generation output unit 13.

Further, the dialogue management unit 12 processes spoken words including omissions and demonstrative pronouns, thereby improving speech comprehension performance and reducing the amount of processing, as well as enabling a natural dialogue. Further, the dialogue management unit 12 outputs, to the response generation output unit 13, person image information of a speaker which is displayed and output on the display 14 and makes a voice response, and visualization information which is information for visualizing understanding contents related to the content of the voice response. Output to

The output semantic expression generated by the dialog managing unit 12 is sent to the speech comprehension unit 11 to narrow down keywords and syntactic semantic rules of the next utterance from the output semantic expression, and to evaluate the speech understanding performance of the next utterance. It is possible to improve.

The response generation and output unit 13 outputs a response sentence generated based on the response content information input from the dialog management unit 12 from the speaker 15 as a synthesized voice, and performs an operation and a response based on the human image information and the response sentence. The image of the person making the voice response with the determined facial expression is visually displayed on the display 14, and is generated based on the visualization information, which is information for visualizing the contents understood by the system through the previous conversation in an easy-to-understand manner. The content visualization information is visually displayed on the display 14, and a response is presented to the user in a multimodal manner using a plurality of media. That is, by presenting the audio information and the visual information to the user in combination, the human interface of the voice interaction system is improved, and a natural interaction becomes possible.

Further, information indicating that a response is currently being output from the response generation unit 13 is sent to the dialog management unit 12. The dialogue management unit 12 sends the above information to the speech understanding unit 11 and can improve the speech understanding performance by controlling, for example, the timing of the input / output end detection process and the keyword detection process.

Next, each part of the above-described voice dialogue system will be described in more detail, assuming an order task in fast food as an application.

First, the speech understanding unit 11 will be described. As described above, the role of the voice comprehension unit 11 is to understand the semantic content of the voice and the intention and situation of the speaker, instead of recognizing the character surface as in text input or a voice word processor. It is an object.

In this case, a ticket vending machine for an unspecified user, a seat reservation system for an airplane or a train, an automatic cash dispenser at a bank, and the like may have different voices among speakers, differences in unnecessary words and spoken language, Due to the influence of noise and the like, sufficient recognition performance may not be expected even when speech recognition technology is actually applied, and it is extremely difficult to perform highly accurate recognition of a sentence in particular. In this regard, a method of first analyzing candidate strings of keywords from continuously uttered speech to understand the utterance content is described in, for example, the literature (Hiroyuki Tsuboi, Hideki Hashimoto, Yoichi Takebayashi: "Analysis of keyword lattice for understanding continuous speech") Proceedings of the Acoustical Society of Japan, 1-5-11, pp. 21-22, 199
1-10), and using this method, in a limited application, there is no limit on the user's utterance as much as possible.
You will be able to understand free utterances at high speed.

FIG. 2 shows a schematic configuration of the speech understanding unit 11 using the above-described keyword.

In this case, the voice comprehension unit 11 comprises a keyword detection unit 21 and a syntactic and semantic analysis unit 22. The keyword detecting unit 21 includes a voice analyzing unit 21a and a keyword spotting processing unit 21b, and the syntactic and semantic analyzing unit 22 includes a sentence start end determining unit 22a, a sentence candidate analyzing unit 22b, a sentence end determining unit 22c, and a sentence candidate table 22d. It consists of.

In the keyword detecting section 21, the voice analyzing section 2
1a, the input voice is passed through a low-pass filter (LPF), and is sampled at a sampling frequency of 12 kHz and a quantization bit of 12 bits.
The digital signal is converted to a digital signal by D / D conversion, and then the spectrum is analyzed, and further, the smoothing in the frequency domain is performed after using the FFT, and the logarithmic conversion is further performed to obtain 16
A voice analysis result is output from the channel band pass filter (BPF) every 8 ms, and the keyword spotting process is performed on the output. In this case, the keyword spotting process is performed, for example, according to the literature (Kanazawa, Tsuboi,
Takebayashi: "Detection of words from continuous speech including unnecessary words", IEICE Spoken Language Study Group, sp.
33-39, 1991-6).

In this way, the keyword detecting section 21 determines a keyword candidate sequence (lattice) from continuous input speech.
Will be extracted. Fig. 3 shows an example of keyword candidate sequences extracted from continuous input speech "Please have three hamburgers, potatoes and coffee" when applying the exchange of orders at a fast food restaurant at the store by voice dialogue. Is shown.

The above-described speech analysis and keyword detection processing are described in other documents (“Development of a speech recognition system using a high-speed DSP board”, Proceedings of the Acoustical Society of Japan, 3-5-1.
Real-time processing is also possible by using a DSP board as described in 2, 1991-3).

Next, the keyword candidate sequence detected in this manner is subjected to syntactic and semantic analysis by the syntactic and semantic analysis unit 22 to obtain an input semantic expression for a voice input as shown in FIG.

Here, the application is limited to the fast food task, and the input semantic expression in the frame format is composed of an ACT frame indicating the type of order processing of the input utterance and an order item frame indicating the contents of the order. And AC
In the T frame, semantic information about the processing related to the order such as “order”, “add”, “delete”, and “replace” is expressed. On the other hand, the order frame includes the slot of the product name, size, and number. The contents of the order can be expressed.

That is, the keyword lattice obtained by the keyword detecting section 21 is sent to the syntactic and semantic analyzing section 22. The syntactic and semantic analysis unit 22 includes a sentence end determination unit 22a, a sentence candidate processing unit 22b, and a sentence end determination unit 22c, and has a sentence candidate table 22d. The syntactic and semantic analysis unit 22 processes each word in the keyword lattice from left to right.

The sentence start determination unit 22a determines whether or not the currently processed word can be the start of a sentence based on syntactic and semantic constraints. If it can be the beginning of a sentence,
The word is set as a new sub-sentence candidate and sentence candidate table 2
Register in 2d.

The sentence candidate analyzing unit 22b determines whether or not the word and each partial sentence candidate in the sentence candidate table 22d can be connected from the syntactic, semantic, and temporal constraints. If it can be connected, copy the partial sentence candidate, connect the input word to it, and connect it to the sentence candidate table 2
Register in 2d.

The sentence end determining unit 22c determines whether or not the partial sentence candidate processed immediately before by the sentence candidate analyzing unit 22b can be syntactically formed as a sentence. Is output as an output of the syntactic and semantic analysis unit 22.

The output sentence candidate is subjected to semantic analysis at the same time as syntactic analysis, and therefore, this directly means the input semantic expression. The above processing is performed on the input in a pipeline manner. Thus, a plurality of input meaning expressions for the input voice can be obtained.

In the fast food task in this case, keywords shown in FIG. 5 are used. However, depending on the situation of the dialog, another utterance may have the same meaning. That is, in speech understanding based on keywords, "one" and "one" have the same semantic expression, and "please" and "please" may have the same semantic expression. Is different from the expression of the input voice. This is the difference between speech recognition and speech understanding. In the speech dialogue system handled by the present invention, task-dependent speech understanding processing using knowledge of the application field is required.

Next, the dialog managing unit 12 will be described. In the speech dialogue system according to the present invention, as shown in FIG. 1, the input semantic expression output from the speech understanding unit 11 is sent to the dialogue management unit 12, and the knowledge of the dialogue, the knowledge of the application field, the information of the history of the dialogue and the state of the dialogue are displayed. Is used to make a semantic determination of the response content, to generate an output semantic expression as response content information for an acknowledgment, and to output it to the response generation output unit 13. Note that, as shown in FIG. 6, the output semantic expression uses a frame format expression like the input semantic expression.

In this embodiment, the input meaning expression shown in FIG. 4 is used as the content expression of one utterance of the input voice.
In addition, as shown in FIG. 7, an order table for storing the contents of the order up to that point is separately prepared as the contents understood by the system from the start of the dialogue. Further, a change in the order table accompanying the progress of the dialog is prepared in the form of an order table (old order table) at the time of answering the previous question as shown in the example of FIG. Further, the dialog management unit 12 holds dialog status information indicating the status of the dialog. This dialog status information includes information such as the current state of the dialog, the next transition state, the number of times the dialog is repeated, the degree of certainty, the emphasis items, the history of the dialog, and the like.
3 is used as person image information.

The order table is rewritten based on the ACT information of the input semantic expression and the order contents, and has a format of only the order contents obtained by removing the ACT information from the input semantic expression. In other words, this order table reflects the contents understood in the dialogue from the start of the dialogue. The old order table has the same configuration as the order table, holds the order table in response to the question at the time of the immediately preceding conversation, and records the state of the order table as history information of the conversation.

As described above, the dialogue management unit 12 obtains knowledge of the dialogue progress method and application fields based on the meaning expression of the input voice (input meaning expression), the history information of the dialogue (old order table), and the state of the dialogue system. To generate response content information (output meaning expression) representing the content of the response output. That is,
Referring to the input semantic expression and the order table, a process depending on the state (state number) of the system at that time is performed, and an output semantic expression expressing the contents of the response generation and the contents of the response generation including the response ACT is generated. I have to. As described above, the output semantic expression in this case uses a frame format expression, like the input semantic expression. Further, based on the history information of the dialogue (old order table) and the state of the dialogue system, dialogue status information is generated so that the response generation / output unit 13 can refer to the person image information on the response screen display.

FIG. 9 shows an example of a state transition inside the dialog management unit.

In this example, the dialogue is managed and progressed by expressing the state transition based on the method of proceeding the dialogue and the knowledge of the application field. The dialogue management unit 12 largely stores the user 72
And system 71 respectively.

Here, the role of the state of the user 72 is to make a transition to the state of the system 71 in accordance with the input meaning expression of the utterance of the user, while the role of the state of the system 71 is to Accordingly, the contents of the order table are changed, the output meaning expression of the response is output, the flow of the dialogue is advanced, and the state of the user 72 is changed. In this way, by having the internal state of the system in two parts, various exchanges between the user and the system can be expressed,
Flexible dialogue can be performed.

The dialog status information used at this time indicates the name of the status of the dialog management being processed and the number of times the partial dialog is repeated. The status name of the ongoing dialog, the name of the next transition state,
The number of times a partial dialogue such as repeating the same question on the same topic is repeated is recorded sequentially and can be easily referred to. It is used as personal image information such as the expression, movement, and emotion and emphasis of voice response of the human image, and is used for response generation output.

In FIG. 9, when the presence of a user (customer) is detected by the dialogue management unit 12, the system 71
The dialogue starts from the initial state S0, and generates an output semantic expression related to a greeting and an order request.
To the initial state U0 of the user 72. In addition, the conversation history information (old order table) is initialized,
Dialog status information is generated based on the transition of the system state from the state S0 to the state U0. This conversation situation information,
The response generation output unit 13 can refer to the image information of the person on the response screen display.

Then, the response generation unit 13 generates a voice response, a human image, a text, and content visualization information while referring to the system state, the history information of the conversation, and the order table based on the output semantic expression.

At this time, in the initial state U0 of the user 72,
When the input semantic expression ACT information of the next utterance is “order”, a transition is made between the dialog progress state SP of the system 71 and the user dialog progress state UP in a general order flow.

On the other hand, if the ACT information of the input semantic expression is other than an order, the utterance of the user is regarded as an unexpected user's utterance, and the dialog correction state S
Transition is made to 10.

If the dialogue modification state S10 of the system 71
In the case of the transition to, the system 71 uses the input semantic expression, the order table and its history information to determine whether the voice input received from the user 72 has unexpected content or did not hear well according to the situation. An appropriate response is sent to the user 72 or an output semantic expression for confirming the details of the order one item at a time is output.
The state transitions to P. Then, the response generation unit 13 generates a voice response, a person image, a text, and content visualization information while referring to the system state, the conversation history information, and the order table based on the output semantic expression.

Thus, the system 71 and the user 72
And the state of the user 72 is changed and the utterance of the user 72 and the response of the system 71 are performed.
2 completes the desired order, ie system 7
When the user 72 utters an affirmative response to the acknowledgment of all the orders, the process shifts to the end state S9 of the system 71 and ends the dialog.

FIG. 10 shows a flowchart of the processing in the state of the user.

In this case, a plurality of input semantic expressions are first read (step S81), abbreviations are inferred (step S82), and a score (scoring) relating to the certainty of each input semantic expression is performed (step S83). ).
Next, an input semantic expression having the highest score is selected from the input semantic expression candidates (step S84), an utterance act is determined (step S85), and a transition is made to a system state based on the utterance act (step S85). S8
6).

On the other hand, FIG. 11 shows a flowchart of the processing of the state of the system.

In this case, first, the contents of the order table are changed based on the input semantic expression (step S91), an output semantic expression is generated in consideration of the state of the system at that time (step S92), and the response content is output. (Step S93), and the state is changed to the user state (step S94). The response generator 13 generates a voice response, a human image, a text, and content visualization information based on the generated output semantic expression.

As described above, in the system according to the present embodiment, when a message is received from the other party separately for the user and the system, it is possible to perform processing in consideration of various knowledges, situations, and message contents. Flexible and likely processing is possible.

FIG. 12 shows a specific example of the interactive processing in the system of this embodiment.

In this case, if it is assumed that the order table shown in FIG. 12B and the output meaning expression shown in FIG. , As shown in FIG. 12 (c), a confirmation message "Order is one hamburger, two coffees, and four colas" is generated, and a voice response, a confirmation text, and an order based on the message are generated. The picture and the number of items on the table are presented to the user in a multimodal manner using audio media and visual media.

On the other hand, assuming that the user voice-inputs "Please add one cola." As shown in FIG. 12C, the voice understanding unit 11 shown in FIG. After the keyword candidates have been detected, the analysis (parsing) of the keyword candidate series (keyword lattice) is performed, and FIG. 12D and FIG.
An input meaning expression candidate 1 and an input meaning expression candidate 2 shown in FIG.

Each candidate here has a score (score) D regarding the likelihood, the input semantic expression candidate 1 has a score of D1, and the input semantic expression candidate 2 has a score of D2. .

In this case, in the input meaning expression candidate 1 of the score D1, the ACT information is added, the product name is Cola, the size is indefinite, and the number is 1. In the input meaning expression candidate 2 of the score D2, the ACT information is added. Is deleted, the product name is potato, the size is large, and the number is 2.

The processing in the state of the user is executed as shown in FIG.

First, the input meaning expression candidate 1 is omitted by referring to the fact that the size of the cola in the previous output semantic expression was large, and inferring that the size of the additional order of the cola this time is also large. The expression is supplemented (step S111).
This inference is not performed for the input semantic expression candidate 2 because there is no particular omission (step S113).

Next, the validity is checked. That is, the contents of the order table are compared with the input meaning expression candidates to check for inconsistency (step S112, step 114).

In this example, as for the input semantic expression candidate 2, although the utterance ACT of the input semantic expression is "delete" and the product name is large, there is no potato in the order table, so the score D2 is D2 '= D2. A process for reducing the size is performed by the process of × α (α <1.0).

Next, the scores of the input meaning expression candidates are compared (step S115). In this case, since D2 '<D1', candidate 1 is selected. On the other hand, the utterance ACT of the input from the user is determined to be “added” (step S11).
6), the state transits to the state SA of the system for performing the addition confirmation (step S117). Here, the state SA of the system that performs the addition confirmation rewrites the order table so as to add one large cola based on the input semantic expression.

In this case, the processing of the state in the system is as shown in FIG.
This is performed as shown in FIG.

That is, in this state, the system selects an acknowledgment to the user from the response ACT list shown in FIG. 15B and determines an output meaning expression. In this example, since the ACT of the input semantic expression is additional, the fourth additional confirmation is selected as the response ACT, and a response output (response sentence) is determined and output using these information.

First, one cola size is added to the order table (step S121). And here, FIG.
Since the utterance ACT of the input meaning expression shown in (a) is added,
The response ACT is selected as an additional confirmation, and the output semantic expression shown in FIG. 15C is obtained from the information (step S122). Next, a response output (response sentence) is determined and output (step S123). The response sentence in this case is determined based on an example of the response expression of the system shown in FIG. 16, and is output as, for example, "I confirm. One large cola is added." Then, step S124
To the state U of the user who has received the response ACT of the additional confirmation.
The state transits to A, and processing in the state of the user is performed.

The dialog management unit 12 sends the user's utterance speed and the likelihood of each keyword received from the speech understanding unit 11 to the response generation output unit 13 together with the output semantic expression obtained as described above.

Here, the utterance speed of the user is obtained as follows. That is, the keyword spotting processing unit 21b of the speech understanding unit 11 in FIG. 2 detects the obtained keyword, its start and end, and the user's utterance speed obtained based on the keyword. Also, the likelihood of each keyword of the speech understanding result is output to the dialog management unit 12 together with the input meaning expression. The utterance speed of the user can be obtained, for example, as shown in FIG. 17 if the start and end of the word obtained from the keyword spotting processing unit 21b and the word are known. That is, when three keywords “hamburger”, “potato”, and “please” are detected from the user's utterance as the start end t1 and end t2, the start end t3 and end t4, the start end t5 and end t6, respectively, the mora number of these keywords Is 6,3,4, so the average utterance speed of the user is

１6 / (t2-t1) + 3 / (t4-t3) +
4 / (t6−t5)｝ / 3.

The utterance speed and likelihood of the user obtained in this way are input to the dialog management unit 12 together with the input meaning expression.

The dialog management unit 12 responds to the output semantic expression generated based on the processing described with reference to FIGS. 9 to 16 by outputting the utterance speed of the user input from the speech understanding unit 11 and the likelihood of the keyword. Input to the unit 13. FIG. 18 shows an example of the output meaning expression at this time.

Next, the response generation / output unit 13 will be described. In the response generation / output unit 13 according to the present invention, the output semantic expression as response content information, the human image information including the dialog situation information and the dialog history information, and the visualization for easily visualizing the contents understood by the system through the previous dialog. Based on the information, a voice response, a person image, a text, and content visualization information are generated and output. Where voice response, person image,
The text is generated based on the output semantic expression and the human image information, with expressions, feelings, and emphasis so that the contents to be conveyed in consideration of the dialogue state can be presented in an easily understandable manner. Further, the content visualization information is for displaying the content of the dialogue understood by the system to make it easy to understand the progress of the dialogue, and is generated and output based on the visualization information output from the dialogue management unit 12. Things.

FIG. 19 shows an example of the configuration of the response generation / output unit 13. The response generation unit 13 includes a response sentence generation unit 131,
Person image expression determination unit 132, person image generation unit 133, emotion /
It comprises an emphasis determination unit 134, a voice response generation unit 135, a content visualization information generation unit 136, and an output integration control unit 137.

The response generation and output unit 13 receives the output semantic expression and the person image information from the dialog management unit 12 and
At step 31, a sentence responding and its structural information are generated. Based on the generated response sentence and the human image information from the dialog management unit 12,
The person image expression determination unit 132 determines the expression of the person image responding by voice on the screen display, generates the person image of the determined expression by the person image generation unit 133, and outputs the generated image to the output integration control unit 137. Further, based on the generated response sentence, sentence structure information, and human image information from the dialogue management unit 12, the emotional expression and the part to be emphasized of the voice response are determined by the emotion / emphasis determination unit 134,
A voice response having emotion and emphasis is generated by the voice response generation unit 135 and output to the output integrated control unit 137. Further, the generated response sentence is output as text information by the output integration control unit 137.
Output to In addition, in order to visualize and display the understanding content related to the response content, the response generation output unit 13 receives the visualization information output from the dialog management unit 12, and generates the content visualization information in the content visualization information generation unit 136. Output to the output integration control unit 137.

The output integrated control unit 137 receives a human image having an expression, a voice response having emotions and emphasis, text information, and content visualization information from each unit, and controls the temporal presentation order while controlling the screen display output unit 14. The response is output to the audio output unit 15 and the contents of the response are integrated and displayed to the user.

Next, the operation of each unit of the response generation / output unit 13 will be described with reference to FIG.

First, information processed by each unit of the response generation / output unit 13 will be described.

The output semantic expression has the same frame format as the input semantic expression as shown in FIG. 6, and the ACT information indicates an action in the response.

The person image information is information on the expression of the person image responding by voice displayed on the screen display output unit 14 and the emotion / emphasis of the voice response, and has a structure as shown in FIG. The system state number and the user state number indicate the numbers of the respective states when transitioning from the system state to the user state in the dialog processing of the dialog management unit 12 to generate the output semantic expression. SP1 and UP3 shown in FIG. 20 respectively indicate one state of the state set SP on the system side and one state of the state set SU on the user side in the dialog state transition of FIG. The number of repetitions is the number of times when the same question is partially repeated in a dialog or the same content is repeatedly confirmed. Emphasis items indicate items that need to be particularly confirmed in the output semantic expression. The certainty indicates the certainty of the corresponding content based on the output semantic expression, and is a score D obtained when the dialog management unit 12 interprets the input semantic expression based on the likelihood of the input semantic expression. The response sentence generation unit 131 generates a response sentence and its sentence structure from the output semantic expression generated by the dialog management unit 12.
There are generally known sentence generation methods using rewriting rules, filling with holes, methods for combining from tree structures, and methods for combining from semantic structures. Here, the method of filling with holes will be described.

For generating an output response sentence, for example, as shown in FIG. 21, a response sentence type with a hole for embedding an item, a size, and the number for each ACT information and its sentence structure are prepared, and FIG.
This can be realized by a method of filling a space based on the output semantic expression according to the flowchart shown in FIG. That is, first, in step S141, a variable n indicating the number of repetitions is set to 0.
The number of items of the output semantic expression is set in a variable M in step S142. In the case of the output meaning expression of FIG. 22B, M is 2. Next, in step 143, the name, size, and quantity of one item of the order are embedded in the response sentence.
Next, while adding the variable n repeatedly in step S144, the process is repeated until the embedding is completed in step S145. When the output semantic expression of FIG. 22 (b) is embedded in the response sentence pattern of FIG. 22 (c), as shown in FIG. 22 (d), "confirmation. One large cola and three small potatoes. . "
And the response sentence is obtained.

[0092] The person image expression determining unit 132 determines the expression of the person image from the sentence generated by the response sentence generating unit 131 and the person image information input from the dialog management unit 12. FIG. 23 shows an example of the person image expression determining unit 132. System state number,
The user state number, the number of repetitions, and the degree of certainty are obtained from the human image information, and express the human image and its expression for each value in the form of a table in advance. For example, if the degree of certainty is high when the number of repetitions is up to one, the confirmation is performed with a normal expression, and if the degree of certainty is low, the confirmation is performed with a confused expression.

The person image generation unit 133 generates an image to be displayed on the screen from the information on the person image and the expression output from the person image expression determination unit 132. At this time, control for changing the display time and the human image is performed. For example, when using a still image, a plurality of images are prepared, and when using a moving image, a continuous operation is performed so that a mouth operation when a human image responds by voice or a bowing operation when greeting is generated. Is specified. When using computer graphics, an image of a specified operation is generated.

The emotion / emphasis determination section 134 determines the emphasis or emotion of the voice to be responded from the person image information. FIG. 24 shows an example of the emotion / emphasis determination unit 134. Human image expression determination unit 1
32, in which the emotions of the human image and the voice response for each value are expressed in advance in the form of a table from the system state number, the user state number, the number of repetitions, and the certainty factor. For example, if the degree of certainty is high when the number of repetitions is up to one, the confirmation is made with a normal voice, and if the certainty is low, the confirmation is made with a confused voice. In addition, in order to emphasize the confirmation when the confirmation is made and to convey the confirmation contents to the user in an easy-to-understand manner, the person image information includes an emphasis item. This is the item that should be confirmed when the dialog management unit 12 generates the response content as the output semantic expression. The emotion / emphasis determination unit 134 extracts the item to be emphasized in the response sentence from the output semantic expression and transmits it to the next voice response generation unit 135.

The voice response generation unit 135 includes the response sentence generation unit 1
Speech synthesis is performed based on the output from the emotion / emphasis determination unit 134 and the emotion / emphasis determination unit 134. As a voice generation method, a conventional recording / editing type or the like can be used. However, in the present embodiment, a response having emphasis and emotion is characterized. desirable.

FIG. 2 shows an example of the configuration of the voice response generator 135.
It is shown in FIG. The voice response generation unit 135 includes the phoneme processing unit 15
1, prosody processing unit 152, control parameter generation unit 153,
It is composed of an audio waveform generator 154.

Here, based on the words (phrases) to be emphasized and the types of emotions input from the emotion / emphasis determining unit 134, and the generated response sentence and its structure, the phonological processing unit 151 and the prosody processing unit 152 respectively. The phoneme processing and the prosody processing are performed, and the time series of the control parameters used in the voice waveform generation section 154 are output from the control parameter generation section 153 to the voice waveform generation section 154.

The phoneme processing unit 151 includes a response sentence generation unit 131
Based on the response sentence and the sentence structure generated in step (1), the output response sentence is determined according to generally well-known phonological rules such as nasalization, voicelessness, and rendaku, and a phonetic symbol sequence is output.

In the prosody processing unit 152, the response sentence and its structure,
Based on the word information to be emphasized and the type of emotion, prosody components such as a fundamental frequency pattern, power, duration, and a pause position are determined.

Particularly, as shown in the model of FIG. 26, the fundamental frequency pattern generation analyzes the degree of accent components and phrase components of each response sentence in the case of not emphasizing in advance indicated by a dotted line and in the case of emphasizing it indicated by a solid line. It can be realized by using the components for words and phrases at the time of synthesis. Alternatively, sentence types such as declarative sentences, question sentences, and command sentences may be classified, and rules for accents and phrases may be created for each sentence type. For example, as described in a document (Hirose, Fujisaki, Kawai, “Continuous Speech Synthesis System-Especially Synthesis of Prosodic Features-”, Japanese Society of Acoustical Society Speech Research Society S85-43 (1985)) The rules of declarative sentences can be determined from the word position and the modification relation.

The prosody with emotion is described in the literature (K. Sheahan, Y.
Yamashita, Y. Takebayashi, “Synthsis of Nonverbal Ex
As described in “Compressions for Human-Computer Interaction”, Proceedings of the Acoustical Society of Japan, 2-4-6 (1990.3)), it is mainly controlled by the rate of change of the fundamental frequency, dynamic range, utterance time, and energy. Therefore, as shown in Fig. 27, in the case of joy, the accent is increased 1.2 times, the utterance time is increased 0.9 times, the energy is increased by 2dB, and the sadness shown in Fig. 28 is increased. In the case of, the accent is 0.9 times, the vocalization time is 1.1 times, and the energy is reduced by 2 dB. However, when sadness is involved, it is possible to generate a slightly slower synthetic sound with less intonation.

The control of the fundamental frequency is not limited to the one used in FIG. 41, but may be a method using linear approximation or a method of expressing the fundamental frequency pattern at the level of the sound level. Various methods may be used without departing from the spirit of the invention.

The control parameter generation section 153 determines the control parameters used in the speech waveform generation section 154 based on the phoneme symbols and the prosody symbols from the phoneme processing section 151 and the prosody processing section 152. This control parameter generator 153
Since the utterance speed is also controlled, it is possible to synthesize speech in accordance with the utterance speed of the user, and it is also possible to proceed with the conversation at the utterance speed of the user.

For this reason, the utterance time length obtained by the control parameter generation unit is output to the output integration control unit 137 in order to synchronize the mouth operation of the human image with the voice response.

In the response generation and output unit 13, the generation of the response sentence is performed by the response sentence generation unit 131, the emotion / emphasis determination unit 134, and the voice response generation unit 135 as described above. The speed is referred to reflect the length of the response sentence. This is because when a conversation with a good tempo is being made, it is better that the response is short, and when the user utters slowly for reasons such as being confused, it is better to respond carefully without omission. For example, if the utterance speed is faster than 9 mora per second, try to choose a short response sentence pattern,
This is achieved.

The likelihood of each keyword given from the dialogue management unit 12 is used, for example, to properly use “is / is?” At the end of a sentence in a confirmation scene. That is, the average likelihood of the keyword is lower than the set threshold value 0.5, for example.
Or, if one of the keywords has a very low likelihood, use "is", and if the likelihood is high, use "is". As a result, in addition to other response outputs, the degree of understanding of the computer can be understood from the response sentence, and the user can easily interact.

[0107] It should be noted that it is also possible to make it possible to change the sentence pattern after determining the sentence pattern without having "is / was" as a sentence pattern table. Alternatively, another word may be used, such as using information on whether or not the response is polite, such as “I am / Is it?”.

The speech waveform generator 154 uses rule synthesis by a formant type synthesizer as shown in FIG. 29, for example. This means, for example, that the sampling frequency is 12 kHz, 8 ms
Speech synthesis can be performed by updating the synthesis parameters for each ec and using a sound source that has been subjected to a low-pass filter for the impulse. However, the composition of the synthesizer, the type of sound source,
A generally known sampling frequency can be used.

In the speech waveform generator 154 composed of a formant type synthesizer shown in FIG. 29, the control parameters input from the control parameter synthesizer 169 include an impulse generator 161, a noise generator 162, a low-pass filter 163A, 163B, an amplitude controller 167, a high-pass filter 168, and resonators 166A and 166B.

The visualization information is information such as the content transmitted to the system during the dialogue, the content understood by the system, the state of the system, and the like. Based on the visualization information, the content visualization information generation unit 136 generates the content visualization information. By generating and visually presenting it to the user, the state of the system and the contents of the understanding can be shared by the system and the user, and the dialogue can proceed naturally and easily.

In this embodiment, the order table is used as visualization information. The order table has already recorded all the items, sizes, and the number of items ordered by the user, and the order contents at each point of the dialog can be confirmed. Thus, for example, when an order with a large number of items is placed, the order, the size, and the number of each item are visually displayed and the contents of the order are conveyed in parallel, rather than being heard in a timely manner only by voice response. Becomes possible. Content visualization information generation unit 13
6 generates an image based on the visualized content information.
As the image generation method here, the method described in the person image generation unit 133 can be used. That is, control for changing the display time and the display image is performed. When a still image is used, a plurality of images are prepared, and when a moving image is used, a display image of a continuous operation is designated. When using computer graphics, an image of a specified operation is generated.

The output integration control unit 137 is provided with the person image generation unit 1
33, image information of a person image having a facial expression which is an output, signal information of a voice response having an emotion or emphasis which is an output of a voice response generation unit 135, text information which is a character string of a response sentence, and a content visualization information generation unit 136 While receiving the content visualization information output from each unit and controlling the temporal presentation order,
The contents are output to the screen display output unit 14 and the audio output unit 15, and the response contents are integrated and presented to the user.

What is important here is that it is not necessary to present each output individually, but it is necessary to present each output information in consideration of the temporal relationship. For example, it is necessary to synchronize the voice response output with the control of the mouth operation when the person image moves the mouth while following the voice response, or to synchronize the image output and the voice output when greeting while greeting the bow. It is also important to control the order in which the outputs are presented.

FIGS. 30, 31, 32 and 33 show examples of time control of the output presentation order. FIG. 30 is a control of the first greeting scene, and since there is no order yet, the content visualization information is not displayed, but the text information of the greeting is displayed at time t0, and at the same time, the person image is “Welcome”, and Prompt the order while operating the mouth in response to the voice response "Please place an order." In this way, the uttered content is synchronized with the person image screen, and all the text information is displayed at t0 for easy understanding in advance.

In FIG. 31, one hamburger and one cola have already been prepared.
In this case, the text information of "the order is one hamburger and one coffee" is displayed until the time t0. Next, the content visualization information is updated anew at t0 when the human image and voice response are started, and three hamburgers, two coffees, and one cola are displayed. Also, the person image is t
The mouth is controlled to move from 0 to t3.

The time control shown in this example is determined based on the length of the voice response. That is, in FIG. 30, the duration from t0 to t1 is determined by "Welcome," and the duration from t1 to t2 is determined by "Please place an order." The respective durations are determined by the voice response generation unit 135, and the voice response signal and the duration are sent to the output integrated control unit 137 and used for time control. In addition to the above, it is also possible to perform time control based on the content visualization information to be presented or the display time of a human image.

FIG. 32 shows the control of the scene of confirming all orders after receiving the first order. The items to be confirmed are three items: two hamburgers, one cheeseburger, and three coffees. In FIG. 32, the response content "Order is hamburger 2
, One cheeseburger and three coffees "at the time t0, and start the voice response and the movement of the mouth of the person image in accordance with the voice response. Until the voice "Order", the content visualization information is not displayed, but two hamburgers, one cheeseburger, and three coffees are displayed as the content visualization information at the time t1 when the order is started to be uttered. To do. The human image is controlled so that the mouth is moved from t0 to t4 in accordance with the utterance of the voice.

Here, the response sentence for confirmation of all orders is generated by the response sentence generation unit 131. However, as the number of items to be confirmed increases, the generated response sentence becomes longer and the length of the voice response becomes longer. However, in the example of FIG. 32, the user can understand the response content of the system or the state or understanding of the system without listening to the voice response to the end by the content visualization information displayed at time t1. The voice response ends from t1 after displaying the content visualization information.
The information output up to 4 is a redundant response for the user.

For this reason, in this embodiment, as shown in FIG. 33, when all the orders are confirmed and there are three or more items to be confirmed, the output presentation order is changed, and the immediately preceding response is returned at the time t0. The sentence text is deleted once, and as the content visualization information,
2 hamburgers, 1 cheeseburger,
Display three coffees. Next, at time t1 when the display processing of the content visualization information is completed, a response sentence text “Is this OK?” Is displayed, and a human image and voice response are started. The time control shown in this example is performed by the output integrated control 137 based on the ACT information of the output semantic expression generated by the dialog management unit 12 and the number of items, and the response sentence is generated by the response sentence generation unit 131. You.

Further, this is not limited to the confirmation of all orders. In the case of other confirmations, when there are many items of the response confirmation contents or when it is complicated and difficult to understand, the visual response output is first given. After performing the dialogue, the dialogue can be efficiently performed in a short time by giving a short-form voice response using a demonstrative pronoun or the like.

Note that, instead of the number of items to be confirmed, an index indicating the length of another voice response, for example, the number of words or moras in the voice response, is used to control the change in response output as described above. May be.

Further, the output integration control section 137 controls the display location of each image display information. For example, on the screen of the image output device 14, a person image can be controlled and displayed on the upper left, the content visualization information on the right, and the text information on the lower left. This display position can be changed under the control of the output integrated control unit 137.

As described above, the present invention is characterized in that the dialogue is advanced by using both the input and output of voice and the screen display. Here, an actual example of the screen display in the present invention will be specifically described.

First, FIG. 34 shows the initial screen. When the customer is not at the store or does not come near, only a sentence is displayed on the screen such as "Welcome to" and no voice response is output. .

Here, when the user (customer) approaches the system (counter, drive-through window, etc.), for example, if the user is detected by sensor information such as a floor mat with a pressure sensor or an image of a monitoring camera, etc. As shown in FIG. 35, the system displays "Please come and order now." In kanji or kana mixed sentences, and displays a smiley clerk on the screen to output a voice response (FIG. 9). State S0).

At this time, it is important for the detection of the user to take into account the movement and position of the person, and in particular, to execute the above-described processing when the stop is detected, and to start the voice dialogue with the user with peace of mind. In particular, the smile of the clerk has the effect of relaxing the customer, and it is also desirable to synthesize a bright voice. These techniques have already been developed, and it is also possible to use recorded synthetic sounds and natural sounds.

In this state, the user looks at the screen,
Suppose you make a quick order and say, "Eh, please give me two hamburgers and two coffees.
Then, the system processes the pronunciation of the user in the state U0 of FIG. 9, but there is a part that cannot be heard right now, and no result is obtained from the voice understanding unit 11 shown in FIG. 1.
The dialog manager 12 responds to the reject.

In this case, as shown in FIG. 36, the system displays a kanji / kana mixed sentence saying "I couldn't hear clearly. Please try again." And a clerk with an expression that seems to be apologetic is displayed on the screen. And answer voice. In this state, the system cannot hear the user's order at all, and there is no order table at that time (empty).
Since it is in the state, nothing is displayed on the screen regarding the order.
In addition, the generation of the expression of the clerk is output as the involvement of the response sentence. In this case, the process moves from the user state U0 in FIG. 9 to the reject dialogue correction state S10, where the response and the expression are determined.

Next, assuming that the user who has received the response from this system places an order of "two hamburgers and two coffees" in a manner clearer than the previous time, the above-described speech understanding processing and interactive processing are performed. Is executed to generate the input semantic expression and the order table, and then the output semantic expression is determined. Then, when the response ACT of the output meaning becomes “all confirmation”, the screen display shown in FIG. 37 and the voice response are performed together as the next response.

[0130] In this case, the system displays "Your order is two hamburgers and two coffees" in kanji or kana mixed sentences, and displays the clerk's face on the screen and outputs a voice response. I will be. At this time, the expression of the clerk and the emotion of the voice response are determined in consideration of the sentence and the state as described above, and a normal expression and a voice response are output here. In addition, the contents of the order table are displayed on the screen together with the response sentence, and the user can quickly confirm whether or not the item is the item requested by the user and the number of items.

In this case, the display of the articles may be displayed on the screen in a state in which the articles are arranged by the order quantity without the numbers being represented by numerals. Here, since the numerical information is important, an area having the same height as an article such as a hamburger is provided to display the numerical value. That is, the information on the number (number) is important, and the user can be naturally informed of the information. In addition, as for the display size of numbers, since approximate information is transmitted by size, it is effective to change the size and display it.In addition, it is also possible to use character information or color information together and output By doing so, it is possible to naturally convey a more realistic image to the user more quickly than a voice response or a text response, and a higher-speed confirmation dialog is realized. On the other hand, as for a person image relating to a clerk, a picture with a small amount of information that conveys a point to be conveyed is more effective than a realistic expression. In addition, the above-described image display can be naturally performed using three-dimensional graphics.

Now, it is assumed that the user confirms the order from the system side and sounds a little hesitantly saying "Oh, well, that's okay". Then, in the system, since no result is obtained from the speech understanding unit 11, the dialog management unit 12 responds to the rejection. In this case, as shown in FIG. 38, the system displays a kanji / kana mixed sentence saying "Sorry, please enter again.", And displays a clerk on the screen for voice response. The message in this case is shorter than that in the case of FIG. 36 described above, and the response sentence is determined using the state and the history information of the dialog so as to convey the voice dialog briefly. As for the expression of the clerk's face, an excuse is output corresponding to the response sentence.

The point of this screen display is that the contents of the order currently understood by the system are displayed in the right area. Since the display of the order product displays the order table as it is, it is possible to make up for the temporary defect of the voice. That is, confirmation of addition, replacement, and deletion is performed for a partial order in the voice response and response sentence, but the effect of displaying the accumulated order and the result of the confirmation as the result of the confirmation is large. .

[0134] Such display can be easily realized by the processing in the dialog management section 12 described above. It is also possible to use a visual display for the partial confirmation, and it is also possible to continuously display all the ordered items in another display area. Furthermore, a partial confirmation screen can be displayed to temporarily hide the display of all the ordered items and concentrate the user's consciousness on the partial confirmation. That is, it is effective to use the display method combining the advantages of the two in accordance with the situation.

Thereafter, when the user utters "Okay, so" with a clear voice, the system recognizes this as affirmative and shifts to S9 in FIG. 9, and as shown in FIG. Thank you. "And a kanji kana mixed sentence is displayed, and the clerk whose head is bowed down is displayed on the screen to give a voice response and end the dialogue.

At this time, the generation of the response sentence, the generation of the smile, and the generation of the bowing gesture are also performed by the same processing as described above. Also, the display of the total amount and the like can be performed by various responses (voice, screen display).

In the case of the above-described confirmation of FIG. 38, FIG.
It is also effective to display the screen so as to guide the answer “yes” or “no” as shown in FIG. In this case, information on the number of times of reflection and correction can be used, and the system uses a dialogue such as "I'm sorry. Is your order two hamburgers and two coffees? There is an advantage that can be performed.

FIG. 41 briefly summarizes the processing procedure in such an embodiment.

In this case, when the floor mat 220 detects a user, a keyword is detected by the keyword detection section 21 with respect to a voice input from the user, and the word candidate sequence 2 is detected.
Then, the syntactic and semantic analysis unit 22 performs speech understanding based on the keyword, and obtains an input semantic expression 224. Further, the dialogue control unit 12 performs a dialogue and a dialogue process based on the knowledge of the application field, obtains an output semantic expression 226, and supplies the output semantic expression 226 to the response generating unit 13, and outputs a voice response by rule synthesis and screen display here. A multimodal consisting of an output and a response.

Although the above description has been made with reference to the example of ordering fast food, the present invention can be applied to an information service, a multimedia, a workstation, and a seat reservation system using a communication network.

Next, another embodiment of the present invention will be described with reference to FIG.

FIG. 42 shows a voice dialogue system according to the present invention in which a function of detecting a motion state of a person is incorporated. In this case, human state detection is a function that is necessary for the system to automatically start and end the dialogue, and to understand the user's state and reaction at the start and end of the dialogue so that the dialogue can proceed naturally. Is made possible.
As a method for detecting a human state, it is conceivable to perform processing by processing light, ultrasonic waves, infrared pressure, and the like. Here, an example using a floor mat that can detect one adult will be described.

In FIG. 42, a speech input unit 231, a speech understanding unit 232, a dialogue management unit 234, a response generation unit 235, a display 236, and a speaker 237 similar to those described in FIG.
In addition, a configuration is provided in which a human state detection unit 233 is provided.

In this case, the human state detection section 233
As shown in the figure, when a person is on the mat, a human state detection meaning expression 1 is output, and when a person gets off the mat, a human state detection meaning expression 2 is output. Is notified to the dialogue management unit 234.

[0145] The dialogue management section 234 includes a human state detection section 233.
In addition to the human state detection meaning expression from, the input meaning expression is also fetched from the speech understanding unit 232 as in the above-described embodiment,
An output semantic expression for acknowledgment is generated using the dialog knowledge and the dialog history information.

In this case, when receiving the input meaning expression from the speech understanding unit 232 and the human state detection meaning expression from the human state detection unit 233, the dialog management unit 234 sequentially processes each meaning expression according to the state of the dialog. And can be processed preferentially, so that the user's state and various reactions can be understood and the dialogue can be advanced.

Thus, when the user gets on the mat, the human state detection meaning expression 1 is output from the human state detection unit 233 and sent to the dialog management unit 234. Then, the output meaning expression 1 of the greeting is sent from the dialog management unit 234 to the response generation output unit 235, and “Please come and place an order” is output from the display 236 and the speaker 237 as a response output.

Next, the user selects “Hamburger and coffee 2”.
When the user inputs "continuously", the input meaning expression 1 is output from the speech understanding unit 232 and sent to the dialog management unit 234. Thus, the dialogue management unit 234 refers to the input semantic expression and the contents of the order table, outputs the output semantic expression 2, and outputs a response of “hamburger 2 to coffee 2” through the response generation output unit 235. Become so.

In this case, normally, the dialogue proceeds as shown in FIG. 44, such as "2 hamburgers and 2 coffees", "Yes" and "Thank you." FIG. 45 shows a case where the user has left the camera.

That is, after the response of the output semantic expression 2 “2 hamburgers and 2 coffees” is output,
The human state detection meaning expression 2 is output from the human state detection unit 233 and input to the dialog management unit 234. In this case, since the user left without confirming the utterance content, the dialog management unit 234 does not register the order content, and outputs a natural response of “Thank you for using” in the output semantic expression 4. Output.

In this way, by combining the human state detection section 233 with the dialog management section 234, it is possible to understand the state and reaction of the user, and the dialog can proceed naturally.

In this embodiment, the mat is used for detecting the state of a person. However, the present invention is not limited to this, and another method such as a surveillance camera may be used.

Next, such processing will be described with reference to the flowchart of FIG.

In this case, the system has states # 0, # 1, # 2, and # 3 in the dialogue management 234, and the initial state is state # 0 (step S281).
In state # 0, the human state ACT of the human state detection meaning expression
Is "human presence" (step S282),
If there is a person, the state is set to # 1, and a response is generated and output based on the output meaning expression of the greeting (step S2).
83).

Next, in the state # 1, the speech understanding unit 23
If the utterance ACT of the input semantic expression is an order from Step 2 (steps S284 and S285), an output semantic expression for confirming the contents of the order is transmitted based on the conversation knowledge, and a response is output. If the utterance ACT is yes (step S2
87) The state is set to # 2, and a response is output in the output semantic expression corresponding to the utterance act yes (step S288). If the utterance ACT is no (step S289), an output meaning expression for reconfirming the order details is transmitted and a response is output. Furthermore, when it is confirmed that the human state ACT of the human state detection meaning expression is “absent” (step S291), the state is changed to # 3.
To

Then, in state # 2, "Thank you" is output as thanks 1 (step S2).
93, S294), in state # 3, "Thank you for using" is output as thanks 2 (steps S295, S296).

Next, another embodiment of the present invention will be described with reference to FIG.

In this embodiment, the speech comprehension unit 1 described with reference to FIG.
1. A part for performing voice input / output and human state detection in the response generation / output unit 13 will be described in detail.

In this case, the entire speech dialogue system is shown in FIG.
As shown in the figure, the operation unit 291, the memory unit 292, the storage unit 2
93, storage unit interface 2931, communication unit 29
4. Communication unit interface 2941, A / D unit 29
5, mat part 296, arithmetic processing part 297, D / A part 29
8, a display unit 299.

Here, the A / D unit 295 is connected to the microphone 295
1, filter amplifying section 2952, A / D converting section 2953,
An A / D converter interface 2954 is provided. The filter amplifying section 2952 has a high-frequency cutoff filter function for amplifying an input from the microphone 2951 and performing A / D conversion. The cutoff frequency of the filter here is
It is determined by the sampling frequency of A / D conversion.
When sampling at 2 kHz, high frequency components are cut off at 5.4 kHz. The A / D converter 2953 converts the amplified input sound to, for example, 16 kHz.
The data is digitized at z or 12 kHz, temporarily stored in the A / D converter interface 2954, and transferred to the memory 292 under the control of the arithmetic unit 291.

The mat section 296 includes a mat 2961, a mat control section 2962, and a mat control section interface 296.
The presence / absence of a person on the mat 2961 is detected by the mat control unit 2962, and the result is transferred through the mat control unit interface 2963.

The arithmetic processing section 297 is provided with a high-speed arithmetic processing section 29.
71, a high-speed processing unit interface 2972. The high-speed arithmetic processing unit 2971 is used for processing necessary for a large amount of arithmetic such as voice understanding processing, response generation processing, and human state detection processing by image processing. In this case, such processing needs to be performed simultaneously, so that a plurality of high-speed operation processing units 2971 can be used simultaneously. In each arithmetic processing, the input data is transferred from the memory unit 292 to the high-speed arithmetic processing unit 2971 under the control of the arithmetic unit 291, and the result is stored in the memory unit 292 after the processing is completed.
To be forwarded to.

The D / A section 298 comprises a D / A conversion section interface 2981, a D / A conversion section 2982, a filter amplification section 2983, and a speaker 2984.
The digital data stored in the memory unit 292 is controlled by the D / A conversion unit interface 2981 under the control of
/ A conversion unit 2982, where the data is converted into analog data at a constant period, for example, 12 kHz, and output to the speaker 2984 through the filter amplification unit 2983. In this case, the D / A conversion unit 2982 has a data temporary storage unit, and performs high-speed data transfer from the memory unit 292 so that the arithmetic unit 291 can perform other processing.

The display unit 299 includes a display control unit interface 2991, a display control unit 2992, and a display 299.
Under the control of the arithmetic unit 291, information such as images, characters, graphics, moving image information, changes in color, brightness, and density information is displayed on the display 2993 by the display control unit 2992.

The communication unit 294 communicates control information data with an external computer, information processing device, service processing device, and the like. Each data is exchanged through the communication unit interface 2941 under the control of the arithmetic unit 291.

The storage unit 293 stores data, control information, programs, intermediate information, and the like necessary for speech understanding, dialog management, and response generation under the control of the arithmetic unit 291.

The arithmetic section 291 uses the information of each section, the execution program, and the program for the information stored in the memory section 292 to execute the A / D section 295, the mat section 296, the arithmetic processing section 297, the D / A section 298, and the communication section. Unit 294, storage unit 29
3 is performed.

Here, the program executed by the operation unit 291 includes the speech understanding unit 11, the dialog management unit 12,
The processing in the response generation / output unit 13 is performed, and the processing is executed in a multitask format. The task switching for that is performed sequentially at regular intervals, but when it is necessary to give priority to the processing, such as when the processing of each unit or input / output is completed,
Priority is given to that processing by interruption.

In the above description, the A / D section 295 and the D / A section 298
Can be operated individually. As a result, the voice input and the synthesized voice output can be handled simultaneously and separately, so that the voice input can be performed even during the synthesized voice output, and the input voice can be detected and recognized by canceling the synthesized voice.

However, as a configuration of the A / D section 295 and the D / A section 298, a common A / D and D / A conversion section interface 301 may be used as shown in FIG. 48 (b), the common A /
D, D / A converter interface 302, A / D, D
The / A conversion filter section 303 and the amplification section 304 may be used.

However, in such a configuration, data cannot be exchanged in both directions at the same time and is limited to either input or output. Therefore, the synthesized sound is output simultaneously with the reception of the voice input during the output of the synthesized sound. Can not do it.

In this case, when the user has no means of knowing the state of accepting the voice input, inconveniences such as waiting for a response to an unaccepted utterance or not inputting the first half of the utterance occur. Therefore, by displaying the audio input / output permission state as an image, the computer side can inform the user of the audio input / output permission state. In particular, by combining image display and character display, for example, FIG.
As shown in FIG. 50, it is possible to display a state in which utterance can be performed by "lips" and "SPEAK", and a state in which lips cannot be produced by "lips" and "LISTEN" as shown in FIG.

As described above, the status and status of each input / output device can be transmitted by another output device, and a more natural and easy-to-understand conversation can be performed. Furthermore, not only the status of each input / output device, but also when the user needs to listen carefully for important matters or when he / she does not want to use voice input in dialogue management, etc. Attention can be drawn by a combination or a change in color, luminance, or density.

The present invention is not limited only to the above embodiments,
The present invention can be appropriately modified and implemented without changing the gist.

[0175]

As described above, according to the present invention, when a response is output from the system side to the user, in addition to the voice response, the person (face) is given a sense of closeness to the display system, and at the same time, the voice is output. The mouth is moved in synchronization with the response to determine the user's attention and improve the usability. Also, on the same screen, a voice response sentence is displayed in the form of text data in order to cover the low quality of the voice response, so that the user can receive the response text faster than the voice utterance speed. Furthermore, the response content is visualized on the same screen (Visu
display of the important message to be transmitted in a form suitable for various applications in synchronism with the voice response and voice response of the person display, so that the user can see at a glance High-speed transmission of messages from the system side to the user can be realized.

As described above, since various types of responses are output from the system side to the user in parallel on the same screen, the user selects appropriate individual responses according to the situation, or selects two or three types of responses. It is possible to receive response data of various types at the same time, and it is possible to obtain the effect of utilizing the characteristics of each media, increasing the degree of freedom for the user,
An easy-to-use multi-modal human interface can be realized.

As a result, imperfections based on erroneous confirmation and ambiguity of the speech confirmation unit of the speech dialogue system, which were problems in the past, can be quickly and efficiently covered by the dialogue, and the user's intention can be determined by the progress of the dialogue. It becomes easy to understand that.

Further, a human state detecting means such as a mat or a camera is provided on the input side to not only emit a synthetic sound but also brighten the face of a person on the display screen upon user detection.
Since it is possible to make the user smile, not only the timing of the dialogue is improved, but also a user-friendly voice interface that can be used with ease without the user being surprised can be realized. Further, by applying the present invention to a multi-modal voice interaction system, a natural system that is easy to use is obtained, and the effect of significantly improving the operability of a computer is obtained.

Also, by adding a voice response cancel function, the user can always see the screen display result even during a voice response and interrupt (Interrupt), thereby enabling speedy voice interaction. Even if the voice recognition performance is low, it can be covered by dialogue exchanges, and the efficiency of communication of intention and data input can be greatly improved.

Summarizing the above, the present invention provides a speech dialogue system having speech recognition, speech synthesis, and dialogue management functions. And visualization of facial expressions and gestures, items and sizes,
Since the display of the compensation object (object) such as the type and the character output of the response sentence can be performed in parallel, the user can receive the response from various viewpoints at the same time, and the degree of freedom increases. Accurate information can be selected as needed, which is effective for improving familiarity, efficiency, and comfort, and reducing eyes and ear fatigue.

[Brief description of the drawings]

FIG. 1 is a diagram showing a schematic configuration of an embodiment of the present invention.

FIG. 2 is a diagram showing a detailed configuration of a speech understanding unit.

FIG. 3 is a diagram illustrating a keyword candidate sequence.

FIG. 4 is a diagram showing an example of an input meaning expression.

FIG. 5 is a diagram showing the contents of a keyword.

FIG. 6 is a diagram showing an example of an output semantic expression.

FIG. 7 is a diagram showing an example of an order table.

FIG. 8 is a diagram showing an example of an old order table.

FIG. 9 is a diagram showing an example of a state transition inside a dialog management unit.

FIG. 10 is a flowchart illustrating processing in a user state.

FIG. 11 is a flowchart for explaining processing in a system state.

FIG. 12 is a diagram showing a specific example of the dialog processing.

FIG. 13 is a view for explaining processing of a user state in the interactive processing shown in FIG. 12;

FIG. 14 is a view for explaining processing of the system state in the interactive processing.

FIG. 15 is a view showing a specific example of the dialog processing in FIG. 14;

FIG. 16 is a diagram showing an example of an output response sentence from the system.

FIG. 17 is a view for explaining how to determine a user's utterance speed;

FIG. 18 is a diagram illustrating an example of an output of a dialog management unit.

FIG. 19 is a diagram showing a detailed configuration of a response generation output unit.

FIG. 20 is a diagram showing an example of person image information.

FIG. 21 is a diagram showing an example of a response sentence pattern.

FIG. 22 is a diagram illustrating a flowchart and a specific example of generating a response sentence in a response sentence generation unit.

FIG. 23 is a diagram illustrating an example of a person image expression determining unit.

FIG. 24 is a diagram illustrating an example of an emotion / emphasis determination unit.

FIG. 25 is a diagram showing a detailed configuration of a voice response generation unit.

FIG. 26 is a diagram showing an example of a fundamental frequency pattern model.

FIG. 27 is a diagram showing a change in a fundamental frequency pattern in the case of a response accompanied by joy.

FIG. 28 is a diagram showing a change in a fundamental frequency pattern in the case of a response with sadness.

FIG. 29 is a diagram showing an example of a specific configuration of an audio waveform generation unit.

FIG. 30 is a timing chart showing an example of time control of an output presentation order.

FIG. 31 is a timing chart showing another example of time control of the output presentation order.

FIG. 32 is a timing chart showing another example of time control of the output presentation order.

FIG. 33 is a timing chart showing another example of time control of the output presentation order.

FIG. 34 is a view showing a display example on a display screen.

FIG. 35 is a view showing a display example on a display screen.

FIG. 36 is a view showing a display example on a display screen.

FIG. 37 is a view showing a display example on a display screen.

FIG. 38 is a view showing a display example on a display screen.

FIG. 39 is a view showing a display example on a display screen.

FIG. 40 is a view showing a display example on a display screen.

FIG. 41 is a view simply showing the procedure of the dialog processing;

FIG. 42 is a view showing a schematic configuration of another embodiment of the present invention.

FIG. 43 is a view for explaining a human state detection unit;

FIG. 44 is a view for explaining the operation of the other embodiment shown in FIG. 42;

FIG. 45 is a view for explaining the operation of the other embodiment shown in FIG. 42;

FIG. 46 is a flowchart for explaining the operation of the other embodiment shown in FIG. 42;

FIG. 47 is a view showing a schematic configuration of another embodiment of the present invention.

FIG. 48 is a view showing an example in which a part of another embodiment shown in FIG. 47 is modified.

FIG. 49 is a view showing a display example on a display screen.

FIG. 50 is a view showing a display example on a display screen.

[Explanation of symbols]

 11,232 Voice comprehension unit 12,234 Dialogue management unit 13,235 Response generation unit 14,236 Display 15,237 Speaker 21 Keyword detection unit 21a Voice analysis unit 21b Keyword spotting processing unit 22 Syntax / meaning analysis unit 22a Sentence start determination unit 22b Sentence candidate analysis unit 22c Sentence end determination unit 22d Sentence candidate table 231 Voice input unit 233 Human state detection unit 291 Operation unit 292 Memory unit 293 Storage unit 294 Communication unit 295 A / D unit 296 Mat unit 297 Operation unit 298 D / A Part 299 Display part

──────────────────────────────────────────────────続 き Continued on the front page (51) Int.Cl. ⁷ Identification FI FI Theme Court ゛ (Reference) G06F 17/60 336 G10L 3/00 571U (72) Inventor Hiroyuki Tsuboi Komukai Toshiba-cho, Kochi-ku, Kawasaki-shi, Kanagawa Prefecture. No. 1 In the Toshiba R & D Center (72) Inventor Yoichi Sadamoto 1 in Komukai Toshiba-cho, Koyuki-ku, Kawasaki-shi, Kanagawa Prefecture In-house Toshiba R & D Center (72) Inventor Yasuki Yamashita Kita-ku, Osaka-shi, Osaka 1-1-30 Oyodonaka Toshiba Kansai Branch Office (72) Inventor Hitoshi Nagata 1-1-30 Oyodonaka Kita-ku, Osaka-shi, Osaka Toshiba Kansai Branch Office (72) Inventor Shigenori Seto Kanagawa (1) Inventor Hideaki Shinchi 1385, Shinmachi, Ome-shi, Tokyo Toshiba Pier E A Engineering within Co., Ltd. (72) inventor Hideki Hashimoto Ome, Tokyo Shinmachi 1385 address Toshiba software engineering within Co., Ltd.

Claims (4)

[Claims] 1. An understanding means for receiving an input and understanding the meaning of the input, a dialog management means for determining a meaning of a response content based on an understanding result of the understanding means, and a decision made by the dialog management means. Response generation means for generating a voice response output and a screen display output based on the content of the response, and output means for outputting the voice response output and the screen display output generated by the response generation means. Spoken dialogue system. 2. An understanding step in which an input is given and the meaning of the input is understood, a dialogue management step of making a semantic determination of a response content based on an understanding result in the understanding step, and a dialogue management step. A response generation step of generating a voice response output and a screen display output based on the determined response content; and an output step of outputting the voice response output and the screen display output generated in the response generation step. Spoken dialogue method. 3. A voice comprehension means which is provided with a voice input and understands the meaning of the input voice, a dialogue management means for making a semantic determination of a response content based on an understanding result of the voice understanding means, A response generation unit that generates a voice response output and a screen display output based on the response content determined by the dialog management unit; and an output unit that outputs the voice response output and the screen display output generated by the response generation unit. A speech dialogue system comprising: 4. A voice understanding step in which a voice input is given and the meaning of the input voice is understood, and a dialogue management step in which a response content is semantically determined based on an understanding result in the voice understanding step. A response generation step of generating a voice response output and a screen display output based on the response content determined in the dialog management step; and an output step of outputting the voice response output and the screen display output generated in the response generation step. A speech dialogue method comprising: