JP2006313287A - Speech dialogue apparatus - Google Patents

Abstract

<P>PROBLEM TO BE SOLVED: To provide a speech dialogue apparatus capable of smoothly performing a dialogue with a user. <P>SOLUTION: The speech dialogue apparatus, by which the transfer of information with the user is performed by speech, comprises an intention presuming function for presuming the intention of the user included in a speech spoken by the user based on both of language information obtained by speech recognition processing of utterance and non-language information capable of showing emotional or physiological state of the user. <P>COPYRIGHT: (C)2007,JPO&INPIT

Description

  The present invention relates to a voice interactive apparatus that exchanges information with a user by voice.

Conventionally, input means such as a touch panel and a keyboard for inputting destination position data, storage means for storing road information such as map data and intersection data, a plurality of sound output means for outputting sound, and a vehicle The destination input from the input means by the current position detection means such as GPS or gyro sensor for detecting the current position of the vehicle, the current position detected by the current position detection means and the road information stored in the storage means If the direction determining means for determining the direction to the vehicle and the direction determining means are assumed to be the right direction as the direction of travel of the vehicle, the voice output means A voice output device is known that includes voice output control means for controlling the output of an instruction voice so that it can be heard from the right direction (see, for example, Patent Document 1).
Japanese Patent No. 2602158

  By the way, a general speech recognition apparatus performs speech recognition processing on a user's utterance using a given speech recognition engine, and recognizes the user's intention based on the obtained speech recognition result (language information). It has a function to do. For example, if the user utters “Toyota Eki” when setting the destination in the navigation system, the user wants to go to “Toyota Eki” based on the speech recognition result (language information) “Toyota Eki”. Recognize intent. Thereby, the user can transmit (recognize) the user's intention (destination) to the voice recognition device (navigation system) without manually inputting.

  By the way, in recent years, with the improvement of the voice recognition ability of the voice recognition apparatus, the ability to interact with the user is increasing. In other words, it is not a simple interactive application that directly inputs a predetermined destination by voice as in the previous example, but for example, the user inputs the voice of the user regarding the destination and the system side suggests that There are applications in which a user's intention is determined through a plurality of dialogues such as a user's response (acceptance or rejection) to the user. In such an application, it is important for the system side to quickly capture the user's intention in order to smoothly realize the dialogue with the user, but it is difficult to realize this only with the language information described above.

  SUMMARY OF THE INVENTION An object of the present invention is to provide a voice dialogue apparatus that can more smoothly realize a dialogue with a user.

In order to solve the above-described problem, according to one aspect of the present invention, in a voice interaction apparatus that performs voice exchange of information with a user,
Intention to estimate the user's intention included in the words spoken by the user based on both linguistic information obtained by speech recognition processing for the utterance and non-linguistic information capable of representing the user's emotion or physiological state A spoken dialogue apparatus characterized by having an estimation function is provided.

  In this aspect, the non-linguistic information may include image information obtained by performing image processing on a captured image of the user by a camera, and prosodic information obtained by analyzing the user's uttered voice. The non-linguistic information may include user physiological information based on a measurement result of an electrocardiogram or a pulse.

  According to the present invention, both the linguistic information obtained by the speech recognition processing for the utterance and the non-linguistic information capable of representing the user's emotion or physiological state, the user's intention included in the words uttered by the user. By having the intention estimation function that performs estimation based on the above, it is possible to obtain a voice conversation apparatus that can more smoothly realize a conversation with the user.

  The best mode for carrying out the present invention will be described below with reference to the drawings.

  FIG. 1 is a system configuration diagram showing an embodiment of a voice dialogue system in which a voice dialogue apparatus according to the present invention is incorporated. The voice interaction device 10 includes a dialogue control ECU 20, a vehicle interior microphone (microphone) 30 that picks up sound (sound) in the vehicle interior, an amplifier 42, a speaker 40, and a display 50. The amplifier 42 and the speaker 40 may be the same as those used in the in-vehicle audio system.

  The dialogue control ECU 20 comprehensively controls the dialogue with the user. A response from the system side (interaction control ECU 20) constituting one direction of the dialogue to the user is realized by an acoustic and / or visual output via the speaker 40 and / or the display 50. The other direction of the dialogue, that is, the communication of intention from the user to the system side is realized in principle by voice input through the in-vehicle microphone 30.

  The dialog control ECU 20 includes, as its basic configuration, a CPU, a memory, and an A / D (analog-to-digital) converter connected via a bus. The memory stores programs and data for realizing the functions of the dialog control ECU 20 described below.

  The analog audio input to the in-vehicle microphone 30 is subjected to predetermined processing such as amplification processing and noise removal by a microphone amplifier, converted into a digital audio signal by an A / D converter, and sent to the dialog control ECU 20. The dialogue control ECU 20 extracts a feature amount from the voice signal, and then obtains a recognition result by a matching process using a given acoustic / language model. Note that the present invention is not particularly limited by the speech recognition method, and can be applied to speech recognition processing using any software (speech recognition engine) with any hardware configuration.

  An occupant state estimation ECU 70 is connected to the dialog control ECU 20 via an appropriate bus such as a CAN (controller area network). The occupant state estimation ECU 70 is connected to the in-vehicle camera 60 and the biological sensor 80, and acquires / generates non-linguistic information that can represent the emotion or physiological state of the user.

  The in-vehicle camera 60 is constituted by a CCD camera, for example, and is mounted at a position where an image of a user in the passenger compartment (that is, an occupant including a driver) can be taken. The occupant state estimation ECU 70 receives and processes image data captured by the in-vehicle camera 60 using a built-in image processing processor, thereby acquiring image information representing a user's facial expression, gesture, gesture, and the like. Alternatively, the in-vehicle camera 60 may be a thermographic camera (infrared camera). In this case, the occupant state estimation ECU 70 acquires image information representing a heat distribution state in a predetermined part such as a user's face.

  The biosensor 80 includes a sensor for measuring an electrocardiogram and a pulse. Specifically, the biosensor 80 is an electrocardiograph or a heart rate monitor (for example, a heart rate detection sensor set in a wristband such as a wristwatch), or A pulse meter that measures the pulse rate, and a sphygmomanometer that measures blood pressure. The biosensor 80 may be set, for example, in a wristwatch carried by the user, or may be embedded in a steering wheel operated by a driver.

  The occupant state estimation ECU 70 also analyzes voice data input to the in-vehicle microphone 30 as described above, thereby prosodic features of the user's uttered voice (voice pitch, strength, magnitude, length, etc.). Prosodic information representing is acquired.

  FIG. 2 shows a characteristic operation flow realized by the voice interaction apparatus of the present embodiment.

  As shown in FIG. 2, the voice interaction apparatus according to the present embodiment performs intention estimation based on language information based on the voice recognition result with respect to the voice data input to the in-vehicle microphone 30 (step 100) and the voice recognition result. Based on (prosodic information, etc.) or occupant state detection results, emotion estimation based on non-linguistic information is performed (step 110), and these intention estimation results are integrated to perform final intention estimation (step 120). That is, the speech dialogue apparatus according to the present embodiment is based on both the non-linguistic information representing the user's emotions and changes thereof and the linguistic information representing the speech recognition result. It has an intention estimation function for estimating the intention (meaning).

  Specifically, in step 110, the dialogue control ECU 20 analyzes the electrocardiogram and pulse data from the electrocardiograph based on the physiological information from the biosensor 80, thereby determining the user's tension or excitement state or calm state. presume. Further, the dialogue control ECU 20 estimates the user's emotion (for example, calmness, anger, joy, sadness) based on the prosodic information and the image information. At this time, the dialogue control ECU 20 uses a user database 22 in which user specific information is stored. That is, in the user database 22, physiological information serving as a reference for each user is stored in the user database 22 in a form classified according to the mental state (tension, excitement state, or calm state) of the user at that time. Yes. Similarly, for the prosody information and image information, the prosody information and image information serving as a reference for each user are stored in the user database 22 in a form classified according to the emotion of the user at that time. These reference data may be acquired in advance or may be learned through an actual application. Thereby, the dialogue control ECU 20 can estimate the mental state or emotion of the user with high accuracy by comparing the physiological information and prosodic information detected this time with the data in the database 22.

  When the final intention estimation is performed in step 120, the estimation results obtained in step 100 and step 110 are naturally synchronized with each other. That is, language information for a certain user's utterance is associated with non-linguistic information at the time of the utterance or before and after the utterance.

  FIG. 3 is a table showing an example of a method for fetching a user's intention (meaning) by integrating non-linguistic information and linguistic information. The example shown in FIG. 3 relates to a case where the user utters “why”.

  Here, the utterance “why” generally has multiple possibilities in its meaning, such as a question about what the other party said, a question (quickly blaming the other party), doubt, etc. Therefore, the user's intention cannot be accurately estimated only based on the language information.

  On the other hand, in this embodiment, as shown in FIG. 3, based on the estimation result of the user's emotion estimated from the prosodic information and the image information, and the estimation result of the user's mental state estimated from the physiological information. Thus, by estimating and judging the user's intention (whether it is a question, a question, or a suspicion) included in the utterance “why”, it is possible to accurately estimate the user's intention.

  For example, in the example shown in FIG. 3, when the estimation result of the emotion of the user who uttered “why” is “calm”, the word “why” is estimated to be a language used in the meaning of “question”. The When the emotion estimation result is “anger”, the word “why” is estimated to be a language used in the meaning of “question”. Similarly, when the emotion estimation result is “joy” or “sadness”, the word “why” is estimated to be a language used in the meaning of “suspect”.

  Note that the user's emotion is estimated to be “seduce” based on prosodic information and image information, for example, when the prosody is the same as normal and / or when there is no particular change in facial expression or gesture. Good. In addition, when the prosody changes greatly, and when a facial expression or gesture is characterized by anger (for example, rising eyes or pointed mouth), the user's emotion is estimated to be “angry”. It's okay. Similarly, if the prosody changes significantly and if facial expressions or gestures show joyful features (for example, smiles or laughter or clapping hands), the user's emotion is “joy”. It may be estimated that there is. Also, if the prosody changes greatly and if a facial expression or a characteristic that expresses sadness (such as a gesture of wiping or tearing) appears, the user's emotion is assumed to be "sadness" Good.

  Further, in the example shown in FIG. 3, when the estimation result of the mental state of the user who uttered “why” is “calm”, the word “why” is estimated to be a language used in the meaning of “question”. Is done. When the mental state estimation result is “tension”, the word “why” is presumed to be the language used in the meaning of “question”, and the mental state estimation result is “tension (close to excitement)”. ”, The word“ why ”is presumed to be a language used in the sense of“ suspect ”. Note that the mental state of the user may be estimated simply based on whether or not a large change in the electrocardiogram or the pulse appears regardless of the pattern matching described above.

  FIG. 4 is a table showing another example. The example shown in FIG. 4 relates to a case where the user utters “OK”.

  Similarly, the utterance of “OK” generally has multiple possibilities in its meaning, such as consent to what the other party said or giving up (giving up that it ’s okay). The user's intention cannot be accurately estimated only based on the language information.

  On the other hand, in this embodiment, as shown in FIG. 3, based on the estimation result of the user's emotion estimated from the prosodic information and the image information, and the estimation result of the user's mental state estimated from the physiological information. Thus, it is possible to accurately estimate the user's intention by specifying the user's intention (whether to accept or give up) included in the utterance “OK”.

  For example, in the example shown in FIG. 4, when the estimated result of the emotion of the user who uttered “good” is “calm” or “joy”, the word “good” was used to mean “acceptance”. Presumed to be language. When the emotion estimation result is “anger” or “sadness”, the word “good” is estimated to be a language used in the meaning of “give up”.

  Similarly, in the example shown in FIG. 4, when the estimation result of the mental state of the user who uttered “good” is “calm”, the word “good” is a language used to mean “acceptance”. Presumed to be. When the mental state estimation result is “tension”, the word “good” is presumed to be a language used in the sense of “give up”.

  The correspondence relationship between the user's emotion and mental state and the user's intention as shown in FIGS. 3 and 4 is created for each language that can have such a plurality of meanings, and can be accessed by the dialog control ECU 20. Pre-stored in memory.

  As a result, for example, when the system side outputs a voice message “Would you like to inform about the availability of the parking lot around the destination?” Through the speaker 40 and inquires the user, When you receive a response that says “OK”, by adding the emotion and mental state of the user as described above, the response is “good” in the sense of “fun” or “not necessary” It is possible to accurately determine whether the meaning is “good”. As a result, it does not bother the user by re-listening, and the next output (operation) from the system side to the response can be made accurate, and the dialog with the user is made smoother. Is possible. For example, the dialogue control ECU 20 immediately acquires the availability of the parking lot around the destination, and the “vacation status of the parking lot around the destination” only in the case of “good” meaning “good to meet”. The parking space availability can be displayed on the display 50, while outputting the voice "I will inform you."

  As described above, according to the present embodiment, there are a plurality of meanings of the language spoken by the user, and even if the user's intention cannot be uniquely specified from the language information, the user's emotion and mental state are estimated. By taking these into account, it is possible to estimate the user's intention with high accuracy (exactly). That is, about 7% of humans can communicate with linguistic information, and the research results (study results by psychologist Albert Melavian) that voice prosody and facial expression play an important role in communication are known. However, in this example, as can be seen from the results of this study, the intentions of users who are difficult to recognize using only linguistic information are more accurately considered by taking into account the prosodic and facial expressions of speech in addition to linguistic information. It becomes possible to grasp.

  The preferred embodiments of the present invention have been described in detail above. However, the present invention is not limited to the above-described embodiments, and various modifications and substitutions can be made to the above-described embodiments without departing from the scope of the present invention. Can be added.

  For example, in the above-described embodiment, in order to improve the estimation accuracy of the user's intention, both the user's emotion and mental state are estimated, but a configuration in which only one of them is estimated is also possible. Also, according to the estimation accuracy at that time, it is possible to preferentially use one estimation result and use the other estimation result as an auxiliary.

  In the above description, prosodic information is illustrated as non-linguistic information input to the in-vehicle microphone 30. However, the present invention is not particularly limited to this, for example, breath input to the in-vehicle microphone 30. -You may detect a user's emotion and mental state based on the mode of breathing (mode of sighing and breathing).

1 is a system configuration diagram showing an embodiment of a voice interaction system in which a voice interaction device according to the present invention is incorporated. It is a figure which shows the characteristic operation | movement flow implement | achieved by the voice interactive apparatus of a present Example. It is a table | surface figure which shows an example of the method of integrating non-linguistic information and linguistic information and estimating a user's intention (intention). It is a table | surface figure which shows another example of the method of integrating a non-linguistic information and linguistic information and estimating a user's intention (intention).

Explanation of symbols

10 Spoken Dialogue Device 20 Dialogue Control ECU
22 User Database 30 Car Microphone 40 Speaker 42 Amplifier 50 Display 60 Car Camera 70 Crew State Estimation ECU
80 Biosensor

Claims (3)

In a voice interaction device that exchanges information with the user by voice,
Intention to estimate the user's intention included in the words spoken by the user based on both linguistic information obtained by speech recognition processing for the utterance and non-linguistic information capable of representing the user's emotion or physiological state A spoken dialogue apparatus characterized by having an estimation function.   The spoken dialogue apparatus according to claim 1, wherein the non-linguistic information includes image information obtained by performing image processing on a user-captured image by a camera, and prosodic information obtained by analyzing a user's speech.   The spoken dialogue apparatus according to claim 1, wherein the non-linguistic information includes user physiological information based on a measurement result of an electrocardiogram or a pulse.

Images