JP2001215993A - Device and method for interactive processing and recording medium - Google Patents

Abstract

PROBLEM TO BE SOLVED: To conduct interactive operations having rich variations depending on the feeling condition of a user. SOLUTION: In a voice recognition section 2, user's voice is recognized and phoneme information of the voice is extracted. In an interactive control section 3, conceptual information of the words and the phrases included in the voice recognition result obtained by the section 2 is extracted. An image inputting section 6 photographs the face of the user and outputs face image information. In a physiological information inputting section 7, physiological information such as the pulse rate of the user is detected. Then, a user feeling information updating section 8 estimates the feeling of the user based on the phoneme, the conceptual, the face image and the physiological information. In the section 3 and a sentence generating section 4, an output sentence is generated and outputted to the user based on the estimated result of the feeling.

Description

DETAILED DESCRIPTION OF THE INVENTION

[0001]

BACKGROUND OF THE INVENTION 1. Field of the Invention The present invention relates to a dialogue processing apparatus, a dialogue processing method, and a recording medium, and more particularly to a dialogue processing apparatus and a dialogue processing method capable of performing a dialogue in consideration of, for example, a user's emotion. And a recording medium.

[0002]

2. Description of the Related Art In a so-called interactive system, when a user inputs, a response sentence corresponding to the meaning of the input is generated and output.

[0003]

Therefore, in the conventional dialogue system, the same response sentence is output regardless of the state of the user's emotion if the meaning of the input is the same. As a result, a similar dialogue will take place.

[0004] The present invention has been made in view of such a situation, and enables a variety of dialogues to be performed depending on the emotional state of the user.

[0005]

According to the present invention, there is provided a dialogue processing apparatus for extracting a concept of a phrase input by a user, and estimating a user's emotion based on the concept of a phrase input by the user. The apparatus further includes an emotion estimating unit that outputs emotion information indicating the emotion, and an output sentence generating unit that generates an output sentence to be output to the user based on the emotion information.

[0006] The emotion estimation means, based on the output sentence,
The emotion of the user can be estimated.

Further, the emotion estimation means can estimate the user's emotion based on an image obtained by imaging the user.

Further, the emotion estimation means can estimate the user's emotion based on the user's physiological phenomenon.

The dialogue processing apparatus of the present invention may further include sound processing means for processing a sound signal obtained from the outside. In this case, the emotion estimation means may be provided based on the processing result of the sound processing means. Thus, the user's emotion can be estimated.

[0010] The dialogue processing apparatus of the present invention may further include voice recognition means for recognizing the user's voice. In this case, the concept extracting means includes a concept of a phrase included in the voice recognition result of the user's voice. Can be extracted.

The emotion estimating means can estimate the user's emotion based on the prosodic information of the user's voice.

[0012] The output sentence generation means can change the expression of the output sentence based on the emotion information.

[0013] The output sentence generation means can change the number of output sentences based on the emotion information.

[0014] The output sentence can be meant to be a combination.

[0015] The interactive processing device of the present invention may further include a storage means for storing emotion information.
The output sentence generating means can generate an output sentence based on the emotion information stored in the storage means.

The interactive processing device according to the present invention may further include output sentence output means for outputting an output sentence.

The output sentence output means can output the output sentence as a synthesized sound.

Also, the output sentence output means can control the prosody of the synthesized sound based on the emotion information.

According to the dialogue processing method of the present invention, a concept extracting step of extracting a concept of a phrase input from a user, and estimating the user's emotion based on the concept of the phrase input by the user, and expressing the emotion An emotion estimation step for outputting emotion information and an output sentence generation step for generating an output sentence to be output to the user based on the emotion information are provided.

A recording medium according to the present invention includes a concept extracting step of extracting a concept of a phrase input by a user, and estimating a user's emotion based on the concept of a phrase input by the user, and an emotion expressing the feeling. A program is recorded which includes an emotion estimation step of outputting information, and an output sentence generation step of generating an output sentence to be output to a user based on the emotion information.

In the dialogue processing apparatus, the dialogue processing method, and the recording medium of the present invention, the concept of a phrase input by the user is extracted, and the emotion of the user is estimated based on the concept. Then, an output sentence to be output to the user is generated based on the emotion information obtained as a result.

[0022]

DESCRIPTION OF THE PREFERRED EMBODIMENTS FIG. 1 shows a dialogue system to which the present invention is applied (a system refers to a system in which a plurality of devices are logically assembled, and whether or not the devices of each configuration are in the same housing. The configuration example of one embodiment is shown.

The audio input unit 1 is composed of, for example, a microphone (microphone), an amplifier, etc., converts a user's voice into an audio signal as an electric signal, amplifies it as necessary, and converts the audio signal into an audio signal. It is supplied to the voice recognition unit 2.

The voice recognition unit 2 performs audio processing on the voice signal from the voice input unit 1, and further recognizes the user's voice based on the result of the voice processing. This speech recognition result is supplied to the dialogue management unit 3. Further, the voice recognition unit 2 supplies the user emotion information updating unit 8 with the prosody information of the user's voice obtained by acoustically processing the voice signal.

The dialog management unit 3 includes a user emotion information recording unit 9
Generates the contents of an output sentence to be output to the user as a response to the speech recognition result from the speech recognition unit 2 in consideration of the emotion information indicating (e.g., emotions of) the user, which is held (stored) by the user. Is supplied to the sentence generation unit 4. Further, the dialogue management unit 3 extracts a concept of a phrase included in the speech recognition result from the speech recognition unit 2 and a concept of a phrase included in an output sentence corresponding to the content information generated by itself, and conceptual information representing the concept. Is supplied to the user emotion information updating unit 8.

The sentence generation unit 4 generates an output sentence of, for example, a text corresponding to the content information from the dialog management unit 3 while considering the emotion information held by the user emotion information recording unit 9. A speech signal of a synthesized sound corresponding to the output sentence is generated and supplied to the speech output unit 5.

The audio output unit 5 is composed of, for example, an amplifier and a speaker. The audio output unit 5 amplifies the audio signal from the sentence generation unit 4 as necessary, and outputs the amplified signal from the speaker.

The image input unit 6 includes, for example, a lens, a CCD,
(Charge Coupled Device), an A / D converter, etc., images a user's face and the like, and converts the resulting face image digital data (image data) of the face image into a user emotion information updating unit. 8

The physiological information input unit 7 comprises, for example, a pulse meter, a sensor for measuring the amount of sweat and heat, etc., and senses physiological information such as the user's pulse, amount of sweat and heat, and obtains the result. The supplied physiological information is supplied to the user emotion information updating unit 8.

The user emotion information updating unit 8 includes the voice recognition unit 2
Based on the prosody information of the user's voice, the concept information of words and phrases included in the speech recognition results and the like from the dialog management unit 3, the face image information from the image input unit 6, and the physiological information from the physiological information input unit 7. Estimate the emotional state of the user. further,
The user emotion information updating unit 8 updates the emotion information stored in the user emotion information recording unit 9 with the emotion information obtained as a result of the estimation.

The user emotion information recording unit 9 holds emotion information that represents the user's emotions, such as joy, anger, surprise, sadness, and the like, in numerical values within a predetermined range.

Next, referring to the flowchart of FIG.
The basic processing flow of the dialog system of FIG. 1 will be described.

When the user speaks, the speech input unit 1 performs a speech input process on the spoken speech in step S 1, and outputs a speech signal obtained as a result to the speech recognition unit 2. . That is, the voice input unit 1
Convert the user's voice into a voice signal as an electrical signal,
The voice signal is amplified as needed, and the voice recognition unit 2
To supply.

In step S2, the voice recognition unit 2
Based on the voice signal from the voice input unit 2, the voice of the user is recognized, and the voice recognition result is supplied to the dialog management unit 3. Further, the voice recognition unit 2 extracts the prosody information of the user's voice from the voice signal from the voice input unit 2 and supplies the information to the user emotion information updating unit 8.

Thereafter, the process proceeds to step S3, and a process for preparing to update the emotion information stored in the user emotion information recording unit 9 is performed.

That is, in step S3, the dialogue management unit 3
Performs an emotion information updating dialogue management process for obtaining the above-mentioned concept information, which is used to update the emotion information, based on the speech recognition result of the user's voice from the speech recognition unit 2, and converts the concept information into The information is supplied to the user emotion information updating unit 8. Further, in step S3, the image input unit 6 performs an image input process of capturing the face of the user and obtaining face image information, and supplies the face image information to the user emotion information updating unit 8. In step S3, the physiological information input unit 7 performs a physiological information input process for obtaining physiological information of the user, and supplies the physiological information to the user emotion information updating unit 8.

The user emotion information updating unit 8 determines in step S4
, Based on the prosody information of the user's voice from the voice recognition unit 2, the concept information from the dialogue management unit 3, the face image information from the image input unit 6, and the physiological information from the physiological information input unit 7, Estimate emotional state. Further, in step S4, the user emotion information updating unit 8 updates the emotion information stored in the user emotion information recording unit 9 with the emotion information obtained as a result of the estimation.

Thereafter, in step S5, the dialog management unit 3 considers the emotion information representing the emotion of the user, which is stored (stored) in the user emotion information recording unit 9, and performs the speech recognition from the speech recognition unit 2. A sentence generation dialogue management process for generating content information representing the content of an output sentence to be output to the user as a response to the result or the like is performed, and the content information is supplied to the sentence generation unit 4.

In step S 6, the sentence generation unit 4 generates a text output sentence corresponding to the content information from the dialog management unit 3 while considering the emotion information held by the user emotion information recording unit 9. (Sentence generation processing is performed), and further, an audio signal of a synthesized sound corresponding to the output sentence is generated and supplied to the audio output unit 5.

The audio output unit 5 determines in step S7
The sound signal from the sentence generation unit 4 is amplified, sound output processing for outputting from the speaker is performed, and the processing ends.

In the above case, in the interactive system, the output of a synthesized sound (hereinafter, also referred to as an utterance of the interactive system as appropriate) is triggered by the fact that the user utters some utterance. The sound is a response to the utterance of the user, but in the interactive system, the utterance can be triggered by a trigger other than the utterance of the user.

That is, in the interactive system, for example,
It is possible to make an utterance every predetermined time. Further, for example, when the image input unit 6 obtains a user's face image (including not only when a face image is obtained but also when a face image with a predetermined expression is obtained), or when physiological information is obtained. When predetermined physiological information is obtained in the input unit 7, it is also possible to make an utterance. Further, for example, when the emotion information stored in the user emotion information recording unit 9 becomes equal to or more than a predetermined value, it is possible to make an utterance. In these cases, the dialogue will take place in such a way that the dialogue system speaks to the user and the response is returned by the user.

Next, FIG. 3 shows an example of the configuration of the speech recognition section 2 in FIG.

The audio signal from the audio input unit 1 is AD (Ana
(Log Digtal) conversion unit 11, and the AD conversion unit 11 converts the audio signal from an analog signal to a digital signal, and supplies the resulting audio data to the feature extraction unit 12. I do. Feature extraction unit 12
Performs audio processing on the audio data from the AD conversion unit 11 for each appropriate frame to obtain, for example, a spectrum, a linear prediction coefficient, a cepstrum coefficient, a line spectrum pair, and a MFCC (Mel Frequency Cepstrum Coefficient).
Are extracted and supplied to the matching unit 13.

The feature extracting unit 12 supplies the user emotion information updating unit 8 with prosody information, such as speech speed, pitch frequency, and power, obtained by performing audio processing on the audio data. As the utterance speed, for example, the number of mora per frame or the like can be used.

The matching section 13 refers to the acoustic model database 14, the dictionary database 15, and the grammar database 16 as needed based on the feature parameters supplied from the feature extracting section 12, and, if necessary, the user's voice (input voice). Recognize.

That is, the acoustic model database 14 stores acoustic models representing acoustic features such as individual phonemes and syllables in the language of the speech to be recognized. Here, as the acoustic model, for example, HMM (Hidden Mark
ov Model) can be used. The dictionary database 15 stores a word dictionary in which information about pronunciation is described for each word to be recognized. The grammar database 16 stores grammar rules that describe how words registered in the word dictionary of the dictionary database 15 are linked (connected). Here, the grammar rules include, for example, context-free grammar (CFG) and H
PSG (Head-driven Phrase Structure Grammar) (head-driven phrase structure grammar), statistical word chain probability (N-gr
am) or the like.

[0048] The matching section 13 is a dictionary database 1
By connecting the acoustic models stored in the acoustic model database 14 by referring to the word dictionary of No. 5, an acoustic model (word model) of the word is formed. Further, the matching unit 13 connects several word models by referring to the grammar rules stored in the grammar database 16, and uses the word models connected in this manner, based on the feature parameters, for example, , HMM method, etc., to recognize the user's voice.

The phoneme information as a result of the speech recognition by the matching unit 13 is output to the dialog management unit 3 in the form of, for example, a text or a word graph.

Next, FIG. 4 shows an example of the configuration of the dialog management section 3 of FIG.

The speech recognition result of the user output from the speech recognition unit 2 is supplied to a language processing unit 21.
, Language processing database 24, history database 2
The speech recognition result is processed while referring to 5 as necessary, and the meaning and concept represented by the speech recognition result are supplied to the dialog processing unit 22.

That is, the thesaurus database 23 contains
A thesaurus in which words are classified into a hierarchical structure based on the concept is stored, and the language processing unit 21 recognizes the concept of the word included in the speech recognition result by referring to the thesaurus.

Here, as the thesaurus, for example, a classification vocabulary table published by the National Institute for Japanese Language and the like can be used.

The database 24 for language processing includes a word dictionary in which the notation of each word and necessary part-of-speech information are described.
Based on the information of each word described in the word dictionary, a syntax / semantic rule in which restrictions on word chains and the like are described is stored. The language processing unit 21 uses the word dictionary and the syntax / semantic rule based on the word dictionary and the syntax / semantic rule. Then, a morphological analysis of the speech recognition result input thereto is performed. Further, the language processing unit 21 analyzes the syntax of the speech recognition result based on the morphological analysis result,
In addition, understand the meaning. Then, the language processing unit 21 interprets the concept of each word constituting the speech recognition result obtained as described above and the result of understanding the meaning and content of the speech recognition result (hereinafter, collectively referred to as the language processing result as appropriate).
Is output to the interaction processing unit 22.

Here, the language processing section 21 can perform syntactic analysis and understanding of semantic contents using, for example, regular grammar, context-free grammar, HPSG, and statistical word chain probability.

Further, the language processing unit 21 may, if necessary,
The processing is performed with reference to the history database 25. That is, in the history database 25, a set of a speech recognition result of a speech uttered by the user and a response output by the dialogue system with respect to the utterance, or an output of the dialogue system and an output thereof, The history of the conversation between the user and the conversation system (conversation history) is stored in the form of a combination of the speech uttered by the user with the speech recognition result, and the language processing unit 21 stores the conversation history. , Omission of the subject, etc. in the speech recognition result, analysis of the anaphoric expression, etc., and thereby, for example, what the pronoun included in the user's speech recognition result specifically means And so on.

The information stored in the thesaurus database 23 and the language processing database 24 is as follows.
Basically, it is not updated, so it can be called static information. On the other hand, the dialogue history stored in the history database 25 is updated by the dialogue processing unit 22 described later when the user utters or the dialogue system performs some output to the user. As it goes, it can be called dynamic information.

As described above, the language processing unit 21 extracts the concept of each word (vocabulary) constituting the speech recognition result by referring to the thesaurus database 23, and the concept represents emotion. At some point, a concept representing the emotion is supplied to the user emotion information updating unit 8 as concept information. That is, on the thesaurus, the language processing unit 21 outputs, for example, "joy,""anger,""surprise,""sadness,""suffering,""embarrassment,""fun," and the like.
When a word belonging to a concept representing an emotion is included in the speech recognition result, concept information representing the concept is supplied to the user emotion information updating unit 8.

The language processing unit 21 extracts not only the concept information of the words included in the speech recognition result but also the concept information of the words included in the output of the dialog system stored as the dialog history as necessary. , And to the user emotion information updating unit 8.

That is, the user emotion information updating section 8 estimates the emotion state of the user as described above. In the estimation, not only the concept information of the words included in the speech recognition result but also the dialogue information is used. Word conceptual information in the output of the system may also be useful. Specifically, for example, in a dialogue system, when an utterance that fools the user is performed, the user is expected to be angry. For this reason, the language processing unit 21 extracts the concept information of the words included in the output of the dialog system stored as the dialog history by referring to the thesaurus, and extracts the concept information of the words included in the speech recognition result, The information is supplied to the user emotion information updating unit 8.

The dialogue processing unit 22 is based on the language processing result from the language processing unit 21 and the emotion information indicating the emotional state of the user, which is stored in the user emotion information recording unit 9, based on the history database 25 and the scenario database. 26
, The contents of an output sentence to be output to the user, such as a response to the user's voice recognition result, and the like, and the content information representing the contents is supplied to the sentence generation unit 4.

That is, the scenario database 26 stores, for example, a scenario as a rule of an interaction pattern with the user for each task (topic).
2 basically determines, from among the scenarios stored in the scenario database 26, those to be used for dialogue with the user based on the language processing result from the language processing unit 21, and according to the scenario, Generate content information.

More specifically, for the purpose-oriented task such as video reservation, for example, the following scenario is stored.

(Action (Question (date, start_time, end_time, channel))) (date ???) #date (start_time ???) # start time (end_time ???) #end time (channel ???) #Channel ・ ・ ・ (1)

Here, according to the scenario (1), if the result of the language processing by the language processing unit 21 indicates a request for recording, the interactive processing unit 22 sets the date of recording and the recording date. Content information is generated to ask the start time, the end time of the recording, and the channel to be recorded in such an order.

For example, the following is stored as a scenario for performing a purposeless conversation (so-called chat).

If X exist then speak (Y) # X: keyword, Y: response sentence (money what do you want) # (X Y) (you want to eat hungry) ... (2)

Here, according to the scenario (2), if the keyword “money” is included in the language processing result by the language processing unit 21,
Content information for asking a question, “What do you want?” Is generated. If the keyword “I want to eat” is included in the language processing result by the language processing unit 21,
In the interactive processing unit 22, content information indicating that a question is to be made, such as "I am hungry," is generated.

The dialogue processing unit 22 determines a scenario to be used, for example, based on not only the language processing result from the language processing unit 21 but also the emotion information stored in the user emotion information recording unit 9. . That is, for example, when the language processing result from the language processing unit 21 indicates that the user has greeted, the emotion information indicates that “fun” or “joy” is at the normal level. when there is or when it indicates that "anger" and "irritability" of the character is large, dialogue processing unit 22, a "Hello" to the user, simply determine the use of scenarios that return greeting. Further, for example, when the language processing result from the language processing unit 21 indicates that the user has made a greeting, the emotion information may be “fun” or “joy”.
Is large, the dialogue processing unit 22
Decides to use a scenario that asks the user "What's good?"

The scenario database 26 also stores general knowledge for interacting with the user in addition to the scenario. That is, scenario database 2
For example, in the case where the result of the language processing by the language processing unit 21 indicates that the user has made a greeting, information indicating that a greeting is given to the greeting is stored as general knowledge. ing. Further, in the scenario database 26, for example, topics (topics) used at the time of chatting are also stored as general knowledge.

Further, the dialogue processing unit 22 includes the language processing unit 2
The language processing result from the language database 1 and the content information generated by itself, as well as information on the scenario used to generate the content information, etc.
5 is stored.

Further, the dialogue processing unit 22 may, if necessary,
By referring to the conversation history, for example, it is possible to deal with a case where it is later found that there is an error in the speech recognition result or in understanding the meaning thereof, for example.

Next, FIG. 5 shows a configuration example of the sentence generation unit 4 of FIG.

The text sentence generation unit 31 includes the dialogue management unit 3
The text sentence generating unit 31 supplies the content information from the dictionary database 34 as necessary.
While referring to the generated grammar database 35, a text output sentence corresponding to (conforming to) the content information is generated.

That is, the dictionary database 34 stores a word dictionary in which part-of-speech information of each word, and information such as readings and accents are described.
Reference numeral 5 stores a template of an example of an output sentence, as well as a rule for utilizing words necessary for generating the output sentence and a grammatical rule for generation such as constraint information on word order. Then, the text sentence generation unit 31 selects a template corresponding to the content information, and further selects a necessary word from the word dictionary. Further, the text sentence generation unit 31 refers to the grammar rules for generation and applies words to the template while appropriately changing the ending or the like, thereby generating an output sentence corresponding to the content information.

The text sentence generation unit 31 is also supplied with the emotion information held in the user emotion information recording unit 9. Change the expression of the output statement. That is, the generation grammar database 35 stores templates having the same content but different expressions, and the text sentence generation unit 31 converts the template having the same content from the template having the same content into an emotional expression. Make an informed choice. In addition, the text sentence generation unit 31 also selects a word to be applied to the template with a predetermined expression based on the emotion information. Further, the text sentence generation unit 31
Also changes the ending and the like based on the emotion information.

Thus, for example, when the emotion information indicates that the level of “anger” or “sadness” is high, the text generator 31 generates an output sentence of a relatively polite expression. . Further, for example, when the emotion information indicates that the level of “fun” or “joy” is high, the text generating unit 31 generates a so-called rough output sentence.

As a method of generating an output sentence, besides the method using a template, for example, a method based on a case structure can be adopted.

When the output sentence is generated, the text sentence generation unit 31 performs morphological analysis and syntax analysis on the output sentence, and extracts information necessary for the rule speech synthesis performed by the rule synthesis unit 32 at the subsequent stage. Here, the information necessary for the rule speech synthesis includes, for example, information for controlling the position of a pause, accent and intonation, other prosody information, and phoneme information such as pronunciation of each word.

The information obtained by the text sentence generator 31 is
The text data is supplied to the rule synthesizing unit 32, and the rule synthesizing unit 32 generates speech data (digital data) of synthesized speech corresponding to the output sentence generated by the text sentence generating unit 31 using the phoneme segment database 36.

That is, the speech segment database 36 stores speech segment data in the form of, for example, CV (Consonant, Vowel), VCV, CVC, and the like.
Based on the information from the text sentence generation unit 31, the necessary sentence segment data is connected, and further, pauses, accents, intonations, etc. are appropriately added to correspond to the output sentence generated by the text sentence generation unit 31. The voice data of the synthesized sound to be generated is generated.

The rule synthesizing unit 32 is supplied with the emotion information stored in the user emotion information recording unit 9, and the rule synthesizing unit 32 is connected based on the emotion information. It controls pause, accent, intonation, and prosody information such as utterance speed and pitch frequency to be added to the phoneme segment data. That is,
In the rule synthesizing unit 32, for example, when the emotion information indicates that the user is excited, voice data of a synthesized sound with a slow and calm tone is generated. Further, for example, when the emotion information indicates that the user is likely to be happy, the rule synthesizing unit 32 also generates speech data of a synthesized sound having a fun tone.

The relationship between emotions and voices is described in, for example, Maekawa, "Transmission of Paralinguistic Information by Voice; From the Perspective of Linguistics", Proceedings of the Acoustical Society of Japan 1997 Fall Meeting 1-3. -10, pp. 381-384, September 1997, etc., the details are described.

The sound data of the synthesized sound obtained by the rule synthesizing unit 32 is supplied to a DA (Digital Analog) converting unit 33,
Then, it is converted into an audio signal as an analog signal.
This audio signal is supplied to the audio output unit 5, whereby a synthesized sound corresponding to the output sentence generated by the text sentence generation unit 31 is output.

Next, FIG. 6 shows an example of the configuration of the user emotion information updating unit 8 of FIG.

The prosody information output by the voice recognition unit 2 is output to the prosody information processing unit 41, the concept information output by the dialogue management unit 3 is output to the concept information processing unit 42, and the face image information output by the image input unit 6 is output by the image information The physiological information output from the physiological information input unit 7 to the processing unit 43 is supplied to the physiological information processing unit 44.

The prosody information processing section 41 estimates the user's emotion by processing the prosody information supplied thereto, and outputs the emotion information as the estimation result to the update processing section 45.
Output to

As a method for estimating the emotion of the user from the prosodic information of the voice of the user, for example, a method described in Japanese Patent Application Laid-Open No. H10-55194 can be used.

The concept information processing section 42 estimates the user's emotion by processing the concept information supplied thereto, and outputs the emotion information as the estimation result to the update processing section 45.
Output to That is, the concept information processing unit 42 counts, based on the concept information, the frequency of occurrence of words belonging to concepts representing respective emotions such as “joy” and “anger” in a dialog between the user and the dialog system. Then, concept information processing section 42 estimates the emotion of the user based on the appearance frequency, and outputs emotion information as the estimation result to update processing section 45.

The image information processing section 43 estimates the emotion of the user by processing the face image information supplied thereto, and sends the emotion information as the estimation result to the update processing section 45.
Output to

FIG. 7 shows the image information processing section 43 of FIG.
Is shown.

[0092] The face image information is supplied to the feature extraction unit 51, and the feature extraction unit 51 extracts the feature amount of the face image information. That is, the feature extraction unit 51 performs, for example, a wavelet (Wavelet) transform on the face image information, obtains a feature vector having a component representing a spatial frequency component as a component, and supplies this to the vector quantization unit 52.

The vector quantization unit 52 performs vector quantization on the feature vector from the feature extraction unit 51 according to the codebook stored in the codebook database 54, thereby obtaining one-dimensional symbols (columns).

That is, learning is performed in the code book database 54 by using face images in each emotional state, such as a happy state, an angry state, a surprised state, or a sad state. Is stored. Here, in order to increase the quantization accuracy, for example, a codebook for each emotion, such as a joy codebook or an anger codebook, is created and stored.

The vector quantization unit 52 vector-quantizes the feature vector from the feature extraction unit 51 according to the codebook for each emotion stored in the codebook database 54, and converts the symbol (the codebook of the codebook into a codebook). The assigned code is obtained and output to the matching unit 53. Therefore, the matching section 53 is supplied with a symbol as a vector quantization result for each emotion.

The matching unit 53 includes a vector quantization unit 5
Using the symbol from No. 2 and referring to the HMM database 55, the face image information is, for example, any of emotional states such as a happy state, an angry state, a surprised state, a sad state, and the like. Is performed to determine whether or not the face is a face.

That is, learning is performed in the HMM database 55 by using a face image in each emotional state such as a happy state, an angry state, a surprised state, or a sad state. The obtained model (HMM) for the face in each emotion is stored.

Then, matching section 53 obtains a model having the highest probability of observing the symbol sequence obtained from vector quantization section 52 by the Viterbi method. further,
The matching unit 53 estimates the emotion corresponding to the model as the user's emotion, and outputs the emotion information as the estimation result to the update processing unit 45.

Here, the matching section 53 calculates the probability of observing the symbol sequence obtained from the vector quantization section 52 for each emotion. That is, for example, the calculation of the probability of observing the symbol sequence obtained by performing vector quantization using the joy codebook is performed by using an HMM (learned using an image of a happy face). HMM for pleasure. Further, for example, the calculation of the probability of observing a symbol sequence obtained by performing vector quantization using an anger codebook is performed by using an HMM (learned using an angry face image). This is performed using an anger HMM).

As described above, methods for estimating emotions from face image information are described in, for example, Sakaguchi, Otani, Kishino, “Expression Recognition from Facial Moving Image Using Hidden Markov Model”, Journal of the Institute of Television Engineers of Japan. , VOL.49, no.8, pp.1
060-1067, August 1995, etc., the details are described.

Other methods for estimating emotions from face image information include, for example, Sakaguchi and Morishima, “Real-time recognition of basic facial expressions based on spatial frequency information”, Proceedings of the 2nd Intelligent Information Media Symposium, pp. 75-82, December 1996, etc. can also be used.

Returning to FIG. 6, the physiological information processing section 44 estimates the emotion of the user by processing the physiological information supplied thereto, and outputs the emotion information as the estimation result to the update processing section 45. I do. Here, as a method of estimating the emotion of the user from the physiological information, for example, a function representing a correlation between each emotion and physiological information such as a pulse rate and a sweating amount is statistically obtained in advance, and the function is calculated. And the like.

The update processing unit 45 includes a prosody information processing unit 41,
A final update value for updating the emotion information stored in the user emotion information recording unit 9 by comprehensively using the emotion information from the concept information processing unit 42, the image information processing unit 43, and the physiological information processing unit 44. And updates the emotion information in the user emotion information recording unit 9 with the updated value. That is, the update processing unit 45 weights and adds emotion information corresponding to each emotion from each of the prosody information processing unit 41, the concept information processing unit 42, the image information processing unit 43, and the physiological information processing unit 44, and performs normalization. Then, final emotion information corresponding to each emotion is calculated. Then, the update processing unit 45 updates the emotion information of the user emotion information recording unit 9 with the final emotion information.

Here, FIG. 8 shows the user emotion information recording section 9.
Indicates the emotion information held by. The emotion information corresponding to each emotion indicates the degree of the emotion, for example, from 0 to 1.
The larger the value, the stronger the emotion (the smaller the value, the weaker the emotion). In the update processing unit 45, such a value as emotion information is updated for each emotion.

Next, referring to the flowchart of FIG.
The processing (emotion information update processing) of the user emotion information update unit 8 in FIG. 6 will be described.

First, in step S11, the prosody information processing unit 41, the concept information processing unit 42, the image information processing unit 43, and the physiological information processing unit 44 estimate the user's emotion as described above. The emotion information as the estimation result is output to the update processing unit 45.

In step S12, the update processing unit 45 comprehensively uses the emotion information from the prosody information processing unit 41, the concept information processing unit 42, the image information processing unit 43, and the physiological information processing unit 44, and A final update value for updating the emotion information stored in the information recording unit 9 is obtained, the process proceeds to step S13, the emotion information of the user emotion information recording unit 9 is updated with the updated value, and the process ends. .

Next, the above-mentioned series of processing can be performed by dedicated hardware or software. When a series of processes is performed by software, a program constituting the software is installed in a general-purpose computer or the like.

FIG. 10 shows a configuration example of an embodiment of a computer in which a program for executing the above-described series of processing is installed.

The program is stored in a hard disk 105 or a ROM 1 as a recording medium built in the computer.
03 can be recorded in advance.

Alternatively, the program may be a floppy disk, CD-ROM (Compact Disc Read Only Memory), M
O (Magneto optical) disc, DVD (Digital Versatile)
Disc), a magnetic disk, a semiconductor memory, or another such removable storage medium 111, which can be temporarily or permanently stored (recorded). Such a removable recording medium 111 can be provided as so-called package software.

The program may be installed in the computer from the removable recording medium 111 as described above, or may be wirelessly transferred from a download site to the computer via an artificial satellite for digital satellite broadcasting, or transmitted to a LAN (Local Area). Network), the Internet, and the like, and can be transferred to a computer by wire. In the computer, the transferred program can be received by the communication unit 108 and installed on the built-in hard disk 105.

The computer has a CPU (Central Processing).
Unit) 102. The CPU 102 has a bus 1
01, an input / output interface 110 is connected. The CPU 102 receives a command via the input / output interface 110 by operating the input unit 107 including a keyboard, a mouse, and the like. Then, according to it, ROM (Read Only Memory)
The program stored in 103 is executed. Alternatively, the CPU 102 transmits the program stored in the hard disk 105, a satellite, or a network, receives the program by the communication unit 108, and
The program installed in the hard disk 105 is read from the removable recording medium 111 installed in the drive 109 and loaded into the RAM (Random Access Memory) 104 and executed. Accordingly, the CPU 102 performs the processing according to the above-described flowchart or the processing performed by the configuration of the above-described block diagram. And
The CPU 102 transmits the processing result to an LCD (Liquid
Output unit 1 consisting of CryStal Display) and speakers
06, or transmitted from the communication unit 108, and further recorded on the hard disk 105.

Here, in the present specification, processing steps for describing a program for causing a computer to perform various processes do not necessarily need to be processed in chronological order in the order described in the flowchart, and may be performed in parallel. Alternatively, it also includes processing executed individually (for example, parallel processing or processing by an object).

The program may be processed by one computer, or may be processed in a distributed manner by a plurality of computers. Further, the program may be transferred to a remote computer and executed.

As described above, since the user's emotion is estimated at least based on the concept of the phrase included in the user's speech recognition result, the user's emotion can be estimated relatively accurately. . Further, since the user's emotion is estimated based on the prosody information, face image information, and physiological information, the user's emotion can be estimated more accurately. Furthermore, since the output sentence is generated based on the estimation result of the emotion, it is possible to provide the user with a variety of output sentences depending on the state of the emotion of the user.

In the present embodiment, voice recognition is performed for the sound (voice) input to the voice input unit 1. However, voice recognition is performed for the sound input to the voice input unit 1. Instead, for example, it is also possible to detect that the sound is a sound of striking a desk or a breath of the user, and to estimate the emotion of the user based on the detection result. That is, for example, when it is continuously detected that the user is hitting the desk, it can be estimated that the user is angry. Further, for example, when it is detected that breathing is rough, it can be estimated that the user is excited. Then, in this case, based on such an estimation result, an ad hoc (ad hoc) that increases the value of emotion information representing “anger” or “excitation” is increased.
c) Applicable update rules can be applied.

Further, the dialogue management section 3 controls the number of output sentence generations in accordance with the emotional state,
It is possible to change the number of utterances to the user. Specifically, for example, when the user is in a fun state, for example, the number of reciprocations and the number of utterances from the interactive system are increased, and the user actively interacts with the user. It is possible to do so. Further, for example, when the user is in a sad state, the number of utterances from the interactive system can be reduced so that the user does not feel troublesome.

Further, in the present embodiment, the voice from the user is recognized and the utterance is made as a response to the voice recognition result. However, for example, the voice is input by the user operating the keyboard. It is also possible to make a response to a sentence.

Further, in the present embodiment, the response to the user and the like are output as synthesized sounds.
For example, it is also possible to display in text or the like.

Further, the present invention can be used, for example, as a user interface between a user and a virtual character displayed on a display, or a real robot or the like. In this case, in addition to outputting a synthesized sound as described above as a response to the user, a multi-modal interface is realized by changing the display state of the virtual character or causing the robot to perform a predetermined operation. Can be.

[0122]

According to the dialogue processing apparatus, the dialogue processing method, and the recording medium of the present invention, the concept of a phrase input by the user is extracted, and the emotion of the user is estimated based on the concept. Then, an output sentence to be output to the user is generated based on the emotion information obtained as a result. Therefore, for example, it is possible to perform a variety of dialogues depending on the emotional state of the user.

[Brief description of the drawings]

FIG. 1 is a block diagram illustrating a configuration example of an embodiment of a dialogue system to which the present invention has been applied.

FIG. 2 is a flowchart illustrating a process of the dialogue system in FIG. 1;

FIG. 3 is a block diagram illustrating a configuration example of a speech recognition unit 2 in FIG. 1;

FIG. 4 is a block diagram illustrating a configuration example of a dialog management unit 3 of FIG. 1;

FIG. 5 is a block diagram illustrating a configuration example of a sentence generation unit 4 of FIG. 1;

FIG. 6 is a block diagram showing a configuration example of a user emotion information updating unit 8 of FIG. 1;

FIG. 7 is a block diagram illustrating a configuration example of an image information processing unit 43 in FIG. 6;

FIG. 8 is a diagram showing emotion information.

FIG. 9 is a flowchart illustrating a process of a user emotion information updating unit 8 of FIG. 6;

FIG. 10 is a block diagram illustrating a configuration example of a computer according to an embodiment of the present invention.

[Explanation of symbols]

1 voice input unit, 2 voice recognition unit, 3 dialogue management unit, 4 sentence generation unit, 5 voice output unit, 6 image input unit, 7 physiological information input unit, 8 user emotion information update unit, 9 user emotion information recording unit, 11 AD conversion unit, 12 feature extraction unit, 13 matching unit, 1
4 acoustic model database, 15 dictionary database, 16 grammar database, 21 language processing unit,
22 dialogue processing section, 23 thesaurus database, 24 language processing database, 25 history database, 26 scenario database, 31
Text sentence generator, 32 rule synthesizer, 33 DA
Conversion unit, 34 dictionary database, 35 grammar database for generation, 36 phoneme segment database, 41
Prosody information processing unit, 42 concept information processing unit, 43 image information processing unit, 44 physiological information processing unit, 51 feature extraction unit, 52 vector quantization unit, 53 matching unit, 54 codebook database, 55 HM
M database, 101 bus, 102 CPU,
103 ROM, 104 RAM, 105 hard disk, 106 output unit, 107 input unit, 108
Communication unit, 109 drive, 110 input / output interface, 111 removable recording medium

 ──────────────────────────────────────────────────続 き Continuing on the front page (72) Inventor Sachi Tanaka 6-7-35 Kita-Shinagawa, Shinagawa-ku, Tokyo Inside Sony Corporation (72) Inventor Jun Yokono 6-35, Kita-Shinagawa, Shinagawa-ku, Tokyo Inside Sony Corporation (72) Inventor Toshio Oe 6-7-35 Kita Shinagawa, Shinagawa-ku, Tokyo Sony Corporation F-term (reference) 5D015 AA06 LL07 LL10 5D045 AB01 AB07 AB30 9A001 DZ11 FF03 HH17 HH18 HH33

Claims (16)

[Claims] 1. A dialogue processing device for performing a dialogue with a user, comprising: a concept extracting means for extracting a concept of a phrase input by the user; A dialogue process comprising: an emotion estimation unit that estimates an emotion and outputs emotion information representing the emotion; and an output sentence generation unit that generates an output sentence to be output to the user based on the emotion information. apparatus. 2. The interaction processing apparatus according to claim 1, wherein the emotion estimation unit estimates the emotion of the user based on the output sentence. 3. The interaction processing apparatus according to claim 1, wherein the emotion estimation unit estimates the emotion of the user based on an image obtained by imaging the user. 4. The interaction processing apparatus according to claim 1, wherein the emotion estimation unit estimates the emotion of the user based on a physiological phenomenon of the user. 5. An audio processing unit for processing an audio signal obtained from the outside, wherein the emotion estimation unit estimates the emotion of the user based on a processing result of the audio processing unit. The interactive processing device according to claim 1. 6. The apparatus according to claim 1, further comprising voice recognition means for recognizing said user's voice, wherein said concept extracting means extracts a concept of a phrase included in a voice recognition result of said user's voice. An interactive processing device according to claim 1. 7. The interaction processing apparatus according to claim 6, wherein the emotion estimation unit estimates the emotion of the user based on prosodic information of the voice of the user. 8. The interaction processing apparatus according to claim 1, wherein the output sentence generation unit changes the expression of the output sentence based on the emotion information. 9. The interaction processing apparatus according to claim 1, wherein the output sentence generating means changes the number of the output sentences based on the emotion information. 10. The interaction processing apparatus according to claim 9, wherein the output sentence means a mutual connection. 11. The apparatus according to claim 11, further comprising a storage unit configured to store the emotion information, wherein the output sentence generation unit generates the output sentence based on the emotion information stored in the storage unit. Item 2. The interactive processing device according to Item 1. 12. The apparatus according to claim 1, further comprising output sentence output means for outputting the output sentence. 13. The interactive processing apparatus according to claim 12, wherein the output sentence output unit outputs the output sentence as a synthesized sound. 14. The apparatus according to claim 13, wherein the output sentence output unit controls the prosody of the synthesized sound based on the emotion information. 15. A dialogue processing method for performing a dialogue with a user, comprising: a concept extracting step of extracting a concept of a phrase input by a user; An emotion estimation step of estimating a user's emotion and outputting emotion information representing the emotion, and an output sentence generation step of generating an output sentence to be output to the user based on the emotion information. Interaction method. 16. A concept extracting step for extracting a concept of a phrase input from a user, the recording medium storing a program for causing a computer to perform an interaction process for performing a dialog with a user; An emotion estimation step of estimating the emotion of the user based on the concept of the phrase input from the user and outputting emotion information representing the emotion; and generating an output sentence output to the user based on the emotion information. A recording medium characterized by storing a program having an output statement generating step of performing the following.