KR20140067678A - Method for supplying consulting communication and apparatus using the method - Google Patents

Abstract

Disclosed are a method for supplying a consultation conversation and an apparatus using the same. According to the present invention, the consultation conversation apparatus is configured to comprise a voice recognition module for recognizing and converting the utterance voice to a text; an emotion detection module for analyzing the emotion of a user from an image of the user or the text of the user′s voice outputted by the voice recognition module; and a utterance understanding module for analyzing the intent of the utterance by grasping the travel of the user utterance from the utterance text of the user and analyzing a fact which causes the emotion of the user with five Ws and one H; and a conversation processing module for selecting the analyzed travel by the user utterance and a conversation strategy of the user by the five Ws and one H. Therefore, the present invention can provide a conversation consultation to understand emotions.

Description

TECHNICAL FIELD [0001] The present invention relates to a method and apparatus for providing a consultation dialogue,

More particularly, the present invention relates to a method for providing a consultation dialogue through a virtual human consultation dialogue for recognizing emotions and an apparatus using the same.

A virtual human listener system is a system that listens to user utterances. The virtual human celadon system was developed to analyze human emotions by analyzing human speech, intonation, facial expressions, and gestures, and output appropriate responses, gestures, and facial expressions according to the results of analysis.

The virtual human listener system, however, does not share conversations other than utterances that encourage the user to respond appropriately and continue conversation, rather than talking. An example of such a system is the SEMAINE system.

However, the existing celadon system provides appropriate response and response, but it is not enough to report only the output according to the user's emotion, Do.

The existing dialogue system is also a form of dialogue or command-oriented dialogue, not an emotional dialogue. In addition, most of the conversations are handled by specific devices, or conversations in the form of chats are the main part, which does not really help human life.

The existing celadon system is not focused on understanding the user's story because it focuses on responding appropriately rather than conversationally.

It is an object of the present invention to overcome the above-described problems and provide a method of providing a consultation dialogue that provides a user with an understanding of a user's feelings and a dialogue about what causes the feelings.

Another object of the present invention is to provide a consultation dialogue providing apparatus which provides consultation dialogue using the above method.

According to an aspect of the present invention, there is provided a consultation dialogue apparatus including a speech recognition module for recognizing a speech and converting the speech into text, a user's voice, An emotion detection module for analyzing the user's emotions, a speech recognition module for analyzing the utterance intention by grasping the running of the user utterance from the utterance text of the user, analyzing the factors causing the emotions of the user as six principle, And a conversation processing module for selecting a conversation strategy for the user from the running and 6-principle information analyzed in the speech.

The consultation dialogue apparatus may further include a response generation module for generating a response to the user by using the conversation strategy selected by the conversation processing module and the emotion of the user output by the emotion detection module.

The response generation module may select a conversation template selected by the conversation processing module and a conversation template to be uttered to the user using the emotion of the user analyzed by the emotion detection module, and generate a speaking attitude.

The consultation dialogue apparatus further includes a dialog template database for storing at least one conversation template data generated by the response generation module.

Here, the consultation dialogue apparatus further includes a virtual human output module for expressing and outputting a virtual human using the result output by the response generation module.

Here, the emotion detection module may include a feature extraction module for extracting data necessary to sense a user's emotion, a feature analysis module for analyzing user behavior using data output from the feature extraction module, And an emotion analysis module for analyzing the emotion using the resultant values of the data on the emotion coordinates and the voice recognition module outputted by the feature synthesis module.

Meanwhile, the emotion detection module may also extract the emotion keyword existing in the user's utterance text to determine the emotion of the user.

The dialog processing module may select a conversation strategy by deriving a user's response through an additional question and analyzing a user's voice response if there is an unfulfilled slot among the slots storing data related to the sixth principle.

The conversation strategy may include one or more of a listening strategy, a questioning strategy, a follow-up strategy, and an empathizing strategy.

According to another aspect of the present invention, there is provided a method for providing a consultation dialogue comprising: a speech recognition step of recognizing a speech and converting the speech into text; An emotion detection step of analyzing the user's emotions from the spoken text, an utterance understanding step of analyzing the utterance intention by grasping the running of the user utterance from the utterance text of the user, and analyzing the factors that cause the emotions of the user, And a dialogue processing step of selecting a dialogue strategy for the user from the running and the six-way information analyzed in the user utterance.

The consultation dialogue providing method may further include generating a response to the user using the conversation strategy selected by the dialogue processing module and the emotion of the user output by the emotion detection module.

According to the consultation dialogue apparatus of the present invention as described above, not only a proper response but also appropriate questions, emotional verbalization, and emotional confirmation can be provided to the user, thereby giving the user a feeling that the system understands himself.

In addition, the system judges the complex emotions that could not be expressed by speech or text alone, so that a more natural and complex emotional conversation is possible.

In addition, using virtual human beings can provide an experience that seems to converse with other personality by using appropriate attitudes and gestures, and this experience can induce the user to feel more psychological by making the system more speech .

1 is a block diagram of a consultation dialog system according to the present invention.
Fig. 2 shows an operation sequence of the dialog processing module according to the present invention.
3 is a block diagram of an emotion detection module according to an embodiment of the present invention.

While the invention is susceptible to various modifications and alternative forms, specific embodiments thereof are shown by way of example in the drawings and will herein be described in detail.

It should be understood, however, that the invention is not intended to be limited to the particular embodiments, but includes all modifications, equivalents, and alternatives falling within the spirit and scope of the invention.

The terminology used in this application is used only to describe a specific embodiment and is not intended to limit the invention. The singular expressions include plural expressions unless the context clearly dictates otherwise. In the present application, the terms "comprises" or "having" and the like are used to specify that there is a feature, a number, a step, an operation, an element, a component or a combination thereof described in the specification, But do not preclude the presence or addition of one or more other features, integers, steps, operations, elements, components, or combinations thereof.

Unless defined otherwise, all terms used herein, including technical or scientific terms, have the same meaning as commonly understood by one of ordinary skill in the art to which this invention belongs. Terms such as those defined in commonly used dictionaries should be interpreted as having a meaning consistent with the meaning in the context of the relevant art and are to be interpreted in an ideal or overly formal sense unless explicitly defined in the present application Do not.

Hereinafter, preferred embodiments of the present invention will be described in detail with reference to the accompanying drawings. In order to facilitate the understanding of the present invention, the same reference numerals are used for the same constituent elements in the drawings and redundant explanations for the same constituent elements are omitted.

The present invention relates to a method for providing a virtual human counseling dialog system capable of natural consultation with a human being. The present invention relates to a method for providing a virtual human counseling conversation system in which a user can grasp the situation of a user, Analyze and understand users with appropriate questions. The present invention also relates to a consultation dialogue system that provides consultation for understanding the emotion to the user based on this understanding.

The consultation dialog system according to the present invention grasps the user's emotions through facial expressions, gestures, intonation, and word selection of the user, and can understand what causes the emotions through utterance. To this end, the consultation dialogue system according to the present invention extracts features from data such as voice, facial expression, and the like, and determines the user's emotional state using the extracted data. The consultation dialogue system also provides appropriate questions in order to specify the cause of the emotions through the main questions in the middle of the conversation, outputs the utterance confirming the user 's answer according to the appropriate questions, induces sympathy, .

The counseling dialogue system needs a response in the form of inducing or synchronizing the midterm utterance by expressing "Yes, yes" or "Really? Also, by outputting the utterance in the form that the emotions and causes of the analyzed user are sympathetic or comprehended, the emotions are expressed by the user, and the emotion is grasped properly.

1 is a block diagram of a consultation dialogue apparatus according to the present invention.

The consultation dialogue apparatus according to the present invention shows a block configuration of a consultation dialogue apparatus which recognizes the emotion and speech of the current user and leads a dialogue so as to switch the emotion of the current user to a direction desired by the user.

1 includes a speech recognition module 200, a speech understanding module 210, a dialog processing module 220, an emotion detection module 100, a response generation module 300, a virtual human output module 310, and a dialog template database 400. [

The speech recognition module 200 interprets the user's utterance as sound data. That is, the voice recognition module is a module that converts the voice of the user into text.

The ignition comprehension module 210 analyzes the intention of the user by grasping the running of the user's utterance. Driving defines the utterance before recognizing it, and classifies the utterance expressing the emotion or utterance to explain the cause. Also, the factor that causes the user 's emotions in the user' s utterance is analyzed by the six - principle, and the data is appropriately stored in the six - principle slot.

Here, the six sub-slots are six slots: "who", "when", "where", "what", "how", and "why". For example, if the word "puppy is hit by a car and die" comes in, "Luke" becomes "puppy", "how" is "dead" and "why" is "hit by car".

The method of analysis according to the sixth principle can be a method of training and analyzing corpus tagged with utterance according to the sixth principle. In addition, analysis using semantic role labeling technique can do.

The emotion detection module 100 is a module for analyzing the state of the user's emotions by receiving voice and image data of the user. The emotion detection module 100 analyzes emotion coordinates obtained by analyzing the user's utterance contents, words, intonation, facial expression, gesture, To determine the user's feelings. The emotion determined here affects the operation of the response generation module together with the result of the dialog processing module. The detailed configuration of the emotion detection module 100 will be described in detail with reference to FIG.

The dialog processing module 220 uses the results of the speech understanding module to determine what conversation strategy to select. The conversation strategy generated by the conversation processing module 220 according to an embodiment of the present invention may have four types. The first strategy is listening, the second strategy is asking, the third strategy is speaking along, and the fourth strategy is emotional confirmation. The dialog processing module 220 will be described in more detail below with reference to FIG. 2.

The response generation module 300 generates an appropriate speech using the conversation strategy and the user's emotions, and determines how to set the attitude of the virtual human. Specifically, the response generation module 300 generates utterances and attitudes using the result of the generated dialog processing module and the emotion result value. The generated utterances and attitudes are output through the virtual human output module 310, which is an attitude formation module, in the form of utterance, facial expression, and gesture.

The response generation module 300 generates a dialogue template database by storing data acquired in the above-described operation process.

Conversation template data stored in the conversation template database 400 means a template conversation for appropriate questions, appropriate follow-up, or emotional confirmation. An example of a conversation template is "When did <who> <how> did?".

The method of generating the dialogue template database can be generated by using the emotion of the user, the system utterance strategy, and the six lower principle slots. After analyzing which slot exists in which slot, which slot is not present, and what combination of questions to ask, a conversation template can be created.

For example, if you have <who> and <how> and you want to know <when>, you can create a template with the question <when> <how> did?>. Also, for example, in a situation where knowing <who> and <what>, a user selects a speech strategy for asking information of a <where> slot, and if the user's feelings are anxious, > Did you get angry? You can create a template as a question.

The response generation module 300 is a module for receiving the conversation strategy output from the conversation processing module and the user's emotion as inputs and generating appropriate utterances and attitudes.

When the dialog processing module 220 selects a well-known strategy, it outputs the stored coincidence corpus. In this case, the attitude of the system may be changed according to the user's attitude, such as eye contact, It is an attitude that gives a feeling of concentration.

In the case of a question, refer to the dialog template database. It is necessary to determine which 6 sub-principle slot values have no value and select a dialogue template corresponding to them. For example, if you hear the user's utterance of "the puppy is dead", "Luke" will be the puppy, "how" will die, and to fill the slot "why" ? " Type template and the filled-in slot data of the 6th principle. Which template is selected uses the result value of the dialog processing module 220. When taking this strategy, the attitude of the system is curiosity or anxious attitude because it is in the position of the question. This includes, for example, glaring at the head or raising the end of a question.

In the case of speaking, the utterance is formed by a combination of the six hypothesis slots, and the dialog processing module receives information on which combination is to be uttered. In conversation, take a conversation template similar to a question and speak. For example, "How did <who>"? The template is used to generate the utterance.

The strategy for identifying emotions is a strategy for generating utterances that understand the user 's emotions using the data of the filled six hypothetical slots, sensed user emotions, and dialogue templates. For example, "Oh, so you can use the template <emotion> because <who> is <why> and <when> <how> <where> in <where>. The attitude of the system at this time should be changed according to the user's feelings. If the user 's feelings are' joy ', they should express' joy'. If they are 'joy', they should express' joy 'and if they are' joy 'they should express joy. If you are in a 'gentle' position, you should reassure the user.

The response generation module 300 also outputs facial expressions, beeps, attitudes, hand gestures, and intonation as well as the system utterance. The output of the response generation module 300 is then used in the virtual human output module 310.

The virtual human output module 310 outputs the voice, mouth shape, gesture, etc. of the virtual human on the basis of the attitude and utterance determined by the response generation module 300.

Specifically, the virtual human output module 310 expresses a virtual human using the result output from the response generation module. The virtual human output module 310 outputs a textual utterance as a voice output, and outputs a mouth shape corresponding to the voice output as an image do.

The virtual human output module 310 may also output an eye, an eyebrow, a mouth, a beep, a hand motion, etc. for effective emotional expression, and may also express an accent to express emotion by voice.

The virtual human counseling dialogue apparatus according to the present invention constructed as described above is also confirmed by uttering an appropriate question to the user and understanding that the user's utterance is well understood. Finally, the emotion of the user is emotionally understood by making a speech that understands how and where the emotion of the user originated.

Fig. 2 shows an operation sequence of the dialog processing module according to the present invention.

The dialog processing module 220 selects the conversation strategy using the output of the speech understanding module. There are four types of conversation strategies that the conversation processing module generates. The first strategy is listening, the second strategy is asking, the third strategy is speaking along, and the fourth strategy is emotional confirmation.

The first strategy is to listen to the user and give the impression that he or she is focusing on the user's utterance and he / she is listening to the user by "responding" or " It is a strategy to instill the impression that 'I am concentrating.' This first strategy is expressed in the middle of the conversation.

The second strategy, Queries, is based on how much content is stored in the Sixth Principles Slot. The sub-slots are populated by consultation with the user. If an empty slot is found because the slot is not full, a question is asked to obtain information about the slot. These questions are generated in the response generation module, but which slots are to be queried are handled in the dialog processing module.

For example, to see why a puppy died, the answer would be ask_why. If the user has uttered an utterance that does not contain information for any of the six hypothetical slots, such as "sad," then what would happen if the "what happened" to induce the utterance of any six hypothetical slots? Select the question of the expression.

The third strategy is to introduce the user to the system as 'listening' or 'following the story', and at the same time, as a strategy to receive modification from the user if they misunderstand. The following speaking strategy generates a talking utterance with the contents of the user utterance after the question. In the dialog processing module, a strategy of what slot information is to be combined and output is also included as information. For example, if a dog is killed in a traffic accident, the dog is dead? Or 'Was it a traffic accident?' And a variety of strategies depending on the combination of slots.

The fourth strategy, emotional confirmation strategy, is to grasp the emotions of the user through emotional expressions in the user's utterances, voices, gestures, facial expressions, etc., and to empathize with the users by using the data of the six hypothetical slots Strategy. It is also a strategy to ensure that you understand your utterances well. This strategy is expressed when all 6 sub-prime slots are stored as data.

Hereinafter, an operation procedure of the dialog processing module based on the strategy described above will be described with reference to FIG.

The conversation processing module uses the output of the ignition comprehension module to inquire about inducing emotion for an empty slot among the six principle (S210). For example, "How are you feeling today?" It can be a question. Upon receiving the user input for the question (S220), the user's emotion is detected and then an appropriate question is given to the situation (S230). For example, if the user says "sad", "What's up?" Etc., to obtain additional information from the user.

again. Upon receipt of the user input for the question (S240), it is determined whether the user input satisfies the six principle (S250).

The input of the user mainly appears in the form of an utterance from the user, and a strategy such as the first strategy (S200) can be used for the utterance of the user. For example, you can create an atmosphere in which the user can additionally speak with "chanting, continue" and so on. A listening strategy can be performed whenever there is a user utterance throughout the conversation.

On the other hand, if the input of the user is found to be in accordance with the sixth principle, the speech is performed (S260). For example, it speaks to the utterance of use by saying "I am sad" or "I am dead yesterday" to express the understanding to the user.

If the input of the user is found to be incompatible with the sixth principle, the user further inputs the input through the additional question (S251). That is, when the user answers "Oh, sad" in contradiction with the sixth principle, he or she can induce the user to further emotional expression through an additional question such as "when the dog died".

Thereafter, it is determined whether there are any empty slots based on the user's inputs (S270). If the slots are not full, the user is further questioned regarding the empty principle (S271). For example, "When did the puppy die?" Etc. to guide the user to answer the contents according to the six principles.

On the other hand, if all six sub-slots are full, the user's emotions are finally confirmed (S280). For example, he expresses that he understands the emotion of the user through expressions such as "I am sad because the dog is hit by a car".

2, the information corresponding to the sixth principle of data input by the user, for example, "yesterday", "puppy", "crosswalk", "hit by car", " The information such as "because the driver has not seen it" is stored in the 6th principle slot every time the information is acquired.

3 is a block diagram of an emotion detection module according to an embodiment of the present invention.

The emotion detection module 100 according to the present invention may include a feature extraction module 110, a feature analysis module 120, a feature synthesis module 130, and an emotion analysis module 140.

The feature extraction module 110 extracts features that may be helpful in detecting emotion in the user's video and audio data as data values. Here, the characteristic may be, for example, the coordinates of a human face, the frequency of a voice, and the like.

The feature analysis module 120 analyzes the behavior of the user using the data value output from the feature extraction module 110. For example, if the coordinate value of the face position of a person is captured from left to right, analyze 'moving the head to the left or right'.

The feature synthesis module 130 is a module for displaying the user's emotions on the emotion coordinates using the result of the feature analysis module. When the emotion of a person is expressed in coordinates, it is outputted to which coordinates the emotion of the current user is located.

The emotion analysis module 140 analyzes the emotion using the data on the coordinates of the emotion from the feature combination module and the value output from the voice recognition module 200. The emotion analysis module 140 sets the reference emotions as reference coordinates and selects a method of analyzing the emotion of the user with the nearest reference emotion among the distances between the reference coordinates and the coordinates detected by the feature synthesis module.

However, according to another embodiment of the emotion detection module 100, when the user utters the emotional keyword such as "sad" or " To analyze whether there is an emotional keyword in the user and to finally determine the emotion of the user.

It will be understood by those skilled in the art that various changes in form and details may be made therein without departing from the spirit and scope of the invention as defined in the appended claims. It will be possible.

100: Emotion detection module 110: Feature extraction module
120: Feature analysis module 130: Feature synthesis module
140: Emotion analysis module 200: Voice recognition module
210: ignition understanding module 220: dialog processing module
300: response generation module 310: virtual human output module
400: Conversation Template Database

Claims (20)

A speech recognition module for recognizing the uttered speech and converting it into text;
An emotion detection module for analyzing a user's emotion from a user's image or a user's voice text output by the voice recognition module;
A spoken understanding module for analyzing the intention of the user by grasping the running of the user's utterance from the user's utterance text and analyzing the factors causing the user's emotions based on the six principle; And
And a dialogue processing module for selecting a dialogue strategy for the user from the running and analyzed information from the user's utterance. The method according to claim 1,
Further comprising a response generation module for generating a response to the user using the conversation strategy selected by the conversation processing module and the emotion of the user output by the emotion detection module. The method of claim 2,
The response generation module includes:
Wherein the conversation processing module selects a conversation template to be uttered to the user using the conversation strategy selected by the conversation processing module and the emotion of the user analyzed by the emotion detection module, and generates a speaking attitude. The method of claim 2,
And a dialog template database for storing at least one dialog template data generated by the response generation module. The method of claim 2,
And a virtual human output module for outputting a virtual human using the result output from the response generation module. The method according to claim 1,
Wherein the emotion detection module comprises:
A feature extraction module for extracting data necessary for sensing the user's feelings;
A feature analysis module for analyzing a user's behavior using data output from the feature extraction module;
A feature synthesis module for displaying the emotion of the user on the emotional coordinates using the output of the feature analysis module; And
And an emotion analysis module for analyzing emotion using data on emotion coordinates output by said feature synthesis module. The method of claim 6,
The emotion analysis module includes:
And sets the reference coordinates for the one or more reference emotions and analyzes the emotion of the user with the closest reference emotion among the coordinates outputted by the feature synthesizing module and the distance between the reference coordinates. The method according to claim 1,
Wherein the emotion detection module comprises:
And extracting emotional keywords present in the user's utterance text to determine emotions of the user. The method according to claim 1,
Wherein the dialog processing module comprises:
Wherein if there is an unfulfilled slot among the slots for storing the data related to the sixth principle, the user selects a conversation strategy by deriving a response from the user through the additional question and analyzing the voice response of the user. . The method of claim 8,
The conversation strategy,
A counselor dialog, comprising one or more of a listener strategy, a questioning strategy, a follow-up strategy, and an empathizing strategy. A voice recognition step of recognizing the uttered voice and converting it into text;
An emotion detecting step of analyzing a user's emotion from a user's image or a user's voice text output by the voice recognition module;
An ignition understanding step for analyzing the intention of the user by grasping the running of the user utterance from the user's utterance text and analyzing the factors causing the user's emotion by the six principle; And
And a dialogue processing step of selecting a dialogue strategy for the user from the running and analyzed information of the six lower principles analyzed by the user utterance. The method of claim 11,
Further comprising generating a response to the user using the conversation strategy selected by the conversation processing module and the emotion of the user output by the emotion detection module. The method of claim 12,
Wherein generating a response to the user comprises:
Selecting a conversation template to be uttered to the user using the selected conversation strategy and the emotions of the analyzed user and generating a speaking attitude. 14. The method of claim 13,
Further comprising storing at least one dialog template data to create a dialog template database. The method of claim 12,
Further comprising the step of outputting a virtual human using a response to the user. The method of claim 11,
Wherein the emotion detecting step comprises:
Extracting characteristic data necessary for sensing the user's emotion;
Analyzing a user's behavior using the extracted feature data;
Displaying the user's emotions on the emotional coordinates using the user behavior analysis; And
And analyzing emotions using data on the emotional coordinates. The method of claim 11,
Wherein the emotion detecting step comprises:
Setting reference coordinates for the one or more reference emotions and analyzing the user's emotions with a reference emotion that is closest to a distance between coordinates of the user's emotions and the reference coordinates. The method of claim 11,
Wherein the emotion detecting step comprises:
And extracting a feeling keyword existing in the user's utterance text to judge the emotion of the user. The method of claim 11,
Wherein, in the dialog processing step,
Selecting a conversation strategy by analyzing a user's voice response and inducing a user's response via an additional question if there is an unfulfilled slot among the slots storing data related to the sixth principle; Way. The method of claim 11,
The conversation strategy,
A method of providing a consultation dialogue comprising at least one of a listening strategy, a questioning strategy, a follow-up strategy, and an emotional verification strategy.

Images