JP2005237668A - Interactive device considering emotion in computer network - Google Patents

Abstract

<P>PROBLEM TO BE SOLVED: To provide a physiological information abnormality detecting device capable of not only detecting physical abnormality but also expressing the situation such as the delicate emotional change and the simultaneous rise of a plurality of emotions, and a care support system capable of providing effects of mental care. <P>SOLUTION: The interactive device extracts a plurality of emotions from the image of the face, uttered voice, physiological information and language information obtained from a user. If an abnormality is found in the extracted emotions or physiological information, the user, a server, and a health management institution are informed of the detected abnormality; and if the calculated psychological state of the user is not psychologically desirable, such a response as to soften the psychological state is returned. As the interactive device has the functions mentioned above, the face-to-face natural language interactive communication and the abnormality detection function on both mental and physical phases can be achieved. <P>COPYRIGHT: (C)2005,JPO&NCIPI

Description

The present invention detects abnormalities of a user's physiological state and psychological state through interactive communication with a human being in a computer connected to the Internet, notifies a health care organization, and also detects those states through interactive communication. The present invention relates to an interactive device that takes into account emotions intended to ease.

As a conventional abnormality detection apparatus, an apparatus is known that monitors physiological information of a subject and automatically contacts the nearest institution when an abnormality occurs (see, for example, Patent Document 1). This conventional abnormality detection device monitors pulse information and blood pressure information from a pulse sensor and a blood pressure sensor built in a bracelet-shaped envelope, and detects a greatly different value from registered normal information. This is to contact the organization (see FIG. 37).

However, since physiological information is greatly influenced by the emotional state of the subject such as anger and sadness, there is a problem that many false alarms occur when the difference from the normal value is simply used as a condition as in these methods.

Moreover, there is a risk that an abnormality in the psychological aspect of the subject or a state requiring support may be missed only by crisis management from the physical aspect of the subject. In particular, when it is assumed that a caregiver is supported by a machine due to a shortage of caregivers due to a declining birthrate and an aging society, leaving a psychological abnormality just because it is a machine may cause a serious problem.

As a device aiming for both mind and body care via a machine, a camera, a measuring instrument, a microphone, and a speaker are installed on the care receiver side and helper side as shown in FIG. (See Patent Document 2). However, this device does not reduce the burden on human beings, so it cannot be used as a substitute for mental care.

As an interactive device that takes emotions into account, an interactive device that determines and responds to emotions according to the personal information of each user, the situation at that time, and the content of the dialogue is known (see, for example, Patent Document 3). This conventional interactive device determines emotions from input speech, face images, language information, etc., and stores pre-stored responses based on semantic structures obtained by analyzing language information and user personal information A pattern is selected and an output sentence is generated (see FIG. 39).

However, with this method, there is only one kind of emotion to be determined, and the reaction to the user is simple and reflexive.

In view of this, we have proposed a technique called an emotion calculation method in Non-Patent Document 1 for the problem that there are few emotions to be distinguished. This method calculates pleasantness / discomfort based on the user's preference information for the content of language information issued by the user, and generates 20 types of emotions in consideration of the situation and the viewpoints of others. can do. At that time, multiple emotions can occur simultaneously.

Also, when using multiple types of information such as language information, facial expressions, physiological information, etc. for emotion recognition, there may be discrepancies in emotions analyzed from these information, so in order to recognize emotions more accurately, It is necessary to analyze emotions from each information and calculate complex emotions from them.

JP 2002-34940 (5th page, FIG. 1) JP 2001-204774 (5th page, FIG. 2) JP 2000-207214 A Kazuya Mera, “Emotion Oriented Intelligent Interface”, Doctoral Dissertation, Tokyo Metropolitan University of Science and Technology, 2003

The present invention extracts a plurality of emotions from face images, speech sounds, physiological information, and language information obtained from a user, and if any abnormality is found in the extracted emotions or physiological information, the user, server, health management organization The purpose is to notify. Furthermore, if the calculated psychological state is not very psychologically good, it is intended to return a reaction that softens it.

In an interactive device considering emotions in a computer network, means for acquiring a user's face image, speech voice, physiological information, text information through a camera, a microphone, a keyboard, a physiological information measuring device, a touch panel, etc., and emotions from those information A means for measuring and analyzing, a means for storing the value of the analysis result in the device, a means for transmitting the value to the server, and the degree of emotion (emotion value) of the user on the server side and its change over time Means for generating utterance content and means for generating a face image of a conversation person based on the value.

The interactive apparatus according to claim 1, further comprising means for notifying a medical institution or an external organization for emergency management when abnormality of physiological information and emotion information acquired from a user is detected.

2. The interactive apparatus according to claim 1, wherein first, a means for extracting each emotion analysis element based on the acquired face image, speech voice, physiological information, and text information, and a user's composite from each extracted emotion analysis element. Emotion analysis means for calculating a value representing a typical emotion.

5. The dialogue apparatus according to claim 1, further comprising means for generating and displaying a reply utterance to the user and a face image of the dialogue person on the server side in accordance with the content of the dialogue, and further storing the history.

The interactive apparatus according to claim 1, further comprising means for displaying the reply utterance and face image generated by the reply utterance and face image generating means according to claim 4 on a terminal on the user side.

The interactive apparatus according to claim 1, wherein the acquired physiological information is determined as means for determining the presence / absence of an abnormality based on a normal value stored in the system and a state of occurrence emotion according to claim 2. It has a means to notify the organization that can support the results on a daily basis.

The interactive apparatus according to claim 1, wherein means for determining the emotional unstable state of the user from the transition of the emotion value of the user obtained by the emotion analyzing means according to claim 3, and an organization that can support the determined result on a daily basis Means to notify

Dialogue by displaying sentences, voices, and facial images calculated as dialogue's emotions on the medical diagnosis, health support, and consulting systems in medical areas such as medical diagnosis support and health support. It has a graphical user interface that can express emotions.

As described above, the emotion analysis means of the present invention calculates emotion values from each of the user's face image, speech voice, physiological information, and linguistic information, and treats the integrated value as a personal emotion value. It can express multimodal emotions that could not be realized, and the co-occurrence and conflict of multiple emotions.

Physiological information anomaly detection means detect anomalies in consideration of normal values and generated emotions. Can be excluded.

Since the facial expression generation means can easily learn the facial image input from the camera and the facial image prepared by the user, it is easy to add facial expressions to the actual image. Further, even if a plurality of 20 types of emotions occur at the same time, it is possible to express facial expressions corresponding to the respective strengths.

The emotion information abnormality detection means can monitor not only the physical aspect but also the mental aspect and notification of anomalies by monitoring the transition of the personal emotion value by the server.

Furthermore, when the current emotional state is not so good, the reaction generating means can return a response that reduces the state to the user.

In addition, using the emotion-oriented intelligent dialogue device according to the present invention, it is possible to make up for a lack of communication in a care facility or care at home.

Embodiments of the present invention will be described below with reference to the drawings. A basic embodiment of the interface device of the present invention is shown in FIG. First, a user's face image is input to a client machine through a camera connected to a computer. Similarly, a user's uttered voice through a microphone connected to a computer, text information through a keyboard, a touch panel, and the like, and physiological information through a physiological information measuring device are input to the client machine. Speech speech is analyzed in a computer, and non-linguistic information such as intonation and stress and text information expressing speech content in character strings are extracted. The face image acquired from the camera is input to a neural network that learned the user's face, and each occurrence emotion and its intensity (emotion value) are calculated. The non-linguistic information acquired from the uttered voice is also input to the emotion value extraction neural network, and the emotion value is calculated. The text information acquired through speech voice, keyboard, etc. is subjected to syntactic analysis, and then the emotion value is calculated by applying an emotion calculation method (EGC). The user's physiological information acquired from the measuring device is compared with the normal value stored in the user personal computer, and then input to the emotion value extraction neural network to calculate the emotion value. If any abnormality is found when compared with the normal value, an external medical diagnosis support system is notified. The emotion values obtained by the above methods are combined into one emotion value (individual emotion value) in consideration of each characteristic. The personal emotion value is sent to the server and stored in the user personal computer as the user's emotion occurrence history. The emotion occurrence history stored in the server is compared with the personal emotion value, and if it is recognized that the situation should be supported, the emergency management system is notified.

The personal emotion value sent to the server is also accumulated as an emotion occurrence history like the user personal computer. Then, based on the transition of the current emotion and the past emotion, a face image and a response sentence are generated as a reaction that does not make the user uncomfortable. When the medical information to be commented is sent from the medical diagnosis support system, the generated face image and response text are output with priority. The face image is also made according to the comment. The generated response utterance and face image are sent to the user personal computer and presented to the user through a speaker, a display, or the like.
The facial expression image is presented to the user through the display.

The basic hardware configuration of the computer device used in the interface system is as shown in FIG. CPU, RAM, ROM, system control means, etc., microphone for inputting voice, camera for inputting user's face image, keyboard and touch panel for inputting text information, measurement for acquiring physiological information It comprises a device, storage means for storing programs and data, a display for outputting and displaying face images and data, and a speaker for outputting a reply utterance from the agent by voice.

In the physiological information abnormality detecting means of the present invention, first, physiological information such as blood pressure, myoelectric potential, skin conductivity, respiration, heart rate and the like is input from a physiological information measuring device. Next, the user's current emotion is extracted from the user's face image input from the camera, voice information input from the microphone, and text information input using a keyboard, touch panel, or the like. Then, the input physiological information is examined for the presence or absence of abnormality in consideration of the value of the normal state and the influence of the current emotion. If an abnormality is found, it is notified as an alarm to an external area / workplace / school health management network (see FIG. 4).

The inputted physiological information is converted into the following values so that the intensity and change can be captured more precisely (see Non-Patent Document 2). In this means, six types of μ _X , σ _X , δ _X , δ ′ _X , γ _X , and γ ′ _X are used for each physiological information (see Formula 1).

In this method, these values are compared with normal values. As normal values to be compared, three types are used: a normal value for ordinary people, a normal value considering the health condition of the user, and an average value from the start of the survey on that day. This is to exclude the influence of external factors such as differences in the measurement position and contact degree of the detector of the physiological information measuring device from the beginning of the normal normal value for users with chronic illness from the beginning. is there.

On the other hand, the inclination of the user's face image input to obtain the current user's emotion is first corrected by Affine transformation (see FIG. 5), and then the two-dimensional discrete cosine transformation is performed on the face image. Perform (see FIG. 6). Emotions are extracted based on the low frequency components obtained in this way.

The utterance voice includes two types of information, that is, the utterance content and the voice waveform.

From the speech waveform, emotions are extracted based on the strength and speed. At that time, six factors of μ _X , σ _X , δ _X , δ ′ _X , γ _X , and γ ′ _X used for the physiological information processing are calculated from the waveform of the input speech, and these values are used.

In the speech recognition means and the text information analysis means, first, a user's speech is input from a microphone connected to the computer, and converted into a character string through software on the computer. Next, morphological analysis and syntax analysis are performed on the obtained utterance character string to extract a semantic structure. Next, the semantic structure is converted into a deep case frame (see FIG. 7). Since the semantic structure obtained by parsing is a surface case frame expression, it must be converted into a deep case frame expression used in the following processing. The correspondence table for conversion is as shown in FIG.

In the emotion calculation means from the text information, first, the pleasantness / discomfort felt for the utterance content is calculated according to the preference information of the emotion creator. Next, more detailed 20 types of pleasure / discomfort calculated based on the situation represented by the utterance content, the likes / dislikes of others, and the pleasure / discomfort caused by others to the utterance content. Classify into emotions.

First, it is determined whether the event represented by the utterance content is pleasant or unpleasant for the user. For the discrimination, the user's likability that the agent has in advance for the case element in the event is used. For example, in the event of “I win Taro”, the calculation is performed based on the favorability of three words “I”, “Taro”, and “Win”. Then, the likability of these words is arranged as a vector on the orthogonal axis of the three-dimensional space (emotional space), and a combined vector thereof is calculated (see FIG. 9). Then, the pleasant / unpleasant feeling is discriminated as shown in FIG. The intensity of comfort / discomfort is calculated from the length of the vector.

The main elements in each predicate type are defined as shown in FIG. 11 along the predicate classification.

The calculated comfort / discomfort is determined by referring to the conditions of others' perspective, future prediction, approval / blame, etc. As you can see, there are 20 types of emotions: disappointment, pride, praise, shame, reproach, appreciation, anger, self-satisfaction, self-responsibility. The classification conditions for each emotion are as shown in FIG.

The emotions that belong to “the fate of others” are caused by how others feel when an event occurs. This is happy, laughing, jealous, sorry. There are both cases where you are pleased and uncomfortable about what is desirable for others. In this study, we judge others who have had a desirable event as joyful if they like it and uncomfortable if they don't like it. In other words, if you like something good for someone you like, you'll be happy. Based on this, using the desirability of the event for the other person and the favorability of the other person for himself, the emotion related to the fate of the other person is obtained as shown in FIG. A in the table represents some other person other than yourself. A processing procedure created based on FIG. 13 is shown in FIG.

Emotions that belong to the “future” arise from considering what is expected to happen in the future. There is hope and fear in this. If the event that you suspect is going to happen is desired, calculate the fear if it is not desired. Predicted events are stored in a predicted event list and used for evaluation of confirmation emotions. FIG. 15 shows the processing procedure.

The emotion that belongs to “confirmation” is caused by the desirability of the event for itself when the event that was predicted has occurred or has not occurred. There are satisfaction, relief, and, as feared, disappointment. If the currently recognized event is one that was previously predicted, that is, it is in the list of predicted events accumulated in future processing, or contains an “adverb that implies that it was as predicted” Calculate what you were satisfied and afraid of depending on your desirability. Also, if it becomes clear that the predicted event has not occurred by the currently recognized event, relief and disappointment are calculated according to the desirability of the predicted event. FIG. 16 shows this relationship, and FIG. 17 shows the processing procedure.

Emotions that belong to “happiness” are caused by whether or not an event is desirable to you. This has joy and suffering. If a pleasant reaction occurs in response to an event, it will be a pleasure, and if an unpleasant reaction occurs, it will cause suffering. First, let us focus on the phenomenon that causes joy. According to FIG. 12, this “certain event” includes not only an event that is desired by itself but also the following emotions included in the emotional group of the destiny, future, and confirmation of others.
・ Preferable events for yourself ・ Preferable events for others you like (happy)
・ Undesirable events for others who dislike
・ Desired future events (hope)
・ Confirmed desirable event (satisfaction)
・ Unwanted events that were not confirmed (anxiety)
Of these, the determination of “desirable for myself” may be performed by directly applying the output of the conventional emotion calculation method. Other than that, it can be judged from the emotions generated by other methods. It should be noted that the conditions described here are for comfort, and vice versa for discomfort. If an event meets both joyful conditions and suffering conditions, it is likely that there will be conflicts depending on its intensity. However, in this study, we don't consider conflicts, and we classify them as both joy and suffering. The processing procedure is shown in FIG.

The emotion that belongs to “Attribution” occurs for an actor when an event occurs. This has pride, praise, shame and reproach. For a desired event, it is pride if the person has performed the event, and praise is calculated if it is others. Also, for an undesired event, a shame is calculated if the event is performed by the user, and a reprimand is calculated if the other is performed. FIG. 19 shows this relationship, and FIG. 20 shows the processing procedure.

Emotions that belong to “happiness / attribute” occur as a combination of emotions related to happiness and emotions related to attribution. There are gratitude, anger, self-satisfaction and remorse. As shown in FIG. 21, a composite emotion is further calculated by a combination of emotions of happiness and belonging. In FIG. 21, (C) is considered that the two emotions do not combine and cause a conflict, but the conflict is not considered and is treated as an opposite emotion of the same strength. The processing procedure is shown in FIG.

In this way, in this method, one emotional occurrence triggers another emotional occurrence, and emotions are generated in a chained manner. The dependency relationship of emotion and animation is as shown in FIG. First, calculate the desirability of the event for you. If it has an aspect of the future, emotions about the future are calculated. Future calculated events can be further validated depending on whether they actually happened. On the other hand, the emotion of the other person's fate is calculated from the desirability of the event for the other person. The emotion of happiness is calculated from these future, confirmation, destiny of others and desirability of events for me. If the target event predicate is an event of type V (S, O, *), the attribution emotion is calculated based on the actor's preference for the subject. If both happiness and attribution are calculated, the happiness / attribute is calculated.

FIG. 24 shows an example of the result of applying this method and the flow of processing. In this example, if you have favorable information such as “I hate annoying neighbors” or “I do n’t like or dislike landlords”, “Yuwe landlord scolded annoying neighbors” Calculate the emotions. First, since the event type of the predicate “speak” is the verb VI, a vector is synthesized from the favorable sensitivity of the three elements of the subject, the object, and the predicate. As a result, since the composite vector exists in the region VI, this event is regarded as “good”. Furthermore, when calculating the pleasant discomfort from the viewpoint of the landlord and the neighbor for the same event, it can be seen that the neighbor feels “discomfort” for the event. Here, “smile” occurs because “disliked people feel uncomfortable”. Furthermore, “joy” arises from this “smile”. Also, because others (landers) have done what they feel comfortable with, “praise” and “thanks” to the landlords arise.

Elements for emotion extraction obtained from face images, voice information, and text information are input to a neural network (see FIG. 25), and 20 types of emotions are extracted based on the state of the intermediate layer.

In the neural network, information input to the input layer is propagated to the output layer via the intermediate layer. However, if correct learning is not performed, information to be output is different from information to be actually output. Therefore, there is a BP learning method (error back propagation method) as a method of learning the neural network itself by learning to reduce this error. In BP learning, not only forward calculation to input information to the input layer and output information to the output layer, but also backward calculation to input information to be output to the output layer and output information corresponding to the input layer . As a result, an error occurs in the network weight due to the forward calculation and the backward calculation. A technique for changing the network weight so that this error is minimized is the BP learning technique.

In general, BP learning is used for neural network learning, but we use a bias function such as Equation 3 instead of a sigmoid function such as Equation 2 in the third and fifth layers. .

ω ₂₁ represents the weight between the first layer and the second layer. The output activation for the second layer is as shown in Equation 4. X ₁ is the first layer output activation.

In the third layer, the output activation of Formula 5 is used.

Similarly, the output activations of the fourth layer and the fifth layer are expressed by Equation 6 and Equation 7, respectively.

In the neural network used in the proposed method, element values for emotion extraction obtained from face images, voice information, and text information are input to the first layer, and emotions appear as output in the third layer. Therefore, the number of neurons in the first layer and the fifth layer is 305 (255 + 30 + 20), and the number of neurons in the third layer is 20.

In this method, the neural network learns the relationship between each data and the corresponding emotion in advance. Then, element values for emotion extraction obtained from the face image, voice information, and text information are input to the first layer, and each emotion and its intensity are extracted from the third layer.

The abnormality detection means from the physiological information detects the abnormal state by comparing the emotion value and the normal value of the physiological information thus obtained with the physiological information currently acquired.

In the abnormality detection means based on physiological information, if each numerical value obtained from blood pressure, heart rate, and breathing deviates from a normal value by a certain threshold value or more, it is notified to the local / work / school health management network that there is some abnormality To do.

The alarm information that informs the abnormality is represented by a continuous value, and has the meanings of “observation required”, “warning required”, “notification required” depending on the degree.

Next, in the means for detecting abnormalities in the mental aspect, first, physiological information input from a physiological information measuring device, user's face image input from a camera, voice information input from a microphone, a keyboard, a touch panel, and the like are used. The user's current emotion is extracted from the input text information. And the presence or absence of abnormality is investigated with respect to the obtained emotion in consideration of the history of occurrence emotion and the health and mental state of the user. If an abnormality is found, it is notified as an alarm to an external area / work area / school health management network (see FIG. 26).

The means for detecting abnormalities in the physiological state did not use physiological information for emotion extraction, but this emotion information abnormality detecting means also uses physiological information for emotion extraction in order to increase the accuracy of the extracted emotion. Therefore, the number of neurons in the first layer and the fifth layer of the neural network used for emotion extraction is increased to 335.

The emotions that occur are managed and stored on both the user computer and the server. At that time, instead of discarding other than the latest emotion, the past emotion occurrence is also left as a history.

The abnormality detection means from the emotion information compares the extracted emotion value with the history of occurrence emotion, the health state and the mental state of the user, and determines whether or not there is an abnormality or a state requiring support. If there are any abnormalities, notify the local / work / school health management network.

Here, as shown in FIG. 27, the emotional state that is regarded as an abnormality or a state requiring support is, as shown in FIG. 27, “having negative emotions for a long time”, “exciting emotions strongly occurred”, “overall “Emotions are decreasing”. There is no problem with these emotional occurrences themselves, but alarms are for those emotions that could have an adverse effect on the body or suggest a deterioration in health.

The alarm information that informs the abnormality is represented by a continuous value, and has the meanings of “observation required”, “warning required”, “notification required” depending on the degree.

The alarm value is calculated using fuzzy inference based on the corresponding condition and the corresponding degree.

Fuzzy reasoning introduces "ambiguity" to the reasoning process based on propositional logic. Essentially in propositional logic, a proposition is represented by a binary value, true or false. However, in this fuzzy reasoning, the proposition can express not only the truth but also the degree. For example, an inference rule that “if a strong emotion of excitement occurs, blood pressure increases” is expressed as Equation 8.

Since the propositions such as “the occurrence emotion is excitability” and “the occurrence emotion is strong” in Formula 8 cannot be expressed by a true binary value, the degree is determined using a fuzzy set. This degree is obtained by a membership function prepared for each proposition. When there are two or more conditions in the antecedent part as in this example, the lower degree is set as the degree (grade value) of the entire antecedent part. And the grade value of a consequent part is calculated | required by applying the obtained grade value of the antecedent part to the membership function of the whole Formula 8 (refer FIG. 28).

Furthermore, as shown in FIG. 27, when a consequent part value (in this method, an alarm value) is obtained by a plurality of inference rules, it is necessary to synthesize these values. In such a situation, there is a Min-Max method as a method for calculating the grade value of the consequent part. In the Min-Max method, the fuzzy set of the consequent part is cut out with the grade value of the antecedent part to create a trapezoidal fuzzy set, and the center of gravity of the sum of the set is used as the grade value of the consequent part (see FIG. 29). ).

The relevant conditions have different membership functions depending on their importance. For example, in the case of the condition “exciting emotions are strongly generated”, this is not a problem as long as the corresponding degree is not so large (see FIG. 30), but the condition that “the overall generation of emotions is reduced”. In the case of, attention should be paid even if the degree of correspondence is not large (see FIG. 31).

The response generation means calculates the response of the interlocutor to the user based on the calculated emotion value of the user and the counseling comment received from the medical diagnosis support system. At that time, in order to express synchronization with the user, it is necessary to express a facial expression that matches the user's emotion, but it is impossible for depressed users who return a smile to reassure if a dark topic continues. When something is voluntarily spoken from the system, such as a counseling comment received from the medical diagnosis support system, a facial expression image is generated according to the emotion associated with the utterance content.

In this method, the neural network learns the characteristics of facial expression images in each of six types of emotions (joy, sadness, anger, disgust, fear, and surprise). In this neural network, since the third layer is composed of two neurons, the features of the facial image in each facial expression are developed on a two-dimensional plane (see FIG. 32). Then, when one part of the two-dimensional plane is given as input to the third layer, a facial expression image based on the face image learned in the fifth layer is output (see FIG. 33).

In order to express the 20 types of emotions calculated by the emotion calculation means as one facial expression image, these 20 types of emotions are first converted into 6 types of emotions for facial expressions, It is necessary to calculate where in the two-dimensional plane. First, the rules for reclassifying 20 types of emotions into 6 types are as shown in FIG. 34 in consideration of the characteristics of each emotion.

For the emotions classified into six types, a value obtained by adding the intensity of each emotion to the corresponding emotion part on the emotion plane is added as a weight, and the center of gravity is used as a facial expression obtained by synthesizing the personal emotion values (see FIG. 35).

The response utterance generating means generates a greeting based on the transition of the personal emotion value so far and the dialogue history with the user. Aizu is basically generated based on a table corresponding to the current emotion value (see FIG. 36). In addition, if the speech cannot be recognized well, it is listened back, if the same topic continues for a long time, a new topic is presented, and if a counseling result arrives from the server, the result is output.

In the reaction output means, the reaction such as the face image generated by the server and the response utterance is sent again to the user personal computer, and is output through the display of the user personal computer, the voice synthesizer, and the speaker.

In the region / work / school health management network, alarms are received from user personal computers and servers, and depending on the severity, notifications are sent to medical institutions, messages are sent to users, and monitoring is continued with caution.

By introducing the method of the present invention into the conventional occurrence information management system, it becomes possible to consider the influence of emotions, and the accuracy of abnormality detection can be increased without increasing excessive detection. In addition, since the method of the present invention can detect not only physical but also mental problems, it can be an effective care support system in situations where 24-hour response is impossible. Furthermore, interaction with anthropomorphic agents that return emotional responses can also be expected to have mental effects that tend to be insufficient in situations such as elderly care.

Processing procedure of one embodiment of the present invention Configuration diagram of the embodiment Hardware configuration diagram of the embodiment Processing procedure of physiological information danger detection means Affine transformation 2D discrete cosine transform Speech recognition and input sentence analysis procedure in the same embodiment Correspondence table of surface case frame expression and deep case frame expression Synthetic vector on emotion space Correspondence between quadrants in emotional space and pleasant discomfort Main elements in each predicate type Output emotions and conditions for their occurrence Emergence of emotions related to “the fate of others” Procedures for determining emotions related to “the fate of others” Procedure for discriminating “future” emotions Occurrence of emotions related to “confirmation” Procedure for discriminating emotion related to “confirmation” Procedure for determining emotions related to “happiness” Occurrence of emotions related to attribution How to determine the emotion related to “Attribution” Emergence of emotions related to “happiness / attribute” Procedure for discriminating emotions related to “happiness / attribute” Dependence of emotion Example of emotion occurrence processing neural network Processing procedure of emotion information abnormality detection means Attention required emotional state Fuzzy reasoning Min-Max method Membership function of “excited emotions” Membership function of “decreasing overall emotional occurrence” Emotional plane built on the third layer Facial expression image reproduced from the emotion plane Emotional reclassification rules Synthesis of all-occurring emotions Response utterance database Conventional physiological information abnormality detection device Conventional care communication support system A conventional dialogue device that takes emotions into account

Explanation of symbols

Er Right eye center point
El left eye center point
M Center of mouth
c1 to c4, d Image cropping variables
D01 user
D02 Speech input microphone
D03 Input audio
D04 Speech recognition result
D05 Morphological analysis results
D06 Parsing result
D07 Parsing result expressed in surface case frame representation
D08 Deep case frame expression
F01 Favorability of essential case element 1
F02 Favorability of essential case element 2
F03 Favorability of essential case element 3
F04 Composite vector
T01 Pleasure / discomfort obtained by the emotion calculation method
T02 Emotional process for the future
T03 Emotion generation processing related to "confirmation"
T04 Emotional process related to "the fate of others"
T05 Emotional process for “happiness”
T06 Emotional process related to “Attribution”
T07 Emotional process for “happiness / attribute”
V01 Area where angry facial expression is output
V02 Area for outputting expressions of disgust
V03 Area that outputs fear expressions
V04 Area where sadness expression is output
V05 Area where happiness expression is output
V06 Area that outputs surprising facial expressions
V07 Neutral expression output area
V08 Neural network third layer activity of first neuron
V09 Neural network third layer second neuron activity
S01 Center of gravity of the first emotion
S02 Center of gravity of the second emotion
S03 The center of gravity of the third emotion
S04 The center of gravity of the fourth emotion
S05 The center of gravity of the fifth emotion
S06 The center of gravity of the sixth emotion
S07 The intensity of the first emotion
S08 The intensity of the second emotion
S09 Third emotional intensity
S10 Fourth emotional intensity
S11 Fifth emotional intensity
S12 Strength of sixth emotion
S13 Synthesis point of all emotions
S14 Strength of synthetic emotion

Claims (6)

In a dialogue device considering emotions in a computer network, a camera, microphone, keyboard, physiological information measuring device, means for acquiring a user's face image, speech voice, physiological information, text information through a touch panel, and measuring emotions from those information Then, a means for obtaining an element of emotion, a means for storing the value of the analysis result in the device, a means for transmitting the value to the server, and the user's face image, speech voice, physiological information, and text information on the server side Based on the degree of emotion that has been found (emotion value) and the emotion value of the user according to the change over time, a means for generating the utterance content of the conversation by the computer, the user's face image, speech voice on the server side, Emotions in computer networks that have means to generate dialogue face images based on emotion values obtained from physiological information and text information Into account the interaction device. When an abnormality that is dangerous to life is detected from physiological information acquired from a user through a camera, microphone, or physiological information measuring device, the health / health network that regularly manages the user's health is notified. The interactive apparatus in consideration of the emotion in the computer network according to claim 1, further comprising means. A means for extracting each emotion analysis element based on the face image, uttered voice, physiological information, and text information acquired through the camera, microphone, keyboard, physiological information measuring device, and touch panel, and the user from each extracted emotion analysis element Emotion analysis means for calculating a value representing a complex emotion. 2. The emotion in the computer network according to claim 1, further comprising means for generating a response utterance to the user on the server side and generating and displaying a face image of a conversation person by a computer set in advance, and further storing the history. An interactive device that takes into account. The interactive apparatus according to claim 1, wherein the physiological information acquired through the camera, the microphone, and the physiological information measuring device is determined based on the normal value stored in the system and the state of the occurrence emotion described in claim 3. The user obtained by the emotion analysis means according to claim 3 based on means for determining presence / absence, and a face image, speech voice, physiological information, and text information acquired through a camera, a microphone, a keyboard, a physiological information measuring instrument, and a touch panel The means for determining the emotional instability state of the user from the transition of the emotion value of the user, and the determination result is notified to the region / work area / school health management network that regularly manages the user's health. An interactive device that takes into account emotions in computer networks. By displaying sentences, voices, and facial images calculated as emotions of the interlocutor on the medical diagnosis, health support, and consulting system in the local occupational area or school network or medical institution for the purpose of medical diagnosis support and health support, An interactive device with a graphical user interface that can express the emotions of the interlocutor.