JP2005199403A - Emotion recognition device and method, emotion recognition method of robot device, learning method of robot device and robot device - Google Patents

Abstract

<P>PROBLEM TO BE SOLVED: To stably estimate the state of emotion of a recognition object by taking account of a context. <P>SOLUTION: An emotion estimate system 50 is provided with; an emotion estimate device 1 outputting an estimated emotion Es obtained by estimating a user's emotion by taking account of the context of sensor information based on time-series sensor information; an emotion prediction device 40 referring a predicted emotion change database learned by taking account of the context of actions, finding a predicted emotion change dEb predicted to change after the execution of the action and outputting the predicted emotion based on the estimated result Es' of the emotion estimate device 11 before the action; and an emotion integration part 52 fusing the estimated emotion Es and the predicted emotion Eb by parameters η obtained by taking account of time deterioration of the predicted emotion Eb and outputting a final estimated emotion E. <P>COPYRIGHT: (C)2005,JPO&NCIPI

Description

  The present invention relates to an emotion recognition device and method for recognizing emotion to be recognized for recognizing emotion based on the feature quantity of the recognition target extracted from the sensor information, a learning method for a robot apparatus equipped with the emotion recognition device, and The present invention relates to a robot apparatus for recognizing emotion to be interacted with and a method for recognizing the emotion.

  Examples of conventional emotion recognition technology and systems to which the technology is applied include Patent Documents 1 to 4 listed below. For example, Patent Document 1 discloses a technology of an interactive movie system in which a story is developed in response to emotions included in a user's voice. Patent Document 2 discloses a hierarchical emotion recognition apparatus that prepares a recognizer that recognizes emotion from speech and a recognizer that recognizes emotion from an image, and weights and integrates them according to the emotion recognized by the recognizer. The technology is disclosed.

  Further, Patent Document 3 discloses a technology of a presentation support apparatus that recognizes a presenter's own emotion according to a reaction of an audience and gives feedback to a presenter such as an academic society. Furthermore, Patent Document 4 discloses a behavior pattern processing device for recognizing human emotions by weighting and integrating recognition results of a plurality of emotion recognizers and proposing a behavior pattern in consideration of the recognized emotions. Has been.

The emotion recognition techniques in the following Patent Documents 1 to 4 have the following characteristics. That is,
A dedicated sub-recognition device is provided for each emotion, and the output is logically synthesized to obtain the final output. The weighting parameters that combine the recognition results based on multiple feature quantities by weighting and combine the recognition results that are the final output are The parameters used by each recognizer calculated based on experimentally collected teacher data are prepared for each individual to be recognized.

Japanese Patent No. 2874858 Japanese Patent No. 2996758 Japanese Patent No. 2960029 Japanese Patent Laid-Open No. 2002-73634

  However, the techniques described in Patent Documents 1 to 4 have the following problems. That is, each recognizer determines its output depending only on the vector of sensor input information at that moment. Therefore, if the emotion recognition device is configured with a model that learns the mapping relationship between sensor input vector information related to facial expressions, speech, gestures, and the like and the corresponding emotion, for example, sensor input information at that time, In other words, since only the superficial state change of the recognition target is trusted as absolute and the output is determined, the recognition result easily changes with a slight change in sensor input, and does not become stable.

  For example, while talking in good mood, just thinking and making a little frown will determine that you are feeling disgusted for a moment. In the conventional technology, it has been proposed to use the information obtained from the vulnerable recognizer as described above for selecting the function of the device or selecting the action of the robot device as it is. When a function is selected based on a change in output, the function is frequently selected, resulting in trouble in using the device. What is important in device function selection, robot device action selection, or communication is the state of emotion that is constantly at the base of the recognition target. The blurring of the recognition result caused by a momentary change in facial expression or a slight change in speech tone needs to be smoothed by some processing.

  The present invention has been proposed in view of such a conventional situation, and an emotion estimation device and method for stably estimating the state of emotion to be recognized by considering the context, and such an emotion estimation device It is an object of the present invention to provide a robot apparatus equipped with a robot apparatus, an emotion recognition method thereof, a robot apparatus equipped with an emotion estimation apparatus, and a learning method thereof.

  In order to achieve the above-described object, an emotion recognition apparatus according to the present invention includes a feature amount extraction unit that extracts time-series feature amounts related to a recognition target for recognizing emotion from time-series sensor information, and the time And an emotion estimation means for estimating the emotion of the recognition target in consideration of the context of the sensor information based on the feature quantity of the sequence.

  In the present invention, since the emotion of the recognition target is estimated based on the time-series feature quantity, that is, the emotion estimation is performed in consideration of the context of the sensor information, for example, from the momentary emotion change or the instantaneous sensor information of the recognition target An error in the estimation result based on the extracted feature amount is not reflected, and the emotion estimation result of the recognition target can be recognized as smoothed.

  Further, the feature quantity extraction means extracts the time-series feature quantity for each modal for a plurality of modals as a modal feature quantity sequence, and the emotion estimation means performs each modal based on the modal feature quantity sequence. A plurality of modal emotion recognition means for recognizing the emotion and a recognition result integration means for estimating the emotion based on the recognition results of the modal emotion recognition means. Extracts time-series features for each modal such as speech, and can recognize emotions by modal. The recognition result integration means uses the emotion estimated for one modal as the recognition result or estimates for multiple modals. It is possible to integrate emotions into recognition results.

  Further, the emotion estimation means includes prediction error calculation means for calculating a prediction error of the recognition result obtained by the modal emotion recognition means, and the recognition result integration means is obtained by the modal emotion recognition means. The emotion can be estimated based on the recognition result obtained and its prediction error, and when the emotion recognition rate differs depending on the modal, the emotion estimation rate of the emotion estimation means is further reflected by reflecting this in the prediction error. (Emotion recognition rate) can be improved.

  Furthermore, the recognition result integration means converts the prediction error obtained by the prediction error calculation means into reliability, and weighted recognition results obtained by weighting the recognition results obtained by the modal emotion recognition means. The emotion can be estimated based on the above, and the estimation result can be further smoothed by obtaining the weighted recognition result by averaging the prediction error to some extent to obtain the reliability.

  In addition, when the emotion estimation unit uses, as input data, a time-series feature amount related to the recognition target extracted from the time-series sensor information, the emotion of the recognition target when the time-series sensor information is acquired is output data. Thus, it can be learned in advance by recursive learning, and the emotion estimation means can be learned as a recurrent network, for example.

  The robot apparatus according to the present invention receives one or more sensors for acquiring information related to an interaction target in the robot apparatus acting autonomously based on an internal state and / or an external stimulus, and time-series sensor information from the sensor. Feature amount extraction means for extracting a time-series feature amount related to the interaction target, and emotion estimation means for estimating the emotion of the interaction target in consideration of the context of the sensor information based on the time-series feature amount It is characterized by that.

  In the present invention, since there is an emotion estimation means for estimating the emotion of the user who is the interaction target of the robot apparatus based on the time-series data of the feature quantity of the user, the emotion of the user is misrecognized or temporarily recognized. It can be more reliably estimated that it does not depend.

  In addition, an action execution means for selecting and executing one action from a plurality of actions, an emotion prediction means for predicting the emotion of the interaction target according to the type of action executed by the action execution means, and the emotion estimation And an emotion integration means for estimating the emotion of the interaction target based on the estimation result obtained by the means and the prediction result obtained by the emotion prediction means, and can more stably and accurately estimate the emotion. it can.

  Further, the emotion prediction means predicts an emotion change after the execution of the action with reference to an expected emotion change database in which one action is associated with an expected emotion change expected to change after the execution of the one action. The emotion after the execution of the action can be predicted based on the prediction result and the estimation result estimated by the emotion estimation unit before the execution of the action. Can be predicted.

  Furthermore, the predicted emotion change database may be such that the expected emotion change is learned based on the emotion change of the interaction target before and after the execution of each action, for example, one action is executed. Update each time an action is executed based on the estimation result by the emotion estimation means before and after the execution of the action and the emotional change before and after the execution of the action given from outside. It is possible to construct a predicted emotion database that takes into account the context of behavior.

  In addition, the emotion integration unit weights the estimation result and the prediction result with a parameter that changes so that the estimation result by the emotion estimation unit is more important than the prediction result by the emotion prediction unit as time elapses after execution of the action. Then, the emotion can be estimated based on the weighted result, and the estimation result in the emotion estimation means is top-down corrected by the prediction result of the emotion prediction means, and is always obtained regardless of the execution of the action. The estimation result of the emotion estimation means can be emphasized as the elapsed time after executing the action becomes longer.

  A learning method for a robot apparatus according to the present invention is a learning method for a robot apparatus equipped with an emotion recognition apparatus for recognizing an emotion of an interaction target from given input data, and relates to an interaction target extracted from time-series sensor information. It has a learning step in which the emotion recognition device learns using the feature quantity of the series as input data and the emotion of the interaction target when the time-series sensor information is acquired as the output target value.

  In the present invention, the emotion recognition device of the robot apparatus is configured by, for example, a recurrent network, and learning is performed by recognizing the emotion using time-series feature amounts as input data. A robot apparatus equipped with an emotion recognition apparatus that can be estimated in consideration of the context of information can be obtained.

  According to the emotion recognition apparatus and method of the present invention, a time-series feature amount related to an interaction target is extracted from time-series sensor information, and based on this, the emotion of the interaction target is considered in consideration of the context of the sensor information. For example, it is possible to prevent changes in the facial expression of the interaction target and instantaneous tone changes during the utterance from being reflected in the recognition result based on instantaneous sensor information. The estimation result can be smoothed and recognized as more natural.

  Furthermore, according to the robot apparatus and the emotion recognition method according to the present invention, the time-series feature amount related to the interaction target is extracted from the time-series sensor information, and based on this, the above-described sensor information context is taken into consideration. Equipped with an emotion recognition device that estimates the emotion of the interaction target, the robot device can affect the emotion of the user or the like that is subject to interaction without being influenced by the momentary facial expression change or the instantaneous tone change of the utterance. If you consider the results of predicting the user's emotional change based on their own action history in addition to time-series sensor information, , Correct the recognition result based on sensor information, recognize the user's emotions more accurately, and behave more biologically Rukoto is possible.

  Further, according to the robot apparatus and the learning method thereof according to the present invention, the time series feature quantity of the interaction target extracted from the time series sensor information is input, and the actual interaction target emotion at that time is used as the output target value. For example, by performing learning by recursive learning such as a recurrent network, the robot apparatus can learn an emotion recognition apparatus capable of recognizing the emotion of an interaction target in consideration of the context of sensor information.

  Hereinafter, specific embodiments to which the present invention is applied will be described in detail with reference to the drawings. In this embodiment, the present invention is applied to a robot apparatus equipped with an emotion recognition system that recognizes an emotion to be recognized in consideration of the context. The robot apparatus according to the present embodiment stably and correctly estimates an emotion of an interaction target (hereinafter referred to as a user) as an emotion recognition target from time series data of an external environment (external stimulus) and an action history. It is. Here, the emotion recognition system will be described first, and then a configuration example of a robot apparatus suitable for mounting the emotion recognition device will be described.

A: Emotion recognition system As described above, the conventional emotion recognizer recognizes the emotion of the interaction target from the feature amount obtained from the instantaneous voice or image of the interaction target of the user, for example, and includes context information. The result of recognition becomes unstable because of ignoring. Specifically, the context here refers to what kind of emotional state the interaction target is just now, what action the robot apparatus itself has just performed on the interaction target, and the like.

  Therefore, the robot apparatus according to the present embodiment performs emotion recognition in consideration of the context, and therefore uses a recurrent neural network or the like as one of algorithms for predicting the next state based on the previous state. In this embodiment, the emotion is estimated in consideration of the temporal change (context) of the feature amount extracted from the sensor information.

  Furthermore, a model that predicts the correspondence between the action selection performed by the robot apparatus itself and the emotional change of the interaction target due to the selected action is prepared. The emotion estimation result is further smoothed by predicting whether an emotional change will occur and applying top-down correction to the emotion estimation result estimated from the sensor information.

  That is, the emotion recognition system according to the present embodiment includes an emotion estimation device that estimates a user's emotion from time-series sensor information, and an emotion prediction device that predicts a user's emotion change according to the type of action performed. The estimation result estimated by the emotion estimation device (hereinafter referred to as estimated emotion Es) and the prediction result predicted by the emotion prediction device (hereinafter referred to as predicted emotion Eb) are multiplied by a weight that takes into account the time decay of the action history. And an emotion integration unit that outputs a finally obtained recognition result (hereinafter referred to as recognition emotion E).

(1) Emotion estimation device First, an emotion estimation device that outputs estimated emotion Es in consideration of the context of sensor information from time-series data of sensor information will be described. FIG. 1 is a block diagram showing an emotion estimation apparatus according to the present embodiment. As shown in FIG. 1, the emotion estimation apparatus 1, preprocessing the sensor unit 2 _m for detecting the external situation, gesture, speech, from the sensor information from the sensor unit 2 _m for each modality, such as a human facial expressions And a feature quantity extraction unit 3 _n that extracts a feature quantity, and an estimated emotion Es _n and a prediction error x _n ^ (x _n ^ indicates x _n hat, the same applies hereinafter) for each modal. Modal emotion estimation unit 4 _n , softmax calculator 7 _n for calculating the reliability λ _n for the estimated emotion Es _n from the prediction error x _n ^, and a multiplier for multiplying the estimated emotion Es _n and the reliability λ _n 8 _n and an adder 9 that calculates the sum of the outputs of the multiplier 8 _n and outputs the estimated emotion Es.

Sensor unit 2 _m, for example, the camera 2 ₁ as an imaging means for capturing an image of the surroundings _consists and a microphone 2 ₂ as a sound input means for inputting the voice of the surrounding, when the sensor information of the sequence (hereinafter, sensor information Is also called a column.)

Amount extracting unit 3 _n features, performs a pre-process for each modal, for example the facial expression analyzer 3 _1, gesture analysis unit 3 _{2 has} a like speech analysis unit 3 _3, from the sensor information of the time series, each modal Then, a time-series feature quantity vector to be recognized is extracted and output as a modal feature quantity sequence.

The emotion estimation unit 4 _n is prepared for each modal, and the feature quantity vector sequence obtained by preprocessing by the feature quantity extraction unit 3 _n corresponding to each modal is input, and the estimated emotion Es _n is obtained from this feature quantity vector sequence. Is estimated and output. The emotion estimation unit 4 _n estimates an emotion by an emotion forward model (forward model or forward model) that predicts the current emotion in consideration of the past emotion, and outputs the estimated emotion as a modal-specific estimate _n . It includes an emotion recognizer 5 _n as modal emotion recognition means, and an error predictor 6 _n that calculates a prediction error x _n ^ of the estimated emotion Es _n output from the emotion recognizer 5 _n . Also in the error predictor 6 _n, be made to calculate the prediction error x _{n ^} of the estimated emotional Es _n by the forward model to calculate the current prediction error x _{n ^} in consideration of past prediction error x _{n ^} it can.

The forward model only needs to realize the mapping relationship of input / output, that is, in this embodiment, the mapping relationship that outputs the estimated emotion Es _n when a time-series feature quantity vector is input. Any expression method such as a parameter matrix or a correspondence table may be used. In the present embodiment, the direction in which emotion is predicted from the sensor input, that is, the direction in which the feature vector sequence is input and the estimated emotion Es _n is output is referred to as the forward direction (emotional forward model). A model in which input / output data is reversed from the forward model is referred to as an inverse model or an inverse model.

The softmax computing unit 7 _n converts a prediction error x _n ^ into a reliability λ _n for the estimated emotion Es _n calculated by each emotion estimator 5 _n using a function to be described later for obtaining the softmax, and a multiplier 8 _n multiplies the estimated emotion Es _n by the reliability λ _n . The adder 9 calculates the sum of the weighted estimated emotions λ _n Es _n weighted with the reliability λ _n by the multiplier 8 _{n and} outputs the sum as the estimated emotion Es. These softmax computing units 7 _n , multipliers 8 _n and adders 9 constitute recognition result integration means for integrating the recognition results of the modal emotion recognizers 5 _n .

Here, in the present embodiment, as the emotion recognizer (emotional forward model) and the error predictor, those previously learned using, for example, a recurrent neural network are used. That is, the emotion recognizer 5 _n uses learning data including time-series feature quantity vectors s _n ^* and time-series emotion classification information Es _n ^* corresponding to teacher data into which emotions are classified in advance. This learns the emotion recognizer 5 _n that outputs time-series emotion information from a time-series feature vector given as an input. That is, among the learning data, the time-series feature vector s _n ^* is used as input data, and the time-series emotion information Es _n ^* , which is teacher data, is learned in advance for each modal as the target value of the output data.

Further, the error predictor 6 _n uses learning data including time-series feature quantity vectors s _n ^* and corresponding time-series emotion classification information Es _n ^* which is teacher data into which emotions are classified in advance. learns error predictor 6 _n for outputting an output of the emotion recognizer 5 _n given as input (estimated emotion Es _n) and / or feature vectors s _n prediction error from ^* for each emotion recognizer 5 _n. Here, these learning methods will be described first, and then the emotion estimation method (reproduction method) in the emotion estimation apparatus using the emotion estimation unit 4 _n having the learned emotion recognizer 5 _n and error predictor 6 _n will be described. To do.

(2) Learning method of modal-specific emotion recognizer In the present embodiment, an emotion estimation unit 4 _n , that is, an emotion recognizer 5 _n is prepared for each modal. The emotion recognizer 5 _n can be operated by one emotion recognizer, for example, “joy”, “sadness”, “anger”, “surprise”, “disgust”, And a plurality of emotions such as “fear” can be recognized, and a feature amount corresponding to each emotion is learned in advance in a certain modal. Preprocessing for extracting feature values is performed, and the user's emotion is estimated and output based on the extracted feature values. Note that emotion recognizer of the present embodiment, estimated emotions Es which shall recognize the six basic 6 emotions, therefore modal-specific emotion recognizers 5 estimated emotions Es _n and emotion estimation apparatus 1 _n outputs outputs Is a vector having six types of emotions as elements, but the types of emotions to be recognized are not limited to this.

The emotion recognizer 5 _n to consider modal, e.g. expression, gesture, and the like speech. In learning, first, learning data for learning the emotion recognizer 5 _n is prepared, and then the emotion recognizer 5 _n is learned for each modal.

The learning data necessary for learning the emotion recognizer 5 _n includes the time-series feature vector serving as input data and the target of interaction when obtaining the input data serving as the target value of the output data. This is the emotion of the user. It is assumed that the user's emotion is appropriately digitized.

The input data, the facial expression analyzer 3 _1, and the feature extraction unit 3 _n made of the gesture analysis unit 3 ₂ supplies the sensor information of the time series, such as the image feature extraction made of speech analysis unit 3 ₃ The time-series sensor information such as voice is supplied to the unit 3 _n , and the time-series user feature vector corresponding to each modal is collected based on the time-series sensor information.

For example, if the facial expression analyzer 3 _1, as a result of the filtering process for extracting a frequency component and direction component of the entire image or be extracted as a feature quantity, for example the forehead and glabella, cheeks wrinkles density and direction, eye Time-series data of feature vectors (feature vectors), such as vector data that numerically represents the features of elements visually appearing on the face, such as the spread of lips, the shape of lips, etc. Column) is output.

Further, if the gesture analysis unit 3 _2, the moving amount of the hand position, moving speed, such as frequency of crosscut of the hand trajectory is output feature vector sequence is extracted as the feature quantity.

Further, if the speech analysis unit 3 _3, average sound pressure of the recognized utterance (power), the fundamental frequency (repetition frequency pattern appears in homothetic wave), and the feature amount data such as spectra, extracted The feature vector sequence is output.

Collecting training data of emotion recognizer 5 _n corresponding to each modal can experimentally performed that, for example, as follows. In other words, in front of the feature quantity extraction unit 3 _n , human beings actually perform the performance according to the determined scenario, and each modal is performed by the feature quantity extraction unit 3 _n from the data (sensor information sequence) observed at that time. Learning data can be collected by extracting a time-series feature vector (feature vector) corresponding to, and recording it together with information indicating human emotion at that time. In addition, when an emotion recognition system is mounted on a robot apparatus to be described later, it is only necessary to collect learning data by acting by a human in the same manner at least in front of the robot apparatus on which the feature amount extraction unit 3 _n is mounted. . Here, the information indicating the human emotion when the feature vector is extracted is a vector (emotional classification information) in which the basic six emotions described above are digitized, and is output by the emotion recognizer 5 _n at the time of recognition. This is the teacher data for the estimated emotion. Hereinafter, this learning data is referred to as estimated emotion (teacher) Es _n ^* . Further, among the learning data, a feature vector used for learning is described as a feature vector (teacher) s _n ^* .

Thus the time series feature vector (teacher) s _n ^* and as an input data collected by the feature extraction unit 3 _n, a target value of the output, the feature vector (teacher) when acquiring the s _n ^* Learning of each emotion recognizer 5 _n is performed using learning data consisting of the estimated emotion (teacher) Es _n ^* of the user.

The learning of the emotion estimation unit 4 _n is performed individually for each modal, and the feature quantity extraction unit 3 _n and the emotion estimation unit 4 _n are prepared for each modal, and the emotion recognizer 5 _n and the error predictor 6. _n is provided for each emotion estimation unit 4 _n . In the following description, the feature quantity extraction unit 3, the emotion estimation unit 4, the emotion recognizer 5, and the error predictor 6, unless particularly necessary. I will do it.

FIG. 2 is a diagram schematically showing the emotion estimation unit 5 during learning. As shown in FIG. 2, at the time of learning, learning data obtained by preprocessing for the “expression” is stored in the emotion recognition device 5 for facial expression that recognizes the emotion from one modal, here “expression”. A database 10 is connected. The emotion recognizer 5 receives the learning data supplied from the database 10, that is, given the time-series feature vector (teacher) s _n ^* and the time-series estimated emotion (teacher) Es _n ^* at that time. An emotion model for estimating an emotion from a sequence feature vector is learned. That is, when the feature vector (teacher) s _n ^* of learning data is input, the emotion recognizer (emotional forward model) 5 is trained for each modal so as to output the estimated emotion (teacher) Es _n ^* .

  In the present embodiment, an emotion recognizer (emotional forward model) 5 that estimates emotion from time-series data is configured by a recurrent neural network. The algorithm used when learning and recognizing emotions is not limited to a recurrent neural network, and any algorithm can be used as long as time series data can be reflected in an output. For example, as a method for clustering information having a certain time width, there are methods such as DP (Dynamic Programming) matching (dynamic programming), Hidden Markov Model (HMM), and the like. The context in the present embodiment refers not only to the state vector at that time, but also to a state vector data string having a time width that goes back in the past, and obtains an output (estimated emotion) using such data as an input. If you can.

The emotion recognizer 5 shown in FIG. 2 is provided with the feature vector (teacher) s _n ^* as input data of the recurrent neural network among the learning data of the character series extracted by the preprocessing for each modal. Then, the estimated emotion (teacher) Es _n ^* when the input data is collected is given as output data, and learning of the emotion recognizer 5 as an emotion forward model is performed. As shown in FIG. 3, the recurrent neural network has an input layer 11a to which a feature vector in the corresponding modal is input, an intermediate layer 12 composed of one or more layers, and an output that outputs an estimated emotion related to the corresponding modal. Layers 13a and 13b, and an input layer 11b to which the estimated emotion of the output layer 13b is input. That is, the feature vector at a certain timing and the estimated emotion output from the output layer 13b at the previous timing are input to the input layers 11a and 11b, which are output from the output layers 13a and 13b via the intermediate layer 12. Some of them are returned to the input layer 11b at the next timing.

  In this way, the recurrent neural network has a group of units (input layer 11b and output layer 13b) that feed back from the output layer to the input layer (or intermediate layer), and these include context information (context) based on time-series input. Expressed internally. The feedback from the output layer 13b to the input layer 11b is called a context loop. By using this recurrent neural network, it is possible to realize current emotion estimation in consideration of context information rather than instantaneous input / output information mapping. Details of the recurrent neural network will be described later.

(3) Learning method of the error predictor It has been experimentally known that emotions that humans express through each modal do not necessarily include expressions corresponding to each emotion on average. For example, in Patent Document 2, when emotions are recognized by voice or image, “sadness” and “fear” are highly recognized only by voice, and “anger”, “happiness”, and “surprise”. Is based on a high degree of recognition only with images, and when an emotion recognition unit is provided for recognizing emotions from audio data and image data, for example, "sadness" in the emotion recognition unit that recognizes emotions from audio data, Or, when “fear” is recognized, the weight is increased, and when “anger”, “happiness”, or “surprise” is recognized by the emotion recognition unit that recognizes emotion from the image data, the weight is increased. Thus, the recognition results of these two emotion recognition units are integrated.

  In this way, sensor information that is good at recognition differs depending on the emotion, and therefore a difference occurs in how reliable the recognition result for each modal for each emotion is. The reliability of the recognition result can be increased by acquiring this difference as the reliability of each recognizer by learning in advance and weighting the recognition result of each modal at the time of recognition to obtain the final recognition result. .

  As described above with reference to FIG. 1, the emotion estimation apparatus according to the present embodiment estimates emotion from multimodal sensor information, and the emotion recognizer 5 corresponding to each modal is an element recognizer ( Emotion forward model). Further, the emotion recognition results output from each emotion recognizer 5 are weighted according to the reliability of the recognition results in the context to determine the final output. For this reason, the module (emotion estimation unit 4) corresponding to each modal has an error predictor 6 for pairing with the emotion recognizer 5 to calculate the reliability. The context here can be assumed to be a sensor input state (time-series sensor information string) or an emotion recognition state (time-series estimated emotion). In accordance with each assumption, input data to the error predictor 6 is set. For example, when considering the context of sensor information, a time series feature vector extracted from time series sensor information is used as input data. When the context of the emotion recognition state is taken into consideration, the output of the emotion recognizer 5 may be used as input data.

  As described above, even in the learning of the error predictor 6 that outputs how reliable the output of the element recognizer (emotion recognizer) individually learned for each modal is, the sensor input information or the emotion recognition state A model that can be applied based on history information can be applied, and a model that predicts an error of a pair of element recognizers using sensor information (or feature values extracted from these) and / or emotion recognition states as inputs. Realized by a recurrent neural network. Here, a model will be described in which a time-series feature vector extracted from time-series sensor information and a modal-specific estimated emotion recognized by an emotion recognizer are input.

That is, as shown in FIG. 4, the error predictor 6 uses again the learning data collected for each modal, and uses the feature vector (teacher) s _n ^* and the feature vector (teacher) s _n ^* . Estimated emotion Es _n ^* which is output data of the emotion recognizer 5 when input is used as input data, and the estimated emotion (teacher) Es _n ^* corresponding to the output Es _n of the emotion recognizer 5 and the feature quantity vector s _n ^{* .} the error information which is the difference between an ideal output x _n ^*, performs learning of the error predictor 6 corresponding to each modality as a target value of the output of the ideal output power x _n ^*. The feature quantity vector (teacher) s _n ^* is a vector having a plurality of feature quantities as elements, and the estimated emotion Es _n ^* , the estimated emotion Es _n , and the ideal output x _n ^* are elements of the six types of emotion described above. Is a vector. Moreover, the learning data used for learning may be prepared differently from the learning data used for learning of the emotion recognizer 5, whereby the error predictor 6 with higher generalization ability can be obtained. .

FIG. 5 shows an error predictor learning method using a recurrent neural network. In the case of learning by the error predictor, the corresponding modal time-series feature vector s _n and the estimated emotion Es _n which is emotion prediction data in the corresponding modal are input to the input layers 21a and 21b, and from the output layer 23a. The emotion prediction error x _n ^ is output. Further, the prediction error x _n ^ output from the output layer 23b is fed back by the context loop and supplied to the input layer 11c. That is, the prediction error x _n ^ at time t + 1 is output from the feature vector s _n , the estimated emotion Es _n , and the prediction error x _n ^ output at time t.

(4) Recurrent Neural Network Next, an example of a recurrent neural network will be described. Not only the recurrent neural network described in Japanese Patent Laid-Open No. 8-6916 described below, but as described above, learning and recognition can be performed in consideration of the context using time series data as input. I just need it.

  A neural network is a simplified model of a neural network in the human brain, which is a network in which neuronal neurons are connected via synapses through which signals pass only in one direction. Signal transmission between neurons is performed through the synapse, and various information processing is possible by appropriately adjusting the resistance of the synapse, that is, the weight. In each neuron, outputs from other connected neurons are input with synaptic weighting, and their sum is added to a non-linear response function and output to another neuron again.

  One of the neural network structures is a multilayer network as shown in FIG. This type of network has a layered structure, only coupling between layers is allowed, and there is no intra-layer coupling or autoregressive coupling. This multilayer network is considered to be suitable for recognition of spatially spreading patterns and information compression.

  On the other hand, as shown in FIG. 7, a network that does not have such a restriction and allows arbitrary coupling between units is called a recurrent network. Strictly speaking, there is no concept of layers in the recurrent network, but here the concept of layers is adopted for convenience in order to correspond to the multilayer network. Hereinafter, a unit group to which input data is input is referred to as an input layer, a unit group that outputs network output is referred to as an output layer, and other unit groups are referred to as intermediate layers.

  In a recurrent network, there is a connection where the past output of each unit is returned to another unit in the network or to itself. Therefore, the network has dynamics in which the state of each neuron changes depending on time. Thus, since the system has time in the network, the recurrent network is considered to be suitable for time series pattern recognition and prediction. A multi-layer network can be viewed as a special case of a recurrent network.

  The equation of state followed by each neuron of the recurrent neural network is given by the following equations (1) and (2).

Here, x _i (t) is the internal state of unit i at time t, and the output value y _i is determined by nonlinearly transforming the internal state. Also, τ _i and X _i are the time constant and external input of unit i, respectively, and w _ij is the weight of coupling from unit j to unit i. N is the total number of units.

When obtaining the network status at each time from time t ₀ to time t _n, it can be obtained by the following procedure. First, as an initial state, the internal state and output value of the network at time t ₀ are appropriately set. Thereafter, the above formulas (1) and (2) are solved from t ₀ to t _n in the forward direction of time.

  As described above, when using a recurrent neural network for time series pattern recognition and prediction as in this embodiment, time series data such as a feature vector sequence is used so that the network outputs a correct output. Prepare learning data in which input data and teacher data (output data) that is time-series data such as estimated emotions are paired, and use them to learn network weight values, time constants, and initial states in advance It is necessary to keep. This is usually done using an optimization technique called backpropagation.

The feature of this method is that the weight is corrected based on the steepest descent method so as to reduce an error in a finite time interval from t ₀ to t _n as given by the following equation (3).

Here, Y _k (t) is teacher data presented to unit k at time t. Here, k is the unit belonging to the output layer, N _v denotes the number of units belonging to the output layer.

  The steepest descent direction used when calculating the weight value, the time constant, and the correction amount of the initial state is given by the following equations (4), (5), and (6).

Here, P _i (t) is the back propagation error of unit i. The back propagation error is calculated according to the following equation (7).

Where δ _ik is the Kronecker delta symbol. In the above equation (7), the back propagation error of the neurons belonging to the output layer is such that the error of the output value is added at each time in addition to the sum of the back propagation errors and weights from other neurons. . Also, df (x _i ) / dx _i , (y _i (t) −Y _i (t)) cannot be calculated unless x _i (t) and y _i (t) are determined. Therefore, after calculating x _i (t) and y _i (t) in the forward direction of time, assuming that the back-propagation error is 0 at time t _n , the boundary condition given by the following equation (8) is Then, the back propagation error is calculated in the reverse direction of the time t _n → t ₀ .

The calculation procedure for the steepest descent direction is as follows. That is, first, the initial state of each unit at time t ₀ is appropriately set, then the internal state and output value at each time are calculated according to the above formulas (1) and (2), and the values are stored. Next, the boundary condition of the back propagation error given by the above equation (8) at time t _n is set. Thereafter, using the internal state and the output value calculated earlier, the back propagation error at each time is calculated and stored while reversing the time according to the above equation (7). Finally, the steepest descent direction given by the above formulas (4) to (6) is calculated using the obtained output value and back propagation error at each time.

The details of the backpropagation learning process procedure will be described with reference to FIG.
First, the learning data in FIG. 8 is, for example, learning data for a network having two input layer neurons and one output layer neuron as shown in FIG. As shown in FIG. 9, the learning data is composed of input data and teacher data, and has the number of data corresponding to (number of input layer neurons × number of time series sample points) and (number of output layer neurons × time series sample points), respectively.

  First, a weight value, a time constant, and an initial value of an initial state are set using an appropriate random number (step S1). Then, the processing from the next step S2 to step S7 is repeated until the back propagation error converges.

  First, after appropriately setting the initial state of the network, that is, the internal state and output value of the network, the internal state of each unit in the forward direction of time using the input data according to the above formulas (1) and (2), An output value is calculated and stored (step S2).

Then, the boundary condition of the above equation (8) is set, and then the back propagation error P _i (t) of each unit is calculated in the reverse direction of time using the teacher data according to the above equation (7). Is stored (step S3).

  Next, the steepest descent direction is calculated according to the above formulas (4) to (6) using the internal state and output value obtained in step 2 and the back propagation error obtained in step 3 (step S4).

  Then, the current weight correction amount is calculated according to the following formulas (9) to (11) based on the steepest descent direction obtained in step 24 and the previous weight correction amount (step S5).

  Here, γ is a learning coefficient, α is a moment coefficient, and n is the number of learnings. The second term on the right side is called a moment term and is an empirically added term to accelerate learning.

  Next, the weight of each unit is corrected according to the following formulas (12) to (14) (step S26).

  Then, in step 7, it is determined whether or not the error converges below a certain value. If the error is larger than the certain value, the processing from step 2 to step 6 is repeated. As described above, the emotion recognizer 5 and the error predictor 6 can be learned, and the emotion estimation apparatus 1 is configured using the emotion estimation unit 4 including the learned emotion recognizer 5 and error predictor 6. Is done.

(5) Integration of recognition results by modal (reproduction method)
Next, the emotion estimation method of the emotion estimation device 1 will be described. First, the various sensor means such as a camera 2 ₁ and microphone 2 ₂ having the robot apparatus acquires sensor information in time series. Then, the feature quantity extraction unit 3 extracts a time series feature quantity vector as preprocessing from the time series sensor information. That is, the feature amount extraction unit 3 extracts a time-series feature amount vector as a modal feature amount sequence for each modal such as an expression, a gesture, and an utterance.

Next, the extracted feature amount is supplied to the corresponding emotion estimation unit 4. For example, if the feature quantity of the facial expression, the expression for the emotion estimation unit 4 _1, if the feature amount of the gesture to a gesture for emotion estimation unit 4 _2, to the utterance for emotion estimation unit 4 ₃ If a feature quantity of speech , A feature vector sequence is supplied.

Each emotion estimation unit 4 outputs the estimated emotional Es _n from the feature quantity vector s _n in time series by the emotion recognizer 5 calculates and outputs a prediction error x _{n ^} by the error estimator 6. Note that the feature quantity extraction unit 3 may be arranged in the emotion recognizer 5 or the error predictor 6. As described above, the modal-specific estimated emotion Es _n is an emotion estimation vector calculated for each emotion recognizer 5 provided for each modal, and each emotion (for example, basic six emotions) defined as an emotion element. The value of is included. The estimated emotion Es that is the final output is also vector information including the value of each emotion.

The prediction error x _n ^ calculated by the error predictor 6 is supplied to the softmax calculator 7 in a corresponding modal manner. The softmax calculator 7 calculates the reliability λ _n of the emotion recognizer 5 according to the softmax function of the following equation (15) based on the prediction error x _n ^ output from the error predictor 6. Here, x _n ^, x _l ^ are outputs of the respective error predictors, and σ is a parameter that determines how much the error of each recognizer is averaged to calculate the responsibility signal (reliability).

As shown in the following equation (16), each reliability λ _n is multiplied by the estimated emotion Es _n output from the corresponding emotion recognizer 5, and the sum of these is added to the time-series sensor input information. This is the estimated emotion Es as the final estimated emotion recognition result vector.

  As described above, the emotion estimation apparatus 1 according to the present embodiment is configured so that the emotion recognizer (element recognizer) 5 corresponding to each modal does not estimate the emotion only from the instantaneous sensor information, Recognizing emotions in consideration of the context of sensor information by taking out sensor information and estimating emotions by extracting features from sensor information (time-series sensor information) having time spread it can. That is, the emotion recognizer 5 and the error predictor 6 are used in advance by using a recursive learning method, for example, an algorithm such as a recurrent neural network. By learning the forward model that outputs the prediction error from the quantity vector and / or the estimated emotion, the time series data of the feature quantity vector obtained from the input information of each sensor is input and the emotion is estimated in consideration of the context. An emotion estimation device can be obtained.

  In addition, by converting the prediction error obtained from the error predictor into reliability by processing of a softmax function and weighting the recognition result for each modal, it is possible to make a more accurate determination for each emotion The estimated emotion Es can be output as an integration result that places importance on the expected modal output.

(6) Emotion Prediction Device Next, an emotion prediction device that takes into consideration the context of the action performed by the robot apparatus and predicts the user's emotion according to the action will be described. The emotion prediction device, in parallel with the module (emotional estimation device) that estimates the emotion of the current interaction target based on the sensor input information history, the robot device itself and the corresponding interaction target (user) emotion This module estimates changes. This emotion prediction device predicts an emotional change of the user after the execution of the action according to the executed action k, the predicted emotional change (hereinafter referred to as an expected emotional change dEb _k ), and the above-mentioned before the execution of the action. Are integrated with the estimation result Es ′ of the emotion estimation apparatus, and a predicted emotion Eb is output.

  FIG. 10A is a diagram illustrating a main part related to the emotion prediction apparatus in the robot apparatus. As shown in FIG. 10 (a), the robot apparatus autonomously executes actions based on the internal state 31 and / or the external stimulus 32, and a plurality of element actions (schema) 33 are configured in a tree structure. A behavior controller 30 having a structured schema tree is provided. When the internal state 31 and the external stimulus 32 are input, the schema 33 calculates an action value (Activation level) AL indicating the execution priority of the action described in each schema. This is a module (behavior description module) that selects an action to be executed based on the action value AL. A state machine is prepared for each module, and a sensor input is input depending on the previous action (action) and situation. The recognition result of external information is classified and the action is expressed on the aircraft. Each schema 33 defines a predetermined internal state and external stimulus according to the behavior described in itself.

  Here, the external stimulus 32 is perceptual information or the like of the robot apparatus, and includes, for example, object information such as color information, shape information, and face information processed for an image input from the camera. Specifically, for example, color, shape, face, 3D general object, hand gesture, movement, voice, contact, distance, place, time, number of times of interaction with the user, and the like can be mentioned.

  The internal state 31 is emotions such as instinct and emotion managed by an internal state management unit (not shown). For example, fatigue (FATIGUE), pain (PAIN), nutritional state (NOURISHMENT), dryness ( THURST), love (AFFECTION), curiosity (CURIOSITY). For example, the internal state “nutrient state” can be determined based on the remaining battery level, and the internal state “fatigue” can be determined based on power consumption.

  For example, the schema 33 whose behavior output is “eat” handles the type of the object, the size of the object, the distance of the object, and the like as the external stimulus 32, and “NOURISHMENT” (“nutrition state”) as the internal state 31. ), “FATIGUE” (“fatigue”), etc. Thus, for each schema 33, the types of external stimuli 32 and / or internal states 31 to be handled are defined, and the action value AL of the action (elemental action) corresponding to the corresponding external stimulus 32 and / or internal state 31 is set. Calculated. Of course, one internal state or external stimulus may be associated with not only one elemental action but also a plurality of elemental actions.

  The action value AL indicates how much the robot device wants to execute the schema 33 (execution priority). Based on this action value AL, the selected schema 33 outputs the action described in itself. This action value AL is calculated from the internal state 31 and the external stimulus 32. Specifically, for example, from the internal state 31, a motivation vector (Motivation Vector) indicating how much the corresponding action is desired is calculated, and whether or not the corresponding action can be performed from the internal state 31 and the external stimulus 32. A releasing vector (Releasing Vector) is calculated, and an action value AL can be calculated from these two vectors. Then, for example, the schema with the highest activation level can be selected, or two or more schemas with the activation level exceeding a predetermined threshold value can be selected and the actions can be executed in parallel. However, when executing in parallel, it is assumed that there is no hardware resource conflict between schemas.

  The behavior controller 30 outputs an identification ID (schema ID) for identifying the selected schema 33, and this schema ID is input to the emotion prediction device 40.

  The emotion prediction device 40 has a predicted emotion change database 41 in which a value of a predicted emotion change is stored, and an emotion change (expected emotion) that is expected to change after execution of the behavior based on the behavior indicated by the input schema ID. Change). The expected emotion change has a close relationship with the behavior control algorithm in the behavior controller 30 of the robot apparatus, and the expected emotion change values corresponding to all the schemas 33 defined in the behavior control algorithm are defined.

  The predicted emotion change database 41 can be dynamically updated using values observed through actual interactions. Normally, a common database of expected emotional changes is constructed for all interaction target persons, but the robot device actually interacts so far, and keeps data for each person who stores face, voice, name, etc. Thus, the predicted emotion change database may be constructed so that the predicted emotion change database as a prediction model can be switched for each person. In addition, by combining these, an expected emotion change database may be constructed for each person who knows well, and an expected emotion database that can be used in common for the first person to be met may be constructed.

The predicted emotion change vector (predicted emotion change) defined for each behavior unit (schema) k is dEb _k, and the emotion before execution of the behavior is estimated from the sensor information observed before the behavior is executed. The estimated emotion Es ′, that is, the estimated emotion before the action execution calculated by the above-described emotion estimation device, is combined, and the predicted emotion shown in the following formula (17) estimated based on the behavior history information Eb is determined. The predicted emotion Eb is also a vector having the basic six emotions as elements.

(7) Learning Method of Expected Emotion Change Database Next, a construction method (learning method) of the expected emotion change database in the emotion prediction device will be described. The predicted emotion change database is a database that expresses the difference between the emotion before and after the action is executed, and the predicted emotion change database 41 of the emotion prediction device 40 in the initial state is all initialized to 0, and sensor information The emotion to be interacted with is determined using only the estimated emotion Es estimated from. After that, when the robot apparatus actually selects an action and interacts with it, if an emotion estimation result of the interaction target before and after the action is obtained, as shown in the following equation (18), expected emotional change database using the difference value of the estimated emotional Eb based on action history after performing a final emotional prediction result E _a and action is updated.

Here, dEb _k is a predicted emotion change vector corresponding to each behavior unit k, E _a is an estimated emotion Es estimated from sensor information, and a predicted emotion Eb estimated based on behavior history information, which will be described later. As a final output, the estimated emotion E, α is a learning coefficient, time T ′ is the time when the previous action k was performed, time T is the time when the same action k is performed next time, and the value of dEb is the action k Only the value corresponding to is updated. Further, E _a ^{T ′} indicating the estimated emotion E _a at time T ′ is, for example, estimated emotion estimated based on the user's time-series feature amount extracted from time-series sensor information after execution of the action (time T ′). The predicted emotion change database may be updated as Es. In addition, for example, when it is desired to artificially create a database, the user's emotion at time T ′ may be supplied from the outside as teacher data or may be taught to the robot apparatus.

Here, when the predicted emotion change dEb _k ^{T is} to be updated, the target value is the estimated emotion E _a , and the value calculated in consideration of the previous predicted emotion change dEb _k ^{T ′} is predicted emotion _Eb ^{k T,} and therefore, taking the difference of the expression ^'a prediction affective Eb _k ^T' (18) estimates the emotion _{E a} ^T as shown in the right-hand side of the expected emotional changes dEb if this value is positive it is necessary to increase the _k ^T, indicating a need to further reduce the expected emotional changes DEB _k ^T if negative.

As described above, the predicted emotion change dEb _k ^T at time ^T is changed from the expected emotion change dEb _k ^{T ′} at time T ′ to the user emotion E _a ^{T ′} at time T ′ and the predicted emotion Eb _k ^{T ′ at} time T ′. And a value obtained by multiplying the difference by the learning coefficient α, and the action history, that is, the action context is taken into consideration.

  For example, as shown in FIG. 10B, in the predicted emotion change database 41 at a certain point in time, when an action described in, for example, schema ID (k) = 1 is executed, the internal state “JOY” increases (10 ) The internal state “DISGUST” has decreased (−25). Further, when the action described in the schema ID (k) = 6 is executed, the internal state “JOY” increases (+5), and the internal state “DISGUST” decreases (−10).

In this way, the robot apparatus can learn the expected emotion change database 41 by updating the value of the expected emotion change by executing the action. Note that the expected emotion change database 41 may be constantly updated, but may be updated after being updated according to the action execution result of a predetermined period, and the expected emotion change dEb _k after the action execution is previously determined. A defined expected emotion change database may be used.

(8) Estimation Method of Estimated Emotion E in Emotion Recognition System Next, a method of merging the recognition result based on time-series sensor input information and the emotion prediction result based on the action history will be described. The estimated emotion Es obtained as the recognition result based on the sensor input information by the emotion estimation device described above is sequentially output according to the time-series information of the sensor input, but the emotion prediction device uses the recognition result based on the action history as the recognition result. The obtained estimated emotion Eb is a value obtained at the moment when a certain elemental action is completed. Therefore, the reliability of the predicted emotion predicted based on the action history decreases with the passage of time from the completion of the action. After reflecting this effect, an algorithm for performing top-down correction on the emotion recognition result (estimated emotion Es) calculated based on the sensor input is formulated by the following equations (19) and (20). These formulas (19) and (20) emphasize the predicted emotion Eb, which is the emotion prediction result based on the behavior history, immediately after the completion of the behavior, and the emotion recognition result based on the sensor information as time elapses from the behavior completion. This means that the estimated emotion Es is changed so as to be emphasized. Here, t is an elapsed time after completion of a certain action (reset to 0 when a different action is completed), and τ ₀ and τ are parameters for determining the degree of attenuation with respect to the passage of time.

  FIG. 11 is a diagram illustrating a portion related to the emotion integration unit 52 that integrates the estimated emotion Es that is the estimation result of the emotion estimation device 1 and the predicted emotion Eb that is the prediction result of the emotion prediction device 40 in the emotion recognition system. is there. As shown in FIG. 11, the result integrating unit 52 receives the estimated emotion Es from the emotion estimation device 1 and the predicted emotion Eb from the emotion prediction device 40, and estimates the estimated emotion E according to the above equations (19) and (20). Is output.

  Here, as described above, the emotion estimation apparatus 1 extracts a time-series feature amount from a time-series sensor information sequence, and outputs an estimation result estimated in consideration of the context as an estimated emotion Es. Acts as

In addition, the emotion prediction device 40 refers to an expected emotion change database 41 learned in consideration of an action history, that is, an action context after the action is executed, and an emotion change prediction unit (behavior history analysis) outputs the expected emotion change dEb _k. System) 42 and a predicted emotion calculation unit 43 that calculates a predicted emotion from the predicted emotion change dEb _k and the estimated emotion Es ′ of the emotion estimation device 1 ′ before the execution of the action. This emotion prediction device 40 outputs the predicted emotion Eb after the action is executed, receives the estimated emotion Es ′ before the action execution from the emotion estimation device 1 ′, and is associated with the action k executed from the emotion prediction unit 42. The expected emotion change dEb _k is received and added to output the estimated emotion Eb. 11 shows the emotion estimation device 1 ′ that outputs the estimated emotion Es ′. However, the estimated emotion Es ′ before the start of the action may be input from the emotion estimation device 1.

  These output data are as shown in FIG. FIG. 12 is a graph showing changes in the estimated emotion when the robot apparatus executes the three actions B1 to B3. From the upper stage, the estimated emotion calculated by the emotion estimation device based on time-series sensor information. Es, the next stage, shows a value = (1−η) Es in consideration of the time decay of the estimated emotion Es.

  The third row shows the predicted emotion Eb predicted based on the action history, and the subsequent row shows a value = ηEb considering the time decay of the predicted emotion Eb. The fifth row shows the estimated emotion E that is the recognition result of the emotion recognition device 50 obtained by integrating (1-η) Es and ηEb.

  In the present embodiment, for a plurality of modals, an emotion recognizer is prepared for each modal, an estimated emotion is calculated from a time-series feature vector extracted from time-series sensor information input by modal, and Since the prediction error of the recognition result is obtained, converted into the reliability parameter, and the recognition result is weighted, it is possible to estimate the emotion in consideration of the time series (context) of the sensor input information. As a result, it is possible to obtain an extremely stable recognition result by suppressing the estimated emotion from being unstable due to sensor noise, i.e., the recognition result changing to incoherent, and the recognition result of the recognizer by the reliability λ. It is possible to estimate the emotion very accurately by weighting and integrating them into the estimated emotion Es.

  Furthermore, not only the estimated emotion Es obtained by the recognizer from the sensor input information, but also a predicted emotion Eb that predicts the emotion change from the context of the behavior of the robot device itself, and applies a top-down correction to the estimated emotion Es, Considering the temporal error between the estimated emotion Es that can be obtained as soon as the sensor information is input and the predicted emotion Eb that is output only after the behavior result, that is, at the stage immediately after the completion of the behavior, the predicted emotion Eb based on the behavior history. The parameter η is changed so that the emotion recognition result based on the sensor information (estimated emotion Es) is emphasized as time elapses from the completion of the action, the recognition result based on the sensor input (estimated emotion Es), By combining the emotion prediction result (predicted emotion Eb) based on the action history, the estimation finally obtained by the emotion estimation system Emotion, for emotion change of the user, to allow natural recognition closer organism.

  Moreover, the predicted emotion change database (forward model) for estimating what kind of emotional change of the interaction target is caused by the user's own behavior is sequentially performed using the results obtained from the emotion estimation device 1 as teacher data. -Real-time learning of an emotion prediction model can be performed, and the prediction accuracy can be improved.

  In this way, not only the estimated emotion Es obtained from the sensor information, but also the experience of how one's own action changes the other's emotion based on the previous experience with human interaction, that is, the robot apparatus itself The estimated emotion E is estimated by constructing a predicted emotion change database that becomes an emotion transition model of the other person in consideration of the context of the other person's behavior, obtaining the predicted emotion Eb, and judging together with the estimated emotion Es from the emotion estimation device 1 Accuracy can be improved. Therefore, the robot apparatus can select an action according to the estimated emotion E indicating the emotion of the interaction target. For example, the robot apparatus expresses an action such as pleasing or entertaining the interaction target, thereby improving the entertainment property. be able to.

B: Robot Device Next, a specific example of a robot device equipped with the emotion recognition system described above will be described. In the present embodiment, a biped walking robot device will be described as an example, but it is needless to say that the present invention is not limited to a biped walking robot device and can be applied to a robot device that can be moved by four feet or wheels. .

  This humanoid robot device is a practical robot that supports human activities in various situations in the living environment and other daily life, and can act according to the internal state (anger, sadness, joy, fun, etc.) It is an entertainment robot that can express the basic actions performed by humans. Moreover, it is also possible to express an action according to the emotion of the user recognized by the above emotion recognition system instead of the internal state of the user. FIG. 13 is a perspective view showing an overview of the robot apparatus according to the present embodiment.

  As shown in FIG. 13, the robot apparatus 101 has a head unit 103 connected to a predetermined position of the trunk unit 102, two left and right arm units 104R / L, and two left and right leg units 105R /. L is connected to each other (provided that R and L are suffixes indicating right and left, respectively, and the same applies hereinafter).

  FIG. 14 schematically shows a joint degree-of-freedom configuration of the robot apparatus 101. The neck joint that supports the head unit 103 has three degrees of freedom: a neck joint yaw axis 111, a neck joint pitch axis 112, and a neck joint roll axis 113.

  Each arm unit 104R / L constituting the upper limb includes a shoulder joint pitch axis 117, a shoulder joint roll axis 118, an upper arm yaw axis 119, an elbow joint pitch axis 120, a forearm yaw axis 121, and a wrist. A joint pitch shaft 122, a wrist joint roll wheel 123, and a hand portion 124 are included. The hand part 124 is actually a multi-joint / multi-degree-of-freedom structure including a plurality of fingers. However, since the movement of the hand part 124 has little contribution or influence on the posture control or walking control of the robot apparatus 101, it is assumed in this specification that the degree of freedom is zero. Therefore, it is assumed that each arm portion has seven degrees of freedom.

  The trunk unit 102 has three degrees of freedom: a trunk pitch axis 114, a trunk roll axis 115, and a trunk yaw axis 116.

  Each leg unit 105R / L constituting the lower limb includes a hip joint yaw axis 125, a hip joint pitch axis 126, a hip joint roll axis 127, a knee joint pitch axis 128, an ankle joint pitch axis 129, and an ankle joint. It comprises a roll shaft 130 and a foot 131. In the present specification, the intersection of the hip joint pitch axis 126 and the hip joint roll axis 127 defines the hip joint position of the robot apparatus 101. The human foot 131 is actually a structure including a multi-joint / multi-degree-of-freedom sole, but in this specification, for simplicity, the bottom of the robot apparatus 101 has zero degrees of freedom. . Accordingly, each leg is configured with 6 degrees of freedom.

  In summary, the robot apparatus 101 as a whole has a total of 3 + 7 × 2 + 3 + 6 × 2 = 32 degrees of freedom. However, the robot device 1 for entertainment is not necessarily limited to 32 degrees of freedom. Needless to say, the degree of freedom, that is, the number of joints, can be increased or decreased as appropriate in accordance with design / production constraints or required specifications.

  Each degree of freedom of the robot apparatus 101 as described above is actually implemented using an actuator. It is preferable that the actuator be small and light in light of demands such as eliminating the extra bulge on the appearance and approximating the shape of a natural human body, and performing posture control on an unstable structure such as biped walking. .

  Such a robot apparatus includes a control system that controls the operation of the entire robot apparatus, for example, in the trunk unit 102. FIG. 15 is a schematic diagram illustrating a control system configuration of the robot apparatus 101. As shown in FIG. 15, the control system dynamically controls the whole body cooperative movement of the robot apparatus 1 such as driving of the actuator 450 and the thinking control module 300 that controls emotion judgment and emotional expression in response to user input and the like. And a control module 400.

  The thought control module 300 includes a central processing unit (CPU) 311, a random access memory (RAM) 312, a read only memory (ROM) 313, and an external storage device (hard disk This is an independent drive type information processing apparatus that is composed of 314 and the like and can perform self-contained processing in the module.

  The thought control module 300 determines the current emotion and intention of the robot apparatus 101 according to stimuli from the outside such as image data input from the image input apparatus 351 and audio data input from the audio input apparatus 352. That is, as described above, by recognizing the user's facial expression from the input image data and reflecting the information on the emotion and intention of the robot apparatus 101, it is possible to express an action according to the user's facial expression. Here, the image input device 351 includes a plurality of CCD (Charge Coupled Device) cameras, for example, and the audio input device 352 includes a plurality of microphones, for example.

  The thought control module 300 issues a command to the motion control module 300 to execute an action or action sequence based on decision making, that is, exercise of the limbs.

  One motion control module 400 includes a CPU 411 that controls the whole body cooperative motion of the robot apparatus 101, a RAM 412, a ROM 413, an external storage device (hard disk drive, etc.) 414, etc., and performs self-contained processing within the module. It is an independent drive type information processing apparatus that can be performed. Also, the external storage device 414 can store, for example, walking patterns calculated offline, target ZMP trajectories, and other action plans.

  The motion control module 400 includes an actuator 450 that realizes the degrees of freedom of joints distributed throughout the body of the robot apparatus 101 shown in FIG. 14, a distance measurement sensor (not shown) that measures the distance from the object, a body Posture sensor 451 for measuring the posture and inclination of the trunk unit 102, grounding confirmation sensors 452 and 453 for detecting left or right foot floor getting off or landing, load sensor provided on the foot sole 131 of the sole 131, power source such as a battery Various devices such as a power supply control device 454 for managing the network are connected via a bus interface (I / F) 401. Here, the posture sensor 451 is configured by, for example, a combination of an acceleration sensor and a gyro sensor, and the grounding confirmation sensors 452 and 453 are configured by proximity sensors, micro switches, or the like.

  The thought control module 300 and the motion control module 400 are constructed on a common platform, and are interconnected via bus interfaces 301 and 401.

  The motion control module 400 controls the whole body cooperative motion by each actuator 450 in order to embody the action instructed from the thought control module 300. That is, the CPU 411 extracts an operation pattern corresponding to the action instructed from the thought control module 300 from the external storage device 414 or generates an operation pattern internally. Then, the CPU 411 sets a foot movement, a ZMP trajectory, a trunk movement, an upper limb movement, a waist horizontal position, a height, and the like according to a specified movement pattern, and a command for instructing an action according to these setting contents. The value is transferred to each actuator 450.

  In addition, the CPU 411 detects the posture and inclination of the trunk unit 102 of the robot apparatus 101 based on the output signal of the posture sensor 451, and each leg unit 105R / L detects the free leg based on the output signals of the grounding confirmation sensors 452 and 453. Alternatively, the whole body cooperative movement of the robot apparatus 101 can be adaptively controlled by detecting whether the robot is standing or standing. Further, the CPU 411 controls the posture and operation of the robot apparatus 101 so that the ZMP position always moves toward the center of the ZMP stable region.

  In addition, the motion control module 400 is configured to return to the thought control module 300 the degree to which the intended behavior determined by the thought control module 300 has been expressed, that is, the processing status. In this way, the robot apparatus 101 can determine its own and surrounding conditions based on the control program, and can act autonomously.

  Such a robot apparatus is required to have a human interface technology that can respond within a predetermined time in a dynamically changing work environment. The robot apparatus 101 according to the present embodiment includes an emotion estimation system, thereby recognizing the emotions of surrounding users (owners, friends, or legitimate users) and controlling reactions based on the recognition results. By this, higher entertainment properties can be realized.

  It should be noted that the present invention is not limited to the above-described embodiments, and various modifications can be made without departing from the scope of the present invention. In addition, the emotion recognition system in the above-described embodiment may be realized by hardware or may be realized by causing an arithmetic unit to execute a computer program. In this case, the computer program can be provided by being recorded on a recording medium, or can be provided by being transmitted via the Internet or another transmission medium.

It is a block diagram which shows the emotion estimation apparatus in embodiment of this invention. It is a figure which shows typically the emotion estimation part 5 at the time of learning in the said emotion estimation apparatus. It is a figure for demonstrating the learning method in the recurrent neural network of the emotion estimation part in the said emotion estimation apparatus. It is a figure which shows typically the error predictor at the time of learning in the said emotion estimation apparatus. It is a figure for demonstrating the learning method in the recurrent neural network of the error predictor in the said emotion estimation apparatus. It is a figure which shows an example of a multilayer type neural network. It is a figure which shows an example of a recurrent type network. It is a flowchart which shows the back pro vacation learning method with respect to the weight value in a recurrent type network, a time constant, and an initial state. It is a figure which shows an example of learning data. (A) is a figure which shows the principal part in connection with the prediction emotion apparatus in a robot apparatus, (b) is a figure which shows a specific example of a prediction emotion change database. It is a figure explaining the principal part of the emotion recognition apparatus in embodiment of this invention. It is a figure which shows an example of the emotion estimation result in the embodiment of this invention, an emotion prediction result, and the result of integrating these. 1 is a perspective view showing an overview of a robot apparatus according to an embodiment of the present invention. It is a figure which shows typically the joint freedom degree structure which the robot apparatus comprises. It is a schematic diagram which shows the control system structure of the robot apparatus.

Explanation of symbols

DESCRIPTION OF SYMBOLS 1 Emotion estimation apparatus, 2 _m sensor part, 3 _n feature-value extraction part, 4 _n modal separate emotion estimation part, 7 _n softmax computing unit, 8 _n multiplier, 9 adder, 5 _n emotion recognizer, 6 _n error Predictor, 10 database, 31 internal state, 32 external stimulus, 30 behavior controller, 33 schema, 40 emotion prediction device, 41 predicted emotion change database, 42 predicted emotion calculation unit, 50 emotion estimation system, 52 emotion integration unit

Claims (28)

A feature quantity extraction means for extracting a time series feature quantity related to a recognition target for recognizing emotion from time series sensor information;
An emotion estimation device comprising: an emotion estimation unit that estimates the emotion of the recognition target in consideration of the context of the sensor information based on the time-series feature amount. The feature amount extraction means extracts the time-series feature amount for each modal for a plurality of modals as a modal-specific feature amount sequence,
The emotion estimation means includes a plurality of modal emotion recognition means for recognizing the emotion for each modal based on the modal feature quantity sequence, and a recognition for estimating the emotion based on a recognition result of the modal emotion recognition means. The emotion estimation apparatus according to claim 1, further comprising a result integration unit. The emotion estimation means includes prediction error calculation means for calculating a prediction error of the recognition result obtained by the modal emotion recognition means,
The emotion estimation apparatus according to claim 2, wherein the recognition result integration unit estimates the emotion based on a recognition result obtained by the modal emotion recognition unit and a prediction error thereof. The prediction error calculation means calculates a prediction error of the recognition result based on the recognition result obtained by the modal feature quantity sequence and / or the modal emotion recognition means. Emotion estimation device. The recognition result integration unit converts the prediction error obtained by the prediction error calculation unit into reliability, and based on the weighted recognition result obtained by weighting the recognition result obtained by each modal emotion recognition unit. The emotion estimation apparatus according to claim 3, wherein the emotion is estimated. When the emotion estimation means uses the time-series feature quantity related to the recognition target extracted from the time-series sensor information as input data, the emotion of the recognition target when the time-series sensor information is acquired is output data. The emotion estimation apparatus according to claim 1, which has been learned in advance by recursive learning. When the prediction error calculation means uses, as input data, a time-series feature quantity for each modal related to a recognition target extracted from time-series sensor information and / or a recognition result obtained by the modal-specific emotion recognition means. It is learned in advance by recursive learning so that the difference between the emotion of the recognition target when acquiring the sensor information of the series and the recognition result obtained by the modal-specific emotion recognition means becomes output data,
When the modal emotion recognition means uses the time-series feature quantity for each modal regarding the recognition target extracted from the time-series sensor information as input data, the emotion of the recognition target when the time-series sensor information is acquired is The emotion estimation apparatus according to claim 3, which has been learned in advance by recursive learning so as to be output data. The emotion estimation apparatus according to claim 6, wherein a neural network is used for the recursive learning. A feature quantity extraction step for extracting a time series feature quantity related to a recognition target for recognizing emotion from time series sensor information;
An emotion estimation method comprising: an emotion estimation step of estimating the emotion of the recognition target in consideration of the context of the sensor information based on the time-series feature amount. In the feature amount extraction step, the time-series feature amount for each modal is extracted as a feature amount sequence by modal for a plurality of modals,
The emotion estimation step includes the modal emotion recognition step for recognizing the emotion for each modal based on the modal feature quantity sequence, and the modal recognition result obtained in the modal emotion recognition step. An emotion estimation method according to claim 9, further comprising a recognition result integration step for estimation. A prediction error calculation step of calculating a prediction error of the recognition result obtained in the emotion recognition process according to modal,
The emotion estimation method according to claim 10, wherein in the recognition result integration step, the emotion is estimated based on a modal recognition result obtained in the modal emotion recognition step and a prediction error thereof. In the recognition result integration step, the prediction error calculated in the prediction error calculation step is converted into a reliability, and the recognition result obtained in each modal emotion recognition step is weighted to the weighted recognition result. The emotion estimation method according to claim 11, wherein the emotion is estimated based on the emotion. In a robotic device that behaves autonomously based on internal conditions and / or external stimuli,
One or more sensors that obtain information about the interaction object;
Feature quantity extraction means for receiving time series sensor information from the sensor and extracting time series feature quantities relating to the interaction target;
A robot apparatus comprising: an emotion estimation unit configured to estimate the emotion of the interaction target in consideration of the context of the sensor information based on the time-series feature amount. The feature amount extraction means extracts the time-series feature amount for each modal as a feature amount sequence by modal for a plurality of modals,
The emotion estimation means includes a plurality of modal emotion recognition means for recognizing the emotion for each modal based on the modal feature quantity sequence, and the emotion based on the recognition result obtained by the modal emotion recognition means. The robot apparatus according to claim 13, further comprising a recognition result integrating unit for estimating. The emotion estimation means includes prediction error calculation means for calculating a prediction error of the recognition result obtained by the modal emotion recognition means,
The robot apparatus according to claim 14, wherein the recognition result integrating unit estimates the emotion based on a recognition result obtained by the modal emotion recognition unit and a prediction error thereof. Action execution means for selecting and executing one action from a plurality of actions;
An emotion prediction means for predicting the emotion of the interaction target according to the type of action executed by the action execution means;
14. The robot apparatus according to claim 13, further comprising emotion integration means for estimating the emotion of the interaction target based on the estimation result obtained by the emotion estimation means and the prediction result obtained by the emotion prediction means. . The emotion prediction means predicts an emotional change after execution of an action by referring to an expected emotional change database in which one action and an expected emotional change expected to change after the execution of the one action are associated with each other, The robot apparatus according to claim 16, wherein the emotion after the execution of the behavior is predicted based on the prediction result and the estimation result estimated by the emotion estimation means before the execution of the behavior. The robot apparatus according to claim 17, wherein the predicted emotion change database is obtained by learning the expected emotion change based on an emotion change of the interaction target before and after the execution of each action. Database learning that updates the predicted emotion change in the predicted emotion change database based on the difference in emotion of the interaction target before and after the execution of the behavior estimated by the emotion estimation unit every time the behavior is executed by the behavior execution unit The robot apparatus according to claim 17, further comprising: means. The emotion integration unit weights the estimation result and the prediction result by a parameter that changes so that the estimation result by the emotion estimation unit is more important than the prediction result by the emotion prediction unit as time elapses after execution of the action, The robot apparatus according to claim 16, wherein the emotion is estimated based on the weighted result. In an emotion recognition method for a robot apparatus that acts autonomously based on an internal state and / or an external stimulus,
A feature amount extraction step of extracting a time series feature amount related to an interaction target from time series sensor information;
An emotion estimation method for a robot apparatus, comprising: an emotion estimation step for estimating an emotion of the interaction target in consideration of the context of the sensor information based on the time-series feature amount. In the feature quantity extraction step, the time series feature quantity for each modal is extracted as a modal-specific feature quantity sequence for a plurality of modals,
The emotion estimation step includes the modal emotion recognition step for recognizing the emotion for each modal based on the modal feature quantity sequence, and the modal recognition result obtained in the modal emotion recognition step. The method of recognizing an emotion of a robot apparatus according to claim 21, further comprising a recognition result integrating step of estimating. A prediction error calculation step of calculating a prediction error of the recognition result obtained in the emotion recognition process according to modal,
The emotion recognition method for a robot apparatus according to claim 22, wherein, in the recognition result integration step, the emotion is estimated based on the recognition result obtained in the modal emotion recognition step and a prediction error thereof. An emotion prediction step of predicting the emotion of the interaction target according to the type of the executed action when the one action is executed by the action executing means for selecting and executing one action from a plurality of actions;
The emotion integration step of estimating the emotion of the interaction target based on the estimation result obtained in the emotion estimation step and the prediction result obtained in the emotion prediction step. An emotion recognition method for a robotic device. In the emotion prediction step, an emotion change after the execution of an action is predicted by referring to an expected emotion change database in which one action and an expected emotion that is expected to change after the execution of the one action are associated with each other. The emotion recognition method for a robot apparatus according to claim 24, wherein the emotion after the execution of the action is predicted based on the result and the estimation result obtained in the emotion estimation step before the execution of the action. In the emotion integration step, the estimation is performed according to a parameter that changes so as to emphasize the estimation result estimated in the emotion estimation step from the prediction result obtained in the emotion prediction step as time elapses after execution of the action. 25. The emotion recognition method for a robot apparatus according to claim 24, wherein a result and a prediction result are weighted, and the emotion is estimated based on the weighted result. In the learning method of the robot apparatus equipped with the emotion recognition device that recognizes the emotion of the interaction target from the given input data,
Learning the emotion recognition device using the time-series feature quantity related to the interaction target extracted from the time-series sensor information as input data and the emotion of the interaction target when the time-series sensor information is acquired as the output target value. A learning method for a robot apparatus, characterized by comprising a learning step. In a robotic device that behaves autonomously based on internal conditions and / or external stimuli,
An emotion recognition device for recognizing the emotion of an interaction target from given input data;
One or more sensors that obtain information about the interaction object;
Feature quantity extraction means for receiving time series sensor information from the sensor and extracting a time series feature quantity relating to the interaction target;
Learning of the emotion recognition apparatus using, as input data, a time-series feature amount related to an interaction target extracted from the time-series sensor information, and using the emotion of the interaction target when the time-series sensor information is acquired as an output target value Learning means to
The emotion recognition apparatus recognizes the emotion using the time-series feature quantity extracted by the feature quantity extraction unit as the input data.

Images