JP5437297B2 - Dialog state estimation apparatus, method, and program - Google Patents

Description

  The present invention relates to a dialog state estimation apparatus, method, and program, and in particular, a dialog state estimation apparatus, method, and program for estimating a state of empathy between a plurality of conversation persons or a state of facial expressions of a plurality of conversation persons. About.

  Face-to-face dialogue is the most basic form of communication for sharing information with others, understanding other people's emotions, and making decisions when a person lives in a social life. Therefore, research on automatic meeting analysis has attracted attention. If there is a variety of information, such as how the turn-taking (speech switching) is performed in the meeting, who is the main person, and what emotions the interlocutor has? If there is a system that can support the meeting well, the meeting will be more enjoyable, meaningful and productive. However, in the state of such a meeting, the feelings of the interlocutor have hardly been dealt with in the field of automatic meeting analysis so far. Moreover, although it is important to automatically estimate the state of emotions of the interlocutor in the group meeting, no attempt has been made.

  Here, a facial expression recognition technique is known as a related prior art. The subject of recent facial expression recognition has shifted from the six basic facial expressions of happiness, surprise, anger, sadness, disgust, and fear, which were the subjects of early facial expression recognition research, to spontaneous natural facial expressions. For example, Non-Patent Document 1 proposes a method for identifying the smile, the ridicule, and the other three types expressed during the meeting.

  In addition, a technique for estimating personal feelings is known. In Non-Patent Document 1 above, the network automatically recognizes who is sending a smile to who in the meeting scene, and counts how long it has been done in total. We have proposed a method to visualize interpersonal feelings between interlocutors by displaying them. At this time, who sent the smile to whom is determined from the facial expression and head direction of each conversation person automatically estimated from the video.

  In addition, a technique for estimating a meeting state as a whole group is known. For example, as a high-level state related to a meeting, a technology for automatically estimating a state of a dialogue regime such as a mononogue or a dialog and a person having initiative is proposed (for example, Patent Document 1). Patent Document 1 proposes a dialog model having a three-layered hierarchical structure consisting of a regime state, who and who are interacting with each other, and the actions of a dialog person.

  In addition, a method for estimating a network about whether there is a direct relationship between persons in a large-scale group called a social network or sociometry has been proposed (Non-patent Documents 2 and 3). These networks are directly required from relatively simple information such as physical proximity (Non-Patent Document 2) and motion information in an image (Non-Patent Document 3).

JP 2006-338529 A

S. Kumano, K. Otsuka, D. Mikami and J. Yamato, "Recognizing communicative facial expressions for discovering interpersonal emotions in group meetings", In Proc. Int'l Conf.Multimodal Interfaces and Workshop on Machine Learning for Multimodal Interaction (ICMI -MLMI 2009), 2009. N. Eagle, A. Pentland, and D. Lazer.Inferring social network structure using mobile phone data.In Proc.the National Academy of Sciences (PNAS), volume 106, pages 15274-15278, 2009. L. Ding and A. Yilmaz. Learning relations among movie characters: A social network perspective.In In Proc.European Conference on Computer Vision (ECCV2010), volume IV, pages 410-423, 2010.

  However, the technique described in Non-Patent Document 1 has a problem in that attention is paid only to a person who recognizes facial expressions, and the context in which those facial expressions are expressed is not taken into consideration. Furthermore, the problem of the technique described in Non-Patent Document 1 described above is that the facial expression directed toward someone directly expresses emotions for that person. In other words, in this method, a scene that conveys emotions held by a third party or an object to others, for example, negative feelings for an object that the two parties do not like, expresses negative expressions to each other. Can't handle scenes that come out and sympathize with each other.

  In addition, the technique described in Patent Document 1 has a problem that the state relating to the emotion of the conversation person is not handled.

  Further, the techniques described in Non-Patent Documents 2 and 3 have a problem that little attention is paid to the state of a group related to a short-time emotion.

  The present invention has been made in order to solve the above-described problem, and a dialogue state estimation device and a dialogue state estimation method capable of estimating a state of empathy among a plurality of talkers or a facial expression state of each talker. And to provide a program.

  In order to achieve the above object, an apparatus for estimating a conversation state according to the present invention uses a line-of-sight state that detects a state of a line of sight between a plurality of interlocutors by using an image obtained by imaging an area including a plurality of conversational person faces. Setting initial values of parameters of the dialogue model representing the co-occurrence of the state of expression according to the combination of the state of the sympathy between the plurality of dialogues and the state of the line of sight between the plurality of dialogues And an initial value setting means for setting an initial value of a state indicating empathy between the plurality of interactors and an initial value indicating a state of each of the plurality of interactors according to the dialog model; The initial values of the parameters set by the initial value setting means, the state showing the empathy, and the state showing the expression, or the previously determined parameters, the state showing the empathy, and the expression Based on the state shown and the state of the line of sight between the plurality of interrogators detected by the line-of-sight state detecting means, according to the dialogue model, State determining means for determining a state indicating each expression of the interlocutor, a state indicating empathy between the plurality of interrogators determined by the state determining means, and a state indicating the expression of each of the plurality of interrogators Based on the parameter determination means for determining the parameters of the interaction model, the determination by the state determination means and the determination by the parameter determination means are repeated until a predetermined convergence condition is satisfied, And an estimation means for estimating a state showing empathy or a state showing each facial expression of the plurality of interlocutors.

  The dialog state estimation method according to the present invention is a dialog state estimation method in a dialog state estimation device including a line-of-sight state detection unit, an initial value setting unit, a state determination unit, a parameter determination unit, and an estimation unit. The apparatus receives, as an input, an image obtained by imaging a region including a plurality of conversational person faces by the visual line state detection unit, and detects the state of the visual line between the plurality of conversational parties, and the initial value setting unit, Setting an initial value of a parameter of a dialogue model representing the co-occurrence of a state of expression according to a combination of a state showing empathy between a plurality of dialogues and a state of gaze between the plurality of dialogues; Setting an initial value of a state indicating empathy among the plurality of interlocutors and an initial value indicating a state of each of the plurality of interrogators according to a dialog model; The initial value of each of the parameter set by the initial value setting unit, the state indicating the empathy, and the state indicating the facial expression by the state determination unit, or the parameter determined last time, the state indicating the empathy And a state showing empathy among the plurality of interlocutors according to the dialog model based on the state indicating the facial expression and the state of the line of sight between the plurality of interlocutors detected by the gaze state detecting means. Determining a state indicating the facial expression of each of the plurality of interlocutors; and a state indicating sympathy between the plurality of interrogators determined by the state determining unit by the parameter determining unit; A step of determining parameters of the dialogue model based on a state indicating each expression of the dialogue person; and And the determination by the parameter determining means are repeated until a predetermined convergence condition is satisfied, and a state showing empathy among the plurality of talkers or a state showing the facial expressions of the plurality of talkers is estimated. And the step of executing.

  According to the present invention, the line-of-sight state detection means detects the state of the line of sight between the plurality of interlocutors using as input an image obtained by imaging an area including the faces of the plurality of interlocutors. The initial value of the parameter of the dialogue model that represents the co-occurrence of the state of the facial expression according to the combination of the state showing empathy among a plurality of interlocutors and the state of the line of sight between the plurality of interlocutors And an initial value of a state indicating empathy among the plurality of interlocutors and an initial value indicating the expression of each of the plurality of interrogators are set according to the dialog model.

  Then, the initial value of each of the parameter set by the initial value setting unit, the state indicating the empathy, and the state indicating the facial expression, or the parameter determined last time, the empathy by the state determination unit. Based on the state shown, the state showing the facial expression, and the state of the line of sight between the plurality of interrogators detected by the line-of-sight state detecting means, according to the dialogue model, And a state indicating the facial expression of each of the plurality of interactors. The parameter determination means determines the parameters of the conversation model based on the state indicating empathy between the plurality of interrogators determined by the state determination means and the state indicating each facial expression of the plurality of conversation persons. .

  Then, the estimation unit repeats the determination by the state determination unit and the determination by the parameter determination unit until a predetermined convergence condition is satisfied, and the state showing empathy among the plurality of interlocutors or the plurality of dialogs The state which shows each person's facial expression is estimated.

  In this way, by using the dialogue model that represents the co-occurrence of facial expression states according to the combination of the state showing empathy among a plurality of talkers and the state of gaze between the plurality of talkers, It is possible to estimate the state of empathy among the parties or the state of facial expression of each interlocutor.

  The state determining means according to the present invention is characterized in that the initial value of each of the parameters set by the initial value setting means, the state indicating the empathy, and the state indicating the facial expression, or the parameter determined last time, Based on the state of empathy, the state of the facial expression, and the state of the line of sight between the plurality of interlocutors detected by the line-of-sight state detection means, according to the dialog model, between the plurality of interlocutors The first probability distribution indicating the probability that the state exhibiting empathy becomes each state, and the second probability distribution indicating the probability that each state indicating the expression of each of the plurality of interlocutors becomes each state are obtained, and the obtained Determining a state indicating empathy among the plurality of interlocutors according to the first probability distribution, and determining a state indicating each facial expression of the plurality of interlocutors according to the obtained second probability distribution; The parameter determination means is configured to determine whether each parameter of the conversation model is based on a state indicating empathy between the plurality of interactors determined by the state determination means and a state indicating each expression of the plurality of interactors. A third probability distribution indicating a probability that is a value can be obtained, and parameters of the dialogue model can be determined according to the obtained third probability distribution.

  The line-of-sight state detection means according to the present invention detects time-series data of the line-of-sight state between the plurality of interrogators, using the time-series data of the image as input, and the initial value setting means In addition to setting initial values of parameters, initial values of time-series data in a state showing empathy among the plurality of interlocutors, and initial values of time-series data in the state of each facial expression of the plurality of interrogators are set. The state determining means determines time-series data indicating a state of empathy among the plurality of interlocutors and time-series data indicating a state of each of the plurality of interrogators, and the estimating means includes The time-series data indicating the empathy among the plurality of interlocutors or the time-series data indicating the state of each of the plurality of interrogators can be estimated. This makes it possible to accurately estimate the state of empathy among a plurality of interlocutors or the state of facial expression of each interlocutor.

  The dialogue state estimation apparatus according to the present invention further includes a facial expression estimation unit that estimates a facial expression state of each of the plurality of conversation persons based on the image, and the state determination unit is set by the initial value setting unit. The state indicating the facial expression of each of the plurality of interlocutors can be determined based on the obtained parameter and the estimation result by the facial expression estimation means.

  The line-of-sight state between the plurality of interlocutors according to the present invention may be mutual gaze, one-sided gaze, and mutual gaze, or mutual gaze and one-sided gaze. Moreover, the state which shows the sympathy between the said several interlocutors can be made into sympathy, indifference, and a counter feeling. In addition, the facial expression can be positive, neutral, negative, or expressionless, smiling, surprised, and disgusted.

  The estimation unit according to the present invention repeats the determination by the state determination unit and the determination by the parameter determination unit until a predetermined convergence condition is satisfied, and then further determines by the state determination unit and the parameter determination unit. Based on a state indicating empathy among the plurality of interrogators determined by the state determining means when the determination is repeated a plurality of times, or a state indicating the expression of each of the plurality of interrogators Thus, it is possible to estimate a state indicating empathy among the plurality of interlocutors or a state indicating the facial expression of each of the plurality of interlocutors.

  The program according to the present invention is a program for causing a computer to function as each unit of the above-described dialog state estimation device.

  As described above, according to the dialogue state estimation device, the dialogue state estimation method, and the program of the present invention, a combination of a state showing empathy among a plurality of talkers and a line-of-sight state between the plurality of talkers. By using the dialogue model representing the co-occurrence of the corresponding facial expression state, it is possible to estimate the sympathetic state among a plurality of interlocutors or the facial expression state of each interlocutor.

(A), (B) It is a figure for demonstrating a dialogue model. It is a figure for demonstrating (A) and (B) expression co-occurrence matrix. (A), (B) It is a figure for demonstrating the estimation method of the state of a facial expression. It is the schematic which shows the structure of the dialogue state estimation apparatus which concerns on embodiment of this invention. (A) The figure which shows a mode that a conversation is image | photographed, (B) The image figure of the image which image | photographed each dialog person. It is a flowchart which shows the content of the dialog state estimation process routine in the dialog state estimation apparatus which concerns on embodiment of this invention.

  Hereinafter, embodiments of the present invention will be described in detail with reference to the drawings.

<Overview>
First, an outline of the present invention will be described. The key to a deep understanding of the meeting is to understand what interpersonal relationships are built and how consensus is formed as a result through the exchange of emotional messages between the interlocutors. It is to be. We consider the basic units of such emotional messages to be empathy, indifference, and antipathy at the meeting. For example, empathy often represents consent or intimate interpersonal relationships. Sympathy causes a coordinated behavior between the interlocutors called the chameleon effect or motor mimicry. Sympathy can cause not only positive emotions such as smiles, but also negative emotions such as pity. On the other hand, although less clear than empathy, antipathy and indifference are expected to easily cause behavioral disagreement.

  In addition, the combination of facial expression and line of sight is particularly important among the interlocutor behaviors related to empathy and antipathy. This is because they can achieve a large amount of emotional messages in an instant and directional manner (specifying the opponent). Facial expressions are the primary method for expressing emotional messages. For example, smiles express positive emotions and build and prove intimate relationships with others. Neutral (not positive or negative) facial expressions, including no expression, can be regarded as a counterfeit reaction when the other person has a positive facial expression. The other line of sight has various functions, but monitoring and triggering are particularly important in empathic interaction. Monitoring is essential for observing the facial expressions of others and reading their feelings. Triggering causes a person to look at him and to react to that person. With these two functions, the two perform mutual gaze (eye contact), and emotional messages are instantly exchanged and shared via facial expressions. Mutual gaze also causes synchronized behavior more strongly.

  In the following, the word “sympathetic interaction” is defined as a short-time event in which two parties exchange emotional messages by combining facial expressions and line of sight. As a result, sent emotional messages may be shared, or ignored or rejected. In short, sympathetic interaction represents "who is sympathetic, indifferent, or dissatisfied with whom". Moreover, the state of empathic interaction is an example of a state showing empathy.

  Here, we will introduce an example of how emotional messages are exchanged between the two parties by the combination of facial expression and line of sight. Consider the case where person A turns a certain expression toward person B. At this time, if the two are sympathetic with each other, the person B returns an expression similar to the person A. For example, a smile is a smile, and a sad face is a sad expression. On the other hand, if the two are rebelling, person B may express a different expression from person A. For example, they dislike smiles or laugh at embarrassed expressions. Two people who are indifferent to each other often have neutral facial expressions regardless of their facial expressions.

  Therefore, in the present invention, facial expressions, which are the main means of expressing emotional messages exchanged between the interlocutors during the meeting, and empathic interaction performed in a deeply related manner are simultaneously estimated. In the present invention, attention is paid to facial expressions and line of sight as a conversational behavior that is a key of empathic interaction. This is because the combination realizes transmission of a large amount of sympathetic messages that are instantaneous and directed. In the present invention, there is a deep relationship between a sympathetic interaction and a facial expression of a dialogue pair, and both facial expressions co-occur according to the type of sympathetic interaction. Based on this assumption, the present invention uses a probabilistic model and simultaneously estimates empathic interaction and facial expression using video data as observation data.

<Outline of conversation model>
Next, a dialogue model used in the dialogue state estimation apparatus proposed in the present invention will be described.

  In this embodiment, a hierarchical dynamic Bayesian network is used as one method for modeling the relationship between empathic interaction and the behavior of the interlocutor. In this model, the state of the top layer is discrete, it transitions according to a Markov process, and that state dominates the state of the lower layer. Here, the empathic interaction is the upper layer, and the interlocutor action including the facial expression and the line of sight is the middle layer. Furthermore, the conversational person behavior affects the video signal which is the lower layer.

FIG. 1A shows a hierarchical dynamic Bayesian network for estimating the state of empathic interaction and the state of facial expression from a video signal. Here, t represents a discrete time step, where t = 1,. Nodes represent variables, and arrows represent probabilistic causal relationships between variables. In the model of FIG. 1A, there are three variables: empathic interaction {E _t } _{t = 1} ^T , facial expression {F _t } _{t = 1} ^T , and line of sight {X _t } _{t = 1} ^T It is. As shown in FIG. 1B, it is assumed that the state of sympathetic interaction and the state of the line of sight of each pair of interactors influence the expressions of both. This relationship is modeled by a facial expression co-occurrence matrix described later. It is assumed that the type of sympathetic interaction also affects the state of gaze. This is due to the fact that in the data used as a preliminary study, mutual gaze and one-sided gaze occurred frequently in the interaction between empathy and antipathy, and a lot of disorientation occurred in indifference.

A set of states of sympathetic interaction of all pairs of dialoguers at time t is represented by E _t = {e _t ^{(i, j)} } _{(i, j) ∈r} . Here, e ^{(i, j)} ∈ {1,..., N _e } represents the type of interaction between the pair (i, j) of the conversation person i and the conversation person j, and r is N × (N -1) Represents a set of dialogue pairs. N _e is the number of sympathetic interaction types. In the present embodiment, e ^{(i, j)} ∈ {“sympathy”, “indifference”, “disapproval”}, and N _e = 3. A set of facial expression states of all the interrogators at time t is represented by F _t = {f _{i, t} } _{i = 1} ^N. Here, f _i ε {1,..., N _f } represents the expression state of the conversation person i, and N _f represents the number of expression states. In the present embodiment, f _i ∈ {“positive”, “neutral”, “negative”}, and N _f = 3.

A set of line-of-sight states of all the conversation partner pairs at time t is represented by X _t = {x _t ^{(i, j)} }. Here, x _t ^{(i, j)} represents the line-of-sight state of the dialogue pair (i, j), and assumes any one state of mutual gaze, one-side gaze, and mutual gaze. That is, the number of line-of-sight states is N _x = 3.

Image at time t (one frame in the video data) representing at I _t. Assume that all the interlocutors are included in this image. This image I becomes observation data. In this embodiment, only the image is used as the observation data, but the audio data in the video may be used together.

<Expression co-occurrence matrix>
Here, a facial expression co-occurrence matrix that models the influence of empathic interaction and line-of-sight, which is an important part of the present invention, on the pattern of co-occurrence of facial expressions among interlocutors will be described. Here, only the line of sight is included in the dialog model as a dialog person action excluding facial expressions, but other dialog person actions such as head gestures may be included in the dialog model.

  In the present embodiment, the sympathetic interaction is defined between each pair of interactors, that is, two parties, and takes a discrete state. Here, it is assumed that there are three sympathetic interaction states: “sympathy”, “indifference”, and “antisense”.

  Facial expressions are defined for each conversation person, and can be classified into multiple states. Although any state can be used in the present invention, in the present embodiment, three states of “positive”, “neutral”, and “negative” are assumed. In addition to this, states such as “no expression”, “smile”, “surprise”, and “disgust” may be used.

  Here, three states of “mutual gaze”, “one-sided gaze”, and “mutual gaze” are defined as gaze states. Mutual gaze is a state in which the two are looking at each other, that is, in eye contact. A one-sided gaze is a state in which only one conversation person of a conversation person pair is looking at the other conversation person. Mutual distraction is a state where the two are not looking at each other.

  Here, the frequency pattern regarding what facial expressions are likely to be expressed by each other is called a facial expression co-occurrence matrix.

  2A and 2B show facial expression co-occurrence matrices for each combination of the state of sympathetic interaction and the state of gaze. These facial expression co-occurrence matrices use labels manually attached to empathic interactions, facial expressions, and line-of-sight conditions for actual conversation data (total conversation time is about 30 minutes). It has been created. In FIG. 2, the brighter the color of an element of the facial expression co-occurrence matrix, the higher the frequency of the element. The “hard to judge” status indicates that it is difficult for the labeled person to label the frame as either sympathy, indifference, or disagreement. Means. For example, when the two are not directly interacting with each other. This facial expression co-occurrence matrix is symmetric in terms of mutual gaze and mutual gaze by definition, but is generally asymmetric for unilateral gaze. In FIG. 2B, the one-side gaze represents a person who is watching the person i and a person who is watching the person j. In FIG. 2A, there is a clear pattern difference between the facial expression co-occurrence matrices. This shows the validity of the hypothesis in the present invention that the expression co-occurrence matrix changes the expression co-occurrence pattern depending on the combination of the state of sympathetic interaction and the state of gaze. Furthermore, many of the features of these facial expression co-occurrence patterns are generally consistent with the psychological findings that have been clarified so far. a) First, in the state of empathy, many co-occurrences of positive facial expressions are seen. b) Mutual gaze causes more synchronization of actions, that is, matching facial expressions. For example, the co-occurrence of positive facial expressions increases in the order of mutual gaze, one-sided gaze, and mutual gaze.

  Other intuitively relevant features are also obtained. c) In antipathy, negative facial expressions are likely to be expressed by one or both interlocutors. d) With indifference, a combination of a neutral facial expression and another facial expression (positive facial expression or negative facial expression) is remarkable, especially in mutual gaze. e) In the scene labeled as difficult to judge, co-occurrence of neutral facial expressions is conspicuous.

  In the above data, there was a non-negligible difference between the expression co-occurrence matrices even in the state of mutual distraction. The reason for this is that the behavior of others can be obtained from peripheral vision and auditory information, even if they are not necessarily looking at the person in the conversation. Therefore, in this embodiment, the state of mutual distraction is also added to the model. However, in the framework of the present invention, the mutual distraction state may be excluded from the model.

  The facial expression co-occurrence matrix is generated by normalizing a frequency matrix (original frequency matrix) obtained by simply counting frequencies based on the frequency of occurrence of each facial expression combination. Specifically, the expression co-occurrence matrix shown in FIG. 2 (B) is first obtained for each combination of facial expressions in the case of a combination of a specific line-of-sight state and empathic interaction for the interlocutors i and j. The actual frequency is counted (original frequency matrix), and then the matrix is calculated by dividing the frequency matrix by the frequency vector indicating the frequency of each combination of facial expressions expressed by each of the interrogators i and j. , A normalized frequency matrix. This normalization further clarifies the difference in facial expression co-occurrence matrix for each sympathetic interaction.

  On the other hand, the original frequency matrix is greatly influenced by the expression frequency of the facial expression category that changes depending on the theme and type of the conversation and the personality of the individual. In particular, in a scene where there are others around, people tend to express a lot of smiles, so the original frequency matrix has a high co-occurrence of positive facial expressions.

  In the present embodiment, the facial expression co-occurrence is modeled in what combination the two facial expressions occur simultaneously, but the time lag between the facial expressions of both may be considered. In other words, it may be modeled as a co-occurrence of the expression of one conversational person i at a certain time and the expression of the other conversational person j after δt time. In this case, the conversation person i may be a conversation person who first turned his gaze toward the other party, or a conversation person who first expressed some other expression from no expression.

  Furthermore, you may consider co-occurrence of facial expressions with more than three parties. For example, it is effective in situations where a lot of interlocutors participating in the dialogue laugh when someone jokes, or everyone has a difficult expression without a good discussion.

  Moreover, although it is assumed that there are three states as the sympathetic interaction, the number of states may be one. In this case, the state of sympathetic interaction is meaningless, but rather than recognizing each conversational person's facial expression individually using information on what facial expression co-occurrence is likely to occur between the conversational persons. It may be possible to estimate with high accuracy. For example, there are scenes where the face of the interlocutor in the video is not facing the front but sideways or upward, or where the face is partially hidden by a hand or object.

<Prior distribution>
In this embodiment, a conjugate prior distribution with respect to the posterior distribution shown in the following equation is used as the parameter prior distribution because it is easy to handle mathematically.

Here, k is a convenient index indicating which parameter is represented, and φ _k represents π, θ, or λ. As the parameter prior distribution P (φ _k ), the Dirichlet distribution generally used for discrete variables is used.

In the dialog state estimation apparatus according to the present embodiment, the observation data series I _{1: T} is used as an input, and the empathic interaction time series E _{1: T} and facial expression time series F ₁ based on the proposed dialog model. _{: T} and model parameter φ are estimated simultaneously. In the present embodiment, the simultaneous posterior probability distribution p (E _{1: T} , F _{1: T} , φ | I _{1: T} ) of all these random variables is estimated in the framework of Bayesian inference. In Bayesian inference, a prior distribution of model parameters p (φ) is introduced as model prior information.

The joint posterior probability distribution for all variables is expressed by the following equation (1) using the joint probability distribution and the prior probability of the observation data I _{1: T.}

p (E1 _{: T} , F1 _{: T} , X1 _{: T} , φ | I1 _{: T} ) = p (E1 _{: T} , F1 _{: T} , X1 _{: T} , I1 _{: T} , φ) / p (I _{1: T} )
... (1)

Since the observation data I _{1: T} is known and p (I _{1: T} ) can be regarded as a constant, the following equation (2) is obtained.

p (E _{1: T} , F _{1: T} , X _{1: T} , φ | I _{1: T} ) ∝p (E _{1: T} , F _{1: T} , X _{1: T} , I _{1: T} , φ) (2)

  Here, the ∝ symbol represents that the left side is proportional to the right side. Here, according to the Bayesian network shown in FIG. 1A, the simultaneous probability distribution on the right side of the above equation (2) is defined as shown in the following equation (3).

p (E _{1: T} , F _{1: T} , X _{1: T} , φ | I _{1: T} )
: = P (φ) P (E _{1: T} | φ) P (X _{1: T} | E _{1: T} , φ) P (F _{1: T} | E _{1: T} , X _{1: T} , φ) P ( I _{1: T} | F _{1: T} , φ) (3)

  After this, unless otherwise required, the model parameter φ is omitted for simplicity.

Assuming that the sympathetic interaction is independent between the dialogue pairs and follows the first-order Markov process, the prior probability P (E _{1: T} ) of the state of the sympathetic interaction is expressed by the following equation (4) _: expressed.

here,

Represents the initial probability of the state of empathic interaction,

Represents the transition probability of the state of the sympathetic interaction. These parameters are prepared for each dialogue pair. Both of these probabilities are time-invariant and are further expressed by the following equation (7).

The likelihood P (X _{1: T} | E _{1: T} ) of the sympathetic interaction with the line-of-sight state is assumed to be independence with respect to each pair of dialoguers and time, and as a product of those likelihoods. Define. That is, the likelihood P (X _{1: T} | E _{1: T} ) of the sympathetic interaction with the line-of-sight state is expressed by the following equation (8).

  Regarding this likelihood, as shown in the following equation (9), each dialogue pair has a parameter individually.

Parameter vectors π _· = {π _{·, e ′} | e′∈ {1,. . . , N _e }} are distributed according to independent Dirichlet distributions.

The conditional probability P (F _{1: T} | E _{1: T} , X _{1: T} ) of the facial expression given the state of sympathetic interaction and the state of gaze is the prior probability of the facial expression and the facial expression co-occurrence matrix Expressed as the product of That is, the conditional probability P (F _{1: T} | E _{1: T} , X _{1: T} ) of the facial expression is expressed by the following equation (10).

Here, P (f _{i, 1: T} ) is a prior distribution of the expression of the conversation person i, and assumes independence among the conversation persons. The prior distribution of the facial expressions of each conversation person is expressed by the following equation (11), assuming independence of facial expressions between times.

M _{e, x (i, j)} (f, f ′) is the expression co-occurrence matrix for the dialogue pair (i, j) when the empathic interaction state is e and the line-of-sight state is x. The (f, f ′) component is represented. The expression co-occurrence matrix M exists for each combination of the state of sympathetic interaction and the state of gaze, and there are a total of N _e × N _x . The size of each facial expression co-occurrence matrix M is N _f × N _f .

  Each interactor pair and interactor has parameters for facial prior probabilities and facial expression co-occurrence matrices. As for the parameters for the prior distribution, the probability that the state of the expression of the conversation person i is f ′ is expressed by the following equation (12).

On the other hand, for the conversation partner pair (i, j), when the state of empathic interaction is e and the line of sight is x, the conversation person i's facial expression is f and the conversation person j's facial expression is f '. A certain probability M _{e, x} ^{(i, j)} (f, f ′) is expressed by the following equation (13).

The expression co-occurrence matrix which is an N _f × N _f matrix has a sum of 1 elements. Here, each expression co-occurrence matrix is regarded as one vector whose number of elements is N _f × N _f and the sum of those elements is 1. An arbitrary shape can be used as a model of the distribution of each parameter. In this embodiment, the vector of facial expression prior distribution parameters

And a vector of facial expression co-occurrence matrix parameters

The Dirichlet distribution that is independent of each other is used. Here, f ″ ∈ {1,..., N _f × N _f }.

  Here, when all the parameters of the dialogue model are collected, φ = {Π, Θ}. The model parameter に 関 す る relating to the sympathetic interaction is expressed by the following equation (16).

  The model parameter Θ relating to the expression is expressed by the following equation (17).

  The parameter prior probability distribution p (φ) is the product of the prior distributions of these respective parameters.

  In the present embodiment, the model parameters are prepared for each pair of dialoguers or dialogues. However, the parameters may be shared by the dialogue pairs or dialogues.

Further, the likelihood P (I _{1: T} | F _{1: T} ) of the facial expression state with respect to the observed video frame I is assumed to be the following (assuming that the facial expression is independent between the interlocutors and between the times) _: 18) It expresses by a formula.

The likelihood P (I _t | f _{i, t} ) of the expression state may be calculated by any method. First, it is conceivable to use a likelihood that can be directly calculated from video data. For example, the method described in JP2009-110426A can be used. In this method, the distribution of the luminance (brightness of the image) of many points sparsely arranged on the face in each facial expression is modeled using a normal distribution, and the facial expression of the target person is derived from the image. , And the position and orientation of the face are estimated simultaneously.

  Alternatively, an operation of how each facial part such as eyebrows, eyes, and mouth, which is called a face action unit, has moved may be sandwiched between the video signal and the facial expression. In this case, first, it is only necessary to detect which face action unit is generated from the video signal, and then calculate the probability of each facial expression with the detected face action unit as a likelihood. As a detection method of the face action unit, for example, a plurality of feature points on the face as shown in FIGS. 3A and 3B are tracked in an image, and a change in geometric arrangement between these feature points is performed. There is a method for detecting the occurrence of a face action unit. As a feature point tracking method, there is template matching using a small rectangular region centered on a feature point as a template. For the detection of the facial action unit in the mouth, in addition to the image information, information on the presence / absence of utterance detected from the audio signal by threshold processing or the like based on the audio power may be used. The change in the geometry between feature points is the distance between the two points or the change in the angle between the three points, and how they are when the face action unit is occurring and not This can be detected by modeling using a normal distribution.

It is also possible to use a recognizer that outputs only one facial expression state estimated with respect to one input image instead of the likelihood of each facial expression state. In this case, for example, the likelihood of the recognized state is set to an appropriate constant τ (0 << τ <1), and the likelihood is set to (1−τ) / (N _f for facial expression states other than the recognized state. -1). The likelihood of facial expression states other than the recognized state is also set to a small positive value in order to avoid that the a posteriori probability is always 0 and the estimation result is never obtained when the likelihood is 0.

<System configuration>
Next, a case where the present invention is applied to a dialog state estimation device that estimates video data, which is time series of image data, and estimates sympathetic interaction and facial expressions between a plurality of interlocutors, An embodiment will be described.

  As shown in FIG. 4, the dialogue state estimation device according to the present embodiment includes an input unit 1 that accepts input of video data obtained by photographing a plurality of interrogators, the state of sympathetic interaction of all pairs of dialoguers, and each A calculation unit 2 that estimates the state of the expression of the conversation person and an output unit 3 that outputs the estimation result are provided.

  The input unit 1 is connected to known photographing means using a CCD camera, a microphone, and the like, and accepts input of conversation video data including faces of a plurality of conversation persons photographed by the photographing means. The input unit 1 is also connected to a keyboard and mouse, and receives information input by operating the keyboard and mouse.

  As a main camera arrangement method, there are a method in which one omnidirectional camera is arranged in the vicinity of the center between the talkers, and a method in which one camera is arranged in the front direction of each talker. In the framework of the present invention, any arrangement method may be used as long as it is an arrangement that can photograph the faces of all the interlocutors. In this embodiment, an omnidirectional camera is used as the photographing means. This is because, when an omnidirectional camera is used, in order to specify the direction of the viewer's line of sight required by the present invention without obtaining external parameters (translation and rotation with respect to the world coordinate system) of the camera, This is because the positional relationship between them can be easily grasped from the positional relationship in the image of the target person regardless of the number of interlocutors, which is advantageous. On the other hand, in the method using multiple cameras, it is easy to increase the resolution of each conversation person's face, but in addition to the complexity of the hardware configuration, if the number of conversation persons and the seat arrangement differ for each conversation, It is necessary to perform calibration to obtain external parameters of each camera each time. FIG. 5A shows an example of a shooting scene. FIG. 5B shows an example of an input image taken with an omnidirectional camera.

  It is assumed that the internal parameters of the camera connected to the input unit 1 have already been obtained by prior calibration.

  The output unit 3 is implemented by a display, a printer, a magnetic disk, or the like.

  The calculation unit 2 is composed of a computer having a CPU (Central Processing Unit), a RAM (Random Access Memory), and a ROM (Read Only Memory) storing a program for executing a dialog state estimation processing routine described later. Functionally, it is configured as follows. The calculation unit 2 includes a data storage unit 21, an estimation unit 22, and a learning unit 23.

  The data storage unit 21 stores input video data and hyperparameters learned by the learning unit 23. The hyper parameter is a model parameter for the prior distribution of parameters estimated by the estimation unit 22.

  The estimation unit 22 receives video data and hyperparameters as input, and based on the above-described dialogue model, the state of empathic interaction of each dialogue pair in each frame of the video, the state of facial expression of each dialogue person, and dialogue Estimate model parameters simultaneously.

Problems solved by estimator 22, based on the interaction model described above, the observation signals I _{1: when T} is given, sympathetic time series of interactions E _{1: T,} the time series expression F _{1: T,} and models This is to estimate the simultaneous posterior probability distribution p (E _{1: T} , F _{1: T} , φ | I _{1: T} ) of the parameter φ. In the present embodiment, a Gibbs sampler is used as one method for solving this problem. Gibbs sampler is a kind of Markov chain Monte Carlo method, and it is excellent for approximating complex models. The Gibbs sampler repeats this set a number of times, with a set of sampling all the variables to be estimated one by one in order. The distribution on which the sampling is based is an all-conditional posterior distribution in which all variables other than the variable to be sampled are fixed. At this time, the unbiased distribution matches the posterior distribution to be estimated. This simultaneous posterior probability distribution is obtained based on statistics obtained from random variables sampled after the Markov chain has converged.

Further, it is assumed that the number of conversational persons _Ni is known. The present invention targets conversations in which the number of conversations N _i is 2 or more. Although the present invention can target any number of interlocutors, the present embodiment will be described for a four-person dialog (N _i = 4). The number of interlocutors can be automatically calculated by using an existing face detector (for example, a cascade type AdaBoost detector based on a Haar-like feature value). The cascade type AdaBoost detector is described in non-patent literature (P. Viola and M. Jones, “Rapid object detection using a boosted cascade of simple features”, In Proceedings of the IEEE Computer Society Conference on Computer Vision and Pattern. (Recognition, pp. 511-518, 2001)).

  The estimation unit 22 includes a gaze direction detection unit 29, a facial expression estimation unit 30, an initial value setting unit 31, a state sampling unit 32, a parameter sampling unit 33, a convergence determination unit 34, and an estimated value calculation unit 35. The state sampling unit 32 is an example of a state determination unit, and the parameter sampling unit 33 is an example of a parameter determination unit. Moreover, the convergence determination part 34 and the estimated value calculation part 35 are examples of an estimation means.

  The line-of-sight direction detection unit 29 specifies an area representing each interactor's face for each frame of the video data, extracts an eye area, and uses a conventionally known method based on the extracted eye area. Detect gaze direction. Further, the line-of-sight direction detection unit 29 determines, for each conversation person, which conversation person the direction of the line-of-sight is facing based on the detected line-of-sight direction and the positional relationship of all the conversation persons. For all pairs (i, j), the line-of-sight state (mutual gaze, one-sided gaze, mutual gaze) of the dialogue pair (i, j) is detected.

The facial expression estimation unit 30 specifies an area representing each interactor's face for each frame of the video data, estimates the facial expression from the face area, and expresses the likelihood of the facial expression P (I _t | f _{i, t} ). Is calculated.

  The initial value setting unit 31 sets an initial value of the parameter φ = {Π, Θ} using a hyperparameter given in advance. In the present embodiment, a case where a value stored in the data storage unit 21 is used as the hyperparameter value will be described, but an appropriate random number may be used as the hyperparameter value.

In addition, the initial value setting unit 31 uses the initial value of the parameter φ = {Π, Θ}, and the initial value of each random variable, that is, the state of the sympathetic interaction regarding each dialoguer pair (i, j). The initial value e _t ^{(i, j)} of the series data and the initial value f _{i, t} of the time series data of the state of facial expression of each conversation person i are sampled. Specifically, a random number generated according to the distribution given by the above equations (5) and (6) is used as the initial value of e _t ^{(i, j)} . Similarly, random numbers generated according to the distribution given by equation (12) are set as initial values f _{i, t} .

The state sampling unit 32 samples the sympathetic interaction state _et ^{(i, j)} at each time t and the facial expression state f _{i, t at} each time t from all conditional probability distributions, and updates the values. To do.

  The all conditional probability distribution for each variable has the same distribution shape as the conjugate prior distribution because the conjugate prior distribution is used as the prior distribution. In this all-conditional probability distribution, all variables other than the target variable are fixed, so a distribution that does not include the target variable from the simultaneous posterior probability distribution represented by the above equation (3) is a normalized constant. Can be considered.

The state e _t ^{(i, j)} of the sympathetic interaction at time t is sampled according to the all conditional probability distribution expressed by the following equation (19).

  The all-conditional probability distribution represented by the above equation (19) is an example of the first probability distribution.

At the time of the first execution by the state sampling unit 32, first, for each time of the term on the right side of the above equation (19), using the values set by the initial value setting unit 31 for the time t = 1 (first frame). The probability is obtained by the above equations (6), (13), and (9). Then, a random number generated according to the obtained probability distribution is set as e ₁ ^{(i, j)} after update. Similar processing is repeated in order from time t = 2 to t = T, and e _t ^{(i, j)} is updated.

At the second and subsequent executions by the state sampling unit 32, instead of the values set by the initial value setting unit 31, the updated values of e _t ^{(i, j)} , f _{i, t} , φ are used, By performing similar sampling, the value of e ^t _{(i, j)} is updated.

Similarly, the facial expression state f _{i, t at} time t is sampled according to an all conditional probability distribution expressed by the following equation (20).

  Note that the all conditional probability distribution represented by the above equation (20) is an example of the second probability distribution.

At the time of the first execution by the state sampling unit 32, first, for each time t = 1 (first frame), each value set by the initial value setting unit 31 and the likelihood of the facial expression calculated by the facial expression estimation unit 30 are used. Thus, the probability of each term on the right side of the above equation (20) is obtained by the above equations (12) and (13). Then, the random number generated according to the obtained probability distribution is set as f _{i, t} after the update. Similar processing is repeated in order from time t = 2 to t = T, and f _{i, t} is updated.

At the second and subsequent executions by the state sampling unit 32, instead of the values set by the initial value setting unit 31, the updated values of e _t ^{(i, j)} , f _{i, t} , φ are used, By performing similar sampling _, the value of f _{i, t} is updated.

  The parameter sampling unit 33 samples the parameter φ = {Π, Θ} from the all conditional probability distribution and updates the value.

  Since all model parameters are assumed to follow the Dirichlet distribution, the target event occurs in the previous sampling of the sympathetic interaction and facial expression by the state sampling unit 32 with respect to the parameter of the prior distribution (ie, the hyper parameter). A value obtained by adding the number of times of the calculation can be obtained as a parameter of the posterior distribution.

  Each model parameter is sampled according to an all conditional probability distribution expressed by the following equations (21) to (25).

Here, D (π _e | η _e ^* ) is a Dirichlet distribution with respect to π with a parameter η _e ^*, and the symbol “˜” means that the distribution on the left side follows the distribution on the right side. For example, the parameter of the transition probability of the sympathetic interaction is obtained according to the following equations (26) and (27).

Here, η is a hyper parameter, and η ^* indicates a value updated using the sampling result. Also, _ne , _{e ′} (e = 0) is the number of times the state of empathic interaction is e, _{and ne} , _{e ′} (e ≠ 0) is the state of empathic interaction at a certain time. This is the number of changes from e to e ′ at the next time.

Similarly, η _x ^* is {η _{x, e '} ^* } _{e'∈ {1, ..., Ne}} , the line-of-sight state is x, and the state of sympathetic interaction is e'. Η _{e, x} ^* is {η _{e, x, f "} ^* } _{e, x, f"} , and the expression of the dialogue pair (i, j) is f _i , f _j , Represents the number of times that the line-of-sight state was x and the state of empathic interaction was e. Also, η _i ^* is {η _{i, f} ^* } _{i, f} and represents the number of times that the state of facial expression of the conversation person i is f.

  The parameter sampling unit 33 sets the random numbers generated according to the distributions on the right side of the equations (21) to (25) as the updated values of the parameters π, λ, and θ, respectively, so that the parameter φ = {Π, Θ} Update. Note that the all-conditional probability distribution represented by the equations (21) to (25) is an example of a third probability distribution.

The convergence determination unit 34 determines whether or not a predetermined convergence condition is satisfied. If the predetermined convergence condition is not satisfied, the state sampling unit 32 and the parameter sampling unit 33 repeat the processing, and the state of empathic interaction The time series e _t ^{(i, j)} , the time series f _{i, t of} the facial expression state _, and the parameter φ are updated.

  Here, the predetermined convergence condition is “whether a predetermined number of iterations has been reached” or “an error between each time series (or parameter) before update and each time series (or parameter) after update. Is used or not ”or the like is used.

The estimated value calculation unit 35 repeats the convergence determination unit 34 until a predetermined convergence condition is satisfied, and after the parameters of the conversation model have converged, the M number of samples {E _{1: T} ^{( q)} , F _{1: T} ^(q) , φ (q)} _{q = M ′ + 1} ^{M ′ + M} is used to calculate the estimation results of the state of empathic interaction and the state of facial expression (M ′ is the convergence The number of iterations required until the parameter converges in the determination unit 34). The subscript (q) is a sample number (q = 1,..., M). The number of repetitions may depend on, for example, M ′ = 800 and M = 200, depending on the number of sympathetic interactions and facial expressions to be estimated.

The estimated value ^ e _t ^{(i, j)} of the state of sympathetic interaction at time t and the estimated value ^ f _{i, t} of the expression state are expressed by the simultaneous posterior probability distribution p (f _t , e _t | I _{1: t} ). For example, these estimated values are calculated as a state that maximizes the peripheral probability for each of the sympathetic interaction or facial expression of the simultaneous posterior probability distribution, as shown in the following equations (28) and (29). The estimated parameter ^ φ is calculated as the expected value of the peripheral distribution for the parameter of the simultaneous posterior probability distribution, as shown in the following equation (30).

Here, δ _y (z) is a function that returns 1 if y = z and returns 0 otherwise.

Empathetic estimate interaction state ^{_{^ e t (i, j)}} , the estimated value of the facial expression of the state ^ f _{i, t,} and the estimated value of the parameter ^ phi is output to the user from the output unit 3.

  The learning unit 23 includes a learning data storage unit 40, a teacher label creation unit 41, and a hyper parameter learning unit 42.

  The value of the hyper parameter used in the initial value setting unit 31 may be determined in advance. However, the hyper parameter of the model is learned by inputting a dialogue video (learning video) other than the estimation dialogue video. You can also. By performing learning, the initial value of the parameter φ can be set to a more appropriate value, so that it is possible to converge to a more accurate model with a predetermined number of iterations, resulting in empathic interaction and facial expression. This has the effect of improving the accuracy of estimation.

  The learning data storage unit 40 stores learning video data.

  The teacher label creating unit 41 receives learning video data as an input, and creates a state of empathic interaction, a facial expression state, and a line-of-sight state in each frame as a teacher label. For empathic interactions and facial expressions that are effective to be estimated according to the present invention, the input of the teacher label from the operator is accepted by the input unit 1 and these labels are given. In addition, even if it takes enormous calculation time, if there is a method capable of estimating these with high accuracy, the teacher label creating unit 41 may be used. On the other hand, the line-of-sight state may be labeled by performing the same processing as the line-of-sight direction detection unit 29.

The hyper parameter learning unit 42 receives the teacher label as an input, learns the hyper parameter of the dialogue model, and outputs it to the data storage unit 21. Here, the probability that the event represented by each hyper parameter has occurred in the learning data is set as the value of the hyper parameter. For example, with regard to the transition probability P (e _t = e ′ | e _t−1 = e) of the sympathetic interaction, e is changed to e ′ within two times when the state of the sympathetic interaction is continuous in the teacher label. And the value obtained by dividing the number of times by the sum of the number of times of all transitions (probability) may be set as the value of the hyperparameter.

<Operation of dialog state estimation device>
Next, the operation of the dialog state estimation apparatus according to this embodiment will be described. First, the dialogue state estimation device causes the operator to input the state of empathic interaction between the conversation parties and the state of facial expression of each conversation person for each frame of the learning video data stored in the learning data storage unit 40. Then, the labeling is performed, and the calculation unit 2 of the dialog state estimation apparatus detects the state of the line of sight between the interrogators and performs the labeling. Then, the calculation unit 2 of the dialog state estimation apparatus learns hyperparameters using the labeled results, and stores the learning results in the data storage unit 21.

  In addition, the photographing unit photographs a state in which a plurality of interlocutors are interacting, and the photographed video data is input to the conversation state estimation device via the input unit 1 and stored in the data storage unit 21.

  Then, a dialogue state estimation processing routine shown in FIG. 6 is executed in the calculation unit 2 of the dialogue state estimation device.

  First, in step 100, the video data and hyper parameters stored in the data storage unit 21 are read and acquired. Then, in step 102, the line-of-sight states of all pairs of the interactors are detected for each time (each frame) of the video data. In step 103, for each time (each frame) of the video data, the state of the facial expression of each conversation person is estimated, and the likelihood of each state of the facial expression is calculated.

  In the next step 104, initial values of various parameters of the dialogue model are set based on the hyper parameters acquired in the above step 100, and all the pairs of the dialoguers are set based on the initial values of the various parameters of the set dialogue model. The initial value of the time-series data of the state of empathic interaction and the time-series data of the state of facial expression of each dialogue person is set.

  Then, in step 106, time-series data of the state of empathic interaction of all pairs of the talkers set in step 104, time-series data of the state of expression of each talker, and initial values of various parameters of the dialogue model Or the time series data of the state of empathic interaction of all pairs of interactors previously determined in step 106 and step 108 to be described later, the time series data of the state of expression of each interactor, and various parameters of the conversation model, Based on the time-series data of the line-of-sight states of all pairs of the conversation persons detected in step 102, and the time-series data of the likelihood of each state of each conversation person's facial expression calculated in step 103, The probability distributions represented by Equations (19) and (20) are obtained, and sampling is performed according to the obtained probability distributions for all pairs of the interlocutors. Time-series data of sensitive Interaction state, and to determine the time-series data of the facial expression of the state of each interlocutor.

  In the next step 108, the time series data of the state of empathic interaction of all pairs of the conversation parties determined in the previous step 106, and the time series data of the state of each conversation person's facial expression are detected in the above step 102. On the basis of the time-series data of the line-of-sight states of all the pairs of conversational persons and the hyperparameters acquired in step 100, the probability distributions expressed by the above equations (21) to (25) are obtained and obtained. According to the probability distribution, the values of various parameters of the dialogue model are determined by sampling.

  In step 110, it is determined whether a predetermined number of iterations has been reached as a predetermined convergence condition. If the predetermined number of iterations has not been reached, it is determined that the predetermined convergence condition has not been established. Then, the process returns to step 106 and the processes of steps 106 and 108 are repeated. On the other hand, if the predetermined number of iterations has been reached, it is determined that a predetermined convergence condition has been established, and the routine proceeds to step 112.

  In step 112, as in step 106, the time-series data of the sympathetic interaction state of all the previously determined pairs of the interlocutors, the time-series data of the facial expression states of the interlocutors, and various parameters of the dialogue model On the basis of the time series data of the gaze state of all the detected pairs of the interlocutors and the time series data of the likelihood of each state of the facial expressions of each interrogator calculated in the above step 103 (19) The probability distribution represented by the equation (20) is obtained, and according to the obtained probability distribution, the time series data of the state of the sympathetic interaction of all the pairs of the talkers and the state of the facial expression of each talker are obtained by sampling. Determine time-series data.

  Then, in step 114, as in step 108 above, time-series data of the state of empathic interaction of all pairs of interactors determined in previous step 112, and time-series data of the states of facial expressions of each interactor Based on the time-series data of the line-of-sight states of all pairs of interrogators detected in the step 102 and the hyperparameters acquired in the step 100, they are expressed by the above equations (21) to (25). A probability distribution is obtained, and values of various parameters of the dialogue model are determined by sampling according to the obtained probability distribution.

  In step 116, it is determined whether or not the processes of steps 112 and 114 have been repeated M times. If the processes have not been repeated M times, the process returns to step 112 and the processes of steps 112 and 114 are repeated. On the other hand, when the processes of steps 112 and 114 are repeated M times, in step 118, the time series data of the state of empathic interaction of all pairs of conversation persons determined in steps 112 and 114 above. , Time series data on the state of facial expression of each conversation person, time series data on the state of empathic interaction of all pairs of conversation persons, and time series data on the state of facial expression of each conversation person, based on various parameters of the conversation model And each estimated value of various parameters of the dialogue model. Then, in step 120, the time-series data of the state of empathic interaction of all pairs of interactors calculated in step 118, the time-series data of the state of expression of each interactor, and the various parameters of the conversation model are estimated. The value is output by the output unit 3, and the dialog state estimation processing routine is terminated.

  As described above, according to the dialogue state estimation device according to the present embodiment, the co-occurrence of the state of the facial expression according to the combination of the state of sympathetic interaction in the dialogue partner pair and the state of the gaze of the dialogue pair By using the dialogue model representing, it is possible to estimate the state of the sympathetic interaction of the dialogue partner pair and the state of the facial expression of each dialogue person.

  In addition, taking into account the context in which facial expressions are expressed for dialogues by multiple people, the state of facial expressions expressed by each dialoguer is correctly estimated from the video data of the dialogue, and at the same time, It is possible to correctly estimate the state of empathic interaction between them.

  It is possible to estimate the state of the dialogue person's facial expression with higher accuracy than the conventional estimation framework focusing only on the target person. In particular, it is difficult to correctly estimate with the conventional method, even in a scene where the face direction of the person in the video is not front but sideways or upward, or where the face is partially hidden by a hand or object The state of facial expression can be estimated accurately.

Also, as the parameters of the dialogue model, the probability of occurrence of each state of empathic interaction (parameter Π ₀ ) and the dialogue behavior (= facial expression) for each target dialogue video It is possible to obtain the probability of occurrence (parameter Θ) and how much they co-occur (parameter λ). These differences in parameters can characterize each interaction.

  Note that the present invention is not limited to the above-described embodiment, and various modifications and applications are possible without departing from the gist of the present invention.

  For example, it is possible to output an estimated value of only one of the state of the sympathetic interaction of the conversation partner pair and the state of the facial expression of each conversation person.

  In addition, the state of empathic interaction, the state of facial expression of each conversation person, and the parameters of the conversation person model may be estimated only for a specific pair of conversation persons among a plurality of conversation persons.

  Further, based on a specific frame in the video data, that is, still image data at one time, the state of empathic interaction at that time, the state of facial expression of each conversation person, and the parameters of the conversation person model are estimated. Also good.

  Moreover, although the above-mentioned dialog state estimation apparatus has a computer system inside, if a "computer system" is using the WWW system, it shall also include a homepage provision environment (or display environment). .

  In the present specification, the embodiment has been described in which the program is installed in advance. However, the program can be provided by being stored in a computer-readable recording medium.

DESCRIPTION OF SYMBOLS 1 Input part 2 Calculation part 3 Output part 21 Data storage part 22 Estimation part 29 Gaze direction detection part 30 Facial expression estimation part 31 Initial value setting part 32 State sampling part 33 Parameter sampling part 34 Convergence determination part 35 Estimated value calculation part

Claims (10)

Eye gaze state detecting means for detecting an eye gaze state between the plurality of interlocutors, using as an input an image obtained by imaging an area including the faces of the plurality of interlocutors,
Setting an initial value of a parameter of a dialogue model representing the co-occurrence of a state of expression according to a combination of a state showing empathy between a plurality of dialogues and a state of gaze between the plurality of dialogues; According to a dialogue model, an initial value setting means for setting an initial value of a state showing empathy between the plurality of talkers, and an initial value showing a state of each of the plurality of talkers,
The initial value of each of the parameter set by the initial value setting means, the state showing the empathy, and the state showing the expression, or the previously determined parameter, the state showing the empathy, and the expression Based on the state shown and the state of the line of sight between the plurality of interrogators detected by the line-of-sight state detecting means, according to the dialogue model, State determining means for determining a state indicating each expression of the dialogue person;
Parameter determining means for determining parameters of the dialog model based on a state indicating empathy among the plurality of interrogators determined by the state determining means and a state indicating the expression of each of the plurality of interrogators;
The determination by the state determination unit and the determination by the parameter determination unit are repeated until a predetermined convergence condition is satisfied, and a state showing empathy between the plurality of interlocutors or each facial expression of the plurality of interlocutors is obtained. An estimation means for estimating the state to be indicated;
A dialogue state estimation device including: The state determination means indicates the initial value of the parameter set by the initial value setting means, the state indicating the empathy, and the state indicating the facial expression, or the parameter determined last time, the empathy Based on the state, the state indicating the facial expression, and the state of the line of sight between the plurality of interlocutors detected by the line-of-sight state detecting means, the empathy between the plurality of interlocutors is shown according to the dialog model. A first probability distribution indicating a probability that each state becomes each state, and a second probability distribution indicating a probability that each state indicating the expression of each of the plurality of interlocutors becomes each state, and the obtained first probability distribution. And determining a state indicating empathy among the plurality of interlocutors, and determining a state indicating each facial expression of the plurality of interrogators according to the obtained second probability distribution,
The parameter determination unit is configured so that each parameter of the conversation model is determined based on a state indicating empathy between the plurality of interactors determined by the state determining unit and a state indicating each expression of the plurality of interactors. The dialog state estimation apparatus according to claim 1, wherein a third probability distribution indicating a probability that is a value is obtained, and parameters of the dialog model are determined according to the obtained third probability distribution. The line-of-sight state detection means receives time-series data of the image as input and detects time-series data of the line-of-sight state between the plurality of interlocutors,
The initial value setting means sets an initial value of a parameter of the dialogue model, an initial value of time-series data in a state showing empathy between the plurality of dialogues, and each facial expression of the plurality of dialogues Set the initial value of the time series data of the state,
The state determination means determines time-series data indicating a state of empathy between the plurality of interlocutors and time-series data indicating a state of each of the plurality of interrogators,
The dialogue according to claim 1 or 2, wherein the estimation means estimates time-series data indicating a state of empathy among the plurality of interactors, or time-series data indicating a state of each of the plurality of interactors. State estimation device. Further comprising facial expression estimation means for estimating the state of facial expression of each of the plurality of interlocutors based on the image;
The said state determination means determines the state which shows each facial expression of these dialog persons based on the said parameter set by the said initial value setting means, and the estimation result by the said facial expression estimation means. 4. The dialogue state estimation apparatus according to any one of items 3.   The conversation state estimation apparatus according to any one of claims 1 to 4, wherein the gaze state between the plurality of interlocutors is a mutual gaze, a one-sided gaze, and a mutual gaze, or a mutual gaze and a one-sided gaze.   The dialog state estimation apparatus according to any one of claims 1 to 5, wherein states indicating empathy among the plurality of interrogators are sympathy, indifference, and anti-feeling.   The dialog state estimation device according to any one of claims 1 to 6, wherein the facial expression states are positive, neutral, and negative, or no expression, smile, surprise, and disgust.   The estimation unit repeats the determination by the state determination unit and the determination by the parameter determination unit until a predetermined convergence condition is satisfied, and then further includes a plurality of determinations by the state determination unit and determinations by the parameter determination unit. Based on a state indicating empathy among the plurality of interrogators determined by the state determination means when repeated a plurality of times, or a state indicating the expression of each of the plurality of interrogators. The conversation state estimation apparatus according to claim 1, wherein a state indicating sympathy between the plurality of conversational persons or a state indicating each facial expression of the plurality of conversational persons is estimated. A dialog state estimation method in a dialog state estimation apparatus including a line-of-sight state detection unit, an initial value setting unit, a state determination unit, a parameter determination unit, and an estimation unit,
The dialog state estimation device includes:
Detecting a line-of-sight state between the plurality of interlocutors by inputting an image obtained by imaging an area including the faces of the plurality of interlocutors by the line-of-sight state detecting unit;
By the initial value setting means, initial values of parameters of the dialogue model representing the co-occurrence of facial expression states according to a combination of a state showing empathy among a plurality of talkers and a line-of-sight state between the plurality of talkers Setting a value and setting an initial value of a state indicating empathy between the plurality of interactors and an initial value indicating a state of each of the plurality of interactors according to the interaction model; and
The initial value of each of the parameter set by the initial value setting unit, the state indicating the empathy, and the state indicating the facial expression by the state determination unit, or the parameter determined last time, the empathy Based on the state, the state indicating the facial expression, and the state of the line of sight between the plurality of interlocutors detected by the line-of-sight state detecting means, the empathy between the plurality of interlocutors is shown according to the dialog model. Determining a state and a state indicating each facial expression of the plurality of interactors;
The parameter determination means determines the parameters of the conversation model based on the state indicating empathy among the plurality of interrogators determined by the state determination means and the state indicating each facial expression of the plurality of conversation persons. And steps to
The estimation means repeats the determination by the state determination means and the determination by the parameter determination means until a predetermined convergence condition is satisfied, or a state showing empathy among the plurality of interactors, or the plurality of interactors Estimating a state indicating each facial expression of
The dialog state estimation method characterized by including and performing.   The program for functioning a computer as each means of the dialog state estimation apparatus of any one of Claims 1-8.