JP2015075907A - Emotion information display control device, method of the same, and program - Google Patents

Abstract

PROBLEM TO BE SOLVED: To provide a technology to control the display of emotion information so as for persons to easily catch the emotion information.SOLUTION: An emotion information display control device includes: an emotion information acquisition unit that takes the degree of emotion between two persons forming each of pairs, which is formed of two persons among a plurality of persons, as emotion information, and acquires emotion information between two persons among three or more persons from a video capturing the three or more persons; and a control unit that controls to switch at least two or more among (1) emotion information, of a pieces of emotion information, between a first person, who is one of three or more persons, and another person, (2) emotion information, of pieces of emotion information, other than the emotion information between the first person and another person, and (3) all the emotion information, and display the resulting pieces of information on a display device.

Description

  The present invention relates to a technique for displaying the degree of emotion between a plurality of interlocutors.

  As a technique for automatically estimating the sympathy / antisense between the two parties, it has been proposed to estimate how the sympathy / antisense of the two parties interacting with each other is interpreted from a group of external observers ( Patent Document 1 and Non-Patent Document 1). Non-Patent Document 1 considers that the interpretation is different for each external observer, and the variability of interpretation within the external observer group, that is, the votes of each of the three states, empathy / disapproval / neither The problem of estimating the rate is set. In Non-Patent Document 1, the degree of emotion (“sympathy”, “disapproval”, and “neither”) among a plurality of interrogators is displayed on one display device.

JP 2012-185727 A

Shiro Kumano, Kazuhiro Otsuka, Amami Mikami, Junji Yamato, “Estimation model and evaluation of empathy / antisense based on facial expression and gaze for multi-person dialogue”, IEICE Technical Report, Human Communication Fundamentals Study Group, HCS 111 (214), pp. 33-38, 2011.

  However, since the conventional technique basically displays all the feelings between a plurality of interlocutors on a single display device regardless of the usage situation, it is difficult to grasp the degree of emotions between the plurality of interrogators. . When the two target parties are not looking at each other (the mutual distraction state), the degree of emotion between the two is not displayed, but the mutual distraction state changes with time, for example, there is It is not easy to grasp the degree of emotion between one particular person and the other person in time series.

  An object of this invention is to provide the technique which controls the display of emotion information so that emotion information can be grasped easily.

  In order to solve the above-described problem, according to the first aspect of the present invention, the emotion information display control device configures each pair of each pair composed of two persons among a plurality of persons. An emotion information acquisition unit for obtaining emotion information between two persons out of three or more from a video in which three or more persons are photographed, with the degree of emotion between the two persons performing emotion information as the emotion information; (1) Among emotion information, emotion information between the first person who is one of three or more and others, (2) Among emotion information, emotion information between the first person and others (3) a control unit that controls to switch at least two or more of all emotion information and display them on the display device.

  In order to solve the above problems, according to another aspect of the present invention, the emotion information display control device configures each pair of each pair composed of two persons among a plurality of persons. An emotion information acquisition unit for obtaining emotion information between two persons from an image in which three or more persons are photographed with the degree of emotion between the two persons as emotion information, The emotion information between the first person who is one of the three or more and the other person is displayed on the first display device which is one of the plurality of display devices. The emotion information other than the emotion information between the first person and the other person is controlled to be displayed on the second display device which is one of the plurality of display devices other than the first display device. And a control unit.

  In order to solve the above-described problem, according to another aspect of the present invention, an emotion information display control method configures each pair of each pair composed of two persons among a plurality of persons. An emotion information acquisition step for obtaining emotion information between two persons from the image in which three or more persons are photographed with the degree of emotion between the two persons as emotion information, and (1) emotion Of the information, emotion information between the first person who is one of three or more and the other person, (2) Of emotion information, other than the emotion information between the first person and the other person Emotion information (3) including a control step of switching so that at least two or more of all emotion information is switched and displayed on the display device.

  In order to solve the above-described problem, according to another aspect of the present invention, an emotion information display control method configures each pair of each pair composed of two persons among a plurality of persons. Emotion information acquisition step for obtaining emotion information between two people out of three or more people from an image in which the degree of emotion between two people is taken as emotion information, and among the emotion information The emotion information between the first person who is one of the three or more and the other person is displayed on the first display device which is one of the plurality of display devices. The emotion information other than the emotion information between the first person and the other person is controlled to be displayed on the second display device which is one of the plurality of display devices other than the first display device. Control steps.

  According to the present invention, it is possible to control the display of emotion information so that the emotion information can be easily grasped.

The figure which shows the example of arrangement | positioning of the emotion information display control apparatus which concerns on 1st embodiment. The figure which illustrates the function structure of the emotion information display control apparatus which concerns on 1st embodiment. The figure which illustrates the processing flow of the emotion information display control apparatus which concerns on 1st embodiment. The figure which illustrates the functional structure of an emotion information acquisition part. The figure which illustrates the function structure of a parameter learning part. The figure which illustrates the processing flow of a learning phase. The figure which illustrates the processing flow of an estimation phase. The figure explaining a time difference function. The figure explaining the time difference of a dialogue person's action and empathy interpretation. The figure explaining a change timing function. The figure explaining the effective range of a change timing function. The figure explaining the effective range of a change timing function. Emotion information between the first person 2-i (2-1 in the figure) and the other person 2-j (2-2, 2-3, or 2-4 in the figure) in the first embodiment FIG. 5 is a diagram illustrating an example of an image displayed on a display device. Emotion information between the first person 2-i (2-1 in the figure) and the other person 2-j (2-2, 2-3, or 2-4 in the figure) in the first embodiment The figure showing the example of the image | video by which emotion information other than is displayed on a display apparatus. The figure showing the example of the image | video by which all emotion information is displayed on a display apparatus in 1st embodiment. The figure showing the example of the image | video displayed on a display apparatus, when it controls so that each posterior probability distribution may be displayed with a bar graph etc. FIG. In the first embodiment, the first person 2-i (not shown in the figure) and the other person 2-j (not shown in the figure) when the first person is controlled not to be displayed on the display device. The figure which shows the example of the image | video by which the emotion information between 2-2, 2-3, or 2-4) is displayed with a display apparatus. In the first embodiment, the first person 2-i (not shown in the figure) and the other person 2-j (not shown in the figure) when the first person is controlled not to be displayed on the display device. The figure which shows the example of the image | video by which emotion information other than the emotion information between 2-2, 2-3, or 2-4) is displayed with a display apparatus. The figure which shows the example of the image | video by which all emotion information is displayed on a display apparatus when a 1st person is controlled not to display on a display apparatus in 1st embodiment. The figure which shows the example of arrangement | positioning of the emotion information display control apparatus which concerns on 2nd embodiment. The figure which illustrates the processing flow of the emotion information display control apparatus which concerns on 2nd embodiment. The figure which illustrates the processing flow of the emotion information display control apparatus which concerns on 3rd embodiment.

  Hereinafter, embodiments of the present invention will be described. In the drawings used for the following description, constituent parts having the same function and steps for performing the same process are denoted by the same reference numerals, and redundant description is omitted. In the following description, the symbol “˜” and the like used in the text should be described immediately above the immediately preceding character, but are described immediately after the character due to restrictions on text notation. In the formula, these symbols are written in their original positions. Further, the processing performed for each element of a vector or matrix is applied to all elements of the vector or matrix unless otherwise specified.

<Emotion information display control apparatus 100 according to the first embodiment>
FIG. 1 shows an arrangement example of the emotion information display control device 100 according to the first embodiment. The emotion information display control device 100 receives a video of N or more people as an input, and outputs a video with emotion information added to the display device 3. N is an integer of 3 or more. In addition, emotion information represents the degree of emotion between two persons constituting each pair of each pair composed of two persons among a plurality of persons. It should be noted that emotions that are common to two persons are targeted, and in this embodiment, three types of emotions of “sympathy”, “antisense”, and “neither” are used as emotions. Here, empathy is a state in which the emotional states of the two are similar, and a feeling of dissatisfaction is a state in which they are different. These three types of emotions are used as emotion information. It can be said that this emotion information indicates a state of empathy between the interlocutors.

  FIG. 2 shows a functional configuration example of the emotion information display control apparatus 100, and FIG. 3 shows a processing flow thereof. The emotion information display control device 100 includes an emotion information acquisition unit 110 and a control unit 120.

<Emotion information acquisition unit 110>
The emotion information acquisition unit 110 receives a video in which N people are photographed, obtains emotion information between two of the N people from this video (s1), and outputs the obtained emotion information to the control unit 120. The video taken by N people may be a video prepared by preparing one camera for each conversation person and multiplexing a plurality of videos taken by a plurality of cameras (see FIG. 1), or using a fisheye lens. For example, it may be an image obtained by photographing all the participants with one omnidirectional camera. For example, as shown in FIG. 1, a person 2-n is photographed from a camera 1-n (where n = 1, 2,..., N), and N images multiplexed may be input. An input of a plurality of persons appearing in one video may be used.

  The control unit 120 receives the video and emotion information taken by N persons, outputs the video with the emotion information added to the display device 3, and controls the display of the display device 3 (s2). Hereinafter, an example of processing of each unit will be described.

<Points of emotion information acquisition unit 110>
The emotion information acquisition unit 110 estimates the state of dialogue and obtains emotion information (degrees of three types of emotions of “sympathy”, “disapproval”, and “none”) from the video. The most important point in the dialogue state estimation technique of the emotion information acquisition unit 110 is that the time difference between the behaviors of the giving person's behavior expression and the recipient's response expression between the two dialogues, and the dialogue two This is a model that stochastically models how the external observer who sees the two interprets the state of the dialogue based on the consistency indicating whether or not their actions are consistent. Behind this is the psychological field that has been obtained so far of how the person reacts to the other person's actions when the external observer interprets the state of dialogue between the two. It is a hypothesis that knowledge will be used consciously or unconsciously. This model makes it possible to estimate the vote rate of how the group of external observers interprets the sympathetic state at each time when a time series of actions of the two dialogues is given.

  For example, in a scene where one interlocutor smiles and the other interlocutor returns smile as a response to that smile, if the reaction smiles quickly, the reaction is spontaneous to the external observer and the two empathize It is easy to see if there is a positive relationship. On the other hand, if the smile of the reaction is delayed a little, it is easy for external observers to see that the reaction is bothersome and the two are in a negative relationship such as dissenting. Furthermore, the interpretation of empathy by external observers is also affected by whether the behavior is consistent, such as returning a smile to a smile, or whether the behavior is inconsistent, such as returning a smile to a smile. This emotion information acquisition unit probabilistically models the relationship between the action time difference between the two conversations and the action consistency.

  Another point of this emotion information acquisition unit is to model the relationship between a combination of instantaneous actions between two parties and an external observer's empathy interpretation for various action channels. An action channel is a type of dialogue person's action. For example, in Patent Document 1, only the facial expression and line of sight of a conversation person are modeled as action channels, but this emotion information acquisition unit can model other arbitrary action channels such as head gestures and the presence or absence of speech. is there. This makes it possible to estimate the sympathy interpretation more accurately, for example, when the external observer interprets the sympathy or disagreement for a scene in which the recipient's smile or tilts his head against the smile of the giver. become able to.

<Configuration of Emotion Information Acquisition Unit 110>
With reference to FIG. 4, the structural example of the emotion information acquisition part 110 of this embodiment is demonstrated. The emotion information acquisition unit 110 includes an input unit 10, an action recognition unit 20, an empathy interpretation giving unit 30, a parameter learning unit 40, a posterior probability estimation unit 50, an output unit 60, a learning video storage unit 70, and an estimation video storage unit 72. A model parameter storage unit 74. The learning video storage unit 70 and the estimation video storage unit 72 include, for example, a main storage device such as a RAM (Random Access Memory), an auxiliary storage device configured by a semiconductor memory element such as a hard disk, an optical disk, or a flash memory. Can be configured. The model parameter storage unit 74 may be configured in the same manner as the learning video storage unit 70, or may be configured by middleware such as a relational database or a key value store.

  With reference to FIG. 5, the structural example of the parameter learning part 40 of this embodiment is demonstrated. The parameter learning unit 40 includes a prior distribution learning unit 42, a timing model learning unit 44, and a static model learning unit 46.

<Learning phase>
With reference to FIG. 6, the operation example in the learning phase of the emotion information acquisition part 110 is demonstrated.

  A learning video is input to the input unit 10 (step S11). The learning video is a video that captures a situation where a plurality of persons interact, and at least the head of the dialog must be captured. The video for learning can be taken by preparing one camera for each conversation person and multiplexing the pictures taken by multiple cameras, or by using a fisheye lens, etc. It may be a video of shooting. The input learning video is stored in the learning video storage unit 70.

  The action recognition unit 20 receives the learning video stored in the learning video storage unit 70 and detects facial expressions, gaze, head gestures, presence / absence of speech, etc. as the actions of each conversation person captured in the learning video. And the time series of the action of the dialogue person generated as a result is outputted (Step S21). In this embodiment, four action channels including facial expression, line of sight, head gesture, and presence / absence of speech are recognized. An action channel is a form of action. Facial expressions are the main pathway for expressing emotions. In this embodiment, six states of no expression / smile / smile / bitter smile / thinking / other are the facial expression recognition targets. The line of sight represents at least one of, for example, who is trying to convey emotions and / or observing the actions of others. In this embodiment, the line-of-sight recognition target is a person who is looking at one of the other persons and who is / is not looking at that person. That is, the number of states is the number of interlocutors. Here, the dialogue person refers to all who participate in the dialogue including the subject who is measuring the line of sight. The method described in Patent Document 1 or Non-Patent Document 1 may be used as a method for recognizing facial expressions and lines of sight. Head gestures are often expressed as an expression of attitude to the opinions of others. In this embodiment, four states of none / whit / neck / tilt / a combination thereof are recognized as head gesture recognition targets. Any known method can be used as a method for recognizing a head gesture. For example, described in “Ejiri Yasushi, Kobayashi Tetsunori,“ Recognition of Head Gestures During Dialogue ”, IEICE Technical Report, PRMU2002-61, pp.31-36, Jul.2002. (Reference 1) This method may be used. The presence or absence of utterance is a major indicator of the conversation role of the speaker / listener. In this embodiment, two states of utterance / silence are recognized as utterance presence / absence recognition targets. As a method for recognizing the presence or absence of utterance, it may be determined that the utterance is made when the audio power in the video is detected and a predetermined threshold is exceeded. Alternatively, the presence or absence of an utterance may be detected by detecting the movement of the conversation person's mouth in the video. Each action may be recognized by a single device, or may be recognized by using a separate device for each action. For example, in the case of facial expression recognition, “Patent No. 4942197 (Reference 2)” may be used as an example of an action recognition device. The action recognition unit 20 may perform manual labeling in the same manner as the empathy interpretation giving unit 30 and output the result.

  For facial expressions and head gestures, “strength” may be estimated and output. The intensity of the facial expression can be obtained from the probability that the facial expression is the target. Further, the strength of the head gesture can be obtained from the ratio of the value of the amplitude of the acquired motion to the maximum value of the amplitude (the maximum value of the scooping angle if it is whispered).

  The empathy interpretation giving unit 30 outputs a learning sympathy interpretation time series in which a plurality of external observers label the sympathy interpretation based on the learning video stored in the learning video storage unit 70 (step S30). The learning sympathy interpretation time series is a time series in which learning videos are presented to a plurality of external observers, and the external observers manually label the sympathetic interpretations between the two conversations at each time. In this embodiment, the three states of empathy / disapproval / neither are targeted as the conversation state between the two parties. The state of dialogue between the two is deeply related to the pressure of entrainment (feeling that many others have to follow the same opinion different from their own) and build consensus building and relationships The basic element above. In addition, these states interpreted by an external observer are collectively referred to as empathy interpretation. That is, the dialogue state interpretation in this embodiment is a sympathy interpretation.

  The learning action time series output from the action recognition unit 20 and the learning empathy interpretation time series output from the empathy interpretation assigning unit 30 are input to the parameter learning unit 40. The parameter learning unit 40 learns model parameters that relate the sympathy interpretation of the external observer and the behavior of the dialog person. Model parameters include a timing model that represents the likelihood of empathy interpretation based on prior distribution of empathy interpretation among the interlocutors, the time difference between the behaviors of the interlocutors and the consistency of the behavior between the interlocutors, And a static model representing the likelihood of sympathy interpretation based on the co-occurrence of.

  The prior distribution learning unit 42 included in the parameter learning unit 40 learns the prior distribution using the learning sympathy interpretation time series (step S42). The timing model learning unit 44 included in the parameter learning unit 40 learns a timing model using the learning action time series and the learning sympathy interpretation time series (step S44). The static model learning unit 46 included in the parameter learning unit 40 learns a static model using the learning action time series and the learning sympathy interpretation time series (step S46). The obtained model parameters are stored in the model parameter storage unit 74.

<< Overview of model >>
The model of this embodiment will be described in detail. In this embodiment, it is assumed that the empathy interpretation given by the external observer is independent for each combination of two dialogues. Therefore, in the following, it is assumed that there are only two participants. When there are three or more interlocutors, learning and estimation may be performed by paying attention to only the combination of the two interrogators.

In this embodiment, the posterior probability distribution P (e _t | B) of the sympathetic interpretation e of the external observer at each time t given the time series B of the conversation person's behavior is _expressed using a naive Bayes model. Model and estimate. The naive Bayes model assumes that the stochastic dependence between the dependent variables (here, empathy interpretation) and each explanatory variable (here, the actions of each interactor) is independent among the explanatory variables. The Naive Bayes model is an excellent model that has been confirmed to show high estimation performance in many fields despite being simple. There are two advantages of using the naive Bayes model in this emotion information acquisition unit. One is that it is easy to avoid over-learning because it does not model all co-occurrence between behavioral channels (for example, a state in which all of facial expressions, gaze, head gestures, and utterances occur simultaneously). This is particularly effective when there are few learning samples for the target variable space. The other is that it is easy to add or delete action channels as observation information.

In the naive Bayes model in this embodiment, the posterior probability distribution P (e _t | B) is defined as shown in Equation (1).

Here, P (dt _t ^b | c _t ^b , e _t ) is a timing model, and when the behavior between the two parties is behavior coherence c _t ^b with time difference dt _t ^b around time t Represents the likelihood that the external observer's sympathy interpretation is e. The coincidence c is a binary state indicating whether or not the behaviors of the two parties are the same, and is determined based on whether or not the categories of the behaviors of the two parties are the same. P (b _t , e _t ) is a static model that models how the action channel b co-occurs between the two parties at the instant of time t. These two models will be described in turn below. P (e _t ) is a prior distribution of the sympathy interpretation e, and represents the probability that each sympathy interpretation e is generated when no action is considered.

<< Timing model >>
The timing model for the action channel b in this embodiment is defined as shown in Equation (2).

As is clear from the equation (2), this timing model has a time difference function P (d ~) representing the likelihood of the empathy interpretation e when the time difference between the actions of the two conversations is dt and the coincidence is c. t _t ^b | c _t ^b , e _t ) and a change timing function π _t representing the timing at which the sympathetic interpretation e changes in the vicinity of the interaction. d to t _t ^b are bin numbers when the time series of the external observer's empathy interpretation is converted into a histogram. For example, the bin size is 200 milliseconds.

  In this embodiment, for each behavior channel, a timing model between two parties is constructed within the behavior channel, but a model between behavior channels may be constructed. For example, the relationship between the time difference dt between the facial expression and the head gesture, the consistency c, and the empathy interpretation e can be modeled. However, in this case, when determining the consistency c, it is necessary to introduce a new category group that can determine the consistency c even between different behavior channels such as positive / neutral / negative, for example. is there. These categories may be recognized when the action channel is detected from the video, or once they are recognized by different category groups for each action channel, and their labels are affirmed if the facial expression is small. You may reclassify later as positive / neutral / negative.

<< Time difference function >>
The time difference function P (d ~ t _t ^b | c _t ^b , e _t ) indicates that the sympathetic interpretation e is based on the coincidence c indicating whether or not the actions between the two parties are matched in the action channel b and the time difference dt The likelihood of which type is likely to be represented. In this embodiment, bin numbers d to t _t ^b when the time series of the sympathy interpretation of the external observer are converted into a histogram are used. For example, the bin size is 200 milliseconds.

FIG. 8 shows an example of the time difference function of this embodiment. Time difference function ^{_{P (d ~ t t b |}} c t b, e t) determines the likelihood of sympathetic interpretation e by the bin number d ~ t _t ^b Consistency c and time difference of behavior of the interlocutor. FIG. 8A is an example of a time difference function when the actions between the interlocutors match, and FIG. 8B is an example of a time difference function when the actions between the interlocutors do not match. For example, if the behaviors of the interlocutors match, and the time difference between the giver's action expression and the receiver's reaction expression is 500 milliseconds, the likelihood that the empathy interpretation e is "sympathy" The likelihood of about 0.3, “Neither” is about 0.2, and the likelihood of “antisense” is about 0.5. The time difference function calculates the time series of empathy interpretations labeled by external observers in units of time difference bins, and normalizes the sum of likelihood in all time difference bins to be 1 for each category of empathy interpretation e. Ask.

<< Change timing function >>
The change timing function π represents at which timing the empathy interpretation e changes. Viewed another way, the change timing function π determines to what extent the time difference function contributes to the estimation of the empathy interpretation e in equation (1) over which range.

  In this embodiment, the change timing function is modeled as shown in Equation (3).

Here, t _a represents the time behavior expression initiation hand given in the interaction of interest. In addition, time t ′ represents the relative time during the interaction when the action expression start time of the giver is t ′ = 0 and the reaction expression start time of the receiver is t ′ = 1. Calculated as = (tt _a ) / dt.

π = 0 represents that the timing model P (dt _t ^b | c _t ^b , e _t ) does not contribute at all in the posterior probability distribution P (e _t | B) represented by the equation (1). π = 1 represents that the timing model P (dt _t ^b | c _t ^b , e _t ) contributes completely in the posterior probability distribution P (e _t | B).

  The condition dt> L represents that the response expression of the receiver is too late with respect to the action expression of the giver. For example, in this embodiment, the threshold value L is 2 seconds. This is a value based on the research results that the expression of the listener's facial expression for the vocabulary important phrase of the speaker occurs in the range of about 500 to 2,500 milliseconds, and it is generally within this range in any action channel. Based on assumptions. For details on the above research results, see “GR Jonsdottir, J. Gratch, E. Fast, and KR Thorisson,“ Fluid semantic back-channel feedback in dialogue: Challenges & progress ”, International Conference Intelligent Virtual Agents (IVA), pp. 154-160, 2007. (Reference 3).

The condition tt _a > W means that the time elapses from the expression of the facial expression of the giving hand that was most recently expressed before time t. When interaction is performed by expressing actions between the two parties, the sympathy interpretation of the external observer is affected by the timing for a certain period of time, but if the next interaction does not occur for a while after that, It is modeled that the effect disappears. The threshold value W may be any value as long as it is a positive value, and there is no problem even if the threshold value W is infinite when interaction between the two parties is continuously generated as in a two-party dialogue. However, in a situation where one person is mainly speaking in a dialogue with a large number of people and the interaction is not always frequent, such as an interaction between two listeners, a threshold is set. It is also possible that W is too long. In this embodiment, the threshold value W is empirically set to 4 seconds. This is based on the experimental result that the estimation accuracy is the highest when the threshold W is set to around 4 seconds.

  FIG. 9 shows an example of sympathy interpretation, behavioral expression of the giver, and response expression of the receiver. The filled pattern in FIG. 9 represents the difference in the category of behavior or empathy interpretation. For the values of α and β, for example, α = 0.2 and β = 0.8 are set. These values are determined so that the change timing function π of Equation (3) approximates the cumulative probability most.

  FIG. 10 shows an example of the change timing function π. The points plotted on the graph are actually the empathetic interpretation labels given by nine external observers to the dialogue data of four dialogue groups of four women (16 people in total). This represents the cumulative probability of the change at time t ′. It can be seen that this change timing function can be approximated well. However, α and β are not limited to these values, and α + β = 1, 0 ≦ α ≦ 1, and 0 ≦ β ≦ 1 are satisfied. As a simple setting, “α = 0, β = 1” may be used.

  11 and 12 are diagrams schematically showing an example of the effective range of the change timing function. A black fill indicates a state in which no action is detected, and a white fill and diagonal hatching indicate a category of action. The vertical hatching of the sympathy interpretation indicates empathy, and the horizontal hatching indicates counteraction. FIG. 11A shows an effective range in the case where actions between the interlocutors coincide. Since the behavior of the giver and the response of the recipient match, “sympathy” continues only during the threshold W. FIG. 11B shows an effective range in the case where the behaviors between the interlocutors do not match. Since the behavior of the giver and the response of the recipient are inconsistent, “disgust” continues only during the threshold W. FIG. 11C shows a situation where the response expression of the receiver is too late with respect to the action expression of the giver, that is, the change timing function is out of the effective range because dt> L. In this case, the “neither” state continues throughout. FIG. 12 shows the effective range when the two dialogues alternately express their actions. The basic concept is the same as that shown in FIGS.

<< Static model >>
The static model P (b _t | e _t ) shows the likelihood that the sympathetic interpretation e is generated when a specific action co-occurs between two parties for the action channel b at time t. Modeled.

  The modeling method has been proposed in Patent Document 1 and Non-Patent Document 1 for facial expression and line of sight, and therefore, it is sufficient to follow the description in these documents. What is necessary is just to combine the model of co-occurrence with the state of the expression for every state. Here, the gaze state between the two may be, for example, three states of mutual gaze / one-side gaze / mutual gaze.

The static model for head gestures is represented by P (g | e). Here, g represents a combination state of head gestures between two parties. When the number of states of the head gestures of interest and N _g, the number of states of combinations of head gestures between two parties becomes N _g × N _g. Arbitrary types and numbers may be targeted as categories, but if the number is too large, overlearning tends to occur when the number of learning samples is small. In that case, the categories prepared first may be further grouped by clustering. For example, Sequential Backward Selection (SBS) is one of the methods. For example, when targeting the category of head gesture, the estimation using only head gesture, that is, the posterior probability is P (e | B): = P (e) P (g '| e) The two categories selected for the best estimation accuracy are integrated into one. It is sufficient to reduce the number of categories one by one by repeating this until just before the estimation accuracy deteriorates. Here, g ′ is a combined state of the head gesture between the two after grouping. As for the presence or absence of utterance, the co-occurrence between two parties is modeled in the same way as the head gesture.

<< Model learning method >>
In this embodiment, since any model is described as a discrete state, in the learning phase, the frequency of how many times the discrete state appears in the learning sample is taken, and finally the frequency is normalized (probabilized). )do it.

  At this time, as a policy to prepare the model, if the group of conversations captured in the learning video used to learn the model parameters is the same as the group of conversations captured in the estimation video for which the conversation state is to be estimated. For example, the parameters are learned independently for each of the two conversations, and the parameters learned from the data of the two conversations may be used for estimation of the two conversations. On the other hand, if the group of interrogators captured in the video for learning differs from the group of interrogators captured in the video for estimation, one model is learned without distinguishing between the two conversations, It is only necessary to make an estimation about two parties who want to estimate using the model.

<Estimation phase>
With reference to FIG. 7, the operation example in the estimation phase of the emotion information acquisition unit 110 will be described.

  The estimation video is input to the input unit 10 (step S12). The estimation video is a video that captures a situation in which a plurality of persons interact, and at least the conversation person's head must be captured. The estimation video is an unknown video different from the learning video. The estimation video capturing method is the same as the learning video capturing method in the learning phase described above. The input estimation video is stored in the estimation video storage unit 72.

  The action recognition unit 20 receives the estimation video stored in the estimation video storage unit 72, and detects facial expressions, gaze, head gestures, presence / absence of speech, etc. as the actions of each person captured in the estimation video Then, the time series B of the behavior of the dialogue person generated as a result is output (step S22). Since the behavior recognition method is the same as the behavior recognition method in the learning phase described above, description thereof is omitted here.

The estimation action time series B output from the action recognition unit 20 is input to the posterior probability estimation unit 50. The posterior probability estimation unit 50 uses the model parameters stored in the model parameter storage unit 74 to determine the posterior probability distribution P (e _t | B) of the sympathy interpretation between the conversationers from the estimation action time series B to the time t. Is estimated (step S50). The posterior probability estimation unit 50 includes a time series B of the conversation person's behavior generated based on the estimation video, a model parameter including each parameter of the prior distribution, timing model, and static model learned by the parameter learning unit 40. As an input, the posterior probability distribution P (e _t | B) of the empathy interpretation e at time t is calculated according to the above equation (1).

The output unit 60 outputs the posterior probability distribution P (e _t | B) of the sympathy interpretation e between the interlocutors (step S60). When it is necessary to output the estimation result of empathy interpretation as one type instead of probability distribution, the type of empathy interpretation with the highest posterior probability, that is, e ~ _t = argmax _{e_t} P (e _t | B) The values e to _t may be output together.

In other words, the emotion information acquisition unit 110, the posterior probability distribution P of the sympathetic interpretation e (e _t | B), or the type of posterior probability is highest sympathy interpretation, namely _{e ~ t = argmax e_t P (} e t | B ) As emotional information. In the present embodiment, the type of posterior probability is highest sympathy interpretation _e ~ _t = argmax e_t P | seeking (e _t B) as the emotion information, and outputs. When the emotion information acquisition unit 110 obtains and outputs the posterior probability distribution P (e _t | B) of the sympathy interpretation e as emotion information, as shown in FIG. 16, each posterior probability distribution P (e _t | B) may be controlled to be displayed as a bar graph or the like. Details will be described in a modification described later.

<Control unit 120>
The control unit 120 receives the video and emotion information in which N people are photographed, and (1) the first person 2-i (i is 1, 2, 1) which is one of the N people in the emotion information. ..., N) and emotion information σ _{i, j} between others 2-j (j = 1, 2,..., N, i ≠ j), (2) emotion information σ _{i, j} Emotion information σ _{i ″, j ″} (i ″ = 1, 2,..., N, where i ″ ≠ i, j ″ = 1, 2,..., N, where i ″ ≠ j ″ and i ≠ j “), (3) All emotion information σ _{i ′, j ′} (i ′ = 1, 2,..., N, j ′ = 1, 2,..., N, where i ′ ≠ j ′) Then, the display device 3 is controlled to display (s2).

For example, _assuming that N = 4 and i = 1, the control unit 120 receives four videos and emotion information σ _{n, n ′} taken by each of four people (where n = 1, 2, 3, 4, (n ′ = 1, 2, 3, 4, n ≠ n ′, σ _{n, n ′} = σ _{n ′, n} )) Four images are synthesized so that they can be displayed on the display device 3.

Based on the switching information, the control unit 120 displays any one of the emotion information σ _{i, j} , σ _{i ″, j ″} , σ _{i ′, j ′} of (1) to (3) described above on the display device 3. Control to do. The switching information is information for specifying which emotion information is to be displayed among the emotion information σ _{i, j} , σ _{i ″, j ″} , σ _{i ′, j ′} of (1) to (3). For example, (A) information selected by a viewer watching the display device 3 or a user of the emotion information display control device 100 (hereinafter also referred to as “selected information”), (B) time information, (C) All emotion information σ _{i ′, j ′} etc. can be considered.

The selection information of (A) is set so as to correspond to the above (1) to (3). For example, “Please select emotion information to be displayed from the following: 1. Empathy information between first person and others 2.1. Information 3. All empathy information "
And the emotion information corresponding to the information input by the user operating an input unit (mouse, keyboard, etc.) (not shown) is displayed on the display device 3. At this time, a process of selecting the first person may be added. For example, “Please select the first person” is displayed on the display unit (not shown) of the emotion information display control device 100, and the first person is selected based on the information input by the user operating the input unit (not shown). Identify. The process for selecting the first person may be before or after selecting emotion information to be displayed.

When the time information of (B) is used as the switching information, the emotion information σ _{i, j} , σ _{i ", j"} , σ of (1) to (3) every elapse of a predetermined time (for example, 30 seconds). Control is performed so that _{i ′ and j ′} are sequentially displayed on the display device 3. At this time, a different time may be set for each of the emotion information σ _{i, j} , σ _{i ″, j ″} , σ _{i ′, j ′ of} (1) to (3). For example, the emotion information σ _{i, j} and σ _{i ", j"} of (1) and (2) are displayed every 10 seconds, and the emotion information σ _{i ', j'} of (3) is switched in 5 seconds. Control is also possible. Furthermore, you may switch a 1st person for every predetermined time progress. For example, first, (1) to (3) are displayed for the person 2-1, and thereafter the same (1) to (3) are sequentially displayed for the other persons. You may perform control to return to 2-1.

When all the emotion information σ _{i ′, j ′} in (C) is used as the switching information, for example, emotion information σ _{i, j} , σ _{i ″, j ″} , σ _{i ′} having a high proportion of empathy or disagreement. _{, j ′} is controlled to be displayed on the display device 3. Further, control is performed so that emotion information σ _{i, j} , σ _{i ", j"} , σ _{i ', j'} having a high ratio of greatly changed emotion information is displayed on the display device 3.

A combination of information (selection information, time information, emotion information) of (A) to (C) may be used as switching information. For example, priorities are given to selection information, time information, and emotion information, and basically the emotion information σ _{i, j} described in (1) to (3) above based on switching information having a high priority. , Σ _{i ″, j ″} and σ _{i ′, j ′} are controlled to be displayed on the display device 3. For example, when the priority is set such that selection information> emotion information> time information and selection information is received, emotion information σ _{i, j} , σ _{i ″, j ”} , σ _{i ′} is based on the selection information. _{, j ′} is controlled to be displayed on the display device 3. The percentage of sympathy if no selection information has been received or if a considerable amount of time has passed without any operation since it was received (a time that can be determined that the intention of selection is no longer felt after entering the selection information) Is controlled so as to display emotion information σ _{i, j} , σ _{i ″, j ″} , σ _{i ′, j ′} having a high value on the display device 3. When selection information has not been received or when no operation has been performed and a considerable amount of time has passed, sympathy can be found in emotion information σ _{i, j} , σ _{i ", j"} , σ _{i ', j'} When there is no emotion information with a high ratio, control is performed so that any one of the emotion information σ _{i, j} , σ _{i ", j"} , σ _{i ', j'} is displayed on the display device 3 based on the time information. To do.

When it is determined which emotion information σ _{i, j} , σ _{i ″, j ″} , σ _{i ′, j ′ is} to be displayed on the display device 3, the control unit 120 displays the emotion information σ _{i, j} , σ _{i ″ to be} displayed. _{, j "} , σ _{i ', j'} , the following processing is performed.

When controlling the emotion information σ _{i, j} of (1) to be displayed on the display device 3, the control unit 120 adds the first person 2-1 and the other persons 2-2, 2-3 to the synthesized video. Emotion information σ _1,2 , σ _1,3 , σ _1,4 between ₂ and ₄ is added and output to the display device 3.

When controlling the emotion information σ _{i ", j"} of (2) to be displayed on the display device 3, the control unit 120 adds the first person 2-1 and others 2-2, 2- to the synthesized video. emotion information sigma _{1, 2} between 3,2-4, σ _1,3, σ _1,4 other emotion information σ _2,3, σ _2,4, by adding a sigma _{3, 4,} the display device 3 is output.

When controlling the emotion information σ _{i ′, j ′} of (3) to be displayed on the display device 3, all emotion information is added to the synthesized video and output to the display device 3.

13 shows an example of an image displayed on the display device 3 in the case of (1), FIG. 14 in the case of (2), and FIG. 15 in the case of (3). In this figure, the solid line connecting the persons indicates that the emotion information σ _{n, n ′} is sympathetic, and the broken line indicates counteracting (in actual use, the color of the line connecting the persons, blinking, etc. Furthermore, any display method can be used as long as the user can perceive a difference in emotional information due to various visual differences such as the position and size of the person without using a line connecting the persons. Also good). Since emotion information indicates the degree of emotion shared by two people, emotion information σ _{n, n ′} = σ _{n ′, n} . In other words, the degree of emotion from one to the other and the degree of emotion from the other to the other are the same.

In addition, as emotion information, a value at each time point may be used, or a representative value such as a mode value of values from each time point before ΔT time may be used. Further, the emotion information and the threshold value may be compared, and control may be performed so that the information is displayed only when the emotion information is equal to or greater than the threshold value. When the emotion information acquisition unit 110 obtains the posterior probability distribution P (e _t | B), an average value of values from each time point before ΔT time may be used as emotion information.

<Effect>
With such a configuration, emotion information can be easily grasped. For example, by displaying emotion information σ _{i, j} between the first person and another person, it is easier to grasp the emotion information of the first person, and the communication ability of the first person can be improved. Effective when viewing. When the first person looks at the display device, the emotional state between himself and others can be easily grasped. If everyone else feels against the first person and all emotion information other than the emotion information between the first person and the other person is sympathetic, the first person has a different opinion You can immediately detect this and respond to it by changing the topic, etc., and you can make the place full. In addition, by displaying emotion information σ _{i ", j"} other than emotion information between the first person and others, it is possible to make it easier to grasp emotion information of persons other than the first person, This is effective when viewing the status of dialogues between persons other than the first person and their communication skills. By switching and displaying emotion information σ _{i, j} and σ _{i ", j",} it is easy to grasp emotion information between persons other than the first person while grasping emotion information of the first person Become. For example, by setting a moderator or facilitator for discussion as the first person and switching and displaying emotion information σ _{i, j} and σ _{i ", j"} , the moderator based on emotion information σ _{i, j} It is possible to make it easier to grasp the situation of the discussion between the participants using the emotion information σ _{i ", j"} while confirming whether the moderator progresses well.

In addition, by switching all emotion information σ _{i ', j'} to emotion information σ _{i, j} and emotion information σ _{i ", j"} , the first person and the first person in the whole conversation It is possible to easily grasp the communication ability and the role of a person other than a person.

<Modification>
In this embodiment, empathy is defined as a state in which the emotional states of the two are similar, and antipathy is defined as a state in which they are different. “Empathy” may be defined as what is felt as “sympathetic”, and “sympathy” may be defined as what is felt as “dissatisfied”. In other words, here we are following the definition of empathy / antisense inside each observer. This is because the ability to read the definition of empathy / antipathy among each observer and the ability to read the empathy / antipathy varies, but rather the variability itself is the essence of communication. It is in a position to define an objective definition of empathy / antisense with a collection of interpretations. In other words, the phenomenon of “sympathy / antisense” that we want to model here is the uncertainties of empathy / antisense among the interlocutors in the field of dialogue and the uncertainty of definition / interpretation of empathy / antisense by external observers. Both are folded.

  In this embodiment, an image in which N people are photographed is input, but an image in which N ′ (> N) people are photographed is input, and emotions related to a part (N of N ′ people). It is good also as a structure controlled so that only information is displayed. However, 3 ≦ N <N ′. More specifically, one of N persons is a first person, and (N-1) persons other than the first person among N persons are other persons. For example, it may be controlled so that video images of six people are input and emotion information regarding four of them is displayed. One of the four persons is set as the first person 2-i, and the remaining three persons are set as the other person 2-j. When receiving a video in which more than N people are photographed, the control unit 120 receives the video and emotion information, selects N people from among more than N people, and further from among the N people. Select the first person. Note that the selection method of the N person and the first person may be specified by the user or a person specified in advance. Other processes are as described in the control unit 120. The selection of the N person and the first person may be before or after the emotion information acquisition unit 110. The emotion information acquisition unit 110 may acquire emotion information between two parties for all the interlocutors (more than N people) included in the captured video, or some people who display emotion information Emotion information between two parties may be acquired for (N people).

  The processing content of the emotion information acquisition unit 110 is not limited to the above-described method, and emotion information (between two persons constituting each pair of each pair composed of two persons among a plurality of persons) Any other method (for example, Patent Literature 1 or Non-Patent Literature 1) may be used as long as the degree of emotion) is obtained. The emotion is not limited to “sympathy”, “antisense”, and “neither”, but may be other emotions.

  In the emotion information acquisition unit 110, the emotion information is obtained for all the combinations between the interlocutors, but the emotion information may not necessarily be obtained for all the combinations. For example, the control unit may perform control so that only some emotion information is not displayed. In that case, emotion information that needs to be displayed may be obtained. With such a configuration, it is possible to reduce the amount of calculation related to the acquisition of emotion information.

The control unit 120 controls the three emotion information σ _{i, j} , σ _{i ″, j ″} , σ _{i ′, j ′} to be switched and displayed on the display device 3, but the three emotion information σ Control may be performed so that at least two of _{i, j} , σ _{i ″, j ″} and σ _{i ′, j ′} are switched and displayed on the display device 3.

  In the case where a video shot of N persons is composed of a single video data, the face portion of each person may be cut out from the single video data and combined so that the display device 3 can display it.

When the emotion information acquisition unit 110 obtains and outputs the posterior probability distribution P (e _t | B) of the sympathy interpretation e as emotion information, the control unit 120 selects the type of sympathy interpretation with the highest posterior probability e ~. A configuration may be adopted in which _t = argmaxP (e _t | B) is displayed by a solid line, a broken line, etc., and each posterior probability distribution P (e _t | B) is displayed by a bar graph or the like (see FIG. 16). . In other words, the control unit 120 may be configured to control so as to display all types of emotion levels obtained by the emotion information acquisition unit 110 as emotion information. In the figure, the diagonal line descending to the right represents “sympathy”, the vertical line represents “antisense”, and the “horizontal line” represents posterior probability of “neither”. You may control so that only a bar graph etc. are displayed.

  You may control so that the 1st person 2-i is not displayed on the display apparatus 3, respectively. 17 to 19 show examples of images displayed on the display device 3 when the first person 2-i is controlled not to be displayed on the display device 3. FIG. 17 to 19 correspond to FIGS. 13 to 15, respectively. In particular, when the first person himself / herself looks at the display device, the emotional state between himself / herself and another person can be more intuitively and easily grasped. Furthermore, by using a head-mounted display as a display device, a more intuitive grasp is possible.

<Second embodiment>
A description will be given centering on differences from the first embodiment.

  FIG. 20 shows an arrangement example of the emotion information display control apparatus 100 according to the first embodiment. Emotion information display control device 100 receives an image of N or more people as an input, and outputs an image with emotion information added to M display devices 3-m (m = 1, 2,..., M). . N is an integer of 3 or more, and M is an integer of 2 or more. FIG. 21 shows a processing flow of the emotion information display control device 100.

  In the second embodiment, the processing content of the control unit 120 is different from that of the first embodiment.

<Control unit 120>
The control unit 120 receives the video and emotion information in which N people are photographed, and among the emotion information, between the first person 2-i who is one of the N people and the other person 2-j. The emotion information σ _{i, j} is controlled to be displayed on the display device 3-p which is one of the M display devices 3-m (s2-1).

Further, the control unit 120, out of the emotion information, the emotion information sigma _i, emotion information other than _{j σ i ", j" (} i "= 1,2, ..., N, however i" ≠ i, j "= 1 , 2,..., N, where i ″ ≠ j ″ and i ≠ j ″) is one of the M display devices 3-m (m = 1, 2,..., M). 3-p ′ (p ′ is any one of 1, 2,..., M, where p ≠ p ′) is controlled (s2-2).

For example, N = 4, M = 2, and i = 1, and the control unit 120 receives four videos and emotion information σ _{n, n ′} taken by each of four persons (where n = 1, 2, (3, 4, n ′ = 1, 2, 3, 4, n ≠ n ′, σ _{n, n ′} = σ _{n ′, n} )) four images are displayed on one display device 3-1 or 3- 2 so that it can be displayed. Emotion information σ _1,2 , σ _1,3 , σ _1,4 between the first person 2-1 and others 2-2, 2-3, 2-4 is added to the synthesized video, The data is output to the display device 3-1. Further, the control unit 120 adds emotion information σ _2,3 , σ _2,4 , σ _3,4 other than the emotion information σ _1,2 , σ _1,3 , σ _1,4 to the synthesized video, The data is output to the display device 3-2. In this case, examples of images displayed on the display devices 3-p and 3-p ′ are the same as those in FIGS. 13 and 14, respectively. Examples of images displayed on the display devices 3-p and 3-p ′ when the first person 2-i is controlled not to be displayed on the display devices 3-p and 3-p ′ are shown in FIG. And it becomes the same as FIG.

<Effect>
With such a configuration, emotion information can be easily grasped. For example, by displaying emotion information σ _{i, j} between the first person and another person, it is easier to grasp the emotion information of the first person, and the communication ability of the first person can be improved. Effective when viewing. When the first person looks at the display device, the emotional state between himself and others can be easily grasped. If everyone else feels against the first person and all emotion information other than the emotion information between the first person and the other person is sympathetic, the first person has a different opinion You can immediately detect this and respond to it by changing the topic, etc., and you can make the place full. In addition, by displaying emotion information σ _{i ", j"} other than emotion information between the first person and others, it is possible to make it easier to grasp emotion information of persons other than the first person, This is effective when viewing the status of dialogues between persons other than the first person and their communication skills. By simultaneously displaying emotion information σ _{i, j} and σ _{i ", j"} on the display devices 3-p and 3-p 'respectively, It becomes easy to grasp emotion information between persons other than the person. For example, a discussion moderator or facilitator is set as the first person, and emotion information σ _{i, j} and σ _{i ", j"} are simultaneously displayed on the display devices 3-p and 3-p ', respectively. that is, while confirming or moderated such as moderator by the emotion information σ _{i, j} is successful, emotion information σ _{i ", j"} it is possible to make it easy to grasp the situation of the discussions among the participants by.

<Third embodiment>
A description will be given centering on differences from the second embodiment.

  In the third embodiment, M is an integer of 3 or more. FIG. 21 shows a processing flow of the emotion information display control device 100. The processing content of the control unit 120 is different from that of the second embodiment.

<Control unit 120>
The control unit 120 receives the video and emotion information in which N people are photographed, and among the emotion information, between the first person 2-i who is one of the N people and the other person 2-j. The emotion information σ _{i, j} is controlled to be displayed on the display device 3-p which is one of the M display devices 3-m (s2-1).

Further, the control unit 120, out of the emotion information, the emotion information sigma _i, emotion information other than _{j σ i ", j" (} i "= 1,2, ..., N, however i" ≠ i, j "= 1 , 2,..., N, where i ″ ≠ j ″ and i ≠ j ″) is one of the M display devices 3-m (m = 1, 2,..., M). 3-p ′ (p ′ is any one of 1, 2,..., M, where p ≠ p ′) is controlled (s2-2).

Further, the control unit 120 converts all the emotion information σ _{i ′, j ′} (i ′ = 1, 2,..., N, j ′ = 1, 2,..., N, where i ′ ≠ j ′), Display device 3-p ″ (p ″ is one of 1, 2,..., M, which is one of the M display devices 3-m other than display devices 3-p and p ′, where p ≠ p ″, p ′ ≠ p ″). (s2-3).

For example, N = 4, M = 3, and i = 1, and the control unit 120 receives four videos and emotion information σ _{n, n ′} taken by each of four persons (where n = 1, 2, (3, 4, n ′ = 1, 2, 3, 4, n ≠ n ′, σ _{n, n ′} = σ _{n ′, n} )), four images are displayed on one display device 3-1, 3- It is synthesized so that it can be displayed in 2 or 3-3. Emotion information σ _1,2 , σ _1,3 , σ _1,4 between the first person 2-1 and others 2-2, 2-3, 2-4 is added to the synthesized video, The data is output to the display device 3-1. Further, the control unit 120 adds emotion information σ _2,3 , σ _2,4 , σ _3,4 other than the emotion information σ _1,2 , σ _1,3 , σ _1,4 to the synthesized video, The data is output to the display device 3-2. Furthermore, the control unit 120 adds all emotion information to the synthesized video and outputs the added emotion information to the display device 3-3.

  In this case, examples of images displayed on the display devices 3-p, 3-p ′, and 3-p ″ are the same as those in FIGS. 13, 14, and 15, respectively. Examples of images displayed on the display devices 3-p, 3-p ′, and 3-p ″ when they are controlled not to be displayed on the display devices 3-p, 3-p ′, and 3-p ″ are respectively This is the same as FIGS. 17, 18 and 19.

<Effect>
With such a configuration, the same effect as that of the second embodiment can be obtained. Furthermore, by displaying all the emotion information σ _{i ′, j ′} on the display device 3-p ″, the communication ability of the first person or a person other than the first person in the entire conversation, its role, etc. Can be easily grasped.

<Other variations>
The present invention is not limited to the above-described embodiments and modifications. For example, the various processes described above are not only executed in time series according to the description, but may also be executed in parallel or individually as required by the processing capability of the apparatus that executes the processes. In addition, it can change suitably in the range which does not deviate from the meaning of this invention.

<Program and recording medium>
In addition, various processing functions in each device described in the above embodiments and modifications may be realized by a computer. In that case, the processing contents of the functions that each device should have are described by a program. Then, by executing this program on a computer, various processing functions in each of the above devices are realized on the computer.

  The program describing the processing contents can be recorded on a computer-readable recording medium. As the computer-readable recording medium, for example, any recording medium such as a magnetic recording device, an optical disk, a magneto-optical recording medium, and a semiconductor memory may be used.

  The program is distributed by selling, transferring, or lending a portable recording medium such as a DVD or CD-ROM in which the program is recorded. Further, the program may be distributed by storing the program in a storage device of the server computer and transferring the program from the server computer to another computer via a network.

  A computer that executes such a program first stores, for example, a program recorded on a portable recording medium or a program transferred from a server computer in its storage unit. When executing the process, this computer reads the program stored in its own storage unit and executes the process according to the read program. As another embodiment of this program, a computer may read a program directly from a portable recording medium and execute processing according to the program. Further, each time a program is transferred from the server computer to the computer, processing according to the received program may be executed sequentially. Also, the program is not transferred from the server computer to the computer, and the above-described processing is executed by a so-called ASP (Application Service Provider) type service that realizes the processing function only by the execution instruction and result acquisition. It is good. Note that the program includes information provided for processing by the electronic computer and equivalent to the program (data that is not a direct command to the computer but has a property that defines the processing of the computer).

  In addition, although each device is configured by executing a predetermined program on a computer, at least a part of these processing contents may be realized by hardware.

Claims (7)

With respect to each pair composed of two persons among a plurality of persons, the degree of emotion between the two persons constituting each pair is set as emotion information, and from the video in which three or more persons are photographed, An emotion information acquisition unit for obtaining emotion information between two of three or more people,
(1) Among the emotion information, emotion information between a first person who is one of the three or more and others, (2) Among the emotion information, the first person and the A control unit that controls at least two or more of the emotion information other than the emotion information (3) other than the emotion information to be displayed on a display device.
Emotion information display control device. With respect to each pair composed of two persons among a plurality of persons, the degree of emotion between the two persons constituting each pair is set as emotion information, and from the video in which three or more persons are photographed, An emotion information acquisition unit for obtaining emotion information between two of three or more people,
Of the emotion information, the emotion information between the first person who is one of the three or more and the other person is displayed on the first display device which is one of the plurality of display devices. The emotion information other than the emotion information between the first person and the other person is one of the plurality of display devices other than the first display device. A control unit that controls to display on the second display device,
Emotion information display control device. The emotion information display control device according to claim 1 or 2,
3 ≦ N <N ′, a video image of N ′ people is input, one of N people who are part of N ′ is the first person, and the first of the N persons is the first person. (N-1) people other than the other person are assumed to be others.
Emotion information display control device. The emotion information display control device according to any one of claims 1 to 3,
The emotion information includes a plurality of types of emotions, and the emotion information acquisition unit obtains all types of emotions as the emotion information,
The control unit controls to display the degree of all kinds of emotions obtained by the emotion information acquisition unit as the emotion information;
Emotion information display control device. With respect to each pair composed of two persons among a plurality of persons, the degree of emotion between the two persons constituting each pair is set as emotion information, and from the video in which three or more persons are photographed, An emotion information acquisition step for obtaining emotion information between two of three or more people;
(1) Among the emotion information, emotion information between a first person who is one of the three or more and others, (2) Among the emotion information, the first person and the A control step of controlling at least two or more of all the emotion information other than the emotion information (3) other than the emotion information to be displayed on a display device.
Emotion information display control method. With respect to each pair composed of two persons among a plurality of persons, the degree of emotion between the two persons constituting each pair is set as emotion information, and from the video in which three or more persons are photographed, An emotion information acquisition step for obtaining emotion information between two of three or more people;
Of the emotion information, the emotion information between the first person who is one of the three or more and the other person is displayed on the first display device which is one of the plurality of display devices. The emotion information other than the emotion information between the first person and the other person is one of the plurality of display devices other than the first display device. A control step for controlling to display on the second display device,
Emotion information display control method.   A program for causing a computer to function as the emotion information display control device according to claim 1.