JP5909472B2 - Empathy interpretation estimation apparatus, method, and program - Google Patents

Description

  The present invention relates to a technique for estimating empathy interpretation.

  Non-Patent Document 1 describes a technique for measuring how well a facial expression movement displayed on a screen can be managed well.

Nintendo, Fumiko Inudo, [online], [July 31, 2013], Internet <URL: http://www.nintendo.co.jp/ds/ykoj/etc/index.html>

  However, the background technology is a technology that measures how well you can manage, and the technology that estimates the empathy between the first person displayed on the screen and the second person who sees the first person through the screen. There wasn't. Sympathetic interpretation is a state of dialogue between two parties, such as empathy and dissensibility.

  An object of the present invention is to provide a sympathy interpretation estimation apparatus, method, program, and recording medium for estimating a sympathy interpretation between a first person displayed on a screen and a second person watching the first person through the screen. And

A video generation device according to an aspect of the present invention includes a person video presentation unit that presents a first video of a first person whose behavior changes to a second person, and a second person that performs the behavior change of the first person and action presentation unit that presents the actions on sympathetic antipathy to the second person, and the image acquisition unit that acquires a second image obtained by photographing the head portion of the second person first image is presented, the first in the second image An action recognition unit that generates a second action time series of the second person by detecting the action of the second person with respect to a change in the action of the person, and the first action time series and the second picture of the first person in the first picture Based on the second action time series of the second person in the model, using the model parameters including the timing model representing the likelihood of empathy interpretation based on the time difference of action and the coincidence of actions, the first person and the second person Estimating the empathy interpretation between Comprising a probability estimator, the estimated sympathy interpreted as a result to changes in behavior of the first person, and a display unit to be presented to the second person, the.

  It is possible to estimate the sympathy interpretation between the first person displayed on the screen and the second person watching the first person through the screen. Moreover, by using model parameters including timing models that represent the likelihood of empathic interpretation based on behavioral time differences and behavioral consistency, the timing of expression can be taken into account, and the empathic interpretation can be estimated with high accuracy. be able to.

The block diagram which shows the example of a sympathy interpretation estimation apparatus. The figure for demonstrating the example of a 1st image | video, a 1st person, and a 2nd person. The figure for demonstrating the change of a 1st person's action. The figure for demonstrating the change of a 1st person's action. The figure for demonstrating the example of the action by an action presentation part. The figure for demonstrating a modification. The figure for demonstrating a modification. The figure which shows the flow of a process of an empathy interpretation estimation method. The block diagram which shows the example of a dialog state estimation apparatus. The block diagram which shows the example of a parameter learning part. The figure which shows the flow of a process of a learning phase. The figure for demonstrating a time difference function. The figure for demonstrating the time difference of a dialogue person's action and empathy interpretation. The figure for demonstrating a change timing function. The figure for demonstrating the effective range of a change timing function. The figure for demonstrating the effective range of a change timing function.

  Hereinafter, embodiments of the present invention will be described in detail.

[Empathetic interpretation estimation apparatus and method]
As shown in FIG. 1, the empathy interpretation estimation apparatus includes a person video presentation unit 82, a video acquisition unit 81, an estimation video storage unit 72, a behavior recognition unit 20, a model parameter storage unit 74, a posterior probability estimation unit 50, and a display unit. 83 and the action presentation part 84 are provided, for example.

  The sympathy interpretation estimation method includes, for example, steps S1 to S4 shown in FIG.

  The person video presentation unit 82 presents the first video of the first person 100 whose behavior changes to the second person 200 (step S1). For example, the person video presentation unit 82 presents the first video to the second person 200 through the display unit 83. The display unit 83 is a display device such as a display. The first video is provided to the action recognition unit 20 as a part of the estimation video. The first video is a video showing at least the head of the first person 100. The first person 100 may not be a person as long as he can express an expression. For example, the first person 100 may be a character or an object. Here, it is assumed that the head is a part including a facial region to such an extent that facial expressions can be recognized.

  An example of the first video is shown on the left of FIG. The action of the first person 100 in the first video changes. For example, in the first video, the first person 100 initially has no expression as shown on the left in FIG. 2, but at a certain timing, the first person 100 smiles as shown in FIG. Such a first video is presented to the second person 200 shown on the right side of FIG.

  The video acquisition unit 81 acquires a second video obtained by photographing the behavior of the second person presented with the first video (step S2). The acquired second video is provided to the action recognition unit 20 as a part of the video for estimation. The video acquisition unit 81 is a device that can capture video such as a video camera. It is assumed that at least the head of the second person 200 is shown in the second video.

  In FIG. 2, the action “please sympathize” that the second person 200 should perform is presented in the first video. For this reason, the second person 200 performs a behavior such as smiling in order to sympathize with the first person 100 smiling. The video acquisition unit 81 acquires a second video obtained by photographing the behavior of the second person 200. The action to be performed by the second person 200 is presented by the action presentation unit 84.

  The behavior recognition unit 20 detects the behavior of the first person 100 in the first video to detect the first behavior time series of the first person 100, and detects the behavior of the second person 200 in the second video by detecting the behavior of the second person 200. A second action time series of the person is generated (step S3). The generated first action time series and second action time series are provided to the posterior probability estimation unit 50. As the behavior, for example, an expression, a line of sight, a head gesture, presence / absence of speech, and the like are detected. Since the behavior recognition method in the behavior recognition unit 20 is the same as the behavior recognition method in [learning phase] described later, the description thereof is omitted here.

  The posterior probability estimation unit 50 uses a model parameter including a timing model representing the likelihood of empathy interpretation based on the time difference between actions and the coincidence of actions based on the first action time series and the second action time series, A sympathetic interpretation between the first person 100 and the second person 200 is estimated (step S4).

  The posterior probability estimation unit 50 uses the model parameter read from the model parameter storage unit 74. It is assumed that the model parameter storage unit 74 stores model parameters learned and generated in a learning phase described later before the posterior probability estimation unit 50 performs the process of step S4.

Specifically, the posterior probability estimation unit 50 uses the model parameters stored in the model parameter storage unit 74, from the first action time series and the second action time series that are the time series B of the conversation person's action. Estimate the posterior probability distribution P (e _t | B) of the sympathy interpretation between the talkers at time t. More specifically, the posterior probability estimation unit 50 includes parameters of a first action time series and a second action time series, which are time series B of a conversation person's action, a prior distribution, a timing model, and a static model. Using the model parameter as an input, a posteriori probability distribution P (e _t | B) of empathy interpretation e at time t is calculated according to equation (1) described later. The model parameters and equation (1) will be described in [Learning Phase] described later.

When it is necessary to output an estimation result of sympathy interpreted as a single kind rather than probability distributions, the posterior probability is highest sympathy interpretation types, namely _e ~ _t = argmax e_t P | a (e _t B) It may be output.

  Further, in addition to the type of sympathy interpretation having the highest posterior probability, the strength of the sympathy interpretation having the highest posterior probability may be output. For example, in the case of empathy, the strength can be obtained from the equation strength = sympathy probability−anti-probability, and in the case of anti-sensation, the strength = probability of probability−sympathy probability. In this case, the intensity is represented by a numerical value between -1 and 1.

  The sympathy interpretation is a state of dialogue between two parties (in this example, a state of dialogue between the first person 100 and the second person 200) such as sympathy or disagreement.

  According to one embodiment of the sympathy interpretation estimating apparatus described above, it is possible to estimate the sympathy interpretation between the first person displayed on the screen and the second person watching the first person through the screen. Moreover, by using model parameters including timing models that represent the likelihood of empathic interpretation based on behavioral time differences and behavioral consistency, the timing of expression can be taken into account, and the empathic interpretation can be estimated with high accuracy. be able to.

[Modified example of empathy interpretation estimation apparatus and method]
The first video generated by the person video presentation unit 82 itself may be provided to the second person 200, or the first video generated in advance by a person other than the person video presentation unit 82 may be provided to the second person 200. . For example, when the first video is generated in advance and stored in the estimation video storage unit 72, the human video presentation unit 82 provides the second person 200 with the first video read from the estimation video storage unit 72. To do.

  The first action time series may be stored in advance in the storage unit 85 in association with the action of the first person in the first video. For example, when a smiling character video is presented as the first video, the action recognition unit 20 generates a first action time series of the character based on the first video, so that the first action time of the character is displayed. This is because the series can be prepared in advance. In this case, the action recognition unit 20 generates only the second action time series. In this case, the posterior probability estimation unit 50 estimates the empathy interpretation based on the first action time series read from the storage unit 85 and the second action time series generated by the action recognition unit 20.

  The empathy interpretation estimated by the posterior probability estimation unit 50 may be displayed on the display unit 83. When information on the action to be performed by the second person 200 is provided from the action presentation unit 84 to the posterior probability estimation unit 50, the posterior probability estimation unit 50 determines the action to be performed by the provided second person 200. Only the related sympathy interpretation may be displayed on the display unit 83. For example, when the action to be performed by the second person 200 is an action “please sympathize”, the posterior probability estimation unit 50 causes the display unit 83 to display only the estimation result of the sympathy interpretation regarding sympathy.

  In the above embodiment, the first person 100 has no expression as shown in the left of FIG. 2 in the first video, but the first person 100 smiles as shown in FIG. 3 at a certain timing. In the first video, the first person 100 may take other actions. For example, the first person 100 may take an action showing a sad expression as shown in FIG.

  In FIG. 4, the action “please sympathize” that the second person 200 should perform is presented in the first video. In general, an action for showing empathy for an action showing a sad expression is an action showing a sad expression generally quickly. Therefore, in the case of FIG. 4, the second person 200 can show empathy by taking an action showing a sad expression at an appropriate timing, like the first person 100.

  The action to be performed by the second person 200 presented to the second person 200 by the action presentation unit 84 is not limited to the action “please sympathize”. In other words, the action presenting unit 84 may present any action to the second person 200 as long as the action should be performed by the second person 200. For example, the action presenting unit 84 may present the action “please feel” to the second person 200 as shown in FIG. Note that FIG. 5 shows an action in which the first person 100 smiles. In general, an action for showing a sense of opposition to the action of becoming a smile is an action that generally shows an angry or sloppy expression generally quickly. Alternatively, even if a smile is delayed, the action that is shown can also be countered. Therefore, in the case of FIG. 5, the second person 200 can show a sense of disapproval by taking an action that quickly shows an angry or sloppy expression or an action that shows a smile late.

  The place where the action presenting unit 84 presents the action to be performed by the second person 200 to the second person 200 is not limited to the first video. In other words, the action presenting unit 84 may use any method as long as the action to be performed by the second person 200 can be presented to the second person 200. For example, the action presenting unit 84 displays the action to be performed by the second person 200 on the display unit 83 before the first video is presented to the second person 200, so that the action to be performed by the second person 200 is displayed. You may show to the 2nd person 200.

The action presentation unit 84 may not be provided in the sympathy interpretation estimation device. in this case,
The second person 200 may or may not know the action to be performed by the second person 200 in advance.

  The person video presentation unit 82 may present each video of a plurality of persons including the first person to the second person 200. For example, the video images of the four persons including the first person shown in FIG. 6 are presented to the second person 200 via the display unit 83 by the person video presentation unit 82. In this case, the behavior of any one of the four persons changes. In FIG. 6, the action presentation unit 84 shows an action “please sympathize” in each of the images of the four persons. For this reason, the second person 200 performs an action to show empathy for any one person whose action among the four persons changes. Any one person whose behavior changes is the first person 100.

  In the example of FIG. 6, the action to be performed by the second person 200 for each of a plurality of persons presented by the action presentation unit 84 is a common action “please sympathize”, but the action presentation The actions to be performed by the second person 200 for each of the plurality of persons presented by the unit 84 may be different.

  For example, in the example of FIG. 7, the action to be performed by the second person 200 for the two persons positioned on the upper side of the page of FIG. 7 among the four persons is an action “please sympathize”. However, among the four persons, the action to be performed by the second person 200 for the two persons positioned on the lower side with respect to the page of FIG.

  In this case, if any one or more of the two persons located in the upper row changes, the second person 200 is required to have an action that shows empathy, but the two persons located in the lower row Among them, when any one or more of the behaviors change, the second person is required to exhibit a feeling of discomfort.

  As long as the second person 200 can indicate an action to be performed, all the actions of the four persons may change.

  For example, when the action to be performed by the second person 200 for all of the four persons is “please sympathize”, when all of the four persons change to a smiling action, the second person 200 Actions that show empathy, such as quickly smiling, are required.

  Further, as illustrated in FIG. 7, the action to be performed by the second person 200 for the two persons positioned on the upper side with respect to the paper surface of FIG. 7 is an action “please sympathize”. Among the persons in FIG. 7, the actions to be performed by the second person 200 for the two persons positioned in the lower stage with respect to the page of FIG. When the person changes to a smiling action and the two persons located in the lower stage change to a dull action, the second person 200 is required to have a sympathetic action such as quickly smiling. .

  In this way, the person video presentation unit 82 may present each video of a plurality of persons including at least one person whose behavior changes to the second person 200. In this case, the empathy interpretation between each first person and the second person is estimated by the processing of the action recognition unit 20 and the posterior probability estimation unit 50 each of which is at least one person whose behavior changes. Is done. For example, when the action to be performed by the second person 200 for all four persons is “please sympathize”, when all of the four persons change to a smiling action, the actions of the four persons The empathy interpretation between each of the four persons and the second person is estimated by the processes of the action recognition unit 20 and the posterior probability estimation unit 50 each of which is the first person.

  The sympathy interpretation estimation apparatus may further include an input unit 10, a learning video storage unit 70, a sympathy interpretation imparting unit 30, and a parameter learning unit 40, which will be described later in “Learning Phase”. In this case, it may have a function of learning and generating model parameters.

[Program, recording medium]
The present invention is not limited to the above-described embodiment, and it goes without saying that modifications can be made as appropriate without departing from the spirit of the present invention. The various processes described in the above-described embodiments are not only executed in time series according to the order described, but may be executed in parallel or individually as required by the processing capability of the apparatus that executes the processes.

  When various processing functions in each device described in the above embodiment are realized by a computer, the processing contents of the functions that each device should have are described by a program. Then, by executing this program on a computer, various processing functions in each of the above devices are realized on the computer.

  The program describing the processing contents can be recorded on a computer-readable recording medium. As the computer-readable recording medium, for example, any recording medium such as a magnetic recording device, an optical disk, a magneto-optical recording medium, and a semiconductor memory may be used.

  The program is distributed by selling, transferring, or lending a portable recording medium such as a DVD or CD-ROM in which the program is recorded. Furthermore, the program may be distributed by storing the program in a storage device of the server computer and transferring the program from the server computer to another computer via a network.

  A computer that executes such a program first stores, for example, a program recorded on a portable recording medium or a program transferred from a server computer in its own storage device. When executing the process, the computer reads a program stored in its own recording medium and executes a process according to the read program. As another execution form of the program, the computer may directly read the program from a portable recording medium and execute processing according to the program, and the program is transferred from the server computer to the computer. Each time, the processing according to the received program may be executed sequentially. Also, the program is not transferred from the server computer to the computer, and the above-described processing is executed by a so-called ASP (Application Service Provider) type service that realizes the processing function only by the execution instruction and result acquisition. It is good. Note that the program in this embodiment includes information that is used for processing by an electronic computer and that conforms to the program (data that is not a direct command to the computer but has a property that defines the processing of the computer).

  In this embodiment, the present apparatus is configured by executing a predetermined program on a computer. However, at least a part of these processing contents may be realized by hardware.

[Learning phase]
Hereinafter, a learning phase for learning and generating model parameters will be described. The learning of the model parameter is performed by, for example, the dialogue state estimation device 1 described in FIG.

<Configuration>
With reference to FIG. 9, the structural example of the dialog state estimation apparatus 1 of this embodiment is demonstrated. The dialog state estimation device 1 includes an input unit 10, an action recognition unit 20, a sympathy interpretation imparting unit 30, a parameter learning unit 40, a learning video storage unit 70, and a model parameter storage unit 74. The learning video storage unit 70 can be configured by, for example, a main storage device such as a RAM (Random Access Memory), an auxiliary storage device including a semiconductor memory element such as a hard disk, an optical disk, or a flash memory. The model parameter storage unit 74 may be configured in the same manner as the learning video storage unit 70, or may be configured by middleware such as a relational database or a key value store.

  With reference to FIG. 10, the structural example of the parameter learning part 40 of this embodiment is demonstrated. The parameter learning unit 40 includes a prior distribution learning unit 42, a timing model learning unit 44, and a static model learning unit 46.

<Learning phase>
With reference to FIG. 11, the operation example in the learning phase of the dialog state estimation apparatus 1 will be described.

  A learning video is input to the input unit 10 (step S11). The learning video is a video that captures a situation where a plurality of persons interact, and at least the head of the dialog must be captured. The video for learning can be taken by preparing one camera for each conversation person and multiplexing the pictures taken by multiple cameras, or by using a fisheye lens, etc. It may be a video of shooting. The input learning video is stored in the learning video storage unit 70.

  The action recognition unit 20 receives the learning video stored in the learning video storage unit 70 and detects facial expressions, gaze, head gestures, presence / absence of speech, etc. as the actions of each conversation person captured in the learning video. And the time series of the action of the dialogue person generated as a result is outputted (Step S21). In this embodiment, four action channels including facial expression, line of sight, head gesture, and presence / absence of speech are recognized. An action channel is a form of action. Facial expressions are the main pathway for expressing emotions. In this embodiment, six states of no expression / smile / smile / bitter smile / thinking / other are the facial expression recognition targets. The line of sight represents at least one of, for example, who is trying to convey emotions and / or observing the actions of others. In this embodiment, the line-of-sight recognition target is a person who is looking at one of the other persons and who is / is not looking at that person. That is, the number of states is the number of interlocutors. Here, the dialogue person refers to all who participate in the dialogue including the subject who is measuring the line of sight. The method of recognizing facial expression and line of sight is “Japanese Laid-Open Patent Publication No. 2012-185727 (Reference 1)” or “Shiro Kumano, Kazuhiro Otsuka, Amami Mikami, Junji Yamato, Estimated model of empathy / antisense and its evaluation ", IEICE Technical Report, Human Communication Fundamentals, HCS 111 (214), pp. 33-38, 2011. (Reference 2)" That's fine.

  Head gestures are often expressed as an expression of attitude to the opinions of others. In this embodiment, four states of none / whit / neck / tilt / a combination thereof are recognized as head gesture recognition targets. Any known method can be used as a method for recognizing a head gesture. For example, “Yasushi Ejiri, Tetsunori Kobayashi,“ Recognition of Head Gesture during Dialogue ”, IEICE Technical Report, PRMU2002-61, pp.31-36, Jul.2002. (Reference 3) This method may be used. The presence or absence of utterance is a major indicator of the conversation role of the speaker / listener. In this embodiment, two states of utterance / silence are recognized as utterance presence / absence recognition targets. As a method for recognizing the presence or absence of utterance, it may be determined that the utterance is made when the audio power in the video is detected and a predetermined threshold is exceeded. Alternatively, the presence or absence of an utterance may be detected by detecting the movement of the conversation person's mouth in the video. Each action may be recognized by a single device, or may be recognized by using a separate device for each action. For example, in the case of facial expression recognition, “Patent No. 4942197 (Reference 4)” may be used as an example of an action recognition device. The action recognition unit 20 may perform manual labeling in the same manner as the empathy interpretation giving unit 30 and output the result.

  For facial expressions and head gestures, “strength” may be estimated and output. The intensity of the facial expression can be obtained from the probability that the facial expression is the target. Further, the strength of the head gesture can be obtained from the ratio of the value of the amplitude of the acquired motion to the maximum value of the amplitude (the maximum value of the scooping angle if it is whispered).

  The empathy interpretation giving unit 30 outputs a learning sympathy interpretation time series in which a plurality of external observers label the sympathy interpretation based on the learning video stored in the learning video storage unit 70 (step S30). The learning sympathy interpretation time series is a time series in which learning videos are presented to a plurality of external observers, and the external observers manually label the sympathetic interpretations between the two conversations at each time. In this embodiment, the three states of empathy / disapproval / neither are targeted as the conversation state between the two parties. The state of dialogue between the two is deeply related to the pressure of entrainment (feeling that many others have to follow the same opinion different from their own) and build consensus building and relationships The basic element above. In addition, these states interpreted by an external observer are collectively referred to as empathy interpretation. That is, the dialogue state interpretation in this embodiment is a sympathy interpretation.

  The learning action time series output from the action recognition unit 20 and the learning empathy interpretation time series output from the empathy interpretation assigning unit 30 are input to the parameter learning unit 40. The parameter learning unit 40 learns model parameters that relate the sympathy interpretation of the external observer and the behavior of the dialog person. Model parameters include a timing model that represents the likelihood of empathy interpretation based on prior distribution of empathy interpretation among the interlocutors, the time difference between the behaviors of the interlocutors and the consistency of the behavior between the interlocutors, And a static model representing the likelihood of sympathy interpretation based on the co-occurrence of.

  The prior distribution learning unit 42 included in the parameter learning unit 40 learns the prior distribution using the learning sympathy interpretation time series (step S42). The timing model learning unit 44 included in the parameter learning unit 40 learns a timing model using the learning action time series and the learning sympathy interpretation time series (step S44). The static model learning unit 46 included in the parameter learning unit 40 learns a static model using the learning action time series and the learning sympathy interpretation time series (step S46). The obtained model parameters are stored in the model parameter storage unit 74.

<< Overview of model >>
The model of this embodiment will be described in detail. In this embodiment, it is assumed that the empathy interpretation given by the external observer is independent for each combination of two dialogues. Therefore, in the following, it is assumed that there are only two participants. Note that when there are three or more interlocutors, it is only necessary to learn by focusing only on the combination of the two interrogators.

In this embodiment, the posterior probability distribution P (e _t | B) of the sympathetic interpretation e of the external observer at each time t given the time series B of the conversation person's behavior is _expressed using a naive Bayes model. Model and estimate. The naive Bayes model assumes that the stochastic dependence between the dependent variables (here, empathy interpretation) and each explanatory variable (here, the actions of each interactor) is independent among the explanatory variables. The Naive Bayes model is an excellent model that has been confirmed to show high estimation performance in many fields despite being simple. There are two advantages of using the naive Bayes model in this invention. One is that it is easy to avoid over-learning because it does not model all co-occurrence between behavioral channels (for example, a state in which all of facial expressions, gaze, head gestures, and utterances occur simultaneously). This is particularly effective when there are few learning samples for the target variable space. The other is that it is easy to add or delete action channels as observation information.

In the naive Bayes model in this embodiment, the posterior probability distribution P (e _t | B) is defined as shown in Equation (1).

Here, P (dt _t ^b | c _t ^b , e _t ) is a timing model, and when the behavior between the two parties is behavior coherence c _t ^b with time difference dt _t ^b around time t Represents the likelihood that the external observer's sympathy interpretation is e. The coincidence c is a binary state indicating whether or not the behaviors of the two parties are the same, and is determined based on whether or not the categories of the behaviors of the two parties are the same. P (b _t , e _t ) is a static model that models how the action channel b co-occurs between the two parties at the instant of time t. These two models will be described in turn below. P (e _t ) is a prior distribution of the sympathy interpretation e, and represents the probability that each sympathy interpretation e is generated when no action is considered.

<< Timing model >>
The timing model for the action channel b in this embodiment is defined as shown in Equation (2).

As is clear from equation (2), this timing model is based on the time difference function P (d ~) representing the likelihood of the empathy interpretation e when the time difference between the actions of the two conversations is dt and the coincidence is c. t _t ^b | c _t ^b , e _t ) and a change timing function π _t representing the timing at which the sympathetic interpretation e changes in the vicinity of the interaction. d to t _t ^b are bin numbers when the time series of the external observer's empathy interpretation is converted into a histogram. For example, the bin size is 200 milliseconds.

  In this embodiment, for each behavior channel, a timing model between two parties is constructed within the behavior channel, but a model between behavior channels may be constructed. For example, the relationship between the time difference dt between the facial expression and the head gesture, the consistency c, and the empathy interpretation e can be modeled. However, in this case, when determining the consistency c, it is necessary to introduce a new category group that can determine the consistency c even between different behavior channels such as positive / neutral / negative, for example. is there. These categories may be recognized when the action channel is detected from the video, or once they are recognized by different category groups for each action channel, and their labels are affirmed if the facial expression is small. You may reclassify later as positive / neutral / negative.

<< Time difference function >>
The time difference function P (d ~ t _t ^b | c _t ^b , e _t ) indicates that the sympathetic interpretation e is based on the coincidence c indicating whether or not the actions between the two parties are matched in the action channel b and the time difference dt. The likelihood of which type is likely to be represented. In this embodiment, bin numbers d to t _t ^b when the time series of the sympathy interpretation of the external observer are converted into a histogram are used. For example, the bin size is 200 milliseconds.

FIG. 12 shows an example of the time difference function of this embodiment. Time difference function ^{_{P (d ~ t t b |}} c t b, e t) determines the likelihood of sympathetic interpretation e by the bin number d ~ t _t ^b Consistency c and time difference of behavior of the interlocutor. FIG. 12A is an example of a time difference function when the actions between the interlocutors match, and FIG. 12B is an example of the time difference function when the actions between the interlocutors do not match. For example, if the behaviors of the interlocutors match, and the time difference between the giver's action expression and the receiver's reaction expression is 500 milliseconds, the likelihood that the empathy interpretation e is "sympathy" The likelihood of about 0.3, “Neither” is about 0.2, and the likelihood of “antisense” is about 0.5. The time difference function calculates the time series of empathy interpretations labeled by external observers in units of time difference bins, and normalizes the sum of likelihood in all time difference bins to be 1 for each category of empathy interpretation e. Ask.

<< Change timing function >>
The change timing function π represents at which timing the empathy interpretation e changes. Viewed another way, the change timing function π determines to what extent the time difference function contributes to the estimation of the empathy interpretation e in equation (1) over which range.

  In this embodiment, the change timing function is modeled as shown in Equation (3).

Here, t _a represents the time behavior expression initiation hand given in the interaction of interest. In addition, time t ′ represents the relative time during the interaction when the action expression start time of the giver is t ′ = 0 and the reaction expression start time of the receiver is t ′ = 1. Calculated as = (tt _a ) / dt.

π = 0 represents that the timing model P (dt _t ^b | c _t ^b , e _t ) does not contribute at all in the posterior probability distribution P (e _t | B) represented by the equation (1). π = 1 represents that the timing model P (dt _t ^b | c _t ^b , e _t ) contributes completely in the posterior probability distribution P (e _t | B).

  The condition dt> L represents that the response expression of the receiver is too late with respect to the action expression of the giver. For example, in this embodiment, the threshold value L is 2 seconds. This is a value based on the research results that the expression of the listener's facial expression for the vocabulary important phrase of the speaker occurs in the range of about 500 to 2,500 milliseconds, and it is generally within this range in any action channel. Based on assumptions. For details on the above research results, see “GR Jonsdottir, J. Gratch, E. Fast, and KR Thorisson,“ Fluid semantic back-channel feedback in dialogue: Challenges & progress ”, International Conference Intelligent Virtual Agents (IVA), pp. 154-160, 2007. (Reference 5).

The condition tt _a > W means that the time elapses from the expression of the facial expression of the giving hand that was most recently expressed before time t. When interaction is performed by expressing actions between the two parties, the sympathy interpretation of the external observer is affected by the timing for a certain period of time, but if the next interaction does not occur for a while after that, It is modeled that the effect disappears. The threshold value W may be any value as long as it is a positive value, and there is no problem even if the threshold value W is infinite when interaction between the two parties is continuously generated as in a two-party dialogue. However, in a situation where one person is mainly speaking in a dialogue with a large number of people and the interaction is not always frequent, such as an interaction between two listeners, a threshold is set. It is also possible that W is too long. In this embodiment, the threshold value W is empirically set to 4 seconds. This is based on the experimental result that the estimation accuracy is the highest when the threshold W is set to around 4 seconds.

  FIG. 13 shows an example of sympathy interpretation, behavioral expression of the giver, and response expression of the receiver. The filled pattern in FIG. 13 represents the difference in the category of behavior or empathy interpretation. For the values of α and β, for example, α = 0.2 and β = 0.8 are set. These values are determined so that the change timing function π of Equation (3) approximates the cumulative probability most.

  FIG. 14 shows an example of the change timing function π. The points plotted on the graph are actually the empathetic interpretation labels given by nine external observers to the dialogue data of four dialogue groups of four women (16 people in total). This represents the cumulative probability of the change at time t ′. It can be seen that this change timing function can be approximated well. However, α and β are not limited to these values, and α + β = 1, 0 ≦ α ≦ 1, and 0 ≦ β ≦ 1 are satisfied. As a simple setting, “α = 0, β = 1” may be used.

  15 and 16 are diagrams schematically showing an example of the effective range of the change timing function. A black fill indicates a state in which no action is detected, and a white fill and diagonal hatching indicate a category of action. The vertical hatching of the sympathy interpretation indicates empathy, and the horizontal hatching indicates counteraction. FIG. 15A shows an effective range in the case where actions between the interlocutors coincide. Since the behavior of the giver and the response of the recipient match, “sympathy” continues only during the threshold W. FIG. 15B shows an effective range in the case where the actions between the dialoguers do not match. Since the behavior of the giver and the response of the recipient are inconsistent, “disgust” continues only during the threshold W. FIG. 15C shows a situation where the response expression of the receiver is too late with respect to the action expression of the giver, that is, the change timing function is outside the effective range because dt> L. In this case, the “neither” state continues throughout. FIG. 16 shows the effective range when the two dialogues alternately express their actions. The basic idea is the same as in FIGS. 15A to 15C.

<< Static model >>
The static model P (b _t | e _t ) shows the likelihood that the sympathetic interpretation e is generated when a specific action co-occurs between two parties for the action channel b at time t. Modeled.

  The modeling method has been proposed in Patent Document 1 and Non-Patent Document 1 for facial expression and line of sight, and therefore, it is sufficient to follow the description in these documents. What is necessary is just to combine the model of co-occurrence with the state of the expression for every state. Here, the gaze state between the two may be, for example, three states of mutual gaze / one-side gaze / mutual gaze.

The static model for head gestures is represented by P (g | e). Here, g represents a combination state of head gestures between two parties. When the number of states of the head gestures of interest and N _g, the number of states of combinations of head gestures between two parties becomes N _g × N _g. Arbitrary types and numbers may be targeted as categories, but if the number is too large, overlearning tends to occur when the number of learning samples is small. In that case, the categories prepared first may be further grouped by clustering. For example, Sequential Backward Selection (SBS) is one of the methods. For example, when targeting the category of head gesture, the estimation using only the head gesture, that is, the posterior probability P (e | B): = P (e) P (g '| e) The two categories selected for the best estimation accuracy are integrated into one. It is sufficient to reduce the number of categories one by one by repeating this until just before the estimation accuracy deteriorates. Here, g ′ is a combined state of the head gesture between the two after grouping. As for the presence or absence of utterance, the co-occurrence between two parties is modeled in the same way as the head gesture.

<< Model learning method >>
In this embodiment, since any model is described as a discrete state, in the learning phase, the frequency of how many times the discrete state appears in the learning sample is taken, and finally the frequency is normalized (probabilized). )do it.

  At this time, as a policy to prepare the model, if the group of conversations captured in the learning video used to learn the model parameters is the same as the group of conversations captured in the estimation video for which the conversation state is to be estimated. For example, the parameters are learned independently for each of the two conversations, and the parameters learned from the data of the two conversations may be used for estimation of the two conversations. On the other hand, if the group of interrogators captured in the video for learning differs from the group of interrogators captured in the video for estimation, one model is learned without distinguishing between the two conversations, It is only necessary to make an estimation about two parties who want to estimate using the model.

  The empathy interpretation estimation apparatus and method can be used, for example, in training or a game for expressing a dialogue act. For example, in a game such as a love simulation, a sympathy interpretation estimation apparatus and method may be used to estimate a sympathy interpretation between a character appearing in the game and a player playing the game.

20 action recognition unit 50 posterior probability estimation unit 72 estimation video storage unit 81 video acquisition unit 82 person video presentation unit 83 display unit 84 action presentation unit 100 first person 200 second person

Claims (9)

A person video presentation unit that presents a first video of the first person whose behavior changes to the second person;
An action presenting unit that presents the second person with an action related to empathy against the change in the behavior of the first person.
A video acquisition unit for acquiring a second video obtained by photographing the head of the second person presented with the first video;
An action recognition unit that generates a second action time series of the second person by detecting the action of the second person with respect to a change in the action of the first person in the second video;
Based on the first action time series of the first person in the first video and the second action time series of the second person in the second video , the empathy interpretation based on the time difference of actions and the consistency of actions A posterior probability estimator that estimates a sympathetic interpretation between the first person and the second person using a model parameter including a timing model representing likelihood;
A display unit that presents the estimated sympathy interpretation to the second person as a result of a change in the behavior of the first person;
A sympathetic interpretation estimation device. In the sympathetic interpretation estimation apparatus according to claim 1,
  The display unit presents only the sympathy interpretation related to the action presented by the action presentation unit among the estimated sympathy interpretations.
  Empathy interpretation estimation device. In the empathy interpretation estimation apparatus of Claim 1 or 2 ,
The action recognition unit further generates a first action time series of the first person by detecting the action of the first person in the first video,
The posterior probability estimation unit estimates the empathy interpretation based on the first action time series and the second action time series generated by the action recognition unit,
Empathy interpretation estimation device. In the empathy interpretation estimation apparatus in any one of Claim 1 to 3 ,
The person video presentation unit presents each video of a plurality of persons including at least one person whose behavior changes to the second person,
By the processing of the behavior recognition unit and the posterior probability estimation unit that each of the at least one person whose behavior changes is the first person, the sympathy interpretation between each first person and the second person is performed. presume,
Empathy interpretation estimation device. A person video presenting step in which the person video presenting section presents the first video of the first person whose behavior changes to the second person;
An action presenting step in which an action presenting unit presents an action relating to empathy against the second person in response to a change in the behavior of the first person;
A video acquisition step in which the video acquisition unit acquires a second video obtained by photographing the head of the second person on which the first video is presented;
A behavior recognition step of generating a second action time series of the second person by the behavior recognizing unit detects the behavior of the second person with respect to a change in behavior of the first person in the above Symbol second image,
The posterior probability estimation unit is configured to determine the time difference between actions and the consistency of actions based on the first action time series of the first person in the first picture and the second action time series of the second person in the second picture. A posterior probability estimation step for estimating a sympathy interpretation between the first person and the second person using a model parameter including a timing model representing a likelihood of the sympathy interpretation based on:
A display step in which the display unit presents the estimated empathy interpretation to the second person as a result of a change in the behavior of the first person;
Empathy interpretation estimation method including   In the sympathetic interpretation estimation method of claim 5,
  The display unit presents only the sympathy interpretation related to the action presented by the action presentation unit among the estimated sympathy interpretations.
  Emotional interpretation estimation method.   In the sympathetic interpretation estimation method according to claim 5 or 6,
  The action recognition unit further generates a first action time series of the first person by detecting the action of the first person in the first video,
  The posterior probability estimation unit estimates the empathy interpretation based on the first action time series and the second action time series generated by the action recognition unit,
  Emotional interpretation estimation method.   In the empathy interpretation estimation method in any one of Claim 5 to 7,
  The person video presentation unit presents each video of a plurality of persons including at least one person whose behavior changes to the second person,
  By the processing of the behavior recognition unit and the posterior probability estimation unit that each of the at least one person whose behavior changes is the first person, the sympathy interpretation between each first person and the second person is performed. presume,
  Emotional interpretation estimation method. The program for functioning a computer as each part of the empathy interpretation estimation apparatus described in any one of Claim 1 to 4 .

Images