JP2006338529A - Conversation structure estimation method - Google Patents

Abstract

<P>PROBLEM TO BE SOLVED: To provide a conversation structure estimation method with which structure of conversation can be automatically estimated by measuring behaviors of participants in the conversation for a scene in which a plurality of persons have conversation by meeting. <P>SOLUTION: The directions of heads of each person who participates in the conversation are measured and the directions of visual lines of each person are calculated based on the directions of heads of each person. In addition, presence of utterance of each person is detected. Then, structural information of the conversation at each point of time in the conversation is calculated based on the directions of visual lines and information about the presence of utterance of each person. <P>COPYRIGHT: (C)2007,JPO&INPIT

Description

  The present invention relates to a technique for automatically measuring or recognizing actions of a plurality of persons, and particularly, a situation in which a plurality of persons have a conversation, and is generated on the spot from observed person actions. The present invention relates to a conversation structure estimation method for automatically estimating a conversation structure.

  It is known that in a face-to-face conversation with a plurality of persons, each participant plays a role such as “speaker”, “recipient”, and “participant”, and these change over time. The automatic extraction of the conversation structure such as the division of roles during conversation and the change in time is a fundamental issue in realizing automatic indexing and automatic video editing for the construction of conference video archives. It is.

  Among the roles of people in conversation, the “speaker”, which is thought to play a central role in linguistic information transmission, has been mainly focused on, and the state of each person's utterance is used as an acoustic signal. A technique for identifying who is the speaker among a plurality of conversation participants has been proposed (see, for example, Patent Document 1).

  However, conversation has not been established by "speaker" alone, and it is gradually beginning to be recognized in recent years that information on the structure of the conversation about who the speaker's utterance was directed to is important. As a clue, it has been suggested that non-verbal information such as the behavior of the line of sight of conversation participants is useful (see Patent Document 2). Further, Patent Document 3 shows an experimental result that it is possible to determine whether the speaker is speaking to one person or a plurality of persons based on the amount of distribution of the speaker's line of sight. However, a method for automatically measuring the line of sight is proposed. Not.

On the other hand, Non-Patent Document 4 has conventionally proposed a method in which the direction of the head is detected and used instead as the direction of the line of sight, but does not estimate the structure of the conversation.
A. Gard, V. Pavlovic, and JMRehg, “Boosted learning in dynamic Bayesian networks for multimodal speaker detection,” Proc. IEEE, Vol.91, No.9, 2003. N. Jovanovic and R. Akker, “Towards automatic addressee identification in multiparty dialogues,” Proc. SIGdial, pp.89-92, 2004. Y. Takemae, K. Otsuka, and N. Mukawa, “An analysis of speakers' gaze behavior for automatic addressee identification in multiparty conversation and its application to video editing,” Proc. Of IEEE International Workshop on Robot and Human Interactive Communication (IEEE / RO-MAN2004), pp.581-586, 2004. R. Stiefelhagen et a1., “Modeling focus of attention for meeting index based on multiple cues,” IEEE Trans. Neural Networks, vo1.13, No.4, 2002.

  As described above, in the prior art, it is not possible to automatically estimate the conversation structure such as the role of the conversation participant and its time change.

  SUMMARY OF THE INVENTION Accordingly, an object of the present invention is to provide a conversation structure estimation method for automatically estimating a conversation structure by measuring a conversation participant's behavior for a scene where a plurality of persons have a conversation in person. Yes.

The present invention has been made to solve the above-described problem, and is a conversation structure estimation method in a conversation structure estimation device, in which a head direction measurement processing unit calculates a head direction of each person participating in a conversation. A gaze direction calculation processing unit calculates a gaze direction of each person based on the head direction of each person, and an utterance presence / absence measurement processing unit detects the presence / absence of each person's utterance, An estimation processing unit calculates conversation structure information at each time point during a conversation based on information on the gaze direction of each person and the presence / absence of the utterance.
In this way, by combining the gaze direction and utterance presence / absence information, it is possible to estimate the structure of the conversation, such as the other party to whom the speaker's utterance is directed, which cannot be known only from the utterance presence / absence information. It becomes. In addition, at present, it is difficult to directly measure a person's gaze direction with a device or the like without disturbing the conversation, but instead of directly measuring the gaze direction, the direction of the head is relatively easy to measure. Since the direction of the line of sight is approximately estimated, the conversation structure can be estimated by external observation without disturbing natural conversation.

In the present invention, the conversation structure estimation processing unit described above is configured such that each person participating in the conversation is a speaker, a receiver that the speaker is speaking to, a participant who is neither a speaker nor a receiver among the conversation participants. It is characterized in that the structure information of the conversation indicating which role is played at each time point among the three types of roles of the person is calculated.
Therefore, it is possible to apply the invention technology to a wide variety of applications, such as adding an index to the video that captures the conversation, by estimating the information of who is talking to whom during the conversation between multiple persons. it can.

The present invention also provides information indicating a situation in which one speaker speaks to all other conversation participants who are receivers based on the calculated conversation structure information, and two specific conversation participants. An output unit for outputting a graph indicating information about a situation in which a person is a speaker or a receiver and another person is an associate and information on a situation that does not match the two situations; Features.
This is equivalent to estimating the pattern of information transmission that governs the place of conversation that is obtained by integrating the roles of individual persons. Inventive technology can be applied to applications.

Further, the present invention calculates the gaze direction of each person based on the head direction of each person, and calculates the structure information of the conversation at each time point during the conversation based on the gaze direction of each person and the presence / absence of speech. , Respectively, a model formula indicating a relationship between the head direction and the line-of-sight direction, a model formula indicating a mutual relationship between the line-of-sight direction, the presence / absence of the utterance, and the structure information of the conversation, and their temporal changes By using the probability model formula, the simultaneous posterior probability distribution of the gaze direction and the structural information of the conversation at each time and the parameters of the probability model is calculated.
By using such a probabilistic model, uncertainties such as ambiguity of gaze when facing a certain head direction, information on gaze direction and presence / absence of speech, and ambiguity between conversation structures Can be handled appropriately using a probabilistic representation. Also, by solving the problem of estimating the gaze direction from the head direction, the problem of estimating the conversation structure from the information of the gaze direction and the presence / absence of speech, and the problem of estimating the parameters of the probability model at the same time, Uncertainty of the solution is eliminated in a complementary manner, and it becomes possible to estimate the direction of the line of sight and the structure of conversation more accurately than in the case where these individual problems are solved independently.

Further, the present invention provides the probability distribution of the gaze direction of each person depending on the conversation structure, the probability that each person speaks depending on the conversation structure, and the gaze direction of each person. Probability distribution representing the possible head direction, transition probability that determines how the conversation structure changes over time, and how the line-of-sight direction of each person depends on the conversation structure over time It is a transition probability that determines how to change.
This makes it possible to incorporate human characteristics such as the appearance of specific gaze behaviors and utterance states depending on the structure of the conversation into the model. When speaking to the receiver, a gaze is thrown at the receiver, and the receiver's person looks closely at the person speaking. In addition, the speaker is more likely to speak. By using such a model, the conversation structure can be accurately estimated from the observed human behavior.

Further, the present invention provides a simultaneous posterior probability distribution of the gaze direction and conversation structure and probability model parameters at each time using a Gibbs sampler, and each of the gaze direction and conversation structure and probability model parameters at each time. Is calculated approximately as a sample set obtained by generating random numbers from their all conditional posterior distributions.
As a result, it is possible to obtain an approximate solution even for a probability model including many unknown variables such as the present invention in which exact calculation of the simultaneous posterior probability distribution is difficult.

Further, the present invention is characterized in that the head direction of each person is measured using a magnetic sensor attached to a conversation participant's head.
This makes it possible to accurately measure the coordinates and rotation angle of each conversation participant's head in the three-dimensional space with high temporal resolution, which is an accurate estimation of the conversation structure in terms of time. It is possible.

In addition, the present invention is characterized in that the presence / absence of speech of each person is detected based on the magnitude of an acoustic signal obtained from a microphone attached to each person.
Thereby, it becomes possible to detect the individual utterance situation for each conversation participant, and it is possible to estimate the conversation structure with high accuracy.

  Further, the present invention is a program executed by a computer of a conversation structure estimation apparatus, the process of measuring the head direction of each person participating in a conversation, and each person based on the head direction of each person Structure information of the conversation at each time point during the conversation based on the process of calculating the gaze direction of each person, the process of detecting the presence or absence of the utterance of each person, and the information on the gaze direction of each person and the presence or absence of the utterance And a program for causing a computer to execute the process of calculating.

  Further, the present invention is a recording medium for storing a program to be executed by a computer of a conversation structure estimation device, the process of measuring the head direction of each person participating in a conversation, and the head direction of each person Based on the processing for calculating the gaze direction of each person based on the above, the processing for detecting the presence / absence of each person's utterance, and the information on the gaze direction of each person and the presence / absence of the utterance, Is a recording medium for storing a program for causing a computer to execute a process for calculating the structure information of the conversation.

  According to the present invention, for a face-to-face conversation between a plurality of persons, the head direction and speech state of the person in conversation are measured, and based on the measured information, the gaze direction, and the probability model related to the conversation structure. Simultaneously estimate the conversation structure, gaze direction, and model parameters at each time point. Therefore, it is possible to automatically estimate the role of each participant such as a speaker, a receiver, and a side participant during the conversation, and the structure of the conversation expressed as a change with time.

  Hereinafter, a conversation structure estimation method according to an embodiment of the present invention will be described with reference to the drawings. FIG. 1 is a block diagram showing the configuration of the conversation structure estimation apparatus according to the embodiment. In this figure, reference numeral 101 is a head direction measuring unit, 102 is an utterance state measuring unit, 103 is an observation data storage unit, 104 is a parameter storage unit, 105 is a sample set storage unit, 106 is a Gibbs sampler, 107 is A statistic calculation unit 108 is an output unit.

  The head direction measuring unit 101 is a magnetic sensor (or geomagnetic sensor) attached to each participant, for example, a head based on a predetermined direction depending on the relationship between the N pole and the head direction due to geomagnetism. Measure the direction of. The utterance state measuring unit 102 is, for example, a pin microphone attached to each participant, and measures the presence or absence of sound generation based on the magnitude of an acoustic signal obtained from the microphone. The observation data storage unit 103 stores data obtained from the head direction measurement unit 101 and the utterance state measurement unit 102 for a certain time interval. The parameter storage unit 104 stores hyperparameter values of the conversation model. The sample set storage unit 105 stores a set of samples generated by the Gibbs sampler 106. The Gibbs sampler 106 receives the observation data stored in the observation data storage unit 103 and the model hyperparameter value stored in the parameter storage unit 104 as input, and represents a simultaneous posterior probability distribution of unknown variables. A set is generated, and the value is stored in the sample set storage unit 105. The statistic calculator 107 calculates a statistic regarding the unknown variable from the sample set recorded in the sample set storage unit 105. The output unit 108 outputs the statistic calculated by the statistic calculation unit 107 to a display or the like.

FIG. 2 is a diagram showing a processing flow of the conversation structure estimation apparatus.
Next, the processing flow of the conversation structure estimation apparatus will be described with reference to FIG.
First, in a certain time interval (1 ≦ t ≦ T), the head direction measuring unit 101 measures the head direction of each participant at regular time intervals (step S501). Similarly, the utterance state measuring unit 102 measures (acquires voice) the utterance state of each participant at a certain time interval in a certain time interval (1 ≦ t ≦ T) (step S502). These measured information is recorded in the observation data storage unit 103. The measurement in steps S501 and S502 is repeated until t <T is determined in each measurement unit (step S503) and t <T is satisfied. Next, the Gibbs sampler 106 is initialized using the parameter values stored in the parameter storage unit 104 (step S504). Then, for each variable, sampling (random number generation) is performed from all conditional posterior distributions, and the process of updating the value of the variable is performed (step S505). Then, it is determined whether or not all variables have been updated by the process of step S504 (step S506). If all the variables have been updated as a result, it is then determined whether or not the number of iterations has reached a predetermined value. (Step S507). When the predetermined value is reached, the statistics for each variable are calculated using the sample set stored in the sample set storage unit 105 (step S508).

Next, the processing flow of the conversation structure estimation apparatus will be described in more detail.
FIG. 3 is a diagram showing the relative positions of the conversation participants.
As shown in FIG. 3, it is assumed that the conversation participants to be targeted in the conversation structure estimation method of the present embodiment are seated at relative coordinates as shown in FIG. 3 and have a conversation. Here, the number of persons N is N ≧ 3. Further, the time interval to be estimated is discretized at a constant time interval, and t = 1, 2,. The line-of-sight direction of the person i at time t is expressed as X _{i, t} . Further, the line-of-sight direction X _ij = j when the person i is looking toward the face of the person j is represented as X _ij = i. Also, the line of sight direction of each person is called a line-of-sight pattern, _expressed as Xa _t = {X _{1, t} , X _{2, t} ,..., X _{N, t} }, and each time interval in the target time interval It is assumed that the line-of-sight pattern series is _expressed as Xa _{1: t} = {X ₁ , X ₂ ,..., X _N }.

The conversation structure at a certain time _t is denoted as St. The structure when one person i talks to all other participants

  . This structure is called a “convergence structure”. Also, the situation where the conversation is progressing only between the two persons i and j among the participants, that is, the case where the person i and the person j are speakers or receivers is “binary join”. The symbol

Represented by Further, a structure other than these structures is referred to as a “divergent structure” and expressed as a symbol S _t = R ⁰ . In a conversation of N (≧ 3) people, there are M = N + _N C ₂ +1 structures in consideration of the combination of target persons for the above three types of structures, Conversation state

Shall be taken. A series of conversation states in the target time range is represented as S _{1: t} = {S ₁ , S ₂ ,..., S _T }.

First, as described above, in step S501, the head direction measuring unit 101 measures the head directions h _{i, t} of each participant i at each time t. As shown in FIG. 3, this measured value is the horizontal rotation angle of the head (based on the positive X-axis direction) and is measured as the angle formed with the coordinate axis when the person is viewed from above. And A set of measurement values in the head direction in the target time interval is represented as H _{1: t} = {H ₁ ,..., H _T }, H _t = {h _{1, t} ,..., H _{N, t} }. Expressed in Further, as described above, in step S502, the utterance state measuring unit 102 measures the utterance state u _{i, t} of each participant i at each time t. This utterance state is the presence or absence of an utterance, and the presence or absence of the utterance is represented by a binary value of 0 or 1. The observation data of the utterance in the target time interval is expressed as U _{1: t} = {U ₁ ,..., U _T }, U _t = {u _{1, t} ,..., U _{N, t} }. These observation data are recorded in the observation data storage unit 103.

FIG. 4 is a diagram showing a conversation model.
As the conversation model, a dynamic Bayesian network as shown in FIG. 4 can be used. In this conversation model, the structure of conversation is the initial probability.

And state transition probability

Suppose we follow a Markov process with Putting these parameters together,

Notation is as follows. Further, it is assumed that the line-of-sight pattern Xa _t appears according to the generation probability P (Xa _t | S _t ) and the transition probability P (Xa _t | Xa _t−1 , S _t−1 ) depending on the conversation structure, The likelihood is

Define as follows. However, here, it is assumed that the gaze direction of each person is conditionally independent when a conversation structure is given. Parameters for gaze direction

Notation is as follows. Also, the likelihood distribution of head direction H _t at a certain gaze pattern Xa _t, using a Gaussian function

It expresses like this.
Here, μ _ij and σ ² _ij represent the mean and variance of the likelihood distribution in the head direction when the person i views the person j, respectively. Also, assume that each conversation participant speaks according to a Bernoulli process that depends on the conversation state, and the likelihood of speech is determined.

And the probability of uttering

It expresses like this.

Based on the above conversation model, in the present invention, all unknown variables, that is, conversation structure series S _{1: T} , line-of-sight pattern series Xa _{1: T} , and conversation model parameters.

The observation data

The goal is to calculate and estimate more. In one embodiment of the invention, the Gibbs sampler 106 takes a Bayesian approach and calculates the joint posterior probability distribution for these unknown variables using a method called Gibbs sampling. In Gibbs sampling, first, the initial value for each variable is set by sampling from the prior probability distribution, and then, for each variable, sampling is performed from the all conditional posterior probability distribution, and the value of the variable is updated. Run repeatedly. The sample set after iterating a sufficient number of times is considered to approximate the simultaneous posterior probability distribution of the unknown variable, and the statistic calculator 107 calculates the statistic for the unknown variable from the sample set.

  In this embodiment, a conjugate prior distribution is adopted as the shape of the prior probability distribution for each unknown variable. The initial probability of the conversation structure, the state transition probability, the line-of-sight pattern generation probability, and the prior distribution of the state transition probability are assumed to follow independent Dirichlet distributions. Further, the mean of the likelihood distribution in the head direction and the prior distribution of the variance follow a Gaussian distribution and an inverse chi-square distribution, respectively. In addition, the prior distribution of the utterance probability follows the beta distribution.

  In order to set the line-of-sight pattern and speech state specific to each conversation structure, the shape of these prior distributions is set as a hyperparameter value. For example, the structure when one person i talks to all other participants

, The distribution of the line-of-sight direction of the speaker i is uniform, and the line-of-sight direction of the receiver j (≠ i) is set to take a high value with respect to the speaker. The utterance probability takes a high value only for the speaker i. Also two-party structure

In this case, the probability of the gaze direction that causes mutual gaze state between the persons of the target pair (i, j) is high, and the gaze direction distribution of the persons other than the pair is uniform. To do. Furthermore, the utterance probability is set to take a high value for the person in this pair. Further, in the divergent structure R ^0, it is assumed that the distribution in the gaze direction of each person is uniform and the utterance probability has a low value. The parameter storage unit 104 stores the values set in this way.

  In step S505, the Gibbs sampler 106 performs Gibbs sampling. First, unknown variables

For each of these, a random number is generated (sampling) from a prior distribution determined by the value stored in the parameter storage unit 104, and the value is set as a variable value. Here, the value stored in the parameter storage unit 104 is a parameter representing the shape of the prior probability distribution. Specifically, for the initial probability of the conversation structure, the Dirichlet distribution that is the prior distribution is used. Regarding the parameter value and the state transition probability of the conversation structure, the parameter value of the Dirichlay distribution that is the prior distribution, and for the generation probability of the line-of-sight pattern, the parameter value of the Dirichlay distribution that is the prior distribution and the state transition of the line-of-sight pattern For the probability, the parameter value of the Dirichlet distribution that is the prior distribution, the average value of the likelihood distribution (Gaussian distribution) in the head direction, the mean and variance values of the Gaussian distribution that is the prior distribution, the head direction For the variance of the likelihood distribution (Gaussian distribution) of, the degree of freedom of the inverse chi-square distribution that is its prior distribution and the scale parameter It is.

  The Gibbs sampler 106 performs sampling from the all conditional posterior distribution for each unknown variable, and updates the value of each variable. When the number of repetitions q is equal to or greater than a certain number of times q ≧ Q ′, the value of the result is stored in the sample assembly memory part 105. Since all conditional posterior distributions are natural conjugate distributions, they have the same function form as the prior distributions, and the initial probability of conversation state, state transition probability, line-of-sight pattern generation probability, and state transition probability prior distribution are respectively Independent Dirichlet distribution. In addition, the average of the likelihood distribution in the head direction and the all-conditional posterior distribution of variance are a Gaussian distribution and an inverse chi-in distribution, respectively. In addition, the utterance probability has a beta distribution. In addition, the all-conditional posterior distribution of the conversation structure at each time is

The state is updated by sampling from this distribution. Furthermore, with regard to the line-of-sight pattern at each time, all conditional posterior distributions

The state is updated by sampling from.

When the Gibbs sampler 106 repeats and executes Gibbs sampling Q times, the Gibbs sampler 106 records the value of each unknown variable obtained as a result of the calculation in the sample set storage unit 105. When the value of each unknown variable obtained as a calculation result is the result of the qth iteration, Xa _{1: t} ^(q) , Sa _{1: t} ^(q) , φ ^(q) <Expression (17 ) Of each of the values> corresponding to the qth iteration of the unknown variable. After that, the statistic calculation unit 107 reads the sample set (value of each unknown variable output by the cast sampler 106) from the sample set storage unit 105, and calculates an estimated value for each unknown variable. For example, for conversational structures and gaze patterns, the maximum posterior probability estimate is

It is calculated as follows. here,

, Other cases

It is. For other unknown variables, the least square error estimate is

It is calculated as follows.

In the following, some of the results obtained by the above example will be described.
The method of the present invention was applied to a time interval of 10000 frames (about 5.6 minutes) with a time interval of 1/30 seconds for a conversation by four people arranged as shown in FIG.

FIG. 5 is a diagram showing a part of observation data (head direction and presence / absence of speech).
FIG. 5 shows a part of the observation data measured by the head direction measurement unit 101 and the utterance state measurement unit 102 and stored in the observation data storage unit 103.

FIG. 6 is a diagram illustrating a part of the estimation result (gaze direction and conversation structure).
As one method of presenting the estimation result value, a form in which the estimated value of the conversation structure at each time is displayed as a time series diagram as shown in FIG. This figure shows who is mainly involved in the conversation at each time. Further, FIG. 6 shows an estimation result output to the output unit 108 after the processing of the statistic calculation unit 107 described above after Q = 700 (Q ′ = 500) iteration processing is performed by the Gibbs sampler 106. An example is shown, and the line-of-sight directions to other participants (person 1 to person 4 = P1 to P4) and the structure of conversation (whether or not conversation is present) are shown.

FIG. 6 is obtained by performing the following procedure at all times.
First, convergence structure to estimate Sb _t a person i conversation state of at each time t, that _is, when the Sb t _{= R} ^{i C} <Equation (1) in the same>, the output unit 108, the person i A band is displayed at the position (Sb represents an estimated value). When the estimated value Sb _{t of the} conversation state is a two-party combination of the persons i and j, that is, when Sb _t = R _{(i, j)} ^DL <same as the expression (2)>, the output unit 108 Bands are displayed at the positions of human figures i and j, respectively. Further, when the estimated value Sb _t conversation state is divergent structure, the output unit 108 does not display the band at that time.

  As another method of presenting the estimation result of the conversation structure other than that in FIG. 6, the output unit 108 switches the video of the corresponding person to determine who is mainly involved in the conversation at each time. It is also possible to present it by displaying it. By viewing such a video, even a person who does not participate in the conversation can easily grasp the structure of the conversation, and can understand the conversation content more accurately.

  In the graph of the gaze direction of each participant in FIG. 6, correct data (broken line) of the gaze direction (solid line) of each person estimated by the conversation structure estimation device and the actual gaze direction labeled manually. And are superimposed. From a comparison between the two, the agreement rate between the two averaged about 70% on average, and a reasonable estimation accuracy was obtained. Further, in the portion showing the estimation result of the conversation structure in FIG. 6, when the band at each time is viewed, at the time when there is a single black band, the convergence of the person with the number i in which the band exists becomes the speaker. Construction

The time when there is a double band is a two-party combination of the two numbers i and j where the band exists.

It has the structure of Further, the time when no band exists has a divergent structure ^R0 .
In other words, in the graph of the conversation structure in FIG. 6, when a band exists for one person P1 to P4 at each time, the conversation structure is a converging structure, and when a band exists for two persons. Indicates that the conversation structure is a two-body combination and that a band exists for three or more persons is a divergent structure.

In FIG. 6, when the estimated conversation state is a convergence structure to the person i, that is, Sb _t = R _i ^C <same as the expression (1)>, this central person i is determined to be a speaker, and other persons Is determined to be the recipient. In addition, when the estimated conversation state is a two-party combination of persons i and j, that is, Sb _t = R _{(i, j)} ^DL <same as in equation (2)>, the two persons i and j are , The speaker and receiver (which is not the speaker and the receiver). Other persons are judged as associates. If the estimated conversation state is a divergent structure, it is determined that no conversation by this group has occurred, and therefore it is determined that there are no speakers, receivers, and associates in that place.

FIG. 7 is an image of a conversation participant at 3 hours.
FIG. 8 is a diagram showing the direction of the line of sight at three times and the estimated conversation structure.
Next, in order to explain the state of time transition of the conversation structure more specifically, an image showing each participant at three times (t ₁ = 310, t ₂ = 485, t ₃ = 578) is shown in FIG. FIG. 8 shows the line-of-sight pattern at each time point and the estimated conversation structure.

In FIG. 8, the thin arrow in the graph indicates the estimated line-of-sight direction, and the thick arrow indicates the correct line-of-sight direction. First, in the vicinity of time t ₁ , the person 4 expresses his / her opinion toward everyone else. After that, around time t ₂ , the person 2 makes an utterance expressing consent to the utterance of the person 4, and the person 4 also pays attention to the utterance, and the conversation proceeds only between the person 4 and the person 2. The other persons 1 and 3 became associates. Thereafter, at time t ₃ , the utterance right was transferred to the person 2, and the person 3 who was an attendant started to listen to the person 2 while paying attention to the person 2. As shown in FIG. 8, the progress of these conversations was correctly estimated as a change in the line of sight direction, and the state of the conversation structure was estimated in such a way as to adapt to such a change in the conversation situation.

  Thus, according to the present invention, it is understood that the direction of the line of sight of each participant that changes with the progress of the conversation and the role in the conversation can be estimated, and the structure of the conversation can be estimated appropriately.

  As described above, in the present invention, for a face-to-face conversation of a plurality of persons, the head direction and the speech state of the person in conversation are measured, and the measured information, the line-of-sight direction, and the conversation structure Based on the probabilistic model, the structure of the conversation, the gaze direction, and the parameters of the model at each time point are estimated simultaneously. Therefore, it is possible to automatically estimate the role of each participant such as a speaker, a receiver, and a side participant during the conversation, and the structure of the conversation expressed as a change with time.

  The conversation structure estimation apparatus described above has a computer system inside. The process described above is stored in a computer-readable recording medium in the form of a program, and the above process is performed by the computer reading and executing this program. Here, the computer-readable recording medium means a magnetic disk, a magneto-optical disk, a CD-ROM, a DVD-ROM, a semiconductor memory, or the like. Alternatively, the computer program may be distributed to the computer via a communication line, and the computer that has received the distribution may execute the program.

  The program may be for realizing a part of the functions described above. Furthermore, what can implement | achieve the function mentioned above in combination with the program already recorded on the computer system, and what is called a difference file (difference program) may be sufficient.

It is a block diagram which shows the structure of the conversation structure estimation apparatus by one Embodiment of this invention. It is a figure which shows the processing flow of the conversation structure estimation apparatus by one Embodiment of this invention. It is a figure which shows the relative position of the conversation participant by one Embodiment of this invention. It is a figure which shows the conversation model by one Embodiment of this invention. It is a figure which shows a part of observation data (head direction and the presence or absence of speech) by one Embodiment of this invention. It is a figure which shows a part of estimation result (gaze direction and the structure of conversation) by one Embodiment of this invention. It is an image of the conversation participant in 3 time for demonstrating the transition of the structure of the conversation in one Example of this invention. It is a figure which shows the direction of the eyes | visual_axis in 3 time for demonstrating the transition of the structure of the conversation in one Example of this invention, and the estimated conversation structure.

Explanation of symbols

DESCRIPTION OF SYMBOLS 1 ... Conversation structure estimation apparatus 101 ... Head direction measurement part 102 ... Speech state measurement part 103 ... Observation data storage part 104 ... Parameter storage part 105 ... Gibbs sampler 106 ... Sample set storage unit 107 ... statistic calculation unit 108 ... output unit

Claims (10)

A conversation structure estimation method in a conversation structure estimation apparatus,
The head direction measurement processing unit measures the head direction of each person participating in the conversation,
The gaze direction calculation processing unit calculates the gaze direction of each person based on the head direction of each person,
The utterance presence / absence measurement processing unit detects the presence / absence of each person's utterance,
A conversation structure estimation processing unit, wherein the conversation structure estimation processing unit calculates conversation structure information at each time point during a conversation based on the gaze direction of each person and the presence / absence of speech. The conversation structure estimation processing unit
Each person participating in the conversation
Among the three types of roles of the speaker, the receiver to whom the speaker is talking, and the attendant who is neither the speaker nor the receiver among the conversation participants, the structure information of the conversation indicating which role is played at each time point. The conversation structure estimation method according to claim 1, wherein the conversation structure is estimated. Based on the calculated conversation structure information,
Information that indicates the situation in which one speaker speaks to all other conversation participants
Information that indicates the situation where two specific participants in a conversation are speakers or receivers and other people are associates,
The conversation structure estimation method according to claim 2, further comprising: an output unit that outputs a graph indicating information on any one of the situation information that does not match the two situations. Calculating the gaze direction of each person based on the head direction of each person;
Calculation of the structure information of the conversation at each time point during the conversation based on the information on the gaze direction of each person and the presence or absence of utterance,
A model expression indicating the relationship between the head direction and the line-of-sight direction, a model expression indicating the mutual relationship between the line-of-sight direction, the presence / absence of the utterance, and the structural information of the conversation, and a probability model expression regarding their temporal changes The conversation structure estimation method according to any one of claims 1 to 3, wherein a simultaneous posterior probability distribution of gaze direction and conversation structure information and probability model parameters at each time is calculated using. The probability model is
Probability distribution of the gaze direction of each person depending on the structure of the conversation,
The probability of each person uttering depending on the structure of the conversation;
Probability distribution representing the possible head direction with respect to the direction of each line of sight of each person,
A transition probability that determines how the structure of the conversation changes over time;
Transition probability that determines how the gaze direction of each person changes in time depending on the conversation structure,
The conversation structure estimation method according to claim 4, wherein: The simultaneous posterior probability distribution of gaze direction and conversation structure and probability model parameters at each time
As a sample set obtained by using a Gibbs sampler to generate random numbers from all the conditional posterior distributions of all unknown variables included in each of the parameters of the gaze direction and conversation structure and probability model at each time The method for estimating the conversation structure according to claim 4 or 5, wherein the calculation is performed approximately. The head direction of each person is
It measures using the magnetic sensor with which the conversation participant's head was mounted | worn. The conversation structure estimation method in any one of Claims 1-6 characterized by the above-mentioned. The presence or absence of each person's utterance is
The conversation structure estimation method according to any one of claims 1 to 7, wherein detection is performed based on a magnitude of an acoustic signal obtained from a microphone attached to each person. A program to be executed by a computer of a conversation structure estimation device,
Processing to measure the head direction of each person participating in the conversation;
A process of calculating the gaze direction of each person based on the head direction of each person;
Processing for detecting the presence or absence of each person's utterance;
Based on the information on the gaze direction of each person and the presence / absence of the utterance, processing for calculating the structure information of the conversation at each time point during the conversation;
A program that causes a computer to execute. A recording medium for storing a program to be executed by a computer of a conversation structure estimation apparatus,
Processing to measure the head direction of each person participating in the conversation;
A process of calculating the gaze direction of each person based on the head direction of each person;
Processing for detecting the presence or absence of each person's utterance;
Based on the information on the gaze direction of each person and the presence / absence of the utterance, processing for calculating the structure information of the conversation at each time point during the conversation;
Medium for storing a program for causing a computer to execute the program.

Images