JP4445348B2 - Conference / discourse state estimation method, apparatus, and conference / discourse state estimation program - Google Patents

Description

  The present invention relates to a technique for estimating and displaying a state of a meeting or a discourse.

  As a technique for searching for a desired scene from moving images and sounds, there is Patent Document 1. In this method, body motion information is obtained from a moving image, and the feature amount is recorded, so that a desired scene can be searched.

  There is Non-Patent Document 1 as a technology related to conference state record retrieval in a conference. This is a technique that allows a conference participant to input a speech intention and organizes and registers the agenda in advance to capture the state of the conference and perform a search.

There is Non-Patent Document 2 as a technique related to estimation of a meeting state in a meeting. This is a technique for estimating a conference state by measuring a heart rate and comparing it between conference participants.
Japanese Patent Laid-Open No. 7-253986 Mitsunobu Tanaka, Mitsunori Fukujuku, Yoshihisa Nishibori, Kaumi Teshigawara, “Development and Evaluation of Multimedia Minutes in Synchronous / Asynchronous Integrated Multimedia Conference System ASSIST”, DICOMO99 Proceedings, pp. 79-84, 1999. Michi Hosoda, Akira Nakayama, Satoshi Kobayashi, Satoshi Iwaki, “Conference State Estimation Based on Biological Information—Observation of Heart Rate Fluctuation Resonance Phenomenon”, Information Processing Research Report 2004-GN-50 (7) 2004-EIP-22 (7) , VR Gakken Bulletin CSVC2004-7, pp. 43-48, 2004.

  The technique of Patent Document 1 can record and search for features such as body movements for data such as moving images, but does not take into account meetings or discourses held by two or more people. The state of cannot be estimated.

  In the technique of Non-Patent Document 1, in order to search for the status of a conference, it is necessary to organize and register the agenda in advance and then let the participants input the intention of the speech, which makes complicated preparations and operations during the conference It has become.

  The technique of Non-Patent Document 2 measures the heart rate of a conference participant in order to estimate the conference state. Therefore, it is necessary to prepare a heart rate measuring device that has not been necessary in the past and attach it to each participant, which makes the use complicated. It is also conceivable that measuring heart rate is a psychological pressure on the participants.

  An object of the present invention is to provide a meeting / discourse state estimation method, apparatus, and program for estimating a state of a meeting or a discourse using information on a participant's physical motion.

The conference / discourse state estimation method of the present invention is a conference / discourse state estimation method for estimating the state of a conference or discourse using a conference / discourse state estimation device,
A body motion information acquiring means for acquiring information representing the body motion of each participant;
Body motion information processing means performs predetermined processing on the obtained body motion information, and outputs the processed body motion information in time series; and
In what state the conference / discourse state estimation means is the time series data that is the body motion information output in time series from the body motion information processing means and the conference or discourse stored in advance in the conference / discourse state dictionary Calculating a correlation coefficient between time-series data indicating whether or not there is, and estimating a meeting / discourse state having the highest correlation coefficient as a participant's individual state; A conference / discourse state estimation step for estimating that the conference / discussion state that was common is the entire conference state .

  The present invention obtains participants' body movement information in a time series in a meeting or discourse and analyzes the time series body movement information to indicate what kind of state the meeting or discourse is. Is estimated.

  In addition, by displaying the current meeting / discourse state during or at the end of a meeting or conversation, it is possible to excite the discussion, or the speaker can know the state of the audience and use it to change the topic. Also, if you record the meeting / discourse status along with the image and sound of the meeting / discussion, you can search for the desired meeting / discussion time during or after the conference / discussion and play back the scene. The contents can be confirmed.

  The present invention acquires the body motion information of the conference / discourse participant and analyzes the relationship of the body motion information between the participants, so that it is not necessary to attach a heart rate measuring device or the like to the participant. Meeting / discourse status can be estimated without psychological pressure on.

  Next, embodiments of the present invention will be described with reference to the drawings.

  FIG. 1 shows the configuration of a conference / discourse state detection / search apparatus according to an embodiment of the present invention.

The conference / discussion participants 1 ₁ , 1 ₂ ,..., 1 _n are each body movement information (head position, head angle, line-of-sight direction, hand position, hand angle of each participant 1 ₁ to 1 _n , A part or all of one or a plurality of body motion information acquisition units 2 ₁ , 2 ₂ ,..., 2 _n which are, for example, 3D position sensors and 3D angle sensors. Wearing. Body movement information processing unit 3 _1, 3 _2, ..., 3 _n each body movement information acquisition unit 2 _1, 2 _2, ..., performs predetermined processing on the acquired body movement information 2 _n. The meeting / discourse state dictionary 4 records the combination of the characteristics of the body movement information and the meeting / discourse state indicating the state of the meeting or the discourse. The conference / discourse state estimation unit 5 estimates the state of the conference or the discourse using the conference / discourse state dictionary 4 from the body motion information sent from the body motion information processing units 3 _{1 to} 3 _n , and the result of the conference / Discourse status display units 10 ₁ , 10 ₂ ,..., 10 _n and the conference / discussion status recording unit 6. The conference / discourse status display sections 10 ₁ to 10 _n indicate the status of the conference or discourse to the conference / discourse participants 11 ₁ to 1 _n , respectively. The meeting / discourse state recording unit 6 records the state of the meeting or the discourse and the time information of the time when the body movement information is acquired in the meeting / discourse database 7. The conference / discourse state search unit 8 searches the conference / discourse database 7 to obtain time information of a desired conference / discourse state, and displays the time information on the conference / discourse state search result display unit 9.

As an example of a three-dimensional position sensor or a three-dimensional angle sensor used for the body motion information acquisition units 2 _{1 to} 2 _n , Polhemus FASTRAK that can obtain a three-dimensional position and angle using a magnetic field can be used. In order to obtain the line-of-sight direction in the body motion information acquisition units 2 ₁ to 2 _n , NAC Image Technology EMR-NC can be used.

As another example of the body motion information acquiring unit 2 ₁ to 2 _n, need not be worn on the body of the conference / discourse participant 1 ₁ to 1 _n, a camera for photographing a 1 ₁ to 1 _n Meeting / discourse participants is there. In this case, in order to detect the head movement from the camera image, the body movement information processing units 3 _{1 to} 3 _n detect the position of the face in the image using the Intel OpenCV library, and the position of the face in the image It is possible to obtain the head position from

The body motion information processing units 3 _{1 to} 3 _n calculate velocity, acceleration, angular velocity, and angular acceleration from the obtained position and angle, and pass noise components through filters that can pass only frequency components of predetermined bands, respectively. The time series data is removed and sent to the conference / discourse state estimation unit 5. When the body motion information acquisition units 2 _{1 to} 2 _n are cameras, the body motion information processing units 3 _{1 to} 3 _n detect body motions from images obtained from the cameras and have information indicating the body motions with predetermined characteristics . The time series data is filtered and sent to the conference / discourse state estimation unit 5.

  The conference / discourse state estimation unit 5 compares the time-series data of a certain participant with the time-series data of other participants and / or the conference / discourse state dictionary 4 within a certain range at regular time intervals. The state of the meeting or discourse at the time when the body movement information is acquired is estimated by comparison.

FIG. 2 shows a processing example of the meeting / discourse state estimation unit 5. In this example, there are four participants. The correlation matrix for four people is a 4 × 4 matrix, and the correlation matrix for n people is an n × n matrix. The element of n rows and m columns of the correlation matrix is a correlation coefficient between the nth participant and the mth participant. The correlation coefficient is close to 1 if the positive correlation is strong, close to 0 if there is no correlation, and close to -1 if the negative correlation is strong. In the conference / discourse state dictionary 4, combinations of correlation matrix and conference / discourse state shown in FIG. 3 are registered in advance. First, time series data for the past 10 seconds is taken out every second (step 101), a correlation coefficient between each participant is calculated (step 102), and a correlation matrix is obtained. The calculated correlation matrix is compared with each correlation matrix in the conference / discourse state dictionary 4 (step 103). The comparison method takes a matrix difference and uses its absolute value. It is estimated that the meeting state has the smallest absolute value (step 104), and the result is sent to the meeting / discourse state display units 10 ₁ to 10 _n and the meeting / discourse state recording unit 6 (step 105).

FIG. 4 shows another process example of the conference / discourse state estimation unit 5. In this example, there are five participants. In the meeting discourse state dictionary 4, a combination of the time series data and the meeting / discourse state shown in FIG. 5 is registered in advance. In the estimation, first, time series data of the past 10 seconds is taken out every second (step 101). A correlation coefficient with the time series data of the conference / discourse state dictionary 4 is taken for the time series data of each participant (step 106). It is estimated that the conference / discourse state having the highest correlation coefficient is the state of the individual participant (step 107), and the most common conference / discourse state is the overall conference state among all participants. Estimate (step 108), and send the result to the conference / discourse state display units 10 ₁ to 10 _n and the conference / discourse state recording unit 6 (step 105).

The conference / discourse state display units 10 ₁ to 10 _n present the estimation results of the conference / discourse state estimation unit 5 to the participants 11 ₁ to 1 _n . A display example is shown in FIG.

  The conference / discourse state recording unit 6 records the estimated information sent from the conference / discourse state estimation unit 5 and the time when the body movement information is acquired in the conference / discourse database 7. An example of a record recorded in the conference / discourse database 7 is shown in FIG.

  The conference / discourse state search unit 8 searches the conference / discourse state time using the conference / discourse database 7 by inputting the conference / discourse state to be searched. The search result is sent to the conference / discourse state search result display unit 9.

  Note that the conference / discourse state estimation method of the present invention is recorded on a computer-readable recording medium, and the program for realizing the function is recorded on the recording medium, in addition to that realized by dedicated hardware. The program may be read by a computer system and executed. The computer-readable recording medium refers to a recording medium such as a flexible disk, a magneto-optical disk, and a CD-ROM, and a storage device such as a hard disk device built in a computer system. Furthermore, a computer-readable recording medium is a server that dynamically holds a program (transmission medium or transmission wave) for a short period of time, as in the case of transmitting a program via the Internet, and a server in that case. Some of them hold programs for a certain period of time, such as volatile memory inside computer systems.

It is a figure which shows the structure of the meeting / discourse state estimation apparatus of one Embodiment of this invention. It is a flowchart which shows the process example of the meeting / discourse state estimation part 5. FIG. It is a figure which shows the example of the correlation matrix registered in the meeting / discourse state dictionary 4, and a meeting / discourse state. 10 is a flowchart illustrating another example of processing of the meeting / discourse state estimation unit 5. It is a figure which shows the example of the time series data registered in the meeting / discourse state dictionary 4, and a meeting / discourse state. It is a figure which shows the example of a display of a meeting / discourse state. It is a figure which shows the example of the record recorded on the meeting / discourse database.

Explanation of symbols

1 ₁ to 1 _n conference participants 2 ₁ to 2 _n body motion information acquisition unit 3 _{1 to} 3 _n body motion information processing unit 4 conference / discourse state dictionary 5 conference / discourse state estimation unit 6 conference / discourse state recording unit 7 conference / Discourse database 8Conference / Discourse state search unit 9Conference / Discourse state search result display unit 101-108 steps

Claims (9)

A conference / discourse state estimation method for estimating a conference or discourse state using a conference / discourse state estimation device,
A body motion information acquiring means for acquiring information representing the body motion of each participant;
Body motion information processing means performs predetermined processing on the obtained body motion information, and outputs the processed body motion information in time series; and
In what state the conference / discourse state estimation means is the time series data that is the body motion information output in time series from the body motion information processing means and the conference or discourse stored in advance in the conference / discourse state dictionary Calculating a correlation coefficient between time-series data indicating whether or not there is, and estimating a meeting / discourse state having the highest correlation coefficient as a participant's individual state; A conference / discourse state estimation method including a conference / discourse state estimation step of estimating that a large number of conference / discourse states are the entire conference state . Body motion information acquisition means for acquiring information representing the body motion of each participant in a meeting or discourse;
Body motion information processing means for performing predetermined processing on the obtained body motion information and outputting the processed body motion information in time series;
A meeting / discourse state dictionary that records a combination of time series data, which is body movement information output in time series from the body movement information processing means, and a meeting / discourse state indicating the state of the meeting or discourse. When,
Calculate the correlation coefficient between the time series data output from the body motion information processing means and the time series data stored in the meeting / discourse state dictionary, and participate in the meeting / discourse state with the highest correlation coefficient who estimated that individuals who are in the state, conference / discourse state estimation and a conference / discourse state estimation means most common conference / discourse state in the state of the participant individual estimates that the entire conference state apparatus. The body motion information acquisition means acquires part or all of each participant's head position, head angle, line-of-sight direction, hand position, hand angle, body position, body angle. 2. The conference / discourse state estimation device according to 2. The body motion information acquisition means is one or a plurality of three-dimensional position and three-dimensional angle sensors, all or a part of which are attached to a participant,
The body motion information processing means detects part or all of the position, velocity, acceleration, angle, angular velocity, and angular acceleration of each sensor from the position and angle acquired by the three-dimensional position sensor and the three-dimensional angle sensor. And apply these to a filter with predetermined characteristics to output time-series data,
The conference / discourse state estimation means is configured to convert the time-series data of a certain participant into the time-series data of other participants and / or the conference / discourse state dictionary or both within a certain range of time at regular intervals. 4. The conference / discourse state estimation device according to claim 2, wherein the state of the conference or discourse at the time when the body motion information is acquired is estimated . The body movement information acquisition means is a camera that acquires images of participants,
The body information processing means detects a body motion from an image obtained by the camera, applies information representing the body motion to a filter having predetermined characteristics, and outputs time series data;
The conference / discourse state estimation means is configured to convert the time-series data of a certain participant into the time-series data of other participants and / or the conference / discourse state dictionary or both within a certain range of time at regular intervals. 4. The conference / discourse state estimation device according to claim 2, wherein the state of the conference or discourse at the time when the body motion information is acquired is estimated. The meeting / conference / discourse state estimated by the discourse state estimating means for displaying further comprises a conference / discourse status display means, conference / discourse state estimating apparatus according to any one of claims 2 to 5. Further comprising a conference / discourse state retrieval means for retrieving the time was the desired conference / discourse state after completion of the meeting / discourse in or conference / discourse from the conference discourse database, according to any one of claims 2 to 6 Meeting / discourse state estimation device. 8. The conference / discourse state estimation device according to claim 7 , further comprising a conference / discourse state search result display means for displaying a search result. Meeting / discourse state estimation program for executing the function of the conference / discourse state estimating device according a computer in any one of claims 2 to 8.

Images