JP2010219733A - Conference recording device, method of recording conference, and conference recording program - Google Patents

Abstract

<P>PROBLEM TO BE SOLVED: To provide a conference recording device capable of recording a videoconference executed by a videoconference system so that reaction of other users to remarks of one user are reproduced in synchronization. <P>SOLUTION: In the videoconference system 1, a terminal unit 3 stores moving image data mutually converted among the terminal units 3, 4, 5 corresponding to time data. The time data correlated with the moving image data are adjusted by buffering of recording delay time calculated based on delay time of a network 2. The terminal unit 3 composites the moving image data in time series based on the time data to create conference recording data to be reproduced while the moving image data generated by the terminal unit 3 are in synchronization with the moving image data generated by the terminal units 4, 5 in the reception of the moving data. <P>COPYRIGHT: (C)2010,JPO&INPIT

Description

  The present invention relates to a conference recording apparatus, a conference recording method, and a conference recording program for recording a remote conference held between computers provided at a plurality of locations.

  2. Description of the Related Art Conventionally, there is a video conference system in which conference terminals provided at a plurality of locations where users exist are connected via a network and a remote conference is performed using these conference terminals. In such a video conference system, there is provided a conference recording function that allows a remote conference to be reproduced later by recording the remote conference. For example, a content processing system is known in which each moving image obtained by photographing a meeting using a plurality of video cameras is integrated and recorded and managed while being synchronized on a time axis (for example, patents). Reference 1).

JP-A-2005-260512

  However, in the conventional video conference system, when each conference terminal is installed at a location far away from each other, a timing shift may occur in a conference video output in real time due to a network delay. Specifically, in a remote conference, a user at another site may show some kind of reaction (for example, whispering or tilting head) in response to a statement made by a user at one site. At this time, if there is a network delay in the conference video, the reaction of the user at the other site is output with a great delay after the message of the user at one site is output. In this case, even if the remote conference is recorded and reproduced, there is a possibility that it is impossible to determine which remark the user's reaction in the conference video is due to a timing shift in the conference video.

  The present invention has been made in order to solve the above-described problems, and records a video conference executed by a video conference system so that the reaction of another user with respect to one user's speech is reproduced in synchronization. An object of the present invention is to provide a possible conference recording device, conference recording method, and conference recording program.

  In order to solve the above-described problem, a conference recording apparatus according to a first aspect of the present invention is a conference terminal that acquires an image and sound of a user participating in a remote conference and generates moving image data including the user's image and sound. Are installed at a plurality of locations where the user exists, and each of the conference terminals exchanges the moving image data with each other via a network, and outputs an image and a sound synthesized based on the moving image data. A conference recording device that is used in a remote conference system to create conference record data recording the remote conference, and exchanges and exchanges the video data between the conference terminals; The moving image data acquired by the moving image data exchanging means is used as a reproduction type of the moving image data when reproducing the conference recording data. Based on a delay time required for the video data acquired by the video data exchanging means to be transmitted between the conference terminals via the network. The delay time correcting means for adjusting the reproduction timing of the moving picture data by correcting the correspondence between the moving picture data stored in the moving picture storage means and the time information, and the moving picture data stored in the moving picture storage means By synthesizing in time series based on the time information, the one moving image data generated by one conference terminal and the other generated by the other conference terminal when receiving the one moving image data The conference record data creating means for creating the conference record data to be played back in synchronization with the moving image data, and the conference record data creating means And a conference recording data output means for outputting the conference record data created.

  According to a second aspect of the present invention, in the conference recording apparatus according to the first aspect, the delay time correcting means is based on an inquiry signal for measuring a transmission time of the network between the conference terminals. The delay time is specified, and the time information associated with the moving image data in the moving image storage means is delayed from the actual timing at which the moving image data is received based on the delay time. To do.

  According to a third aspect of the present invention, in addition to the configuration of the first aspect, the time information is a time stamp indicating a time when the moving image data is transmitted or received, and the delay time The correcting unit adjusts the reproduction timing of the moving image data based on a time stamp associated with the moving image data stored in the moving image storage unit.

  In addition to the configuration of the invention according to any one of claims 1 to 3, the conference recording device of the invention according to claim 4 includes reaction data including a predetermined reaction related to the user detected by the conference terminal, Reaction data exchanging means for transmitting and receiving and exchanging between the conference terminals via the network, and the reaction data acquired by the reaction data exchanging means received at the conference terminal when the reaction data was generated Reaction storage means for storing in association with the time information included in the moving image data, the conference recording data creating means is the moving image data stored in the moving image storage means and the reaction storage means stored in the reaction storage means By synthesizing the reaction data in time series based on the time information, the predetermined recorded data is synchronized with the moving image data when the conference recorded data is reproduced. The reaction is characterized in that to create the conference record data to be displayed.

  According to a fifth aspect of the present invention, in addition to the configuration of the fourth aspect of the present invention, each of the conference terminals is mounted on the user who participates in the remote conference using the conference terminal. Sensor means for detecting at least one of a whisper that is a positive head movement by the user and a head tilt that is a negative head movement by the user as the predetermined reaction; and the sensor means When the predetermined reaction is detected by the method, the apparatus includes reaction data generating means for generating the reaction data including the predetermined reaction detected by the sensor means.

  According to a sixth aspect of the present invention, in addition to the configuration of the fourth aspect of the present invention, each of the conference terminals images the user participating in the remote conference using the conference terminal. An imaging means and a neck which is a positive head movement by the user and a negative head movement by the user by analyzing the image of the user imaged by the imaging means Image analysis means for detecting at least one of the tilts as the predetermined reaction, and the reaction data including the predetermined reaction detected by the image analysis means when the predetermined analysis is detected by the image analysis means Reaction data generating means for generating.

  According to a seventh aspect of the present invention, in addition to the configuration of the fourth aspect of the present invention, each of the conference terminals further includes the video data received by the video data exchanging means. Analyzing the voice included in the data, it is provided with volume judgment means for judging whether or not the voice is higher than a predetermined volume, and the reaction data generating means is configured such that the voice is higher than the predetermined volume by the volume judgment means. If it is determined that the response data is generated, the reaction data is generated.

  In addition to the configuration of the invention according to any one of claims 4 to 7, the conference recording device of the invention according to claim 8 is characterized in that the conference record data creating means stores the reaction data stored in the reaction storage means. Based on the time information, among the moving image data stored in the moving image storage means, the moving image data that does not include the predetermined reaction is replaced with the moving image data that includes the predetermined reaction. Features.

  According to a ninth aspect of the present invention, there is provided a conference recording method in which a plurality of conference terminals that acquire images and sounds of users participating in a remote conference and generate moving image data including the images and sounds of the users exist. Used in a remote conference system that is installed at each site and in which each of the conference terminals exchanges the moving image data with each other via a network and outputs an image and a sound synthesized based on the moving image data. , A conference recording method for recording the remote conference, wherein the video data exchange step of exchanging and exchanging the video data between the conference terminals and the video data acquired by the video data exchange step, The moving image storage unit is associated with time information indicating the reproduction timing of the moving image data at the time of reproducing the conference recording data. Stored in the moving image storage means based on a delay time required for the moving image storage step to be stored and the moving image data acquired by the moving image data exchange step to be transmitted between the conference terminals via the network. A delay time correcting step for adjusting the reproduction timing of the moving image data by correcting the correspondence between the moving image data and the time information, and the moving image data stored in the moving image storage means based on the time information The one video data generated at one conference terminal and the other video data generated at another conference terminal when receiving the one video data are synchronized with each other. Created by the conference record data creation step for creating the conference record data to be played back and the conference record data creation step The and a conference recording data output step of the outputting of the conference record data.

  According to a tenth aspect of the present invention, there is provided a conference recording program in which a plurality of conference terminals that acquire images and sounds of users participating in a remote conference and generate moving image data including the images and sounds of the users exist. Used in a remote conference system that is installed at each site and in which each of the conference terminals exchanges the moving image data with each other via a network and outputs an image and a sound synthesized based on the moving image data A video recording data exchange means for exchanging the video data between the conference terminals by exchanging the video data between the conference terminals, the video data obtained by the video data exchanging means, Video storage means associated with time information indicating the playback timing of the video data during playback A moving image stored in the moving image storage unit based on a delay time required for the moving image data acquired by the storage execution unit to be stored and the moving image data exchange unit to be transmitted between the conference terminals via the network By correcting the correspondence between the data and the time information, delay time correcting means for adjusting the reproduction timing of the moving picture data, and the moving picture data stored in the moving picture storage means are synthesized in time series based on the time information. Thus, the one moving image data generated at one of the conference terminals and the other moving image data generated at the other conference terminal when the one moving image data is received are reproduced in synchronization. Meeting record data creating means for creating the meeting record data, and outputting the meeting record data created by the meeting record data creating means. Characterized in that to function as conference record data output means for.

  In the conference recording apparatus according to the first aspect of the present invention, the moving image data exchanged between the conference terminals is stored in association with time information indicating the reproduction timing of the moving image data at the time of reproducing the conference recording data. The time information associated with the moving image data is adjusted based on the delay time required for transmission via the network. By synthesizing video data in time series based on time information, one video data generated at one conference terminal and another video generated at another conference terminal when receiving one video data Conference recording data is created that is played back in synchronization with the data. Thereby, when meeting record data is reproduced | regenerated, the reaction of the other user with respect to one user's utterance can be output synchronizing.

  In the conference recording apparatus of the invention according to claim 2, the delay time is specified based on the inquiry signal for measuring the transmission time of the network, and the time information is more than the actual timing at which the moving image data is received based on the delay time. Delayed. Thereby, in addition to the effect of the invention of claim 1, the time information associated with the moving image data is delayed based on the transmission time of the network specified by the inquiry signal, so that one user's remark Other users' reactions can be accurately synchronized regardless of network delay.

  In the conference recording apparatus of the invention according to claim 3, the reproduction timing of the moving image data is adjusted based on the time stamp indicating the time when the moving image data is transmitted or received. Thus, in addition to the effect of the invention according to claim 1, since the time at which the moving image data is transmitted or received is specified based on the time stamp, the response of one user to another user's response to the network delay It can be accurately synchronized regardless.

  In the conference recording apparatus of the invention according to claim 4, a predetermined response is displayed in synchronization with the moving image data when the conference recording data is reproduced based on the response data including the predetermined response regarding the user detected by the conference terminal. Meeting record data is created. Thereby, in addition to the effect of the invention according to any one of claims 1 to 3, it is possible to accurately discriminate the reaction of another user to one user's utterance at the time of reproducing the conference record data.

  In the conference recording apparatus of the invention according to claim 5, at each conference terminal, at least one of beating and tilting is detected as a predetermined response by the sensor means attached to the user, and the response is detected when the predetermined response is detected. Data is generated. Thereby, in addition to the effect of the invention of claim 4, it is possible to display whether the user's reaction is positive or negative at the time of reproducing the conference record data.

  In the conference recording apparatus of the invention according to claim 6, at each conference terminal, at least one of whirling and tilting is detected as a predetermined reaction by image analysis, and reaction data is generated when the predetermined reaction is detected. . Thereby, in addition to the effect of the invention of claim 4, it is possible to display whether the user's reaction is positive or negative at the time of reproducing the conference record data.

  In the conference recording apparatus according to the seventh aspect of the present invention, the audio included in the moving image data received by each conference terminal is analyzed, and reaction data is generated when the audio is higher than a predetermined volume. Thereby, in addition to the effect of the invention according to any one of claims 4 to 6, it is possible to prevent a problem that reaction data is generated for another user even though one user does not speak. be able to.

  In the conference recording apparatus according to the eighth aspect of the present invention, the moving image data not including the predetermined reaction is combined with the moving image data including the predetermined reaction based on the reaction data when the conference recording data is created. Thereby, in addition to the effect of the invention according to any one of claims 4 to 7, by simply replacing a part of the moving image data in the conference record data, the reaction of the other user to the utterance of one user is accurately synchronized. be able to.

  In the conference recording method according to the ninth aspect of the present invention, the moving image data exchanged between the conference terminals is stored in association with the time information indicating the reproduction timing of the moving image data when reproducing the conference recording data. The time information associated with the moving image data is adjusted based on the delay time required for transmission via the network. By synthesizing video data in time series based on time information, one video data generated at one conference terminal and another video generated at another conference terminal when receiving one video data Conference recording data is created that is played back in synchronization with the data. Thereby, when meeting record data is reproduced | regenerated, the reaction of the other user with respect to one user's utterance can be output synchronizing.

  In the conference recording program according to the tenth aspect, the moving image data exchanged between the conference terminals is stored in association with the time information indicating the reproduction timing of the moving image data at the time of reproducing the conference recording data. The time information associated with the moving image data is adjusted based on the delay time required for transmission via the network. By synthesizing video data in time series based on time information, one video data generated at one conference terminal and another video generated at another conference terminal when receiving one video data Conference recording data is created that is played back in synchronization with the data. Thereby, when meeting record data is reproduced | regenerated, the reaction of the other user with respect to one user's utterance can be output synchronizing.

It is a figure showing the whole video conference system 1 composition concerning a 1st embodiment. 3 is a block diagram showing an electrical configuration of a terminal device 3. FIG. 2 is a diagram illustrating a memory configuration of an HDD 31. FIG. 10 is a flowchart of a delay calculation process executed by the terminal device 3. It is a figure which shows the data transmission / reception between the terminal devices 3, 4, and 5 in time series. It is a figure which shows one specific example of the video conference screen. 10 is a flowchart of a conference recording process executed by the terminal device 3. It is a figure which shows one specific example of the meeting reproduction | regeneration screen 290. FIG. It is a figure which shows the whole structure of the video conference system 1 which concerns on 2nd Embodiment. It is a figure which shows the whole structure of MUC7. 2 is a diagram illustrating a memory configuration of an HDD 51. FIG. It is a figure which shows the data transmission / reception between the terminal devices 3, 4, 5 and MUC7 in time series. 4 is a flowchart of a reaction data transmission process executed by the terminal device 3. It is a flowchart of the meeting recording process performed by MUC7. It is a flowchart of the reaction display processing process performed by MUC7. It is a flowchart of the reaction display addition process performed by MUC7. 5 is a diagram illustrating a specific example of a conference reproduction screen 300. FIG. It is a flowchart of the delay calculation process which concerns on a modification. It is a block diagram which shows the electric constitution of the terminal device 3 which concerns on a modification. It is a flowchart of the reaction data transmission process which concerns on a modification.

  DESCRIPTION OF EXEMPLARY EMBODIMENTS Hereinafter, embodiments of the invention will be described with reference to the drawings. The drawings to be referred to are used for explaining the technical features that can be adopted by the present invention, and the configuration of the apparatus and the flowcharts of various processes described are not intended to be limited thereto. This is just an illustrative example.

<First Embodiment>
With reference to FIG. 1, the overall configuration of the video conference system 1 according to the first embodiment will be described. As shown in FIG. 1, in the video conference system 1 according to the first embodiment, a plurality of terminal devices respectively provided at a plurality of locations where a user exists are connected to a network 2. Although FIG. 1 shows a case where the video conference system 1 is provided with three terminal devices 3, 4, and 5, the video conference system 1 only needs to be provided with a plurality of terminal devices.

  In the video conference system 1 according to the first embodiment, video and audio acquired by the terminal devices 3, 4, and 5 provided at each site are transmitted and received via the network 2. In each terminal device 3, 4, 5, conference data obtained by synthesizing video and audio acquired at each base is created and output. As a result, in each of the terminal devices 3, 4, and 5, the video and audio of the user existing at each base are synthesized and output in real time, and a remote conference (here, a video conference) is performed. In each of the terminal devices 3, 4, and 5, a video conference executed by the video conference system 1 is recorded as conference record data that can be reproduced after the video conference ends. Below, the case where the video conference performed by the video conference system 1 is recorded by the terminal device 3 is demonstrated and demonstrated.

  With reference to FIG. 2 and FIG. 3, the electrical configuration of the terminal device 3 will be described. Since all the terminal devices 3, 4, and 5 have the same configuration, only the configuration of the terminal device 3 will be described here, and the description of the other terminal devices 4, 5 will be omitted.

  The terminal device 3 is provided with a CPU 20 as a controller that controls the terminal device 3. Connected to the CPU 20 are a ROM 21 that stores BIOS, a RAM 22 that temporarily stores various data, and an I / O interface 30 that mediates data transfer. The I / O interface 30 is connected to a hard disk drive 31 (hereinafter referred to as HDD 31) having various storage areas.

  The I / O interface 30 includes a communication device 25 for communication connection to the network 2, a mouse 27, a video controller 23, a key controller 24, a camera 34 for photographing a user, and a user's voice. A microphone 35 for capturing, a speaker 36 for outputting sound, and a CD-ROM drive 26 are connected to each other. A display 28 is connected to the video controller 23. A keyboard 29 is connected to the key controller 24.

  As shown in FIG. 3, the HDD 31 includes a moving image data storage area 31a for storing moving image data generated by the terminal devices 3, 4, and 5, and a recording delay time storage area 31b for storing a recording delay time described later. A conference record data storage area 31c for storing conference record data recording a video conference, a program storage area 31d for storing various programs executed by the terminal device 3, and another information storage area 31e are provided. Yes. The conference recording data stored in the conference recording data storage area 31c is created based on a plurality of moving image data stored in the moving image data storage area 31a. Details will be described later.

  The program storage area 31d stores a conference execution program for executing a video conference and a conference recording program for recording the video conference. The CD-ROM 114 inserted into the CD-ROM drive 26 stores the above conference execution program and conference recording program. In the terminal device 3, by reading the CD-ROM 114 from the CD-ROM drive 26, these programs and data can be set up in the HDD 31 and stored in the program storage area 31d.

  With reference to FIGS. 4-8, the process regarding the recording of the video conference in the video conference system 1 which concerns on 1st Embodiment is demonstrated. Various processes described below are executed by the CPU 20 based on a conference recording program stored in the HDD 31. Here, the processing executed by the terminal device 3 will be described, but the same applies to the case where the video conference is recorded by the other terminal devices 4 and 5.

  With reference to FIG. 4, the delay calculation process which measures the network delay between each base in the terminal device 3 is demonstrated. The delay calculation process (FIG. 4) may be executed at an arbitrary timing before the video conference is started in the video conference system 1. In the terminal device 3 according to the first embodiment, the delay calculation process (FIG. 4) is executed at the timing when the start of the video conference is instructed, and after the delay calculation process (FIG. 4) ends, each terminal device 3, Assume that a video conference starts between 4 and 5.

  As shown in FIG. 4, in the delay calculation process, first, the delay time with each site is measured with reference to the own site (S1). Specifically, the data transmission time (that is, network delay) via the network 2 between the terminal device 3 at the local site and the terminal devices 3, 4, and 5 at each site is calculated for each other site. The The network delay between the bases may be measured by a known method. For example, the network delay is calculated by using PING (Packet Internet Grouper).

  Specifically, as shown in FIG. 5, the PING transmitted from the terminal device 3 at the timing of time t1 reaches the terminal device 4 at the timing of time t2, and the response from the terminal device 4 is the timing of time t3. It is assumed that the terminal device 3 has been reached. In this case, a time difference obtained by subtracting time t1 from time t3 is calculated as a delay time Δt12 with respect to the base 2 with respect to the base 1. Also, it is assumed that the PING transmitted from the terminal device 3 at the time t1 arrives at the terminal device 5 at the time t4 and the response from the terminal device 5 reaches the terminal device 3 at the time t5. . In this case, a time difference obtained by subtracting the time t1 from the time t5 is calculated as a delay time Δt13 with respect to the base 3 with the base 1 as a reference. Since the base 1 is its own base, the delay time Δt11 with respect to the base 1 is set to “0”.

  Based on the delay time with each base calculated in S1, the recording delay time for each base is calculated (S3). The recording delay time is data for adjusting the recording position of the moving image data generated at each site in the meeting recording data described later. In S3, the recording delay time for each site is calculated by subtracting the delay time for each site from the longest delay time among the delay times acquired in S1.

  Specifically, in the example of FIG. 5, the longest delay time among the delay times Δt11, Δt12, Δt13 with respect to each of the bases 1 to 3 is the delay time Δt13 with respect to the base 3. Therefore, the recording delay time of the base 2 is “Δt13−Δt12”. The recording delay time of the base 3 is “Δt13−Δt13”, that is, “0”. The recording delay time of the site 1 is “Δt13−Δt11”, that is, “Δ13”. The recording delay time for each base calculated in S3 is stored in the recording delay time storage area 31b of the HDD 31.

  When the recording delay time is calculated and stored for all the bases, the delay calculation process (FIG. 4) ends. Then, a video conference is executed between the bases (that is, the terminal devices 3, 4, 5) via the network 2. In the terminal device 3, the video conference is executed by the CPU 20 by a known method based on the conference execution program stored in the HDD 31. For example, in the terminal device 3, a video in a specified time unit (for example, every second) captured by the camera 34 and a sound in a specified time unit (for example, every second) acquired by the microphone 35 are It is generated as moving image data of the base 1. The moving image data of the site 1 is transmitted to another site via the network 2. Similarly, in the terminal devices 4 and 5, moving image data of the bases 2 and 3 are generated and transmitted to other bases via the network 2. In the terminal device 3, conference data is created by combining the moving image data generated at the local site 1 and the moving image data received from the other sites 2 and 3, and the conference data is output from the display 28 and the speaker 36. Similarly, the terminal devices 4 and 5 also generate and output conference data obtained by synthesizing the moving image data generated at each site and the moving image data received from another site.

  For example, in a video conference executed by the video conference system 1, a video conference screen 280 as shown in FIG. 6 is displayed on the display 28 based on conference data created by the terminal device 3 at the site 1. On the video conference screen 280 (FIG. 6), in addition to the user display area 281 where the video of the user at the local site 1 is displayed, the video of the users at the other bases 2 and 3 are displayed in the user display areas 282 and 283, respectively. . In addition, in synchronism with the video conference screen 280, the voices of the users at the respective sites 1 to 3 are output from the speaker 36. Similarly, at the bases 2 and 3, the video and audio of the users at the respective bases 1 to 3 are synthesized and output based on the conference data created by the terminal devices 4 and 5.

  However, the conference data created by each of the terminal devices 3, 4, and 5 is a composite of the moving image data generated at its own site and the moving image data received from another site in real time. On the other hand, from the time when the video data of the local site is transmitted, until the video data of the local site reaches the other site, and the video data of the other site generated at that time reaches the local site. As described above, the transmission time in the network 2 is required. For this reason, in an actual video conference, for example, a user's reaction at another site to a user's speech at the local site is output to the video conference after a time when the user actually reacted at the other site.

  In the example shown in FIG. 6, when the above-mentioned video conference is performed, the user's reaction at the other sites 2 and 3 responds to the remark “Do you agree with my opinion?” Not output due to network delay. The response of the user at the site 1 is output when the above-mentioned delay time Δ12 has passed, and the response of the user at the site 3 is output in response to the above-mentioned delay time. It is the time when Δ13 has elapsed. As described above, in the video conference, the video and audio of the user at each site are synthesized and output in real time, but in reality, the output timing may be shifted due to the network delay with each site. Then, when the conference record data is created by recording the video conference as it is, even if the conference record data is reproduced, it is difficult for the user at the local site to understand the reaction of the user at the other site to the utterance. In the first embodiment, by executing the following processing in the terminal device 3 that records a video conference, the user's speech at the local site and the user's reaction at the other site are synchronized when the recorded conference data is reproduced. Output is enabled.

  With reference to FIG. 7, the meeting recording process for recording a video conference in the terminal device 3 is demonstrated. The conference recording process (FIG. 7) is started and executed by the CPU 20 when a video conference in which the user of the base 1 (terminal device 3) participates is started.

  As shown in FIG. 7, in the conference recording process, moving image data is first acquired from each base (S11). In S <b> 11, the moving image data generated at the local site during the video conference and the moving image data received from another site are temporarily stored in the RAM 22. Note that the moving image data includes identification data indicating the generation source site in addition to video and audio in a predetermined time unit acquired at the site (generation source site) where the moving image data was generated.

  The moving image data temporarily stored in the RAM 22 is buffered according to the recording delay time for each site (S13). In S13, the recording delay time of the generation source base is acquired from the recording delay time storage area 31b with reference to the generation base indicated in the moving image data acquired in S11. The moving image data temporarily stored in the RAM 22 is buffered for the recording delay time of the generation base. The buffering for the recording delay time may be executed by a known delay circuit or delay program.

  After buffering for the recording delay time, the moving image data temporarily stored in the RAM 22 is recorded in the moving image data storage area 31a of the HDD 31 (S15). In the moving image data storage area 31a, moving image data is stored in association with time data indicating the actual time when the buffering is finished. When the moving image data is saved in the moving image data storage area 31a, the moving image data temporarily stored in the RAM 22 is deleted.

  Then, it is determined whether or not the video conference is ended (S17). For example, when the terminal device 3 is instructed to end the video conference (S17: YES), a plurality of video data stored in the video data storage area 31a is synthesized and conference recording data in which the video conference is recorded. Is created (S19). On the other hand, when there is no instruction to end the video conference (S17: NO), the acquisition, buffering, and recording of the moving image data are continued (S11 to S15).

  In S19, the conference record data in which the video conference is recorded by synthesizing the plurality of movie data stored in the movie data storage area 31a in time series based on the time data associated with each movie data. Is created. Therefore, at the time of reproducing the conference recording data, the moving image data having the same time data associated with each moving image data is output at the same timing (that is, synchronously). When the conference record data created in S19 is stored in the conference record data storage area 31c of the HDD 31 (S21), the conference recording process (FIG. 7) ends.

  More specifically, in S13 of the conference recording process (FIG. 7), among the moving image data generated at each of the sites 1 to 3, the moving image data acquired from the own site 1 is buffered for “Δt13” seconds. Executed. Buffering of “Δt13−Δt12” seconds is performed on the moving image data acquired from the other site 2. Buffering of “0” seconds is performed on the moving image data acquired from the other base 3. That is, the buffering time in S13 becomes longer as the moving image data is generated at a base with less network delay (for example, own base 1). For this reason, in S15, time data having a larger delay width is associated with the moving image data generated at the base having a smaller network delay with reference to the actually generated timing. On the other hand, the buffering time in S13 becomes shorter as the moving image data is generated at a base with a large network delay (for example, another base 3). Therefore, in S15, time data having a smaller delay width is associated with the moving image data generated at the base having the larger network delay with reference to the actually generated timing. As a result, in the moving image data storage area 31a, the moving image data transmitted from the local site and the moving image data arrived on the basis of the time when the moving image data generated at the other site having the largest network delay is received at the local site. The same time data is given to the moving image data generated at the other site.

  Thereby, in S19 and S21, by synthesizing a plurality of moving image data stored in the moving image data storage area 31a along with the time data, the user's remarks at the local site 1 and the other sites 2 and 3 in the video conference. Thus, meeting record data is generated and stored so that the reaction of the user who receives the comment is output in synchronization with each other. Then, when the user reproduces the conference record data stored in the terminal device 3 at an arbitrary timing after the end of the video conference, a conference reproduction screen 290 as shown in FIG. 8 is displayed on the display 28, for example.

  The conference playback screen 290 (FIG. 8) is basically the same as the video conference screen 280 (FIG. 6) displayed in the video conference, but unlike the video conference, the output timing shift caused by the network delay is suppressed. Has been. In the example shown in FIG. 8, the user at the other site 3 shows a positive reaction that the user at the other site 2 whispered in synchronization with the statement “Do you agree with my opinion?” Has shown a negative reaction of tilting the head. In addition, in synchronization with the conference playback screen 290 (FIG. 8), each user's voice is output from the speaker 36 at the actual speech timing in the video conference.

  According to the video conference system 1 according to the first embodiment, the moving image data exchanged between the terminal devices 3, 4, and 5 is stored in the terminal device 3 in association with the time data. The time data associated with the moving image data is adjusted by buffering the recording delay time calculated based on the delay time required for transmission via the network. By synthesizing the moving image data in time series based on the time data, the moving image data generated by the terminal device 3 is synchronized with the moving image data generated by the terminal devices 4 and 5 when the moving image data is received. Meeting record data to be played back is created. Thereby, when meeting record data is reproduced | regenerated, the reaction of the other user with respect to one user's utterance can be output synchronizing.

  Further, the delay time is specified based on the inquiry signal (PING) for measuring the transmission time of the network, and the time data associated with the video data is delayed by buffering the recording delay time calculated based on the delay time. Is done. Therefore, it is possible to accurately synchronize the reactions of other users to one user's remarks regardless of network delay.

<Second Embodiment>
With reference to FIG. 9, the overall configuration of the video conference system 1 according to the second embodiment will be described. As shown in FIG. 9, in the video conference system 1 according to the second embodiment, a plurality of terminal devices provided at each site and a multi-site connection device 7 are connected to the network 2. Although FIG. 9 shows a case where the video conference system 1 is provided with three terminal devices 3, 4, and 5, the video conference system 1 only needs to be provided with a plurality of terminal devices.

  The multi-site connection apparatus 7 is an apparatus that realizes a video conference between multi-sites by being connected to user terminals provided at a plurality of bases via the network 2 and relaying video, audio, data, and the like. Hereinafter, the multipoint control unit (Multipoint Control Unit) 7 is abbreviated as MCU7.

  In the video conference system 1 according to the second embodiment, moving image data generated by the terminal devices 3, 4, and 5 provided at each base is transmitted to the MCU 7 via the network 2. In the MCU 7, conference data obtained by synthesizing the moving image data received from each base is created, and the conference data is transmitted to the terminal devices 3, 4, 5 via the network 2. In the terminal devices 3, 4, and 5, the conference data received from the MCU 7 is output. Thereby, in each terminal device 3, 4, and 5, the video and audio of the user existing at each base are synthesized and output in real time, and a remote conference (here, a video conference) is performed. In the MCU 7, the video conference executed in the video conference system 1 is recorded as conference record data that can be reproduced after the video conference is completed. Below, the case where the video conference performed by the video conference system 1 is recorded by MUC7 is demonstrated and illustrated.

  The terminal devices 3, 4, and 5 have the same configuration as that of the first embodiment (FIG. 3), but at least functions as a client that participates in a video conference executed by the MCU 7 (more specifically, the MCU 7 has video and A function of transmitting voice, a function of outputting conference data transmitted from the MCU 7, and the like. For this reason, the above-described conference execution program, conference recording program, and various storage areas (video data storage area 31a, recording delay time storage area 31b, and conference recording data storage area 31c) necessary for video conference recording are not provided. However, the program storage area 31d stores a reaction detection program for detecting the reaction of the user during the video conference.

  The electrical configuration of the MUC 7 will be described with reference to FIGS. 10 and 11. As shown in FIG. 10, the MUC 7 has substantially the same configuration as the terminal device 3 described above, and includes a CPU 40, a ROM 41, a RAM 42, an I / O interface 50, and an HDD 51. The I / O interface 50 includes a communication device 45 for communicating with the network 2, a mouse 47, a key controller 44 that receives key inputs from the keyboard 49, and a video controller 43 that controls display of the display 48. Each is connected.

  As shown in FIG. 11, the HDD 51 stores a moving image data storage area 51 a that stores moving image data generated by the terminal devices 3, 4, and 5, and reaction data that stores reaction data indicating the reaction of the user at each site. A storage area 51b, a conference recording data storage area 51c for storing conference recording data recording a video conference, a program storage area 51d for storing various programs executed in the MUC 7, and other information storage areas 51e are provided. It has been. The conference recording data stored in the conference recording data storage area 51c is created based on the moving image data stored in the moving image data storage area 51a and the reaction data stored in the reaction data storage area 51b. Will be described later. The program storage area 51d stores a conference execution program for executing a video conference and a conference recording program for recording the video conference.

  By the way, in the video conference system 1 according to the second embodiment, the conference data created by the MUC 7 is obtained by combining the moving image data received from each base in real time. On the other hand, when a video conference is executed with MUC7, the conference data including the video data arrives at another base from the point when MUC7 receives the video data of one base and is generated at the time of arrival. It takes a transmission time in the network 2 until the MUC 7 receives the video data of another base. For this reason, in an actual video conference, for example, a user's reaction from another site to a user's statement at one site is output to the video conference later than when the user actually reacted at the other site. .

  Specifically, as shown in FIG. 12, the moving image data transmitted from the terminal device 3 at the timing of time t11 arrives at the MUC 7 at the timing of time t12. The MUC 7 synthesizes a plurality of video data received at the timing of time t12 to create conference data, and transmits the conference data to each of the terminal devices 3, 4, and 5. The conference data arrives at the terminal device 4 at the timing of time t13, and the moving image data returned from the terminal device 4 reaches the MCU 7 at the timing of time t14. In this case, the time difference obtained by subtracting the time t12 from the time t14 becomes, for example, the delay time Δt21 necessary for the MUC 7 to acquire the reaction of the user at the base 2 with respect to the user's utterance at the base 1. The conference data arrives at the terminal device 5 at the timing of time t15, and the moving image data returned from the terminal device 5 reaches the MCU 7 at the timing of time t16. In this case, the time difference obtained by subtracting the time t12 from the time t16 becomes, for example, the delay time Δt22 necessary for the MUC 7 to acquire the reaction of the user at the base 3 with respect to the user's utterance at the base 1.

  With reference to FIGS. 13-17, the process regarding the video conference recording in the video conference system 1 which concerns on 2nd Embodiment is demonstrated. Below, the process performed by the terminal device 3 and the process performed by MUC7 are demonstrated separately among the processes regarding the recording of a video conference. In addition, although the process executed by the terminal device 3 will be described here, the process executed by the other terminal devices 4 and 5 is the same.

  With reference to FIG. 13, the reaction data transmission process performed in the terminal device 3 is demonstrated. The reaction data transmission process (FIG. 13) is a process for returning a user reaction at the time of outputting the conference data acquired from the MUC 7 to the MUC 7 as reaction data. The reaction data transmission process (FIG. 13) is executed by the CPU 20 based on the above-described reaction detection program when a video conference in which the user of the base 1 (terminal device 3) participates is started.

  As shown in FIG. 13, in the reaction data transmission process, first, conference data transmitted from the MCU 7 is received (S51). As described above, during the video conference, video data including video and audio acquired by each terminal device 3, 4, and 5 is transmitted to MUC 7, and conference data obtained by synthesizing these video data is created in MUC 7. Is done. The conference data transmitted from the MUC 7 is output from each terminal device 3, 4, 5. In S51, the conference data transmitted from the MUC 7 to the terminal device 3 is received.

  A reaction time stamp is extracted from the conference data received in S51 (S53). The reaction time stamp indicates the actual time when the conference data is transmitted from the MUC 7, and will be described in detail later. Then, the reaction of the user who uses the terminal device 3 is detected (S55). In S55, the user's reaction included in the image is detected by analyzing the image taken by the camera 34. As an example, when detecting “whit” or “head tilt” as a user's reaction, the shake of the user's head is detected by well-known image processing. “Swing” refers to a state in which the listener's head swings more than a predetermined amount in the vertical direction when the listener is satisfied with the content the speaker is speaking. “Neck tilt” refers to a state in which the listener's head shakes more than a predetermined amount in the left-right direction when the listener is not satisfied with the content the speaker is speaking. These reactions can be detected by an identification method using a state identification device described in, for example, Japanese Patent Application Laid-Open No. 2007-97668.

  When a user reaction is detected in S55 (S57: YES), reaction data relating to the user reaction is generated (S59). The reaction data includes the reaction time stamp extracted in S53, the type of reaction detected in S55 (for example, whispering or tilting the head), and identification data indicating its own base. The reaction data generated in S59 is returned to the MCU 7 via the network 2 (S61). If no user reaction is detected in S55 (S57: NO), S37 and S39 are skipped, and the reaction time stamp extracted in S53 is discarded. Then, it is determined whether or not the video conference is terminated (S63). For example, when a video conference end instruction is issued from the MUC 7 (S63: YES), the reaction data transmission process (FIG. 13) ends. If there is no instruction to end the video conference (S63: NO), the process returns to S51.

  With reference to FIG. 14, the meeting recording process which records a video conference in MUC7 is demonstrated. A process related to the recording of the video conference executed by the MCU 7 will be described. When the video conference is started in the video conference system 1, the conference recording process (FIG. 14) is executed by the CPU 40 based on the above-described conference recording program.

  As shown in FIG. 14, in the conference recording process for recording a video conference in MUC7, when data is first received from any one of terminal devices 3, 4, and 5 (S101: YES), the received data in S101 is a moving image. It is determined whether it is data (S103). When the received data is moving image data (S103: YES), the moving image data is stored in the moving image data storage area 51a in association with the recording time stamp (S105). The recording time stamp associated with the moving image data in S105 is information indicating the actual time when the moving image data is received by the MCU 7.

  On the other hand, in the MUC 7, as described above, the moving image data received from the terminal devices 3, 4, and 5 at the same timing is synthesized in real time, and the conference data is transmitted to the terminal devices 3, 4, and 5. At this time, the conference data including the moving image data stored in S105 is transmitted to each of the terminal devices 3, 4, and 5 after being given a reaction time stamp (S107). The reaction time stamp is information indicating the actual time when the conference data is transmitted from the MUC 7. In the MUC 7 according to the second embodiment, since there is almost no time lag between the time when the moving image data is received and the time when the conference data is transmitted, the reaction time stamp assigned in S107 is associated in S105. Indicates the same time as the recording time stamp. As a result, each terminal device 3, 4, 5 acquires a reaction time stamp indicating the same timing as the recording time stamp managed by the MUC 7 in S 53 described above.

  If the received data in S101 is not moving image data (S103: NO), it is determined whether the received data in S101 is reaction data (S109). When the received data is reaction data (S109: YES), the reaction data is stored in the reaction data storage area 51b in association with the recording time stamp and the reaction time stamp (S111). The recording time stamp associated with the reaction data in S111 indicates the actual time when the reaction data was received by the MCU 7. Further, the reaction time stamp associated with the reaction data in S111 is a reaction time stamp (see S53) extracted at the transmission source site (that is, the reaction site).

  After S107 or S111 is executed, or when the received data is not reaction data (S109: NO), it is determined whether or not the video conference is terminated (S113). For example, when an instruction to end a video conference is given by each terminal device 3, 4, 5, (S113: YES), a plurality of video data stored in the video data storage area 51a are combined to perform the video conference. The recorded conference record data is created (S115). The conference record data created in S115 is stored in the conference record data storage area 51c of the HDD 51 (S117). At this time, in the conference record data storage area 51c, a position (display position for each site) where the moving image data acquired from each site when the conference record data is generated is stored in association with the conference record data. If there is no instruction to end the video conference (S113: NO), the process returns to S101.

  In S115, a conference in which a video conference is recorded by synthesizing a plurality of movie data stored in the movie data storage area 51a in time series based on a recording time stamp associated with each movie data. Recorded data is created. Therefore, at the time of reproducing the conference recording data, the moving image data whose recording time stamps corresponding to the respective moving image data match is output at the same timing (that is, synchronously). However, as described above, the recording time stamp is time data given to the moving image data with reference to the time of reception at the MUC 7. Therefore, due to network delay between the MUC 7 and each base, each moving picture data may be output at a timing different from the time when it is actually generated in the video conference.

  For example, at the time of reproducing the conference record data recorded in the video conference as shown in FIG. 6, in response to the remark “Do you agree with my opinion?” User response is not output at the same timing. The response of the user at the site 1 is output at the time when the above-described delay time Δ22 has passed, and the response of the user at the site 3 is output for the above-described delay time. It is the time when Δ23 has elapsed. As described above, in the conference recording data created based on the recording time stamp based on the reception time of the MUC 7, the video and audio of the user at each site are synthesized and output. The playback timing may be shifted due to the network delay. Then, even if the meeting record data is reproduced, it is difficult for the user at one site to understand the reaction of the user at another site to the speech. In the second embodiment, by executing the following reaction display processing (S119) in the MUC 7, when the conference record data is played back, the user at one site speaks and the user's reaction at another site responds to the statement. Can be output synchronously.

  As shown in FIG. 15, in the reaction display processing (S119), it is first determined whether or not unprocessed reaction data exists in the reaction data storage area 51b (S121). When unprocessed reaction data exists (S121: YES), a time stamp is extracted from the reaction data (S123). In S123, two time stamps (recording time stamp and reaction time stamp) associated with each reaction data are acquired in the reaction data storage area 51b. And the reaction display addition process which adds a user's reaction to meeting record data is performed (S125).

  As shown in FIG. 16, in the response display addition process (S125), first, moving image data at the time when the user shows a response is cut out from the meeting record data (S131). In S131, moving image data that matches the recording time stamp of the reaction data extracted in S123 is cut out from the conference recording data stored in the conference recording data storage area 51c. Each moving image data combined with the conference record data occupies a reproduction time of a predetermined time unit (for example, every second) for each base in the conference record data. Here, among the moving image data included in the conference record data, the moving image data common to the reaction data extracted in S123 and the generation base (reaction base) is based on the identification data indicating each base. Cut out from another display position.

  Next, among the moving image data included in the conference record data, the moving image data at the time of conference data transmission is replaced with the moving image data cut out in S131 (S133). In S133, the video data corresponding to the recording time stamp that matches the reaction time stamp of the reaction data extracted in S123 in the conference recording data stored in the conference recording data storage area 51c is cut out in S131. Replaced with video data. Thereby, the user's utterance at one site and the user's reaction at another site with respect to the utterance are reproduced in synchronization with the reproduction of the conference record data. Further, in S133, the replaced moving image data included in the conference record data is processed so that the user's reaction indicated in the response data is displayed with characters or symbols.

  Finally, the stop image immediately before the moving image data is cut out is inserted into the portion of the meeting record data where the moving image data is cut out in S131 (S135). That is, in the conference record data stored in the conference record data storage area 51c, the portion where the moving image data is cut out in S131 is in a state of no data, so that the cut-out portion is not displayed when the conference record data is reproduced. Is done. Therefore, in S135, the image displayed immediately before the cutout portion when the conference record data is reproduced is inserted as a stop image displayed over the cutout portion. Thereby, generation | occurrence | production of the blank accompanying cutout of the moving image data from meeting recording data can be prevented. After S135 is executed, the process returns to the reaction display processing (FIG. 15).

  Returning to FIG. 15, after the reaction display addition process (S125) is executed, the process returns to S121 to determine whether or not unprocessed reaction data exists. That is, S123 and S125 are repeatedly executed for each unprocessed reaction data until there is no unprocessed reaction data in the reaction data storage area 51b. If there is no unprocessed reaction data (S121: NO), the processed conference record data is stored in the conference record data storage area 51c (S127), and the process returns to the conference recording process (FIG. 14). The meeting recording process (FIG. 14) ends after the reaction display processing process (S119) is executed.

  Then, at an arbitrary timing after the end of the video conference, the user of each terminal device 3, 4, 5 acquires conference record data stored in the MUC 7 via the network 2. Specifically, when the user of the terminal device 3 reproduces the conference record data acquired from the MUC 7, a conference reproduction screen 300 as shown in FIG. 17 is displayed on the display 28, for example. The conference playback screen 300 (FIG. 17) is basically the same as the video conference screen 280 (FIG. 6) displayed in the video conference, but unlike the video conference, the output timing shift caused by the network delay is suppressed. Has been.

  In the example shown in FIG. 17, the user at the site 1 shows a positive response that the user at the site 2 whispered in synchronization with the statement “Do you agree with my opinion?” It is displayed that it showed a negative reaction. Further, the response of the user at the site 2 is displayed in a pop-up “with tilt”, and the response of the user at the site 3 is displayed in a pop-up “with tilt”. In addition, in synchronism with the conference playback screen 300 (FIG. 17), each user's voice is output from the speaker 36 at the actual speech timing in the video conference. Instead of the pop-up, a video that is registered in advance according to the user's reaction, or a graph indicating the strength or history of the user's reaction may be displayed.

  According to the video conference system 1 according to the second embodiment, moving image data exchanged between the terminal devices 3, 4, and 5 is stored in the MUC 7 in association with the recording time stamp. The recording time stamp associated with the moving image data is given according to the timing at which the conference data is transmitted. By synthesizing the moving image data in time series based on the recording time stamp, conference record data recording a video conference is created. When creating the meeting record data, based on the recording time stamp and reaction time stamp included in the reaction data, the moving image data not including the user's reaction is replaced with the moving image data including the predetermined reaction. Is done. Thereby, it is possible to synchronize the reactions of other users with respect to one user's remarks regardless of network delays by simply replacing a part of the moving image data in the conference record data.

  Further, in the MCU 7, the conference record in which the user's reaction is displayed in synchronization with the moving image data when the conference record data is reproduced based on the reaction data indicating the user's reaction detected by the terminal devices 3, 4, and 5. Data is created. Thereby, at the time of reproduction | regeneration of meeting recording data, the reaction of the other user with respect to one user's statement can be discriminate | determined correctly. Further, it can be displayed whether the user's reaction is positive (whispering or the like) or negative (head tilting or the like).

  By the way, in the said embodiment, the video conference system 1 is equivalent to the "remote conference system" of this invention, the terminal devices 3, 4, and 5 are equivalent to a "conference terminal", and the terminal device 3 and MUC7 are " It corresponds to “conference recording device”. The CPU 20 that executes S11 and the like and the CPU 40 that executes S101 and the like correspond to the “moving image data exchanging means” of the present invention, respectively. The moving image data storage area 31a and the moving image data storage area 51a correspond to the “moving image storage means” of the present invention. The CPU 20 that executes S1, S3, and S13 and the CPU 40 that executes S125 correspond to the “delay time correcting means” of the present invention. The CPU 20 that executes S19 and the CPU 40 that executes S115 and S125 respectively correspond to the “conference recording data creation means” of the present invention. The CPU 20 that executes S21 and the CPU 40 that executes S117 and S127 correspond to the “conference recording data output means” of the present invention. The CPU 40 that executes S101 and the like corresponds to the “reaction data exchanging means” of the present invention. The reaction data storage area 51b corresponds to the “reaction storage means” of the present invention. The camera 34 corresponds to the “imaging means” of the present invention. The CPU 20 that executes S55 corresponds to the “image analysis means” of the present invention. The CPU 20 that executes S59 corresponds to the “reaction data generation means” of the present invention.

  S11 and S101 correspond to the “moving image data exchange step” of the present invention. S15 and S105 correspond to the “moving image storing step” of the present invention. 1, S3, S13, and S125 correspond to the “delay time correction step” of the present invention. S19, S115, and S125 correspond to the “conference record data creation step” of the present invention. S21, S117, and S127 correspond to the “conference record data output step” of the present invention, respectively. The conference recording program that causes the CPU 20 to execute the delay calculation process (FIG. 4) and the conference recording process (FIG. 7) and the conference recording program that causes the CPU 40 to execute the meeting recording process (FIG. 14) Corresponds to “recording program”.

  In addition, this invention is not limited to the said embodiment, The change in the range which does not change the summary of invention is possible. Hereinafter, a modification of the present invention will be described with reference to FIGS.

  For example, in the first embodiment, the delay calculation process shown in FIG. 18 may be executed instead of the delay calculation process (FIG. 4) described above. The delay calculation process shown in FIG. 18 is executed when a video conference is started, and first, the delay time with each base is measured as described above (S1). The delay time with each base measured in S1 is stored as a history in the other information storage area 31e of the HDD 31. Then, the latest delay time measured in S1 is compared with the previously measured delay time for each base, and it is determined whether there is a base that has changed by a predetermined threshold (for example, 1 second) (S2). ). Immediately after the video conference is started, it is determined that the delay time has changed by more than a threshold value for all bases (S2: YES). Then, as described above, the recording delay time for each site is calculated and stored in the recording delay time storage area 31b of the HDD 31 (S3). Thereafter, when an instruction to end the video conference is given (S5: YES), the delay calculation process (FIG. 18) ends.

  On the other hand, if there is no instruction to end the video conference (S5: NO), the process returns to S1 after waiting for a predetermined time (for example, 5 minutes). In S1, the delay time with each base is measured, and in S2, it is determined whether or not there is a base where the latest delay time has changed by a threshold value or more than the previous delay time. Here, if there is a base whose delay time has changed by more than the threshold value (S2: YES), it means that the network delay situation for that base has changed greatly. Therefore, the recording delay time is calculated for the site whose delay time has changed by more than the threshold value, and the recording delay time of the site stored in the recording delay time storage area 31b is updated (S3). On the other hand, if there is no site where the delay time has changed by more than the threshold (S2: NO), the record delay time of each site is not updated because the change in the network delay status is less than that at the time of the previous delay time measurement. Proceed to S5.

  According to the delay calculation process shown in FIG. 18, even if the network delay state with each base changes greatly during the video conference, the recording delay time is updated accordingly. Therefore, it is possible to accurately synchronize the reactions of other users with respect to one user's remarks without being affected by changes in the network delay status during the Levi conference.

  In the first embodiment, instead of PING as an inquiry signal, the above-described time stamp may be transmitted and received to measure the delay time with each base. In this case, in the delay calculation process (FIG. 4), a time stamp indicating the time of transmission from the own site is transmitted to the other site, and the other site returns the time stamp to the own site. The own base may calculate a time difference between the time when the time stamp returned from the other base is received and the time indicated by the time stamp as a delay time with respect to the other base.

  In the second embodiment, instead of detecting the user's reaction by image analysis in each of the terminal devices 3, 4, and 5, the reaction may be detected by a sensor attached to the user. Specifically, in the terminal device 3 shown in FIG. 19, a known acceleration sensor 33 is connected to the I / O interface 30 via the sensor control unit 32. The acceleration sensor 33 is built in headphones (not shown), and the user wears the headphones during the video conference. Then, in the reaction data transmission process (FIG. 13), the user's reaction (eg, whispering or tilting the neck) is detected according to the detection signal from the acceleration sensor 33 (S55). As described above, various methods can be applied as a method for detecting a user reaction.

  In the second embodiment, the reaction data transmission process shown in FIG. 20 may be executed instead of the above-described reaction data transmission process (FIG. 13). The reaction data transmission process shown in FIG. 20 is the same as the above-described reaction data transmission process (FIG. 13). However, when the user has a reaction (S57: YES), the conference data output from the terminal device 3 is displayed. It is determined whether or not the sound based on the sound level is equal to or higher than a predetermined volume (S58). The predetermined volume may be, for example, an average volume of past user voices in a video conference. If the voice is equal to or higher than the predetermined volume (S58: YES), reaction data is generated (S59). If the voice does not reach the predetermined volume (S58: NO), S59 and S61 are skipped.

  According to the reaction data transmission process shown in FIG. 20, when the audio output based on the conference data is small at each terminal device 3, 4, 5, it is considered that the user does not speak in the video conference. It is. If a user reaction is detected in such a state, the user reaction is not related to another user's remarks in the video conference. Thereby, it is possible to prevent erroneous detection such that reaction data is generated in each of the terminal devices 3, 4, and 5 even though the user does not speak in the video conference.

1 video conference system 2 network 3 terminal device 4 terminal device 5 terminal device 7 MUC
20 CPU
21 ROM
22 RAM
31 HDD
31a Video data storage area 31b Recording delay time storage area 31c Conference recording data storage area 33 Acceleration sensor 34 Camera 35 Microphone 36 Speaker 40 CPU
41 ROM
42 RAM
51 HDD
51a Video data storage area 51b Reaction data storage area 51c Conference record data storage area 280 Video conference screen 290 Conference playback screen 300 Conference playback screen

Claims (10)

A conference terminal that acquires images and sounds of users participating in a remote conference and generates moving image data including the images and sounds of the users is installed at each of a plurality of locations where the users exist, and each of the conference terminals The video data is exchanged via a network and used in a remote conference system in which images and sounds synthesized based on the video data are output to create conference record data recording the remote conference A conference recording device that
Video data exchanging means for exchanging and exchanging the video data between the conference terminals;
Moving image storage means for storing the moving image data acquired by the moving image data exchanging means in association with time information indicating the reproduction timing of the moving image data at the time of reproduction of the conference record data;
Based on the delay time required for the moving image data acquired by the moving image data exchanging means to be transmitted between the conference terminals via the network, the moving image data stored in the moving image storage means and the time information A delay time correcting means for adjusting the reproduction timing of the moving image data by correcting the correspondence;
One moving image data generated by one conference terminal and one moving image data received by synthesizing the moving image data stored in the moving image storage means in time series based on the time information Meeting record data creating means for creating the meeting record data that is sometimes played back in synchronization with the other video data generated by the other conference terminal;
A conference recording apparatus comprising: conference record data output means for outputting the conference record data created by the conference record data creation means. The delay time correcting means includes
Identify the delay time based on an inquiry signal that measures the transmission time of the network between the conference terminals,
2. The time information associated with the moving image data in the moving image storage means is delayed from an actual timing at which the moving image data is received based on the delay time. Conference recording device. The time information is a time stamp indicating a time when the moving image data is transmitted or received,
2. The conference recording according to claim 1, wherein the delay time correcting unit adjusts the reproduction timing of the moving image data based on a time stamp associated with the moving image data stored in the moving image storage unit. apparatus. Reaction data exchange means for exchanging and exchanging reaction data including a predetermined reaction relating to the user detected at the conference terminal between the conference terminals via the network;
Reaction memory means for storing the reaction data acquired by the reaction data exchanging means in association with the time information included in the video data received when the reaction data is generated at the conference terminal,
The conference record data creating unit synthesizes the video data stored in the video storage unit and the reaction data stored in the reaction storage unit in time series based on the time information. The conference recording apparatus according to claim 1, wherein the conference recording data in which the predetermined reaction is displayed in synchronization with the moving image data during reproduction is generated. Each of the conference terminals
Worn by the user participating in the teleconference using the conference terminal, whispering a positive head movement by the user and tilting a negative head movement by the user Sensor means for detecting at least one of the predetermined reactions;
5. The reaction data generating means for generating the reaction data including the predetermined reaction detected by the sensor means when the predetermined reaction is detected by the sensor means. Conference recording device. Each of the conference terminals
Imaging means for imaging the user participating in the remote conference using the conference terminal;
By analyzing the image of the user imaged by the imaging means, at least one of a whisper that is a positive head movement by the user and a neck tilt that is a negative head movement by the user Image analysis means for detecting the predetermined reaction as
5. The reaction data generating means for generating the reaction data including the predetermined reaction detected by the image analyzing means when the predetermined reaction is detected by the image analyzing means. The meeting recording device described. Each of the conference terminals
Furthermore, it comprises a sound volume judging means for analyzing the sound included in the moving picture data received by the moving picture data exchanging means and judging whether or not the sound is larger than a predetermined sound volume,
7. The conference record according to claim 4, wherein the response data generation unit generates the response data when the volume determination unit determines that the voice is higher than a predetermined volume. apparatus.   The meeting record data creating means includes the predetermined reaction not included in the moving picture data stored in the moving picture storage means based on time information of the reaction data stored in the reaction storage means. The conference recording apparatus according to claim 4, wherein the video recording data is synthesized by replacing the video data with the video data including the predetermined reaction. A conference terminal that acquires images and sounds of users participating in a remote conference and generates moving image data including the images and sounds of the users is installed at each of a plurality of locations where the users exist, and each of the conference terminals The conference recording method is used in a remote conference system in which the moving image data is exchanged via a network and an image and sound synthesized based on the moving image data are output, and the remote conference is recorded. And
A video data exchange step of exchanging and exchanging the video data between the conference terminals;
A moving image storage step of storing the moving image data acquired by the moving image data exchange step in a moving image storage unit in association with time information indicating a reproduction timing of the moving image data at the time of reproduction of the conference recording data;
Based on the delay time required for the video data acquired by the video data exchange step to be transmitted between the conference terminals via the network, the video data stored in the video storage means and the time information A delay time correcting step of adjusting the reproduction timing of the moving image data by correcting the correspondence;
One moving image data generated by one conference terminal and one moving image data received by synthesizing the moving image data stored in the moving image storage means in time series based on the time information A conference record data creating step for creating the conference record data that is sometimes played back synchronously with the other video data generated by the other conference terminal;
A conference record data output step of outputting the conference record data created by the conference record data creation step. A conference terminal that acquires images and sounds of users participating in a remote conference and generates moving image data including the images and sounds of the users is installed at each of a plurality of locations where the users exist, and each of the conference terminals The conference recording program used in a remote conference system in which the video data is exchanged with each other via a network and an image and a sound synthesized based on the video data are output.
Computer
Video data exchanging means for exchanging and exchanging the video data between the conference terminals,
A storage execution unit for storing the moving image data acquired by the moving image data exchanging unit in a moving image storage unit in association with time information indicating a reproduction timing of the moving image data at the time of reproduction of the conference recording data;
Based on the delay time required for the moving image data acquired by the moving image data exchanging means to be transmitted between the conference terminals via the network, the moving image data stored in the moving image storage means and the time information A delay time correcting means for adjusting the reproduction timing of the moving image data by correcting the correspondence;
One moving image data generated by one conference terminal and one moving image data received by synthesizing the moving image data stored in the moving image storage means in time series based on the time information Meeting record data creating means for creating the meeting record data that is played back in synchronization with the other video data generated by the other conference terminal.
A conference recording program that functions as conference record data output means for outputting the conference record data created by the conference record data creation means.

Images