JP2014072599A - Conference server, computer program, remote conference system and terminal - Google Patents

Abstract

PROBLEM TO BE SOLVED: To allow for video recording of image data, transmitted from a predetermined terminal participating in a remote conference, from the timing when a video recording start instruction is acquired.SOLUTION: A conference server acquires first image data from a first terminal and second image data from a second terminal, and transmits the first image data to the second terminal, and the second image data to the first terminal. The conference server acquires a video recording start instruction, transmits a supplement instruction for instructing creation of supplement data to one specific terminal out of the first and second terminals, thus acquiring supplement data. The supplement data is the data corresponding to a period from a first timing when the video recording start instruction is acquired, to a second timing when an intraframe coding image forming the first image data is acquired for the first time after the first timing. The conference server records the first image data from the second timing, and synthesizes the supplement data and video recording data in time series.

Description

  The present invention relates to a conference server for a remote conference, a computer program for the conference server, a remote conference system for a remote conference, and a terminal device connected to a network to which the conference server is connected.

  A technique related to a multi-video chat system has been proposed (see Patent Document 1). In a multi-video chat system, PCs that communicate with each other share a method of recording each other, and by distributing the records themselves, an increase in load can be suppressed and conference recording centered on video can be taken. In the multi-video chat system, communicating PCs are detected, the number of connected PCs is calculated, and the thinning amount of recording timing is calculated. When the recording process is performed and a certain recording amount is reached, the recording data is transmitted to the server.

JP 2006-333417 A

  A remote conference system has been put into practical use. When the remote conference system is used, participants in the conference can participate in the conference by using a plurality of terminal devices connected to a network such as the Internet without gathering at the venue. In a remote conference system, a conference server included in the remote conference system acquires media data acquired by each terminal device used by each participant and transmits (relays) it to other terminal devices. And there are things that can make a meeting happen. The media data includes at least image data. Audio data may be included in the media data.

  Image data transmitted / received between the conference server and each terminal device has a data structure including an intra-frame encoded image and an inter-frame encoded image. An intra-frame encoded image is a frame that serves as a reference when encoding, and is encoded without using inter-frame prediction. Therefore, a still image can be decoded from an intra-frame encoded image without referring to another frame. For example, an intra-frame encoded image can be decoded alone by using image information such as resolution necessary for decoding. The inter-frame encoded image is encoded with reference to other frames. Therefore, the inter-frame encoded image can be decoded into one still image using another frame, for example, the preceding intra-frame encoded image and / or inter-frame encoded image. According to such a data structure, the amount of image data can be reduced.

  Each terminal device can instruct the conference server to record image data of a predetermined terminal device (recording start instruction). When the conference server obtains the recording start instruction, it records the image data of the predetermined terminal device. The image data acquired at the timing when the recording start instruction is acquired may be formed by the inter-frame prediction image. In general, in the conference server, only the process for controlling the remote conference performed between the terminal devices is executed, and the image corresponding to the image data acquired from each terminal device may not be displayed by the conference server itself. is there. In such a type of conference server, when a recording start instruction is acquired at the timing when an inter-frame prediction image is acquired, an image (for example, an intra-frame encoded image or an image required for decoding the inter-frame encoded image) is acquired. The decoded inter-frame encoded image) may not be stored. Therefore, it is considered that the predetermined image data cannot be recorded until the next intra-frame encoded image is acquired. When it is required to record media data transmitted / received between a plurality of terminal devices participating in a remote conference from the timing when the recording start instruction is acquired, regardless of whether or not the intra-frame encoded image is stored There is also.

  The present invention relates to a conference server, a computer program for a conference server, and a remote program capable of recording image data transmitted from a predetermined terminal device participating in a remote conference from the timing when a recording start instruction is acquired. An object is to provide a conference system and a terminal device.

  One aspect of the present invention is a conference server that is included in a remote conference system for a remote conference by a first terminal device and a second terminal device connected to a network, and is connected to the network, the first terminal A first data structure including an intra-frame encoded image that is transmitted from a device, is included at a predetermined time interval, and can be independently decoded; and an inter-frame encoded image that can be decoded based on the intra-frame encoded image First acquisition means for acquiring image data and second image data transmitted from the second terminal device and having the data structure; and the first image data acquired by the first acquisition means, First transmission means for controlling to be transmitted to the second terminal apparatus, and controlling the second image data acquired by the first acquisition means to be transmitted to the first terminal apparatus; Second acquisition means for acquiring a recording start instruction, which is an instruction to start recording of the first image data, input in the first terminal apparatus or the second terminal apparatus; and the recording start instruction by the second acquisition means A third acquisition unit that acquires a timing at which the first acquisition unit acquires an intra-frame encoded image forming the first image data after the first timing, with the first timing acquired as a reference; The first image corresponding to a period from the first timing to a second timing at which the intra-frame encoded image forming the first image data is first acquired by the first acquisition means after the first timing. A complementary instruction that complements the data portion and instructs generation of complementary data having the data structure is one specific terminal of the first terminal device or the second terminal device. Generated from display data generated to display an image corresponding to the first image data in the second transmission means for controlling to be transmitted to the device and the specific terminal device that has received the complement instruction, Acquired by the fourth acquisition means for acquiring the complementary data transmitted from the specific terminal device, the recording means for controlling the recording of the first image data from the second timing, and the fourth acquisition means A conferencing server comprising: synthesizing means for synthesizing the complementary data and the recording data having the data structure recorded under the control of the recording means in time series.

  According to this, complementary data can be generated in one specific terminal device of the first terminal device or the second terminal device. Then, the complementary data generated by the specific terminal device and the recording data recorded by the conference server can be synthesized in time series. Therefore, an image corresponding to the first image data transmitted from the first terminal device can be recorded in response to the recording start instruction. Regarding the first image data and the second image data, the predetermined time interval in which the intra-frame encoded image is included may be either the same interval or a different interval in both image data.

  This remote conference system may be as follows. A determination unit configured to determine a relationship between a period from the first timing to the second timing and a reference time serving as a predetermined reference; and the second transmission unit includes the second timing from the first timing to the second timing. When the determination unit determines that the period until the time is shorter than the reference time, the complement instruction may not be controlled to be transmitted to the specific terminal device.

  According to this, it is possible to reduce the processing load on the conference server required for transmission and synthesis of the complementary instruction. Since the complementary instruction and the complementary data are not transmitted and received, network congestion can be avoided and / or improved.

  Fifth acquisition means for acquiring first capability information indicating the processing capability of the first terminal device and second capability information indicating the processing capability of the second terminal device; the first capability information; and the second Comparing with capability information, the first terminal device or the second terminal device may be provided with first determining means for determining a terminal device having a high processing capability as the specific terminal device. .

  According to this, complementary data can be generated in a terminal device with high processing capability (performance). In the conference server, it is possible to shorten the time until the supplement instruction is transmitted after it is transmitted.

  Sixth acquisition for acquiring first state information indicating a communication state with the first terminal device via the network and second state information indicating a communication state with the second terminal device via the network And the first state information and the second state information are compared, and a terminal device having a good communication state among the first terminal device or the second terminal device is determined as the specific terminal device. 2 determination means.

  According to this, complementary data can be generated in a terminal device having a good communication state. In the conference server, it is possible to shorten the time until the supplement instruction is transmitted after it is transmitted. It is possible to avoid and / or improve network congestion.

  The third acquisition unit acquires a timing at which an intra-frame encoded image forming the second image data is acquired by the first acquisition unit after the first timing on the basis of the first timing, When the specific terminal device is the first terminal device, the second transmission means has a timing different from the timing acquired by the third acquisition means for acquiring the intra-frame encoded image forming the first image data. Is controlled to be transmitted to the first terminal device as the specific terminal device, and the specific terminal device is the second terminal device. A timing different from the timing acquired by the third acquisition means for acquiring the intra-frame encoded image forming the second image data, Second supplemental instructions including a timing of acquiring a complete data controls to be transmitted to the second terminal apparatus as the specific terminal unit, may be.

  According to this, regarding transmission of data from the first terminal device to the conference server or transmission of data from the second terminal device to the conference server, transmission of the intraframe encoded image forming the first image data or the second image data is performed. The timing and the transmission timing of the complementary data can be made different. It is possible to avoid and / or improve network congestion.

  The third acquisition unit acquires a timing at which an intra-frame encoded image forming the second image data is acquired by the first acquisition unit after the first timing on the basis of the first timing, The second transmitting means obtains the timing obtained by the third obtaining means for obtaining the intra-frame coded image forming the first image data, and obtaining the intra-frame coded image forming the second image data. Control is performed so that a third complement instruction including a timing different from the timing acquired by the third acquisition means as a timing for acquiring the complementary data is transmitted to the specific terminal device. Good.

  According to this, regarding the data transmission from the first terminal device and the second terminal device to the conference server, the transmission timing of each intra-frame encoded image forming the first image data and the second image data, and the complementary data Transmission timing can be varied. It is possible to avoid and / or improve network congestion.

  Another aspect of the present invention is included in a remote conference system for a remote conference by a first terminal device and a second terminal device connected to a network, and is read by a control unit that controls a conference server connected to the network. A computer program capable of decoding the control unit based on an intra-frame encoded image that is transmitted from the first terminal device, is included at predetermined time intervals, and can be independently decoded, and an intra-frame encoded image A first acquisition means for acquiring first image data having a data structure including a possible inter-frame encoded image, and second image data transmitted from the second terminal device and having the data structure; The first image data acquired by the first acquisition unit is controlled so that the first image data acquired by the first acquisition unit is transmitted to the second terminal device, and the second image acquired by the first acquisition unit A first transmission means for controlling the data to be transmitted to the first terminal device, and an instruction to start recording of the first image data input in the first terminal device or the second terminal device The second acquisition means for acquiring the recording start instruction and the first timing at which the recording start instruction is acquired by the second acquisition means as a reference, the first acquisition means at the first acquisition means after the first timing. Third acquisition means for acquiring a timing at which an intra-frame encoded image forming image data is acquired, and an intra-frame encoded image forming the first image data from the first timing after the first timing First, the part of the first image data corresponding to the period up to the second timing acquired by the first acquisition unit is complemented, and generation of complementary data having the data structure is instructed. A second transmission means for controlling a completion instruction to be transmitted to one specific terminal apparatus of the first terminal apparatus or the second terminal apparatus; and the specific terminal apparatus that has received the complement instruction, Fourth acquisition means for acquiring the complementary data, generated from display data generated to display an image corresponding to the first image data, and transmitted from the specific terminal device; and the second timing from the second timing Recording means for controlling the recording of the first image data, the complementary data acquired by the fourth acquisition means, and the recording data having the data structure recorded by the control by the recording means in time series A computer program that functions as a combining means for combining.

  According to this, it is possible to realize a conference server capable of generating complementary data in one specific terminal device of the first terminal device or the second terminal device. And the conference server which can synthesize | combine the complementary data produced | generated with the specific terminal device and the video recording data recorded with the conference server according to a time series is realizable. Therefore, the conference server can record an image corresponding to the first image data transmitted from the first terminal device in response to the recording start instruction. The first image data and the second image data are as described above.

  This computer program may be specified as a computer program that can be read by a control unit that controls the conference server further including the above-described elements. According to such a computer program, a conference server further including the above-described elements can be realized.

  Still another aspect of the present invention is a remote conference system including a conference server, a first terminal device, and a second terminal device, which are connected to each other via a network, the conference server including the first server It has a data structure including an intra-frame encoded image that is transmitted from one terminal device and is included at a predetermined time interval and that can be decoded independently, and an inter-frame encoded image that can be decoded based on the intra-frame encoded image First acquisition means for acquiring first image data and second image data transmitted from the second terminal device and having the data structure; and the first image data acquired by the first acquisition means First transmission means for controlling to be transmitted to the second terminal device and controlling the second image data acquired by the first acquisition means to be transmitted to the first terminal device; The first A second acquisition means for acquiring a recording start instruction, which is an instruction to start recording of the first image data, input at the terminal device or the second terminal device; and the recording start instruction is acquired by the second acquisition means A third acquisition unit that acquires a timing at which the first acquisition unit acquires an intra-frame encoded image forming the first image data after the first timing, with the first timing as a reference; The first image data corresponding to a period from one timing to a second timing in which the intra-frame encoded image forming the first image data is first acquired by the first acquisition means after the first timing. A complementary instruction that complements a portion and instructs generation of complementary data having the data structure is sent to one specific terminal device of the first terminal device or the second terminal device. Generated from display data generated to display an image corresponding to the first image data in the second transmission means that controls to be received and the specific terminal device that has received the complement instruction, Fourth acquisition means for acquiring the complementary data transmitted from the terminal device, recording means for controlling recording of the first image data from the second timing, and the complement acquired by the fourth acquisition means Data and recording data recorded under the control of the recording means and having the data structure are combined in time series, and the first terminal device includes the first image data, Third transmission means for controlling to be transmitted to the conference server, seventh acquisition means for acquiring the second image data transmitted from the conference server, and the recording start instruction, Generated by a fourth transmission means for controlling to be transmitted to the conference server, a first generation means for generating the complementary data in response to the complementary instruction transmitted from the conference server, and the first generation means. And a fifth transmission means for controlling the complementary data to be transmitted to the conference server, wherein the second terminal device is controlled to transmit the second image data to the conference server. Sixth transmission means, eighth acquisition means for acquiring the first image data transmitted from the conference server, and seventh transmission means for controlling the recording start instruction to be transmitted to the conference server. In response to the supplement instruction transmitted from the conference server, second generation means for generating the supplement data and the supplement data generated by the second generation means are transmitted to the conference server. System Comprising a ninth transmitting means for, an, a teleconference system.

  According to this, complementary data can be generated in one specific terminal device of the first terminal device or the second terminal device. Then, the conference server can synthesize the complementary data generated by the specific terminal device and the recorded data recorded by the conference server in time series. Therefore, the conference server can record an image corresponding to the first image data transmitted from the first terminal device in response to the recording start instruction. The first image data and the second image data are as described above.

  In this remote conference system, the conference server may be specified as a conference server further including the above-described elements. In this case, a 1st terminal device and a 2nd terminal device are provided with the element corresponding to the conference server further provided with the element mentioned above.

  Still another aspect of the present invention is a terminal device that is connected to a network to which a conference server included in a remote conference system is connected and performs a remote conference via the conference server, and is included in a predetermined time interval and The first image data having a data structure including an intra-frame encoded image that can be decoded independently and an inter-frame encoded image that can be decoded based on the intra-frame encoded image is transmitted to another terminal via the conference server. A tenth transmission means for controlling to be transmitted to the apparatus; a ninth acquisition means for acquiring the second image data transmitted from the other terminal apparatus via the conference server; and the recording of the first image data. An instruction to start recording is supplemented with eleventh transmission means for controlling the recording start instruction to be transmitted to the conference server, and the first image data transmitted from the conference server. In response to a complementary instruction for instructing generation of complementary data, a generating means for generating the complementary data, and a twelfth transmitting means for controlling the complementary data generated by the generating means to be transmitted to the conference server The supplementary data is the first timing when the recording start instruction is acquired by the conference server, and the intra-frame encoded image forming the first image data is first after the first timing. It is a terminal device which is data which complements the part of the 1st image data corresponding to the period to the 2nd timing acquired with a conference server.

  According to this, it can be set as the terminal device which can respond to the conference server mentioned above. When this terminal device corresponds to the first terminal device in the remote conference system described above, the other terminal devices correspond to the second terminal device in the remote conference system. The tenth transmission means of this terminal device corresponds to the third transmission means of the first terminal device. The ninth acquisition means of this terminal device corresponds to the seventh acquisition means of the first terminal device. The eleventh transmission means of this terminal device corresponds to the fourth transmission means of the first terminal device. The generation unit of the terminal device corresponds to the first generation unit of the first terminal device. The twelfth transmission means of this terminal device corresponds to the fifth transmission means of the first terminal device. The first image data in this terminal device corresponds to the first image data transmitted to the conference server by the third transmission means of the first terminal device. The second image data in the terminal device corresponds to the second image data acquired by the seventh acquisition unit of the first terminal device.

  When this terminal device corresponds to the second terminal device in the remote conference system described above, the other terminal devices correspond to the first terminal device in the remote conference system. The tenth transmitting means of this terminal device corresponds to the sixth transmitting means of the second terminal device. The ninth acquisition means of this terminal device corresponds to the eighth acquisition means of the second terminal device. The eleventh transmission means of this terminal device corresponds to the seventh transmission means of the second terminal device. The generation means of this terminal device corresponds to the second generation means of the second terminal device. The twelfth transmission means of this terminal device corresponds to the ninth transmission means of the second terminal device. The first image data in this terminal device corresponds to the second image data transmitted to the conference server by the sixth transmission means of the second terminal device. The second image data in this terminal device corresponds to the first image data acquired by the eighth acquisition means of the second terminal device.

  ADVANTAGE OF THE INVENTION According to this invention, the computer program for a conference server and a conference server which can record the image data transmitted from the predetermined | prescribed terminal device which has participated in the remote conference from the timing when the recording start instruction | indication was acquired. A remote conference system and a terminal device can be obtained.

It is a figure which shows the outline of a remote conference system. It is a figure explaining transmission / reception of media data. It is a figure which shows an example of the remote conference screen displayed on the display part of a terminal device. It is a figure which shows an example of the data structure of image data. It is a figure explaining encoding of the image data based on imaging | photography with a terminal device. It is a sequence diagram of the process regarding the video recording of the remote conference performed with a remote conference system. It is a flowchart (the 1) of the server process performed with a conference server. It is a flowchart (the 2) of the server process performed with a conference server. It is a figure explaining determination of the timing which acquires complementary data. It is a flowchart of the 1st sub process performed by a server process. It is a figure explaining a timing table. It is a flowchart of the 2nd sub process performed by a server process. It is a flowchart of the 3rd sub process (1st example) performed by a server process. It is a figure explaining a performance table. It is a flowchart of the terminal performance process performed by the 3rd sub process of a 1st example. It is a flowchart of the 3rd sub process (2nd example) performed by a server process. It is a figure explaining a communication state table. It is a flowchart of the communication state process performed by the 3rd sub process of a 2nd example. It is a flowchart (the 1) of the terminal process performed with a terminal device. It is a flowchart (the 2) of the terminal process performed with a terminal device.

  Embodiments for carrying out the present invention will be described with reference to the drawings. The present invention is not limited to the configurations described below, and various configurations can be employed in the same technical idea. For example, some of the configurations shown below may be omitted or replaced with other configurations. Moreover, you may make it include another structure.

<Remote conference system>
The remote conference system 10 will be described with reference to FIGS. 1 and 2. As shown in FIG. 1, the remote conference system 10 includes a conference server 20 and a plurality of terminal devices 50A, 50B, and 50C. In this embodiment, a remote conference performed by these three terminal devices 50A, 50B, and 50C will be described as an example. The remote conference by the remote conference system 10 may be performed by three or more terminal devices. In the present embodiment, the terminal devices 50A, 50B, and 50C are referred to as “terminal device 50” when they are not distinguished or when they are collectively referred to.

  The conference server 20 and the terminal devices 50A, 50B, and 50C are connected to the network 80. The network 80 is a network such as the Internet, for example. The remote conference by the terminal devices 50A, 50B, 50C is performed via the conference server 20. The remote conference in the remote conference system 10 may be referred to as a video conference, a video conference, a web conference, or the like, for example.

  The terminal device 50 is, for example, a general-purpose computer (for example, a personal computer or a mobile terminal device such as a smartphone) in which a computer program for remote conference is installed. When the terminal device 50 is a general-purpose computer, a computer program for a remote conference may be installed in advance, but when the terminal device 50 accesses the conference server 20 when participating in the remote conference, It may be transmitted from the conference server 20 to each terminal device 50 via the network 80 and installed at that timing. The prior installation is performed by a computer program stored in a storage medium such as an optical disk or a semiconductor memory being read by the reading unit of the terminal device 50. Alternatively, the prior installation is performed by using a computer program stored in a storage unit such as a hard disk of a server (the conference server 20 or a server not shown) connected to the terminal device 50 via the network 80. 60 (described later). Which form is adopted is appropriately determined in consideration of various circumstances. The terminal device 50 may be a dedicated terminal for a remote conference performed via the conference server 20. The conference server 20 may also be a general-purpose server having a storage medium such as a hard disk in which a computer program for a remote conference is stored.

  In the remote conference system 10, as shown in FIG. 2, the media data A is continuously transmitted from the terminal device 50A to the conference server 20, and the media data B is continuously transmitted from the terminal device 50B to the conference server 20. The media data C is continuously transmitted from the terminal device 50C to the conference server 20. The media data A includes image data (video data) A and audio data A acquired by the terminal device 50A. The media data B includes image data B and audio data B acquired by the terminal device 50B. The media data C includes image data C and audio data C acquired by the terminal device 50C. In FIG. 2, the network 80 is omitted.

  The conference server 20 that receives and acquires the media data A, B, and C continuously transmits the media data D to the terminal devices 50A, 50B, and 50C, respectively. The media data D includes image data A, B, and C and audio data A, B, and C.

  In the terminal devices 50A, 50B, and 50C, an image 65A corresponding to the image data A, an image 65B corresponding to the image data B, and an image 65C corresponding to the image data C included in the received media data D are displayed. The In the terminal devices 50A, 50B, and 50C, the audio data A, B, and C included in the received media data D are reproduced and output in synchronization with the image data A, B, and C. In the remote conference system 10, the remote conference is performed in this way.

<Conference server>
As shown in FIG. 1, the conference server 20 includes a CPU 22, a ROM 24, a storage unit 26, a RAM 28, a timer unit 30, and a communication unit 32. The CPU 22, the ROM 24, the storage unit 26, the RAM 28, the timing unit 30, and the communication unit 32 are connected via a bus 34. CPU22 performs arithmetic processing. The ROM 24 stores a computer program for starting the conference server 20 such as a BIOS (Basic Input / Output System). The storage unit 26 stores various computer programs including, for example, an operation system and computer programs for processing described later (see FIG. 7 and FIG. 8). The storage unit 26 stores recording data generated when recording is executed by the conference server 20. The RAM 28 is a storage area used when the CPU 22 executes various computer programs. The RAM 28 stores predetermined data used in processing during the execution of various types of processing. The CPU 22 controls the conference server 20 by executing an operation system and / or a computer program stored in the storage unit 26. As a result, the conference server 20 executes various processes and realizes various functions.

  The time measuring unit 30 has a clock function and measures the current time. The timekeeping unit 30 may be achieved, for example, by using a timekeeping function that is mounted as a standard in the operation system. The communication unit 32 is connected to the network 80 and executes communication with the terminal devices 50A, 50B, and 50C via the network 80. The communication unit 32 receives media data A, B, and C transmitted from the terminal devices 50A, 50B, and 50C, respectively, and transmits media data D to the terminal devices 50A, 50B, and 50C, respectively. The communication unit 32 receives a recording start instruction transmitted from any of the terminal devices 50A, 50B, and 50C, and transmits a complementary instruction to any of the terminal devices 50A, 50B, and 50C. The communication unit 32 receives complementary data transmitted from any of the terminal devices 50A, 50B, and 50C that have received the complementary instruction. The recording start instruction, the complementary instruction, and the complementary data will be described later.

<Terminal device>
As illustrated in FIG. 1, the terminal device 50 </ b> A includes a CPU 52, a ROM 54, a storage unit 56, a RAM 58, a timer unit 59, and a communication unit 60. The CPU 52, ROM 54, storage unit 56, RAM 58, timing unit 59, and communication unit 60 are connected via a bus 76. A display unit 62, an audio output unit 68, an operation unit 70, an imaging unit 72, and a sound collection unit 74 are connected to the terminal device 50A. The display unit 62, the audio output unit 68, the operation unit 70, the photographing unit 72, and the sound collection unit 74 are connected to the bus 76 via interface circuits (not shown) provided corresponding to the respective units. . Although illustration is abbreviate | omitted in FIG. 1, terminal device 50B, 50C is provided with each part mentioned above with which terminal device 50A is provided.

  The CPU 52 executes arithmetic processing. The ROM 54 stores a computer program for starting up the terminal device 50 such as BIOS. The storage unit 56 stores a computer program for a remote conference between the operation system and the conference server 20. As described above, the computer program for the remote conference may be stored in the storage unit 56 by prior installation, but is installed when the terminal device 50 accesses the conference server 20 when participating in the conference. It may be stored. The RAM 58 becomes a storage area used when the CPU 52 executes various computer programs. The RAM 58 stores predetermined data used in processing during the execution of various types of processing. The CPU 52 controls the terminal device 50 by executing an operation system and / or a computer program stored in the storage unit 56. Thereby, in the terminal device 50, various processes are performed and various functions are realized.

  The timer 59 has a clock function and measures the current time. The timer unit 59 may be achieved, for example, by using a timer function that is mounted as a standard in the operation system. The communication unit 60 is connected to the network 80 and executes communication with the conference server 20 via the network 80. Communication by the communication unit 60 may be either wired communication or wireless communication. The communication units 60 included in the terminal devices 50A, 50B, and 50C respectively transmit the media data A, B, and C to the conference server 20 and receive the media data D transmitted from the conference server 20. Each communication unit 60 transmits a recording start instruction to the conference server 20 and receives a complementary instruction transmitted from the conference server 20. Each communication unit 60 transmits complementary data to the conference server 20. The recording start instruction, the complementary instruction, and the complementary data will be described later.

  The display unit 62 displays various information. For example, a remote conference screen 64 as shown in FIG. The audio output unit 68 outputs audio. For example, when a remote conference is being performed, the voice output unit 68 outputs voices corresponding to the voice data A, B, and C, respectively. The operation unit 70 is an interface that includes predetermined keys and the like and receives operation instructions from the user. When the terminal device 50 is, for example, a personal computer, the operation unit 70 includes a keyboard and a mouse. When the terminal device 50 is a mobile terminal device such as a smartphone, the operation unit 70 may be configured with a touch panel function. The imaging unit 72 captures an image of the outside world where the terminal device 50 exists. The video imaged by the imaging unit 72 is image data. The sound collection unit 74 collects external sounds in which the terminal device 50 exists. The sound collected by the sound collection unit 74 is used as sound data.

<Image data>
Data structures of the image data A, B, and C generated by the terminal devices 50A, 50B, and 50C will be described with reference to FIG. The image data A, B, and C all have the same data structure. Therefore, in this embodiment, when the image data A, B, and C are not distinguished or collectively referred to, they are simply referred to as “image data” (“media data A, B, C”, “audio data A, B, The same applies to “C”).

  The image data is moving image data encoded (compressed) by a predetermined encoding method. As an encoding method, H.264 is used. H.264 is exemplified. As shown in FIG. 4, the image data includes an I frame (Intra-coded Frame) and a P frame (Predicted Frame). That is, the image data is formed by an I frame and a P frame. The I frame is a frame that serves as a reference when coding an image shot by the shooting unit 72 and is encoded without using inter-frame prediction. The I frame may be referred to as an intra-frame encoded image. A P frame is a frame that is encoded using only forward prediction, and forms a single still image by decoding using a forward reference frame (I frame and / or P frame decoded by the I frame). To do. The P frame may be referred to as an inter-frame encoded image.

  For example, it is assumed that the facial expressions of the users participating in the remote conference via the terminal device 50A have changed as shown in FIG. An image related to the first scene (see “Scene 1” in FIG. 5) is encoded as an entire I-frame. Since the entire image of the I frame is encoded, one still image (see “Scene 1” in FIG. 5) can be decoded independently. Subsequent to the scene 1, the photographed image (see “Scene 2” in FIG. 5) changes only the mouth portion of the user as compared with “Scene 1” in FIG. 5. Therefore, only the difference area E (see “difference area E” surrounded by a broken line in “scene 2” in FIG. 5) that is different from “scene 1” in FIG. 5 is encoded into a P frame. That is, the P frame is an image in which only the difference area E from the front I frame as a reference is encoded. The P frame can decode one still image using the region of the reference I frame excluding the difference region E.

  The image forming the image data is an I frame at a predetermined time interval F as shown in FIG. In the image data A, B, and C generated by the terminal devices 50A, 50B, and 50C, the time interval F in which the I frame is repeated is the same. However, the time interval F may be different for each of the image data A, B, and C.

<Processing sequence related to recording>
Processing related to remote conference recording performed in the remote conference system 10 will be described with reference to FIG. In the sequence shown in FIG. 6, the remote conference is started. The terminal devices 50A, 50B, and 50C transmit the media data A, B, and C to the conference server 20, respectively. The conference server 20 includes the terminal devices 50A, 50A, and 50C. It is assumed that media data D is transmitted to 50B and 50C, respectively (see FIG. 2). The recording start instruction is transmitted from the terminal device 50A, and the complement instruction is transmitted to the terminal device 50B. Therefore, in FIG. 6, illustration regarding the terminal device 50C is omitted. In addition, description will be made with respect to image data.

  The user (one of the participants) of the terminal device 50A operates the operation unit 70 and inputs a recording start instruction at the timing when the remote conference is started or when a predetermined time has elapsed after the start. For example, as shown in FIG. 3, the recording start instruction is input by operating and pressing down an input button 66 provided on a remote conference screen 64 displayed on the display unit 62. In this case, a recording start instruction is transmitted from the communication unit 60 of the terminal device 50A to the conference server 20 (T1). In the conference server 20, a recording start instruction is received by the communication unit 32 (T2). In accordance with the recording start instruction, the conference server 20 stores the recording data obtained by recording the image data in the storage unit 26. The target of recording is any one of the image data A, B, and C. For example, the target of recording is image data A related to the terminal device 50A that has transmitted the recording start instruction. However, a plurality or all of the image data A, B, and C may be recorded.

  When the recording start instruction is acquired, the conference server 20 determines whether or not to generate complementary data (T3). A specific determination method at T3 will be described later. When no complementary data is generated (T3: No), the conference server 20 starts recording (T4). Recording data sequentially generated with the start of recording is stored in the storage unit 26. Along with the recording data, recording data corresponding to a predetermined audio data portion corresponding to a predetermined image data portion to be recorded may be generated and stored in the storage unit 26 in association with the recording data.

  When generating complementary data (T3: Yes), the conference server 20 determines the terminal device 50 that generates complementary data from the terminal devices 50A, 50B, and 50C (T5). In the present embodiment, the terminal device 50 that generates complementary data is referred to as a “specific terminal device 50”. A specific determination method at T5 will be described later. Here, as described above, the terminal device 50B is determined as the specific terminal device 50. Thereafter, in the conference server 20, a complement instruction is transmitted from the communication unit 32 to the terminal device 50B (T6). In the conference server 20, when T6 is executed, recording is started (T4) as in the case where complementary data is not generated (see T3: No).

  In the terminal device 50B, the complementary instruction transmitted from the communication unit 32 at T6 is received by the communication unit 60 (T7). In the terminal device 50B, complementary data is generated (T8). The complementary data is generated corresponding to the image data to be recorded. As described above, when the recording data is generated together with the recording data, the audio data corresponding to the complementary data is generated. This audio data is generated by the audio data portion corresponding to the image data portion to be recorded. The complementary instruction and the complementary data will be described later. When the complementary data is generated, in the terminal device 50B, the generated complementary data is transmitted from the communication unit 60 to the conference server 20 (T9). At this time, audio data corresponding to the complementary data is also transmitted together with the complementary data.

  In the conference server 20, the complementary data is received by the communication unit 32 (T10). When the complement data is acquired by the conference server 20, the conference server 20 synthesizes the complement data and the recorded data in time series (T11). Although details will be described later, the supplemental data is synthesized at the beginning (the oldest in time) of the recorded data. The recorded data to which the supplemental data is synthesized is the recorded data that has been determined to generate the supplemental data at T3 (see T3: refer to), has started recording at T4 after T5 and T6. Note that the recording started at T4 ends in response to the recording stop instruction being received by the communication unit 32.

<Server processing>
Server processing executed by the conference server 20 will be described with reference to FIGS. For example, in the remote conference system 10, an email is sent from the conference server 20 to each email address corresponding to each of the terminal devices 50A, 50B, 50C scheduled to participate in the remote conference before the scheduled date and time of the remote conference. Sent. This e-mail includes a URL (Uniform Resource Locator) for a remote conference by the terminal devices 50A, 50B, and 50C. Each user of terminal device 50A, 50B, 50C operates each operation part 70 of terminal device 50A, 50B, 50C etc. for the URL mentioned above on the scheduled date and time of a remote conference. Thereby, the terminal devices 50A, 50B, and 50C access the conference server 20, respectively.

  The server process is started for each terminal device 50A, 50B, 50C participating in the remote conference. When the terminal device 50A accesses the conference server 20, the CPU 22 starts server processing for the terminal device 50A. When the terminal device 50B accesses the conference server 20, the CPU 22 starts server processing for the terminal device 50B. When the terminal device 50C accesses the conference server 20, the CPU 22 starts server processing for the terminal device 50C.

  The CPU 22 that started the server process starts a remote conference with the terminal device 50 that has accessed (S100). Specifically, in S100, the CPU 22 executes various procedures for establishing a session for a remote conference with the terminal device 50 that has accessed. In these procedures, for example, a conference start request (see S800 in FIG. 19) including a device ID and a conference ID is received from the terminal device 50, and the device ID and the conference ID exist in the database stored in the storage unit 26. It includes processing such as determining whether or not. The device ID is information for identifying the terminal device 50. The conference ID is information for identifying a conference in which the terminal device 50 participates. The conference ID corresponds to the URL for the remote conference described above, for example. When the session is established, the CPU 22 temporarily stores the device ID in the RAM 28 in association with the conference ID.

  Subsequently, the CPU 22 executes the first sub-process (S102). The first sub-process is a process for determining the timing at which the conference server 20 (CPU 22) acquires an I frame for the image data included in the media data from the terminal device 50 that has started the remote conference. The determined timing is registered in the timing table (see FIG. 11). The first sub-process and timing table will be described later. The CPU 22 controls the I frame acquisition timing determined in the first sub-process and associated with the accessed terminal device 50 to be transmitted from the communication unit 32 to the terminal device 50 that started the remote conference (S104). ).

  For example, it is assumed that the terminal device 50 that has accessed is the terminal device 50A. In this case, the CPU 22 starts server processing for the terminal device 50A (see S100). When a session is established between the conference server 20 and the terminal device 50A, the CPU 22 executes the first sub-process, and regarding the image data A included in the media data A from the terminal device 50A, the I frame acquisition timing is set. Determine (see S102). The CPU 22 transmits the determined timing (see “timing (I1, I2, I3...)” Associated with the terminal device 50A in the timing table shown in FIG. 11) from the communication unit 32 to the terminal device 50A. (See S104). In the terminal device 50A that has received this timing by the communication unit 60, transmission of the media data A to the conference server 20 is started (see S804 in FIG. 19). The CPU 52 of the terminal device 50A receives the media data A including the I frame forming the image data A from the communication unit 60 from the communication server 60 at a timing corresponding to the I frame acquisition timing acquired with the reception by the communication unit 60. 20 to be transmitted. In the conference server 20, the media data A is received by the communication unit 32.

  The CPU 22 starts server processing for the terminal device 50B and server processing for the terminal device 50C in the same manner as the terminal device 50A described above at each timing when the terminal devices 50B and 50C have accessed. In each server process, S100 to S104 are executed. In the conference server 20, the media data B is received and the media data C is received by the communication unit 32. In the conference server 20, among the terminal devices 50 </ b> A, 50 </ b> B, and 50 </ b> C, media data D including image data and audio data included in the media data from the terminal device 50 with which the session is established is established from the communication unit 32. Is transmitted to the terminal device 50. Media data D including image data A, B, C and audio data A, B, C is transmitted when a session is established with all of the terminal devices 50A, 50B, 50C. In the present embodiment, the terminal device 50 that has accessed the conference server 20 and started server processing in the conference server 20 is referred to as a “target terminal device 50” related to the server processing. For example, in the server process for the terminal device 50A, the terminal device 50A is the target terminal device 50.

  After executing S104, the CPU 22 determines whether or not a recording start instruction has been acquired (S106). The recording start instruction is an instruction to request the start of remote conference recording. It may be determined appropriately as to which of the image data A, B, and C is the image data to be recorded. For example, image data from the target terminal device 50 is set as a recording target. In addition, information for designating the terminal device 50 may be included in the recording start instruction, and the image data from the designated terminal device 50 may be recorded. Two or all of the image data A, B, and C may be recorded. In the present embodiment, it is assumed that the image data to be recorded by the recording start instruction is image data from the target terminal device 50. The recording start instruction is transmitted from the communication unit 60 of the target terminal device 50 (see T1 in FIG. 6 and S810 in FIG. 19) and received by the communication unit 32 (see T2 in FIG. 6). The CPU 22 acquires a recording start instruction via the communication unit 32. In the present embodiment, the timing at which the recording start instruction is acquired is referred to as “first timing”.

  When the recording start instruction has not been acquired (S106: No), the CPU 22 shifts the processing to S134 of FIG. When the recording start instruction is acquired (S106: Yes), the CPU 22 executes the second sub-process (S108). In the second sub-process, the next acquisition period is acquired (see S304 in FIG. 12). The next acquisition period is a period from the first timing to the timing at which the I frame forming the image data from the target terminal device 50 is first acquired after the first timing (see “second timing” described later). is there. The second sub-process and the next acquisition period will be described later. Thereafter, the CPU 22 determines whether or not the next acquisition period is longer than the reference time (S110). S108 and S110 correspond to T3 in FIG. The reference time is a predetermined time and is a time that is not perceived by humans. For example, the reference time is a time of 1 second or less, specifically, a time of about 0.3 seconds.

  When the next acquisition period is longer than the reference time (S110: Yes), the CPU 22 executes a third sub-process (S112). The third sub-process is a process of determining a specific terminal device 50 that generates complementary data from the terminal devices 50A, 50B, and 50C. The third sub process will be described later. The complementary data is data that complements the portion corresponding to the next acquisition period with respect to the image data from the target terminal device 50 (image data to be recorded). When recording image data from the target terminal device 50, in order to acquire an image at the first timing, information representing all the images at that time, that is, an I frame is necessary. However, the image data included in the media data that is received by the communication unit 32 at the first timing and can be acquired by the CPU 22 may be a P frame.

  In such a case, if the next I frame is acquired within a predetermined reference time, the amount of data that is not recorded in accordance with the recording start instruction is equal to or less than the reference time. As described above, the reference time is a time set in advance as a time that is not perceived by humans. Therefore, even if the portion below the reference time is not recorded, it is considered that there is no particular problem. That is, even if the recorded data is reproduced after the end of the remote conference, it is considered that the participants of the remote conference who have visually recognized the data do not notice the presence of the unrecorded portion. On the other hand, if a time longer than the reference time is unrecorded, a participant who visually recognizes the time may be aware that an unrecorded part exists. It can be said that the supplemental data is data for supplementing a part that is not recorded by the conference server 20 when a time longer than the reference time is not recorded. “S110: Yes” corresponds to “T3: Yes” in FIG. 6, and S112 corresponds to T5 in FIG.

  After executing S112, the CPU 22 determines the timing for acquiring complementary data (S114). The timing for acquiring the complementary data is determined based on the I frame acquisition timing registered in the timing table (see FIG. 11) in the first sub-process of S102. Regarding the determination of the timing for acquiring the complementary data, two examples will be described with reference to FIG. In this description, the specific terminal device 50 is assumed to be the terminal device 50B. In the first example, the CPU 22 accesses the timing table (see the lower part of FIG. 11) stored in the RAM 28, and specifies the timing for acquiring the I frame from the terminal device 50B. The CPU 22 determines a predetermined timing among the timings different from the identified timing (see “region G1” (two places) indicated by a broken line in FIG. 9) as the timing for acquiring the complementary data. In the second example, the CPU 22 accesses the timing table (see the lower part of FIG. 11) stored in the RAM 28, and specifies the timing for acquiring I frames from the terminal devices 50A, 50B, and 50C. The CPU 22 determines a predetermined timing among the timings different from the identified timing (see “region G2” (four locations) indicated by a one-dot chain line in FIG. 9) as a timing for acquiring complementary data. In the specific terminal device 50, the complementary data is transmitted at a timing corresponding to the timing determined in S114.

  Thereafter, the CPU 22 controls the supplement instruction to be transmitted from the communication unit 32 to the specific terminal device 50 (S116). S116 corresponds to T6 in FIG. 6, and the complementary instruction is transmitted from the communication unit 32 and received by the specific terminal device 50 (see T7 in FIG. 6 and S820 in FIG. 20: Yes). The complementary instruction is an instruction for requesting the specific terminal device 50 determined in S114 to generate the above-described complementary data. The complement instruction includes the timing determined in S <b> 114 and the terminal ID of the target terminal device 50. The CPU 22 temporarily stores, in the RAM 28, the device ID of the terminal device 50 that has transmitted the complement instruction and the device ID of the target terminal device 50 on which the media data to be complemented with the complement data is recorded in association with each other.

  After executing S116, the CPU 22 acquires the I frame from the first media data including the I frame after the first timing among the media data from the target terminal device 50 sequentially received by the communication unit 32 ( S118). Then, the CPU 22 starts recording the image data included in the media data from the target terminal device 50 (S120). In the recording started in S120, the CPU 22 uses the I frame acquired in S118, and then the P frame and the I frame included in the media data from the target terminal device 50 sequentially received by the communication unit 32. , Generate recording data. Recorded data, like the image data A, B, C, has a data structure including an I frame and a P frame. In addition, the CPU 22 uses the audio data included in the media data together with the I frame in S118, and the audio data included in the media data from the target terminal device 50 that is sequentially received by the communication unit 32 thereafter, to record data. Is generated. The recording data and the recording data that are sequentially generated in response to the reception of the media data by the communication unit 32 are stored as, for example, the storage unit 26 as a media file. When the media file is stored, the media file is associated with the device ID and conference ID of the target terminal device 50. S120 corresponds to T4 executed after T6 of FIG.

  Next, the CPU 22 determines whether or not complementary data has been acquired (S122). The complementary data transmitted from the specific terminal device 50 is received by the communication unit 32. This complementary data is associated with the device ID and conference ID of the specific terminal device 50 that is the transmission source. When the complement data is not received by the communication unit 32 and the complement data is not acquired (S122: No), the CPU 22 repeatedly executes S122 until the complement data is received. When the complement data is received by the communication unit 32 and the complement data is acquired (S122: Yes), the CPU 22 synthesizes the complement data and the recording data in time series (S124). That is, the CPU 22 synthesizes the complementary data with the beginning of the recorded data. The recording data combined with the supplemental data is the recording data stored in the storage unit 26 with the recording started in S120. Specifically, the supplementary data associated with the device ID of the specific terminal device 50 temporarily stored in the RAM 28 in S116 is recorded data included in the media file corresponding to the device ID of the target terminal device 50. Are synthesized. At this time, the CPU 22 synthesizes the audio data for the next acquisition period received and acquired by the communication unit 32 together with the complementary data into the recording data corresponding to the recording data in time series. The synthesis target is specified in the same manner as in the case of complementary data and recorded data. The CPU 22 stores the new recording data synthesized as described above and the media file including the new recording data in the storage unit 26. Thereafter, the CPU 22 proceeds to S130 (see FIG. 8). “S122: Yes” corresponds to T10 in FIG. 6, and S124 corresponds to T11 in FIG.

  When the next acquisition period is equal to or shorter than the reference time (S110: No), the CPU 22 sequentially executes S126 and S128. “S110: No” corresponds to “T3: No” in FIG. S128 corresponds to T4 executed after “T3: No” in FIG. S126 and S128 are executed in the same manner as S118 and S120 described above. Therefore, the description regarding S126 and S128 is omitted. After executing S128, the CPU 22 shifts the process to S130 (see FIG. 8).

  When a recording stop instruction is received by the communication unit 32 after executing S124 or S128, the CPU 22 acquires the recording stop instruction received by the communication unit 32 in S130. The recording stop instruction is an instruction transmitted from the target terminal device 50 (see S814 in FIG. 19) and requesting the end of the recording accompanying the recording start instruction. The CPU 22 continues the recording started in S120 or S128 until the recording stop instruction is acquired, and the image data and the recording data sequentially generated along with the recording are stored in the image file in the media file stored in the storage unit 26. In addition, the recorded data is additionally stored. When the recording stop instruction is acquired, the CPU 22 stops the recording started in S120 or S128 (S132). Thereafter, the CPU 22 shifts the processing to S134.

  In S134, the CPU 22 determines whether or not a conference end instruction has been acquired. The conference end instruction is an instruction to end the established session and request the end of the remote conference. The conference end instruction is transmitted from the communication unit 60 of the target terminal device 50 (see S830 in FIG. 20) and received by the communication unit 32. The CPU 22 acquires a conference end instruction via the communication unit 32. When the conference end instruction has not been acquired (S134: No), the CPU 22 returns the process to S106 in FIG. 7, and executes the processes after S106 in the same manner as described above. When the conference end instruction is acquired (S134: Yes), the CPU 22 ends the server process for the target terminal device 50. As the server process for the target terminal device 50 ends, the CPU 22 ends the transmission of the media data D to the target terminal device 50 and erases the device ID of the target terminal device 50 from the RAM 28. Acquisition of media data from the target terminal device 50 via the communication unit 32 is also terminated.

<First sub-process>
The first sub-process executed in S102 of the server process (see FIGS. 7 and 8) will be described with reference to FIG. The CPU 22 that has started the first sub-process generates a timing table or accesses the timing table stored in the RAM 28 (S200). Specifically, when the timing table corresponding to the conference ID included in the conference start request received in S100 of the server process is not stored in the RAM 28, the CPU 22 generates a timing table. On the other hand, when the timing table corresponding to the conference ID included in the conference start request signal received in S100 has already been stored in the RAM 28, the CPU 22 accesses the timing table associated with the conference ID. As shown in FIG. 11, the timing table is a table in which I frame acquisition timing is registered in association with each of the terminal devices 50A, 50B, and 50C (terminal IDs of the terminal devices 50A, 50B, and 50C). The timing table is generated in association with the conference ID for each remote conference executed by the remote conference system 10, and one common timing table is used in each server process for the terminal devices 50A, 50B, and 50C.

  Subsequently, the CPU 22 registers the I frame acquisition timing in the timing table in association with the device ID of the target terminal device 50 (S202). When the timing table is already stored in the RAM 28 at the time of S200, the I frame acquisition timing does not overlap with the timing already registered in association with the device ID of the other terminal device 50. Thereafter, the CPU 22 ends the first sub-process, and returns the process to S104 in FIG.

  The first sub-process described above will be specifically described. In the remote conference by the terminal devices 50A, 50B, and 50C, the server processing (see FIGS. 7 and 8) for the terminal device 50A is started in a state where the server processing is not executed for any of the terminal devices 50. Assume that the first sub-process is executed in S102 of this server process. In such a case, the conference server 20 has not yet generated the timing table associated with the conference ID corresponding to the remote conference by the terminal devices 50A, 50B, and 50C. Accordingly, in S200, the CPU 22 newly generates a timing table in association with the conference ID corresponding to the remote conference by the terminal devices 50A, 50B, and 50C. The generated timing table is stored in the RAM 28, for example.

  In S202, the CPU 22 registers a plurality of I frame acquisition timings (I1, I2, I3,...) In association with the device ID of the terminal device 50A. The timing of I frame acquisition is set to a time interval F (see FIG. 4) with a predetermined time as a reference. The predetermined time serving as a reference is, for example, the current time measured by the timer unit 30. Since the first sub-process is a process executed immediately after the session is established (see S100 in FIG. 7), the current time as the reference time is established between the conference server 20 and the terminal device 50A. Corresponding to the timing. The time interval F is, for example, 10 seconds. It is assumed that the predetermined reference time is “10:00:00”. In such a case, the CPU 22 associates with the device ID of the terminal device 50A in the timing table newly generated in S200, “10:00: 0” (timing (I1)), “10:00: 10” ( Timing (I2)), “10:00:20” (Timing (I3))... Are registered (see the upper part of FIG. 11).

  Next, it is assumed that the server process (see FIGS. 7 and 8) for the terminal device 50B is started and the first sub-process is executed in S102 of this server process. In this case, in the conference server 20, as described above, in S200 of the first sub-process in the server process for the terminal device 50A, the timing already associated with the conference ID corresponding to the remote conference by the terminal devices 50A, 50B, and 50C. A table is generated and stored in the RAM 28. Accordingly, in S200, the CPU 22 accesses the timing table stored in the RAM 28.

  In S202, the CPU 22 registers a plurality of I frame acquisition timings (I1, I2, I3,...) In association with the device ID of the terminal device 50B. For example, it is assumed that the timing at which a session is established between the conference server 20 and the terminal device 50A and the timing at which a session is established between the conference server 20 and the terminal device 50B are the same timing. In such a case, the CPU 22 delays the reference predetermined time by several seconds, and registers subsequent timings (I2, I3,...) At a time interval F with reference to the delayed timing (I1). To do. Taking the middle part of FIG. 11 as an example, the CPU 22 sets “10:00:02” delayed by 2 seconds from the timing (I1) of the terminal device 50A as the timing (I1) in the timing table stored in the RAM 28. After that, “10:00:12” (timing (I2)), “10:00:22” (timing (I3)),... Are registered (see the middle of FIG. 11).

  Finally, it is assumed that the server process for the terminal device 50C is started and the first sub-process is executed in S102 of this server process. In this case as well, the timing table has already been generated and stored in the RAM 28 as described above. Accordingly, in S200, the CPU 22 accesses the timing table stored in the RAM 28. Subsequently, in S202, the CPU 22 registers a plurality of I frame acquisition timings (I1, I2, I3,...) In association with the device ID of the terminal device 50C. For example, it is assumed that the predetermined time as a reference is “10:00:04”. In such a case, the CPU 22 is associated with the device ID of the terminal device 50C in the timing table stored in the RAM 28, and “10:00:04” (timing (I1)), “10:00:14” ( Timing (I2)), “10:00:24” (Timing (I3))... Are registered (see the lower part of FIG. 11).

  In the remote conference by the terminal devices 50A, 50B, and 50C, server processing (see FIGS. 7 and 8) for the terminal device 50A is performed in a state where the server processing is not executed for any of the terminal devices 50. At the timing when a new timing table is generated in S200 and S202 is executed, the timings of acquiring I frames for all the terminal devices 50A, 50B, and 50C scheduled to participate in the remote conference are registered (lower row in FIG. 11). Reference). When the server processing for the terminal devices 50B and 50C is started, S202 is omitted. When the timing table can be accessed at S200, the CPU 22 ends the first sub-process.

<Second sub-process>
The second sub-process executed in S108 of the server process (see FIGS. 7 and 8) will be described with reference to FIG. CPU22 which started the 2nd sub process acquires the present time measured by the time measuring part 30 (S300). S300 is a process executed immediately after the recording start instruction is acquired in S106 of FIG. Accordingly, the current time acquired in S300 corresponds to the first timing at which the recording start instruction is acquired.

  Subsequently, the CPU 22 acquires the second timing (S302). The second timing is a timing at which the I frame is first acquired from the media data from the target terminal device 50 including the I frame after the first timing. In other words, the second timing is a timing at which the I frame is acquired in S118 of FIG. For example, when the target terminal device 50 is the terminal device 50A and the timing table stored in the RAM 28 is shown in the lower part of FIG. 11, the current time (first timing) acquired in S300 is “10:00:15”. Suppose that In S302, the CPU 22 accesses the timing table stored in the RAM 28, and acquires the timing (I1, I2, I3...) For acquiring the I frame associated with the terminal device 50A. Then, the CPU 22 selects the first “10:00:20” after “10:00:15” from the acquired timing (see “Timing (I3)” associated with the terminal device 50A in the lower part of FIG. 11). ) Is acquired as the second timing.

  Next, the CPU 22 acquires a next acquisition period (S304). As described above, the next acquisition period is a period from the first timing to the second timing. Based on the example described above, the next acquisition period is “5 seconds” from “10:00:15” (current time first timing) acquired in S300 to “10:00:20” (second timing). It is. Thereafter, the CPU 22 ends the second sub-process, and returns the process to S110 of FIG.

<Third sub-process>
Two embodiments of the third sub-process executed in S112 of the server process (see FIGS. 7 and 8) will be described.

<First example>
The third sub-process of the first example will be described with reference to FIG. CPU22 which started the 3rd sub process of the 1st example performs terminal performance processing (S400). The terminal performance processing is processing for acquiring the performance (capability information) of each terminal device 50 from each of the terminal devices 50A, 50B, and 50C and generating a performance table. As shown in FIG. 14, the performance table is associated with each of the terminal devices 50A, 50B, and 50C (terminal IDs of the terminal devices 50A, 50B, and 50C), and the performance of each terminal device 50 (the performance of the CPU 52) is specified. It is the table which registered the frequency | count (determined frequency | count) determined by the terminal device 50 of. The performance table is managed for each remote conference executed by the remote conference system 10, and one common performance table is used in each server process for the terminal devices 50A, 50B, and 50C. The performance table is associated with the conference ID. The terminal performance process will be described later.

  Next, the CPU 22 identifies a terminal device 50 whose performance is equal to or higher than the standard performance (S402). The reference performance is predetermined in consideration of various conditions. Subsequently, the CPU 22 identifies the terminal device 50 with the smallest determined number of times (S404). S402 and S404 are performed based on the performance table associated with the conference ID corresponding to the remote conference by the terminal devices 50A, 50B, and 50C. Thereafter, the CPU 22 determines whether there is a matching terminal device 50 for the terminal device 50 specified in S402 and the terminal device 50 specified in S404 (S406). If there is a matching terminal device 50 (S406: Yes), the CPU 22 determines the matching terminal device 50 as a specific terminal device 50 (S408). When there are a plurality of matching terminal devices 50, the terminal device 50 with the highest performance or the terminal device 50 with the smallest number of determinations may be determined as the specific terminal device 50. When there is no matching terminal device 50 (S406: No), the CPU 22 determines the terminal device 50 having the highest performance as the specific terminal device 50 (S410).

  S402 to S410 will be specifically described based on the performance table shown in FIG. For example, it is assumed that the performance table is the upper part of FIG. 14 and “3.0 GHz × 2 core” is defined as the reference performance. In S402, the CPU 22 accesses the performance table stored in the RAM 28, and specifies that the terminal devices 50A and 50B are the terminal devices 50 having the standard performance or more. In S404, the CPU 22 accesses the performance table stored in the RAM 28, and specifies that the terminal devices 50A, 50B, and 50C are the terminal devices 50 with the smallest number of decisions (0 times). When the terminal devices 50A, 50B specified in S402 and the terminal devices 50A, 50B, 50C specified in S404 are targeted, the terminal devices 50A, 50B are the terminal devices 50 specified in S402 and S404. . Therefore, in S406, the CPU 22 determines that a matching terminal device 50 exists (S406: Yes). In S408, the CPU 22 determines the terminal device 50A having the highest performance (the number of cores: 4 (terminal device 50A)> 2 (terminal device 50B)) as the specific terminal device 50.

  The performance table is the lower part of FIG. 14, and “3.0 GHz × 2 core” is defined as the standard performance. In S402, the CPU 22 accesses the performance table stored in the RAM 28, and specifies that the terminal devices 50A and 50B are the terminal devices 50 having the standard performance or more. In S404, the CPU 22 accesses the performance table stored in the RAM 28, and specifies that the terminal devices 50B and 50C are the terminal devices 50 with the smallest number of determinations (0 times). When the terminal devices 50A and 50B specified in S402 and the terminal devices 50B and 50C specified in S404 are targeted, the terminal device 50B is the terminal device 50 specified in both S402 and S404. Therefore, in S406, the CPU 22 determines that a matching terminal device 50 exists (S406: Yes). In S <b> 408, the CPU 22 determines the terminal device 50 </ b> B as the specific terminal device 50.

  The performance table is the lower part of FIG. 14, and “3.0 GHz × 4 core” is defined as the standard performance. In S <b> 402, the CPU 22 accesses the performance table stored in the RAM 28 and specifies that the terminal device 50 </ b> A is the terminal device 50 having the reference performance or higher. In S404, the CPU 22 accesses the performance table stored in the RAM 28, and specifies that the terminal devices 50B and 50C are the terminal devices 50 with the smallest number of determinations (0 times). When the terminal device 50A specified in S402 and the terminal devices 50B and 50C specified in S404 are targeted, the terminal device 50 specified in S402 and S404 does not exist. Accordingly, in S406, the CPU 22 determines that there is no matching terminal device 50 (S406: No). In S410, the CPU 22 determines the terminal device 50A having the highest performance as the specific terminal device 50. In S410, the specific terminal device 50 may be determined based on the determined number of times (priority is given to the case where the determined number is small).

  After executing S408 or S410, the CPU 22 increases the number of determinations associated with the terminal device 50 determined as the specific terminal device 50 in S408 or S410 by “1” (S412). For example, when the terminal device 50A is determined to be a specific terminal device 50 in the state where the performance table is in the upper part of FIG. 14, the CPU 22 sets the determined number of times associated with the terminal device 50A to “1” (FIG. 14). See below). Thereafter, the CPU 22 ends the third sub-process of the first example, and returns the process to S114 of FIG.

<Terminal performance processing>
The terminal performance process executed in S400 of the third sub process (see FIG. 13) of the first example will be described with reference to FIG. CPU22 which started the terminal performance process controls so that a performance request | requirement may be transmitted to the terminal device 50 from the communication part 32 (S500). The performance request is a command that requests the performance of the terminal device 50 (the performance of the CPU 52) to be transmitted to the conference server 20. Receiving this, the terminal device 50 transmits the performance of the CPU 52 of the own device, the device ID of the terminal device 50, and the conference ID from the communication unit 60 (see S818 in FIG. 19).

  As described above, the server process (FIGS. 7 and 8) is executed for each of the terminal devices 50A, 50B, and 50C that have accessed the conference server 20. The conference server 20 establishes a session with the accessing terminal device 50 and starts a remote conference (see S100 in FIG. 7). In other words, the conference server 20 has not established a session with the terminal device 50 for which server processing has not been started. Therefore, the terminal device 50 to which the performance request is transmitted (receives the performance request) is the terminal device 50 in which a session is established with the conference server 20 at the time when S500 is executed.

  Next, in response to the transmission in S500, the CPU 22 receives and acquires the performance transmitted from the terminal device 50 that has received the transmission (S502). At this time, the terminal ID and the conference ID transmitted together with the performance are also received by the communication unit 32, and the CPU 22 acquires them. Subsequently, the CPU 22 generates a performance table in which the acquired performance of the terminal device 50 is registered in association with the device ID of the terminal device 50 (S504). The performance table is stored in, for example, the RAM 28 in a state associated with the acquired conference ID. Thereafter, the CPU 22 ends the terminal performance process, and returns the process to S402 in FIG.

  In the remote conference by the terminal devices 50A, 50B, and 50C, the server processing (see FIGS. 7 and 8) for the terminal device 50A is started in a state where the server processing is not executed for any of the terminal devices 50. Assume that the third sub-process is executed in S112 of this server process. In such a case, as described above, the performance request is transmitted only to the terminal device 50A (see S500), and in S502, the CPU 22 acquires only the performance of the terminal device 50A. In the case as described above, since S504 is not executed in the relationship with the terminal devices 50A, 50B, and 50C, the performance table associated with the conference ID of the remote conference in which the terminal device 50A participates has been generated. Absent. Therefore, in S504, the CPU 22 newly generates a performance table in association with the conference ID, and registers the performance acquired in S502 in association with the device ID of the terminal device 50A in the generated performance table. The determined number of times is “0”. The performance table is stored in the RAM 28.

  Thereafter, a server process (see FIGS. 7 and 8) for the terminal device 50B is started, and the third sub-process is executed in S112 of this server process. In this case, the performance request is transmitted to the terminal devices 50A and 50B in S500, and the performance of the terminal devices 50A and 50B, the terminal IDs of the terminal devices 50A and 50B, and the conference ID are acquired in S502. Transmission to the registered terminal device 50A may be omitted. In S504, the CPU 22 additionally registers the performance of the terminal device 50B in association with the device ID of the terminal device 50B in the performance table associated with the conference ID of the remote conference in which the terminal device 50B has already been generated. The determined number of times for the terminal device 50B is “0”. As a result, a performance table in which the performance of the terminal devices 50A and 50B and the determined number of times are registered in association with the terminal IDs of the terminal devices 50A and 50B is generated.

  Finally, it is assumed that a server process (see FIGS. 7 and 8) for the terminal device 50C is started and the third sub-process is executed in S112 of this server process. In this case, the performance request is transmitted to the terminal devices 50A, 50B, and 50C in S500, and the performance of the terminal devices 50A, 50B, and 50C, the terminal IDs of the terminal devices 50A, 50B, and 50C, and the conference ID are acquired in S502. Is done. Transmission to the registered terminal devices 50A and 50B may be omitted. In S <b> 504, the CPU 22 additionally registers the performance of the terminal device 50 </ b> C in association with the device ID of the terminal device 50 </ b> C in the performance table associated with the conference ID of the remote conference in which the already generated terminal device 50 </ b> C participates. The determined number of times for the terminal device 50C is “0”. As a result, a performance table (see the upper part of FIG. 14) in which the performance of the terminal devices 50A, 50B, and 50C and the determined number of times are registered in association with the terminal IDs of the terminal devices 50A, 50B, and 50C is generated.

  In addition, although each server process for terminal device 50A, 50B, 50C is started (S100-S104 of FIG. 7 is already performed about all the terminal devices 50), for example, in terminal device 50B, 50C, it is video recording. It is also assumed that the start instruction has not been transmitted (see S810 in FIG. 19), and as a result, in each server process of the terminal devices 50B and 50C, S106 is not affirmed (see S106: Yes). The In such a case, since all the sessions are established between the conference server 20 and the terminal devices 50A, 50B, and 50C, the timing at which the third sub-process is executed in S112 of the server process for the terminal device 50A At the time of execution of S504, a performance table (see the upper part of FIG. 14) is created in which the performance of the terminal devices 50A, 50B, and 50C and the determined number of times are registered in association with the device IDs of the terminal devices 50A, 50B, and 50C. The When the third sub-process is executed by either one of the terminal devices 50B and 50C or both devices, a performance table in which the performance of the terminal devices 50A, 50B, and 50C and the determined number of times are registered is already generated, Stored in the RAM 28. Therefore, the CPU 22 that has started the third sub-process in S112 of the server process for the terminal devices 50B and 50C accesses the performance table stored in the RAM 28 based on the conference ID and is in the registration state as described above. And the third sub-process may be terminated.

<Second example>
The third sub process of the second example will be described with reference to FIG. The CPU 22 that has started the third sub-process of the second example executes a communication state process (S600). The communication status process is a process of acquiring the status of the network 80 between each of the terminal devices 50A, 50B, and 50C, that is, the communication status (status information) and generating a communication status table. In the present embodiment, a communication speed (communication band) will be described as an example of the communication state. The communication state may be another element different from the communication speed. For example, it may be a delay time until a response is acquired after sending a request. As shown in FIG. 17, the communication state table is determined for the communication speed and the specific terminal device 50 in association with each of the terminal devices 50A, 50B, and 50C (the terminal IDs of the terminal devices 50A, 50B, and 50C). It is a table in which the number of times (the number of decisions) is registered. The communication status table is managed for each remote conference executed in the remote conference system 10, and one common communication status table is used in each server process for the terminal devices 50A, 50B, and 50C. The communication state table is associated with the conference ID. The communication state process will be described later.

  Next, the CPU 22 identifies a terminal device 50 whose communication state is equal to or higher than a reference value (S602). The reference value is determined in advance in consideration of various conditions such as the amount of communication data. Subsequently, the CPU 22 identifies the terminal device 50 with the smallest number of decisions (S604). S602 and S604 are performed based on the communication state table associated with the conference ID corresponding to the remote conference by the terminal devices 50A, 50B, and 50C. Thereafter, the CPU 22 determines whether there is a matching terminal device 50 for the terminal device 50 specified in S602 and the terminal device 50 specified in S604 (S606). When there is a matching terminal device 50 (S606: Yes), the CPU 22 determines the matching terminal device 50 as a specific terminal device 50 (S608). When there are a plurality of matching terminal devices 50, the terminal device 50 with the best communication state or the terminal device 50 with the smallest number of determinations may be determined as the specific terminal device 50. If there is no matching terminal device 50 (S406: No), the CPU 22 determines the specific terminal device 50 as the terminal device 50 with the best communication state (S610).

  S602 to S610 will be specifically described based on the communication state table shown in FIG. For example, it is assumed that the communication state table is the upper part of FIG. 17 and “1.00 Mbytes / sec” is defined as the reference value. In S <b> 602, the CPU 22 accesses the communication state table stored in the RAM 28, and specifies that the terminal devices 50 </ b> A and 50 </ b> B are the terminal devices 50 having a reference value or more. In S604, the CPU 22 accesses the communication state table stored in the RAM 28, and specifies that the terminal devices 50A, 50B, and 50C are the terminal devices 50 with the minimum determined number (0 times). When the terminal devices 50A, 50B specified in S602 and the terminal devices 50A, 50B, 50C specified in S604 are targeted, the terminal devices 50A, 50B are the terminal devices 50 specified in S602 and S604. . Accordingly, in S606, the CPU 22 determines that a matching terminal device 50 exists (S606: Yes). In S <b> 608, the CPU 22 determines the terminal device 50 </ b> A having the highest communication speed as the specific terminal device 50.

  It is assumed that the communication state table is in the lower part of FIG. 17 and “1.00 Mbytes / sec” is defined as the reference value. In S402, the CPU 22 accesses the communication state table stored in the RAM 28, and specifies that the terminal devices 50A and 50B are the terminal devices 50 having a reference value or more. In S <b> 604, the CPU 22 accesses the performance table stored in the RAM 28, and specifies that the terminal devices 50 </ b> B and 50 </ b> C are the terminal devices 50 with the smallest number of decisions (0 times). When the terminal devices 50A and 50B specified in S602 and the terminal devices 50B and 50C specified in S604 are targeted, the terminal device 50B is the terminal device 50 specified in both S602 and S604. Accordingly, in S606, the CPU 22 determines that a matching terminal device 50 exists (S606: Yes). In S <b> 608, the CPU 22 determines the terminal device 50 </ b> B as the specific terminal device 50.

  It is assumed that the communication state table is the lower part of FIG. 17 and “1.10 Mbytes / sec” is defined as the reference value. In S <b> 602, the CPU 22 accesses the performance table stored in the RAM 28 and specifies that the terminal device 50 </ b> A is the terminal device 50 having a reference value or more. In S <b> 604, the CPU 22 accesses the performance table stored in the RAM 28, and specifies that the terminal devices 50 </ b> B and 50 </ b> C are the terminal devices 50 with the smallest number of decisions (0 times). When the terminal device 50A specified in S602 and the terminal devices 50B and 50C specified in S604 are targeted, the terminal device 50 specified in both S602 and S604 does not exist. Therefore, in S606, the CPU 22 determines that there is no matching terminal device 50 (S606: No). In S610, the CPU 22 determines the specific terminal device 50 as the terminal device 50A having the highest communication speed. In S610, a specific terminal device 50 may be determined based on the determined number of times (priority is given to the case where the determined number is small).

  After executing S608 or S610, the CPU 22 increases the determined number of times associated with the terminal device 50 determined as the specific terminal device 50 in S608 or S610 by “1” (S612). For example, when the terminal device 50A is determined to be a specific terminal device 50 in the state where the communication state table is in the upper part of FIG. 17, the CPU 22 sets the determined number of times associated with the terminal ID of the terminal device 50A to “1”. (Refer to the lower part of FIG. 17). Thereafter, the CPU 22 ends the third sub-process of the second example, and returns the process to S114 of FIG.

<Communication status processing>
The communication state process executed in S600 of the third sub process (see FIG. 16) of the second example will be described with reference to FIG. The CPU 22 that has started the communication state process controls so that the speed measurement request is transmitted from the communication unit 32 to the terminal device 50 (S700). The speed measurement request is a command for requesting the terminal device 50 to transmit predetermined data (packets) having a predetermined data amount to the conference server 20. Receiving this, the terminal device 50 transmits a predetermined amount of predetermined data (hereinafter referred to as “measurement data”) from the communication unit 60 to the conference server 20. A transmission time is given to the measurement data. The measurement data may include the device ID of the terminal device 50 and the conference ID. The terminal device 50 to which the speed measurement request is transmitted (receives the speed measurement request) is the same as the terminal device 50 to which the performance request is transmitted (see S500 in FIG. 15) in the terminal performance processing described above. The description regarding this is omitted.

  Next, in response to the transmission of the speed measurement request in S500, the CPU 22 receives and acquires measurement data transmitted from the terminal device 50 that has received the request (S702). From the terminal device 50 that has received the speed measurement request, the terminal ID and the conference ID are also transmitted together with the measurement data, and are received by the communication unit 32. CPU22 acquires these with reception of terminal ID and meeting ID in the communication part 32. FIG. In S <b> 702, the CPU 22 acquires the current time measured by the time measuring unit 30. Subsequently, the CPU 22 acquires a communication speed (S704). In S704, the CPU 22 obtains the time required for transmission / reception of the measurement data from the current time acquired in S702 and the transmission time included in the measurement data. Then, the CPU 22 obtains the communication speed by dividing the data amount of the measurement data by the obtained time. The CPU 22 generates a communication state table in which the acquired communication speed is registered in association with the terminal ID of the terminal device 50 (S704). At this time, the determined number of times is set to “0”. The communication state table is associated with the conference ID of the remote conference in which the terminal devices 50A, 50B, and 50C participate. The communication state table is stored in the RAM 28, for example. Thereafter, the CPU 22 ends the communication state process, and returns the process to S602 in FIG.

  Among the terminal devices 50A, 50B, and 50C, the terminal device 50 that can acquire the communication speed is the same as the terminal device 50 that can acquire the performance in the terminal performance processing. The explanation regarding this is the same as above, and is omitted.

<Terminal processing>
Terminal processing executed by each terminal device 50 participating in the remote conference will be described with reference to FIGS. 19 and 20. As described above, the terminal processing starts when the user of the terminal device 50 operates the operation unit 70 for the remote conference URL included in the e-mail received before the scheduled date and time of the remote conference. Is done. The CPU 52 that has started the terminal process controls the conference start request to be transmitted from the communication unit 60 to the conference server 20 (S800). As described above, the transmission of the conference start request in S800 is performed in various procedures for establishing a session that is executed when the conference server 20 is accessed based on the URL described above. The conference start request includes the terminal ID of the terminal device 50 that has started the terminal processing and the conference ID.

  After executing S800 and establishing a session with the conference server 20, the CPU 52 acquires the timing for acquiring the I frame (S802). The I frame acquisition timing is transmitted from the communication unit 32 of the conference server 20 and received by the communication unit 60 in S104 of FIG. With the reception by the communication unit 60, the CPU 52 acquires the timing of acquiring the I frame. Subsequently, the CPU 52 starts transmission of media data including image data and audio data acquired by the own device and reception of media data D transmitted from the conference server 20. In step S <b> 804, the CPU 52 controls so that the media data in the own device is transmitted from the communication unit 60 to the conference server 20. Then, the CPU 52 acquires media data D received by the communication unit 60. The CPU 52 performs control so that the remote conference screen 64 corresponding to the acquired media data D is displayed on the display unit 62. The display unit 62 displays a remote conference screen 64 corresponding to the image data included in the media data D, and the audio output unit 68 outputs audio corresponding to the audio data included in the media data D.

  Next, the CPU 52 determines whether or not recording has been started (S806). When the recording start instruction is transmitted and the recording stop instruction is not transmitted after the start, S806 is affirmed (S806: Yes). If the recording start instruction is not transmitted, or if the recording stop instruction is transmitted after the recording start instruction is transmitted, S806 is denied (No in S806). If S806 is negative (S806: No), the CPU 52 determines whether or not a recording start instruction has been input (S808). As described above, the recording start instruction is input by operating the operation unit 70 and pressing the input button 66 provided on the remote conference screen 64 displayed on the display unit 62. When the input button 66 is pressed at the start of recording, a mark indicating the recording state may be displayed at a predetermined position on the remote conference screen 64. The input button 66 may be displayed in a manner in which it can be recognized that the input button 66 has been pressed. When the recording start instruction is not input (S808: No), the CPU 52 shifts the process to S816. When the recording start instruction is input (S808: Yes), the CPU 52 controls the recording start instruction to be transmitted from the communication unit 60 to the conference server 20 (S810). S810 corresponds to T1 in FIG. Thereafter, the CPU 52 shifts the process to S816.

  When S806 is affirmed (S806: Yes), the CPU 52 determines whether or not a recording stop instruction is input (S812). The recording stop is input by operating the operation unit 70 again and pressing the input button 66 provided on the remote conference screen 64 displayed on the display unit 62 as in the case of the recording start instruction. As described above, when a mark indicating the recording state is displayed at a predetermined position on the remote conference screen 64, when the input button 66 is pressed when the recording is stopped, the mark is not displayed. Is done. Further, when the input button 66 is displayed in such a manner that it can be recognized that the input button 66 has been pressed, the input button 66 is changed to the state before the press. When the recording stop instruction is not input (S812: No), the CPU 52 shifts the process to S816. When the recording stop instruction is input (S812: Yes), the CPU 52 controls the recording stop instruction to be transmitted from the communication unit 60 to the conference server 20 (S814). Thereafter, the CPU 52 shifts the process to S816.

  In S816, the CPU 52 determines whether or not a performance request has been acquired. The performance request is transmitted from the conference server 20 in S500 of the terminal performance process (see FIG. 15) executed in S400 of the third sub-process (see FIG. 13) of the first example. The transmitted performance request is received by the communication unit 60, and accordingly, the CPU 52 acquires the performance request. When the performance request is not acquired (S816: No), the CPU 52 shifts the processing to S820 in FIG. When the performance request is acquired (S816: Yes), the CPU 52 controls the performance of the own device to be transmitted from the communication unit 60 to the conference server 20 (S818). At this time, the terminal ID of the terminal device 50 and the conference ID are transmitted together with the performance. After executing S818, the CPU 52 shifts the processing to S820 in FIG.

  Note that S816 and S818 described above correspond to the case where the third sub-process (see S112 in FIG. 7) executed by the conference server 20 is the third sub-process (see FIG. 13) of the first example. is there. 19 and 20 are flowcharts of terminal processing executed by the terminal device 50 when the third sub processing of the first example is executed by the conference server 20 as the third sub processing. When the third sub-process executed in the conference server 20 is the third sub-process of the second example (see FIG. 16), the conference server 20 executes the third sub-process in S700 of the second sub-process of the second example. In S800 of the communication state process (see FIG. 18), a speed measurement request is transmitted. Therefore, in the terminal process corresponding to the third sub-process of the second example, S816 is determined based on whether or not a speed measurement request has been acquired. In S818, the measurement data (including the transmission time, the device ID, and the conference ID) is transmitted from the communication unit 60 to the conference server 20.

  In S820 of FIG. 20, the CPU 52 determines whether or not a complement instruction has been acquired. The supplement instruction is transmitted from the conference server 20 to the specific terminal device 50 in S116 of FIG. 7 (see T6 of FIG. 6). The complement instruction is received by the communication unit 60 (see T7 in FIG. 6), and accordingly, the CPU 52 acquires the complement instruction. When the complement instruction has not been acquired (S820: No), the CPU 52 shifts the process to S828. When the complementary instruction is acquired (S820: Yes), the CPU 52 generates complementary data according to the complementary instruction (S822).

  It is assumed that the media data D acquired based on the process started in S804 includes image data A, B, C and audio data A, B, C. In this case, in the terminal device 50 (specific terminal device 50) that has acquired the complementary instruction, a remote conference screen 64 as shown in FIG. The audio corresponding to the audio data A, B, and C is output from the audio output unit 68. When the remote conference screen 64 is displayed, the CPU 52 generates display data by decoding the image data A, B, and C, for example. In S822, the CPU 52 generates image data for the next acquisition period corresponding to the image data of the target terminal device 50 from the display data. The target terminal device 50 is specified by the terminal ID of the target terminal device 50 included in the complement instruction. The CPU 52 encodes the image data for the next acquisition period and generates complementary data for the next acquisition period. Further, the CPU 52 generates audio data for the next acquisition period corresponding to the complementary data from the audio data of the target terminal device 50. The generated complementary data and audio data are temporarily stored in the storage unit 56 or the RAM 58, for example. Complementary data and audio data are associated with the device ID and conference ID of the device itself. S822 corresponds to T8 in FIG.

  Next, the CPU 52 determines whether or not the current time measured by the time measuring unit 59 is a timing for transmitting complementary data (S824). The timing for transmitting the complementary data is a timing corresponding to the timing determined in S114 of FIG. When the current time is not the timing for transmitting the complementary data (S824: No), the CPU 52 repeatedly executes S824. When the current time is the timing for transmitting complementary data (S824: Yes), the CPU 52 controls the complementary data generated in S822 to be transmitted from the communication unit 60 to the conference server 20 (S826). At this time, the communication unit 60 also transmits audio data for the next acquisition period corresponding to the complementary data, along with the complementary data. S826 corresponds to T9 in FIG.

  After executing S826, the CPU 52 determines whether or not the conference end has been input in its own device (S828). The end of the conference is input, for example, when the user of the terminal device 50 performs an operation for closing the remote conference screen 64 displayed on the display unit 62 via the operation unit 70. A button (not shown in FIG. 3) for inputting the end of the conference may be provided on the remote conference screen 64 so that the remote conference screen 64 can be input by operating the operation unit 70 and pressing it. When the conference end has not been input (S828: No), the CPU 52 returns the process to S806 in FIG. 19, and executes the processes after S806 in the same manner as described above.

  When the conference end is input (S828: Yes), the CPU 52 controls the conference end instruction to be transmitted from the communication unit 60 to the conference server 20 (S830). Subsequently, the CPU 52 ends the transmission of the media data from the communication unit 60 and the reception of the media data D (S832). Thereafter, the CPU 52 ends the terminal process.

<Effect of this embodiment>
According to this embodiment, the following effects can be obtained.

  (1) With respect to recording in response to a recording start instruction, complementary data is generated in a specific terminal device 50 (see S112 in FIG. 7 (see FIG. 13 or FIG. 16 for details) and S116), and the generated complementary data (See S122 in FIG. 7: Yes), and the complementary data and the recording data based on the recording started in S120 in FIG. 7 are combined in time series (see S124 in FIG. 7). Therefore, in the conference server 20 that acquires the media data A, B, and C and transmits the media data D to the terminal devices 50A, 50B, and 50C, the image data A, B, and C are not stored in the storage unit 26 at all times. Even when the conference screen 64 is not displayed, the recording data from the first timing when the recording start instruction is acquired can be generated for the target terminal device 50.

  When the next acquisition period is equal to or shorter than the reference time (see S110: No in FIG. 7), S112 to S116 and the like are omitted and complementary data is not generated. Therefore, it is possible to reduce the processing load on the conference server 20 required for transmission and synthesis of the complementary instruction.

  (2) When generating complementary data (see S110 in FIG. 7: Yes), the timing for acquiring the complementary data is set to be different from the timing for acquiring the I frame (see S114 in FIG. 7). Therefore, the transmission timing of media data including an I frame from the terminal device 50 to the conference server 20 and the transmission timing of complementary data from the specific terminal device 50 to the conference server 20 can be made different. It becomes possible to avoid and / or improve the congestion of the network 80.

  (3) The specific terminal device 50 that generates the complementary data is changed to the performance of the terminal device 50 (see the third sub-process in the first example, FIG. 13) or the communication state (see the third sub-process in the second example, FIG. 16). Based on this, it was decided to decide. Therefore, it is possible to generate complementary data with the terminal device 50 with high performance and high processing capacity, or the terminal device 50 with good network status and good communication state. Can be obtained.

<Modification>
This embodiment can also be performed as follows.

  (1) In the above, media data A, B, and C including image data A, B, and C and audio data A, B, and C acquired by each of the terminal devices 50A, 50B, and 50C are transmitted to the conference server 20. Then, the conference server 20 transmits the media data D including the image data A, B, C and the audio data A, B, C to the terminal devices 50A, 50B, 50C. The media data D transmitted from the conference server 20 may be data that does not include image data and audio data of the terminal device 50 that is the destination. For example, the media data D transmitted to the terminal device 50A may be data including image data B and C and audio data B and C. In this case, in the terminal device 50A, display data corresponding to the remote conference screen 64 is generated based on the image data A acquired by the own device and the image data B and C included in the media data D, and the remote conference screen 64 is generated. Is displayed on the display unit 62. The same applies to audio.

  (2) In the above description, the third sub-process (S112 in FIG. 7; see FIG. 13 or FIG. 16 for details) is performed after the execution of S110 in the server process (FIGS. 7 and 8). For example, the third sub-process may be performed immediately after S100. Further, when the third sub-process of the second example (see FIG. 16) is employed, such a third sub-process may be performed constantly or periodically during the period in which the remote conference is being executed.

10 remote conference system 20 conference server 22 CPU
24 ROM
26 storage unit 28 RAM
30 Timekeeping Unit 32 Communication Unit 34 Bus 50, 50A, 50B, 50C Terminal Device 52 CPU
54 ROM
56 storage unit 58 RAM
59 Timing unit 60 Communication unit 62 Display unit 64 Remote conference screen 65A, 65B, 65C Image 66 Input button 68 Audio output unit 70 Operation unit 72 Shooting unit 74 Sound collection unit 76 Bus 80 Network

Claims (9)

A conference server included in a remote conference system for a remote conference by a first terminal device and a second terminal device connected to a network, connected to the network,
A data structure that is transmitted from the first terminal device and includes an intra-frame encoded image that is included at a predetermined time interval and can be independently decoded, and an inter-frame encoded image that can be decoded based on the intra-frame encoded image First acquisition means for acquiring first image data having the second image data transmitted from the second terminal device and having the data structure;
The first image data acquired by the first acquisition means is controlled to be transmitted to the second terminal device, and the second image data acquired by the first acquisition means is the first terminal First transmission means for controlling to be transmitted to the apparatus;
Second acquisition means for acquiring a recording start instruction, which is an instruction to start recording of the first image data, input in the first terminal apparatus or the second terminal apparatus;
With reference to the first timing at which the recording start instruction is acquired by the second acquisition means, after the first timing, an intra-frame encoded image forming the first image data is acquired by the first acquisition means. Third acquisition means for acquiring timing;
The first image corresponding to a period from the first timing to a second timing at which the intra-frame encoded image forming the first image data is first acquired by the first acquisition means after the first timing. Control is performed such that a complementary instruction that complements the data portion and instructs generation of complementary data having the data structure is transmitted to one specific terminal device of the first terminal device or the second terminal device. A second transmission means;
The specific terminal device that has received the complementary instruction acquires the complementary data that is generated from display data generated to display an image corresponding to the first image data and transmitted from the specific terminal device. A fourth acquisition means;
Recording means for controlling recording of the first image data from the second timing;
A conference server comprising: synthesizing means for synthesizing the complementary data acquired by the fourth acquisition means and the recorded data having the data structure recorded by the control of the recording means in time series. A determination unit that determines a relationship between a period from the first timing to the second timing and a reference time that is a predetermined reference;
The second transmission unit is configured to transmit the complementary instruction to the specific terminal device when the determination unit determines that a period from the first timing to the second timing is shorter than the reference time. The conference server according to claim 1, which is not controlled. Fifth acquisition means for acquiring first capability information indicating the processing capability of the first terminal device and second capability information indicating the processing capability of the second terminal device;
A first determining unit that compares the first capability information with the second capability information, and determines a terminal device having a high processing capability as the specific terminal device among the first terminal device or the second terminal device; The conference server according to claim 1, further comprising: Sixth acquisition for acquiring first state information indicating a communication state with the first terminal device via the network and second state information indicating a communication state with the second terminal device via the network Means,
Second determining means for comparing the first state information with the second state information and determining, as the specific terminal device, a terminal device having a good communication state among the first terminal device or the second terminal device. The conference server according to claim 1, further comprising: The third acquisition unit acquires a timing at which an intra-frame encoded image forming the second image data is acquired by the first acquisition unit after the first timing on the basis of the first timing;
The second transmission means includes
In the case where the specific terminal device is the first terminal device, a timing different from the timing acquired by the third acquisition unit for acquiring the intra-frame encoded image forming the first image data is set to the complementary data. Control is performed so that a first complement instruction included as the acquisition timing is transmitted to the first terminal device as the specific terminal device,
In the case where the specific terminal device is the second terminal device, a timing different from the timing acquired by the third acquisition unit for the acquisition of the intra-frame encoded image forming the second image data is set to the complementary data. 5. The conference server according to claim 1, wherein control is performed such that a second complementary instruction included as the acquisition timing is transmitted to the second terminal device as the specific terminal device. The third acquisition unit acquires a timing at which an intra-frame encoded image forming the second image data is acquired by the first acquisition unit after the first timing on the basis of the first timing;
The second transmission means obtains the timing obtained by the third obtaining means for obtaining the intra-frame coded image forming the first image data, and obtains the intra-frame coded image forming the second image data. The third complement instruction including a timing different from the timing obtained by the third obtaining means as the timing for obtaining the complement data is controlled to be transmitted to the specific terminal device. The conference server according to claim 4. A computer program included in a remote conference system for a remote conference by a first terminal device and a second terminal device connected to a network and readable by a control unit that controls a conference server connected to the network,
The control unit
A data structure that is transmitted from the first terminal device and includes an intra-frame encoded image that is included at a predetermined time interval and can be independently decoded, and an inter-frame encoded image that can be decoded based on the intra-frame encoded image First acquisition means for acquiring first image data having the second image data transmitted from the second terminal device and having the data structure;
The first image data acquired by the first acquisition means is controlled to be transmitted to the second terminal device, and the second image data acquired by the first acquisition means is the first terminal First transmission means for controlling to be transmitted to the apparatus;
Second acquisition means for acquiring a recording start instruction, which is an instruction to start recording of the first image data, input in the first terminal apparatus or the second terminal apparatus;
With reference to the first timing at which the recording start instruction is acquired by the second acquisition means, after the first timing, an intra-frame encoded image forming the first image data is acquired by the first acquisition means. Third acquisition means for acquiring timing;
The first image corresponding to a period from the first timing to a second timing at which the intra-frame encoded image forming the first image data is first acquired by the first acquisition means after the first timing. Control is performed such that a complementary instruction that complements the data portion and instructs generation of complementary data having the data structure is transmitted to one specific terminal device of the first terminal device or the second terminal device. A second transmission means;
The specific terminal device that has received the complementary instruction acquires the complementary data that is generated from display data generated to display an image corresponding to the first image data and transmitted from the specific terminal device. A fourth acquisition means;
Recording means for controlling recording of the first image data from the second timing;
A computer program that functions as a synthesizing unit that synthesizes the complementary data acquired by the fourth acquisition unit and the recording data having the data structure recorded by the control of the recording unit in time series. A remote conference system including a conference server, a first terminal device, and a second terminal device connected to each other via a network,
The conference server
A data structure that is transmitted from the first terminal device and includes an intra-frame encoded image that is included at a predetermined time interval and can be independently decoded, and an inter-frame encoded image that can be decoded based on the intra-frame encoded image First acquisition means for acquiring first image data having the second image data transmitted from the second terminal device and having the data structure;
The first image data acquired by the first acquisition means is controlled to be transmitted to the second terminal device, and the second image data acquired by the first acquisition means is the first terminal First transmission means for controlling to be transmitted to the apparatus;
Second acquisition means for acquiring a recording start instruction, which is an instruction to start recording of the first image data, input in the first terminal apparatus or the second terminal apparatus;
With reference to the first timing at which the recording start instruction is acquired by the second acquisition means, after the first timing, an intra-frame encoded image forming the first image data is acquired by the first acquisition means. Third acquisition means for acquiring timing;
The first image corresponding to a period from the first timing to a second timing at which the intra-frame encoded image forming the first image data is first acquired by the first acquisition means after the first timing. Control is performed such that a complementary instruction that complements the data portion and instructs generation of complementary data having the data structure is transmitted to one specific terminal device of the first terminal device or the second terminal device. A second transmission means;
The specific terminal device that has received the complementary instruction acquires the complementary data that is generated from display data generated to display an image corresponding to the first image data and transmitted from the specific terminal device. A fourth acquisition means;
Recording means for controlling recording of the first image data from the second timing;
A synthesizing unit that synthesizes the complementary data acquired by the fourth acquisition unit and the recording data having the data structure recorded by the control of the recording unit according to a time series;
The first terminal device
Third transmission means for controlling the first image data to be transmitted to the conference server;
Seventh acquisition means for acquiring the second image data transmitted from the conference server;
Fourth transmission means for controlling the recording start instruction to be transmitted to the conference server;
First generation means for generating the complementary data in response to the complementary instruction transmitted from the conference server;
A fifth transmission means for controlling the complementary data generated by the first generation means to be transmitted to the conference server;
The second terminal device
Sixth transmission means for controlling the second image data to be transmitted to the conference server;
Eighth acquisition means for acquiring the first image data transmitted from the conference server;
Seventh transmission means for controlling the recording start instruction to be transmitted to the conference server;
Second generation means for generating the complementary data in response to the complementary instruction transmitted from the conference server;
A teleconference system comprising: ninth transmission means for controlling the supplementary data generated by the second generation means to be transmitted to the conference server. A terminal device connected to a network to which a conference server included in a remote conference system is connected, and performing a remote conference via the conference server,
First image data having a data structure including an intra-frame encoded image that is included at a predetermined time interval and can be independently decoded, and an inter-frame encoded image that can be decoded based on the intra-frame encoded image, A tenth transmission means for controlling to be transmitted to another terminal device via the conference server;
Ninth acquisition means for acquiring second image data transmitted from the other terminal device via the conference server;
An eleventh transmission means for controlling a recording start instruction, which is an instruction to start recording the first image data, to be transmitted to the conference server;
Generating means for generating the complementary data in response to a complementary instruction for instructing generation of complementary data for complementing the first image data transmitted from the conference server;
Twelfth transmission means for controlling the complementary data generated by the generation means to be transmitted to the conference server,
The supplemental data is acquired by the conference server first from the first timing when the recording start instruction is acquired by the conference server, and after the first timing, the intra-frame encoded image forming the first image data is acquired. A terminal device that is data that complements the portion of the first image data corresponding to a period up to the second timing.

Images