JP2008227968A - Terminal device in video conference system, control method of terminal device, and control program of terminal device - Google Patents

Abstract

<P>PROBLEM TO BE SOLVED: To provide an electronic conference which can smoothly reproduce voices and an images by using a peer-to-peer network. <P>SOLUTION: A terminal device 5 has data storing portions 11a-12c for storing a plurality of packets received from terminal devices 2, 3, and 4 for each of the terminal devices 2, 3, and 4, and a video reproducing portion 13 and a voice reproducing portion 14 reading from the data storing portions 11a-12c for each of the terminal devices 2, 3, and 4 a packet stored first in the data storing portions 11a-12c for each reproduction time of data included in the packet. Thus, the video reproducing portion 13 and the voice reproducing portion 14 can reproduce a packet without interruption. <P>COPYRIGHT: (C)2008,JPO&INPIT

Description

  The present invention relates to a terminal device for conducting a video conference using a plurality of terminals connected via a network, a control method for the terminal device, and a control program for the terminal device.

  In recent years, along with the rapid development of audio and video encoding technology, both voice and video can be exchanged at the same time by exchanging voice and video via a communication network such as the Internet Protocol (IP) in two or more remote locations. Attention has been focused on a multi-point electronic conference system that realizes a multi-point conference.

  In such a multipoint electronic conferencing system, even if conference participants are scattered in multiple locations, the audio and video of the conference are shared at the same time as if they were meeting together in one conference room. Can do. As a result, troubles and expenses such as business trips can be reduced, so that the multi-point electronic conference system is rapidly spreading mainly in companies.

  In addition, in such a multipoint electronic conference system, a communication network that transmits and receives audio and video data divided into a plurality of packets via the Internet or the like, that is, a packet communication network, is used for communication between conference participants. Sometimes. However, in communication via the Internet, if the communication speed decreases due to deterioration of the communication status or the like, the conference participants on the receiving side cannot receive packets continuously, or the arrival order of packets is switched. Problem arises. When such a problem occurs, the audio and video are interrupted, and the multipoint electronic conference cannot be smoothly advanced.

  Therefore, there is a technique for temporarily storing received packets to reduce interruptions in audio and video, and providing a buffer or the like that sequentially outputs the stored packets when a certain time has elapsed. This technique is described in, for example, Patent Documents 1 to 3 shown below.

  Patent Document 1 discloses a gateway device that stores and outputs voice packets received from a packet communication network in the order of reception. Specifically, it is disclosed that a buffer stores voice data and outputs voice data at regular time intervals to absorb variations in IP packet transfer time generated in the IP network.

  Further, Patent Document 2 includes a buffer memory that temporarily stores IP packetized audio frame data received via the Internet, and generates audio information based on the audio frame data included in the IP packet. Has been disclosed.

Also, in Patent Document 3, before storing received packets in the buffer, the buffer holds the received packets that have been input, and the received packets that have been received in order to rearrange the received packets whose arrival order has been changed as much as possible. A network telephone is disclosed in which, in the input order, at regular time intervals.
JP 2005-323029 A (published on November 17, 2005) Japanese Patent Laid-Open No. 10-210074 (released on August 7, 1998) JP 2004-260805 A (published September 16, 2004)

  By the way, a network form of communication between a plurality of terminal devices such as the multipoint electronic conference system includes a server that provides a specific process via the network and a terminal device that is a client requesting the process. There are a client server type configured and a peer-to-peer type in which a plurality of terminal devices have an equal relationship, and each terminal device directly exchanges data without using a server. FIG. 12 is a schematic diagram showing a state of communication using a client server type network form. FIG. 2 is an explanatory diagram of the present invention, and shows a schematic configuration of the video conference system of the present invention using a peer-to-peer network form.

  In the electronic conference using the client server type network form, as shown in FIG. 12, the video conference server 100 mixes the audio and video data received from each of the conference participants a, b, and c, and mixes them. The subsequent data is transmitted to each conference participant a, b, c, and the terminal devices of the conference participants a, b, c are configured to reproduce the transmitted audio and video data. For this reason, in the client-server type, packet data interruptions caused by the deterioration of the communication situation when each conference participant transmits audio and video data to the server and the audio and video data after the server mixes are transmitted to each conference. It is necessary to consider the packet interruption caused by the deterioration of the communication situation when transmitting to the participant. Therefore, it is necessary to provide buffers for reducing the interruption of the packet data and smoothly reproducing the data at two locations of the server and each conference participant.

  On the other hand, in an electronic conference over a peer-to-peer network, as shown in FIG. 2, since the relay server 6 does not perform a specific process, the relay server is transmitted from each conference participant A, B, C, D. The data is transmitted to each conference participant excluding the data sender without being mixed, and the terminal device of each conference participant who has received the data performs mixing and reproduction. For example, the terminal device of the conference participant A receives data from the conference participants B, C, and D via the relay server 6, but the relay server 6 does not process anything, so the conference participants B and C , D is the same as receiving data directly. Therefore, the terminal device of the conference participant A mixes the data from the conference participants B, C, and D, and then reproduces them.

  Therefore, when using the peer-to-peer network form, the relay server does not perform mixing, so that the server buffer required when using the client-server type network form becomes unnecessary.

  However, as described above, in the peer-to-peer type electronic conference, each conference participant directly transmits and receives data. Therefore, due to the deterioration of the communication status between the conference participants, each conference participant There arises a problem that each packet data is interrupted.

  Accordingly, the present invention has been made in view of the above-described conventional problems, and an object thereof is to provide an electronic conference that can smoothly reproduce audio and video using a peer-to-peer network form. There is.

  In order to solve the above problems, a terminal device according to the present invention is a terminal device in a video conference system in which a plurality of terminal devices connected via a peer-to-peer network mutually transmit and receive at least one of video and audio data. A data storage unit that stores a plurality of packets received from the terminal device for each communication partner terminal device, and the data storage unit that is the earliest in the data storage unit for each communication partner terminal device. Reproducing means for reading out and reproducing the stored packet at every reproduction time of the data contained in the packet is provided.

  Also, the terminal device control method according to the present invention is a terminal device control method in a video conference system in which a plurality of terminal devices connected via a peer-to-peer network mutually transmit and receive at least one of video and audio data. A storage step of storing, for each communication partner terminal device, a packet received from the terminal device in a data storage unit capable of storing a plurality of packets; and for each communication partner terminal device, storing the data And a reproduction step of reading out and reproducing the packet stored in the data storage unit earliest at every reproduction time of the data included in the packet.

  According to the above configuration, the terminal device according to the present invention includes the data storage unit that stores the packet received from the communication partner terminal device for each communication partner terminal device. Packets can be stored. That is, video and audio data from each conference participant participating in the video conference can be stored for each conference participant.

  Therefore, video and audio data from each conference participant can be reproduced without interruption. Therefore, a video conference using a peer-to-peer network can be performed smoothly.

  Furthermore, the terminal device according to the present invention includes a communication unit having a plurality of ports assigned to each of the communication partner terminal devices, and a packet input to each port from the communication partner terminal device. Is further provided with temporary storage control means for storing the data in a data storage unit corresponding to each of the terminal devices.

  According to the above configuration, the communication partner that is the transmission source of the received packet can be determined based on which port of the communication unit the received packet is received. Can be recognized.

  Thereby, the temporary storage control means can easily store the received packet in the data storage unit corresponding to the terminal device of the transmission source.

  Further, in the terminal device according to the present invention, the delay time until the reproduction unit reproduces the packet stored in the data storage unit first is determined in a communication environment with a plurality of communication partner terminal devices. According to the worst communication environment, there is provided temporary storage control means that is set in common to a plurality of terminal devices of the communication partner.

  According to the above configuration, the time from when the temporary storage control unit first stores the packet after the start of the video conference until the reproduction unit reproduces the packet depends on the communication partner having the worst communication environment. Set common to time.

  As a result, the common time for storing the received data can be set to the minimum necessary time for eliminating the delay caused in the received packet from the communication partner having the worst communication environment. Therefore, the video and audio data from each conference participant can be reproduced without interruption, and the time when the data is received and the data generated by storing the data in the data storage unit are reproduced. It becomes possible to minimize the time lag with time.

  The terminal device may be realized by a computer. In this case, a control program for the terminal device that causes the terminal device to be realized by the computer by causing the computer to operate as each of the above means falls within the scope of the present invention. enter.

  As described above, for each communication partner terminal device, the data storage unit that stores a plurality of packets received from the terminal device, and for each communication partner terminal device, from the data storage unit to the data storage unit Reproducing means for reading out and reproducing a packet stored earlier every reproduction time of data included in the packet.

  The terminal device control method according to the present invention includes a storage step of storing, for each communication partner terminal device, a packet received from the terminal device in a data storage unit capable of storing a plurality of packets; A reproduction step of reading out and reproducing from the data storage unit the packet stored earliest in the data storage unit at every reproduction time of the data included in the packet.

  Therefore, it is possible to provide an electronic conference that can smoothly reproduce audio and video using a peer-to-peer network form.

  An embodiment of the present invention will be described with reference to FIGS. 1 to 9 as follows.

  In this embodiment, a video conference system realized by transmitting and receiving video and audio data simultaneously and bidirectionally between a plurality of personal computers connected via a peer-to-peer network. explain.

  In the peer-to-peer network, a plurality of terminal devices are in an equal relationship, and each terminal device directly exchanges data without going through a server. The outline of the video conference via the peer-to-peer network will be specifically described with reference to FIG.

  As shown in FIG. 2, the audio and video data output from the terminal device of the conference participant A is transmitted to the conference participants B, C, and D via the relay server 6. The audio and video data output from the terminal device of the conference participant B is sent to the conference participants A, C and D, and the voice and video data from the terminal device of the conference participant C is sent to the conference participant A. , B, and D, audio and video data from the terminal device of the conference participant D are transmitted to the conference participants A, B, and C, respectively. Then, in the terminal device of the conference participant A that has received data from the conference participants B, C, and D, the audio and video data of the conference participant A and the received conference participants B, C, and D Mix and play back video data. The same operation is performed for the conference participants B, C, and D.

  In addition, since communication between each terminal apparatus in this Embodiment is packet communication using a packet, the said transmitted / received data is comprised from the packet divided | segmented into plurality.

  First, the structure of the video conference system 1 in this Embodiment is demonstrated using FIG. 1 and FIG.

  FIG. 1 is a functional block diagram illustrating a schematic configuration of the terminal device. FIG. 3 is an explanatory diagram showing the overall configuration of the video conference system 1.

  As shown in FIG. 3, the video conference system 1 includes terminal devices 2 to 5, a relay server 6, and a network 7 that are used by conference participants (hereinafter simply referred to as users) A to D, respectively.

  A personal computer is used for the terminal devices 2 to 5. A detailed description of the configuration of each terminal device will be given later.

  Further, since the relay server 6 is a server in a peer-to-peer network, the relay server 6 relays communication between terminal devices without performing specific management.

  The network 7 includes one or a plurality of communication networks such as the Internet, a public communication network such as a mobile phone network and a wired telephone network, and a non-public communication network such as a dedicated communication line. The network 7 is configured to divide and transfer data into a plurality of fixed-length packets using the Internet protocol (IP), and each device connected to the network 7 such as the terminal devices 2 to 5 includes , An IP address is assigned.

(Configuration of terminal device)
Next, although the structure of each terminal device 2-5 is demonstrated, here, the structure of the terminal device 5 which the user D uses is demonstrated using FIG. 1 and FIG.

  FIG. 4 is a diagram illustrating an example of a table stored in the input / output information storage unit 24.

  The video conference system 1 is a system in which a plurality of terminal devices 2, 3, 4, and 5 connected in a peer-to-peer network form perform a video conference by mutually transmitting and receiving at least one of video and audio data. It is.

  The terminal device 5 includes a communication unit 8, an input / output control unit 9, a cache control unit 10, a video cache unit 11, an audio cache unit 12, a video reproduction unit 13, an audio reproduction unit 14, and a video conference GUI. (Graphical user interface) 15, a camera 16, a microphone 17, a display unit 18, an operation unit 19, a speaker 20, and a storage unit 21.

  The communication unit 8 communicates with the terminal devices 2 to 4 via the relay server 6. Specifically, the communication unit 8 has a plurality of input / output ports (ports) assigned to the terminal devices 2, 3, and 4. Each input / output port is assigned to the relay server 6 as an interface for each connection established for each communication partner user and each data type to be communicated. The communication unit 8 outputs the input data to the video / audio input management unit 23 (FIG. 1) together with input / output port information indicating from which input / output port the data is input. The communication unit 8 relays the data input from the video / audio output management unit 22 (FIG. 1) via the input / output port specified by the input / output port information from the video / audio output management unit 22. Output to the server 6. Moreover, the communication part 8 is provided with the function to detect the environment of communication with each terminal device 2-4.

  The input / output control unit 9 controls data input / output to / from the terminal device 5 performed via the communication unit 8, and includes a video / audio output management unit 22 and a video / audio input management unit 23. .

  The video / audio output management unit 22 manages output of data input by the user D to another terminal device, and outputs data input from the camera 16 and the microphone 17 to the communication unit 8. . Specifically, the video / audio output management unit 22 specifies a user who transmits audio or video data input by a user D via a microphone or a camera, and a type of data to be transmitted. Then, the video / audio output management unit 22 refers to the table stored in the input / output information storage unit 24 (FIG. 1) based on the specified information, and specifies the user of the specified transmission destination and the type of data. Select the input / output port corresponding to. Then, information on the selected input / output port (input / output port information) and data input by the user D are output to the communication unit 8.

  For example, if it is set to transmit data to all the participants participating in the video conference, and the user D inputs only audio data, the video / audio output management unit 22 first selects the user. It is specified that audio data is output to all of A to C. Then, based on this information, the video / audio output management unit 22 refers to the table shown in FIG. 4 stored in the input / output information storage unit 24, and corresponds to the input / output port 1, 3 and 5 are selected. Then, the information of the selected input / output ports 1, 3, 5 is output to the communication unit 8.

  The video / audio input management unit 23 manages data input from the terminal devices 2 to 4 to the terminal device 5. Specifically, the video / audio input management unit 23 acquires data (received data) received from the terminal devices 2 to 4 and input / output port information when receiving the data from the communication unit 8. Then, the video / audio input management unit 23 refers to the table stored in the input / output information storage unit 24 based on the acquired input / output port information, and receives received data corresponding to the input / output port from the table. Get the information of the user who is the sender of and the type of received data. The video / audio input management unit 23 outputs the information on the transmission source user and the information on the type of received data acquired from the input / output information storage unit 24 to the cache control unit 10 together with the received data.

  For example, if the terminal device 5 receives certain data via the input / output ports 1 and 3 of the communication unit 8, the video / audio input management unit 23 inputs and outputs together with the received data from the communication unit 8. Input / output port information indicating that it has been received via ports 1 and 3 is acquired. Then, based on the input / output port information, the video / audio input management unit 23 reads from the table of the input / output information storage unit 24 the participant who is the transmission source of the received data corresponding to each of the ports 1 and 3, and Get data type information. Here, since the data is received via the input / output port 1, the data is audio data transmitted from the user A.

  The storage unit 21 is a memory, a hard disk, or the like that stores programs and data necessary for executing various functions of the terminal device 5, and includes an input / output information storage unit 24.

  The input / output information storage unit 24 stores information to be referred to in order to specify through which input / output port when the video / audio output management unit 22 outputs data from the terminal device 5. In addition, the video / audio input management unit 23 stores a terminal device that is a transmission source of the data input to the terminal device 5 and information that is referred to in order to identify the type of the input data. .

  Specifically, the input / output information storage unit 24 inputs / outputs port information indicating which input / output port is passed when data is input / output to / from the terminal device 5 and the input / output via each input / output port. Table in which information indicating which user is connected to which data is connected to information indicating whether the data input / output via each input / output port is video data or audio data Is stored. An example of this table is as shown in FIG.

  In the table of FIG. 4, the user information is indicated as A, B, and C. Actually, the video conference ID is assigned to each user of the video conference system, and the TV assigned to each user. Information on the conference ID is stored.

  The cache control unit 10 (temporary storage control means) controls the video cache unit 11 and the audio cache unit 12 based on information input from the video / audio input management unit 23 or based on an instruction from the video conference GUI 15. To do. Specifically, the cache control unit 10 corresponds to a packet input to each port of the communication unit 8 from each terminal device 2, 3, 4 to the terminal device 2, 3, 4 to which the port is assigned. The data is stored in data storage areas 11a, 11b, and 11c (described later) of the video cache unit 11 or data storage areas 12a, 12b, and 12c (described later) of the audio cache unit 12.

  In addition, after the video conference starts, the cache control unit 10 first stores the packet in the data storage area 11a, 11b, 11c, or 12a, 12b, 12c after the video playback unit 13 or the audio playback unit 14 stores the packet. , The time until the stored packet is replayed is determined according to the worst communication environment among the plurality of terminal devices 2, 3, and 4. Set in common.

  The video cache unit 11 temporarily stores data whose type of received data is video, and stores it for each user who is the transmission source of the received data.

  Specifically, as shown in FIG. 1, the video cache unit 11 includes a user A data storage area 11a for storing user A data, a user B data storage area 11b for storing user B data, and a user C. And a user C data storage area 11c for storing the data.

  The user data storage areas 11a, 11b, and 11c store a plurality of packets including video data received from the terminal devices 2, 3, and 4 for each of the terminal devices 2, 3, and 4.

  The voice cache unit 12 temporarily stores data in which the type of received data is voice, and stores it for each user who is the transmission source of the received data.

  Specifically, as shown in FIG. 1, the voice cache unit 12 includes a user A data storage area 12a for storing user A data, a user B data storage area 12b for storing user B data, and a user C. And a user C data storage area 12c for storing the data.

  The user data storage areas 12a, 12b, and 12c store a plurality of packets including voice data received from the terminal devices 2, 3, and 4 for the terminal devices 2, 3, and 4, respectively.

  Details of the cache control unit 10, the video cache unit 11, and the audio cache unit 12 will be described later.

  The video reproduction unit 13 (reproduction means) reproduces each user's video data output from the video cache unit 11 on the display unit 18 via the video conference GUI 15. In addition, the video reproduction unit 13 reproduces the video data input when the user D photographs the user D using the camera 16 on the display unit 18 via the video conference GUI 15.

  Specifically, for each of the terminal devices 2, 3, and 4, the data stored in the data storage units 11 a, 11 b, and 11 c from the data storage units 11 a, 11 b, and 11 c is reproduced as the data included in the packet. It is read and reproduced every time.

  For example, in the video conference screen 30 as shown in FIG. 9 displayed by the video conference GUI 15 when reproducing the video of the user A, the video playback unit 13 displays the video of the user A. Play video data. In addition, when the video playback unit 13 plays back the video of the user B, the video playback unit 13 plays back the video data of the user B in an area where the video of the user B is displayed on the video conference screen 30. When the user C video is played back, the video data of the user C is played back in the area where the video of the user C is displayed on the video conference screen 30.

  The audio reproduction unit 14 (reproduction unit) mixes each user's audio data output from the audio cache unit 12 into one continuous audio data, and causes the speaker 20 to reproduce the audio based on the audio data. It is. In addition, the voice reproduction unit 14 mixes the voice data of the user D input to the terminal device 5 using the microphone 17 with the voice data of other users and reproduces it as one voice data on the speaker 20.

  Specifically, for each of the terminal devices 2, 3, 4, the data stored in the data storage unit 12 a, 12 b, 12 c is reproduced from the data storage unit 12 a, 12 b, 12 c, and the data contained in the packet is reproduced. It is read out every time, and the audio data from each terminal device constituted by the read packet is mixed and reproduced by a speaker.

  The video conference GUI 15 displays a video conference screen 30 as shown in FIG. 9 on the display unit 18. Also, the video conference GUI 15 acquires various operation information since it is input by the user D using the operation unit 19 via the video conference screen 30.

  The video conference screen 30 is a screen for displaying the user participating in the video conference and his / her video on the display unit 18. Specifically, the video conference screen 30 is an operation for performing various settings in the video conference by the user D such as an area for displaying videos of other users A to C participating in the video conference and a cache size setting. And a button area. The user D can check other users participating in the video conference or his / her video via the video conference screen 30.

  For example, when the user D selects and reproduces only the voice of the user B, the user D uses the operation unit 19 to reproduce only the voice of the selected user on the video conference screen 30 “voice reproduction user selection button” ”Is selected, and an area in which the image of the user B is displayed is selected. Thereby, the video conference GUI 15 recognizes that the user D has selected the user B via the video conference screen 30. Then, the video conference GUI 15 notifies the sound reproduction unit 14 of the fact, and the sound reproduction unit 14 reproduces the sound of only the user B.

  Similarly, when the user D selects and reproduces only the video of the user A, the user D selects the “video reproduction user selection button” that reproduces only the video of the selected user, and displays the video conference screen 30. The area in which the video of user A is displayed is selected. Thereby, the video conference GUI 15 recognizes that the user D has selected the user A, and reproduces the video of only the user A via the video reproduction unit 13.

  The camera 16 captures the video of the user D into the terminal device 5. The microphone 17 is for capturing the voice of the user D into the terminal device 5.

  The display unit 18 displays the video conference screen 30 and is, for example, a PC display.

  The operation unit 19 is used when the user operates the terminal device 5 such as selecting a menu displayed on the video conference screen 30, and is, for example, a PC mouse.

  The speaker 20 outputs each user's voice.

  Next, the cache control unit 10, the video cache unit 11, and the audio cache unit 12 will be described in detail.

  Here, a case where the terminal device 5 acquires video data from the user A will be described.

  The cache control unit 10 receives the received data received from the terminal device 2, the video conference ID of the user A of the terminal device 2 that is the transmission source of the received data, and information that the received data is video data. Obtained from the video / audio input management unit 23.

  When acquiring information from the video / audio input management unit 23, the cache control unit 10 first sets a cache time. The cache time can be set automatically by the cache control unit 10 or manually by the user D. Specifically, as shown in FIG. 9, when the user D checks the automatic column, the video conference GUI 15 recognizes that the cache time is automatically set, and cache control that the cache time is automatically set. To the unit 10.

  At this time, the cache control unit 10 recognizes the terminal device having the worst communication environment out of the communication environments with the terminal devices 2 to 4 acquired from the communication unit 8 via the input / output control unit 9, and performs the communication. The cache time is set to be long so that data can be smoothly reproduced even in a terminal device having a poor environment.

  On the other hand, as shown in FIG. 9, when the user D sets the data cache size, that is, the cache time to a desired size via the video conference screen 30, the video conference GUI 15 caches the set desired cache time. Output to the control unit 10. Then, the cache control unit 10 sets the number of blocks in a regular storage area described later according to the cache time.

  The cache time here is assumed to be set to 5 blocks in a regular storage area to be described later.

  After the cache control unit 10 sets the cache time to 5 blocks in the regular area, the cache control unit 10 recognizes that the received data is video data, and selects the cache unit that stores the received data packet as the video cache unit 11. Next, since the cache control unit 10 recognizes that the received data is data from the user A based on the video conference ID of the user A, the cache control unit 10 selects the user A data storage area 11 a in the video cache unit 11. Then, the cache control unit 10 stores the received data packet received from the user A in the user A data storage area 11 a of the video cache unit 11.

  FIG. 5 is an explanatory diagram showing how packets are cached in the user A data storage area 11a.

  A packet caching method in the user A data storage area 11a will be described with reference to FIG.

  Although only the user A data storage area 11a has been described here, the other data storage areas 11b, 11c, and 12a to 12c have the same configuration.

  As shown in FIG. 5, the user A data storage area 11a has a configuration in which ten blocks for storing one packet are connected. Of these ten blocks, five blocks i to v on the side where the packet is output (right side as viewed in FIG. 5) are regular storage areas, which store packets, that is, receive data. Used for storage. The output of the packet means that the packet is read out from the video reproduction unit 13.

  On the other hand, five blocks on the packet input side (left side as viewed in FIG. 5) are spare storage areas, which are areas for storing packets overflowing from the regular storage area.

  In addition, since the number of blocks in the regular storage area in each of the data storage areas 11a to 11c and 12a to 12c is always a common number, the number of blocks in the regular storage area of the data storage areas 11b, 11c, and 12a to 12c is regular storage. The number is the same as the region 11a.

  Specifically, as shown in FIG. 5, when the cache control unit 10 receives four consecutive packets d to g at timing 6 (hereinafter, time t = 6), the cache storage unit 10 has already stored in the regular storage area. Since the packets b and c are stored, there are three blank blocks in the regular storage area. However, since four packets are received, the packets df that arrive earlier are included in the regular storage area, and the packet g that arrives the latest among the four packets overflows from the regular storage area. Since the spare storage area stores the overflowing packet g, it is possible to prevent data interruption caused by the storage leak of the packet g.

  Further, the cache control unit 10 first stores a packet that has arrived first since the start of the video conference, in a block i that is located closest to the packet input side of the regular storage area in the video cache unit 11. Then, for each count, the packets are sequentially moved to the packet output side by one block, that is, the packets are stored in blocks ii to iv. When the packet is stored in the block v on the most packet output side, the cache control unit 10 outputs the packet to the video reproduction unit 13 at the next count.

  From now on, a cache method in the video cache unit 11 will be described using the video data of the user A transmitted from the terminal device 2 and composed of nine packets a to i as shown in FIG. Further, when the nine consecutive packets a to i are interrupted due to deterioration of the communication environment between the terminal device 2 and the terminal device 5, the terminal device 5 receives the interrupted packet as shown in FIG.

  First, after starting the video conference, the cache control unit 10 firstly transmits the packet a of the video data of the user A that reaches the terminal device 5 among the five blocks in the regular storage area farthest from the packet output side. The data is stored in the block i, that is, the fifth block from the right side in FIG.

  In addition, the cache control unit 10 sets t = 1 as the time when the packet a that first arrived at the terminal device 5 after the start of the video conference is input to the cache control unit 10, and then at a predetermined time interval, t = 2 , T = 3, t = 4. Then, for each count, the stored packets are moved one by one to the output side block. Here, the predetermined time, that is, 1 count refers to the time for reproducing one packet.

  Since the cache control unit 10 moves one packet per count to the output side block, the cache control unit 10 counts t = 2 and simultaneously moves the packet a to the block ii located on the packet output side rather than the block i. .

  Further, the cache control unit 10 stores two consecutive packets b and c input at a count of t = 2 in the video cache unit 11. At this time, the cache control unit 10 stores the packet b in the block i in which the packet a is stored, and stores the continuous packet c in the block located on the most packet output side of the spare storage area.

  In addition, the cache control unit 10 counts t = 3 and simultaneously moves the packets a, b, and c to adjacent blocks on the packet output side. That is, packet a is stored in block iii, packet b is stored in block ii, and packet c is stored in block i.

  Further, the cache control unit 10 counts t = 4 and simultaneously moves each of the packets a, b, and c to adjacent blocks on the packet output side. That is, packet a is stored in block iv, packet b is stored in block iii, and packet c is stored in block ii. In addition, the cache control unit 10 counts t = 5, and simultaneously stores the packet a in the block v, the packet b in the block iv, and the packet a in the block iii.

  Then, the cache control unit 10 counts t = 6 and simultaneously outputs the packet a to the video reproduction unit 13 to move the packet b to the block v and the packet c to the block iv.

  At this time, packets d to g that have arrived at the terminal device 5 with a delay due to deterioration of the communication status or the like are input to the cache control unit 10.

  When the packets d to g are input, the cache control unit 10 stores the packet d in the block iii and stores the packets e and f in the blocks ii and i, respectively. Then, the packet g overflowing from the regular storage area is stored in a block located on the most packet output side of the spare storage area.

  Further, the cache control unit 10 counts t = 7, and at the same time outputs the packet b stored in the block v on the most packet output side to the video reproduction unit 13, blocks the packet c into the block v, and blocks the packet d. In iv, the packet e is moved to block iii and the packet f is moved to block ii for storage. Further, since the blank area is generated in the regular storage area for the output of the packet b, the cache control unit 10 stores the packet g in the block i of the regular storage area.

  At this time, the packets h and i that have arrived at the terminal device 5 with a delay due to deterioration of the communication status or the like are input to the cache control unit 10. Since there is no blank area in the regular storage area at the time of counting t = 7, the cache control unit 10 stores the packets h and i in order from the block located closest to the packet output side in the spare storage area.

  The cache control unit 10 counts each of t = 8 to 14, and simultaneously outputs the packets c to i stored in the block v at each timing to the video reproduction unit 13 and also to each block of the video cache unit 11. The stored packets are moved one by one to the block on the packet output side.

  According to the above control method, since the cache control unit 10 can control the timing of outputting a packet from the video cache unit 11, the data interrupted at the time of input to the video cache unit 11 is transferred from the cache control unit 10. Can be output as continuous data without interruption.

  Further, the user B data storage area 11b and the user C data storage area 11c have the same configuration as the user A data storage area 11a, and thus description thereof is omitted here. Further, the user A data storage area 12a, the user B data storage area 12b, and the user C data storage area 12c in the voice cache unit 12 have the same configuration as the user A data storage area 11a, and thus the description thereof is omitted here. To do.

(Operation of the entire video conference system)
Next, the operation of the entire video conference system 1 will be described with reference to FIG. FIG. 6 is a flowchart showing the overall operation of the video conference system 1.

  First, in order for the video conference system 1 to operate, any one of the plurality of terminal devices serves as a session manager, and transmits a video conference start request to other terminal devices participating in the video conference participation.

  When the user of the terminal device that has received the video conference start request participates in the video conference, a participation response notification is transmitted to the session manager. As a result, a connection is established between the terminal device and the session manager (S1). When the connection is established, a port is assigned to each terminal device with which the connection is established, and input / output port information is created (S1).

  Then, the video conference starts, and various video conference processes are performed between the terminal devices (S2). Details of the various video conference processes will be described later.

  Further, when one of the users participating in the video conference, for example, the user A makes a video conference end request to the users B to D and receives a video conference end response notification from all the users B to D, The conference ends (YES in S3). Then, the connection between the users is disconnected (S4).

  If no video conference end response notification is received from all users, the video conference is not ended (NO in S3), and the video conference is continued.

  Here, details of various video conference processes will be described below.

(Processing of the terminal device when receiving data)
First, the processing operation of the terminal device 5 when data is received from another terminal device will be described.

  FIG. 7 is a flowchart showing the processing operation of the terminal device 5 when data is received from another terminal device in the video conference process.

  As illustrated in FIG. 7, when the terminal device 5 receives data from the other terminal devices 2 to 4 via the communication unit 8, first, the input / output port of the communication unit 8 determines which data has been input. The input / output port information shown and the input data are input to the video / audio input management unit 23 (FIG. 1).

  The video / audio input management unit 23 specifies the user who is the transmission source of the received data from the input / output information storage unit 24 based on the input / output port information (S11) and specifies the type of the received data (S12).

  The video / audio input management unit 23 outputs to the cache control unit 10 the user who is the transmission source of the specified received data and the type of the received data.

  First, the cache control unit 10 automatically sets the cache time for received data (YES in S13). As shown in FIG. 9, this is because the video conference GUI 15 caches, and if the cache control unit 10 does not automatically set the cache time (NO in S 13), the user D caches via the video conference GUI 15. Time is set (S18).

  Then, if the received data is video data from user A (YES in S14), the cache control unit 10 stores the received data in the user A data storage area 11a of the video cache unit 11. The video cache unit 11 caches video data that is received data based on the cache time set by the cache control unit 10 (S15, storage step).

  When the cache time has elapsed, the cache control unit 10 causes the video data to be output from the video cache unit 11 to the video playback unit 13 (S16).

  Then, the video reproduction unit 13 reproduces the input video data on the display unit 18 via the video conference GUI 15 (S17, reproduction step).

  If the received data is voice data from the user A (NO in S14), the cache control unit 10 stores the received data in the user A data storage area 12a of the voice cache unit 12. The voice cache unit 12 caches voice data that is received data based on the cache time set by the cache control unit 10 (S19, storage step).

  When the cache time elapses, the cache control unit 10 causes the audio data to be output from the audio cache unit 12 to the audio reproduction unit 14 (S20).

  Then, when the audio data of the user B or the user C is obtained by the same processing operation as described above, the audio reproducing unit 14 mixes the audio data of the user A input in S20 and the audio data of the user B or the user C. (S21). The audio data that has become one continuous data by the mixing in S21 is reproduced via the speaker 20 (S22, reproduction step).

(Processing of terminal device during data transmission)
Next, the processing operation of the terminal device 5 when the user D captures his / her video and audio as data in the terminal device 5 and transmits the video data and audio data to the other terminal devices 2 to 4 will be described.

  FIG. 8 is a flowchart showing a processing operation when video or audio data is transmitted from the terminal device 5 to another terminal device in the video conference process.

  First, when the video of the user D is captured by the terminal device 5 via the camera 16 (YES in S31), and the captured video is transmitted to the other terminal devices 2 to 4 (YES in S32), the terminal device 5 inputs video data taken into the video / audio output management unit 22.

  Then, the video / audio output management unit 22 recognizes the transmission destination user (S33). Specifically, if there is a designation from the user D, the user of the transmission destination is recognized according to the instruction, and if it is set to transmit to all users, the transmission destination is recognized as all users.

  The video / audio output management unit 22 refers to the input / output information storage unit 24 based on the transmission destination user recognized in S33, the type of input data (S34), and transmits the data to each terminal device. The input / output port to be used is specified (S35).

  Then, the video / audio output management unit 22 outputs the specified input / output port information to the communication unit 8 together with the video data. The communication unit 8 transmits video data to each terminal device based on the input / output port information (S36).

  Also, when capturing audio data via the microphone 17 without capturing video via the camera 16 (NO at S31) (YES at S37), when transmitting the captured audio data to another terminal device, , The same operation as S32 to S36 is performed.

  When displaying video data or audio data captured in the terminal device 5 via the camera 16 or the microphone 17 via the video conference GUI 15 on the display unit 18 of the terminal device 5 (NO in S32). The camera 16 outputs the video data to the video reproduction unit 13, and the microphone 17 outputs the audio data to the audio reproduction unit 14 (S38).

  Then, the video reproduction unit 13 reproduces the video of the user D on the display unit 18 via the video conference GUI 15 (S39). On the other hand, if there is other user's voice data, the voice reproducing unit 14 mixes the voice data of the user D with the voice data and outputs it through the speaker 20 (S39).

  In the present embodiment, the cache control unit 10 is configured to set the number of blocks in the regular storage area according to the worst communication environment among the communication environments with each terminal device, but the present invention is not limited to this. If the communication environment with any terminal device is good, the number of blocks that can be communicated with the smallest amount of communication can be set.

  For example, when the communication environment with all the terminal devices is good and the delay time of the packet d with respect to the packet c in FIG. 5 is two timings, the number of blocks in the regular storage area is four blocks ii to v. Can be set. In this case, packet a is first stored in block ii instead of block i.

  In the present embodiment, each of the data storage areas 11a to 11c and 12a to 12c includes a regular storage area and a spare storage area. However, the present invention is not limited to this, and corresponds to a regular storage area. It may consist only of things.

  In the present embodiment, the number of blocks in each of the data storage areas 11a to 11c and 12a to 12c is set to 10. However, the present invention is not limited to this and may be set to any number.

  In the present embodiment, the packets are arranged and stored in the order of the received packets, and the stored packets are sequentially moved to the packet output side and output when the packet is most output. The cache method of the video cache unit 11 or the audio cache unit 12 by the cache control unit 10 of the present invention is not limited to the cache method described above. For example, an index may be given to the input packets in the order of input, and the video reproduction unit 13 may read the packets in the order of the index numbers. In this case, unlike the present embodiment, it is not necessary to store the packets in the order of input.

  In the present embodiment, the case where the audio or video data input by the user D is transmitted to all participants has been described. However, the present invention is not limited to this, and the transmission destination specified by the user D is not limited to this. Only the transmission may be possible. For example, when the user D designates the data transmission destination as the users B and C, the user D uses the operation unit 19 to display the area where the user B is displayed on the video conference GUI 15 and the user C. By selecting the area, the video conference GUI 15 recognizes that the data transmission destinations are designated as the users B and C. Then, the video conference GUI 15 outputs an instruction to set the data transmission destination to the users B and C to the video / audio output management unit 22. When the video data is input from the camera, the video / audio output management unit 22 receives an instruction from the video conference GUI 15, and the participants are the users B and C and the data type is video from the input / output information storage unit 24. Information on input / output ports, that is, input / output ports 4 and 6, which are data, is acquired and output to the communication unit 8. The video data may be transmitted to the users B and C via the input / output ports 4 and 6 based on the input / output port information acquired by the communication unit 8.

  In addition, the video cache unit 11 and the audio cache unit 12 in the present embodiment are configured to have no data storage area for caching the video data and audio data of the user D, but the video cache unit 11 and the audio cache unit 12 may each have a user D data storage area.

  Further, for example, when the user E participates in the video conference from the middle, the cache control unit 10 stores the data from the user E in the video cache unit 11 and the audio cache unit 12. You may make it create.

  The present invention can also be applied to SOBA (registered trademark) (Session Oriented Broadband Application).

  SOBA provides a mechanism for users of terminal devices connected via a communication network to realize real-time communication. Examples of real-time communication include a chat function, a communication function such as a whiteboard function, a communication function using voice or video, a remote screen operation function, and the like. In SOBA, the range of communication connections shared by the communication functions as described above is called a session.

  FIG. 10 shows a state in which the terminal device 201 and the terminal device 202 on which the SOBA is mounted are connected via a communication network. Here, it is assumed that both are equal and can transmit and receive data to each other.

  In each of the terminal device 201 and the terminal device 202, three layers of an application layer, a session layer, and a service layer are formed. The application layer is a layer configured by applications running on SOBA. This application layer includes, for example, a component application and a screen sharing application. More specific applications include a chat function application, a whiteboard function application, and an audio / video communication function application.

  The session layer is a layer that manages session state and data distribution destinations.

  The service layer is a layer configured by functions for realizing a communication function by SOBA. Examples of the service layer include a synchronization service function, a resource service function, a directory service function, a security service function, and a network service function. The synchronization service function is a function that controls synchronization of communication with the terminal devices included in the session. The resource service function is a function for controlling voice and moving image data. The directory service function is a function that provides a directory for searching for a communication destination terminal device or user. The security service function is a function that provides an access permission / authorization function, a communication data encryption function, and the like. The network service function is a function for performing processing required for performing communication on a communication network.

  Data that is communicated by SOBA includes component application data, screen sharing application data, and session control data. The component application data is data that notifies the component application state change event. The screen sharing application data corresponds to transmission data for which a transmission instruction is issued in the screen sharing application, data for notifying a state change event of the screen sharing application, and the like. The session control data is data for notifying a session state change event.

FIG. 11A shows a packet configuration when component application data flows on the communication network. As shown in the figure, the packet is composed of a TCP / IP header, a SOBA header, a component application header, and component application data. The TCP / IP header is header information required when communication based on TCP / IP is performed, and is added by a network service function in the service layer. The SOBA header is header information added by the network service function in the service layer, and represents that the data is based on the SOBA protocol. The component application header is header information added by the component application in the application layer. Yes, it shows information for identifying a component application in which data included in the packet is used. The component application data is data for notifying the component application state change event as described above.

  FIG. 11B shows a packet configuration when the screen sharing application data flows on the communication network. As shown in the figure, the packet is composed of a TCP / IP header, an SOBA header, a screen sharing application header, and screen sharing application data. The TCP / IP header and SOBA header are as described above. The screen sharing application header is header information added by the screen sharing application in the application layer, and indicates information specifying a screen sharing application in which data included in the packet is used. As described above, the screen sharing application data is transmission data for which a transmission instruction is issued in the screen sharing application, and data for notifying a state change event of the screen sharing application.

  FIG. 11C shows a packet configuration when session control data flows on the communication network. As shown in the figure, the packet is composed of a TCP / IP header, a SOBA header, a session header, and session control data. The TCP / IP header and SOBA header are as described above. The session header is header information added by the session layer, and indicates information specifying the type of session control. As described above, the session control data is data for notifying a session state change event.

  For example, when a state change event is generated by a component application in the terminal device 201, component application data and a component application header are first generated in the component application. Thereafter, in the service layer, the SOBA header and the TCP / IP header are disabled, and the packet shown in FIG. 11A is transmitted to the terminal device 202 via the communication network. When the terminal device 202 receives the packet, the TCP / IP header is removed in the service layer, and the SOBA header is removed after the SOBA header is confirmed. Then, the service layer confirms the component application header, and transmits the component application data and the component application data in the component application header and the component application to the corresponding component application. In the component application layer, the state change event is recognized based on the received component application header and component application data, and processing corresponding to this is performed. Similar processing is performed for the screen sharing application data and the session control data.

  Various functions can be realized by installing the SOBA system as described above in each of the terminal devices 2 to 5 of the video conference system 1 shown in FIG.

(Configuration example by software)
Finally, each block included in the terminal device 5, particularly the input / output control unit 9, the cache control unit 10, the video cache unit 11, the audio cache unit 12, the video playback unit 13, the audio playback unit 14, and the video conference GUI 15, etc. Each functional block may be configured by hardware logic, or may be realized by software using a CPU as follows.

  That is, the terminal device 5 includes a CPU (central processing unit) that executes instructions of a control program for realizing each functional block, a ROM (read only memory) that stores the program, and a RAM (random access memory) that expands the program. ), A storage device (recording medium) such as a memory for storing the program and various data. An object of the present invention is to provide a recording medium in which a program code (execution format program, intermediate code program, source program) of a control program, which is software for realizing the functions described above, is recorded so as to be readable by a computer. This can also be achieved by reading the program code recorded on the recording medium and executing it by the computer (or CPU or MPU).

  Examples of the recording medium include tapes such as magnetic tapes and cassette tapes, magnetic disks such as floppy (registered trademark) disks / hard disks, and disks including optical disks such as CD-ROM / MO / MD / DVD / CD-R. Card system such as IC card, IC card (including memory card) / optical card, or semiconductor memory system such as mask ROM / EPROM / EEPROM / flash ROM.

  The terminal device 5 may be configured to be connectable to a communication network, and the program code may be supplied via the communication network. The communication network is not particularly limited. For example, the Internet, intranet, extranet, LAN, ISDN, VAN, CATV communication network, virtual private network, telephone line network, mobile communication network, satellite communication. A net or the like is available. Also, the transmission medium constituting the communication network is not particularly limited. For example, even in the case of wired such as IEEE 1394, USB, power line carrier, cable TV line, telephone line, ADSL line, etc., infrared rays such as IrDA and remote control, Bluetooth ( (Registered trademark), 802.11 wireless, HDR, mobile phone network, satellite line, terrestrial digital network, and the like can also be used. The present invention can also be realized in the form of a computer data signal embedded in a carrier wave in which the program code is embodied by electronic transmission.

  The present invention is not limited to the above-described embodiments, and various modifications can be made within the scope of the claims. That is, embodiments obtained by combining technical means appropriately changed within the scope of the claims are also included in the technical scope of the present invention.

  The terminal device according to the present invention can be applied to a video conference or the like using a peer-to-peer network form.

It is a functional block diagram which shows one Embodiment of the terminal device in this invention. It is the block diagram which showed schematically the whole video conference system in this invention. It is explanatory drawing which shows the mode of communication of the whole video conference system in this invention. It is a figure which shows an example of the table stored in the memory | storage part in this invention. It is explanatory drawing which shows the mode of the cache with the terminal device in this invention. It is a flowchart which shows the operation | movement of the whole video conference system in this invention. It is a flowchart which shows the operation | movement at the time of data reception of the terminal device in this invention. It is a flowchart which shows the operation | movement at the time of data transmission of the terminal device in this invention. It is a figure which shows an example of the video conference screen in this invention. It is a figure which shows the state in which the terminal device carrying SOBA is connected via the communication network. (A) is a figure which shows the packet structure when component application data flows on a communication network, (b) is a figure which shows the packet structure when screen sharing application data flows on a communication network, c) is a diagram showing a packet configuration when session control data flows on a communication network. It is the schematic which shows the mode of communication using the conventional client server type | mold network form.

Explanation of symbols

DESCRIPTION OF SYMBOLS 1 Video conference system 2, 3, 4, 5 Terminal device 6 Relay server 7 Network 8 Communication part 9 Input / output control part 10 Cache control part (temporary storage control means)
11a, 11b, 11c, 12a, 12b, 12c Data storage area (data storage unit)
13 Video playback unit (playback means)
14 Audio playback unit (playback means)
15 Video conference GUI
16 camera 17 microphone 18 display unit 19 operation unit 20 speaker 21 storage unit 22 video / audio output management unit 23 video / audio input management unit 24 input / output information storage unit

Claims (5)

A plurality of terminal devices connected via a peer-to-peer network are terminal devices in a video conference system in which at least one of video and audio data is mutually transmitted and received,
A data storage unit for storing a plurality of packets received from the terminal device for each communication partner terminal device;
Reproducing means for reading out and reproducing the packet stored earliest in the data storage unit from the data storage unit at every reproduction time of data included in the packet, for each terminal device of the communication partner A terminal device. A communication unit having a plurality of ports assigned to each terminal device of the communication partner;
Temporary storage control means for storing a packet input from the communication partner terminal device to each port in a data storage unit corresponding to each terminal device to which each port is allocated. The terminal device according to claim 1.   A delay time until the reproduction unit reproduces the packet stored in the data storage unit first is set in a plurality of communication environments with a plurality of communication partner terminal devices according to the worst communication environment. The terminal device according to claim 1, further comprising temporary storage control means that is commonly set for the terminal device of the communication partner. A terminal device control method in a video conference system in which a plurality of terminal devices connected via a peer-to-peer network mutually transmit and receive at least one of video and audio data,
A storage step of storing a packet received from the terminal device for each communication partner terminal device in a data storage unit capable of storing a plurality of packets;
A reproduction step of reading out and reproducing from the data storage unit the packet stored earliest in the data storage unit at each reproduction time of the data included in the packet, for each terminal device of the communication partner. A method for controlling the terminal device. A control program for a terminal device in a video conference system in which a plurality of terminal devices connected via a peer-to-peer network mutually transmit and receive at least one of video and audio data,
A storage step of storing a packet received from the terminal device for each communication partner terminal device in a data storage unit capable of storing a plurality of packets;
For each terminal device of the communication partner, a reproduction step of reading out and reproducing from the data storage unit the packet stored earliest in the data storage unit at every reproduction time of data contained in the packet is executed on the computer Control program to let you.

Images