JP4994320B2 - Group communication device and group communication program - Google Patents

Description

  The present invention relates to a group communication apparatus and a group communication program for controlling a mixing time according to a change in a situation of a user participating in group communication in group communication in which different types of terminals such as mobile terminals and fixed terminals are mixed.

2. Description of the Related Art Conventionally, a media synchronization system is known that can acquire data of a plurality of types of media via the Internet and perform synchronized playback with a small CPU load (see, for example, Patent Document 1).
In addition, user-specific data and content corresponding to public content displayed on a large screen monitor installed on the street etc. can be individually transmitted in response to user requests, and even different network environments and playback platforms Also in the above, a content distribution apparatus that can distribute and reproduce media data of public content and individual user content while synchronizing them is also known (for example, see Patent Document 2).
JP 2005-006037 A JP 2006-244060 A

However, in the media synchronization system disclosed in Patent Document 1, transmission and reception of a plurality of contents such as video and audio can be received by the same network and the same terminal, and efficient synchronization between contents can be realized. However, there is a problem that it is difficult to synchronize in different networks and different terminals.
Further, in the content distribution apparatus shown in Patent Document 2, it is possible to achieve content synchronization between different networks and different terminals, but the user terminal moves during reception of the content, and the connection to the network is achieved. There is a problem that it is difficult to follow synchronization when the environment changes.

  The present invention has been made in view of such circumstances, and even when the communication terminal used by the user or the network to which the terminal is connected changes, the mixing time of the media data is dynamically changed. It is an object of the present invention to provide a group communication apparatus and a group communication program that can realize synchronization in the same kind of media by controlling to the above.

The present invention is a group communication device that establishes a communication line with a plurality of user terminals and performs information communication using media data in which priority is set for each media type between the user terminals. A data receiving means for receiving media data from each of the terminals, a maximum data arrival delay time is obtained from the data arrival delay time of each of the received media data, and a difference between the maximum data arrival delay time and another data arrival delay time is A determination unit that determines that the correction of the mixing time is necessary when the time is equal to or longer than the predetermined time, and a medium of a medium type having a high priority when the determination unit determines that the correction of the mixing time is necessary In the data generation time (T ^Mix ), the media data generation time (T ^Si x) of the media type with the lower priority is selected. Based on the nearest high priority media type of the media data Mixing time ^{(M Mi} x), mixing the ideal time to ^{(I Si x),} determined by the ^{+ I Si x = M Mi x} (T Si x-T Mi x) a correction means for the data arrival time a ^{Si x} and the mixing ideal time I ^{Si x} and the mixing time the slower time by comparing, on the basis of the mixing time corrected by said correction means, the received Mixing means for mixing media data, and distribution means for distributing the media data mixed by the mixing means to each of the user terminals.

The present invention relates to a computer on a group communication device that establishes a communication line with a plurality of user terminals and performs information communication using media data in which priority is set for each media type between the user terminals. A group communication program for performing a process for correcting a mixing time of media data required at the time of communication, comprising: a data reception step of receiving media data from each of the user terminals; and a data arrival delay time of each of the received media data A determination step of obtaining a maximum data arrival delay time and determining that correction of mixing time is necessary when a difference between the maximum data arrival delay time and another data arrival delay time is equal to or greater than a predetermined time; and If the correction of the mixing time is determined to be necessary, the priority Among the high media type of the media data generation time (T ^{Mi x),} mixing the nearest high priority media type of the media data to the generation time (T ^{Si x)} of the media data low priority media type Based on the time (M ^Mix ), the mixing ideal time (I ^Si x) is obtained by I ^Si x = M ^Mix x (T ^Si x−T ^Mix x), and the data arrival time A ^Si x and the mixing ideal time I are calculated. a correction step of the ^{Si x} and the mixing time the slower time by comparing, on the basis of the mixing time corrected by said correction step, and mixing step of mixing the media the received data, by said mixing step A distribution step of distributing the mixed media data to each of the user terminals; It is characterized by having a computer perform.

  According to the present invention, when performing group communication, the media data mixing time is dynamically controlled even when the communication terminal used by the user or the network to which the terminal is connected changes according to the user's situation. Therefore, the same kind of media data can be prevented from being out of synchronization.

  Hereinafter, a group communication apparatus according to an embodiment of the present invention will be described with reference to the drawings. FIG. 1 is a block diagram showing the configuration of the embodiment. In this figure, the code | symbol S is a management server which implement | achieves bi-directional information communication between several user terminals. Reference numerals A1, A2, B1, B2, C1, and C2 are user terminals (hereinafter simply referred to as terminals) used by the user A, the user B, and the user C, respectively, and bidirectional information communication between the terminals of each user. I do. Here, an example of three users is shown, but four or more users may be used. In addition, each of users A, B, and C shows an example in which two terminals are used. However, the user only needs to use at least one terminal, and one user uses three or more terminals. Also good. Here, as an example, one of the two terminals of each of the users A, B, and C is a terminal (for example, a personal computer) having a function (display) for displaying an image and a function (camera) for capturing an image. The other will be described on the assumption that the sound collection function (microphone) and sound generation function are terminals (for example, mobile phone terminals) having (speakers). Reference numeral N denotes an IP communication network to which six terminals A1, A2, B1, B2, C1, C2, and the management server S are connected. Each of the six terminals A1, A2, B1, B2, C1, and C2 may be always connected to the IP communication network N, or connected by performing a connection operation to the IP communication network N as necessary. It may be. Further, although the IP communication network N shown in FIG. 1 is expressed as one communication network, a configuration in which a plurality of IP communication networks are connected to each other may be employed.

  Next, a basic operation when performing group communication will be described with reference to FIG. Here, as an application example of group communication, a description will be given assuming that a video conference by a plurality of users is performed. The management server S realizes a function of receiving media data such as video and audio from each terminal, mixing each received media data, and distributing the mixed media data to each terminal. When the user activates the terminal, each terminal transmits user information of each terminal to the management server S. The user information includes a user name, an address (IP address, etc.) of each terminal on the network, a type of media data to be transmitted / received (video, audio, etc.), and media data priority information.

  Next, when the management server S acquires user information from each terminal, the management server S stores the user information in an internal user management table, and groups terminals used by each user based on the user name. Then, each terminal transmits the generation time and data output time of media data with high priority to the management server. The priority is determined in advance by the user. For example, the priority of video is set high and the priority of audio is set low. When the management server S acquires the information of the media data, the management server S compares the data output delay (time from the generation of the media data to the output) of the media data close to the generation time. As a result of the comparison, a difference between other media data is calculated in accordance with the media data having the maximum data output delay. The difference is transmitted to the corresponding terminal. Each terminal controls the output time of media data by sliding the output interval of media data based on the information.

  Next, the management server S acquires from the user management table the addresses on the network of other terminals (terminals that transmit / receive low-priority media data) used by the same user as the terminals that transmit / receive high-priority media data. To do. The generation time and output time of the corresponding high-priority media data are transmitted to the acquired address. The terminal that receives the information controls the output time of the media based on the information.

  On the other hand, each terminal transmits the changed information to the management server S when the user information is changed. For example, when a video conference is held by a plurality of users, a certain user goes out, so that only the user changes to voice communication using a mobile phone. At this time, when the priority of the voice communication is high, the management server stops the video communication with the low priority, and stops transmitting the generation time and output time of the voice data to the terminal. On the other hand, when the priority of video communication is high, the generation time and output time to be referenced disappear. Through the processing operation of the management server S, a video conference using group communication is performed.

  Here, with reference to FIG. 3, the loss of synchronization of media data in group communication will be described. When a voice media data packet is transmitted from one user's terminal (for example, terminal A2) to the other user's terminal (for example, terminal B2) (see FIG. 3 (a)), the mth packet is the transmitting terminal. It is assumed that a loss of synchronization occurs when the time A (this is referred to as an output delay) from generation to A2 to output at the receiving terminal B2 exceeds 150 ms. In addition, an interval time B (this is referred to as a generation interval) at which a continuous packet is generated at the terminal A2 on the transmission side, and an interval time C (this is referred to as a generation interval) at which this continuous packet is output at the terminal B2 on the reception side An out-of-synchronization occurs when a difference D (C−B) from the output interval exceeds 160 ms.

  Also, when transmitting image and audio media data packets from one user's terminal (for example, terminals A1 and A2) to the other user's terminal (for example, terminals B1 and B2) (see FIG. 3B). The time B (generation interval) between the time when the image data is generated at the terminal A1 on the side, the time when the audio data is generated at the terminal A2 to be synchronized with this image, and the two at the terminal B2 on the reception side It is assumed that loss of synchronization occurs when the difference D (C−B) between the interval C at which the packet is output and the time C (output interval) exceeds 160 ms. The present invention controls the output time of media data so as not to cause such a loss of synchronization.

  Next, with reference to FIG. 3, an operation in which the management server S shown in FIG. 1 performs mixing of media data to be distributed to each terminal without causing loss of synchronization will be described. Here, it is assumed that the user A transmits / receives video media data using the terminal A1, and transmits / receives audio media data using the terminal A2. Similarly to the user A, the users B and C also transmit and receive video media data by the terminals B1 and C1, and transmit and receive audio media data by the terminals B2 and C2. In the following description, it is assumed that the video medium is a master medium, the audio medium is a slave medium, and the master medium has higher priority than the slave medium.

  First, when the management server S receives audio or video media data from each of the terminals A1, A2, B1, B2, C1, and C2 of the users participating in the group communication (step S1), the data arrives for each media data. Find the delay time. The data arrival delay time is the time from when each media data is generated in the terminal until it arrives at the management server S. Then, the management server S obtains the maximum data arrival delay time among the obtained data arrival delay times, and calculates the difference between the maximum data arrival delay and the data arrival delay time of other media (step S2).

  Next, the management server S determines whether or not it is necessary to correct the mixing time based on the difference between the obtained data arrival delay times (difference between the maximum data arrival delay and the data arrival delay time of other media). Determine (step S3). In this determination, if the obtained difference is equal to or greater than a threshold value (for example, 80 ms), it is determined that the mixing time of the master medium needs to be corrected. This determination is performed for all master media data (here, video data) to be mixed.

  As a result of this determination, if it is not necessary to correct the mixing time, the subsequent steps S4 and S5 are not executed. On the other hand, when it is necessary to correct the mixing time, the management server S corrects the mixing time of the master media data (step S4). In this correction, the mixing time of the master media data with a small data arrival delay time is delayed so that the difference between the maximum data arrival delay time and the data arrival delay time of the master media with a small data arrival delay time becomes zero. This is done by correcting. That is, the mixing time is corrected so that the mixing is performed in accordance with the latest master media data.

  Next, the management server S corrects the mixing time of the slave media data (step S5). The slave media data corrects the mixing time in accordance with the mixing time of the master media data transmitted by the terminal of the same user. For example, when the terminal A1 transmits the master medium and the terminal A2 transmits and receives the slave medium, the mixing time of the slave media data is corrected in accordance with the mixing time of the terminal A1.

The slave media data mixing time is corrected as follows. First, in the generation time of the master media data ^{(T Mi} _x), the generation time of the slave media data ^{(T Si} _x) to the nearest master media data Mixing time ^{(M Mi} _x), utilizing the media synchronizing To determine the ideal mixing time (I ^Si _x ). The ideal mixing time (I ^Si _x ) is obtained by the equation (1).
_{^{I Si x = M Mi x +}} (T Si x -T Mi x) ··· (1)
However, before mixing time of the slave media data, condition _{^{| T Si x -T Mi x |}} > | T Si x -T Mi x + y | when satisfying modifies the ideal time by equation (2).
_{^{I Si x = M Mi x +}} y + (T Si x -T Mi x + y) ··· (2)

Next, the data arrival time A ^Si _x in the management server S is compared with the ideal time I ^Si _x , and the mixing correction time (m ^Si _x ) is determined by setting the maximum value according to the equation (3) as the mixing correction time. To do.
m ^Si _x = max (I ^Si _x , A ^Si _x ) (3)

  Next, the management server S mixes each of the master media data and slave media data according to the corrected mixing time (step S6), and distributes the mixed media data to each terminal (step S7). . The management server S repeats the processing operation shown in FIG. 2 until the group communication ends. Since the processing operation shown in FIG. 2 is repeatedly performed, the arrival delay time difference is obtained for each repeated processing to determine whether or not the mixing time needs to be corrected, and if the correction is necessary, the mixing time is corrected. Even if the number of terminals increases or decreases during processing, the mixing time is corrected whenever necessary, so that group communication can be performed without causing loss of synchronization.

  If the terminal that transmits / receives the master media is disconnected due to changes in the user's surroundings, if there are one or more terminals that transmit / receive the master media, the mixing is first performed based on the mixing time of the master media data. Accordingly, the mixing time of the slave media data may be corrected accordingly. In addition, when a terminal that transmits / receives master media data is disconnected and one or more terminals that transmit / receive slave media exist, the management server S assumes that slave media data is master media data when mixing slave media data. The mixing may be performed based on the mixing time of the master media data.

  As described above, even when the terminal used by the user or the network to which the terminal is connected changes according to the user's situation, the mixing time of the media data is dynamically controlled. Can be prevented. In addition, according to the data arrival delay time of each medium, the mixing time of other media is dynamically corrected and controlled in accordance with the mixing time of the media data to be output preferentially. Can be suppressed.

  As described above, in group communication using a plurality of different types of terminals, even when the arrival time of each media data is different, it is possible to realize mixing time synchronization while maintaining real-time characteristics, Since media data can be received without a sense of incongruity, good information communication can be performed. In addition, synchronization of different contents between different terminals can be realized, and the user can use various terminals. In addition, even when the terminal used by the user or the network to be connected changes, it is possible to achieve synchronization between different types of content while maintaining real-time characteristics, and the user can enjoy content without any discomfort. Can be received.

  Note that a group communication is performed by recording a program for realizing the function of the management server S shown in FIG. 1 on a computer-readable recording medium, causing the computer system to read and execute the program recorded on the recording medium. Media data mixing control processing may be performed. Here, the “computer system” includes an OS and hardware such as peripheral devices. The “computer-readable recording medium” refers to a storage device such as a flexible medium, a magneto-optical disk, a portable medium such as a ROM and a CD-ROM, and a hard disk incorporated in a computer system. Further, the “computer-readable recording medium” refers to a volatile memory (RAM) in a computer system that becomes a server or a client when a program is transmitted via a network such as the Internet or a communication line such as a telephone line. In addition, those holding programs for a certain period of time are also included.

  The program may be transmitted from a computer system storing the program in a storage device or the like to another computer system via a transmission medium or by a transmission wave in the transmission medium. Here, the “transmission medium” for transmitting the program refers to a medium having a function of transmitting information, such as a network (communication network) such as the Internet or a communication line (communication line) such as a telephone line. The program may be for realizing a part of the functions described above. Furthermore, what can implement | achieve the function mentioned above in combination with the program already recorded on the computer system, what is called a difference file (difference program) may be sufficient.

It is a block diagram which shows the structure of one Embodiment of this invention. It is a flowchart which shows operation | movement of the management server S shown in FIG. It is explanatory drawing which shows the definition of a loss of synchronization.

Explanation of symbols

  S ... Management server, N ... IP communication network, A1, A2, B1, B2, C1, C2 ... Terminal

Claims (2)

A group communication device that establishes communication lines with a plurality of user terminals and performs information communication using media data in which priority is set for each media type between the user terminals,
Data receiving means for receiving media data from each of the user terminals;
The maximum data arrival delay time is obtained from the data arrival delay time of each of the received media data, and if the difference between the maximum data arrival delay time and other data arrival delay times is equal to or greater than a predetermined time, the mixing time needs to be corrected. Determination means for determining that there is,
Media data of a media type with a low priority among media data generation times (T ^Mix ) of media types with a high priority when it is determined by the determination means that the mixing time needs to be corrected. Based on the mixing time (M ^Mix ) of the media data of the media type having the highest priority closest to the generation time ( ^TSix ), the ideal mixing time ( ^ISix ) is
^{I Si x = M Mi x +} (T Si x-T Mi x)
A correction means for comparing the data arrival time A ^Si x with the mixing ideal time I ^Si x and setting the later time as the mixing time ;
Based on the mixing time corrected by said correction means, and mixing means for performing mixing of media data received;
A group communication apparatus comprising: distribution means for distributing media data mixed by the mixing means to each of the user terminals. Necessary at the time of group communication to a computer on a group communication device that establishes a communication line with a plurality of user terminals and performs information communication using media data in which priority is set for each media type between the user terminals. A group communication program for performing processing for correcting the mixing time of media data,
A data receiving step of receiving media data from each of the user terminals;
The maximum data arrival delay time is obtained from the data arrival delay time of each of the received media data, and if the difference between the maximum data arrival delay time and other data arrival delay times is equal to or greater than a predetermined time, the mixing time needs to be corrected. A determination step for determining that there is,
When it is determined in the determination step that the mixing time needs to be corrected, the media data of the media type with the lower priority among the generation times (T ^Mix ) of the media data of the media type with the higher priority. Based on the mixing time (M ^Mix ) of the media data of the media type having the highest priority closest to the generation time ( ^TSix ), the ideal mixing time ( ^ISix ) is
^{I Si x = M Mi x +} (T Si x-T Mi x)
A correction step of comparing the data arrival time A ^Si x with the mixing ideal time I ^Si x and setting the later time as the mixing time ;
Wherein based on the mixing time was corrected by the correction step, and mixing step of mixing the media data received;
A group communication program which causes a computer to perform a distribution step of distributing the media data mixed in the mixing step to each of the user terminals.

Images