WO2022269723A1

WO2022269723A1 - Communication system that performs synchronous control, synchronous control method therefor, reception server, and synchronous control program

Info

Publication number: WO2022269723A1
Application number: PCT/JP2021/023482
Authority: WO
Inventors: 真二深津; 麻衣子井元
Original assignee: 日本電信電話株式会社
Priority date: 2021-06-21
Filing date: 2021-06-21
Publication date: 2022-12-29
Also published as: JPWO2022269723A1

Abstract

The present invention: calculates a frame size for transferring a plurality of second videos captured in a plurality of second venues in response to a first video transferred from a first venue via a network, so that the second videos are transferred within a duration set in advance from the plurality of second venues back to the first venue, the calculation being made on the basis of acquired throughput and round-trip time of transfer between the first venue and each of the second venues; and synchronizes the plurality of second videos by editing the second videos to the calculated frame size.

Description

Communication system performing synchronization control, synchronization control method thereof, receiving server, and synchronization control program

Embodiments of the present invention relate to a communication system that performs synchronization control in information communication including video and audio, a synchronization control method thereof, a receiving server, and a synchronization control program.

In recent years, an event held at one main venue is instantly distributed (live distribution) to remote venues set up in multiple remote locations, allowing audiences at multiple venues to participate in or appreciate the event at the same time. There is a communication system that can

In such live distribution, in order for the organizer of the main venue to feel a sense of unity with the audience at each remote venue, the response or reaction of the audience at each remote venue must have an ultra-low delay. It is important to come back to On the other hand, if there is a significant delay in the response from each remote venue to the organizer, or if there is a time lag due to inconsistent responses, the organizer will not be able to determine the timing of the next action, and the organizer and the audience will not be able to I can't feel the sense of unity. Therefore, if the response delays from each remote venue are different, it is necessary to synchronize the reactions of the audience.

As a technique for synchronization without delay, for example, the MMT (MPEG Media Transport) method described in Non-Patent Document 1 is known as one of the media transport methods.

In the above-mentioned MMT method, by setting the presentation time stamp (Presentation Time Stamp) of video and audio to the STC (system time clock), which is a common reference clock set for each communication device, multiple Synchronization in absolute time is realized for communication devices. By setting an offset according to the communication device (stream) with the largest delay when establishing this synchronization, it is possible to achieve complete synchronization of a plurality of target communication devices.

However, when synchronizing with a communication device with the longest delay, a waiting period for time adjustment occurs in the communication device with the shortest delay, so the frame of the received video is repeated. . Such correspondence can realize communication synchronization, but if even one communication device with a large delay is included, even if the other communication devices are low-delay communication devices, the delay is adjusted to the communication device with a large delay. Therefore, it is not possible to respond to the organizer with good response.

As a countermeasure for this, there is a method of varying the bit rate. For example, ABR (Adaptive Bit Rate) described in Non-Patent Document 2 is known as one of the variable methods. This ABR reduces delay by preparing moving images of a plurality of bit rates based on captured images in advance and selecting a suitable bit rate according to the communication status of the network NW (Network Work).

When using ABR, regardless of whether it is used or not, it is necessary to prepare videos with multiple bitrates before transmission. As described above, in order to create a sense of unity between the host and the audience in the live distribution, it is necessary to send the video of the inside of the remote venue back to the host. However, ABR is controlled by the transmitting side, and since images of the inside of each venue taken from a plurality of remote venues are transmitted to the organizer, the organizer cannot uniquely determine the frame size.

According to the present invention, a plurality of second videos shot at a plurality of second venues in response to a first video transmitted from a first venue through a network are returned from a plurality of second venues to the first venue. Another object of the present invention is to provide a technique for synchronizing a plurality of secondary images that are generated.

In order to solve the above problems, a communication system for performing synchronization control according to one aspect of the present invention includes a transmission server that transmits a first video signal; a plurality of receiving servers that receive the first video signal transmitted from the transmitting server through the transmission server and transmit back to the transmitting server a second video signal that responds to the first video signal; a first timing control unit that sets an allowable waiting time that does not cause a visual difference between the scenes of the plurality of the second video signals when the plurality of the second video signals are displayed simultaneously; and each of the receiving servers. is the frame size of the second video signal based on the throughput and round trip time of the signal transmitted between the transmitting server and the receiving server so that the return transmission is completed within the allowable waiting time and a second timing control unit that edits the second video signal with the calculated frame size.

According to one aspect of the present invention, a first image is transmitted from a transmission server to a plurality of reception servers respectively arranged at a plurality of remote locations, and a second image is sent in response to the first image signal received by each of the reception servers. When transmitting the signal back to the transmitting server, an allowable waiting time is set in advance based on the previously obtained throughput of the signal transmitted between the transmitting server and the plurality of receiving servers and the round-trip time. Each receiving server calculates the frame size of the second video signal so that return transmission is completed within the allowable waiting time, and adjusts the resolution, bit depth, and chroma subsampling so as to achieve the calculated frame size. It is possible to provide a technique for synchronizing a plurality of second video signals returned from a plurality of second venues to the first venue by changing and editing the second video and transmitting it to the transmission server. Therefore, when a plurality of second video signals are displayed in the first venue, synchronized video can be displayed without visual difference between the scenes of the plurality of second video images.

FIG. 1 is a block diagram showing the overall configuration of a communication system. FIG. 2 is a schematic diagram showing an example of the configuration of a computer functioning as a transmission server according to the first embodiment. FIG. 3 is a flowchart for explaining the calculation of the permissible waiting time for the return video. FIG. 4 is a flowchart for explaining the calculation of the frame size of the return video. FIG. 5 is a diagram showing setting patterns of the frame size of the return video signal in the communication system according to the first embodiment. FIG. 6 is a diagram showing setting patterns of the frame size of the return video signal in the communication system according to the modification of the first embodiment. FIG. 7 is a diagram showing setting patterns of the frame size of the return video signal in the communication system according to the first embodiment. FIG. 8 is a communication timing flow diagram for explaining transmission of a video signal and return video in the communication system according to the first embodiment. FIG. 9 is a timing flow diagram of communication in the communication system according to the modification of the first embodiment. FIG. 10 is a timing flow diagram of communication in the communication system according to the second embodiment.

BEST MODE FOR CARRYING OUT THE INVENTION Hereinafter, embodiments of the present invention will be described in detail with reference to the drawings. FIG. 1 is a diagram illustrating a configuration example of a communication system according to one embodiment.
The communication system 1 of the present embodiment includes a main venue side communication system 2 (hereinafter referred to as a communication system 2) provided in the main venue [first venue] where the organizer holds the event, and a plurality, in this example, three , remote venue side communication system 3 (hereinafter referred to as communication system 3) provided at the remote venue [second venue] set up at each remote location, and video signals etc. transmitted between these communication systems are relayed. It is composed of a network 4 that In order to clarify the explanation, the video shot in the main venue will be referred to as the event video [first video] or the event video signal, and the state of the remote venue including the audience will be filmed and sent to the main venue. The resulting image is called a return image [second image] or a return image signal. Although referred to as a video signal here, it is assumed that the video signal includes a sound signal including voice.

The communication system 2 on the main venue side includes at least one camera 11, a microphone 12 that collects the host's voice, etc., a monitor 13 that displays return images taken at each remote venue, and a transmission server. 14.

The monitor 13 is a display device having a display screen, and can use, for example, a liquid crystal display, a plasma display, an organic EL display, a projector, or the like. Also, although not shown, a speaker for emitting sound is separately provided. In addition, in FIG. 1 of the present embodiment, a plurality of monitors are installed to display the return video of each remote site on each. A configuration may be used in which the display area is divided into a plurality of display areas, and the return image of each remote site is assigned to each of the divided display areas and displayed.

The transmission server 14 includes a first input unit 21 , a first transmission unit 22 , a reception unit 23 , a presentation unit 24 and a timing control unit 25 . In this embodiment, the receiving unit 23 and the presenting unit 24 are the first receiving unit 23a and the first presenting unit 24a, the second receiving unit 23b and Three sets of the second presentation unit 24b, the third reception unit 23c, and the third presentation unit 24c are shown as examples. Of course, the number of reception units and presentation units is set according to the number of remote venues to be set up. If there are a large number of remote venues set up, a switching device may be installed to switch or select the remote venue as appropriate to display the return video.

The first input unit 21 inputs an event video signal captured by at least one camera 11 and a sound signal such as sound collected by the microphone 12, digitally encodes the event video signal and the sound signal, After performing signal processing such as level adjustment and noise removal, the signal is output to the first transmission unit 22 .

The first sending unit 22 compresses the data of the event video signal and the sound signal, converts them into communication signals in a format compatible with the network 4 used for communication, and attaches the time information described later to the converted communication signals. , to the receiving server 31 via the network 4 .
The receiving unit 23 receives the return video signal of each remote site transmitted from the receiving server 31 (first to third receiving

servers

31a, 31b, 31c) installed at each remote site.
The presentation unit 24 performs image data processing such as decompression of compressed data on the received return video signal, converts it into a video signal, and immediately outputs it to the monitor 13 .

The first timing control unit 25 adds a time stamp or the like indicating the transmission time to the event video signal sent from the first sending unit 22 as time information, and transmits the signal to the receiving server 31. The 2-timing control unit 46 acquires the throughput (Thr x ). The second timing control section 46 sends the acquired throughput Thr x to the first timing control section 25 . The throughput Thr x indicates the amount of data transferred per unit time.
Further, the first timing control unit 25 sends out the event video signal from the first sending unit 22 of the main venue, and then the return video signal sent back by the receiving server 31 of each remote venue is received by the sending server 14 . A round trip time (RTT x) is obtained as the time until the call arrives at the unit 23 .
In both cases, the x at the end of the code is an identifier for identifying each remote venue. Since the round trip time RTT x is the round trip time between the sending server 14 and the receiving server 31, the one-way transmission time from the receiving server 31 to the sending server 14 is assumed to be half the round trip time RTT x.

Based on the obtained throughput Thrx and round trip time RTTx, the first timing control unit 25 sets the allowable waiting time ThresW for return video from the remote site, and is notified to a second timing control unit, which will be described later.

Various networks can be applied to the network 4, and for example, the Internet, a dedicated line network, or an optical communication network can be used. In recent years, video transmission methods compatible with multicast, such as the IP/MPLS (Multi-Protocol Label Switching) network method, have been adopted for live distribution. The differences between these networks are, for example, the amount of data that can be transmitted and the data communication speed. The network 4 has different communication speeds (or line speeds) depending on the communication environment, for example, the communication method, communication distance, or communication route. ) 4a, a medium-speed network (NW) 4b for medium-speed communication, which is a normal communication speed, and a high-speed network (NW) 4c for high-speed communication.

In the communication system 1 of the present embodiment, remote venues set up at a plurality of remote locations are connected to the communication system 2 via the network 4, respectively, with respect to the communication system 2 installed at one main venue. A communication system 3 is installed.

These communication systems 3 include a receiving server 31, a camera 33 for photographing the situation in the remote hall, a microphone 32 for collecting sounds such as cheers in the remote hall, and an event image photographed in the main hall. and a monitor 34 . Note that the cameras, microphones, and monitors in the receiving

servers

31b and 31c are omitted from the drawing.

The receiving server 31 will be described as an example of three receiving

servers

31a, 31b, and 31c respectively connected to the networks 4 (4a, 4b, and 4c) having different communication speeds. Each receiving server 31 includes a receiving section 41 , a presentation section 42 , an input section 43 , a sending section 44 , a frame size calculating section 45 and a second timing control section 46 .

The reception unit 41 receives the event video signal and the sound signal from the transmission server 14 of the main venue via the network 4 and outputs the event video signal and the sound signal to the presentation unit 42 . The presentation unit 42 performs image data processing such as decompression of data compression, outputs an event video signal to the monitor 34, and outputs a sound signal to a speaker (not shown) arranged in the remote hall. The monitor 34 displays an event video being held at the main venue, and the speaker reproduces the sound (for example, music) and voice produced by the organizer.

The input unit 43 digitally encodes, adjusts the level of, and adjusts the level of the return video signal of the video including the spectators in the remote venue captured by at least one camera 33 and the sound signal such as the sound collected by the microphone 32 . Signal processing such as noise removal is performed on the signal, and the signal is output to the sending unit 44 .

The second timing control unit 46 outputs to the frame size calculation unit 45 the allowable waiting time Thres W for the return video, which is transmitted from the communication system 2 and stored in an updatable manner.
The frame size calculator 45 uses the allowable waiting time Thres W as a reference to calculate the frame size by a calculation method to be described later, and sets the frame size in the sending unit 44 .

The sending unit 44 edits the return video input from the input unit 43 to a set frame size, compresses the data, processes it into a signal form compatible with the network 4, and indicates the transmission time to the processed return video signal. Attached with time information such as a time stamp, it is transmitted via the network 4 to the transmission server 14 on the side of the main venue.

FIG. 2 is a schematic diagram showing an example of the configuration of a computer functioning as the transmission server 14 according to the first embodiment.
As shown in FIG. 2, the transmission server 14 is composed of a computer device and has a processor 51 such as a CPU. In the first timing control section 25 , a program memory 52 , a data memory 53 , a storage 54 , an input/output section 55 and a communication section 56 are connected to the processor 51 via a bus 57 .

The program memory 52 is a non-temporary tangible computer-readable storage medium, for example, a combination of a non-volatile memory such as a flash memory that can be written and read at any time and a non-volatile memory such as a ROM (Read Only Memory). It has been used. The program memory 52 stores programs necessary for the processor 51 to execute various control processes.

The data memory 53 is used as a tangible computer-readable storage medium, for example, by combining the above nonvolatile memory and a volatile memory such as RAM (Random Access Memory). This data memory 53 is used to store various data obtained and created in the process of performing various processes.

The storage 54 is a non-temporary tangible computer-readable storage medium, such as a HDD (Hard Disk Drive) or SSD (Solid State Drive). have a medium. The storage 54 has a large storage capacity for rewritably storing various programs and data necessary for the processor 51 to execute various control processes. The input/output unit 55 functions as an interface with the camera 11 , the microphone 12 and the monitor 13 , and the communication unit 56 functions as an interface for communicating signals via the network 4 .

The reception server 31 on the remote venue side can also be configured by a computer, similar to the transmission server 14 . The program memory 52 of the computer that constitutes the receiving server 31 stores, for example, an arithmetic program for calculating a frame size, which will be described later.

Next, the setting of the permissible waiting time for the return video by the first timing control unit 25 will be described with reference to the flowchart shown in FIG.
First, an initial connection (session) is established through the network 4 between the communication system 2 on the main venue side and a plurality of communication systems 3 on the remote venue side, and the test video signal is transmitted from the communication system 2 to each remote venue. is transmitted to the communication system 3 on the side (step S1). In addition, each communication system 3 loops back the test video signal or the image of the remote venue and transmits it to the communication system 2 (step S2).

From the round-trip transmission of this test video signal, the throughput Thr x between the main venue and each remote venue and the round-trip time RTT x due to return transmission are obtained (step S3). However, x is an identifier for distinguishing each remote venue. Based on the throughput Thrx and the round trip time RTTx, the permissible waiting time (Thres W) of the return video is set (step S4). This allowable waiting time is a time that is arbitrarily set, and when multiple images are displayed at the same time, there is no visual difference (and/or auditory difference), and the actions of the displayed scenes are almost synchronized. It is the time difference (delay difference) or the time within the allowable range. In this embodiment, the process of transmitting and displaying a plurality of return images to the main venue within the allowable waiting time is called synchronous control. It should be noted that the visual difference referred to here includes the difference in motion within the scene that is felt when viewing a plurality of reversed images at the same time, for example, the time difference between similar motions occurring in the audience. At the same time, the auditory difference is defined as the time difference between sounds and voices (spectators) that are reproduced together with the scene of the return video. Here, sound (audio) is also added to the return video, and when there is a visual difference in the video, an auditory difference is also generated. For example, when images of spectators from two remote venues are displayed simultaneously on the monitor at the main venue, the organizer sees the scene where the spectators at each remote venue cheered, The time difference, ie, the delay, is such that the cheers raised up do not feel the delay, for example, about several hundred msec.

The allowable waiting time (Thres W) for this return video can be set arbitrarily. The longer the time, the longer the delay time, although the quality of the video can be maintained. Of course, this allowable waiting time (time difference) varies depending on the type of event being held, and the numerical value is not limited.

This allowable waiting time (Thres W) is transmitted to the communication system 3 and stored in the second timing control section 46 of the reception server 31 of each remote venue (step S5). The acquisition of throughput Thr x and round trip time RTT x is performed periodically during the event, that is, during the period in which the video signal is continuously transmitted, and if necessary, the allowable waiting time ( Thres W) is updated (step S6). In addition, it is not necessary to update the allowable waiting time (Thres W) every time it is measured periodically. may be updated as an allowable waiting time (Thres W).

Calculation of the frame size (image size) of the return video at each remote venue will be described with reference to the flowchart shown in FIG.
First, the delay time when the video is folded back at the highest quality is obtained.

A frame size IS_max at the highest quality is calculated (step S11). for example,
IS_max=3840*2160(4K)*10(10bit)*3(4:4:4)=249Mbit/frame …(1)
A delay time Delay is calculated by the following equation (2) (step S12).

Delay_x=IS_max/throughput Thr_x + round trip time RTT_X/2 …(2)
Next, it is determined whether or not the calculated delay time is within a preset allowable waiting time (step S13). If it is determined in step S12 that the delay time is within the allowable waiting time (YES), the return video from the remote venue is set to be transmitted with the highest quality video signal (step S14).

However, if the calculated delay time is greater than the allowable waiting time (NO) in the determination of step S12, the delay is made within the allowable waiting time in order to absorb the difference by reducing the quality of the video (or image quality). A frame size that fits the time is calculated (step S15). The return video from the remote venue is edited with the calculated frame size and set to be transmitted as the return video with reduced quality (step S16).

Next, frame size setting patterns for reducing the amount of data will be described with reference to FIGS. 5 to 7. FIG.
As described above, the frame size that reduces the amount of data is set by a setting pattern that combines the three elements of pixel count, bit depth, and color information (chroma subsampling). Here, there are a total of 24 frame size setting patterns, ie number of pixels: 4 patterns, bit depth: 2 patterns, and color information: 3 patterns. The number of pixels is 4 patterns of 4K, 2K, HD and SD. There are two patterns of bit depth: 10 bits and 8 bits. Further, the color information (chroma subsampling) has three patterns of 4,4,4(3), 4,2,2,(2), and 4,2,0(1.5). 5 to 7, the reduction ratio of the data amount by setting the frame size indicates the reduction ratio of the frame size by changing the setting pattern when the highest image quality is 1. . The smaller the reduction ratio, the more the amount of data can be reduced.

In this embodiment, setting patterns are divided into the following six groups.
As the first group shown in FIG. 5, the frame size is set by sequentially decreasing the number of pixels to 4K, 2K, HD, and SD without changing the bit depth of 10 bits and the color information 4.4.4(3). . With this setting, the reduction ratio decreases from 1 to 0.25, 0.0625, and 0.041667.
The second group is a setting in which the bit depth is changed from 10 bits to 8 bits without changing the color information 4.4.4(3). Sets the reduced frame size.

The third group shown in FIG. 6 is a setting pattern in which the color information is reduced from 4.4.4.(3) to 4.2.2(2) without changing the bit depth of 10 bits. A frame size is set by decreasing the number in order of 4K, 2K, HD, and SD.
In addition, as the fourth group, a frame size with a bit depth of 8 bits and color information of 4.2.2 (2), respectively, is reduced by one step, and the number of pixels is sequentially reduced to 4K, 2K, HD, and SD. set.

As the fifth group shown in FIG. 7, without changing the bit depth of 10 bits, color information 4.4.4. The frame size is set by decreasing the number of pixels from (3) to 4.2.0 (1.5) by two steps, and further decreasing the number of pixels to 4K, 2K, HD, and SD.
As a sixth group, the bit depth is reduced to 8 bits, and color information 4.4.4. The frame size is set by decreasing the number of pixels from (3) to 4.2.0 (1.5) by two steps, and further decreasing the number of pixels to 4K, 2K, HD, and SD.

Next, with reference to the timing flow shown in FIG. 8, the transmission of return video between the main venue and each remote venue in the communication system according to the first embodiment will be described. In this embodiment, two receiving servers 31 at remote venues connected to the transmission server 14 at the main venue are connected via a low-speed network 4a to a first receiving server 31a, and connected via a high-speed network 4c to a third receiving server 31a. The receiving server 31c will be described as an example.

The first receiving server 31a assumes that the communication environment with the main venue is, for example, a throughput Thr_x of 1 Gbps and a round trip time RTT of 20 msec. It is also assumed that the third receiving server 31c has a throughput Thr_x of 10 Gbps and a round trip time RTT of 10 msec. Also, in this embodiment, the allowable waiting time is set to 100 msec.

First, the delay time when the video is folded back at the highest quality is calculated. The frame size IS_max at the highest quality is obtained by formula (1) above, with the number of pixels set to 4K, bit depth set to 10 bits, and color information set to 4.4.4 (3).
IS_max=3840*2160(4K)*10(10bit)*3(4:4:4)=249Mbit/frame
The frame size at the highest quality is 249 Mbit/frame. Therefore, the calculation of the delay time Delay uses the formula (2) described above. Here, as described above, the communication status between the transmission server 14 at the main venue and the first reception server 31a at the remote venue has a throughput Thr_x of 1 Gbps and a one-way round trip time RTT from the remote venue to the main venue of 10 msec. do. The required delay time is given by the formula (2) above.
Delay_x=249M/Thr_x+RTT_x/2=259msec
, the delay time is 259 msec.

Also, the communication status between the main venue and the third receiving server 31c of the remote venue is, as described above, throughput Thr_x of 10 Gbps, and the one-way round trip time RTT/2 from the remote venue to the main venue is 5 msec. The required delay time is given by the formula (2) above.
Delay_x=249M/Thr_x+RTT_x/2=29.9msec
, the delay time is 29.9 msec.

The quality of the image to be transmitted is set depending on whether or not these calculated delay times are within the preset allowable waiting time (100 msec). Note that this allowable waiting time is a one-way time from the remote venue to the main venue, unlike the round trip time RTT. Here, if the calculated delay time is within the allowable waiting time, the return video from the remote venue is transmitted with the highest quality video signal. On the other hand, if the calculated delay time is longer than the allowable waiting time (NO), the difference is absorbed by lowering the image quality, and the frame size that allows the delay time to fall within the allowable waiting time is calculated.

Therefore, the third receiving server 31c, whose delay time is 29.9 msec, is within the allowable waiting time (100 msec), so the frame size IS_max of the set pattern with the highest quality, 4K pixels, 10-bit bit depth, and color information In 4.4.4(3), the return video can be transmitted.
On the other hand, the first receiving server 31a, whose delay time is 259 msec, exceeds the allowable waiting time (100 msec). Calculate size.

Here, we will explain the point of lowering the image quality of the return video in order to reduce the amount of data. The return image transmitted from each remote venue is an image in which a large number of independent images are returned and aggregated, and from the organizer's point of view, it is clear that each image is displayed in synchronization without loss. is important. In other words, it is considered that the organizer side does not care much about the quality of the individual video of the return video because it is enough for the organizer to grasp the collected return video as a whole. In other words, even if there is deterioration in the image quality of individual images, there is no big impact. On the organizer side, presentation with low delay is more important than the image quality of the video. Therefore, in the present embodiment, priority is given to low delay over image quality for the return video.

Then the frame size is
IS_x =Thres W/Delay_x*IS_max …(3)
is required. In this embodiment, the allowable waiting time is set to 100 ms. In this case, the frame size is 100/259*249M=96.1Mbit/frame.
In order to realize the frame size of 96.1 Mbit/frame calculated here, as described above, the number of pixels, the bit depth, and the color information are adjusted in order to decrease the set values, and the data amount is to derive the desired frame size.

The data amount reduction, that is, the reduction rate, is calculated by dividing the desired frame size by the highest quality frame size (IS_x/IS_max). For example, divide the calculated frame size of 96.1 Mbit/frame by the highest quality frame size of 249 Mbit/frame. As a result, the reduction rate is 96.1/249=0.39. When a reduction rate close to the calculated reduction rate of 0.39 is selected from the setting patterns shown in FIGS. 5 to 7, "4k, 8bit, 4: A setting pattern of 2:0 (1.5)" is most suitable. Specifically, since sqrt{96.1M/(10*1.5)/(16*9)}=210.9, the screen size is 3374×1898 (px).

The transmission unit 44 edits the folded video input from the input unit 43 with the frame size output from the frame size calculation unit 45, and then compresses the data and processes it into a signal format compatible with the network 4. Time information such as a time stamp indicating transmission time is attached to the processed return video signal, and the signal is transmitted to the transmission server 14 on the side of the main venue via the network 4 .

Next, the first receiving unit 23a of the transmission server 14 transfers the time information added to the received return video to the first timing control unit 25, decompresses the return video, and outputs the decompressed return video to the first presentation unit 24a. . The first presentation unit 24a immediately displays the return video on the monitor 13a.

As described above, in the present embodiment, when different delay times occur in each return video from a plurality of remote venues to the main venue, the return video whose delay exceeds the preset allowable waiting time is reduced in image quality. By reducing the data capacity and adjusting the frame size so that the return video is transmitted within the allowable waiting time, the return video from each delayed venue is displayed on the monitor in the main venue in a synchronized manner. can be displayed. In addition, by lowering the resolution, bit depth, and color information (chroma subsampling) for the folded video, it is possible to reduce the data capacity as well as the image quality, and edit it with the frame size transmitted within the allowable delay time. can. Therefore, the delay time can be set without the need to match the maximum delay time.

In this embodiment, the main venue, which is the receiving side, can perform synchronous control of the return video sent from a plurality of remote venues. Also, as long as the allowable waiting time or the delay time from the remote site does not change due to changes in the communication status through the network, the communication system at the main site, which is the transmitting side, needs to adjust the frame size only once. In addition, even if the delay time from the remote venue changes, the main venue can adjust the image quality and readjust the frame size based on the allowable waiting time, so that the return video can be synchronously controlled.

[Modification of First Embodiment]
A communication system according to a modification of the first embodiment will be described with reference to FIG. The configuration of the communication system of this modified example is the same as that of the communication system of the first embodiment, and the description of the configuration is omitted here.
This modified example shows an example in which the quality of the return video is lowered without lowering the quality of the event video transmitted from the main venue to a plurality of remote venues. In the first embodiment described above, synchronous control with little delay was performed by adjusting the frame size of the loopback video so that it falls within the delay time. It is conceivable that the transmitted event video is also delayed. Delay can also be reduced by adjusting the frame size for this event video as well. However, lowering the quality of the event video presented from the main venue to the remote venue impairs the quality of the user, so in this modified example, only the quality of the return video is reduced.

　The first receiving server and the third receiving server shown in FIG. 9 are equivalent to the first receiving server 31a and the third receiving server 31b of the first embodiment described above. Assume that the first reception server installed at the remote venue has a communication environment with the main venue with a throughput Thr_x 1 Gbps and a round trip time RTT of 20 msec. Similarly, the third receiving server assumes a throughput Thr_x of 10 Gbps and a round trip time RTT of 10 msec. Also, in this embodiment, the allowable waiting time is set to 140 msec.

Since the round trip time RTT is the round trip transmission time between the main venue and the remote venue, it is conceivable to absorb the delay time from the main venue to the remote venue. As mentioned above, the delay time at the first receiving server (one way from the remote venue to the main venue) is
Delay_x=249M/Thr_x+RTT_x/2=259ms
becomes. Also, the frame size is
IS_x ={ThresW-(Delay_x-ThresW)}/Delay_x*IS_max
is required. For example, if the allowable waiting time is set to 140 ms,
IS_x={140-(259-140)}/259*249M=20.19Mbit/frame
becomes.
Therefore, the reduction rate is 20.19/249=0.081 from IS_x/IS_max.

A reduction ratio that approximates the reduction ratio of 0.081 is the reduction ratio of 0.0625 shown in FIG. become resolution. Specifically, since sqrt{20.19M/(8*3)/(16*9)}=76.43, the screen size is 1223×688 (px). Incidentally, when the permissible waiting time (one side) in the first embodiment described above is set to 100 ms, IS_x becomes negative, and even if the frame size is reduced, transmission with that permissible waiting time cannot be performed. , increasing the allowable waiting time.

Therefore, according to this modified example, by setting the image size of the return video to 1223x688 (px), even if the number of pixels for transmission of the event video is maintained at 4K, the round-trip transmission between the main venue and the remote venue is possible. Even so, the delay time can be kept within 280 msec.

[Second embodiment]
In this embodiment, delay adjustment is handled not by transmission of one frame but by transmission of a plurality of frames. At this time, processing of a multithread program, for example, is used to process a plurality of frames in parallel. This makes it possible to reduce the degree of reduction in the frame size compared to the case of dealing with one frame, and to reduce deterioration in video quality.
Specifically, the permissible waiting time per frame of video: Thres WF is obtained by the following formula, where the number of corresponding frames: NumF and the parallel processing interval: Pt.
Pt=Thres W/(NumF-1) (4)
Thres WF=Thres W*(NumF^2-1)/{NumF*(NumF-1)} …(5)
Here, when the delay time of the first receiving server 31a is 259 msec, the allowable waiting time is 100 msc, and the return video is handled by transmitting 11 frames, from the above equations (4) and (5), the parallel processing interval Pt=10 msec, Allowable waiting time Thres WF=109 msec.

At this time, if the relationship between the parallel processing interval and the round trip time RTT/2 is Pt≤RTT_x/2, single-thread processing is possible, and if Pt>RTT_x/2, multi-thread processing is required. Become. Here, if the parallel processing interval Pt=10 msec and the round trip time RTT=10 msec, then Pt>RTT_x/2, so multithread processing is performed. Derive the frame size for transmission with the determined allowable latency Thres WF.

The total allowable waiting time when transmitting in 11 frames is 11*100=1100msec with an allowable waiting time of 100msc when processed with a normal single thread. On the other hand, in the case of multi-thread processing according to this embodiment, the allowable wait time is 109 msec, but the overlapped portion of each allowable wait time includes a parallel processing section of 10 msec, so the total allowable wait time is 109*11-100. = 1099msec.

As described above, according to the present embodiment, as the number of times of transmission of the return video increases, deterioration in video quality is reduced, and the total delay time is gradually reduced by the parallel processing time, which is the overlapping portion of the delay time. Therefore, when the transmission of the set number of frames is completed, the data can be transmitted within the set allowable waiting time.

Next, a modified example will be described.
In the above-described first embodiment, modified example of the first embodiment, and second embodiment, the allowable waiting time is set arbitrarily. set to

Calculate and set the frame size for the return video transmitted from other remote venues so that the return video is transmitted to the transmission server within the set allowable waiting time. At this time, in order to calculate the frame size, 24 setting patterns that combine the three elements of the number of pixels, bit depth, and color information (chroma subsampling) described above were used. is not limited to the numerical value of By setting more setting patterns, it becomes possible to completely synchronize the return video from each remote venue.

In the above embodiment, the transmission server 14 on the main venue side measures the throughput and round trip time for each remote venue side, but the throughput and round trip time are measured on the reception server 31 on each remote venue side. You can make it work. In this case, it is only necessary to transmit only the allowable waiting time to the remote venue from the transmission server 14 on the main venue side.

The method described in each embodiment can be executed by a computer (computer) as a distance measurement program (software means), such as a magnetic disk (floppy (registered trademark) disk, hard disk, etc.), optical disk (CD-ROM). , DVD, MO, etc.), a semiconductor memory (ROM, RAM, flash memory, etc.), or the like, or may be transmitted and distributed via a communication medium. The programs stored on the medium also include a setting program for configuring software means (including not only execution programs but also tables and data structures) to be executed by the computer. A computer that realizes this apparatus reads a program recorded on a recording medium, and in some cases, builds software means by a setting program, and executes the above-described processes by controlling the operation by this software means. The term "recording medium" as used herein is not limited to those for distribution, and includes storage media such as magnetic disks, semiconductor memories, etc. provided in computers or devices connected via a network.

In short, the present invention is not limited to the respective embodiments described above, and can be modified in various ways without departing from the gist of the present invention at the implementation stage. Also, the configurations described in the above embodiments may be combined as appropriate as possible, in which case the combined effect can be obtained. Furthermore, the above-described embodiments include inventions at various stages, and various inventions can be extracted by appropriately combining a plurality of disclosed constituent elements.

DESCRIPTION OF

SYMBOLS

1, 2, 3... Communication system 4...

Network

11, 33...

Camera

12, 32...

Microphone

13, 34... Monitor 14... Sending server 21... Input part 22... Sending

part

23, 41... Receiving

part

24, 42... Presentation part 25 First timing control unit 31 Reception server 43 Input unit 44 Sending unit 45 Frame size calculation unit 46 Second timing control unit 51 Processor 52 Program memory 53 Data memory 54 Storage 55 Input/output Part 56... Communication part 57... Bus

Claims

a transmission server that transmits the first video signal;
and receiving the first video signal transmitted from the transmission server through a network, and transmitting a second video signal in response to the first video signal to the transmission server. a plurality of receiving servers that transmit back to
The transmission server includes a first timing control unit that sets an allowable waiting time that does not cause a visual difference between the scenes of the plurality of the second video signals when the plurality of the second video signals are displayed at the same time;
Each of the receiving servers performs the second processing based on the throughput of signals transmitted between the sending server and the receiving server and the round trip time so that the return transmission is completed within the allowable waiting time. a second timing control unit that calculates the frame size of a video signal and edits the second video signal with the calculated frame size;
A communication system that performs synchronous control.
The plurality of receiving servers,
2. The frame size calculator according to claim 1, further comprising a frame size calculator that calculates the frame size by changing the resolution, bit depth, and chroma subsampling of the second video signal so that transmission is completed within the allowable waiting time. A communication system that performs synchronous control.
A first video signal transmitted from a transmission server through a network is received by a plurality of reception servers respectively located at different remote locations, a plurality of second video signals responding to the first video signal are returned, and a plurality of the A synchronization control method for performing synchronization control so as not to cause a visual difference between scenes of the plurality of second video signals when the transmission server simultaneously displays the plurality of second video signals transmitted from the receiving server to the transmission server. and
a processor of the delivery server setting an allowable waiting time;
Throughput of signals transmitted between the transmitting server and the receiving server, wherein each of the processors of the plurality of receiving servers determines a frame size for transmitting the second video signal to be returned and transmitted within the allowable waiting time. and a round-trip time, and editing and transmitting the second video signal with the calculated frame size.
The frame size of the second video signal is
4. The synchronization control method according to claim 3, wherein the resolution, bit depth and chroma subsampling of said second video signal are changed so that transmission is completed within said allowable waiting time.
A first video signal transmitted from a transmission server through a network is received by a plurality of reception servers respectively located at different remote locations, a plurality of second video signals responding to the first video signal are returned, and a plurality of the In a communication system that performs synchronous control that does not cause a visual difference between the scenes of the plurality of second video signals when the plurality of second video signals transmitted from a receiving server to the sending server are simultaneously displayed by the sending server a receiving server,
receiving, from the transmission server, an allowable waiting time set as a waiting time at which visual difference does not occur between the scenes of the plurality of second video signals, and the sending server so that the return transmission is completed within the allowable waiting time. A timing control unit that calculates the frame size of the second video signal based on the throughput and round trip time of the signal transmitted between the receiving server and the receiving server, and edits the second video signal with the calculated frame size. a receiving server, comprising:
A first video signal transmitted from a transmission server through a network is received by a plurality of reception servers respectively located at different remote locations, a plurality of second video signals responding to the first video signal are returned, and a plurality of the In a communication system that performs synchronous control that does not cause a visual difference between the scenes of the plurality of second video signals when the plurality of second video signals transmitted from a receiving server to the sending server are simultaneously displayed by the sending server A synchronization control program that causes a processor to function as the timing control unit of the receiving server according to claim 5 .