KR102597852B1 - System for Providing Online Collaboration - Google Patents

Abstract

Launch of online collaboration system.
In this embodiment, when a collaboration participant generates performance data, a high-resolution video for broadcasting and a low-resolution video for synchronization are created, and the low-resolution video is composed on a separate ultra-low latency server and transmitted back to the collaboration participant to each An online collaboration system that can quickly decode the composing video from the peer and use reference video for collaboration, thereby implementing an on-tact platform that can satisfy the collaboration or collaboration platform of participants and high-quality video transmission of broadcasters. to provide.

Description

Online Collaboration System {System for Providing Online Collaboration}

One embodiment of the present invention relates to an online collaboration system.

The content described below simply provides background information related to this embodiment and does not constitute prior art.

Various types of sound reproduction systems are proposed to music lovers. A compact disc player is an example of a sound reproduction system. A set of audio data codes representing discrete values as waveforms of analog audio signals is stored on a compact disk. As the compact disc rotates within the compact disc player, audio data codes are sequentially read from the compact disc and converted back into analog audio signals. The analog audio signal is transmitted to the sound system, and tones are reproduced from the analog audio signal along with the musical piece.

Since the audio data code and the music data code represent discrete values in the waveform of an analog audio signal and a key event, respectively, the audio data code and the music data code are not compatible. That is, the automatic playing system cannot drive the keys of an acoustic piano based on the audio data code, and the compact disc player cannot generate tones from the music data code along with the music piece. In this case, there is difficulty in reproducing music in collaboration between the compact disc player and the automatic player piano.

In particular, with conventional technology, the image and sound of another peer received due to compression transmission delay is already hundreds of mS of past images or time, so real-time concert performance is not possible. Therefore, there is a problem that natural concerto performance is difficult with current video and voice communication technologies.

In this embodiment, when a collaboration participant generates performance data, a high-resolution video for broadcasting and a low-resolution video for synchronization are created, and the low-resolution video is composed on a separate ultra-low latency server and transmitted back to the collaboration participant to each An online collaboration system that can quickly decode the composing video from the peer and use reference video for collaboration, thereby implementing an on-tact platform that can satisfy the collaboration or collaboration platform of participants and high-quality video transmission of broadcasters. The purpose is to provide

According to one aspect of the present embodiment, when generating performance data using a provided camera, low-resolution data for synchronization, including shooting, transmitting, receiving, and playing general video, is encoded into ultra-low-latency encoded data. A plurality of peers that render ultra-low latency decoded data that decodes corresponding compositing data; an ultra-low-latency compositing media server that receives the ultra-low-latency encoded data from each of the plurality of peers and then generates the composed data; A media server that receives and streams encoded data encoding high-resolution data of each peer generated for broadcast based on the ultra-low latency decoding data from the plurality of peers. do.

As described above, according to this embodiment, when collaborative performance participants generate performance data, they generate high-resolution video for broadcasting and low-resolution video for synchronization, and the low-resolution video is composited in a separate ultra-low latency server. By transmitting it back to the collaboration participants, each peer can quickly decode the compositing video and use the reference video for collaboration, which has the effect of satisfying the high-quality video transmission of the participant's collaboration or collaboration platform and the broadcaster.

According to this embodiment, it can be used for real-time collaborative broadcasting online and can be applied to classes such as choir, ensemble, school music, dance, and acting.

Figure 1 is a diagram showing an online collaboration system according to this embodiment.
Figure 2 is a diagram showing the operation of peers in the online collaboration system according to this embodiment.
Figure 3 is a diagram showing a synchronization method of the online collaboration system according to this embodiment.

Hereinafter, this embodiment will be described in detail with reference to the attached drawings.

Figure 1 is a diagram showing an online collaboration system according to this embodiment.

The online collaboration system according to this embodiment includes a peer 110, a media server 130, an ultra-low latency compositing media server 120, and a broadcasting streamer 140. The components included in the online collaboration system are not necessarily limited to this.

A plurality of peers (Peer1 to PeerN) 110 each system includes an application for collaboration. A plurality of peers 110 have two codecs therein. Here, codecs are divided into ultra-low latency codecs and general latency codecs that process large sizes of information that can actually be broadcast. A plurality of peers 110 encode performance data (video or sound) using both an ultra-low latency codec and a general latency codec simultaneously and transmit it to the outside.

A plurality of peers 110 generate ultra-low-latency encoding data encoded using an ultra-low-latency codec using only low-resolution performance data and transmit it to a separate ultra-low-latency compositing media server 120. The ultra-low latency compositing media server 120 receives ultra-low latency encoding data from a plurality of peers 110 and then returns compositing data combining the ultra-low latency encoding data to each of the plurality of peers 110. have

A plurality of peers 110 can receive composing data and perform while checking their own parts. In other words, since compositing data is composed of data (video or sound) from all channels, it is immediacy enough for each peer to play its own part using only composing data.

Composing data generated in the ultra-low latency compositing media server 120 is accurately synchronized even if there is a time delay. Each of the plurality of peers 110 can check the composing data and play in sync.

Thereafter, the plurality of peers 110 checks the compositing data, generates general latency encoding data by encoding the synchronized performance data of normal size using a general latency codec, and transmits it to the media server 130. The media server 130 transmits general latency encoding data to the broadcasting streamer 140.

The plurality of peers 110 naturally includes a plurality of encoders. A plurality of peers 110 initially receive a time code telling them to start together under the control of the media server 130. A plurality of peers 110 transmits performance data captured by the camera 210 as two streams using a provided application. A plurality of peers 110 displays and outputs compositing data received from the ultra-low latency compositing media server 120.

When generating performance data using the provided camera 210, a plurality of peers 110 generate ultra-low latency decoding data by decoding compositing data corresponding to ultra-low latency encoding data encoding low-resolution data for synchronization. renders.

When each of the plurality of peers 110 captures performance data, it generates low-resolution data for synchronization. Each of the plurality of peers 110 encodes low-resolution data and generates ultra-low latency encoded data. Each of the plurality of peers 110 transmits ultra-low latency encoded data to the ultra-low latency compositing media server 120.

Each of the plurality of peers 110 receives compositing data from the ultra low latency compositing media server 120. Each of the plurality of peers 110 generates ultra-low latency decoded data by decoding composing data. Each of the plurality of peers 110 renders and outputs ultra-low latency decoded data.

When each of the plurality of peers 110 captures performance data, it generates ultra-low latency decoding data and high-resolution data for broadcasting in the same time zone. Each of the plurality of peers 110 encodes high-resolution data and generates high-resolution encoded data. Each of the plurality of peers 110 transmits high-resolution encoded data to the media server 130.

A plurality of peers 110 generate ultra-low-latency encoding data encoding low-resolution data based on the reference clock information received from the ultra-low-latency compositing media server 120 to create ultra-low-latency compositing media. Transmit to server 120.

A plurality of peers 110 generate ultra-low latency decoded data by decoding compositing data based on inter-tile error information received from the ultra-low latency compositing media server 120.

A plurality of peers 110 generate ultra-low latency decoded data by decoding compositing data from the ultra-low latency compositing media server 120 based on meta information.

The media server 130 is a server for processing high-definition or high-quality sound and refers to a server that performs streaming regardless of latency. The media server 130 receives and streams encoded data encoding high-resolution data generated for broadcasting based on ultra-low latency decoding data from a plurality of peers.

The ultra-low latency compositing media server 120 refers to a server that performs processing to accurately sync by reducing the amount of data to minimize latency.

The ultra-low latency compositing media server 120 receives ultra-low latency encoded data from a plurality of peers 110 and generates compositing data encoded in time units by keeping the time until decoding as short as possible.

When the ultra-low latency compositing media server 120 decodes ultra-low latency encoded data, a codec with a low compression ratio (1:2 to 1:6) with a short decoding time is used.

The ultra-low latency compositing media server 120 receives ultra-low latency encoded data from each of the plurality of peers 110 and then generates the composed data.

The ultra-low latency compositing media server 120 periodically transmits reference clock (Ref. Clock) information to each of the plurality of peers 110 that are encoding participants.

When each of the plurality of peers, which are all encoding participants, is ready, the ultra low latency compositing media server 120 simultaneously transmits a run command to the plurality of peers 110 and encodes low-resolution data corresponding to the start command. Receive ultra-low latency encoded data.

The ultra-low latency compositing media server 120 transmits inter-peer error information to correct synchronization of screens from a plurality of peers on the decoding side. The ultra-low latency compositing media server 120 transmits meta information about the synthesized video to a plurality of peers on the decoding side.

Figure 2 is a diagram showing the operation of peers in the online collaboration system according to this embodiment.

Each of the plurality of peers 110 according to this embodiment includes a high-resolution capture unit 222, an encoder 224, a streamer 226, a low-resolution capture unit 232, an ultra-low latency encoder 234, and an ultra-low latency It includes a trimmer (236), an ultra-low latency receiver (242), an ultra-low latency decoder (244), and an ultra-low latency renderer (246). Components included in peer 110 are not necessarily limited thereto.

Each component included in the peer 110 is connected to a communication path connecting software modules or hardware modules within the device and can operate organically with each other. These components communicate using one or more communication buses or signal lines.

Each component of the peer 110 shown in FIG. 2 refers to a unit that processes at least one function or operation, and may be implemented as a software module, a hardware module, or a combination of software and hardware.

Each of the plurality of peers 110 generates performance data captured using a camera 210 provided therein.

When each of the plurality of peers 110 captures performance data, it generates high-resolution data using the high-resolution capture unit 222. Each of the plurality of peers 110 uses the encoder 224 to generate high-resolution data as high-resolution encoded data. Each of the plurality of peers 110 transmits high-resolution encoded data to the media server 130 using the streamer 226.

When each of the plurality of peers 110 captures performance data, it generates low-resolution data using the low-resolution capture unit 232. Each of the plurality of peers 110 uses the ultra low latency encoder 234 to generate low resolution data as ultra low latency encoded data. Each of the plurality of peers 110 transmits ultra-low latency encoded data to the ultra-low latency compositing media server 120 using the ultra-low latency streamer 236.

The ultra-low latency compositing media server 120 receives ultra-low latency encoding data from each of the plurality of peers 110 and then transmits compositing data obtained by combining the ultra-low latency encoding data to each of the plurality of peers 110.

Each of the plurality of peers 110 receives compositing data from the ultra low latency compositing media server 120 using the ultra low latency receiver 242. Each of the plurality of peers 110 generates ultra-low latency decoded data by decoding compositing data using the ultra-low latency decoder 244. Each of the plurality of peers 110 outputs ultra-low latency decoded data using the ultra-low latency renderer 246. At this time, the time error metadata of each peer video or tile for video sync received from the ultra-low latency compositing media server can be interpreted to elaborate the render of each part.

When processing low-resolution video, the final rendering delay of composite video/audio after communication with multiple peers 110 and the ultra-low latency compositing media server 130 is 10ms (example) for audio and 33ms for video. (1 frame) This is possible with less than a delay. Using JPEG-XS, Ultra Low Latency Video is possible with a delay of several ms due to line latency, and a delay of less than 1 frame is possible. If there are a large number of peers 110, it is possible to separate the servers into specific groups and process them separately.

Figure 3 is a diagram showing a synchronization method of the online collaboration system according to this embodiment.

The ultra-low latency compositing media server 120 periodically transmits reference clock (Ref. Clock) information to each of the plurality of peers 110 that are encoding participants (S310).

When each of the plurality of peers 110, which are all encoding participants, is ready, the ultra-low latency compositing media server 120 simultaneously transmits a run command to the plurality of peers 110, which are all encoding participants (S320). .

A plurality of peers 110, which are encoding participants, generate an encoding stream (normal stream or HEVC MCTS Stream) based on the requested start reference clock (Ref. Clock) information and transmit it to the ultra-low latency compositing media server 120. (S330).

The ultra-low latency compositing media server 120 synchronizes the video or MCTS tiles with a plurality of peers 110 on the decoding side, combines them in a composite or bitstream state, and transmits them to each peer (S340).

The ultra-low latency compositing media server 120 transmits time error meta information for each video or MCTS tile to each of the plurality of peers 110 on the decoding side (S350).

In Figure 3, steps S310 to S350 are described as being sequentially executed, but this is not necessarily limited. In other words, the steps shown in FIG. 3 may be applied by modifying them or executing one or more steps in parallel, so FIG. 3 is not limited to a time-series order.

As described above, the synchronization method of the online collaboration system according to this embodiment shown in FIG. 3 may be implemented as a program and recorded on a computer-readable recording medium. The computer-readable recording medium on which the program for implementing the synchronization method of the online collaboration system according to this embodiment is recorded includes all types of recording devices that store data that can be read by the computer system.

The above description is merely an illustrative explanation of the technical idea of the present embodiment, and those skilled in the art will be able to make various modifications and variations without departing from the essential characteristics of the present embodiment. Accordingly, the present embodiments are not intended to limit the technical idea of the present embodiment, but rather to explain it, and the scope of the technical idea of the present embodiment is not limited by these examples. The scope of protection of this embodiment should be interpreted in accordance with the claims below, and all technical ideas within the equivalent scope should be interpreted as being included in the scope of rights of this embodiment.

110: Peer
120: Ultra-low latency compositing media server
130: media server
140: Broadcasting Streamer
210: camera
222: High-resolution capture unit
224: encoder
226: Streamer
232: Low-resolution capture unit
234: Ultra low latency encoder
236: Ultra low latency streamer
242: Ultra low latency receiver
244: Ultra low latency decoder
246: Ultra low latency render

Claims (9)

Low-resolution data for collaboration is generated as ultra-low-latency encoding data encoded using an ultra-low-latency codec and transmitted to the outside, and after receiving compositing data corresponding to the ultra-low-latency encoding data from the outside, the composing data Render the decoded ultra-low latency decoded data,
A plurality of peers that generate high-resolution performance data for synchronized broadcasting as encoded data based on the ultra-low latency decoding data and then transmit it to the outside;
an ultra-low-latency compositing media server that receives the ultra-low-latency encoded data from each of the plurality of peers and then generates the composed data;
a media server that streams the high-resolution encoded data received from each of the plurality of peers using a broadcasting streamer;
An online collaboration system comprising: According to paragraph 1,
Each of the plurality of peers
When capturing the performance data, a low-resolution capture unit that generates the low-resolution data for synchronization;
an ultra-low latency encoder that encodes the low-resolution data to generate ultra-low latency encoded data;
An ultra-low latency streamer that transmits the ultra-low latency encoded data to an ultra-low latency compositing media server.
An online collaboration system comprising: According to paragraph 1,
Each of the plurality of peers
an ultra-low latency receiver that receives the compositing data from the ultra-low latency composing media server;
An ultra-low latency decoder that generates ultra-low latency decoded data by decoding the compositing data;
Ultra low latency render that renders and outputs the ultra low latency decoded data
An online collaboration system comprising: According to paragraph 1,
Each of the plurality of peers
When photographing the performance data, a high-resolution capture unit that generates the high-resolution data for broadcasting based on the ultra-low latency decoding data;
an encoder that encodes the high-resolution data and generates high-resolution encoded data;
A streamer transmitting the high-resolution encoded data to the media server
An online collaboration system comprising: According to paragraph 1,
The ultra-low latency compositing media server is
An online collaboration system characterized in that reference clock (Ref. Clock) information is periodically transmitted to each of the plurality of peers that are encoding participants. According to paragraph 1,
The ultra-low latency compositing media server is
When each of the plurality of peers, which are all encoding participants, is ready, simultaneously transmitting a start command (Run Command) to the plurality of peers and receiving the ultra-low latency encoded data encoding the low-resolution data corresponding to the start command. An online collaboration system characterized by: According to paragraph 1,
The plurality of peers is
Characterized by generating the ultra-low latency encoded data encoding the low-resolution data based on reference clock (Ref. Clock) information received from the ultra-low latency compositing media server and transmitting it to the ultra-low latency compositing media server. An online collaboration system with . According to paragraph 1,
The ultra-low latency compositing media server transmits inter-tile time error information for correcting video or tile synchronization to the plurality of peers on the decoding side,
An online collaboration system wherein the plurality of peers reproduce the ultra-low latency decoded data obtained by decoding the compositing data based on time error information between images or tiles. According to paragraph 1,
The ultra-low latency compositing media server transmits meta information of time error for each video or tile to the plurality of peers on the decoding side,
The plurality of peers decodes the composing data into the ultra-low latency decoding data based on the meta information and performs precise playback by applying time error values of images or tiles for each peer.