KR101821145B1 - Video live streaming system - Google Patents

Abstract

The present invention discloses a video live streaming system. The video live streaming system comprises a photographing device; a time synchronization module for providing a time stamp; an encoder which performs encoding for each data unit for the streaming of photographing data using a time stamp; a streaming server for comparing correction timestamps of the data units of the photographing data based on a universal time and synchronizing the data units of the photographing data; and an image processor for combining photographed images with the photographing data to eliminate duplication between the photographed images and generating an output image formed by merging the photographed data having no duplication. It is possible to provide the live streaming of a high-capacity output video to a client.

Description

VIDEO LIVE STREAMING SYSTEM [0002]

The present invention relates to an image live streaming system, and more particularly, to an image live streaming system that provides an output image obtained by merging a plurality of captured images captured in real time by a live streaming method.

With the emergence of various video services in the recent Internet environment, demand for high-definition live broadcasting is increasing, and live streaming of images is being attempted.

When a plurality of photographing apparatuses are used, it is possible to generate special images such as a surround view showing an image photographed at 360 degrees or a panoramic view showing a wide-width image.

In order to satisfy the above-mentioned demands, there is a need to develop a technique of synchronizing captured images of a plurality of photographing apparatuses into a single output image and then performing live streaming to a client.

However, there is a difficulty in synchronizing captured images transmitted in real time from a plurality of photographing apparatuses that can be configured as heterogeneous systems, and it is difficult to merge various captured images into one output image and live streaming them to a client.

The present invention provides a video live streaming system for synchronizing photographed images of a plurality of photographing apparatuses into a single output image and live streaming an output image to a client in order to solve a delay difference between photographed images in a live streaming process .

In order to solve the delay difference between photographed images, the present invention synchronizes the photographed data of the photographed image transmitted in real time through a plurality of channels, eliminates redundancy by comparing the photographed data with a small amount of data, The present invention provides a video live streaming system in which live streams are streamed to a client.

In addition, the present invention provides an output such as a surround view showing a video shot at 360 degrees in synchronism with a live streaming of photographed images shot by a plurality of photographing apparatuses, or a panoramic view (panoramic view) showing a wide- Another object of the present invention is to provide a video live streaming system for live streaming an image to a client.

The video live streaming system of the present invention includes: photographing devices for transmitting photographed data of a photographed image in real time; Time synchronization modules for providing time stamps to the photographing devices; Encoders for encoding the correction time stamp, which is obtained by correcting the time stamp by using a time correction value, for each data unit for streaming to each shot data, and then live streaming the shot data; A streaming server for comparing the correction timestamps of the data units of the photographing data based on a universal time to synchronize the data units of the photographing data and then live streaming the photographing data; And an image processor that removes duplication between the captured images by the shooting data that is streamed live on the streaming server, and then performs live streaming of an output image obtained by merging the shooting data to a client.

In addition, the video live streaming system of the present invention includes photographing devices for transmitting photographed data on photographed images in real time; A time synchronization module and an encoder, wherein the time synchronization module provides a time stamp, and the encoder receives the imaging data and the time stamp from the imaging devices and the time synchronization module, respectively, A streaming source device for encoding the correction time stamp, which is obtained by correcting the time stamp using a time correction value corresponding to the data, for each data unit for streaming the shooting data, and then live streaming the shooting data; A streaming server for comparing the correction timestamps of the data units of the photographing data based on a universal time to synchronize the data units of the photographing data and then live streaming the photographing data; And an image processor that removes duplication between the captured images by the shooting data that is streamed live on the streaming server, and then performs live streaming of an output image obtained by merging the shooting data to a client.

The present invention relates to a surveillance system for synchronizing shot images of a plurality of photographing apparatuses transmitted in real time with time stamps and merging them into one output image, It is possible to provide special images such as a panoramic view in real time.

In addition, the present invention synchronizes photographed image data of photographed images transmitted through multiple channels in real time using time stamps, and eliminates redundancy by comparing photographed image data with a small amount of data, A special image such as a view (Surround View) or a panoramic view (Panoramic View) showing a wide-width image can be generated as a high-capacity output image, and an output image can be provided in real time.

1 is a block diagram showing an embodiment of a video live streaming system of the present invention;
2 is a diagram illustrating an arrangement of photographing apparatuses for a surround view.
3 is a diagram illustrating an arrangement state of photographing apparatuses for a panoramic view.
Fig. 4 is a view for explaining a method of encoding a time stamp on photographing data; Fig.
5 is a view for explaining a time difference in which photographic data are received by a server;
Fig. 6 is a diagram for explaining how data units of photographing data are arranged in a streaming server; Fig.
7 is a view for explaining a method of generating a vector value for eliminating duplication of photographed images.
8 is a diagram illustrating a method of merging an output image using vector values generated by the method of FIG. 7;
9 is a block diagram showing another embodiment of a video live streaming system of the present invention.
10 is a block diagram showing still another embodiment of a video live streaming system of the present invention;

Hereinafter, preferred embodiments of the present invention will be described in detail with reference to the accompanying drawings. It is to be understood that the terminology used herein is for the purpose of description and should not be interpreted as limiting the scope of the present invention.

The embodiments described in the present specification and the configurations shown in the drawings are preferred embodiments of the present invention and are not intended to represent all of the technical ideas of the present invention and thus various equivalents and modifications Can be.

1 is a block diagram showing an embodiment of a video live streaming system of the present invention.

1, a video live streaming system according to the present invention includes a plurality of photographing apparatuses 100, a plurality of encoders 120 and a plurality of time synchronization modules 140, a streaming server 200, an image processor 220, Client 300 as shown in FIG.

Here, one photographing apparatus 100, one encoder 120, and one time synchronization module 140 may be configured as the streaming source apparatuses M10_1 to M10_n, respectively. That is, the video live streaming system may include a plurality of streaming source devices M10_1 to M10_n,

The photographing apparatus 100 of each of the streaming source apparatuses M10_1 to M10_n is configured to generate photographing data for the photographed image and provide it to the encoder 120. The encoder 120 of each of the streaming source apparatuses M10_1 to M10_n And is configured to live stream the shot data to the streaming server (200) via the communication network (10).

The streaming server 200 arranges photographed data, the image processor 220 generates an output image using the photographed data, and outputs output data for the output image to the client 30 through the communication network 20 Stream.

Here, the communication network 10 and the communication network 20 may be configured using the same or different types of networks. For example, the communication network 10 and the communication network 20 may be formed of one or more of a wired communication network, a wireless communication network, a wired Internet network, Lt; / RTI >

In the video live streaming system of the present invention, transmission and reception of pickup data between the pickup apparatus 100 and the encoder 120 and transmission and reception of pickup data between the encoder 120 and the streaming server 200 are performed in a first communication protocol Transmission and reception of the shooting data between the streaming server 200 and the image processor 220 and transmission and reception of the output image between the image processor 220 and the client 300 via the communication network 20 2 communication protocol.

Here, the first communication protocol and the second communication protocol may support live streaming, and the first and second communication protocols may be dynamic adaptive streaming over HTTP (DASH), MPEG2- TS: MPEG2 Transport System, HLS (HTTP Live Streaming), RTP (Real Time Transport Protocol), RTMP (Real Time Messaging Protocol), RTSP (Real Time Streaming Protocol) (RTCP), and an MPEG media transport (MMT), using the same or different communication protocols.

The configuration of each component of the video live streaming system according to the present invention will be described in more detail.

Each photographing apparatus 100 transmits photographed data on a photographed image in real time, and real-time transmission can be realized by live streaming. The above-mentioned live streaming may include mobile live streaming.

The photographing apparatuses 100 of the streaming source apparatuses M10_1 to M10_n can be arranged as shown in FIG. 2 to implement a surround view showing an image photographed at 360 degrees, and a panorama image And can be arranged as shown in FIG. 3 to implement a panoramic view.

In FIGS. 2 and 3, the photographing apparatuses are represented by 100_1 to 100_n in order to indicate that they correspond to the streaming source apparatuses M10_1 to M10_n. In Fig. 2, it is understood that the photographing apparatus 100_1 is included in the streaming source apparatus M10_1, and the photographing apparatus 100_2 is included in the streaming source apparatus M10_2.

In a case where photographing apparatuses of the streaming source apparatuses M10_1 to M10_n arranged to implement a special image such as a surround view or a panoramic view are arranged to photograph adjacent areas, images taken by photographing apparatuses include overlapping regions .

FIG. 2 and FIG. 3 only illustrate a method of arranging a plurality of photographing apparatuses 100_1 to 100_n for photographing and providing a special image such as a surround view or a panoramic view in real time. In order to generate various special images, The devices 100_1 to 100_n may be arranged in various ways. The number of photographing apparatuses 100_1 to 100_n may be variously determined according to the manufacturer's selection.

The plurality of photographing apparatuses 100_1 to 100_n may use homogeneous apparatuses or heterogeneous apparatuses, or use apparatuses having the same or different angle of view.

For example, the plurality of photographing apparatuses 100_1 to 100_n include a camera function, a camera capable of wired transmission or wireless transmission of photographed data on a photographed image, a mobile communication terminal such as a mobile phone or a smart phone, Multimedia Player), a MID (Mobile Internet Device), a Tablet PC, a Notebook, and a Netbook.

Also, the plurality of photographing apparatuses 100_1 to 100_n may be combined using heterogeneous apparatuses having a narrow view angle and a wide view angle, or homogeneous apparatuses having different view angles.

The client 300 may include a device capable of receiving output data that is live streamed through the communication network 20 and reproducing an output image. Examples of the device include a mobile communication terminal such as a mobile phone or a smart phone, a portable multimedia player (PMP) ), A MID (Mobile Internet Device), a tablet computer (Tablet PC), a notebook, and a net book.

The time synchronization module 140 is composed of a number corresponding to the photographing apparatus 100 and the encoder 120, and provides a time stamp for the current time.

More specifically, the time synchronization module 140 may be configured to provide a first time stamp that codes the universal time and a second time stamp that is generated by counting based on the first time stamp.

Here, the time stamp is time information displayed at a specific position to prove that the data was present or received at a certain point, and can be understood as a time displacement parameter indicating the origin of time.

The universal time may be a global standard time that is shared with a wireless network, a wired network, or the Internet network.

The encoder 120 receives photographed data and a time stamp from the photographing apparatus 100 and the time synchronization module 140 and outputs a correction time stamp obtained by correcting the time stamp using the time correction value corresponding to the photographed data, Encodes it for each data unit for streaming, and live streaming capturing data including the data units for which the correction time stamp is encoded.

The operation of the encoder 120 will be described with reference to FIG.

The photographing apparatuses 100 of the respective streaming source apparatuses M10_1 to M10_n start photographing from the time t1 and transmit the photographing data 1 to the photographing data n to the encoders 120. [

Therefore, the encoders 120 of the respective streaming source apparatuses M10_1 to M10_n receive the shooting data 1 to shooting data n from time t1, respectively.

Each of the shooting data 1 to n is divided into predetermined data units when the shooting device 100 transmits the shooting data 1 to the encoder 120. [ The data unit may be defined and configured in various formats by the image pickup apparatus 100 and may be exemplarily divided into a line unit, a frame unit, or a data amount unit of an image. Therefore, the encoder 120 receives sequential data units 1 to m sequentially from time t1.

That is, as shown in FIG. 4, the shooting data 1 to the shooting data n of the streaming source devices M10_1 to M10_n include successively consecutive data units 1 to m. Here, n and m mean a natural number equal to or different from each other.

The encoder 120 encodes the correction time stamps in units of data for streaming the shooting data 1 to the shooting data n, when the shooting data 1 to the shooting data n are received. Taking the case of the shooting data 1 as an example, the correction time stamps TS1_1 to Ts1_m are encoded for each data unit 1 to data unit m.

The correction time stamps TS1_1 to Ts1_m are obtained by correcting the time stamps provided by the time synchronization module 140 by the encoder 120 using the time correction values provided by the streaming server 200. [

There is no time correction value to be applied to the first data unit (exemplarily, data unit 1) of the shooting data, or the initial value may be set to "0" as an example. The time correction value to be applied to the data units (illustratively, data unit 2 to data unit m) subsequent to the original data unit corresponds to the time difference between the reception time of the previous data unit and the correction time stamp to the streaming server 200 May be provided in the streaming server 200 as a value.

In an embodiment of the present invention, the encoder 120 may be configured to limit the number of times the time stamp is corrected using the time correction value, so that if the time stamp exceeds the number of corrections, The correction time stamp can be generated using the corrected time correction value.

Referring to Fig. 4, at time t1, the data units 1 of the respective photographing data 1 to n are received by the encoder 120. Fig. At this time, since the time correction value is absent or the initial value may be set to "0" as an example, the encoder 120 sets the time (t1) provided in the time synchronization module 140 The stamp is encoded with a correction timestamp.

Thereafter, the data unit 2 to the data unit m of the shooting data 1 to the shooting data n are successively received by the encoder 120 from time t2 to tm. At this time, the time correction value is received by the streaming server 200, and the streaming server 200 provides the time correction value with a value corresponding to the time difference between the reception time of the previous data unit and the correction time stamp. Therefore, the encoder 120 sequentially obtains the correction timings obtained by correcting the time stamps provided by the time synchronization module 140 using the time correction values for the data units 2 to the data units m of the photographic data 1 to the photographic data n, Encode the stamp.

Illustratively, the encoder 120 receives the data unit 1 of the shooting data 1 at time t1 and encodes the timestamp provided by the time synchronization module 140 to the correction time stamp TS1_1 corresponding to the time t1.

Then, the encoder 120 receives the data unit 2 of the shooting data 1 at time t2, and the previous data unit 1 receives the reception time D1 (see FIG. 5) received by the streaming server 200 and the correction time stamp And uses the value corresponding to the time difference of Ts1_1 to encode the time stamp provided by the time synchronization module 140 to the correction time stamp TS1_2 corresponding to the time t2. At this time, the correction time stamp TS1_2 may be expressed as "t2- (t1-D1)". That is, the correction time stamp TS1_2 encoded in the data unit 2 of the photographing data 1 has a corrected value to include the information compensating the delayed time difference of the previous data unit 1.

Here, the encoder 120 may use the first time stamp coding the universal time corresponding to the time t1, and use the second time stamp which is generated by counting the first time stamp after the time t2 .

Unlike the above, the encoder 120 transmits the time stamp of the time synchronization module 140 to the streaming server 200 in advance before receiving the shooting data 1 to the shooting data n, A time correction value can be received. In this case, when the data units 1 of the image pickup data 1 to the image pickup data n are received, the encoder 120 uses the correction time stamps obtained by correcting the current time stamp of the time synchronization module 140 using previously received time correction values Data units 1 can be encoded.

On the other hand, the streaming server 200 receives the shooting data 1 to shooting data n, which are live streamed by the encoders 120 of the streaming source devices M10_1 to M10_n, via the communication network 10 as shown in FIG. In Fig. 5, D1 indicates the time when the shooting data 1 is received by the streaming server 200, D2 indicates the time when the shooting data 2 is received by the streaming server 200, Dn indicates that the shooting data n is received by the streaming server 200, Quot; means the time that is received at the base station 200.

The streaming server 200 generates time correction values corresponding to the data units of the shooting data 1 to the shooting data n and supplies the time correction values to the encoders 100 of the corresponding streaming source apparatuses M10_1 to M10_n do.

The streaming server 200 compares the correction timestamps of the data units of the shooting data 1 to the shooting data n based on the universal time to synchronize the data units of the shooting data 1 to the shooting data n as shown in FIG. 6, (N) to the image processor 220. The image processor 220 generates the image data (n)

The streaming server 200 repeats the process of providing the delay time for the delay difference of the data units of the shooting data 1 to the shooting data n as the time correction value on the basis of the universal time several times, Repeat the generation of the compensation time stamp using the time correction value several times.

As a result, the streaming server 200 can provide a fixed time correction value so that it can be aligned at the same time on the basis of universal time, and data units of the shooting data 1 to shooting data n are synchronized .

The image processor 220 compares photographed images by photographed data (photographed data 1 to photographed data n) that are streamed live in the streaming server 200 to eliminate duplication between photographed images, And outputs the output data of the output video to the client 300 through the communication network 20. [

First, the image processor 220 receives photographed data synchronized with data units as shown in FIG. 6 in the streaming server 200, and generates an output image by the method of FIGS. 7 and 8. FIG. The photographed image in the image processor 220 can be analyzed as information included in the photographed data.

The image processor 220 sequentially compares the photographed images of the photographed data pair by pair. The case of comparing the first and second images V1 and V2 photographed at the adjacent positions can be described with reference to FIG.

The first captured image V1 and the second captured image V2 include overlapping areas. In order to eliminate redundancy between the first and second photographed images V1 and V2, the overlapping area OV1 of the first photographed image V1 and the overlapped area OV2 of the second photographed image V2 One should be removed.

The photographed data contains high-capacity data. Therefore, comparative analysis of a large amount of data is required when the captured data is directly compared and analyzed.

In order to solve this problem, the image processor 220 downscales the first shot image V1 and the second shot image V2 to reduce the amount of data, and outputs the downscaled first shot image DV1, Compare the video (DV2).

The image processor 220 generates the second captured image DV2 downscaled from the overlapping areas DOV1 and DOV2 existing in the downscaled first captured image DV1 and the downscaled second captured image DV2, And generates and stores a vector value indicating the position of the image of the overlap area DOV2. In Fig. 7, the vector value is represented by VP.

The vector value VP is generated so as to include information about a position at which the photographed image is to be removed and information about a direction to be removed. Illustratively, when the left area of the 300th vertical line is determined as a comparison result in the overlap area DVO2, the vector value VP may include information about the position to be removed, i.e., (0,300), which means the 300th vertical line Quot; - " which means that the direction to be removed is the left region. That is, the vector value VP may be generated and stored as - (0,300). The above-mentioned vector value VP is only illustrative and can be expressed in various formats by the manufacturer.

The vector value VP is generated for the downscaled second captured image DV2. Therefore, the image processor 230 may convert and store the vector value VP corresponding to the second captured image V2 before the downscale according to the design of the manufacturer.

After storing the vector value VP as described above, the image processor 220 removes the overlap area OV2 using the vector value VP with respect to the second captured image V2, and outputs the first captured image V1 And the second photographed image V2 can be generated as shown in FIG.

Referring to FIG. 8, it can be understood that the output image is generated by merging the first photographed image V1 including the second photographed image V2 with the overlapped area OV2 removed, and the overlapped area OV1.

The image processor 220 may generate an output image as shown in FIG. 8 after eliminating redundancy while sequentially comparing captured images using the captured data. The output data of the output image is transmitted to the client 300 through the communication network 20, Can be streamed live.

In the embodiment of the present invention, the streaming source apparatuses M10_1 to M10_n are exemplified as including the photographing apparatus 100, the encoder 120 and the time synchronization module 140 one by one.

However, the present invention is not limited thereto, and the photographing apparatus 100, the encoder 120, and the time synchronization module 140 may be variously configured according to the manufacturer's intention.

For example, the present invention may be implemented such that the imaging device 100, the encoder 120, and the time synchronization module 140 are not included in one streaming source device but are configured as separate devices, .

As another example, the present invention can be applied to the streaming source apparatuses M20_1 to M20_n including the encoder 120 and the time synchronization module 140, . ≪ / RTI >

The above-described two embodiments are different from the streaming source apparatus of FIG. 1 only in terms of the structure and operation of the elements, and a duplicate description thereof will be omitted.

As another example, the present invention can be applied to a single streaming source apparatus M30 (100) including the encoder 120 and the time synchronization module 140, as shown in FIG. 10, ). ≪ / RTI >

In this case, the time synchronization module 140 provides the time stamp, the encoder 120 receives the photographing data and the time stamp from the photographing apparatuses 100_1 to 100 - n and the time synchronization module 140, A time stamp correction unit configured to encode a correction time stamp obtained by correcting a time stamp using the time correction value corresponding to the shooting data for each data unit for streaming the shooting data and to transmit the shooting data including the data units, And live streaming to the server 200.

As described above, the embodiment of the present invention synchronizes shot data and shot images using a time stamp in order to eliminate a delay difference due to live streaming between shot images, and generates an output image from which overlapping shot images are removed And output data of the output video can be live streamed.

As described above, the present invention can solve the delay difference caused by live streaming of the photographed images of a plurality of photographing apparatuses and merge them together. It is also possible to provide a surround view for displaying images photographed at 360 degrees, Live streaming with special images such as Panoramic View.

In addition, the present invention is characterized in that a special image such as a surround view that shows an image photographed at 360 degrees or a panoramic view (panoramic view) that shows a wide-width image after removing redundant areas by comparing the photographed data with a small amount of data, And the output image can be streamed live.

Claims (17)

A photographing device for transmitting photographing data of a photographed image in real time;
Time synchronization modules for providing time stamps to the photographing devices;
Encoders for encoding the correction time stamp, which is obtained by correcting the time stamp by using a time correction value, for each data unit for streaming to each shot data, and then live streaming the shot data;
A streaming server for comparing the correction timestamps of the data units of the photographing data based on a universal time to synchronize the data units of the photographing data and then live streaming the photographing data; And
And an image processor for performing live streaming of an output image obtained by merging the photographed data after removing the overlap between the photographed images by the photographed data that is live streamed in the streaming server. . The method according to claim 1,
Wherein the streaming server generates the time correction values corresponding to the time difference between the reception times of the data units and the correction timestamps for each shot data and provides the time correction values to the corresponding encoders, system. The method according to claim 1,
A plurality of the streaming source apparatuses are connected to the streaming server through a communication network, and the video live Streaming system. The method according to claim 1,
One of the time synchronization modules and one of the encoders is configured as one streaming source device and a plurality of the streaming source devices live stream the shot data to the streaming server through a communication network. 2. The method of claim 1, wherein the time synchronization modules comprise at least one of a first time stamp coding the universal time and a second time stamp generated by counting the first time stamp on the basis of the first time stamp, Streaming system. delete 6. The method of claim 5,
Wherein the encoder receives the time correction value corresponding to the first time stamp by transmitting the first time stamp before reception of the photographing data, and when the data units of the photographing data are received, Wherein the data units are encoded using the correction timestamps corrected for the second timestamp. The method according to claim 1,
Wherein the encoders perform the correction of the time stamp using the time correction value for each shot data several times and then encode the time stamp using the fixed correction time stamp. The method according to claim 1,
Wherein the image processor sequentially compares the photographed images by the photographed data in pairs and outputs the photographed images to the overlapping region of the second photographed image among overlapping regions existing respectively in the first photographed image and the second photographed image, And stores the vector value in the storage unit, and removes the overlapping area of the second captured image using the stored vector value. 10. The method of claim 9,
Wherein the image processor down-scales the first and second images to reduce the amount of data, compares the down-scaled first and second images with each other, And generates the vector value indicating the image position of the region. 10. The method of claim 9,
Wherein the image processor generates the vector value so as to include information on a position to remove the photographed image and information on a direction to be removed. 10. The method of claim 9,
Wherein the image processor converts the vector value to correspond to the second captured image before the downscale and stores the converted vector value. The method according to claim 1,
Wherein the image processor merges the photographed images to generate the output image for a surround view or a panoramic view. The method according to claim 1,
The reception of the photographing data of the encoder and the transmission and reception of the photographing data between the encoder and the streaming server are performed in accordance with a first communication protocol,
Wherein transmission and reception of the photographing data between the streaming server and the image processor and transmission and reception of the output image between the image processor and the client are performed according to a second communication protocol,
Wherein the first communication protocol and the second communication protocol support live streaming. 15. The method of claim 14,
The first and second communication protocols may be dynamic adaptive streaming over HTTP (DASH), MPEG2 transport system (MPEG2-TS), HTTP live streaming (HLS) Real Time Transport Protocol (RTMP), Real Time Messaging Protocol (RTMP), Real Time Streaming Protocol (RTSP), Real Time Control Protocol (RTCP), and MMT A video live streaming system defined as any one of different. A photographing device for transmitting photographing data of a photographed image in real time;
A time synchronization module and an encoder, wherein the time synchronization module provides a time stamp, and the encoder receives the imaging data and the time stamp from the imaging devices and the time synchronization module, respectively, A streaming source device for encoding the correction time stamp, which is obtained by correcting the time stamp using a time correction value corresponding to the data, for each data unit for streaming the shooting data, and then live streaming the shooting data;
A streaming server for comparing the correction timestamps of the data units of the photographing data based on a universal time to synchronize the data units of the photographing data and then live streaming the photographing data; And
And an image processor for performing live streaming of an output image obtained by merging the photographed data after removing the overlap between the photographed images by the photographed data that is live streamed in the streaming server. . 17. The method of claim 16,
Wherein the streaming server generates the time correction values corresponding to the time difference between the reception times of the data units and the correction timestamps for each shot data and provides the time correction values to the corresponding encoders, system.

Images