WO2020138536A1

WO2020138536A1 - System and method for transmitting image on basis of hybrid network

Info

Publication number: WO2020138536A1
Application number: PCT/KR2018/016726
Authority: WO
Inventors: 김동호; 서봉석
Original assignee: 서울과학기술대학교 산학협력단
Priority date: 2018-12-24
Filing date: 2018-12-27
Publication date: 2020-07-02
Also published as: KR102158007B1; KR20200078818A

Abstract

The present invention relates to a system and method for transmitting stereoscopic images on the basis of a hybrid network. A specific embodiment of the present invention can provide immersive media such as monoscopic 360 VR in HD resolution, monoscopic 360-degree VR in UHD resolution, stereoscopic 360-degree VR in UHD resolution due to signaling with an MPD structure for a base layer stream, enhancement layer stream, and upsampled base layer stream, which are derived from high-resolution left-eye view images and right-eye view images captured by a 360-degree camera, and can reduce system complexity and improve compression efficiency by employing a single encoder to derive the base layer stream, enhancement layer stream, and upsampled base layer stream.

Description

Hybrid network based video transmission system and method

The present invention relates to a hybrid network-based video transmission system and method, and more specifically, using a single encoder/decoder, ATSC 3.0-based ROUTE / DASH protocol based mono HD-class resolution 360 VR (Virtual Reality), It relates to a technology capable of providing a 360-degree VR in mono UHD-class resolution and a 360-degree VR video service in stereo UHD-class resolution.

With the recent increase in IP-based requirements, the standardization of ATSC 3.0, the world's first IP-based broadcasting system, has been completed. One of the ultimate objectives of ATSC 3.0 is to focus on compatibility with existing fixed TVs and users' mobile devices, and to enable service to both categories of users through a single transmission method.

The transmission method used by ATSC 3.0 is largely divided into two categories: MPEG Media Transport Protocol (MMPT) and Real-Time Object delivery over Unidirectional Transport (ROUTE). Last year, the ROSC method was used to successfully test the ATSC 3.0 broadcasting system.

Recently, as virtual reality devices have become popular, requirements for tangible media such as 360-degree VR (Virtual Reality) videos have also increased. Currently, these contents are provided to users mainly on video sharing sites, and they are also used in various fields such as education, navigation, and games. These 360-degree VR videos are expected to greatly affect the broadcasting industry in the future, and many scholars expect it to completely change the existing broadcasting paradigm.

However, based on the recently completed ATSC 3.0 broadcast protocol, high resolution stereoscopic 360 degree VR video transmission is not considered.

The present invention is to provide a hybrid network-based image transmission system and method capable of reducing complexity and maximizing compression efficiency by using the MPD structure of one encoder and decoder.

In addition, according to the present invention, signaling is performed in MPD structures for a base layer stream, an enhancement layer stream, and an up-sampled base layer stream derived from a high-resolution left-eye image and a right-eye image captured through a 360-degree camera that is sensible to a viewer. It is intended to provide a hybrid network-based video transmission system and method capable of providing sensational media services such as a 360-degree VR with mono HD-definition resolution, a 360-degree VR with mono UHD-definition resolution, and a 360-degree VR with stereo UHD-definition resolution.

The object of the present invention is not limited to the above-mentioned object, other objects and advantages of the present invention which are not mentioned can be understood by the following description, and will be more clearly understood by embodiments of the present invention. It will also be readily appreciated that the objects and advantages of the present invention can be realized by means of the appended claims and combinations thereof.

Accordingly, the hybrid network-based video transmission system according to an embodiment of the present invention

An image acquisition unit acquiring high-resolution left-eye images and right-eye images respectively using a 360 camera; A pre-processing unit that performs tiling with a predetermined size for each of the obtained left-eye image and right-eye image; A plurality of down-sampling units for down-sampling each of the tiled left-eye images and right-eye images in a predetermined number of unit resolutions; An encoder for encoding each of the down-sampled left-eye and right-eye images of each unit resolution to output a base layer stream and an enhancement layer stream; And an up-sampling unit performing up-sampling the base layer stream at a high resolution to output the base layer stream, the enhancement layer stream, and the up-sampled base layer stream. And a broadcasting unit that standardizes and multiplexes at least one of the base layer stream, the enhancement layer stream, and the up-sampled base layer stream according to a broadcast network or Internet standard, and then transmits the same.

Preferably, the plurality of down-sampling units set an arbitrary resolution of a predetermined original resolution as a reference resolution, and then sequentially perform down-sampling on a tiled right-eye image at a unit resolution less than the preset reference resolution, and exceed the reference resolution It may be provided to sequentially perform down-sampling on the left-eye image tiled with a unit resolution of.

The encoder performs encoding on a down-sampled right-eye image of each unit resolution to output a base layer stream of each unit resolution, and for each down-sampled left-eye image of each unit resolution, based on the base layer stream, each unit resolution It may be provided to output the enhancement layer stream of.

Preferably, the broadcasting unit generates MPD (Media Presentation Description) information for the base layer stream and the enhancement layer stream, and then generates at least one of the MPD information, the base layer stream and the enhancement layer stream, and the upsampled base layer stream. It may be provided to propagate.

Preferably, the MPD includes tiling information that is provided as a spatial relationship description (SRD) and describes a viewpoint and a position between the photographed left-eye image and the right-eye image tile; And dependency information describing a base layer stream ID of each unit resolution depending on each of a plurality of enhancement layer streams for each unit resolution generated based on the base layer stream of each unit resolution.

According to another embodiment of the present invention, a hybrid network-based image transmission method includes an image acquisition step of obtaining a high-resolution left-eye image and a right-eye image respectively using a 360 camera; A pre-processing step of performing tiling on each of the obtained left-eye image and right-eye image in a predetermined size; A plurality of down-sampling steps for down-sampling the tiled left-eye image and the right-eye image, respectively, in predetermined unit resolutions; An encoder that performs encoding on each of the down-sampled left-eye and right-eye images and outputs a base layer stream and an enhancement layer stream; And an up-sampling step of performing up-sampling the base layer stream at a high resolution to output the base layer stream, the enhancement layer stream, and the up-sampled base layer stream. And a broadcasting step of standardizing and multiplexing at least one of the base layer stream, the enhancement layer stream, and the up-sampled base layer stream according to a broadcast network or Internet network standard specification, and then transmitting the same.

Preferably, the plurality of down-sampling steps include setting an arbitrary resolution from a predetermined original resolution as a reference resolution; Sequentially performing down-sampling on the tiled right-eye image at a unit resolution less than the preset reference resolution; And sequentially performing downsampling on the left-eye image tiled with a unit resolution exceeding the reference resolution.

According to the present invention, mono HD-level resolution is performed by signaling MPD structures for a base layer stream, an enhancement layer stream, and an up-sampled base layer stream derived from a high-resolution left-eye image and a right-eye image captured through a 360-degree camera. Can provide immersive media such as 360 VR, mono UHD-level resolution 360-degree VR, and stereo UHD-level resolution 360-degree VR.

According to the present invention, complexity of the system can be reduced and compression efficiency can be improved by deriving a base layer stream, an enhancement layer stream, and an up-sampled base layer stream using one encoder.

The following drawings attached in this specification are intended to illustrate preferred embodiments of the present invention, and serve to further understand the technical idea of the present invention together with the detailed description of the invention described below, and thus the present invention is described in such drawings. It is not limited to interpretation.

1 is a block diagram of an image transmission system of a hybrid network in this embodiment.

2 is a detailed configuration diagram of a downsampling unit of the system of this embodiment.

3 and 4 are exemplary diagrams of the MPD structure of the system of the present embodiment.

5 is a flowchart showing an image transmission process according to another embodiment.

Hereinafter, exemplary embodiments of the present invention will be described in detail with reference to the accompanying drawings so that those skilled in the art to which the present invention pertains may easily practice. However, the present invention can be implemented in many different forms and is not limited to the embodiments described herein. In addition, in order to clearly describe the present invention in the drawings, parts irrelevant to the description are omitted, and like reference numerals are assigned to similar parts throughout the specification.

Throughout the specification, when a part “includes” a certain component, it means that the component may further include other components, not to exclude other components, unless otherwise stated.

In an embodiment of the present invention, a transport stream applied to a broadcast network may be Real-time Object Delivery over Unidirectional Transport (ROTUE) or MPEG Media Transport Protocol (MMTP). ROUTE and MMTP represent a broadcast network transport stream multiplexing standard that is currently being standardized in ATSC 3.0. In addition, the transport stream transmitted through the Internet network conforms to the MPEG-DASH (Dynamic Adaptive Streaming over HTTP) standard.

According to the present invention, a high-resolution left-eye image and a right-eye image photographed through a 360-degree camera are tiled according to a service request mode, and downsampling is performed on a tiled right-eye image for each unit resolution below a predetermined reference resolution to be a base layer stream. Encoding, down-sampling the tiled left-eye image for each unit resolution higher than a predetermined reference resolution, encoding based on the base layer stream, outputting an enhancement layer stream, and then performing upsampling on the base layer stream to perform basic Based on the hierarchical stream, the enhanced hierarchical stream, and the up-sampled base hierarchical stream, immersive media such as HD 360 VR, mono UHD-level resolution 360-degree VR, and stereoscopic UHD-level resolution 360-degree VR are provided to the user.

Here, the hybrid network is a network that transmits a video stream to a broadcasting network and/or an Internet network.

1 is a configuration diagram of a hybrid network-based image transmission system according to an embodiment of the present invention, and FIG. 2 is a detailed configuration diagram of the down-sampling unit 300 illustrated in FIG. 1, referring to FIGS. 1 and 2 The system S according to an embodiment of the present invention includes an image acquisition unit 100, a pre-processing unit 200, a down-sampling unit 300, an encoder 400, an up-sampling unit 500, and a broadcasting unit ( 600).

The image acquisition unit 100 outputs each left-eye image and right-eye image captured through two cameras that support a predetermined resolution (eg, 8K resolution). Images taken from each camera may be used as UHD (Ultra High Definition) images or HD images, or images of some regions may be applied as stereoscopic UHD images in 360 degree VR.

Accordingly, the pre-processing unit 200 tiles each of the acquired left-eye image and right-eye image according to service requirements, and transmits the tiled left-eye image and right-eye image to the down-sampling unit 300. Here, for the left-eye image and the right-eye image, the tiles are the same size and the same number.

The down-sampling unit 300 is provided with a plurality of down-simpling modules 311 to 318 and sets any resolution (preferably set to a medium 4K resolution) of a predetermined original resolution (8K resolution) as a reference resolution, For the tiled right-eye image, down-sampling is sequentially performed at each unit resolution (1K to 3K) below the preset reference resolution, and down-sampling is sequentially performed for a left-eye image tiled at a unit resolution (4K to 8K) greater than or equal to the next reference resolution. Perform.

Here, the unit resolution is a randomly determined 1K unit resolution, and downsampling is performed in 1K, 2K, and 3K for a right eye image, respectively, and 4K, 5K, and 6K for a left eye image. Downsampling is performed in 7K and 8K, respectively. Here, the unit resolution is limited to 1K units, but it can be changed according to the specifications of the system, which can be understood by those skilled in the art related to the present embodiment. Each sampled left-eye and right-eye image is delivered to the encoder 400.

The encoder 400 encodes a down-sampled left-eye image of each unit resolution and outputs it in a stream form. Here, the encoder 400 is provided with a scalable high efficiency video codec (SHVC).

That is, the encoder 400 encodes a down-sampled left-eye image of each unit resolution, outputs a base layer stream of each unit resolution, and outputs a down-sampled right-eye image of each unit resolution based on a base layer stream of each unit resolution. Outputs the enhancement enhancement layer stream. Accordingly, a plurality of unit resolution enhancement layer streams are output to the base layer stream having one unit resolution.

In addition, the base layer stream is a transport stream of the right-eye image, and the enhancement layer stream is a transport stream of the left-eye image for providing high resolution (eg, 8K resolution) image quality for a user-selected or watched view (VoI).

The base layer stream of the encoder 400 is delivered to the up-sampling unit 500.

The up-sampling unit 500 performs up-sampling of the base layer stream of the encoder 400 at a high resolution, and then transmits the up-sampled base layer stream to the broadcasting unit 600. Accordingly, the broadcasting unit 600 receives a base layer stream of each unit resolution, an enhancement layer stream of each unit resolution based on each base layer stream, and a base layer stream of each unit resolution up-sampled.

When decoding the broadcasted stream, the broadcaster 600 performs MPD (Media Presentation Description) signaling that describes tiling information and information about the base layer stream and the up-sampled base layer stream, and the enhancement layer stream and MPD information. Produces

3 and 4 are exemplary diagrams showing the structure of MPD information. Referring to FIG. 3, MPD information includes tiling information and dependency information between a base layer and an enhancement layer stream.

That is, the tiling information includes a spatial relationship description (SRD) indicating a viewpoint and a position between tiles of the captured left-eye image and right-eye image, and the SRD of each tile is a component that can be independently encoded and decoded, and an adaptation set (Adaptation Set) ). For example, SRDs are specified as schemed Uri and value using Supplemental Property within each adaptation set. Accordingly, in the decoding process, a tile or a tile is identified by the scheme and Uri of the components in the corresponding adaptation set, and in the case of a tile, the location and time of the tile. For example, the adaptation set includes an adaptation set right tile 0 of the base layer, an adaptation set left tile 0-1 of the enhancement layer for 3K resolution, an adaptation set right tile 0-2 of the enhancement layer for 2K resolution, and the like. The supplemental property of the adaptation set right tile 0 of the base layer includes representation 1 of 5 Mbps (3K), representation 2 of 2 Mbps (2K), representation 3 of 1 Mbps (1K), and representation 4 of 0.5 Mbps (SD).

For example, the schemed Uri for SRD is "urn:m.peg:dash:srd:2014", and the value is expressed as "source_id, object_x, object_width, object_hight, total_width, total_high, spatial_set_id", and corresponding tile ID , It is possible to check the horizontal and vertical position of the upper left, the width and height of the corresponding tile, the width and height of the original image, and the ID of the grouped tiles.

That is, when tiled with 16 tiles in consideration of the original resolution of 8K, the resolution of each tile is 1920X1080. Therefore, as in the 19th line, the value of the Supplemental Property of the first tile in the left image is (2, 0, 0, 1920, 1080, 7680, 4320). The value of Supplemental Property of the second tile is (2, 1920, 0, 1920, 1080, 7680, 4320).

Meanwhile, referring to FIG. 4, MPD information further includes dependency information between a base layer and an enhancement layer for transmission based on a hybrid network. For example, when decoding the enhancement layer stream, the dependency ID indicating which of the base layer streams of each unit resolution depends on which base layer stream is used is represented by Representation of the adaptation set.

That is, as shown in FIGS. 3 and 4, since the base layer stream of each unit resolution includes a plurality of enhancement layer streams, the relationship between the base layer and the enhancement layer must be specified in the Representation, and the dependency ID is applicable. Since Representation is a signal for deciding which Representation should be dependent on, the dependency ID of the enhancement layer must match the Representation value of the base layer stream of the corresponding unit resolution.

The broadcasting unit 600 receiving the base layer stream, the enhancement layer stream, the up-sampled base layer stream, and the MPD information, the base layer stream, the enhancement layer stream according to the ATSC 3.0 broadcast platform (ROUTE) and the Internet platform (DASH) , Up-sampled base layer stream and MPD information are standardized and multiplexed, and thus, a mono UHD resolution 360-degree VR broadcast service, an HD resolution 360-degree VR broadcast service, and a stereoscopic UHD resolution 360-degree VR Experienced media such as broadcast services are provided to viewers.

For example, in an embodiment of the present invention, a base layer stream applied to a broadcast network, an enhancement layer stream, and an up-sampled base layer stream may be multiplexed by ROTUE. The broadcast network transport stream multiplexing standard follows ROUTE standardized in ATSC 3.0, and the transport stream transmitted through the Internet network follows a dynamic adaptive streaming over HTTP (DASH) standard.

To this end, the broadcasting unit 600 multiplexes the base layer stream and the enhancement layer stream and the up-sampled base layer stream by ROUTE/DASH to generate a transport stream, and the transport stream is a mono UHD 360-degree VR broadcast service and HD 360 It is transmitted to each channel or PLP for providing a VR broadcast service in FIG. and a 360 degree VR broadcast service in stereoscopic. Accordingly, the transport stream is produced and transmitted as an enhancement layer stream having a unit resolution of 4K to 8K in the base layer stream sea. Accordingly, the base layer stream is played as a mono HD class resolution 360 video through a mobile or low resolution display, and the base layer stream and the enhancement layer stream is played as a mono UHD class resolution 360 video through a high resolution display, and the base layer stream The up-sampled base layer stream and enhancement layer stream have a good band condition and are reproduced as a stereo UHD-level 360 video through a 3D display.

That is, as the base layer stream, the enhancement layer stream, and the up-sampled base layer stream and MPD information are additionally transmitted through the broadcasting network, the viewer may be provided with a mono UHD class or HD class resolution 360 VR broadcast service and a stereo UHD class resolution 360 VR. Broadcast service can be provided.

Accordingly, according to the present embodiment, signaling is performed in MPD structure for a base layer stream, an enhancement layer stream, and an up-sampled base layer stream derived from a high-resolution left-eye image and a right-eye image captured through a 360-degree camera. It can provide tangible media such as 360 VR with HD resolution, 360 degree VR with mono UHD resolution, and 360 degree VR with stereo UHD resolution.

In addition, complexity of the system may be reduced and compression efficiency may be improved by deriving a base layer stream, an enhancement layer stream, and an up-sampled base layer stream using one encoder.

5 is an overall flow chart showing a hybrid network-based image transmission process according to another embodiment of the present invention. Referring to FIG. 5, a hybrid network-based image transmission method according to another embodiment of the present invention includes an image acquisition step. (10), a pre-processing step 20, a down-sampling step 30, an encoding step 40, an up-sampling step 50, and a broadcasting step 60 may be included.

Here, the image acquiring step 10 acquires each of a high-resolution left-eye image and right-eye image using a 360 camera, and each of the obtained left-eye and right-eye images is transmitted to the pre-processing unit 200.

Then, each of the obtained left-eye and right-eye images performs tiling in a predetermined size, and the tiled left-eye and right-eye images are transmitted to the down sampling unit 300.

That is, in the down-sampling step 30, each of the tiled left-eye and right-eye images is set to a reference resolution of an arbitrary resolution of the intermediate resolution (4K) among the predetermined original resolution 8K resolution, and then the set reference for the tiled right-eye image. Down-sampling is sequentially performed at each unit resolution (1K, 2K, 3K) below the resolution.

Then, the down-sampling step 30 sequentially performs down-sampling on the left eye image tiled with each unit resolution (4K, 5K, 6K, 7K, 8K) exceeding the reference resolution, and the right eye down-sampled with each unit resolution. The image and the left eye image are delivered to the encoder 400.

Each of the right-eye image and the left-eye image down-sampled at each unit resolution performs encoding by the encoding step 40 to output a base layer stream and an enhancement layer stream. That is, a base layer stream of each unit resolution is output for a right-eye image downsampled at each unit resolution, and an enhancement layer stream of each unit resolution is output for each base layer stream of each unit resolution.

And the base layer stream is delivered to the up-sampling step 50, and the up-sampling step 50 performs up-sampling the base layer stream at a high resolution to perform the base layer stream and the enhancement layer stream and the up-sampled base. Output the hierarchical stream.

At least one of the base layer stream, the enhancement layer stream, and the up-sampled base layer stream is transmitted to a broadcasting step 60, and the broadcasting step 60 includes a base layer stream and the enhancement layer stream and the up sampling At least one of the base layer streams is standardized and multiplexed according to a standard specification of a broadcasting network or an Internet network, and then transmitted.

Accordingly, according to the present embodiment, as the signaling is performed in MPD structure for the base layer stream, the enhancement layer stream, and the up-sampled base layer stream derived from the high-resolution left-eye image and right-eye image captured through a 360-degree camera, mono HD It can provide sensational media such as 360 VR of class resolution, 360 degree VR of mono UHD class resolution, and 360 degree VR of stereo UHD class resolution.

Although described above with reference to preferred embodiments of the present invention, those skilled in the art can variously modify and change the present invention without departing from the spirit and scope of the present invention as set forth in the claims below. You will understand that you can.

Claims

An image acquisition unit acquiring high-resolution left-eye images and right-eye images respectively using a 360 camera;

A pre-processing unit that performs tiling with a predetermined size for each of the obtained left-eye image and right-eye image; A plurality of down-sampling units performing down-sampling on the tiled left-eye and right-eye images;

An encoder that performs encoding on each of the down-sampled left-eye and right-eye images and outputs a base layer stream and an enhancement layer stream;

An up-sampling unit configured to perform the up-sampling of the base layer stream at a high resolution to output the base layer stream, the enhancement layer stream, and the up-sampled base layer stream; And

And a broadcasting unit that standardizes and multiplexes the at least one of the base layer stream, the enhancement layer stream, and the up-sampled base layer stream according to a broadcast network standard specification, and then transmits the same.
The method of claim 1, wherein the plurality of down-sampling unit,

After setting an arbitrary resolution among the predetermined original resolutions as a reference resolution, downsampling is sequentially performed on a tiled right-eye image at a unit resolution less than the preset reference resolution, and the tiled right-eye image is tiled at a unit resolution exceeding the reference resolution. Sequential downsampling,

The encoder

Encoding of the down-sampled right-eye image of each unit resolution is performed to output a base layer stream of each unit resolution, and for the down-sampled left-eye image of each unit resolution, an enhancement layer of each unit resolution is based on the base layer stream. Hybrid network-based video transmission system characterized in that it is provided to output the stream.
According to claim 2, The broadcasting unit

It is provided to generate media presentation description (MPD) information for the base layer stream and the enhancement layer stream, and then propagate at least one of the MPD information, the base layer stream and the enhancement layer stream, and the upsampled base layer stream,

The MPD information

Tiling information that is provided as a spatial relationship description (SRD) and describes a viewpoint and a position between a left-eye image and a right-eye image tile; And dependency information describing a base layer stream ID of each unit resolution depending on each of the plurality of enhancement layer streams generated based on the base layer stream of each unit resolution. .
An image acquiring step of acquiring high-resolution left-eye images and right-eye images respectively using a 360 camera;

A pre-processing step of performing tiling with a predetermined size for each of the obtained left-eye image and right-eye image;

A plurality of down-sampling steps for down-sampling the tiled left-eye and right-eye images;

An encoding step of encoding each of the down-sampled left-eye image and right-eye image and outputting a base layer stream and an enhancement layer stream;

An up-sampling step of performing up-sampling the base layer stream at a high resolution to output the base layer stream, the enhancement layer stream, and the up-sampled base layer stream; And

And a broadcasting step of standardizing and multiplexing at least one of the base layer stream, the enhancement layer stream, and the up-sampled base layer stream according to a broadcast network standard specification, and then transmitting the same. .
The method of claim 4, wherein the plurality of down-sampling steps

Setting an arbitrary resolution among predetermined original resolutions as a reference resolution; Sequentially performing down-sampling on the tiled right-eye image at a unit resolution less than the preset reference resolution; And sequentially performing downsampling on the left-eye image tiled with a unit resolution exceeding the reference resolution,

The encoding step,

Encoding a down-sampled right-eye image of each unit resolution to output a base layer stream of each unit resolution; And outputting an enhancement layer stream of each unit resolution based on the base layer stream for a down-sampled left-eye image of each unit resolution.