KR20200076529A - Indexing of tiles for region of interest in virtual reality video streaming - Google Patents

Abstract

An image transmission method of an image transmission device disclosed in the present specification is a method performed in the image transmission device including a processor. The image transmission method includes steps of: receiving first signaling data including a sphere coordinates for a user′s region of interest within a 360-degree virtual reality space; mapping the region of interest to a 2D plane based on the sphere coordinates and preset projection information; and calculating the indexing information of the region of interest tile corresponding to the mapped region of interest; and transmitting video data and second signaling data for a 360-degree virtual reality space based on the indexing information.

Description

INDEXING OF TILES FOR REGION OF INTEREST IN VIRTUAL REALITY VIDEO STREAMING}

This specification relates to streaming virtual reality video.

Recently, with the development of virtual reality technology and equipment, wearable devices such as a head-mounted display (HMD) have been introduced. Among various service scenarios through virtual reality technology and a head-mounted video display device, there are not only movie viewing and games, but also video conferencing and telesurgery.

Since the head-mounted video display device is played in front of the user by wearing it on the head and needs to play a spherical 360-degree screen, the virtual reality technology and the head-mounted video display device allow the user to experience virtual reality without awkwardness. In the service scenario, ultra high-definition video of UHD (Ultra High-Definition) or higher is required.

Although high computing power is required for the reproduction of ultra-high-definition 360-degree images, and high bandwidth is required for the transmission of ultra-high-definition 360-degree image data, it is possible to display ultra-high-definition 360-degree images with the currently available head-mounted video display device and virtual reality technology. There is a limit to transferring and playing. Accordingly, there is an increasing need for an image processing technology for efficiently transmitting an ultra-high-definition 360-degree image and increasing an image reproduction speed.

The representative efficient transmission methods include a user gaze-based method and a user gaze-based method. The user gaze ratio-based method lowers transmission bandwidth by adjusting or downsampling QP values for each main area of the image to be transmitted instead of transmitting data for the entire 360-degree image to the head-mounted image display device. The user gaze-based method transmits only the tile corresponding to the area viewed by the user in high quality, considering that a part of the 360-degree video is partially viewed by the user. The user gaze-based method has a large reduction in transmission bandwidth due to selective transmission of image data, and a low computational complexity of the decoder and encoder. In addition, the area that the user does not look at is transmitted with low image quality to alleviate the delay.

The user gaze-based method needs to accurately determine the user gaze area. However, in order to accurately determine the user's gaze, there are the following problems. The direction in which the head mounted image display device (HMD) looks is different from the direction in which the user's gaze is directed, and the tile index of the user's region of interest is different according to the type of projection. Accordingly, there is an increasing need for an efficient transmission technology in consideration of user gaze and projection.

This specification presents a video transmission method of the video transmission device. The image transmission method is a method performed by an image transmission device including a processor, and includes information on a region of interest based on a spherical coordinate of a user's region of interest and a type of the region of interest in a 360-degree virtual reality space. Receiving first signaling data, mapping the region of interest to a two-dimensional plane based on the sphere coordinates and preset projection information, and calculating indexing information of a region of interest tile corresponding to the mapped region of interest And transmitting video data and second signaling data for a 360-degree virtual reality space based on the indexing information.

The above method and other embodiments may include the following features.

The second signaling data may include the projection information and mapped region of interest information based on the projection information.

Also, the region of interest type information may be information indicating either a viewport based on a head-mounted image display device or a viewport based on a user's viewpoint.

In addition, the method further includes preparing basic quality video data for at least a portion of the 360-degree virtual reality space, and preparing high-definition video data related to the region of interest tile, and the 360-degree virtual reality space The video data for may include the basic quality video data and the high quality video data.

In addition, the base quality video data may include base layer video data for the 360-degree virtual reality space, and the high definition video data may include base layer video data and enhancement layer video data for the region of interest tile.

In addition, the spherical coordinates are based on the yaw coordinates and pitch coordinates of the second central point where the segments connected between the first central points of the two sides opposite to the central points of the four sides of the region of interest and the second central point. It may be represented by one horizontal length information and vertical length information.

In addition, the first signaling data and the second signaling data are generated based on image configuration information, and the image configuration information indicates gaze information indicating the viewports in the 360-degree virtual reality space and the viewing angle of the user It may include zoom area information.

In addition, the first signaling data and the second signaling data include High-Level Syntax Protocol (SEI) for carrying session information, Supplement Enhancement Information (SEI), video usability information (VUI), Slice Header, And a file describing the video data.

In addition, the projection method is Equi-Rectangular Projection (ERP), Cube Map Projection (CMP), Adjusted Equal-Area Projection (AEP), Octahedron Projection (OHP), Icosahedron Projection (ISP), Truncated Square Pyramid (TSP), Segmented Sphere Projection (SSP), Adjusted Cube Map Projection (ACP), Rotated Sphere Projection (RSP), Equatorial Cylindrical Projection (ECP) and Equi-Angular Cube Map (EAC).

Meanwhile, this specification proposes an image transmission device. The image transmission device is a communication unit for receiving first signaling data including sphere coordinates for a user's region of interest and type of region of interest based on a type of the region of interest in a 360 degree virtual reality space, the sphere coordinates And a controller that maps the region of interest to a two-dimensional plane based on preset projection information, a signaling data generator that calculates indexing information of a region of interest tile corresponding to the mapped region of interest, and 360 based on the indexing information. Also may include a transmitter for transmitting video data and second signaling data for the virtual reality space.

The device and other embodiments may include the following features.

The encoder may further include basic quality video data for at least a portion of the 360-degree virtual reality space and high-definition video data for the region of interest tile.

In addition, the base quality video data includes base layer video data for at least a portion of the 360 degree virtual reality space, and the high definition video data includes base layer video data and enhancement layer video data for the region of interest tile. Can.

Also, the region of interest type information may be information indicating either a viewport based on a head-mounted image display device or a viewport based on a user's viewpoint.

In addition, the spherical coordinates are based on the yaw coordinates and pitch coordinates of the second central point where the segments connected between the first central points of the two sides opposite to the central points of the four sides of the region of interest and the second central point. It may be represented by one horizontal length information and vertical length information.

On the other hand, the present specification proposes a video reproducing method of the video reproducing apparatus. The image reproducing method is a method performed by an image reproducing apparatus including a processor, and includes sphere coordinates for a user's region of interest in the 360-degree virtual reality space and region of interest type information based on the type of the region of interest. Transmitting first signaling data, receiving second signaling data and video data for the 360-degree virtual reality space, and video based on video data for the 360-degree virtual reality space and the second signaling data And reproducing, wherein the second signaling data includes indexing information of a region of interest tile corresponding to the region of interest mapped to a 2D plane based on the sphere coordinates, the region of interest type information, and preset projection information. The video data for the 360-degree virtual reality space may include basic quality video data for at least a portion of the 360-degree virtual reality space and high-definition video data for the tile of interest.

The above method and other embodiments may include the following features.

The operation of reproducing an image based on the video data for the 360-degree virtual reality space and the second signaling data includes the region of interest based on the video data for the 360-degree virtual reality space and the second signaling data. The image for the user viewport can be rendered and displayed.

On the other hand, this specification proposes a computer program for image transmission stored on the medium. The computer program, in the computing device, receiving first signaling data including sphere coordinates for a user's region of interest in the 360-degree virtual reality space and region-of-interest type information based on the type of the region of interest , Mapping the region of interest to a two-dimensional plane based on the spherical coordinates and preset projection information, calculating indexing information of a region of interest tile corresponding to the mapped region of interest, and based on the indexing information An operation of transmitting video data and second signaling data for a 360-degree virtual reality space may be executed.

According to the embodiments disclosed herein, a user's region of interest for a 360-degree virtual reality space is divided into a viewport of a head-worn video display device and a user's point of view based viewport, and a user's region of interest is mapped according to a projection type to correct the user. By transmitting the information of the region of interest, it is possible to transmit a high-definition video only to the region of interest of the user, thereby securing the transmission bandwidth.

In addition, according to the embodiments disclosed herein, video data in a 360-degree virtual reality space includes high-definition video data only for a user's region of interest, and other areas include video data of a basic quality, so that the video is reproduced in the video reproducing apparatus. This has the effect of reducing the delay of reproduction.

In addition, according to the embodiments disclosed in the present specification, information about a user's region of interest, jeop for a projection type, mapping information of a user's region of interest according to a projection type, etc. are transmitted through signaling data, and transmitted in a virtual reality service system. It has the effect of securing the bandwidth for and improving the service speed.

In addition, according to the embodiments disclosed in the present specification, in consideration of projection in a 360-degree virtual reality space, information on a region of interest to which a user's gaze is actually transmitted is transmitted by expressing metadata with respect to a position viewed by the head-mounted display device. , It has the effect of delivering accurate user interest area information and achieving video optimized for bandwidth through adaptive video transmission according to the network environment.

1 shows an example virtual reality system that provides a virtual reality image.
2 is a diagram illustrating an example scalable video coding service.
3 is a block diagram showing an exemplary configuration of a server device.
4 is a block diagram showing an exemplary configuration of an encoder of a server device.
5 is a diagram illustrating an exemplary method of signaling a region of interest.
6 is a block diagram showing an exemplary configuration of a client device.
7 is a block diagram showing an exemplary configuration of a control unit of a client device.
8 is a block diagram showing an exemplary configuration of a decoder of a client device.
9 is a diagram illustrating a tile indexing method of a user's region of interest according to projection information.
10 is a block diagram of an exemplary video transmission device for a video streaming service.
11 is a flowchart of an exemplary video transmission method for a video streaming service.
12 is a flowchart of an exemplary video playback method for a video streaming service.
13 is a diagram illustrating an exemplary OMAF syntax in the international video standard for signaling in tile indexing of a user's region of interest according to projection.
14 is a diagram illustrating an example of a tile indexing information syntax of a region of interest expressed in the form of XML.

The technology disclosed herein can be applied to a 360-degree virtual reality system that provides image streaming between an image transmission device and an image replay device. However, the technology disclosed in this specification is not limited thereto, and may be applied to all electronic devices and methods to which the technical spirit of the technology can be applied.

It should be noted that the technical terms used in this specification are only used to describe specific embodiments, and are not intended to limit the spirit of the technology disclosed herein. In addition, technical terms used in this specification should be interpreted as meanings generally understood by a person having ordinary knowledge in the field to which the technology disclosed in this specification belongs, unless defined otherwise. It should not be interpreted as a comprehensive meaning or an excessively reduced meaning. In addition, when the technical term used in this specification is an incorrect technical term that does not accurately represent the spirit of the technology disclosed in the present specification, a technical term that can be correctly understood by a person having ordinary knowledge in the field to which the technology disclosed in this specification belongs Should be understood. In addition, the general terms used in this specification should be interpreted as defined in the dictionary or in context before and after, and should not be interpreted as an excessively reduced meaning.

Terms including ordinal numbers such as first and second used in the present specification may be used to describe various components, but the components should not be limited by the terms. The terms are used only for the purpose of distinguishing one component from other components. For example, the first component may be referred to as a second component without departing from the scope of the present invention, and similarly, the second component may also be referred to as a first component.

Hereinafter, exemplary embodiments disclosed herein will be described in detail with reference to the accompanying drawings, but the same or similar elements are assigned the same reference numbers regardless of the reference numerals, and overlapping descriptions thereof will be omitted.

In addition, in the description of the technology disclosed in the present specification, when it is determined that the detailed description of the related known technology may obscure the gist of the technology disclosed herein, the detailed description will be omitted. In addition, it should be noted that the accompanying drawings are only for easily understanding the spirit of the technology disclosed in the present specification, and should not be interpreted as limiting the spirit of the technology by the accompanying drawings.

1 shows an example virtual reality system that provides a virtual reality image.

The virtual reality system includes a virtual reality image generating apparatus that generates a virtual reality image, a server device that encodes and transmits the input virtual reality image, and one or more client devices that decode and output the transmitted virtual reality image to a user. It can be configured to.

Referring to FIG. 1, the exemplary virtual reality system 100 includes a virtual reality image generating apparatus 110, a server device 120, and one or more client devices 130, and each component illustrated in FIG. 1 The number of examples is illustrative only and is not limited thereto. The virtual reality system 100 may also be referred to as a 360-degree image providing system.

The virtual reality image generating apparatus 110 may generate a spatial image by capturing an image of a space in which the device itself is located, including one or more camera modules.

The server device 120 generates a 360-degree image by stitching, projecting, and mapping an image of the virtual reality space generated and input by the virtual reality image generating apparatus 110 and , The generated 360-degree image may be adjusted to video data of a desired quality and then encoded.

Further, the server device 120 may transmit a bitstream including video data for the encoded 360-degree video and signaling data for the video data to the client device 130 through a network (network).

The client device 130 may decode a received bitstream (Decoding) to output a 360-degree image to a user wearing the client device 130. The client device 130 may be a near-eye display device such as a head-mounted display (HMD).

On the other hand, the virtual reality image generating apparatus 110 may be configured with a computer system to generate an image of a virtual 360-degree space implemented with computer graphics. In addition, the virtual reality image generating apparatus 110 may be a provider of virtual reality content such as a virtual reality game.

The client device 130 may obtain user data from a user who uses the client device 130. The user data may include user image data, audio data, viewport data (gaze data), region of interest data, and additional data.

For example, the client device 130 may include at least one of a 2D/3D camera and an Immersive camera that acquire image data of a user. 2D/3D cameras can take images with a viewing angle of 180 degrees or less. The Immersive camera can shoot images with a viewing angle of 360 degrees or less.

For example, the client device 130 may include a first client device 131 obtaining user data of a first user located in a first place, and a second client device obtaining user data of a second user located in a second place. (133), and a third client device 135 for acquiring user data of a third user located in a third place.

Each client device 130 may transmit user data obtained from a user to the server device 120 through a network.

The server device 120 may receive at least one user data from the client device 130. The server device 120 may generate an entire image of the virtual reality space based on the received user data. The entire image generated by the server device 120 may represent an immersive image providing a 360-degree image in the virtual reality space. The server device 120 may generate the entire image by mapping the image data included in the user data to the virtual reality space.

The server device 120 may transmit the generated entire image to each user.

Each client device 130 may receive the entire image and render and/or display only the area viewed by each user in the virtual reality space.

2 is a diagram illustrating an example scalable video coding service.

The scalable video coding service is a video compression method for providing various services in terms of time, space, and image quality according to various user environments such as network conditions or terminal resolution in various multimedia environments. A scalable video coding service generally provides scalability in terms of spatial resolution, quality, and time.

Spatial scalability can be serviced by encoding the same image with different resolution for each layer. It is possible to adaptively provide video content to devices having various resolutions, such as digital TVs, laptops, and smart phones, using spatial hierarchies.

Referring to the drawings, a scalable video coding service may simultaneously support TVs having one or more different characteristics from a video service provider (VSP) through a home gateway in a home. For example, the scalable video coding service may simultaneously support HDTV (High-Definition TV), SDTV (Standard-Definition TV), and LDTV (Low-Definition TV) having different resolutions.

Temporal scalability may adaptively adjust the frame rate of an image in consideration of a network environment in which content is transmitted or performance of a terminal. For example, when a local area network is used, a service is provided at a high frame rate of 60 frames per second (FPS), and when a wireless broadband communication network such as a 3G mobile network having a slower transmission speed than the local area network is used, By providing the content at a low frame rate of 16FPS, a service can be provided so that the user can receive the video seamlessly. However, if a high-speed wireless broadband communication network such as a 5G mobile network is used, the service can be provided at a high frame rate of 60 FPS.

Quality scalability also provides contents of various image quality according to network environment or terminal performance, so that users can stably play video contents.

The scalable video coding service may include a base layer and one or more enhancement layers(s), respectively. When the receiver receives only the base layer, the image can be provided in general quality, and when the base layer and the enhancement layer are received together, the receiver can provide high quality. That is, when there is a base layer and one or more enhancement layers, the more an enhancement layer (eg, enhancement layer 1, enhancement layer 2, ..., enhancement layer n) is received while the base layer is received, the more image quality or quality it provides. The quality is improved.

As described above, since the video of the scalable video coding service is composed of a plurality of layers, the receiver receives a small amount of base layer data quickly, processes and reproduces the image of general quality quickly, and adds enhancement layer image data if necessary. You can increase the quality of the service.

3 is a diagram showing an exemplary configuration of a server device.

The server device 300 may include a control unit 310 and/or a communication unit 320.

The control unit 310 may generate the entire image for the virtual reality space and encode the generated whole image. Also, the control unit 310 may control all operations of the server device 300. Details will be described below.

The communication unit 320 may transmit and/or receive data to an external device and/or a client device. For example, the communication unit 320 may receive user data and/or signaling data from at least one client device. Also, the communication unit 320 may transmit an entire image of the virtual reality space and/or an image of a partial region to the client device.

The control unit 310 may include at least one of a signaling data extraction unit 311, an image generation unit 313, a region of interest determination unit 315, a signaling data generation unit 317, and/or an encoder 319. have.

The signaling data extraction unit 311 may extract signaling data from data transmitted from the client device. For example, the signaling data may include image configuration information. The image configuration information may include gaze information indicating a user's gaze direction in a virtual reality space and zoom area information indicating a user's viewing angle. In addition, the image configuration information may include user's viewport information in a virtual reality space.

The image generating unit 313 may generate an entire image for the virtual reality space and an image for a specific area in the virtual reality space.

The ROI determining unit 315 may determine the ROI corresponding to the user's gaze direction within the entire region of the virtual reality space. In addition, it is possible to determine a user's viewport within the entire area of the virtual reality space. For example, the ROI determination unit 315 may determine the ROI based on gaze information and/or zoom region information. For example, the region of interest is the location of a tile in which the important object will be placed in the virtual space that the user will see (eg, a location where new enemies appear in a game, etc., the speaker's location in the virtual reality space), and/or the user It may be the place where the gaze of is looking. In addition, the region of interest determining unit 315 may generate region of interest information indicating a region of interest corresponding to a user's gaze direction and information about a user's viewport within the entire region of the virtual reality space.

The signaling data generation unit 317 may generate signaling data for processing the entire image. For example, the signaling data may transmit region of interest information and/or viewport information. The signaling data may be transmitted through at least one of SEI (Supplement Enhancement Information), VUI (video usability information), a slice header, and a file describing video data.

The encoder 319 may encode the entire image based on signaling data. For example, the encoder 319 may encode the entire image in a manner customized to each user based on the gaze direction of each user. For example, when a user views a specific point in the virtual reality space, the encoder encodes an image corresponding to the specific point in high quality based on the user's gaze in the virtual reality space, and the image corresponding to the point other than the specific point is low quality. Can be encoded as According to an embodiment, the encoder 319 may include at least one of a signaling data extraction unit 311, an image generation unit 313, a region of interest determination unit 315, and/or a signaling data generation unit 317. have.

In addition, the control unit 310, a signaling data extracting unit 311, an image generating unit 313, a region of interest determining unit 315, a signaling data generating unit 317, and an encoder 319 other than the processor (not shown) ), a memory (not shown), and an input/output interface (not shown).

The processor may include one or more of a central processing unit (CPU), an application processor (AP), or a communication processor (CP). The processor may execute, for example, calculation or data processing related to control and/or communication of at least one other component of the control unit 310.

Also, the processor may be implemented with, for example, a System on Chip (SoC). According to an embodiment, the processor may further include a graphic processing unit (GPU) and/or an image signal processor.

In addition, the processor may control, for example, a plurality of hardware or software components connected to the processor by driving an operating system or an application program, and may perform various data processing and operations.

In addition, the processor may load and process instructions or data received from at least one of other components (eg, non-volatile memory) into a volatile memory, and store various data in a non-volatile memory. have.

The memory may include volatile and/or non-volatile memory. The memory may store, for example, commands or data related to at least one other component of the control unit 310. According to one embodiment, the memory may store software and/or programs.

The input/output interface, for example, may serve as an interface that can transmit commands or data input from a user or other external device to other component(s) of the controller 310. In addition, the input/output interface may output commands or data received from other component(s) of the control unit 310 to a user or another external device.

Hereinafter, an exemplary image transmission method using a region of interest will be described.

The server device may receive video data and signaling data from at least one client device using a communication unit. Further, the server device may extract signaling data using the signaling data extraction unit. For example, the signaling data may include viewpoint information and zoom area information.

The gaze information may indicate which area (point) the user views in the virtual reality space. When the user views a specific area in the virtual reality space, gaze information may indicate a direction from the user to the specific area.

The zoom area information may indicate an enlargement range and/or a reduction range of video data corresponding to a user's gaze direction. Also, the zoom area information may indicate a user's viewing angle. When the video data is enlarged based on the value of the zoom area information, the user can see only a specific area. When the video data is reduced based on the value of the zoom area information, the user can view not only a specific area, but also part and/or all of the areas other than the specific area.

The server device may generate an entire image of the virtual reality space using the image generation unit.

The server device may use the region of interest determination unit to grasp image configuration information for a viewpoint and a zoom region viewed by each user in the virtual reality space based on the signaling data. The region of interest determining unit may determine the region of interest of the user based on the image configuration information.

When signaling data (eg, at least one of viewpoint information and zoom area information) is changed, the server device may receive new signaling data. In this case, the server device may determine a new region of interest based on the new signaling data.

The control unit of the server device may determine whether the data currently being processed is data corresponding to the ROI based on the signaling data.

When the signaling data is changed, the server device may determine whether the data currently being processed is data corresponding to the ROI based on the new signaling data. When the data currently being processed is data corresponding to a region of interest, the server device may encode video data (eg, a region of interest) corresponding to the user's viewpoint using a encoder. For example, the server device may generate base layer video data and enhancement layer video data for the video data corresponding to the user's viewpoint and transmit them.

In addition, when signaling data is changed, the server device may transmit video data (a new region of interest) corresponding to a new viewpoint as a high-quality image. In the past, when the server device was transmitting the image of the basic quality, but the signaling data is changed and the server device needs to transmit the image of the high quality, the server device may additionally generate and/or transmit enhancement layer video data.

If the data currently being processed based on the new signaling data is data that does not correspond to the region of interest, the server device may encode video data (eg, non-interested region) that does not correspond to the user's point of view with the basic image quality. have. For example, the server device may generate and transmit only the base layer video data for video data that does not correspond to the user's viewpoint.

When the signaling data is changed, the server device may transmit video data (a new non-interested region) that does not correspond to a new user's viewpoint as an image having a basic quality. If the server device has been transmitting high-definition video in the past, but the signaling data has changed and the server device has to transmit the video with the basic quality, the server device no longer generates and/or transmits at least one enhancement layer video data. , Only base layer video data can be generated and/or transmitted.

That is, since the quality of the video data when the base layer video data is received is lower than the quality of the video data when the enhancement layer video data is received, the client device is the user at the moment when the user obtains information from the sensor or the like. Enhancement layer video data may be received for video data corresponding to the gaze direction of (for example, video data for a region of interest). And, the client device can provide high quality video data to the user in a short time.

4 is a diagram showing an exemplary configuration of an encoder of a server device.

The encoder 400 (image encoding apparatus) may include at least one of a base layer encoder 410, at least one enhancement layer encoder 420, and a multiplexer 430.

The encoder 400 may encode the entire image using a scalable video coding method. The scalable video coding method may include SVC (Scalable Video Coding) and/or SVC (Scalable High Efficiency Video Coding).

The scalable video coding method is an image compression method for providing various services in a hierarchical (Scalable) view in terms of time, space, and image quality according to various user environments such as network conditions or terminal resolution in various multimedia environments. For example, the encoder 400 may generate a bitstream by encoding two or more different quality (or resolution, frame rate) images of the same video data.

For example, the encoder 400 may use inter-layer prediction tools, which is an encoding method using inter-layer redundancy, to increase compression performance of video data. The inter-layer prediction tool is a technique for improving compression efficiency in an enhancement layer (EL) by removing redundancy of images existing between layers.

The enhancement layer may be encoded by referring to information of a reference layer using an inter-layer prediction tool. The reference layer refers to a lower layer referenced when encoding an enhancement layer. Here, since a dependency exists between layers by using the inter-layer prediction tool, bitstreams of all the referenced lower layers are required to decode the image of the highest layer. In the middle layer, decoding can be performed by acquiring only the bitstream of the layer to be decoded and its lower layers. The bitstream of the lowest layer is a base layer (BL), and may be encoded by encoders such as H.264/AVC and HEVC.

The base layer encoder 410 may encode the entire image to generate base layer video data (or a base layer bitstream) for the base layer. For example, the base layer video data may include video data for the entire area viewed by the user in the virtual reality space. The base layer image may be the lowest quality image.

The enhancement layer encoder 420 may encode at least one enhancement layer for at least one enhancement layer predicted from the base layer by encoding the entire image based on signaling data (eg, region of interest information) and base layer video data. Video data (or enhancement layer bitstream) can be generated. The enhancement layer video data may include video data for a region of interest within the entire region.

The multiplexer 430 may multiplex base layer video data, at least one enhancement layer video data, and/or signaling data, and generate one bitstream corresponding to the entire image.

5 exemplarily shows a method of signaling a region of interest, and shows a method of signaling a region of interest in scalable video coding.

Referring to FIG. 5, the server device (or encoder) includes one video data (or picture) composed of enhancement layers in scalable video data 500 composed of a base layer BL and at least one enhancement layer EL. Can be divided into several tiles (510) having a rectangular shape. For example, video data may be divided into Coding Tree Unit (CTU) units as boundaries. For example, one CTU may include Y CTB, Cb CTB, and Cr CTB.

For quick user response, the server device may encode the video data of the base layer BL without dividing it into tiles.

The server device may encode video data of one or more enhancement layers by dividing some or all of them into several tiles as necessary. That is, the server device may divide the video data of the enhancement layer into at least one tile and encode tiles corresponding to a region of interest (ROI) 520.

At this time, the region of interest 520 is the location of tiles in which the important object to be viewed by the user in the virtual reality space is located (eg, a location where a new enemy appears in the game, a speaker in the virtual space in video communication) Location), and/or the user's gaze.

In addition, the server device may generate region of interest information including tile information for identifying at least one tile included in the region of interest. For example, the region of interest information may be generated by the region of interest determination unit included in the server device, the signaling data generation unit, and/or the encoder.

The tile information of the region of interest 520 is continuous, and thus can be effectively compressed even if all of the tile numbers are not obtained. For example, the tile information may include all tile numbers corresponding to the region of interest, as well as tile start and end numbers, coordinate point information, a CU (Coding Unit) number list, and tile numbers expressed by formulas. .

Also, the region of interest 520 may be a user's current viewport.

The tile information of the non-interested region may be transmitted to other client devices, image processing computing equipment, and/or servers after undergoing entropy coding provided by the encoder.

The region of interest information may be transmitted through a high-level syntax protocol carrying session information. In addition, the region of interest information may be transmitted in packet units such as SEI (Supplement Enhancement Information) of the video standard, video usability information (VUI), and slice header. In addition, the region of interest information may be transmitted as a separate file describing a video file, for example, an MPD of DASH.

Hereinafter, a method of signaling a region of interest in a single screen video is shown.

In the exemplary technology of the present specification, in a single screen image that is not a scalable video, a technique of degrading an image quality by using a method of downscaling (downsampling) an area other than a region of interest (ROI) may be used. have.

The prior art does not share filter information used for downscaling between terminals using a service, and promises with one technique from the beginning or only the encoder knows the filter information.

However, the server device of the present specification uses filter information used at the time of encoding to improve the image quality of a region outside the region of interest downscaled by the client device (or HMD terminal) receiving the encoded image. It can be delivered to a client device. This technology can actually significantly reduce image processing time and provide image quality improvement.

As described above, the server device can generate the region of interest information. For example, the region of interest information may further include filter information as well as tile information. For example, the filter information may include the number of promised filter candidates and values used in the filter.

6 is a diagram showing an exemplary configuration of a client device.

The client device 600 includes an image input unit 610, an audio input unit 620, a sensor unit 630, an image output unit 640, an audio output unit 650, a communication unit 660, and/or a control unit 670 It may include at least one. For example, the client device 600 may be a head-mounted display (HMD). In addition, the control unit 670 of the client device 600 may be included in the client device 600, or may exist as a separate device, such as a smartphone of a VR device.

The image input unit 610 may capture video data. The image input unit 610 may include at least one of a 2D/3D camera and/or an Immersive camera that acquires a user's image. 2D/3D cameras can take images with a viewing angle of 180 degrees or less. The Immersive camera can shoot images with a viewing angle of 360 degrees or less.

The audio input unit 620 may record a user's voice. For example, the audio input unit 620 may include a microphone.

The sensor unit 630 may acquire information about the movement of the user's gaze. For example, the sensor unit 630 may include a gyro sensor that detects a change in the orientation of an object, an acceleration sensor that measures the acceleration or impact intensity of a moving object, and an external sensor that senses a user's gaze direction. . According to an embodiment, the sensor unit 630 may include an image input unit 610 and an audio input unit 620.

The image output unit 640 may output image data received from the communication unit 660 or stored in a memory (not shown).

The audio output unit 650 may output audio data received from the communication unit 660 or stored in a memory.

The communication unit 660 may communicate with an external client device and/or server device through a broadcast network, a wireless communication network, and/or broadband. The communication unit 660 may further include a transmission unit (not shown) for transmitting data and/or a reception unit (not shown) for receiving data.

The control unit 670 may control all operations of the client device 600. The control unit 670 may process video data and signaling data received from the server device. Details of the control unit 670 will be described in detail in FIG. 7 below.

7 is a block diagram showing an exemplary configuration of a control unit of a client device.

The control unit 700 may process signaling data and/or video data. The control unit 700 may include at least one of the signaling data extraction unit 710, the decoder 720, the gaze determination unit 730, and/or the signaling data generation unit 740.

The signaling data extraction unit 710 may extract signaling data from data transmitted from a server device and/or another client device. For example, signaling data may include information of interest.

The decoder 720 may decode video data based on signaling data. For example, the decoder 720 may decode the entire image in a manner customized to each user based on the gaze direction of each user. For example, when a user views a specific area in a virtual reality space, the decoder 720 decodes an image corresponding to a specific area in high quality based on a user's gaze in the virtual reality space, and an image corresponding to a specific area. Can be decoded with low quality. According to an embodiment, the decoder 720 may include at least one of a signaling data extraction unit 710, a gaze determination unit 730, and/or a signaling data generation unit 740.

The gaze determining unit 730 may determine a user's gaze in the virtual reality space and generate image configuration information. For example, the image configuration information may include gaze information indicating a gaze direction and/or zoom area information indicating a user's viewing angle.

The signaling data generation unit 740 may generate signaling data for transmission to a server device and/or another client device. For example, signaling data may transmit image configuration information. In addition, the signaling data can be transmitted through a high-level syntax protocol (SSL) carrying session information, and the signaling data is SEI (Supplement Enhancement Information), VUI (video usability information), slice header (Slice Header), And files describing video data.

8 is a diagram showing an exemplary configuration of a decoder of a client device.

The decoder 800 may include at least one of an extractor 810, a base layer decoder 820, and/or at least one enhancement layer decoder 830.

The decoder 800 may decode a bitstream including video data using an inverse process of the encoding process of the scalable video coding method.

The extractor 810 may receive a bitstream (video data) including video data and signaling data, and selectively extract the bitstream according to an image quality to be reproduced. For example, the bitstream (video data) includes a base layer bitstream (base layer video data) for the base layer and at least one enhancement layer bitstream (enhancement layer video data) for at least one enhancement layer predicted from the base layer. ). The base layer bitstream (base layer video data) may include video data for the entire area of the virtual reality space. The at least one enhancement layer bitstream (enhancement layer video data) may include video data for a region of interest within the entire region.

Also, the signaling data may include interest region information indicating a region of interest corresponding to a user's gaze direction within the entire region of the virtual reality space for the video conference service.

The base layer decoder 820 may decode a bit stream (or base layer video data) of a base layer for a low-quality image.

The enhancement layer decoder 830 may decode at least one enhancement layer bitstream (or enhancement layer video data) for a high-quality image based on signaling data and/or a base layer bitstream (or base layer video data). have.

On the other hand, if the video data included in the bitstream is not encoded by scalable video coding, the decoder 800 may output video data of the basic quality for the entire area of the virtual reality space and the entire area of the virtual reality space from the video stream. High quality video data for a region of interest can be extracted from within.

Hereinafter, a method of generating image configuration information for responding to a user's gaze movement in real time will be described.

The image configuration information may include at least one of gaze information indicating a user's gaze direction and/or zoom area information indicating a user's viewing angle. The user's gaze means the direction the user looks in the virtual reality space, not the real space. In addition, the gaze information may include information indicating a gaze direction of the user in the future, as well as information indicating a gaze direction of the user in the future (eg, information about a gaze point that is expected to receive attention).

The client device may sense and process an operation of looking at a specific area located in the virtual reality space centering on the user.

The client device may receive the sensing information from the sensor unit using the control unit and/or the gaze determination unit. The sensing information may be an image captured by a camera or a voice recorded by a microphone. Further, the sensing information may be data sensed by a gyro sensor, an acceleration sensor, and an external sensor.

In addition, the client device may check the movement of the user's gaze based on the sensing information, using the control unit and/or the gaze determination unit. For example, the client device may check the movement of the user's gaze based on the change in the value of the sensing information.

Also, the client device may generate image configuration information in the virtual reality space using the control unit and/or the gaze determination unit. For example, when the client device physically moves or the user's gaze moves, the client device may calculate the user's gaze information and/or zoom area information in the virtual reality space based on the sensing information.

Also, the client device may transmit the video configuration information to the server device and/or another client device using the communication unit. In addition, the client device may transmit the image configuration information to other components of its own.

In the above, the method in which the client device generates image configuration information has been described. However, the present invention is not limited thereto, and the server device may receive sensing information from the client device and generate image configuration information.

Also, an external computing device connected to the client device may generate image configuration information, and the computing device may deliver the image configuration information to its client device, other client device, and/or server device.

Hereinafter, a method of signaling video configuration information by a client device will be described.

It is very important to signal the video configuration information (including viewpoint information and/or zoom area information). If the signaling of the video configuration information is too frequent, a burden may be placed on the client device, the server device, and/or the entire network.

Accordingly, the client device may signal the image configuration information only when the image configuration information (or gaze information and/or zoom area information) of the user is changed. That is, the client device may transmit the user's gaze information to another client device and/or the server device only when the user's gaze information is changed.

In the above, it has been mainly described that the client device generates and/or transmits image configuration information, but the server device receives the sensing information from the client device, generates image configuration information based on the sensing information, and at least the image configuration information. It can also be transmitted to one client device.

The above-mentioned signaling may be signaling between a server device, a client device, and/or an external computing device (if present). In addition, the above-mentioned signaling may be signaling between a client device and/or an external computing device (if present).

Hereinafter, an exemplary method of transmitting a high/low level image will be described.

The method of transmitting high/low level video based on the user's gaze information is the method of switching the layer of the scalable codec, the rate control method using QP (Quantization Parameter) for single bitstream and real-time encoding, DASH, etc. Includes a method of switching in chunks for a single bitstream, a downscaling/upscaling method, and/or a high definition rendering method using more resources in case of rendering can do.

Although the above-described exemplary technique refers to a differential transmission technique through scalable video, even when a general video coding technique having a single layer is used, by adjusting a quantization parameter or a degree of downscaling/upscaling, It can provide advantages such as lowering the overall bandwidth and rapidly responding to user gaze movements. In addition, when files transcoded into a bitstream having a plurality of bitrates in advance are used, the exemplary technique of the present specification switches between a high-level image and a low-level image in chunk units. Can provide.

In addition, although this specification uses a virtual reality system as an example, this specification may be equally applied to a VR (Virtual Reality) game using an HMD or an Augmented Reality (AR) game. That is, all of the techniques for providing a region corresponding to the gaze viewed by the user as a high level image and signaling only when looking at a region other than an object or an object expected to be viewed by the user are all of the virtual reality system. It can be applied as in the example.

The technology of receiving an entire image as a bitstream of a compressed image, decoding it, and rendering the area viewed by the user in a virtual space is a whole image (for example, a 360 degree immersive ( Immersive) video) are all transmitted as a bitstream. Scalable high-efficiency video, which is a scalable extension standard of SVC and HEVC among international video standard technologies, to prevent the total bandwidth of the bitstream from becoming very large, since the total bandwidth of this video bitstream where each high-resolution video is collected is very large. Scalable video technologies such as scalable high efficiency video coding may be used.

Hereinafter, an example of an indexing method for tiles corresponding to a user's region of interest according to the projection type will be described with reference to FIGS. 9 to 11.

9 is a diagram illustrating a tile indexing method of a user's region of interest according to projection information.

Referring to FIG. 9, when a user wearing a client device 910 (a head-mounted video display device) views a specific direction in a virtual reality space, a user interest area in the client device 910 is based on a client device. It can be divided into a user interest region 911 and a user gaze-based user interest region 913.

The client device 910 may calculate the sphere coordinates of the region of interest selected by the user, and transmit the calculated information about the sphere coordinates to the server device (not shown) together with the type information about the region of interest.

For example, when the gaze-based region of interest 913 is selected as the user's region of interest, the spherical coordinates of the region of interest 920 are horizontally passing through the yaw and pitch coordinates of the central point 929 and the central point 929 ( yaw), vertical (pitch) length. Here, the center point 929 is four center points of the four sides of the region of interest 920 (921, 923, 925, 927), for example, cAzimuth1 (921), cAzimuth2 (923), cElevation1 (925) ), cElevation2 (927) is connected to the two points by Yaw and Pitch horizontally and vertically to determine the intersection point (929).

When the server device receives the sphere coordinates for the user's region of interest and type information about the region of interest (the region of interest based on the user's gaze in the drawing) from the client device 910, the projection type is used using the received sphere coordinates (in FIG. 9). The type of projection can be mapped to a two-dimensional plane of user interest according to equirectangular projection. Since the mapped region of interest 931 is a rectangular 2D image, the coordinates of the region of interest may be represented by x,y coordinates of the upper left corner of the rectangle and x,y coordinates of the lower right corner of the rectangle.

The server device obtains the tile index corresponding to the region of interest 931 mapped to the projection from the image 930 for the virtual reality space divided by the tile, and uses the obtained tile index information to obtain metadata about the video data to be transmitted. It can be generated and included in the signaling data to be transmitted to the client device. In the example, the number of tiles including the mapped region of interest is Tile1, Tile2, Tile4, and Tile5.

10 is a block diagram of an exemplary video transmission device for a video streaming service, and FIG. 11 is a flowchart of an exemplary video transmission method for a video streaming service.

10 to 11, the video transmission apparatus (server device) 1000 includes a communication unit 1010, a control unit 1020, a signaling data generation unit 1030, an encoder 1040, and/or a multiplexer 1050 It may include.

The communication unit 1010 is the first signaling data including the sphere coordinates of the user's region of interest and the region of interest type information based on the type of the region of interest in a 360-degree virtual reality space from a client device (not shown). It may be received (S1101).

The region of interest type information may include information indicating either a viewport based on a client device (eg, a head-mounted video display device) or a viewport based on a user's viewpoint.

The control unit 1020 is based on the spherical coordinates of the user's region of interest received through the communication unit 1010, type (type) information for the user's region of interest and/or projection information for projecting a 360-degree image into a two-dimensional plane. The user's region of interest may be mapped to the 2D plane (S1103).

In order to encode a video for a 360-degree virtual reality space and transmit it to a client device, since the encoding apparatus can encode only a rectangular video, a process of projecting a spherical video image into a two-dimensional plane is essential. .

Accordingly, the control unit 1020 may map the user's region of interest to the two-dimensional plane based on the projection information indicating the projection type.

In addition, the controller 1020 may calculate and generate indexing information of a tile for a region of interest corresponding to the region of interest mapped to the 2D plane. That is, the controller 1020 may generate identification information of tiles including the mapped region of interest (S1105).

The signaling data generation unit 1030 may generate second signaling data based on at least part of indexing information of a region of interest tile corresponding to the mapped region of interest in the virtual reality space.

The second signaling data may include projection information for a 360-degree virtual reality space and mapped region of interest information based on the projection information. The mapped region of interest information may be identification information of tiles including the mapped region of interest.

The encoder 1040 may generate video data for an image streaming service for a 360-degree virtual reality space. The video data may encode basic quality video data for at least a portion of the 360-degree virtual reality space and high-definition video data related to an area corresponding to a user's region of interest in the entire virtual reality space.

The region corresponding to the user's region of interest may be divided into at least one tile, so that the high-definition video data may be a high-definition image of the at least one tile.

Video data for a 360-degree virtual reality space may include basic definition video data and high-definition video data, and the basic-definition video data includes base layer video data for the 360-degree virtual reality space, and the high-definition video data May be encoded to include base layer video data for a region of interest tile and at least one enhancement layer video data.

In addition, the encoder 1040 may include a base layer encoder and an enhancement layer encoder, where the base quality video data may be base layer video data for an entire 360 degree virtual reality space, and the high definition video data may be 360. Also, it may be base layer video data and at least one or more enhancement layer video data for tiles included in a viewport or a view of a client device viewed by a user in a virtual reality space.

The base layer can be encoded entirely without tiling for fast user response times. One or more enhancement layers may be coded by dividing some or all of the tiles into multiple tiles as necessary.

The multiplexer 1050 may generate a bitstream including video data for a 360 degree virtual reality space and second signaling data for the video data.

Also, the communication unit 1010 may transmit a bitstream including the video data generated by the multiplexer 1050 and the second signaling data to the client device according to the command of the controller 1020 (S1107).

Here, since the tile numbers in the viewport or the region of interest are continuously assigned, when transmitting the tile information included in the viewport to the first signaling data and the second signaling data, the client device and the server device send all the number information for the tiles. It can be compressed effectively without sending it all out. For example, the devices can effectively compress tile number information by using tile start and end numbers, tile coordinate point information, a list of coding unit (CU) numbers in a tile, and a method of expressing a tile number as a formula. have.

The first signaling data and the second signaling data are generated based on the image configuration information for the 360-degree virtual reality space, and the image configuration information includes the gaze information indicating the viewports and the user's viewing angle in the 360-degree virtual reality space. It may include indicating zoom area information.

The signaling data may be transmitted through a high-level syntax protocol carrying session information, or may be transmitted by packet units such as a video standard SEI, VUI, or slice header, or a video file. It may be included in a separate file to be described (eg, DASH MPD) and delivered.

Equi-Rectangular Projection (ERP), Cube Map Projection (CMP), Adjusted Equal-Area Projection (AEP), Octahedron Projection (OHP), Icosahedron Projection (ISP) , Truncated Square Pyramid (TSP), Segmented Sphere Projection (SSP), Adjusted Cube Map Projection (ACP), Rotated Sphere Projection (RSP), Equatorial Cylindrical Projection (ECP) and Equi-Angular Cube Map (EAC).

The video of a specific area of the 360-degree virtual reality space has a different position and shape depending on the projection type due to various projection methods. Therefore, after mapping the region of interest according to the projection type, and encoding the projected image together with the information about the mapped region of interest and the information about the projection type, and transmitting the projected image to the image reproducing apparatus, the image reproducing apparatus is accurately located in a 360 degree virtual space. It is possible to decode the video, and the high-definition video data can be transmitted only to the tile corresponding to the region of interest through mapping information on the region of interest, thereby securing a transmission bandwidth and increasing the playback speed. .

In addition, projection coordinates and tile index information, which can be obtained through sphere coordinates for a user's region of interest, can be calculated on both sides according to situations such as the performance of each device and the state of the network between the video transmission device and the video playback device.

Hereinafter, a method of reproducing an image received from an image reproducing apparatus using an indexing method for a tile corresponding to a user's region of interest will be described with reference to FIG. 12.

12 is a flowchart of an exemplary video playback method for a video streaming service.

Referring to FIG. 12, an exemplary image playback method may be performed by a processor included in an image playback device through the following process.

First, the processor of the video reproducing apparatus may transmit first signaling data including sphere coordinates for a user's region of interest and type of region of interest based on the type of region of interest in a 360 degree virtual reality space ( S1201).

Next, the processor of the video reproducing apparatus may receive the second signaling data and the video data for the 360-degree virtual reality space from the video transmitting apparatus (S1203).

The second signaling data is generated by the video transmission device, and within the 360-degree virtual reality space, the spherical coordinates of the user's region of interest, type information of the region of interest, projection information performed by the video transmission device, and the projection information. It may include indexing information of a region of interest tile corresponding to a region of interest in the 360-degree virtual reality space mapped on a 2D plane based on the basis.

Next, the processor of the image reproducing apparatus may reproduce the received image based on the video data and the second signaling data for the 360-degree virtual reality space (S1205).

The video reproducing apparatus decodes video data for a user viewport including the region of interest based on the received video data for the 360-degree virtual reality space and the received second signaling data, and renders the decoded video, thereby It can be output to a display device.

Further, the video data for the 360-degree virtual reality space may include at least a portion of the basic-definition video data for at least a portion of the 360-degree virtual reality space and the high-definition video data for the tile of interest.

Hereinafter, signaling (signal system) in tile indexing of a user's region of interest according to projection will be described with reference to FIGS. 13 to 14.

13 is a diagram illustrating an exemplary OMAF syntax in the international video standard for signaling in tile indexing of a user's region of interest according to projection.

Referring to FIG. 13, an exemplary OMAF (Omnidirectional Media Application Format) syntax shows syntax used in international video standards such as H.264 AVC or H.265 HEVC.

The syntaxes of reference numerals 1301, 1303, and 1305 in the drawings are contents to be newly added as an embodiment of the present specification, and all other syntaxes are existing standard syntaxes. Hereinafter, each syntax will be described in detail.

viewport_id means the kind of user interest area. For example, if the value is "0", the user interest area is a head-mounted video display device-based interest area, and if it is "1", the user interest area is a user gaze-based interest. This indicates that it is an area.

The center_azimuth syntax indicates the yaw value of the center point of the user's region of interest based on spherical coordinates.

The center_elevation syntax indicates the pitch value of the center point of the user's region of interest based on spherical coordinates.

The azimuth_range syntax represents the yaw horizontal range value passing through the center point of the user's region of interest based on the spherical coordinates.

The elevation_range syntax represents the pitch vertical range value passing through the center point of the user's region of interest based on the spherical coordinates.

The projection_id syntax means a 2D projection type. For example, if the value is "0", 2D projection is ERP (Equi-Rectangular Projection), and if "1", 2D projection is CMP (Cube Map Projection). .

The left_top_x syntax represents the x-coordinate value of the upper left corner of the region of interest based on 2D projection.

The left_top_y syntax indicates the y coordinate value of the upper left corner of the region of interest based on the 2D projection.

The right_bottom_x syntax represents the x-coordinate value of the lower right corner of the region of interest based on 2D projection.

The right_bottom_y syntax indicates the y-coordinate value of the lower right corner of the region of interest based on the 2D projection.

The tile_id[] syntax represents index array information of a tile corresponding to the projected region of interest.

Accordingly, by including the user's region of interest type information and the tile information of the region of interest according to the projection method in signaling data through the OMAF syntax presented herein, transmission and reception between the client device and the server device is reduced, thereby reducing the transmission bandwidth and the client. It is possible to achieve an improvement in playback speed on the device.

Information related to syntax and semantics defined in the detailed description of FIG. 13 may be respectively expressed in the form of XML in HTTP-based video communication such as MPEG DASH.

14 is a diagram illustrating an example of a tile indexing information syntax of a region of interest expressed in the form of XML, and illustrated XML is a type of a user region of interest, a sphere coordinate of a region of interest, a type of projection, a projection coordinate of a region of interest, and a region of interest. It may include index arrangement information of a corresponding tile.

Referring to FIG. 14, in the example expressed in the form of XML, the type of the user's region of interest is the region of interest based on the user's gaze, the spherical coordinate of the region of interest is 45 horizontally, 127 vertically, and the type of projection is ERP. The projection coordinates of the area are the coordinates of the upper left corner (312, 523), the coordinates of the lower right corner are (408, 845), and the index array information of the tile corresponding to the region of interest is tile numbers 1, 2, 4, 5 It may include information indicating that.

The virtual reality system according to the embodiments disclosed herein may be embodied as computer readable code in a computer readable recording medium. The computer-readable recording medium includes all kinds of recording devices in which data that can be read by a computer system are stored. Examples of computer-readable recording media include ROM, RAM, CD-ROM, magnetic tape, floppy disks, and optical data storage devices. In addition, the computer-readable recording medium can be distributed over network coupled computer systems so that the computer readable code is stored and executed in a distributed fashion. And, functional programs, codes, and code segments for implementing the present invention can be easily inferred by programmers in the technical field to which the technology of this specification belongs.

In the above, a preferred embodiment of the technology of the present specification has been described with reference to the accompanying drawings. Here, the terms or words used in the specification and claims should not be interpreted as being limited to ordinary or dictionary meanings, but should be interpreted as meanings and concepts corresponding to the technical spirit of the present invention.

The scope of the present invention is not limited to the embodiments disclosed herein, and the present invention can be modified, changed, or improved in various forms within the scope described in the spirit and claims of the present invention.

1000: server device
1010: Communication Department
1020: control unit
1030: signaling data generation unit
1040: encoder
1050: multiplexer

Claims (17)

As a method performed in a video transmission apparatus including a processor,
Receiving first signaling data including sphere coordinates for a user's region of interest and type of region of interest based on a type of the region of interest in a 360-degree virtual reality space;
Mapping the region of interest to a two-dimensional plane based on the spherical coordinates and preset projection information;
Calculating indexing information of a region of interest tile corresponding to the mapped region of interest; And
And transmitting video data and second signaling data for a 360-degree virtual reality space based on the indexing information. According to claim 1,
The second signaling data is a video transmission method of the video transmission apparatus including the projection information and mapped region of interest information based on the projection information. According to claim 1,
The region-of-interest type information is information indicating either a viewport based on a head-mounted video display device or a viewport based on a user's viewpoint, and the video transmission method of the video transmission device. According to claim 1,
The method further includes preparing basic quality video data for at least a portion of the 360-degree virtual reality space and high-definition video data related to the region of interest tile,
The video data of the 360-degree virtual reality space includes the basic quality video data and the high-definition video data. According to claim 4,
The base quality video data includes base layer video data for the 360-degree virtual reality space,
The high-definition video data includes a base layer video data and an enhancement layer video data for the region of interest tile. According to claim 1,
The spherical coordinates are the yaw coordinates and pitch coordinates of the second central point where the segments connected by the first central points of the two sides opposite to the central points of the four sides of the region of interest and the horizontal point based on the second central point. Video transmission method of the video transmission device represented by length information and vertical length information. According to claim 3,
The first signaling data and the second signaling data are generated based on image configuration information,
The image configuration information includes a gaze information indicating the viewports in the 360-degree virtual reality space and zoom area information indicating the user's viewing angle. The method of claim 7,
The first signaling data and the second signaling data are high-level syntax protocol (SEI) carrying session information, supplemental enhancement information (SEI), video usability information (VUI), slice header, and the A video transmission method of a video transmission device that is transmitted through at least one of files describing video data. According to claim 1,
Equi-Rectangular Projection (ERP), Cube Map Projection (CMP), Adjusted Equal-Area Projection (AEP), Octahedron Projection (OHP), Icosahedron Projection (ISP), Truncated Square Pyramid (TSP), Segmented Sphere Video transmission method of an image transmission device which is one of Projection (SSP), Adjusted Cube Map Projection (ACP), Rotated Sphere Projection (RSP), Equatorial Cylindrical Projection (ECP) and Equi-Angular Cube Map (EAC). A communication unit for receiving first signaling data including sphere coordinates for a user's region of interest and type of region of interest based on a type of the region of interest within a 360-degree virtual reality space, and transmitting the generated bitstream;
A controller for mapping the region of interest to a two-dimensional plane based on the spherical coordinates and preset projection information;
A signaling data generation unit generating second signaling data including indexing information of a region of interest tile corresponding to the mapped region of interest; And
And a multiplexer generating a bitstream including the video data and the second signaling data for the 360 degree virtual reality space. The method of claim 10,
And an encoder for generating basic quality video data for at least a portion of the 360 degree virtual reality space and high quality video data for the region of interest tile. The method of claim 11,
The base quality video data includes base layer video data for at least a portion of the 360-degree virtual reality space,
The high definition video data includes a base layer video data and enhancement layer video data for the region of interest tile. The method of claim 10,
The region of interest type information is information indicating either a viewport based on a head-mounted image display device or a viewport based on a user's viewpoint. The method of claim 10,
The spherical coordinates are the yaw coordinates and pitch coordinates of the second central point where the segments connected by the first central points of the two sides opposite to the central points of the four sides of the region of interest and the horizontal point based on the second central point. An image transmission device represented by length information and vertical length information. As a method performed in a video playback device including a processor,
Transmitting first signaling data including sphere coordinates for a user's region of interest and type of region of interest based on a type of the region of interest in a 360-degree virtual reality space;
Receiving second signaling data and video data for the 360-degree virtual reality space; And
And reproducing an image based on the video data for the 360-degree virtual reality space and the second signaling data,
The second signaling data includes indexing information of a region of interest tile corresponding to the region of interest mapped to a two-dimensional plane based on the sphere coordinates, the region of interest type information, and preset projection information,
The video data for the 360-degree virtual reality space includes a basic-definition video data for at least a part of the 360-degree virtual reality space and high-definition video data for the region of interest tile. The method of claim 15,
Reproducing an image based on the video data and the second signaling data for the 360-degree virtual reality space,
A video reproducing method of an image reproducing apparatus that renders and displays an image for a user viewport including the region of interest based on the video data for the 360-degree virtual reality space and the second signaling data. On a computing device,
Receiving first signaling data including sphere coordinates for a user's region of interest and type of region of interest based on a type of the region of interest in a 360-degree virtual reality space;
Mapping the region of interest to a two-dimensional plane based on the spherical coordinates and preset projection information;
Calculating indexing information of a region of interest tile corresponding to the mapped region of interest; And
A computer program stored in a medium to execute an operation of transmitting video data and second signaling data for a 360-degree virtual reality space based on the indexing information.

Images