KR20180107007A - Method and apparatus for processing a video signal - Google Patents

Abstract

The present invention relates to a method and an apparatus for processing a video signal. An image decoding method according to the present invention can comprise the steps of: decoding viewport related information from a bit stream; restoring a 360-degree video; and setting a viewport on the restored 360-degree video based on the viewport-related information, wherein the view port-related information can include information of the number of viewports. Therefore, the efficiency of encoding/decoding of the 360-degree video is improved.

Description

TECHNICAL FIELD [0001] The present invention relates to a video signal processing method and apparatus,

The present invention relates to a video signal processing method and apparatus.

Recently, the demand for high resolution and high quality images such as high definition (HD) image and ultra high definition (UHD) image is increasing in various applications. As the image data has high resolution and high quality, the amount of data increases relative to the existing image data. Therefore, when the image data is transmitted using a medium such as a wired / wireless broadband line or stored using an existing storage medium, The storage cost is increased. High-efficiency image compression techniques can be utilized to solve such problems as image data becomes high-resolution and high-quality.

An inter picture prediction technique for predicting a pixel value included in a current picture from a previous or a subsequent picture of a current picture by an image compression technique, an intra picture prediction technique for predicting a pixel value included in a current picture using pixel information in the current picture, There are various techniques such as an entropy encoding technique in which a short code is assigned to a value having a high appearance frequency and a long code is assigned to a value having a low appearance frequency. Image data can be effectively compressed and transmitted or stored using such an image compression technique.

On the other hand, demand for high-resolution images is increasing, and demand for stereoscopic image content as a new image service is also increasing. Video compression techniques are being discussed to effectively provide high resolution and ultra-high resolution stereoscopic content.

An object of the present invention is to improve coding / decoding efficiency of 360-degree video by encoding / decoding view port information.

It is an object of the present invention to allow a plurality of view ports to be set in 360-degree video.

An object of the present invention is to improve coding / decoding efficiency of 360-degree video by encoding / decoding rotation information of a view port.

It is to be understood that both the foregoing general description and the following detailed description are exemplary and explanatory and are not restrictive of the invention, unless further departing from the spirit and scope of the invention as defined by the appended claims. It will be possible.

The video decoding method and apparatus according to the present invention decodes viewport-related information from a bitstream, restores 360-degree video, and sets a viewport on the restored 360-degree video based on the viewport-related information have.

The image encoding method and apparatus according to the present invention can set a view port to 360-degree video and encode the 360-degree video and the view port related information.

In the image decoding / encoding method and apparatus according to the present invention, the view port related information may include the view port number information.

In the video decoding / encoding method and apparatus according to the present invention, when the number information of the view port indicates that a plurality of view ports exist, the view port related information includes position information and range information of each of the plurality of view ports .

In the image decoding / encoding method and apparatus according to the present invention, the position information of the first viewport among the plurality of viewports represents a difference value from the position information of the reference viewport, and the range information of the first viewport is And the difference value with the range information of the reference view port.

In the video decoding / encoding method and apparatus according to the present invention, the reference view port may have a specific view port index among the plurality of viewports.

In the video decoding / coding method and apparatus according to the present invention, the reference view port may be a previous viewport of the first viewport.

In the image decoding / encoding method and apparatus according to the present invention, the view port related information may further include rotation information of a view port.

In the image decoding / encoding method and apparatus according to the present invention, the rotation information of the view port may include at least one of a rotation angle in the Yaw direction, a rotation angle in the Pitch direction, or a rotation angle in the Roll direction .

The features briefly summarized above for the present invention are only illustrative aspects of the detailed description of the invention which are described below and do not limit the scope of the invention.

According to the present invention, there is an advantage that coding / decoding efficiency of 360-degree video can be improved by encoding / decoding view port information.

According to the present invention, a plurality of view ports can be set in 360-degree video, which is advantageous in that it can provide user convenience during 360-degree video play.

According to the present invention, there is an advantage that coding / decoding efficiency of 360-degree video can be improved by encoding / decoding rotation information of the view port.

The effects obtained by the present invention are not limited to the above-mentioned effects, and other effects not mentioned can be clearly understood by those skilled in the art from the following description will be.

1 is a block diagram illustrating an image encoding apparatus according to an embodiment of the present invention.
2 is a block diagram illustrating an image decoding apparatus according to an embodiment of the present invention.
3 is a diagram illustrating a partition mode that can be applied to a coding block when a coding block is coded by inter-picture prediction.
4 to 6 are views illustrating a camera apparatus for generating a panoramic image.
7 is a block diagram of a 360-degree video data generation apparatus and a 360-degree video play apparatus.
8 is a flowchart showing the operation of a 360-degree video data generation apparatus and a 360-degree video play apparatus.
Figure 9 shows a 2D projection method using the isometric quadrature method.
10 shows a 2D projection method using a cube projection method.
11 shows a 2D projection method using a bipartite projection technique.
12 shows a 2D projection method using an octahedral projection technique.
13 shows a 2D projection method using a cutting pyramid projection technique.
14 is a diagram illustrating the conversion between face 2D coordinates and three-dimensional coordinates.
15 is a view for explaining the view port mode.
16 is a view showing an example in which the position and size of the view port are specified.
17 is a diagram showing a syntax related to a view port encoded in a bitstream.
18 is a diagram showing an example in which a syntax indicating the number of viewports is coded.
19 is a view showing an example of deriving current viewport information using a reference viewport.
20 is a diagram showing an example in which information related to rotation of a sphere is encoded.

While the invention is susceptible to various modifications and alternative forms, specific embodiments thereof are shown by way of example in the drawings and will herein be described in detail. It should be understood, however, that the invention is not intended to be limited to the particular embodiments, but includes all modifications, equivalents, and alternatives falling within the spirit and scope of the invention. Like reference numerals are used for like elements in describing each drawing.

The terms first, second, etc. may be used to describe various components, but the components should not be limited by the terms. The terms are used only for the purpose of distinguishing one component from another. For example, without departing from the scope of the present invention, the first component may be referred to as a second component, and similarly, the second component may also be referred to as a first component. And / or < / RTI > includes any combination of a plurality of related listed items or any of a plurality of related listed items.

It is to be understood that when an element is referred to as being "connected" or "connected" to another element, it may be directly connected or connected to the other element, . On the other hand, when an element is referred to as being "directly connected" or "directly connected" to another element, it should be understood that there are no other elements in between.

The terminology used in this application is used only to describe a specific embodiment and is not intended to limit the invention. The singular expressions include plural expressions unless the context clearly dictates otherwise. In the present application, the terms "comprises" or "having" and the like are used to specify that there is a feature, a number, a step, an operation, an element, a component or a combination thereof described in the specification, But do not preclude the presence or addition of one or more other features, integers, steps, operations, elements, components, or combinations thereof.

Hereinafter, preferred embodiments of the present invention will be described in detail with reference to the accompanying drawings. Hereinafter, the same reference numerals will be used for the same constituent elements in the drawings, and redundant explanations for the same constituent elements will be omitted.

1 is a block diagram illustrating an image encoding apparatus according to an embodiment of the present invention.

1, the image encoding apparatus 100 includes a picture division unit 110, prediction units 120 and 125, a transform unit 130, a quantization unit 135, a reordering unit 160, an entropy encoding unit An inverse quantization unit 140, an inverse transform unit 145, a filter unit 150, and a memory 155. [

Each of the components shown in FIG. 1 is shown independently to represent different characteristic functions in the image encoding apparatus, and does not mean that each component is composed of separate hardware or one software configuration unit. That is, each constituent unit is included in each constituent unit for convenience of explanation, and at least two constituent units of the constituent units may be combined to form one constituent unit, or one constituent unit may be divided into a plurality of constituent units to perform a function. The integrated embodiments and separate embodiments of the components are also included within the scope of the present invention, unless they depart from the essence of the present invention.

In addition, some of the components are not essential components to perform essential functions in the present invention, but may be optional components only to improve performance. The present invention can be implemented only with components essential for realizing the essence of the present invention, except for the components used for the performance improvement, and can be implemented by only including the essential components except the optional components used for performance improvement Are also included in the scope of the present invention.

The picture division unit 110 may divide the input picture into at least one processing unit. At this time, the processing unit may be a prediction unit (PU), a transform unit (TU), or a coding unit (CU). The picture division unit 110 divides one picture into a plurality of coding units, a prediction unit, and a combination of conversion units, and generates a coding unit, a prediction unit, and a conversion unit combination So that the picture can be encoded.

For example, one picture may be divided into a plurality of coding units. In order to divide a coding unit in a picture, a recursive tree structure such as a quad tree structure can be used. In a coding or decoding scheme in which one picture or a largest coding unit is used as a root and divided into other coding units A unit can be divided with as many child nodes as the number of divided coding units. Under certain constraints, an encoding unit that is no longer segmented becomes a leaf node. That is, when it is assumed that only one square division is possible for one coding unit, one coding unit can be divided into a maximum of four different coding units.

Hereinafter, in the embodiment of the present invention, a coding unit may be used as a unit for performing coding, or may be used as a unit for performing decoding.

The prediction unit may be one divided into at least one square or rectangular shape having the same size in one coding unit, and one of the prediction units in one coding unit may be divided into another prediction Or may have a shape and / or size different from the unit.

If a prediction unit performing intra prediction on the basis of an encoding unit is not the minimum encoding unit at the time of generation, intraprediction can be performed without dividing the prediction unit into a plurality of prediction units NxN.

The prediction units 120 and 125 may include an inter prediction unit 120 for performing inter prediction and an intra prediction unit 125 for performing intra prediction. It is possible to determine whether to use inter prediction or intra prediction for a prediction unit and to determine concrete information (e.g., intra prediction mode, motion vector, reference picture, etc.) according to each prediction method. At this time, the processing unit in which the prediction is performed may be different from the processing unit in which the prediction method and the concrete contents are determined. For example, the method of prediction, the prediction mode and the like are determined as a prediction unit, and the execution of the prediction may be performed in a conversion unit. The residual value (residual block) between the generated prediction block and the original block can be input to the conversion unit 130. [ In addition, the prediction mode information, motion vector information, and the like used for prediction can be encoded by the entropy encoding unit 165 together with the residual value and transmitted to the decoder. When a particular encoding mode is used, it is also possible to directly encode the original block and transmit it to the decoding unit without generating a prediction block through the prediction units 120 and 125.

The inter-prediction unit 120 may predict a prediction unit based on information of at least one of a previous picture or a following picture of the current picture, and may predict a prediction unit based on information of a partially- Unit may be predicted. The inter prediction unit 120 may include a reference picture interpolation unit, a motion prediction unit, and a motion compensation unit.

In the reference picture interpolating section, the reference picture information is supplied from the memory 155 and pixel information of an integer pixel or less can be generated in the reference picture. In the case of a luminance pixel, a DCT-based interpolation filter having a different filter coefficient may be used to generate pixel information of an integer number of pixels or less in units of quarter pixels. In the case of a color difference signal, a DCT-based 4-tap interpolation filter having a different filter coefficient may be used to generate pixel information of an integer number of pixels or less in units of 1/8 pixel.

The motion prediction unit may perform motion prediction based on the reference picture interpolated by the reference picture interpolating unit. Various methods such as Full Search-based Block Matching Algorithm (FBMA), Three Step Search (TSS), and New Three-Step Search Algorithm (NTS) can be used as methods for calculating motion vectors. The motion vector may have a motion vector value of 1/2 or 1/4 pixel unit based on the interpolated pixel. The motion prediction unit can predict the current prediction unit by making the motion prediction method different. Various methods such as a skip method, a merge method, an AMVP (Advanced Motion Vector Prediction) method, and an Intra Block Copy method can be used as the motion prediction method.

The intra prediction unit 125 can generate a prediction unit based on reference pixel information around the current block which is pixel information in the current picture. In the case where the neighboring block of the current prediction unit is the block in which the inter prediction is performed so that the reference pixel is the pixel performing the inter prediction, the reference pixel included in the block in which the inter prediction is performed is referred to as the reference pixel Information. That is, when the reference pixel is not available, the reference pixel information that is not available may be replaced by at least one reference pixel among the available reference pixels.

In intra prediction, the prediction mode may have a directional prediction mode in which reference pixel information is used according to a prediction direction, and a non-directional mode in which direction information is not used in prediction. The mode for predicting the luminance information may be different from the mode for predicting the chrominance information and the intra prediction mode information or predicted luminance signal information used for predicting the luminance information may be utilized to predict the chrominance information.

When intraprediction is performed, when the size of the prediction unit is the same as the size of the conversion unit, intra prediction is performed on the prediction unit based on pixels existing on the left side of the prediction unit, pixels existing on the upper left side, Can be performed. However, when intra prediction is performed, when the size of the prediction unit differs from the size of the conversion unit, intraprediction can be performed using the reference pixel based on the conversion unit. It is also possible to use intraprediction using NxN partitioning only for the minimum encoding unit.

The intra prediction method can generate a prediction block after applying an AIS (Adaptive Intra Smoothing) filter to the reference pixel according to the prediction mode. The type of the AIS filter applied to the reference pixel may be different. In order to perform the intra prediction method, the intra prediction mode of the current prediction unit can be predicted from the intra prediction mode of the prediction unit existing around the current prediction unit. In the case where the prediction mode of the current prediction unit is predicted using the mode information predicted from the peripheral prediction unit, if the intra prediction mode of the current prediction unit is the same as the intra prediction mode of the current prediction unit, The prediction mode information of the current block can be encoded by performing entropy encoding if the prediction mode of the current prediction unit is different from the prediction mode of the neighbor prediction unit.

In addition, a residual block including a prediction unit that has been predicted based on the prediction unit generated by the prediction units 120 and 125 and a residual value that is a difference value from the original block of the prediction unit may be generated. The generated residual block may be input to the transform unit 130. [

The transform unit 130 transforms the residual block including the residual information of the prediction unit generated through the original block and the predictors 120 and 125 into a DCT (Discrete Cosine Transform), a DST (Discrete Sine Transform), a KLT You can convert using the same conversion method. The decision to apply the DCT, DST, or KLT to transform the residual block may be based on the intra prediction mode information of the prediction unit used to generate the residual block.

The quantization unit 135 may quantize the values converted into the frequency domain by the conversion unit 130. [ The quantization factor may vary depending on the block or the importance of the image. The values calculated by the quantization unit 135 may be provided to the inverse quantization unit 140 and the reorder unit 160.

The reordering unit 160 can reorder the coefficient values with respect to the quantized residual values.

The reordering unit 160 may change the two-dimensional block type coefficient to a one-dimensional vector form through a coefficient scanning method. For example, the rearranging unit 160 may scan a DC coefficient to a coefficient in a high frequency region using a Zig-Zag scan method, and change the DC coefficient to a one-dimensional vector form. Instead of the jig-jag scan, a vertical scan may be used to scan two-dimensional block type coefficients in a column direction, and a horizontal scan to scan a two-dimensional block type coefficient in a row direction depending on the size of the conversion unit and the intra prediction mode. That is, it is possible to determine whether any scanning method among the jig-jag scan, the vertical direction scan and the horizontal direction scan is used according to the size of the conversion unit and the intra prediction mode.

The entropy encoding unit 165 may perform entropy encoding based on the values calculated by the reordering unit 160. For entropy encoding, various encoding methods such as Exponential Golomb, Context-Adaptive Variable Length Coding (CAVLC), and Context-Adaptive Binary Arithmetic Coding (CABAC) may be used.

The entropy encoding unit 165 receives the residual value count information of the encoding unit, the block type information, the prediction mode information, the division unit information, the prediction unit information and the transmission unit information, and the motion information of the motion unit from the reordering unit 160 and the prediction units 120 and 125 Vector information, reference frame information, interpolation information of a block, filtering information, and the like.

The entropy encoding unit 165 can entropy-encode the coefficient value of the encoding unit input by the reordering unit 160. [

The inverse quantization unit 140 and the inverse transformation unit 145 inverse quantize the quantized values in the quantization unit 135 and inversely transform the converted values in the conversion unit 130. [ The residual value generated by the inverse quantization unit 140 and the inverse transform unit 145 is combined with the prediction unit predicted through the motion estimation unit, the motion compensation unit and the intra prediction unit included in the prediction units 120 and 125, A block (Reconstructed Block) can be generated.

The filter unit 150 may include at least one of a deblocking filter, an offset correction unit, and an adaptive loop filter (ALF).

The deblocking filter can remove block distortion caused by the boundary between the blocks in the reconstructed picture. It may be determined whether to apply a deblocking filter to the current block based on pixels included in a few columns or rows included in the block to determine whether to perform deblocking. When a deblocking filter is applied to a block, a strong filter or a weak filter may be applied according to the deblocking filtering strength required. In applying the deblocking filter, horizontal filtering and vertical filtering may be performed concurrently in performing vertical filtering and horizontal filtering.

The offset correction unit may correct the offset of the deblocked image with respect to the original image in units of pixels. In order to perform offset correction for a specific picture, pixels included in an image are divided into a predetermined number of areas, and then an area to be offset is determined and an offset is applied to the area. Alternatively, Can be used.

Adaptive Loop Filtering (ALF) can be performed based on a comparison between the filtered reconstructed image and the original image. After dividing the pixels included in the image into a predetermined group, one filter to be applied to the group may be determined and different filtering may be performed for each group. The information related to whether to apply the ALF may be transmitted for each coding unit (CU), and the shape and the filter coefficient of the ALF filter to be applied may be changed according to each block. Also, an ALF filter of the same type (fixed form) may be applied irrespective of the characteristics of the application target block.

The memory 155 may store the reconstructed block or picture calculated through the filter unit 150 and the reconstructed block or picture stored therein may be provided to the predictor 120 or 125 when the inter prediction is performed.

2 is a block diagram illustrating an image decoding apparatus according to an embodiment of the present invention.

2, the image decoder 200 includes an entropy decoding unit 210, a reordering unit 215, an inverse quantization unit 220, an inverse transform unit 225, prediction units 230 and 235, 240, and a memory 245 may be included.

When an image bitstream is input in the image encoder, the input bitstream may be decoded in a procedure opposite to that of the image encoder.

The entropy decoding unit 210 can perform entropy decoding in a procedure opposite to that in which entropy encoding is performed in the entropy encoding unit of the image encoder. For example, various methods such as Exponential Golomb, Context-Adaptive Variable Length Coding (CAVLC), and Context-Adaptive Binary Arithmetic Coding (CABAC) may be applied in accordance with the method performed by the image encoder.

The entropy decoding unit 210 may decode information related to intra prediction and inter prediction performed in the encoder.

The reordering unit 215 can perform reordering based on a method in which the entropy decoding unit 210 rearranges the entropy-decoded bitstreams in the encoding unit. The coefficients represented by the one-dimensional vector form can be rearranged by restoring the coefficients of the two-dimensional block form again. The reordering unit 215 can perform reordering by receiving information related to the coefficient scanning performed by the encoding unit and performing a reverse scanning based on the scanning order performed by the encoding unit.

The inverse quantization unit 220 can perform inverse quantization based on the quantization parameters provided by the encoder and the coefficient values of the re-arranged blocks.

The inverse transform unit 225 may perform an inverse DCT, an inverse DST, and an inverse KLT on the DCT, DST, and KLT transformations performed by the transform unit on the quantization result performed by the image encoder. The inverse transform can be performed based on the transmission unit determined by the image encoder. In the inverse transform unit 225 of the image decoder, a transform technique (e.g., DCT, DST, KLT) may be selectively performed according to a plurality of information such as a prediction method, a size of a current block, and a prediction direction.

The prediction units 230 and 235 can generate a prediction block based on the prediction block generation related information provided by the entropy decoding unit 210 and the previously decoded block or picture information provided in the memory 245. [

As described above, when intra prediction is performed in the same manner as in the image encoder, when the size of the prediction unit is the same as the size of the conversion unit, pixels existing on the left side of the prediction unit, pixels existing on the upper left side, However, when the size of the prediction unit differs from the size of the prediction unit in intra prediction, intraprediction is performed using a reference pixel based on the conversion unit . It is also possible to use intra prediction using NxN division only for the minimum coding unit.

The prediction units 230 and 235 may include a prediction unit determination unit, an inter prediction unit, and an intra prediction unit. The prediction unit determination unit receives various information such as prediction unit information input from the entropy decoding unit 210, prediction mode information of the intra prediction method, motion prediction related information of the inter prediction method, and identifies prediction units in the current coding unit. It is possible to determine whether the unit performs inter prediction or intra prediction. The inter prediction unit 230 predicts the current prediction based on the information included in at least one of the previous picture of the current picture or the following picture including the current prediction unit by using information necessary for inter prediction of the current prediction unit provided by the image encoder, Unit can be performed. Alternatively, the inter prediction may be performed on the basis of the information of the partial region previously reconstructed in the current picture including the current prediction unit.

In order to perform inter prediction, a motion prediction method of a prediction unit included in a corresponding encoding unit on the basis of an encoding unit includes a skip mode, a merge mode, an AMVP mode, and an intra block copy mode It is possible to judge whether or not it is any method.

The intra prediction unit 235 can generate a prediction block based on the pixel information in the current picture. If the prediction unit is a prediction unit that performs intra prediction, the intra prediction can be performed based on the intra prediction mode information of the prediction unit provided by the image encoder. The intraprediction unit 235 may include an AIS (Adaptive Intra Smoothing) filter, a reference pixel interpolator, and a DC filter. The AIS filter performs filtering on the reference pixels of the current block and can determine whether to apply the filter according to the prediction mode of the current prediction unit. The AIS filtering can be performed on the reference pixel of the current block using the prediction mode of the prediction unit provided in the image encoder and the AIS filter information. When the prediction mode of the current block is a mode in which AIS filtering is not performed, the AIS filter may not be applied.

The reference pixel interpolator may interpolate the reference pixels to generate reference pixels in units of pixels less than or equal to an integer value when the prediction mode of the prediction unit is a prediction unit that performs intra prediction based on pixel values obtained by interpolating reference pixels. The reference pixel may not be interpolated in the prediction mode in which the prediction mode of the current prediction unit generates the prediction block without interpolating the reference pixel. The DC filter can generate a prediction block through filtering when the prediction mode of the current block is the DC mode.

The restored block or picture may be provided to the filter unit 240. The filter unit 240 may include a deblocking filter, an offset correction unit, and an ALF.

When information on whether a deblocking filter is applied to a corresponding block or picture from the image encoder or a deblocking filter is applied, information on whether a strong filter or a weak filter is applied can be provided. In the deblocking filter of the video decoder, the deblocking filter related information provided by the video encoder is provided, and the video decoder can perform deblocking filtering for the corresponding block.

The offset correction unit may perform offset correction on the reconstructed image based on the type of offset correction applied to the image and the offset value information during encoding.

The ALF can be applied to an encoding unit on the basis of ALF application information and ALF coefficient information provided from an encoder. Such ALF information may be provided in a specific parameter set.

The memory 245 may store the reconstructed picture or block to be used as a reference picture or a reference block, and may also provide the reconstructed picture to the output unit.

As described above, in the embodiment of the present invention, a coding unit (coding unit) is used as a coding unit for convenience of explanation, but it may be a unit for performing not only coding but also decoding.

The current block indicates a block to be coded / decoded. Depending on the coding / decoding step, the current block includes a coding tree block (or coding tree unit), a coding block (or coding unit), a transform block (Or prediction unit), and the like. In this specification, 'unit' represents a basic unit for performing a specific encoding / decoding process, and 'block' may represent a sample array of a predetermined size. Unless otherwise indicated, the terms 'block' and 'unit' may be used interchangeably. For example, in the embodiments described below, it can be understood that the encoding block (coding block) and the encoding unit (coding unit) have mutually equivalent meanings.

One picture may be divided into a square block or a non-square basic block and then encoded / decoded. At this time, the basic block may be referred to as a coding tree unit. The coding tree unit may be defined as a coding unit of the largest size allowed in a sequence or a slice. Information regarding whether the coding tree unit is square or non-square or about the size of the coding tree unit can be signaled through a sequence parameter set, a picture parameter set, or a slice header. The coding tree unit can be divided into smaller size partitions. In this case, if the partition generated by dividing the coding tree unit is depth 1, the partition created by dividing the partition having depth 1 can be defined as depth 2. That is, the partition created by dividing the partition having the depth k in the coding tree unit can be defined as having the depth k + 1.

A partition of arbitrary size generated as the coding tree unit is divided can be defined as a coding unit. The coding unit may be recursively divided or divided into basic units for performing prediction, quantization, transformation, or in-loop filtering, and the like. In one example, a partition of arbitrary size generated as a coding unit is divided may be defined as a coding unit, or may be defined as a conversion unit or a prediction unit, which is a basic unit for performing prediction, quantization, conversion or in-loop filtering and the like.

Alternatively, if a coding block is determined, a prediction block having the same size as the coding block or smaller than the coding block can be determined through predictive division of the coding block. Predictive partitioning of the coded block can be performed by a partition mode (Part_mode) indicating the partition type of the coded block. The size or shape of the prediction block may be determined according to the partition mode of the coding block. The division type of the coding block can be determined through information specifying any one of the partition candidates. At this time, the partition candidates available to the coding block may include an asymmetric partition type (for example, nLx2N, nRx2N, 2NxnU, 2NxnD) depending on the size, type, coding mode or the like of the coding block. In one example, the partition candidate available to the coding block may be determined according to the coding mode of the current block. For example, FIG. 3 illustrates a partition mode that can be applied to a coding block when the coding block is coded by inter-picture prediction.

When the coding block is coded by the inter-picture prediction, one of eight partitioning modes can be applied to the coding block, as in the example shown in Fig.

On the other hand, when the coding block is coded by the intra prediction, the coding mode can be applied to the partition mode PART_2Nx2N or PART_NxN.

PART_NxN may be applied when the coding block has a minimum size. Here, the minimum size of the coding block may be one previously defined in the encoder and the decoder. Alternatively, information regarding the minimum size of the coding block may be signaled via the bitstream. In one example, the minimum size of the coding block is signaled through the slice header, so that the minimum size of the coding block per slice can be defined.

In another example, the partition candidates available to the coding block may be determined differently depending on at least one of the size or type of the coding block. In one example, the number or type of partition candidates available to the coding block may be differently determined according to at least one of the size or type of the coding block.

Alternatively, the type or number of asymmetric partition candidates among the partition candidates available to the coding block may be limited depending on the size or type of the coding block. In one example, the number or type of asymmetric partition candidates available to the coding block may be differently determined according to at least one of the size or type of the coding block.

In general, the size of the prediction block may have a size from 64x64 to 4x4. However, when the coding block is coded by inter-picture prediction, it is possible to prevent the prediction block from having a 4x4 size in order to reduce the memory bandwidth when performing motion compensation.

Depending on the angle of view of the camera, the view of the video captured by the camera is limited. In order to overcome this problem, it is possible to capture a video using a plurality of cameras and stitch the photographed video to form one video or one bit stream. For example, FIGS. 4 to 6 show an example in which a plurality of cameras are used to photograph up and down, right and left, or front and back at the same time. As described above, a video generated by stitching a plurality of videos can be referred to as a panoramic video. In particular, an image having a degree of freedom (Degree of Freedom) based on a predetermined center axis can be referred to as a 360-degree video. For example, the 360 degree video may be an image having rotational degrees of freedom for at least one of Yaw, Roll, and Pitch.

The camera structure (or camera arrangement) for acquiring 360-degree video may have a circular arrangement, as in the example shown in Fig. 4, or a one-dimensional vertical / horizontal arrangement as in the example shown in Fig. Or a two-dimensional arrangement (i.e., a combination of vertical arrangement and horizontal arrangement) as in the example shown in Fig. 5 (b). Alternatively, as in the example shown in Fig. 6, a plurality of cameras may be mounted on the spherical device.

The embodiments described below will be described with reference to 360-degree video, but it will be within the technical scope of the present invention to apply the embodiments described below to panoramic video that is not 360-degree video.

FIG. 7 is a block diagram of a 360-degree video data generation apparatus and a 360-degree video play apparatus, and FIG. 8 is a flowchart illustrating operations of a 360-degree video data generation apparatus and a 360-degree video data apparatus.

7, the 360-degree video data generation apparatus includes a projection unit 710, a frame packing unit 720, an encoding unit 730, and a transmission unit 740, A parsing unit 750, a decoding unit 760, a frame deblocking unit 770, and an inverse decoding unit 780. The encoding unit and the decoding unit shown in FIG. 7 may correspond to the image encoding apparatus and the image decoding apparatus shown in FIG. 1 and FIG. 2, respectively.

The data generation apparatus can determine a projection transformation technique of a 360-degree image generated by stitching an image photographed by a plurality of cameras. In the projection unit 710, the 3D shape of the 360-degree video is determined according to the determined projection transformation technique, and the 360-degree video is projected on the 2D plane according to the determined 3D shape (S801). Here, the projection transformation technique can represent a 3D shape of 360-degree video and an aspect in which 360-degree video is developed on the 2D plane. 360 degree images can be approximated to have shapes such as spheres, cylinders, cubes, octahedrons, or regular twins, etc., in 3D space according to projection transformation techniques. According to the projection transformation technique, an image generated by projecting a 360-degree video onto a 2D plane can be referred to as a 360-degree projection image.

The 360 degree projection image may be composed of at least one face according to the projection transformation technique. For example, when a 360-degree video is approximated as a polyhedron, each face constituting the polyhedron can be defined as a pace. Alternatively, the specific surface constituting the polyhedron may be divided into a plurality of regions, and each divided region may be configured to form a separate face. 360 degree video, which approximates spherical shape, can have multiple faces according to the projection transformation technique.

Frame packing may be performed in the frame packing unit 720 in order to increase the encoding / decoding efficiency of the 360-degree video (S802). The frame packing may include at least one of rearranging, resizing, warping, rotating, or flipping the face. Through the frame packing, the 360 degree projection image can be converted into a form having a high encoding / decoding efficiency (for example, a rectangle) or discontinuous data between faces can be removed. The frame packing may also be referred to as frame reordering or Region-wise Packing. The frame packing may be selectively performed to improve the coding / decoding efficiency for the 360 degree projection image.

In the encoding unit 730, the 360-degree projection image or the 360-degree projection image in which the frame packing is performed may be encoded (S803). At this time, the encoding unit 730 may encode information indicating a projection transformation technique for 360-degree video. Here, the information indicating the projection transformation technique may be index information indicating any one of a plurality of projection transformation techniques.

In addition, the encoding unit 730 can encode information related to frame packing for 360-degree video. Here, the information related to the frame packing may include at least one of whether or not frame packing has been performed, the number of paces, the position of the pace, the size of the pace, the shape of the pace, or the rotation information of the pace.

The transmitter 740 encapsulates the encapsulated data and transmits the encapsulated data to the player terminal (S804).

The file parsing unit 750 can parse the file received from the content providing apparatus (S805). In the decoding unit 760, the 360-degree projection image can be decoded using the parsed data (S806).

If frame packing is performed on the 360-degree projection image, the frame deblocking unit 760 may perform a frame de-packing (Region-wise depacking), which is opposite to the frame packing performed on the content providing side (S807). The frame de-packing may be to restore the frame-packed 360 degree projection image to before the frame packing is performed. For example, frame de-packing may be to reverse the pacing, resizing, warping, rotation, or flipping performed at the data generating device.

The inverse transformation unit 780 can perform inverse projection on the 360 degree projection image on the 2D plane in 3D form according to the projection transformation technique of 360 degree video (S808).

Projection transformation techniques include ERP, Equirectangular Procction, Cube Map Projection (CMP), Icosahedral Projection (ISP), Octahedron Projection (OHP), Cutting Pyramid And may include at least one of Truncated Pyramid Projection (TPP), Shpere Segment Projection (SSP), Equatorial Cylindrical Projection (ECP), and Rotated Sphere Projection (RSP).

Figure 9 shows a 2D projection method using the isometric quadrature method.

The isometric method is a method of projecting a pixel corresponding to a sphere into a rectangle having an aspect ratio of N: 1, which is the most widely used 2D transformation technique. Here, N may be 2, or may be 2 or less or 2 or more real numbers. When using the isometrical method, the actual length of the sphere corresponding to the unit length on the 2D plane becomes shorter as the sphere becomes closer to the sphere. For example, the coordinates of both ends of the unit length on the 2D plane may correspond to a distance difference of 20 cm in the vicinity of the sphere of the sphere, and a distance difference of 5 cm in the vicinity of the sphere of the sphere. As a result, the isochronous quadrature method has a disadvantage in that the image is distorted in the vicinity of the sphere and the coding efficiency is lowered.

10 shows a 2D projection method using a cube projection method.

The cube projection method approximates a 360 degree video with a cube and then transforms the cube into 2D. When projecting a 360 degree video into a cube, one face (or plane) is configured to be adjacent to the four faces. Since the continuity between faces is high, the cube projection method has an advantage in that the coding efficiency is higher than that of the isotropic square method. After the 360 degree video is projected and converted into 2D, the 2D projection converted image may be rearranged into a rectangular shape to perform encoding / decoding.

11 shows a 2D projection method using a bipartite projection technique.

The trilateral projection method is a method of approximating a 360-degree video to a twenty-sided shape and transforming it into 2D. The twin-sided projection technique has a strong continuity between faces. As in the example shown in FIG. 11, it is also possible to perform coding / decoding by rearranging the faces in the 2D projection-converted image.

12 shows a 2D projection method using an octahedral projection technique.

The octahedron projection method is a method of approximating a 360 degree video to an octahedron and transforming it into 2D. The octahedral projection technique is characterized by strong continuity between faces. As in the example shown in FIG. 12, it is possible to perform encoding / decoding by rearranging the faces in the 2D projection-converted image.

13 shows a 2D projection method using a cutting pyramid projection technique.

The truncated pyramid projection technique is a method of approximating a 360 degree video with a cutting pyramid and transforming it into 2D. Under the truncated pyramid projection technique, frame packing may be performed such that the face at a particular point in time has a different size from the neighboring face. For example, as in the example shown in FIG. 13, the Front face may have a larger size than the side face and the back face. When the cutting-type pyramid projection technique is used, the image data at a specific point in time is large, and the encoding / decoding efficiency at a specific point of time is higher than other points.

The SSP is a method of performing 2D projection transformation by dividing spherical 360 degree video into high latitude regions and mid-latitude regions. Specifically, if you follow the SSP, you can map two high-latitude regions of the north and south of the sphere to two circles on the 2D plane, and map the mid-latitude region of the sphere to a rectangle on the 2D plane like ERP.

ECP is a method of transforming spherical 360 degree video into cylindrical shape and then 2D cylindrical projection of 360 degree video. Specifically, when the ECP is followed, the upper and lower surfaces of the cylinder can be mapped to two circles on the 2D plane, and the body of the cylinder can be mapped to a rectangle on the 2D plane.

The RSP represents a method of projecting and transforming a sphere-shaped 360-degree video around a tennis ball into two ellipses on a 2D plane.

Each sample of the 360 degree projection image can be identified by face 2D coordinates. The face 2D coordinates may include an index f for identifying the face where the sample is located, and coordinates (m, n) representing a sample grid in the 360 degree projection image.

Through the conversion between face 2D coordinates and three-dimensional coordinates, 2D projection transformation and image rendering can be performed. For example, FIG. 14 is a diagram illustrating the conversion between face 2D coordinates and three-dimensional coordinates. (X, y, z) and the face 2D coordinates (f, m, n) can be performed using the following equations (1) have.

In the 360 degree projection image, the current picture may include at least one face. At this time, the number of faces may be 1, 2, 3, 4 or more natural numbers, depending on the projection method. In the face 2D coordinates, f may be set to a value equal to or less than the number of faces. The current picture may include at least one pace having the same temporal order or output order (POC).

Alternatively, the number of paces constituting the current picture may be fixed or variable. For example, the number of paces constituting the current picture may be limited so as not to exceed a predetermined threshold value. Here, the threshold value may be a fixed value promised in the encoder and the decoder. Alternatively, information regarding the maximum number of paces constituting one picture may be signaled through the bit stream.

Paces can be determined by partitioning the current picture using at least one of horizontal, vertical, or diagonal lines, depending on the projection method.

Each face in the picture may be assigned an index to identify each face. Each face may be capable of parallel processing, such as a tile or a slice. Accordingly, when intra prediction or inter prediction of the current block is performed, a neighboring block belonging to a different face from the current block can be judged as unavailable.

Pairs that do not allow parallel processing (or non-parallel processing regions) may be defined, or interdependent paces may be defined. For example, paces for which parallel processing is not allowed or interdependent paces may be sequentially encoded / decoded instead of being parallel-encoded / decoded. Accordingly, even if the neighboring block belongs to a different pace than the current block, the neighboring block may be determined to be available for intra prediction or inter prediction of the current block, depending on whether inter-face parallel processing is possible or dependency.

In the embodiments described below, although the embodiment is described based on the specific projection method, the embodiments described later may be extended to the projection method other than the specific projection method.

Since the human viewing angle is limited to about 120 degrees, the user can not see the entire 360 degree video at a time. Due to the limited viewing angle, the user only sees a partial image of the 360 degree video. A partial image corresponding to the viewing area of the user in the entire area of the 360-degree video can be defined as a view port. The play device may decode the 360 degree image and render the portion corresponding to the viewport in the 360 degree video.

The viewport can be classified into a static view port and a dynamic view port according to fluidity. The static viewport refers to a viewport with no positional change, and the dynamic viewport refers to a viewport whose position changes as the user's gaze changes. A static viewport is rarely used for a whole sequence of 360-degree video, and a dynamic viewport whose position changes according to the user's gaze movement is used.

An area that a user should watch carefully in a 360 degree video can be set. As an example, a content producer can set the viewport trajectory that must be watched centrally in 360 degree video. A 360-degree video may refer to a viewport corresponding to an area that a producer or a viewer may be interested in as a region of interest (ROI), a recommended view port, or a director's cut .

Considering the view port, only a part of the 360 degree video can be encoded. As an example, 360 degree video can be encoded around a view port in a 360 degree video. Here, the meaning of encoding a 360-degree video around the viewport may mean encoding a tile / slice that includes a rectangular area corresponding to the viewport, a predetermined size area including the viewport, or a viewport .

For example, 360 degrees of video can be encoded centered on the region of interest of the entire region of the 360 degree video. The encoding of the 360 degree video around the region of interest may be referred to as ROI mode.

Alternatively, the entire 360-degree video may be encoded and decoded, and then only the partial image corresponding to the view port may be extracted from the decoded 360-degree image, and the extracted partial image may be re-encoded. As described above, the re-encoding of only the portion corresponding to the view port in the decoded 360-degree image after encoding and decoding the entire 360-degree video can be referred to as a view port mode.

There is a limitation in watching the area of interest provided by the content creator instead of viewing the area of the user's gaze in the ROI mode. However, in the view port mode, there is an advantage that a dynamic viewport can be provided in accordance with a viewer's gaze movement.

15 is a view for explaining the view port mode.

In the example shown in FIG. 15, the encoder of the server (Encoder A 1510) can encode the whole 360 degree video. The encoded 360 degree video can be transmitted over the network to the intermediate node.

If the server transmits a bitstream for the entire 360 degree video, the intermediate node decoder (Decoder A, 1520) can decode the entire 360 degree video through the bitstream received from the server. Here, the intermediate node may include a Media Aware Network Element (MANE) conforming to the MMT transmission standard. The linear region extractor 1530 of the intermediate node can receive the position data related to the view port from the play device and extract the rectangular area corresponding to the view port from the decoded 360 degree video. When a rectangular region corresponding to the view port is extracted, the encoder (B) 1540 of the intermediate node encodes the extracted rectangular image and transmits a bitstream of the encoded partial image through the network .

Decoder B 1550 of the play apparatus can receive a bitstream of a partial image received via a network and decode the received partial image. The play device can output the area corresponding to the view port in the 360-degree video.

As in the above example, in order to encode / decode / render the partial image corresponding to the view port, it is necessary to specify the position, size or area of the viewport. The position and size (or width) of the viewport and the like can be defined based on at least one of the coordinates of the viewport, the width and height of the viewport, and the angle between the predetermined center axis and the viewport. In one example, the location and size of the viewport can be determined by the angular difference between the viewport center point and the viewport center point and viewport boundary.

The center point of the viewport can be specified by spherical coordinates including a rotation angle Yaw about the z axis, a rotation angle Pitch about the x axis, and a rotation angle Roll about the z axis. In addition, the width and height of the viewport can be specified by information indicating the range of the viewport. Here, the range of the view port may include a Yaw range indicating the angle between the left boundary and the right boundary of the view port, and a pitch range indicating an angle between the upper boundary and the lower boundary of the view port.

16 is a view showing an example in which the position and size of the view port are specified.

For convenience of explanation, it is assumed that the position of the viewport center point and the range of the viewport are defined using spherical coordinates represented by (Yaw, Picth, Roll) as in the example shown in Fig. 16 (a) .

As in the example shown in FIG. 16 (b), the position and size of the viewport can be specified by the viewport center point and the viewport range. For example, the position and size of the viewport can be specified by the coordinates (Yaw, Roll, Pitch) of the viewport, the Yaw range, and the pitch range. Here, the Yaw range is used to specify the left and right range of the viewport, and the pitch range can be used to specify the upper and lower range of the viewport.

The image encoding apparatus for encoding a 360-degree image can encode viewport-related information to specify the position and size of the viewport, and transmit the encoded image to the image decoding apparatus through a bitstream. Here, the view port related information may include information indicating the position of the view port center point and information for specifying the view port range. In addition, the view port related information may further include at least one of angle precision related information, information for determining a picture range to which the view port is applied, and encoding mode information related to the view port.

17 is a diagram showing a syntax related to a view port encoded in a bitstream.

The image encoding apparatus can encode information indicating the angle accuracy. In one example, in the example shown in Fig. 17, spherical_viewport_precision may be a syntax indicating angular precision. The angle precision can be set to a value obtained by multiplying the value of spherical_viewport_precision by 10 ^-2 . In one example, if the value of spherical_viewport_precision is 1, the angular precision can be set to 0.01 degrees. This indicates that the position and range of the viewport can be displayed in units of 0.01 degree.

Alternatively, based on the signaled information, one of a plurality of angular precision candidates may be selected. For example, spherical_viewport_precision may be an index specifying any one of a plurality of angular precision candidates.

The image encoding apparatus can encode information indicating the center point of the view port. For example, in the example shown in FIG. 17, spherical_viewport_yaw indicates the Yaw coordinate of the viewport center point, spherical_viewport_pitch indicates the Pitch coordinate of the viewport center point, and spherical_viewport_roll indicates the Roll coordinate of the viewport center point.

The image encoding apparatus can encode information for specifying the viewport range. 17, spherical_viewport_range_yaw is for specifying the left and right range (i.e., Yaw range) of the viewport, and spherical_viewport_range_pitch is for specifying the upper and lower range (i.e., the pitch range) of the viewport.

spherical_viewport_range_yaw and spherical_viewport_range_pictch may represent angles from the viewport center point to one side of the viewport. For example, spherical_viewport_range_yaw represents the angle from the center of the viewport to the left or right border of the viewport, and spherical_viewport_range_pictch may represent the angle from the center of the viewport to the top or bottom border of the viewport. In this case, the Yaw range extends to the left and right symmetrical areas of the area specified by spherical_viewport_range_yaw, and the pitch range can be extended to the upper and lower symmetrical areas of the area designated by spherical_viewport_range_pitch. As described above, by encoding information corresponding to half of the view port range, it is possible to reduce the amount of bits required to determine the view port range.

Equations (4) and (5) show an example of deriving a Half view port range based on the syntax associated with the view port.

If the value of spherical_viewport_precision is 1, the value of spherical_viewport_range_yaw is 4000, and the value of spherical_viewport_range_pitch is 5000, the half Yaw range can be derived to 40 and the half pitch range to 50. [ Accordingly, the view port can be defined as a yaw area between -40 ° and 40 ° from the center point of the viewport, and a pitch area between -50 ° and 50 ° from the center point of the viewport.

As another example, spherical_viewport_range_yaw and spherical_viewport_range_pictch may be set to represent angles between the two side boundaries of the viewport. For example, a value obtained by multiplying the spherical_viewport_range_yaw value by the angular precision may be induced to the Yaw range, or a value obtained by multiplying the spherical_viewport_range_pitch value by the angular precision may be derived to the Pitch range.

The image encoding apparatus can encode information indicating a picture range to which the view port information is applied. For example, in the example shown in FIG. 17, spherical_viewport_persistence_flag is a syntax indicating whether viewport information is applied to the current picture or to another picture. For example, if the value of spherical_viewport_persistence_flag is 0 (or 1), viewport related syntax (e.g., angle precision, viewport center point, and viewport range related information) may be applied only to the current picture. On the other hand, if the value of spherical_viewport_persistence_flag is 1 (or 0), the viewport related syntax of the current picture can be applied to another picture until a new viewport related syntax appears. Here, another picture may be a picture having an output order (e.g., Picture Order Count, POC) or a decoding order different from the current picture. For example, if the spherical_viewport_persistence_flag is 1, the viewport related syntax of the current picture can be applied to a POC larger than the current picture or a picture to be decoded later than the current picture.

The image encoding apparatus can encode information related to the encoding mode. As an example, in the example shown in Fig. 17, spherical_viewport_mode may indicate the encoding mode of the current sequence or the current picture. For example, a spherical viewport_mode of 0 (or 1) may indicate that the region of interest is applied to the current sequence or the current picture, and spherical_viewport_mode of 1 (or 0) indicates that the viewport mode is applied to the current sequence or the current picture .

In the example shown in Fig. 17, encoding / decoding for angle accuracy may be omitted, and the angular precision already defined in the encoder and decoder may be used.

The view port related information shown in FIG. 17 can be encoded through at least one of a Supplementary Enhancement Information (SEI) message, a sequence header, a picture header, or a slice header.

It is also possible to set a plurality of viewports in one picture. Taking the viewport mode as an example, when a plurality of viewers view 360-degree video, the viewports of a plurality of viewers may be different. In this case, the intermediate node can re-encode the partial area corresponding to the view port of each viewer from the decoded 360-degree video and transmit the re-encoded plural partial images to each play device.

Alternatively, a plurality of regions of interest may be set in the 360 degree video. For example, when a plurality of interest areas are set in a sports game, the viewer can select one of a plurality of interest areas and watch a partial image corresponding to the selected interest area. In the ROI mode, the encoding apparatus encodes a partial image corresponding to each ROI, and may transmit any one of the plurality of encoded partial images to the play apparatus according to the ROI selected by the user.

If multiple viewports are available, the viewport information may further include information about the number of viewports. In this case, the image encoding apparatus can further encode information on the number of viewports.

18 is a diagram showing an example in which a syntax indicating the number of viewports is coded.

The image encoding apparatus can encode information indicating the number of viewports. As an example, in the example shown in FIG. 18, viewport_num may represent the number of viewports. Unlike the illustrated example, it is also possible to encode a syntax obtained by subtracting a predefined integer from the number of viewports. For example, viewport_num_minus1 can represent a value obtained by subtracting 1 from the number of viewports.

The number of viewports may represent at least one of a maximum number or a minimum number of viewports that can be accepted in the current picture. Alternatively, the number of viewports may represent a number applied to at least one of a sequence including a current picture, a temporal layer, a picture, or a slice.

When a plurality of viewports are used, the position and range information of each viewport can be encoded. A different viewport index (viewport_idx) is assigned to the multiple viewport, and location and range information can be encoded for each viewport index.

18, spherical viewport_yaw [viewport_idx], spherical_viewport_pitch [viewport_idx], and spherical_viewport_roll [viewport_idx] represent center point spherical coordinates (Yaw, Pitch, Roll) of a viewport whose viewport index is [viewport_idx] .

In addition, spherical_viewport_range_yaw [viewport_idx] and spherical_viewport_range_pitch [viewport_idx] are used to determine the Yaw range and the pitch range of the viewport whose viewport index is [viewport_idx], respectively.

The image decoding apparatus decodes information indicating the number of viewports. When there are a plurality of viewports, the position and size information of each viewport can be decoded to determine the position and size of each viewport.

Although not shown, information (e.g., a flag) indicating whether a plurality of viewports are allowed may be signaled through the bitstream.

When there are a plurality of viewports, information of each viewport can be independently encoded. For example, in the example shown in Fig. 18, it has been described that the position information and the range information are signaled for each of the plurality of view ports.

As another example, the new viewport information may be encoded / decoded using a pre-encoded / decoded viewport or pre-defined viewport. Specifically, after selecting a reference viewport among a plurality of viewports, current viewport information can be derived using the reference viewport information. Here, the current viewport information derived from the reference viewport may include at least one of the viewport related syntaxes described with reference to FIG. 17 and FIG.

The reference view port may refer to a view port that is a reference for encoding or decoding among a plurality of viewports. For example, the reference viewport may refer to a viewport that is first coded or decoded in a picture, a viewport at a specific position in a picture, or a viewport whose viewport index is a predetermined value (e.g., 0). The specific position for determining the reference view port may be predefined in the image encoding apparatus and the image decoding apparatus. Alternatively, information indicating a specific location for determining the reference view port may be signaled via the bitstream.

Alternatively, adjacent viewports adjacent to the current viewport, and viewports coded / decoded prior to the current viewport may be set as reference viewports.

The number of reference view ports may be one or more.

19 is a view showing an example of deriving current viewport information using a reference viewport.

In a case where a plurality of view ports are supported in the view port mode or the ROI mode, the image encoding apparatus encodes information on the reference viewport or the specific viewport, and for the remaining viewports except for the reference viewport, Can be encoded.

For example, in the example shown in Fig. 19, spherical_viewport_yaw [0], spherical_viewport_pitch [0] and spherical_viewport_roll [0] may represent the coordinates of the center point of the viewport that is index [0]. In addition, spherical_viewport_range_yaw [0] and spherical_viewport_range_pitch [0] can represent the half Yaw range and the half pitch range of the view port which is index [0].

Information about the remaining view port can be encoded as a difference value from the reference view port information. For example, in the example shown in FIG. 19, delta_viewport_yaw, delta_viewport_pitch, and delta_viewport_roll may represent the coordinate difference between the center point of the reference viewport and the center point of the current viewport. When the viewport having the viewport index [0] is the reference viewport, the delta_viewport_yaw, delta_viewport_pitch, and delta_viewport_roll of the viewport having the [viewport_idx] can be defined as shown in Equation (6).

In the example shown in FIG. 19, delta_viewport_range_yaw and delta_viewport_range_pitch may represent a half Yaw range difference value and a half pitch range difference value between the reference viewport and the current viewport. For example, the delta_viewport_range_yaw and the delta_viewport_range_pitch of the viewport having [viewport_idx] can be defined as shown in Equation (7).

As another example, a previous viewport of the current viewport (i.e., a viewport whose viewport index is one less than the current index) may be set as the reference viewport. In this case, the delta_viewport_yaw, delta_viewport_pitch, and delta_viewport_roll of the viewport having the [viewport_idx] can be defined as the difference values from the viewport having the [viewport_idx-1] as shown in the following equation (8).

In addition, delta_viewport_range_yaw and delta_viewport_range_pitch of the viewport having [viewport_idx] can be defined as the difference value with respect to the viewport having [viewport_idx-1] as shown in the following Equation (9).

The video decoding apparatus can determine the position and the range of the current viewport by summing the position and range information for the reference viewport and the position and range difference values for the current viewport.

After three-dimensional rotation is applied in units of view ports, the image can be encoded / decoded. Specifically, at the initial position of the viewport, the sphere may be rotated in at least one direction of Yaw, Pitch, or Roll to obtain the position of the final viewport.

The image encoding apparatus can encode information related to the rotation of the view port into a bit stream. The video decoding apparatus can obtain the rotated viewport by applying the decoded viewport rotation information to the viewport set based on the viewport position and range information.

20 is a diagram showing an example in which information related to rotation of a sphere is encoded.

The image encoding apparatus can encode information indicating the rotation angle accuracy. As an example, in the example shown in Fig. 20, spherical_viewport_orientation_precision may be a syntax indicating rotation angle accuracy. The rotation angle accuracy can be applied to the syntax indicating the rotation angle of the view port. The rotation angle accuracy can be set to a value obtained by multiplying the value of spherical_viewport_orientation_precision by 10 ^-2 . For example, if the value of spherical_viewport_orientation_precision is 1, the angular precision can be set to 0.01 degrees. This indicates that the rotation angle of the view port can be expressed in units of 0.01 degree.

Alternatively, based on the signaled information, one of a plurality of rotation angle accuracy candidates may be selected. For example, the spherical_viewport_orientation_precision may be an index that specifies any one of a plurality of rotation angle precision candidates.

The image encoding apparatus can encode information indicating the rotation angle of the view port. For example, in the example shown in FIG. 20, spherical_viewport_orientation_yaw indicates the degree of rotation of the view port in the Yaw direction, spherical_viewport_orientation_pitch indicates the degree of rotation of the viewport in the pitch direction, and spherical_viewport_orientation_roll indicates the degree of rotation of the viewport in the roll direction.

Rotation angle accuracy and view port rotation information can be used to specify the rotation angle of the view port. For example, Equation 10 shows an example of deriving the view port rotation angle.

Although not shown, information indicating whether or not the viewport is rotated may be further encoded. The rotation information of the view port can be selectively signaled based on information indicating whether or not the view port is rotated.

In the example shown in Fig. 20, encoding / decoding for rotation angle accuracy may be omitted, and the rotation angle accuracy defined in the encoding apparatus and the decryption apparatus may be used. Alternatively, it is also possible to omit encoding / decoding of the rotation angle accuracy and set the rotation angle accuracy to the same degree as the angle accuracy.

In the example shown in Fig. 20, spherical_viewport_orientation_yaw, spherical_viewport_orientation_pitch, and spherical_viewport_orientation_roll simultaneously indicate the rotation direction and the rotation angle. In one example, spherical_viewport_orientation_yaw represents the rotation angle with respect to Yaw. Instead of encoding the syntax indicating both the rotation direction and the rotation angle, it is also possible to encode the rotation direction and the rotation angle into separate information. For example, after information indicating whether the view port rotates in a predetermined direction is encoded, information indicating a rotation angle with respect to the predetermined direction may be encoded when it is determined that the view port is rotated in a predetermined direction. For example, the spherical_viewport_yaw_oriented_flag may be a flag indicating whether the view port is rotated in the Yaw direction. If the flag is 1, a spherical_viewport_orientation_yaw indicating the rotation angle of the view port in the Yaw direction may be further signaled.

Although the above-described embodiments have been described on the basis of a series of steps or flowcharts, they do not limit the time-series order of the invention, and may be performed simultaneously or in different orders as necessary. Further, in the above-described embodiments, each of the components (for example, units, modules, etc.) constituting the block diagram may be implemented by a hardware device or software, and a plurality of components may be combined into one hardware device or software . The above-described embodiments may be implemented in the form of program instructions that may be executed through various computer components and recorded in a computer-readable recording medium. The computer-readable recording medium may include program commands, data files, data structures, and the like, alone or in combination. Examples of computer-readable recording media include magnetic media such as hard disks, floppy disks and magnetic tape, optical recording media such as CD-ROMs and DVDs, magneto-optical media such as floptical disks, media, and hardware devices specifically configured to store and execute program instructions such as ROM, RAM, flash memory, and the like. The hardware device may be configured to operate as one or more software modules for performing the processing according to the present invention, and vice versa.

Claims (15)

Decoding the viewport related information from the bitstream;
Restoring the 360 degree video; And
And setting a viewport on the restored 360 degree video based on the viewport related information,
Wherein the viewport related information includes information on the number of viewports. The method according to claim 1,
If the number information of the view port indicates that a plurality of viewports exist,
Wherein the viewport related information further includes positional information and range information of each of a plurality of viewports. 3. The method of claim 2,
The position information of the first viewport among the plurality of viewports represents a difference value from the position information of the reference viewport and the range information of the first viewport represents a difference value from the range information of the reference viewport Wherein the video decoding method comprises the steps of: The method of claim 3,
Wherein the reference view port has a specific view port index among the plurality of view ports. The method of claim 3,
Wherein the reference view port is a previous viewport of the first viewport. The method according to claim 1,
Wherein the viewport related information further includes rotation information of a viewport. The method according to claim 6,
Wherein the rotation information of the view port includes at least one of a rotation angle in the Yaw direction, a rotation angle in the Pitch direction, and a rotation angle in the Roll direction. Decoding the viewport related information from the bitstream;
Setting a viewport to the 360-degree video; And
Encoding the 360-degree video and the viewport related information,
Wherein the viewport related information includes information on the number of viewports. 9. The method of claim 8,
If the number information of the view port indicates that a plurality of viewports exist,
Wherein the viewport related information further includes position information and range information of each of the plurality of viewports. 10. The method of claim 9,
The position information of the first viewport among the plurality of viewports represents a difference value from the position information of the reference viewport and the range information of the first viewport represents a difference value from the range information of the reference viewport Characterized in that the method comprises the steps of: 11. The method of claim 10,
Wherein the reference view port has a specific view port index among the plurality of view ports. 11. The method of claim 10,
Wherein the reference viewport is a previous viewport of the first viewport. 9. The method of claim 8,
Wherein the viewport related information further includes rotation information of a viewport. 14. The method of claim 13,
Wherein the rotation information of the view port includes at least one of a rotation angle in the Yaw direction, a rotation angle in the Pitch direction, and a rotation angle in the Roll direction. Decoding the viewport related information from the bitstream;
Setting a viewport to the 360-degree video; And
Encoding the 360-degree video and the viewport-related information,
Wherein the viewport-related information includes information about the number of viewports. 20. A recording medium readable by a computing device storing an encoded bitstream.

Images