WO2019083120A1

WO2019083120A1 - Image decoding method and device using reference picture derived by projecting rotated 360-degree video in image coding system for 360-degree video

Info

Publication number: WO2019083120A1
Application number: PCT/KR2018/007545
Authority: WO
Inventors: 이령; 임재현
Original assignee: 엘지전자 주식회사
Priority date: 2017-10-23
Filing date: 2018-07-04
Publication date: 2019-05-02

Abstract

An image decoding method performed by a decoding device, according to the present invention, comprises the steps of: deriving rotation parameters for a 360-degree image; acquiring a rotated reference picture by processing the 360-degree image on the basis of a projection type for the 360-degree image, and the rotation parameters; deriving a reference picture list for a current picture on the basis of the rotated reference picture; and decoding the current picture on the basis of inter prediction using the reference picture list, wherein the rotated reference picture is a picture projected so that the specific position of the 360-degree image in a 3D space, derived on the basis of the rotation parameters, is mapped at the center of the rotated reference picture.

Description

A method and an apparatus for decoding an image using a reference picture derived by projecting 360 degrees of rotated video in a video coding system for 360 degree video

The present invention relates to a 360-degree video, and more particularly, to a video decoding method and apparatus using a reference picture derived by projecting a rotated 360-degree video.

360 degree video can refer to video or image content that is required to provide a virtual reality (VR) system, while being captured or played back in all directions (360 degrees). For example, 360 degree video can be represented on a three-dimensional spherical surface. 360 degree video is captured by capturing an image or video for each of a plurality of viewpoints via one or more cameras and connecting the captured plurality of images / videos into one panorama image / video or spherical image / video, And coding and transmitting the projected picture.

When video data is transmitted using a medium such as a conventional wired / wireless broadband line or image data is stored using an existing storage medium because the amount of information or bit amount to be transmitted increases relative to the existing video data, Transmission costs and storage costs are increased.

Accordingly, there is a demand for a highly efficient video compression technique for effectively transmitting, storing, and reproducing 360-degree video information.

SUMMARY OF THE INVENTION The present invention provides a method and apparatus for increasing 360-degree video information transmission efficiency to provide 360-degree video.

It is another object of the present invention to provide a method and apparatus for deriving a reference picture in which rotated 360 degree video is projected.

It is another object of the present invention to provide a method and apparatus for deriving a reference picture list including a reference picture to which rotated 360 degree video is projected.

According to an embodiment of the present invention, an image encoding method performed by an encoding apparatus is provided. The method includes deriving rotation parameters for a 360 degree image, processing the 360 degree image based on projection type and rotation parameters for the 360 degree image to obtain a rotated reference picture, Deriving a reference picture list for the current picture based on a current picture, generating 360-degree video information for the current picture, and encoding and outputting the 360-degree video information for the current picture, And a specific position on the 3D space is mapped to a center of the rotated reference picture.

According to another embodiment of the present invention, an encoding apparatus for performing image encoding is provided. Wherein the encoding device derives rotation parameters for a 360 degree image, processes the 360 degree image based on the projection type and the rotation parameters for the 360 degree image to obtain a rotated reference picture, A prediction unit for deriving a reference picture list for the current picture based on the current picture, and an entropy encoding unit for encoding and outputting 360-degree video information for the current picture.

According to another embodiment of the present invention, an image decoding method performed by a decoding apparatus is provided. The method includes deriving rotation parameters for the 360 degree image, processing the 360 degree video based on the projection type and the rotation parameters for the 360 degree image to obtain a rotated reference picture, Deriving a reference picture list for the current picture based on a reference picture and decoding the current picture based on inter prediction using the reference picture list, And a specific position on the 3D space derived from the rotation parameters is mapped to a center of the rotated reference picture.

According to another embodiment of the present invention, a decoding apparatus for processing 360-degree video data is provided. The decoding apparatus includes an entropy decoding unit for receiving 360-degree video information, and a controller for deriving rotation parameters for the 360-degree image, processing the 360-degree image based on the projection type and the rotation parameters for the 360- And a predictor for obtaining a reference picture list for the current picture based on the rotated reference picture and decoding the current picture based on inter prediction using the reference picture list .

According to the present invention, a rotated reference picture in which a discontinuous boundary of a reference picture for a 360-degree video is reduced can be derived, and a reference picture list can be derived based on the rotated reference picture, thereby improving prediction accuracy and overall coding efficiency .

According to the present invention, a reference picture can be derived by rotating a 360-degree image and projecting a rotated 360-degree image, thereby reducing prediction efficiency deterioration due to discontinuity of the projected picture and improving overall coding efficiency .

1 is a diagram illustrating an overall architecture for providing 360-degree video according to the present invention.

FIG. 2 illustrates an exemplary processing of 360-degree video in the encoding apparatus and the decoding apparatus.

FIG. 3 is a view for schematically explaining a configuration of a video encoding apparatus to which the present invention can be applied.

FIG. 4 is a view for schematically explaining a configuration of a video decoding apparatus to which the present invention can be applied.

Fig. 5 shows an example of constituting a reference picture list.

FIG. 6 exemplarily shows a projected picture derived based on the ERP.

FIG. 7 shows an example of a rectangular coordinate system in which 360-degree video data is represented by a spherical surface.

FIG. 8 is a diagram showing an Aircraft Principal Axes concept for explaining a spherical surface representing 360-degree video.

FIG. 9 exemplarily shows a projected picture derived based on an ERP to which a symmetric projection is applied.

FIG. 10 exemplarily shows a projected picture derived based on the CMP.

FIG. 11 exemplarily shows a projected picture derived based on CMP to which a symmetric projection is applied.

12 shows an example of a reference picture list including a picture derived based on a projection type to which a symmetrical projection is applied.

FIG. 13 exemplarily shows a method for constructing the reference picture list including pictures derived based on the projection type to which the symmetrical projection is applied.

14 schematically shows a video encoding method by the encoding apparatus according to the present invention.

15 schematically shows a video decoding method by a decoding apparatus according to the present invention.

While the invention is susceptible to various modifications and alternative forms, specific embodiments thereof are shown by way of example in the drawings and will herein be described in detail. However, this is not intended to limit the invention to the specific embodiments. It is to be understood that the terminology used herein is for the purpose of describing particular embodiments only and is not intended to limit the scope of the present invention. The singular expressions include plural expressions unless the context clearly dictates otherwise. It is to be understood that the terms "comprises", "having", and the like in the specification are intended to specify the presence of stated features, integers, steps, operations, elements, parts or combinations thereof, It should be understood that they do not preclude the presence or addition of a combination of numbers, steps, operations, components, parts, or combinations thereof.

In the meantime, the configurations of the drawings described in the present invention are shown independently for convenience of description of different characteristic functions, and do not mean that the configurations are implemented as separate hardware or separate software. For example, two or more of the configurations may combine to form one configuration, or one configuration may be divided into a plurality of configurations. Embodiments in which each configuration is integrated and / or separated are also included in the scope of the present invention unless they depart from the essence of the present invention.

Hereinafter, preferred embodiments of the present invention will be described in detail with reference to the accompanying drawings. Hereinafter, the same reference numerals will be used for the same constituent elements in the drawings, and redundant explanations for the same constituent elements will be omitted.

In this specification, a picture generally refers to a unit that represents one image in a specific time zone, and a slice is a unit that constitutes a part of a picture in coding. One picture may be composed of a plurality of slices, and pictures and slices may be used in combination if necessary.

A pixel or a pel may mean a minimum unit of a picture (or image). Also, a 'sample' may be used as a term corresponding to a pixel. A sample may generally represent a pixel or pixel value and may only represent a pixel / pixel value of a luma component or only a pixel / pixel value of a chroma component.

A unit represents a basic unit of image processing. A unit may include at least one of a specific area of a picture and information related to the area. The unit may be used in combination with terms such as a block or an area in some cases. In general, an MxN block may represent a set of samples or transform coefficients consisting of M columns and N rows.

The present invention proposes a method of providing 360 contents in order to provide a virtual reality (VR) to a user. The VR may mean a technique or an environment for replicating an actual or virtual environment. VR artificially provides the user with a sensory experience that allows the user to experience the same experience as in an electronically projected environment.

360 content refers to the entire content for implementing and providing VR, and may include 360-degree video and / or 360 audio. 360 degree video may refer to video or image content that is required to provide VR while being captured or played back in all directions (360 degrees). Hereinafter, 360-degree video may mean 360-degree video. 360 degree video may refer to a video or image represented in various types of 3D space according to the 3D model, for example, a 360 degree video may be displayed on a spherical surface. 360 audio may also refer to audio content for providing VR, which may be perceived as being located on a three-dimensional specific space of a sound source. 360 content can be created, processed and sent to users, and users can consume VR experience using 360 content.

The present invention particularly proposes a scheme for efficiently providing 360-degree video. To provide 360 degree video, a 360 degree video can first be captured through one or more cameras. The captured 360-degree video is transmitted through a series of processes, and the receiving side can process the received data back into the original 360-degree video and render it. This allows 360-degree video to be provided to the user.

Specifically, the entire process for providing 360-degree video may include a capture process, a preparation process, a transmission process, a processing process, a rendering process, and / or a feedback process.

The capturing process may refer to a process of capturing an image or video for each of a plurality of viewpoints via one or more cameras. Image / video data such as (110) in Fig. 1 shown by the capture process can be generated. Each plane of (110) shown in FIG. 1 may mean image / video for each viewpoint. The captured plurality of images / videos may be referred to as raw data. Metadata associated with the capture can be generated during the capture process.

A special camera can be used for this capture. In the case of providing a 360-degree video for a virtual space generated by a computer according to an embodiment, capturing through a real camera may not be performed. In this case, the process of generating the related data may be replaced with the capturing process.

The preparation process may be a process of processing the captured image / video and metadata generated during the capturing process. The captured image / video may be subjected to a stitching process, a projection process, a region-wise packing process and / or an encoding process in the preparation process.

First, each image / video can be subjected to a stitching process. The stitching process may be a process of linking each captured image / video to create one panoramic image / video or spherical image / video.

Thereafter, the stitched image / video may undergo a projection process. In the projection process, the stitched image / video can be projected onto the 2D image. This 2D image may be referred to as a 2D image frame or a projected picture depending on the context. It can also be expressed as mapping a 2D image to a 2D image. The projected image / video data may be in the form of a 2D image as shown in FIG. 1 (120).

Also, in the projection process, a process of dividing the video data projected on the 2D image into regions and applying the process may be applied. Here, a region may mean a region in which a 2D image in which 360-degree video data is projected is divided. The region may correspond to a face or a tile. According to the embodiment, these regions can be divided into 2D images evenly divided or arbitrarily divided. Also, according to the embodiment, the regions may be classified according to the projection scheme.

According to an embodiment, the processing may include rotating each region or reordering on a 2D image to enhance video coding efficiency. For example, by rotating the regions so that certain sides of the regions are located close to each other, the coding efficiency can be increased.

According to an embodiment, the process may include raising or lowering the resolution for a particular region to differentiate resolution by region on a 360 degree video. For example, regions that are relatively more important in 360-degree video can have a higher resolution than other regions. The video data projected on the 2D image may be encoded through a video codec.

According to the embodiment, the preparation process may further include an editing process and the like. In this editing process, editing of image / video data before and after projection can be further performed. Likewise in the preparation process, metadata for stitching / projection / encoding / editing can be generated. Also, meta data regarding the initial point of time of the video data projected on the 2D image, the ROI (Region of Interest), and the like can be generated.

The transmission process may be a process of processing the prepared image / video data and metadata and transmitting the processed image / video data and metadata. Processing according to any transmission protocol can be performed for transmission. The processed data for transmission may be transmitted over the broadcast network and / or broadband. These data may be delivered to the receiving side on an on-demand basis. The receiving side can receive the corresponding data through various paths.

The processing may be a process of decoding the received data and re-projecting the projected image / video data on the 3D model. In this process, the image / video data projected on the 2D images can be re-projected onto the 3D space. This process can be called mapping, projection, depending on the context. The 3D space mapped at this time may have a different shape depending on the 3D model. For example, a 3D model may have a sphere, a cube, a cylinder, or a pyramid.

According to an exemplary embodiment, the processing may further include an editing process, an up scaling process, and the like. In this editing process, editing of image / video data before and after re-projection can be further performed. If the image / video data is scaled down, it can be enlarged by upscaling the samples during upscaling. If necessary, an operation of reducing the size through downscaling may be performed.

The rendering process may refer to the process of rendering and displaying the re-projected image / video data on the 3D space. It can also be expressed that the re-projection and the rendering are combined according to the representation and rendered on the 3D model. The image / video that is re-projected (or rendered on the 3D model) on the 3D model may have the form of (130) shown in FIG. 1 (130) is a case where the projection is re-projected onto a 3D model of a sphere. The user can view some areas of the rendered image / video through the VR display or the like. In this case, the area viewed by the user may be the same as 140 shown in FIG.

The feedback process may be a process of transmitting various feedback information that can be obtained in the display process to the transmitting side. The feedback process can provide interactivity in 360 degree video consumption. According to the embodiment, Head Orientation information in the feedback process, Viewport information indicating the area currently viewed by the user, and the like can be transmitted to the sender. Depending on the embodiment, the user may interact with those implemented in the VR environment, in which case the information associated with that interaction may be conveyed from the sender to the service provider side in the feedback process. Depending on the embodiment, the feedback process may not be performed.

The head orientation information may mean information about a user's head position, angle, motion, and the like. Based on this information, information about the area that the user is currently viewing within the 360 degree video, i.e. viewport information, can be calculated.

The viewport information may be information about an area that the current user is viewing in a 360 degree video. This allows a Gaze Analysis to be performed to see how the user consumes 360 degrees of video, what area of the 360 degree video is staring, and so on. The Gaussian analysis may be performed on the receiving side and delivered via the feedback channel to the transmitting side. A device such as a VR display can extract a viewport area based on a user's head position / direction, vertical or horizontal FOV (field of view) information supported by the device, and the like.

According to the embodiment, the above-described feedback information may be consumed not only at the transmitting side but also at the receiving side. That is, decoding, re-projection, and rendering processes on the receiving side can be performed using the above-described feedback information. For example, only the 360 degree video for the area that the current user is viewing may be preferentially decoded and rendered using head orientation information and / or viewport information.

Here, the viewport or viewport area may refer to an area viewed by a user in a 360-degree video. A viewpoint is a point that a user is viewing in a 360 degree video, which may mean a center point of the viewport area. That is, the viewport is a region around the viewpoint, and the size and the size occupied by the viewport can be determined by the FOV (Field Of View) described later.

Image / video data that undergoes a series of processes of capture / projection / encoding / transmission / decoding / re-projection / rendering within the overall architecture for providing 360-degree video may be called 360-degree video data. The term 360-degree video data may also be used to include metadata or signaling information associated with such image / video data.

FIG. 2 illustrates an exemplary processing of 360-degree video in the encoding apparatus and the decoding apparatus. FIG. 2 (a) illustrates processing of input 360-degree video data performed by the encoding apparatus. Referring to FIG. 2A, the projection processing unit 210 can stitch and project 360-degree video data at the input time point into a 3D projection structure in accordance with various projection schemes, 360 degree video data can be represented as a 2D image. That is, the projection processing unit 210 can stitch the 360-degree video data, and can project the 2D image. Here, the projection scheme may be referred to as a projection type. The 2D image on which the 360 degree video data is projected may be referred to as a projected frame or a projected picture. The projected picture may be divided into a plurality of faces according to the projection type. The face may correspond to a tile. The plurality of paces in a projected picture according to a particular projection type may be the same size and shape (e.g., triangle or square). In addition, the size and shape of the in-picture face projected according to the projection type may be different. The projection processing unit 210 may perform processing such as rotating and rearranging the respective regions of the projected picture, and changing resolutions of the respective regions. The encoding device 220 may encode information on the projected picture and output the encoded information through a bitstream. The process of encoding the projected picture by the encoding device 220 will be described later in detail with reference to FIG. Meanwhile, the projection processing unit 210 may be included in the encoding apparatus, or the projection process may be performed through an external apparatus.

FIG. 2A illustrates processing of information on the projected picture with respect to 360-degree video data performed by the decoding apparatus. Information on the projected picture may be received through a bitstream.

The decoding apparatus 250 may decode the projection picture based on the information on the received projection picture. The process of decoding the projected picture by the decoding apparatus 250 will be described later in detail with reference to FIG.

The re-projection processing unit 260 can re-project 360-degree video data projected on the projected picture derived through the decoding process on the 3D model. The re-projection processing unit 260 may correspond to the projection processing unit. In this process, the 360-degree video data projected on the projected picture can be re-projected onto the 3D space. This process can be called mapping, projection, depending on the context. The 3D space mapped at this time may have a different shape depending on the 3D model. For example, a 3D model may have a sphere, a cube, a cylinder, or a pyramid. The re-projection processing unit 260 may be included in the decoding device 250, or the re-projection process may be performed through an external device. The re-projected 360 degree video data can be rendered on 3D space.

3, the video encoding apparatus 300 includes a picture dividing unit 305, a predicting unit 310, a residual processing unit 320, an entropy encoding unit 330, an adding unit 340, a filter unit 350 And a memory 360. The memory 360 may be a memory. The residual processing unit 320 may include a subtracting unit 321, a transforming unit 322, a quantizing unit 323, a reordering unit 324, an inverse quantizing unit 325, and an inverse transforming unit 326.

The picture dividing unit 305 may divide the inputted picture into at least one processing unit.

In one example, the processing unit may be referred to as a coding unit (CU). In this case, the coding unit may be recursively partitioned according to a quad-tree binary-tree (QTBT) structure from the largest coding unit (LCU). For example, one coding unit may be divided into a plurality of coding units of deeper depth based on a quadtree structure and / or a binary tree structure. In this case, for example, the quadtree structure is applied first and the binary tree structure can be applied later. Or a binary tree structure may be applied first. The coding procedure according to the present invention can be performed based on the final coding unit which is not further divided. In this case, the maximum coding unit may be directly used as the final coding unit based on the coding efficiency or the like depending on the image characteristics, or the coding unit may be recursively divided into lower-depth coding units Lt; / RTI > may be used as the final coding unit. Here, the coding procedure may include a procedure such as prediction, conversion, and restoration, which will be described later.

As another example, the processing unit may include a coding unit (CU) prediction unit (PU) or a transform unit (TU). The coding unit may be split from the largest coding unit (LCU) into coding units of deeper depth along the quad tree structure. In this case, the maximum coding unit may be directly used as the final coding unit based on the coding efficiency or the like depending on the image characteristics, or the coding unit may be recursively divided into lower-depth coding units Lt; / RTI > may be used as the final coding unit. When a smallest coding unit (SCU) is set, the coding unit can not be divided into smaller coding units than the minimum coding unit. Herein, the term " final coding unit " means a coding unit on which the prediction unit or the conversion unit is partitioned or divided. A prediction unit is a unit that is partitioned from a coding unit, and may be a unit of sample prediction. At this time, the prediction unit may be divided into sub-blocks. The conversion unit may be divided along the quad-tree structure from the coding unit, and may be a unit for deriving a conversion coefficient and / or a unit for deriving a residual signal from the conversion factor. Hereinafter, the coding unit may be referred to as a coding block (CB), the prediction unit may be referred to as a prediction block (PB), and the conversion unit may be referred to as a transform block (TB). The prediction block or prediction unit may refer to a specific area in the form of a block in a picture and may include an array of prediction samples. Also, a transform block or transform unit may refer to a specific region in the form of a block within a picture, and may include an array of transform coefficients or residual samples.

The prediction unit 310 may perform a prediction on a current block to be processed (hereinafter, referred to as a current block), and may generate a predicted block including prediction samples for the current block. The unit of prediction performed in the prediction unit 310 may be a coding block, a transform block, or a prediction block.

The prediction unit 310 may determine whether intra prediction or inter prediction is applied to the current block. For example, the prediction unit 310 may determine whether intra prediction or inter prediction is applied in units of CU.

In the case of intraprediction, the prediction unit 310 may derive a prediction sample for a current block based on a reference sample outside the current block in a picture to which the current block belongs (hereinafter referred to as a current picture). At this time, the predicting unit 310 may derive a prediction sample based on (i) an average or interpolation of neighboring reference samples of the current block, (ii) The prediction sample may be derived based on a reference sample existing in a specific (prediction) direction with respect to the prediction sample among the samples. (i) may be referred to as a non-directional mode or a non-angle mode, and (ii) may be referred to as a directional mode or an angular mode. In the intra prediction, the prediction mode may have, for example, 33 directional prediction modes and at least two non-directional modes. The non-directional mode may include a DC prediction mode and a planar mode (Planar mode). The prediction unit 310 may determine the prediction mode applied to the current block using the prediction mode applied to the neighboring block.

In the case of inter prediction, the prediction unit 310 may derive a prediction sample for a current block based on a sample specified by a motion vector on a reference picture. The prediction unit 310 may apply a skip mode, a merge mode, or a motion vector prediction (MVP) mode to derive a prediction sample for a current block. In the skip mode and the merge mode, the prediction unit 310 can use motion information of a neighboring block as motion information of a current block. In the skip mode, difference (residual) between the predicted sample and the original sample is not transmitted unlike the merge mode. In the MVP mode, a motion vector of a current block can be derived by using a motion vector of a neighboring block as a motion vector predictor to use as a motion vector predictor of a current block.

In the case of inter prediction, a neighboring block may include a spatial neighboring block existing in a current picture and a temporal neighboring block existing in a reference picture. The reference picture including the temporal neighboring block may be referred to as a collocated picture (colPic). The motion information may include a motion vector and a reference picture index. Information such as prediction mode information and motion information may be (entropy) encoded and output in the form of a bit stream.

When the motion information of the temporal neighboring blocks is used in the skip mode and the merge mode, the highest picture on the reference picture list may be used as a reference picture. The reference pictures included in the picture order count can be sorted on the basis of the picture order count (POC) difference between the current picture and the corresponding reference picture. The POC corresponds to the display order of the pictures and can be distinguished from the coding order.

The subtractor 321 generates a residual sample which is a difference between the original sample and the predicted sample. When the skip mode is applied, a residual sample may not be generated as described above.

The transforming unit 322 transforms the residual samples on a transform block basis to generate a transform coefficient. The transforming unit 322 can perform the transform according to the size of the transform block and the prediction mode applied to the coding block or the prediction block spatially overlapping the transform block. For example, if intraprediction is applied to the coding block or the prediction block that overlaps the transform block and the transform block is a 4 × 4 residue array, the residual sample is transformed into a discrete sine transform (DST) In other cases, the residual samples can be converted using a DCT (Discrete Cosine Transform) conversion kernel.

The quantization unit 323 may quantize the transform coefficients to generate quantized transform coefficients.

The reordering unit 324 rearranges the quantized transform coefficients. The reordering unit 324 can rearrange the block-shaped quantized transform coefficients into a one-dimensional vector form through a scanning method of coefficients. The rearrangement unit 324 may be a part of the quantization unit 323, although the rearrangement unit 324 has been described as an alternative configuration.

The entropy encoding unit 330 may perform entropy encoding on the quantized transform coefficients. Entropy encoding may include, for example, an encoding method such as exponential Golomb, context-adaptive variable length coding (CAVLC), context-adaptive binary arithmetic coding (CABAC) The entropy encoding unit 330 may encode the information necessary for video restoration (e.g., the value of a syntax element, etc.) besides the quantized transform coefficients separately or separately. The entropy encoded information may be transmitted or stored in units of NAL (network abstraction layer) units in the form of a bit stream.

The inverse quantization unit 325 inversely quantizes the quantized values (quantized transform coefficients) in the quantization unit 323 and the inverse transformation unit 326 inversely transforms the inversely quantized values in the inverse quantization unit 325, .

The adder 340 combines the residual sample and the predicted sample to reconstruct the picture. The residual samples and the prediction samples are added in units of blocks so that a reconstruction block can be generated. Here, the adder 340 may be part of the predicting unit 310, although the adder 340 has been described as an alternative configuration. Meanwhile, the addition unit 340 may be referred to as a restoration unit or a restoration block generation unit.

For the reconstructed picture, the filter unit 350 may apply a deblocking filter and / or a sample adaptive offset. Through deblocking filtering and / or sample adaptive offsets, artifacts in the block boundary in the reconstructed picture or distortion in the quantization process can be corrected. The sample adaptive offset can be applied on a sample-by-sample basis and can be applied after the process of deblocking filtering is complete. The filter unit 350 may apply an ALF (Adaptive Loop Filter) to the restored picture. The ALF may be applied to the reconstructed picture after the deblocking filter and / or sample adaptive offset is applied.

The memory 360 may store a reconstructed picture (decoded picture) or information necessary for encoding / decoding. Here, the reconstructed picture may be a reconstructed picture in which the filtering procedure has been completed by the filter unit 350. The stored restored picture may be used as a reference picture for (inter) prediction of another picture. For example, the memory 360 may store (reference) pictures used for inter prediction. At this time, the pictures used for inter prediction can be designated by a reference picture set or a reference picture list.

4, the video decoding apparatus 400 includes an entropy decoding unit 410, a residual processing unit 420, a predicting unit 430, an adding unit 440, a filter unit 450, and a memory 460 . The residual processing unit 420 may include a rearrangement unit 421, an inverse quantization unit 422, and an inverse transformation unit 423.

When a bitstream including video information is input, the video decoding apparatus 400 can restore video in response to a process in which video information is processed in the video encoding apparatus.

For example, the video decoding apparatus 400 may perform video decoding using a processing unit applied in the video encoding apparatus. Thus, the processing unit block of video decoding may be, for example, a coding unit and, in another example, a coding unit, a prediction unit or a conversion unit. The coding unit may be partitioned along the quad tree structure and / or the binary tree structure from the maximum coding unit.

A prediction unit and a conversion unit may be further used as the case may be, in which case the prediction block is a block derived or partitioned from the coding unit and may be a unit of sample prediction. At this time, the prediction unit may be divided into sub-blocks. The conversion unit may be divided along the quad tree structure from the coding unit and may be a unit that derives the conversion factor or a unit that derives the residual signal from the conversion factor.

The entropy decoding unit 410 may parse the bitstream and output information necessary for video restoration or picture restoration. For example, the entropy decoding unit 410 decodes information in a bitstream based on a coding method such as exponential Golomb coding, CAVLC, or CABAC, and calculates a value of a syntax element necessary for video restoration, a quantized value Lt; / RTI >

More specifically, the CABAC entropy decoding method includes receiving a bean corresponding to each syntax element in a bitstream, decoding decoding target information of the decoding target syntax element, decoding information of a surrounding and decoding target block, or information of a symbol / A context model is determined and an occurrence probability of a bin is predicted according to the determined context model to perform arithmetic decoding of the bean to generate a symbol corresponding to the value of each syntax element have. At this time, the CABAC entropy decoding method can update the context model using the information of the decoded symbol / bin for the context model of the next symbol / bean after determining the context model.

Information about prediction in the information decoded by the entropy decoding unit 410 is provided to the predicting unit 430. The residual value in which the entropy decoding is performed in the entropy decoding unit 410, that is, the quantized transform coefficient, 421).

The reordering unit 421 may rearrange the quantized transform coefficients into a two-dimensional block form. The reordering unit 421 may perform reordering in response to the coefficient scanning performed in the encoding apparatus. The rearrangement unit 421 may be a part of the inverse quantization unit 422, although the rearrangement unit 421 has been described as an alternative configuration.

The inverse quantization unit 422 can dequantize the quantized transform coefficients based on the (inverse) quantization parameter, and output the transform coefficients. At this time, the information for deriving the quantization parameter may be signaled from the encoding device.

The inverse transform unit 423 may invert the transform coefficients to derive the residual samples.

The prediction unit 430 may predict a current block and may generate a predicted block including prediction samples of the current block. The unit of prediction performed by the prediction unit 430 may be a coding block, a transform block, or a prediction block.

The prediction unit 430 may determine whether intra prediction or inter prediction is to be applied based on the prediction information. In this case, a unit for determining whether to apply intra prediction or inter prediction may differ from a unit for generating a prediction sample. In addition, units for generating prediction samples in inter prediction and intra prediction may also be different. For example, whether inter prediction or intra prediction is to be applied can be determined in units of CU. Also, for example, in the inter prediction, the prediction mode may be determined in units of PU to generate prediction samples. In intra prediction, a prediction mode may be determined in units of PU, and prediction samples may be generated in units of TU.

In the case of intraprediction, the prediction unit 430 may derive a prediction sample for the current block based on the neighbor reference samples in the current picture. The prediction unit 430 may derive a prediction sample for the current block by applying a directional mode or a non-directional mode based on the neighbor reference samples of the current block. In this case, a prediction mode to be applied to the current block may be determined using the intra prediction mode of the neighboring block.

In the case of inter prediction, the prediction unit 430 may derive a prediction sample for a current block based on a sample specified on a reference picture by a motion vector on a reference picture. The prediction unit 430 may derive a prediction sample for the current block by applying one of a skip mode, a merge mode, and an MVP mode. At this time, motion information necessary for inter-prediction of a current block provided in the video encoding apparatus, for example, information on a motion vector, a reference picture index, and the like may be acquired or derived based on the prediction information

In the skip mode and the merge mode, motion information of a neighboring block can be used as motion information of the current block. In this case, the neighboring block may include a spatial neighboring block and a temporal neighboring block.

The predicting unit 430 constructs a merge candidate list using the motion information of the available neighboring blocks and uses the information indicated by the merge index on the merge candidate list as the motion vector of the current block. The merge index may be signaled from the encoding device. The motion information may include a motion vector and a reference picture. When the motion information of temporal neighboring blocks is used in the skip mode and the merge mode, the highest picture on the reference picture list can be used as a reference picture.

In the skip mode, unlike the merge mode, the difference between the predicted sample and the original sample (residual) is not transmitted.

In the MVP mode, a motion vector of a current block can be derived using a motion vector of a neighboring block as a motion vector predictor. In this case, the neighboring block may include a spatial neighboring block and a temporal neighboring block.

For example, when the merge mode is applied, a merge candidate list may be generated using a motion vector of the reconstructed spatial neighboring block and / or a motion vector corresponding to a Col block that is a temporally neighboring block. In the merge mode, the motion vector of the candidate block selected in the merge candidate list is used as the motion vector of the current block. The prediction information may include a merge index indicating a candidate block having an optimal motion vector selected from the candidate blocks included in the merge candidate list. At this time, the predicting unit 430 can derive the motion vector of the current block using the merge index.

As another example, when a motion vector prediction mode (MVP) is applied, a motion vector predictor candidate list is generated by using a motion vector of the reconstructed spatial neighboring block and / or a motion vector corresponding to a Col block which is a temporally neighboring block . That is, the motion vector of the reconstructed spatial neighboring block and / or the motion vector corresponding to the neighboring block Col may be used as a motion vector candidate. The information on the prediction may include a predicted motion vector index indicating an optimal motion vector selected from the motion vector candidates included in the list. At this time, the predicting unit 430 may use the motion vector index to select a predictive motion vector of the current block from the motion vector candidates included in the motion vector candidate list. The predicting unit of the encoding apparatus can obtain the motion vector difference (MVD) between the motion vector of the current block and the motion vector predictor, and can output it as a bit stream. That is, MVD can be obtained by subtracting the motion vector predictor from the motion vector of the current block. In this case, the predicting unit 430 may obtain the motion vector difference included in the information on the prediction, and derive the motion vector of the current block through addition of the motion vector difference and the motion vector predictor. The prediction unit may also acquire or derive a reference picture index or the like indicating the reference picture from the information on the prediction.

The adder 440 may add the residual samples and the prediction samples to reconstruct the current block or the current picture. The adder 440 may restore the current picture by adding residual samples and prediction samples on a block-by-block basis. When the skip mode is applied, since the residual is not transmitted, the predicted sample can be the restored sample. Here, the addition unit 440 is described as an alternative configuration, but the addition unit 440 may be a part of the prediction unit 430. [ Meanwhile, the addition unit 440 may be referred to as a restoration unit or a restoration block generation unit.

The filter unit 450 may apply deblocking filtering sample adaptive offsets, and / or ALF, etc. to the reconstructed pictures. At this time, the sample adaptive offset may be applied on a sample-by-sample basis and may be applied after deblocking filtering. The ALF may be applied after deblocking filtering and / or sample adaptive offsets.

The memory 460 may store a reconstructed picture (decoded picture) or information necessary for decoding. Here, the reconstructed picture may be a reconstructed picture whose filtering process has been completed by the filter unit 450. For example, the memory 460 may store pictures used for inter prediction. At this time, the pictures used for inter prediction may be designated by a reference picture set or a reference picture list. The reconstructed picture can be used as a reference picture for another picture. Further, the memory 460 may output the restored picture according to the output order.

On the other hand, unlike the conventional two-dimensional image, the projected picture of the 360-degree video, which is a three-dimensional image, may include discontinuities due to the characteristics of the projection format . That is, consecutive images in the three-dimensional space can be projected onto the 2D image to derive the projected picture, and there is a discontinuity in which successive 360-degree video data in the three-dimensional space is discontinuously mapped into the projected picture .

In addition, when the inter prediction is performed on the current block as described above, the motion information for the inter prediction can be derived. Meanwhile, when the inter prediction is performed on the 360-degree video, if motion estimation is performed to derive motion information based on the projected picture, the motion estimation process may not be efficient due to the discontinuity of the projected picture . Accordingly, a method of constructing a reference picture list for inter-prediction of the 360-degree video in consideration of the discontinuity of the projected picture may be proposed.

Fig. 5 shows an example of constituting a reference picture list. When inter prediction is performed, motion estimation and motion compensation may be performed to derive the block most similar to the current block for encoding / decoding based on reference pictures. That is, a block most similar to the current block among the blocks in the reference pictures can be derived as a reference block of the current block, and inter-prediction of the current block can be performed based on the reference block. Therefore, constructing a more appropriate reference picture list can improve the prediction accuracy and prediction efficiency of the inter prediction.

For example, referring to FIG. 5, the reference picture list may be configured based on I pictures, P pictures, and B pictures. The number of pictures shown in FIG. 5 indicates the picture order count (POC) of the picture.

Here, the I picture can represent a picture to be independently encoded / decoded regardless of the preceding and following pictures in the encoding / decoding order. That is, only the intra prediction is performed and the picture to be encoded / decoded is not .

In addition, the P picture may represent a picture to be encoded / decoded based on unidirectional inter prediction, i.e., uni-prediction. The inter prediction may be referred to as LO prediction when inter prediction is performed based on the L0 motion information and may be referred to as L1 prediction when inter prediction is performed based on the L1 motion information. And bi-prediction when inter prediction is performed based on the L1 motion information. The L0 prediction or the L1 prediction may represent the single prediction. Here, the L0 motion information may indicate information about an index (refidxL0) of a reference picture included in a reference picture list 0 and an associated motion vector (Motion Vector L0, MVL0) with respect to a current block in a current picture to be encoded / The L1 motion information may indicate information about an index (refidxL0) of a reference picture included in a reference picture list 1 and an associated motion vector (Motion Vector L0, MVL0) with respect to a current block in a current picture to be encoded / decoded. The reference picture list 0 may be referred to as L0 (List 0), the reference picture list 1 may be referred to as L 1 (List 1), and various numbers of reference pictures may be referred to as L 1 L0 or the L1. The B picture may represent a picture to be encoded / decoded based on the bipartite prediction.

Referring to FIG. 5, an arrow between pictures may indicate a reference relationship of the pictures. A reference picture list for the corresponding picture can be generated in accordance with the reference relationship. For example, the reference picture list 0 and the reference picture list 1 for the picture B7 may be configured as follows.

More specifically, the reference picture list 0 may be configured through three steps to be described later.

As a first step, short term reference pictures having a POC smaller than the POC of the current picture among the pictures referenced in the encoding / decoding process of the current picture may be included in the reference picture list 0. For example, pictures P1 and B2 that are short-term reference pictures having a POC smaller than the POC of the picture B7 may be included in the reference picture list 0 for the picture B7. In this case, the short-term reference pictures having a POC that is smaller than the POC of the current picture may be included in descending order of the POC. For example, the reference picture list 0 can be represented by {P1, B2}.

In the second step, the reference picture list 0 may include the short-term reference pictures having a POC larger than the POC of the current picture among the pictures referred to in the encoding / decoding of the current picture. For example, a picture B6 and a picture P5, which are short-term reference pictures having a POC smaller than the POC of the picture B7, may be included in the reference picture list 0 for the picture B7. In this case, the short-term reference pictures having a POC larger than the POC of the current picture may be included in descending order of the POC. For example, the reference picture list 0 can be represented by {P1, B2, B6, P5}.

As a third step, a long term reference picture among the pictures referred to in the encoding / decoding of the current picture may be included in the reference picture list 0. For example, the picture 10, which is the long-term reference picture of the picture B7, may be included in the reference picture list 0 for the picture B7. For example, the reference picture list 0 can be represented by {P1, B2, B6, P5, I0}.

On the other hand, the reference picture list 1 can be configured by changing the order of the first step and the second step among the three steps constituting the reference picture list 0.

Specifically, the reference picture list 1 may be configured through three steps described later.

In the first step, the reference picture list 1 may include the short-term reference pictures having a POC larger than the POC of the current picture among the pictures referred to in the encoding / decoding process of the current picture. For example, a picture B6 and a picture P5, which are short-term reference pictures having a POC smaller than the POC of the picture B7, may be included in the reference picture list 1 for the picture B7. In this case, the short-term reference pictures having a POC larger than the POC of the current picture may be included in descending order of the POC. For example, the reference picture list 1 can be represented by {B6, P5}.

As a second step, the reference picture list 1 may include the short-term reference pictures having a POC smaller than the POC of the current picture among the pictures referred to in the encoding / decoding process of the current picture. For example, pictures P1 and B2, which are short-term reference pictures having a POC smaller than the POC of the picture B7, may be included in the reference picture list 1 for the picture B7. In this case, the short-term reference pictures having a POC that is smaller than the POC of the current picture may be included in descending order of the POC. For example, the reference picture list 1 can be represented by {B6, P5, P1, B2}.

As a third step, a long term reference picture among the pictures referred to in the encoding / decoding of the current picture may be included in the reference picture list 1. For example, the picture I0, which is the long-term reference picture of the picture B7, may be included in the reference picture list 1 for the picture B7. For example, the reference picture list 1 can be expressed as {B6, P5, P1, B2, I0}.

The reference picture list 0 and the reference picture list 1 may be configured differently according to the length of the predetermined list, and the length of the list may be set to a maximum of 15.

On the other hand, 360 degree video in 3D space is essentially borderless, but when the 360 degree video is projected onto a 2D image, a boundary where discontinuities appear can be created. Specifically, the 360 degree video can be projected onto a 2D image as described below.

FIG. 6 exemplarily shows a projected picture derived based on the ERP. 360 degree video data may be projected onto a 2D picture, where the 2D picture on which the 360 degree video data is projected may be referred to as a projected frame or a projected picture. The 360 degree video data may be projected onto a picture through various projection types. For example, 360-degree video data can be divided into Equirectangular Projection (ERP), Cube Map Projection (CMP), Icosahedral Projection (ISP), Octahedron Projection (OHP), Truncated Square Pyramid Projection (TSP), Segmented Sphere Projection Can be projected and / or packed into a picture through Area Projection (EAP). Specifically, the stitched 360 degree video data can be represented on a 3D projection structure according to the projection type, that is, the 360 degree video data can be mapped to the face of the 3D projection structure of each projection type, The face may be projected onto the projected picture.

Referring to FIG. 6, 360-degree video data can be projected onto a 2D picture through an ERP. When 360-degree video data is projected through the ERP, for example, stitched 360-degree video data can be represented on a spherical surface, i.e., the 360- Can be mapped, and can be projected into one picture in which continuity on the spherical surface is maintained. The 3D projection structure of the ERP may be a sphere having one face. Thus, as shown in FIG. 6, the 360 degree video data can be mapped to one face in the projected picture.

Also, as another example, 360 degree video data can be projected through the CMP. The 3D projection structure of the CMP may be a cube. Thus, when 360-degree video data is projected through the CMP, the stitched 360-degree video data can be represented in a cube, and the 360-degree video data can be projected onto a 2D image in a cubic 3D projection structure . That is, the 360 degree video data can be mapped to six faces of the cube, and the faces can be projected onto the projected picture.

Also, as another example, 360 degree video data may be projected through the ISP. The 3D projection structure of the ISP may be a twenty-sided form. Therefore, when the 360-degree video data is projected through the ISP, the stitched 360-degree video data can be represented in a twenty-sided form, and the 360-degree video data can be divided into a two- Lt; / RTI > That is, the 360 degree video data may be mapped to twenty faces of a twosopaper, and the faces may be projected onto the projected picture.

Also, as another example, 360 degree video data may be projected through the OHP. The 3D projection structure of the OHP may be octahedral. Therefore, when the 360-degree video data is projected through the OHP, the stitched 360-degree video data can be displayed on the octahedron, and the 360-degree video data is divided into the octahedral 3D projection structure and projected onto the 2D image . That is, the 360 degree video data can be mapped to eight faces of the octahedron, and the faces can be projected onto the projected picture.

Also, as another example, 360 degree video data may be projected through the TSP. The 3D projection structure of the TSP may be a truncated square pyramid. Thus, when the 360-degree video data is projected through the TSP, the stitched 360-degree video data may be represented in the pyramid with the top portion truncated, and the 360-degree video data may be represented as a pyramid- Can be divided into projection structures and projected onto a 2D image. That is, the 360 degree video data may be mapped to six faces of the pyramid with the top portion cut out, and the faces may be projected onto the projected picture.

Also, as another example, 360 degree video data may be projected through the SSP. The 3D projection structure of the SSP may be a spherical surface having six faces. In particular, the faces may include two circular-shaped faces for the anode regions and four square block-shaped faces for the remaining regions. Thus, when the 360 degree video data is projected through the SSP, the stitched 360 degree video data may be represented by a spherical surface having the six faces, and the 360 degree video data may be represented by a rectangle having the six faces And can be divided into a 3D projection structure in the form of a plane to be projected on a 2D image. That is, the 360 degree video data may be mapped to six faces of the spherical surface, and the faces may be projected onto the projected picture.

Also, as another example, 360 degree video data may be projected through the EAP. The 3D projection structure of the EAP may be predefined. Thus, when the 360 degree video data is projected through the EAP, the stitched 360 degree video data can be represented on a spherical surface, i.e., the 360 degree video data can be mapped on the spherical surface, The continuity on the spherical surface is maintained. That is, the 360 degree video data can be mapped to one face of a sphere, and the face can be projected onto the projected picture. Here, the EAP may represent a method of projecting a specific area on the spherical surface onto the projected picture in a size equal to the size on the spherical surface, unlike the ERP.

On the other hand, when the 360-degree video data is projected through the ERP, the 3D space of the ERP, that is, the 360-degree video data on the spherical surface, as shown in FIG. 6, The center point on the spherical surface may be mapped to the center point of the projected picture and the continuity on the spherical surface may be maintained. Here, the center of gravity on the spherical surface may be referred to as an orientation on the spherical surface.

If the spherical surface, that is, the 3D space is represented by a spherical coordinate, the center point may be a point where? = 0 and? = 0, and the 3D space may be defined as a yaw / pitch / Coordinate system), it can mean pitch = 0, yaw = 0, and roll = 0. Where the 3D space may be referred to as a projection structure or VR geometry.

The spherical coordinate system representing the 3D space and the yaw / pitch / roll coordinate system may be as follows.

FIG. 7 shows an example of a rectangular coordinate system in which 360-degree video data is represented by a spherical surface. The 360 degree video data obtained from the camera may be represented by a spherical surface. As shown in FIG. 7, each point on the spherical surface is expressed by r (sphere radius),? (Rotational direction and degree with respect to the z axis),? (Rotational direction and degree with respect to the z axis of the xy plane) Lt; / RTI > According to the embodiment, the spherical surface may coincide with the world coordinate system, or the principal point of the front camera may be a (r, 0, 0) point of the spherical surface.

On the other hand, the position of each point on the spherical surface can be expressed based on the Aircraft Principal Axes. For example, the position of each point on the spherical surface may be expressed through pitch, yaw, and roll.

FIG. 8 is a diagram showing an Aircraft Principal Axes concept for explaining a spherical surface representing 360-degree video. In the present invention, the concept of a plane main axis can be used to express a specific point, a position, a direction, an interval, an area, and the like in 3D space. That is, in the present invention, the concept of the airplane main axis can be used to describe the 3D space before or after the projection, and to perform signaling on the 3D space. Specifically, the position of each point on the spherical surface can be expressed based on the Aircraft Principal Axes. The three-dimensional axes can be referred to as a pitch axis, a yaw axis, and a roll axis, respectively. In the present specification, these may be abbreviated to pitch, yaw, roll to pitch, yaw, and roll directions. The position of each point on the spherical surface may be expressed through pitch, yaw, and roll. When compared with the XYZ coordinate system, the pitch axis may correspond to the X axis, the yaw axis to the Z axis, and the roll axis to the Y axis.

Referring to FIG. 8A, the yaw angle may indicate the direction and degree of rotation based on the yaw axis, and the range of the yaw angle may range from 0 degrees to +360 degrees, or from -180 degrees to +180 degrees Lt; / RTI > 8B, the pitch angle may indicate the rotational direction and the degree with respect to the pitch axis, and the range of the pitch angle may range from 0 degrees to +180 degrees or from -90 degrees to +90 degrees . The roll angle can indicate the direction and degree of rotation with respect to the roll axis, and the range of roll angles can be from 0 degrees to +360 degrees or from -180 degrees to +180 degrees. In the following description, the yaw angle increases in a clockwise direction, and the range of the yaw angle may be assumed to be 0 to 360 degrees. In addition, the pitch angle increases as the polar angle increases, and the range of the polar angle may be assumed to be -90 degrees to +90 degrees.

On the other hand, as shown in FIG. 6, the projected picture includes a boundary 600. As the discontinuous boundary is minimized, more accurate motion information can be output, thereby improving the efficiency of inter prediction. Accordingly, the present invention proposes a projection type in which the discontinuous boundary of the projected picture is minimized. Specifically, a projection type in which 360-degree video data on the spherical surface is rotated in various manners, and the rotated 360-degree video data is projected onto the 2D picture can be proposed.

FIG. 9 exemplarily shows a projected picture derived based on an ERP to which a symmetric projection is applied. Here, when the above-mentioned projection type is applied to the above-mentioned symmetrical projection to project the projected picture, instead of being projected so that the middle point on the existing spherical surface is mapped to the center point of the projected picture, Rotated, rotated 360 degree video data can be projected onto the 2D picture. In other words, instead of mapping the center point on the spherical surface to the center point of the projected picture, a position derived by rotating the center point on the spherical surface by a specific value is mapped to the center point of the projected picture, An ERP with a symmetrical projection that is projected onto a single retained picture may be proposed.

FIG. 9A shows a projected picture derived by projecting 360-degree video data rotated by (180, 0, 90) to one picture in which continuity is maintained. That is, FIG. 7A can show a picture projected so that a specific position of the (180, 0, 90) coordinate is mapped to the center point of the projected picture. Here, the coordinates indicating the specific position may be coordinates represented by pitch, yaw, and roll.

Referring to FIG. 9A, upper and lower boundaries of a picture projected through a conventional ERP may disappear, and a left boundary and a right boundary may be derived as a boundary 900 located at the center. FIG. 9B shows a projected picture derived by projecting 360-degree video data rotated by (180, 0, -90) to one picture in which continuity is maintained. That is, FIG. 9B shows a picture projected so that the specific position of the (180, 0, -90) coordinate is mapped to the center point of the projected picture. Referring to FIG. 9B, the upper and lower boundaries of the picture projected through the existing ERP may disappear, and the left boundary and the right boundary may be derived as the boundary 910 located at the center.

On the other hand, the above-described symmetrical projection can be applied to other projection types other than the ERP.

FIG. 10 exemplarily shows a projected picture derived based on the CMP. Referring to FIG. 10, a projected picture derived based on a conventional CMP is exemplarily shown. Referring to FIG. 10, 360-degree video data can be projected through the CMP, and the 3D projection structure of the CMP can be a cube. Thus, when 360-degree video data is projected through the CMP, the stitched 360-degree video data can be represented in a cube, and the 360-degree video data can be projected onto a 2D image in a cubic 3D projection structure . That is, the 360 degree video data can be mapped to six faces of the cube, and the faces can be projected onto the projected picture.

On the other hand, as shown in FIG. 10, the projected picture includes a boundary 1000. As the discontinuous boundary is minimized, more accurate motion information can be output, thereby improving the efficiency of inter prediction. Accordingly, the present invention proposes a projection type in which the discontinuous boundary of the projected picture is minimized.

FIG. 11 exemplarily shows a projected picture derived based on CMP to which a symmetric projection is applied. When the projected picture is derived by applying the symmetrical projection to the projection type described above, instead of being projected so as to map the center point on the existing spherical surface to the center point of the projected picture, the 360 degree video data on the spherical surface is rotated , The rotated 360 degree video data may be projected onto the 2D picture. When the 360 degree video data is projected through the CMP to which the symmetrical projection is applied, the center point on the spherical surface is mapped to the center point of the projected picture, Can be mapped to six faces of the cube while being mapped to the midpoint of the projected picture, and the faces can be projected onto the projected picture.

FIG. 11 shows a projected picture derived by projecting 360-degree video data rotated by (0, 0, 90) to one picture in which continuity is maintained. That is, Fig. 11 can represent a picture projected so that a specific position of the (0, 0, 90) coordinate is mapped to the center point of the projected picture. Referring to FIG. 11, a projected picture in which discontinuous boundaries are less than a picture projected through a conventional CMP based on the symmetric projection-applied CMP can be derived.

On the other hand, as described above, a discontinuous picture with few boundaries can be derived based on the projection type to which the symmetric projection is applied, and a method of constructing a reference picture list using the picture can be proposed.

12 shows an example of a reference picture list including a picture derived based on a projection type to which a symmetrical projection is applied. L0 and L1 for the current picture can be derived as shown in Fig. For example, the picture order count (POC) of the current picture may be 3, the picture L0 for the current picture may include a picture having a POC of 0 and a picture having a POC of 4, A picture having a POC of 4 and a picture having a POC of 4 and a picture having a POC of 4 and a picture having a POC of 4 and a picture having a POC of 0 As shown in FIG. That is, L0 can be expressed as {POC0, POC4}, and L1 can be expressed as {POC4, POC0}. When inter prediction is applied to the current block in the current picture 1200, a block most similar to the current block 1200 among the blocks in the reference pictures included in the reference picture list is included in the current block 1200 Can be derived as a reference block. For example, a reference block P0 1210 of the current block 1200 may be included in a picture having the POC of 0 in L0. However, as shown in FIG. 12, the reference block P0 1210 may be located in an area adjacent to the boundary of the picture with the POC of 0, and may be partially available.

When the existing inter prediction is performed, the samples of the non-available region may be padded based on the samples located at the left boundary of the reference block P0 (1210).

However, as described above, the reference picture derived based on the projection type to which the symmetric projection is applied can be included in the reference picture list L0 and / or the reference picture list L1 for the current picture. For example, a 360 degree video for the POC of 0 can be rotated at a specific angle, and a reference picture projected from the rotated 360 degree video can be derived. The projected reference picture may be represented as POC0 '.

In this case, since the reference block P0 '1220 at the position corresponding to the reference block P0 1210 in the POC0' can be derived as a reference block of the current block 1200, the reference block P0 ' Can be available without being located in an area adjacent to the boundary of the picture. Therefore, performing inter-prediction on the current block 1200 based on the reference block P0 '1220 is more efficient than performing inter-prediction based on the reference block P0 1210 including the padded region Prediction accuracy and coding efficiency can be improved.

The encoding apparatus / decoding apparatus may include a short-term reference picture of the current picture in a reference picture list for the current picture (S1300). For example, if the current picture is a POC 3 picture, the short reference picture of the current picture may include a picture having a POC of 0 and a picture having a POC of 4. In this case, a picture having the POC of 0 and a picture having the POC of 4 can be included in the reference picture list for the current picture. The reference pictures included in the reference picture list may be arranged based on a picture order count (POC) difference between the current picture and the corresponding reference picture.

The encoding apparatus / decoding apparatus may include a modified short time reference picture of the current picture in the reference picture list for the current picture (S1310). The modified short-term reference picture may represent a reference picture in which a 360-degree video for the short-term reference picture rotated at a certain angle is projected. For example, a 360 degree video for the POC of 0 can be rotated at a specific angle, and a reference picture with a modified POC of 0, from which the rotated 360 degree video is projected, can be derived. Also, a 360-degree video for the POC 4 can be rotated at a specific angle, and a reference picture with the modified POC 4 for which the rotated 360-degree video is projected can be derived.

The encoding apparatus / decoding apparatus may include a long-term reference picture of the current picture in the reference picture list for the current picture (S1320).

14 schematically shows a video encoding method by the encoding apparatus according to the present invention. The method disclosed in Fig. 14 can be performed by the encoding apparatus disclosed in Fig. Specifically, for example, S1400 to S1420 in FIG. 14 can be performed by the projection processing unit and the prediction unit of the encoding apparatus, and S1430 can be performed by the entropy encoding unit of the encoding apparatus.

The encoding device derives the rotation parameters for the 360 degree image (S1400).

The encoding apparatus may derive a specific yaw value, a specific pitch value and a specific roll value, which cause the discontinuous boundary of the reference picture for the current picture to be minimized, with the rotation parameters. Here, the reference picture may be a picture on which the 360-degree image is projected based on a projection type. That is, the reference picture may be a picture projected so that the 360-degree image is mapped to a center of the reference picture in the 3D space. Here, the center of gravity in the 3D space may be referred to as an orientation in the 3D space.

In addition, the discontinuous boundary may represent a boundary where the projected 360 degree image is not continuous. If the projection type is Equirectangular Projection (ERP), the discontinuous boundary of the reference picture may be derived as an upper boundary, a lower boundary, a left boundary, and a right boundary. Also, when the projection type is CMP (Cube Map Projection), the discontinuous boundary of the reference picture may be derived as an upper boundary, a lower boundary, a left boundary, a right boundary, and a center horizontal boundary. Here, the center horizontal boundary may be a horizontal line passing the center of the reference picture. On the other hand, the encoding apparatus may generate information indicating the rotation parameters. That is, the encoding apparatus can generate information indicating the specific desired value, information indicating the specific pitch value, and information indicating the specific roll value.

The encoding apparatus processes the 360-degree image based on the projection type and the rotation parameters for the 360-degree image to obtain a rotated reference picture (S1410). Here, the projection type may be an ERP (Equirectangular Projection), the 3D space may be a spherical surface, the specific desired value may be 180, the specific pitch value may be 0, and the specific roll value may be 90. [ Alternatively, the projection type may be an ERP (Equirectangular Projection), the 3D space may be a spherical surface, the specific yaw value may be 180, the specific pitch value may be 0, and the specific roll value may be -90 . Alternatively, the projection type may be CMP (Cube Map Projection), the 3D space may be a cube, the specific yaw value is 0, the specific pitch value is 0, and the specific roll value is 90 .

The encoding apparatus can project the 360-degree image on the 3D space (3D projection structure) onto the 2D image (or picture) based on the projection type and the rotation parameters for the 360-degree image, and obtains the rotated reference picture can do.

Specifically, the encoding device derives the 360-degree image on the 3D space and the rotated 360-degree image based on the rotation parameters, and the specific position, which is the center of the rotated 360-degree image, The 360-degree image may be projected onto the 2D picture so as to be mapped to derive the rotated reference picture. That is, the rotated reference picture may be a picture that is projected so that the 360-degree image is mapped to a center of the rotated reference picture at a specific position in the 3D space.

The 360-degree image in the 3D space may be projected so that a specific position in the 3D space derived based on the rotation parameters is mapped to a center of the rotated reference picture. Herein, the rotation parameters may include a specific pitch value, a specific yaw value, and a specific roll value, and the specific position may be determined such that the yaw component is the specific yaw value, the pitch component is the specific pitch value, And may be a position that is the specific roll value. That is, the specific position may be a position shifted by the rotation parameters at a midpoint on the 3D space. Alternatively, the 360-degree image may be projected around a center point in the 3D space to derive the rotated reference picture, and the 360-degree video in the rotated reference picture may be rotated based on the rotation parameters.

In addition, the encoding apparatus can perform projection on a 2D image (or a picture) according to a projection type for the 360-degree image among a plurality of projection types, and obtain the rotated reference picture. The projection type may correspond to the projection method described above. The various types of projections include Equirectangular Projection (ERP), Cube Map Projection (CMP), Icosahedral Projection (ISP), Octahedron Projection (OHP), Truncated Square Pyramid Projection (TSP), Segmented Sphere Projection ). The 360 degree image can be mapped to the faces of the 3D projection structure of each projection type, and the faces can be projected onto the rotated reference picture. That is, the rotated reference picture may include paces of a 3D projection structure of each projection type. For example, the 360-degree image may be projected onto the rotated reference picture based on a cube map projection (CMP), in which case the 3D projection structure may be a cube. In this case, the 360 degree image may be mapped to six faces of the cube, and the faces may be projected to the rotated reference picture. As another example, the 360 degree image may be projected onto the rotated reference picture based on ISP (Icosahedral Projection), in which case the 3D projection structure may be a twentieth face. As another example, the 360-degree image may be projected onto the rotated reference picture based on an Octahedron Projection (OHP), in which case the 3D projection structure may be octahedral. In addition, the encoding apparatus may perform processing such as rotating and rearranging the faces of the rotated reference picture, changing resolutions of the respective faces, and the like.

The encoding apparatus derives a reference picture list for the current picture based on the rotated reference picture (S1420). The reference picture list may include a reference picture list 0 (List 0, L0) and / or a reference picture list 1 (List 1, L1). The reference pictures included in the reference picture list may be arranged based on a picture order count (POC) difference between the current picture and the corresponding reference picture. The POC corresponds to the display order of the pictures and can be distinguished from the coding order. The reference picture list may include the rotated reference pictures.

In addition, the reference picture list may include a reference picture in which the 360-degree image is projected. Here, the reference picture may be a picture projected so that the 360-degree image is mapped to the center of the reference picture in the 3D space. In this case, the reference picture list may be arranged in the order of the reference picture and the rotated reference picture. On the other hand, the rotated reference picture may include a discontinuous boundary smaller than the reference picture. The discontinuous boundary may represent a boundary where the projected 360 degree image is not continuous.

The encoding apparatus generates 360-degree video information for the current picture, encodes it, and outputs the 360-degree video information (S1440). The encoding device may generate the 360-degree video information for the current picture, and may encode the 360-degree video information and output it through a bitstream, the bitstream may be transmitted over a network, May be stored in a non-transitory computer readable medium.

In addition, the 360-degree video information may include information indicating a projection type. The projection type may be one of a plurality of projection types, and the various projection types may be classified into Equirectangular Projection (ERP), Cube Map Projection (CMP), Icosahedral Projection (ISP), Octahedron Projection (TSP), Segmented Sphere Projection (SSP), and Equal Area Projection (EAP).

In addition, the 360-degree video information may include information indicating the rotation parameters. That is, the 360-degree video information may include information indicating the specific yaw value, information indicating the specific pitch value, and information indicating the specific roll value.

Meanwhile, although not shown in the figure, when the inter-prediction is performed based on the reference picture list to perform decoding on the current picture, the encoding apparatus can derive a prediction sample for the current picture. In addition, the encoding apparatus may generate residual samples based on the original sample and the derived prediction samples. When the residual samples are generated, the encoding apparatus can generate information on the residuals based on the residual samples. The information on the residual may include transform coefficients relating to the residual sample. The encoding apparatus may derive the reconstructed sample based on the prediction sample and the residual sample. That is, the encoding apparatus may add the prediction sample and the residual sample to derive the reconstructed sample. Also, the encoding apparatus can encode the information on the residual and output it in the form of a bit stream. The bitstream may be transmitted to a decoding device via a network or a storage medium.

15 schematically shows a video decoding method by a decoding apparatus according to the present invention. The method disclosed in Fig. 15 can be performed by the decoding apparatus disclosed in Fig. Specifically, for example, steps S1500 to S1530 of FIG. 15 may be performed by the predicting unit and the re-projection processing unit of the decoding apparatus.

The decoding apparatus derives the rotation parameters for the 360-degree image (S1500). The decoding apparatus may derive a specific yaw value, a specific pitch value, and specific roll values such that the discontinuous boundary of the reference picture is minimized with the rotation parameters. Here, the reference picture may be a picture on which the 360-degree image is projected based on a projection type. That is, the reference picture may be a picture projected so that the 360-degree image is mapped to a center of the reference picture in the 3D space. Here, the center of gravity in the 3D space may be referred to as an orientation in the 3D space.

Also. The discontinuous boundary may represent a boundary where the projected 360 degree image is not continuous. If the projection type is Equirectangular Projection (ERP), the discontinuous boundary of the reference picture may be derived as an upper boundary, a lower boundary, a left boundary, and a right boundary. Also, when the projection type is CMP (Cube Map Projection), the discontinuous boundary of the reference picture may be derived as an upper boundary, a lower boundary, a left boundary, a right boundary, and a center horizontal boundary. Here, the center horizontal boundary may be a horizontal line passing the center of the reference picture. On the other hand, the decoding apparatus may generate information indicating the rotation parameters. That is, the decoding apparatus may generate information indicating the specific yaw value, information indicating the specific pitch value, and information indicating the specific roll value.

Alternatively, the decoding apparatus may receive 360-degree video information through a bitstream, and the 360-degree video information may include information indicative of the rotation parameters. That is, the 360-degree video information may include information indicating the specific yaw value, information indicating the specific pitch value, and information indicating the specific roll value.

The decoding apparatus processes the 360-degree image based on the projection type and the rotation parameters for the 360-degree image to obtain a rotated reference picture (S1510). Here, the projection type may be an ERP (Equirectangular Projection), the 3D space may be a spherical surface, the specific desired value may be 180, the specific pitch value may be 0, and the specific roll value may be 90. [ Alternatively, the projection type may be an ERP (Equirectangular Projection), the 3D space may be a spherical surface, the specific yaw value may be 180, the specific pitch value may be 0, and the specific roll value may be -90 . Alternatively, the projection type may be CMP (Cube Map Projection), the 3D space may be a cube, the specific yaw value is 0, the specific pitch value is 0, and the specific roll value is 90 . On the other hand, the 360-degree video information may include projection type information indicating the projection type, and the projection type may be derived based on the projection type information. Here, the projection type includes an Equal Projection (ERP), a Cube Map Projection (CMP), an Icosahedral Projection (ISP), an Octahedron Projection (OHP), a Truncated Square Pyramid Projection (TSP), a Segmented Sphere Projection EAP).

The decoding apparatus can project the 360-degree image on the 3D space (3D projection structure) onto the 2D image (or picture) based on the projection type and the rotation parameters for the 360-degree image, and obtains the rotated reference picture can do.

Specifically, the decoding apparatus derives the 360-degree image on the 3D space and the rotated 360-degree image based on the rotation parameters, and the specific position, which is the center of the rotated 360-degree image, The 360-degree image may be projected onto the 2D picture so as to be mapped to derive the rotated reference picture. That is, the rotated reference picture may be a picture that is projected so that a specific position on the 3D space derived from the rotation parameters is mapped to a center of the rotated reference picture.

The 360-degree image in the 3D space may be projected so that a specific position in the 3D space derived based on the rotation parameters is mapped to a center of the rotated reference picture. Herein, the rotation parameters may include a specific pitch value, a specific yaw value, and a specific roll value, and the specific position may be determined such that the yaw component is the specific yaw value, the pitch component is the specific pitch value, And may be a position that is the specific roll value. That is, the specific position may be a position shifted by the rotation parameters at a midpoint on the 3D space. Alternatively, the 360-degree image may be projected on a center point in the 3D space to derive the rotated reference picture, and the 360-degree picture in the rotated reference picture may be rotated based on the rotation parameters.

Also, the decoding apparatus can perform projection on a 2D image (or a picture) according to a projection type for the 360-degree image among a plurality of projection types, and obtain the rotated reference picture. The projection type may correspond to the projection method described above. The various types of projections include Equirectangular Projection (ERP), Cube Map Projection (CMP), Icosahedral Projection (ISP), Octahedron Projection (OHP), Truncated Square Pyramid Projection (TSP), Segmented Sphere Projection ). The 360 degree image can be mapped to the faces of the 3D projection structure of each projection type, and the faces can be projected onto the rotated reference picture. That is, the rotated reference picture may include paces of a 3D projection structure of each projection type. For example, the 360-degree image may be projected onto the rotated reference picture based on a cube map projection (CMP), in which case the 3D projection structure may be a cube. In this case, the 360 degree image may be mapped to six faces of the cube, and the faces may be projected to the rotated reference picture. As another example, the 360 degree image may be projected onto the rotated reference picture based on ISP (Icosahedral Projection), in which case the 3D projection structure may be a twentieth face. As another example, the 360-degree image may be projected onto the rotated reference picture based on an Octahedron Projection (OHP), in which case the 3D projection structure may be octahedral. In addition, the decoding apparatus may perform processing such as rotating and rearranging the faces of the rotated reference picture, changing resolutions of the respective faces, and the like.

The decoding apparatus derives a reference picture list for the current picture based on the rotated reference picture (S1520). The reference picture list may include a reference picture list 0 (List 0, L0) and / or a reference picture list 1 (List 1, L1). The reference pictures included in the reference picture list may be arranged based on a picture order count (POC) difference between the current picture and the corresponding reference picture. The POC corresponds to the display order of the pictures and can be distinguished from the coding order. The reference picture list may include the rotated reference pictures.

The decoding apparatus decodes the current picture based on the inter prediction using the reference picture list (S1530). The decoding apparatus may perform inter-prediction on the current block of the current picture to generate a prediction sample. The decoding apparatus may derive motion information on the current block and generate a prediction sample of the current block based on a reference block in the reference picture included in the reference picture list indicated by the motion information.

If there is no residual sample for the current block of the current picture, the decoding apparatus may derive the prediction sample as a reconstructed sample for the current picture, and if the residual sample for the current block of the current picture does not exist, If there is a residual sample, the residual sample may be added to the prediction sample to generate a reconstruction sample for the current picture.

On the other hand, if there is a residual sample for the current picture, although not shown in the figure, the decoding apparatus can receive information on the residual for each quantization processing unit. The information on the residual may include a transform coefficient relating to the residual sample. The decoding apparatus may derive the residual sample (or residual sample array) for the current block based on the residual information. The decoding apparatus may generate a reconstructed sample based on the prediction sample and the residual sample, and may derive a reconstructed block or a reconstructed picture based on the reconstructed sample. Hereinafter, it is described that the decoding apparatus can apply an in-loop filtering procedure such as deblocking filtering and / or SAO procedure to the restored picture in order to improve subjective / objective picture quality as necessary.

In addition, the decoding apparatus can re-project a 360-degree image of the decoded current picture on the 3D space (3D projection structure) based on the projection type. Here, the projection type may be the ERP (Equirectangular Projection), and the 3D space may be a spherical surface.

According to the present invention described above, a rotated reference picture in which a discontinuous boundary of a reference picture for a 360-degree video is reduced can be derived, and a reference picture list can be derived based on the rotated reference picture. Through the prediction accuracy and the overall coding efficiency Can be improved.

In addition, according to the present invention, a reference picture can be derived by rotating a 360-degree image and projecting a rotated 360-degree image, thereby reducing prediction efficiency deterioration due to discontinuity of the projected picture, Can be improved.

In the above-described embodiments, while the methods are described on the basis of a flowchart as a series of steps or blocks, the present invention is not limited to the order of steps, and some steps may occur in different orders or in a different order than the steps described above have. It will also be appreciated by those skilled in the art that the steps depicted in the flowchart are not exclusive and that other steps may be included or that one or more steps in the flowchart may be deleted without affecting the scope of the invention.

The above-described method according to the present invention can be implemented in software, and the encoding apparatus and / or decoding apparatus according to the present invention can perform image processing of, for example, a TV, a computer, a smart phone, a set- Device.

When the embodiments of the present invention are implemented in software, the above-described method may be implemented by a module (a process, a function, and the like) that performs the above-described functions. The module is stored in memory and can be executed by the processor. The memory may be internal or external to the processor and may be coupled to the processor by any of a variety of well known means. The processor may comprise an application-specific integrated circuit (ASIC), other chipset, logic circuitry and / or a data processing device. The memory may include read-only memory (ROM), random access memory (RAM), flash memory, memory cards, storage media, and / or other storage devices.

Claims

A video decoding method performed by a decoding apparatus,

Deriving rotation parameters for the 360 degree image;

Processing the 360 degree video based on the projection type and the rotation parameters for the 360 degree image to obtain a rotated reference picture;

Deriving a reference picture list for the current picture based on the rotated reference picture; And

And decoding the current picture based on inter prediction using the reference picture list,

Wherein the rotated reference picture is a picture projected so that a specific position on the 3D space derived from the rotation parameters is mapped to a center of the rotated reference picture. .
The method according to claim 1,

Wherein the reference picture list includes a reference picture to which the 360-degree picture is projected,

Wherein the reference picture is a picture that is projected so that the 360-degree image is mapped to the center of the reference picture in the 3D space.
The method according to claim 1,

Processing the 360-degree image based on the projection type and the rotation parameters for the 360-degree image to obtain the rotated reference picture,

Deriving the 360-degree image and the rotated 360-degree image based on the rotation parameters; And

And projecting the rotated reference picture by projecting the 360-degree picture onto a 2D picture such that the specific position, which is the center of the rotated 360-degree picture, is mapped to the center of the rotated reference picture based on the projection type And decodes the image data.
The method of claim 3,

The rotation parameters are derived as a specific yaw value, a specific pitch value and a specific roll value,

Wherein the specific position is a position in which the yaw component is the specific yaw value, the pitch component is the specific pitch value, and the roll component is the specific roll value in the 3D space.
5. The method of claim 4,

The projection type is Equirectangular Projection (ERP)

Wherein the specific yaw value is 180, the specific pitch value is 0, and the specific roll value is 90.
5. The method of claim 4,

The projection type is Equirectangular Projection (ERP)

Wherein the specific yaw value is 180, the specific pitch value is 0, and the specific roll value is -90.
5. The method of claim 4,

The projection type is CMP (Cube Map Projection)

Wherein the specific yaw value is 0, the specific pitch value is 0, and the specific roll value is 90.
3. The method of claim 2,

And the reference picture list is arranged in the order of the reference picture and the rotated reference picture.
The method according to claim 1,

And re-projecting the 360-degree image of the decoded current picture on the 3D space based on the projection type.
3. The method of claim 2,

Wherein the rotated reference picture includes a discontinuous boundary smaller than the reference picture.
A video encoding method performed by an encoding apparatus,

Deriving rotation parameters for a 360 degree image;

Processing the 360-degree image based on the projection type and the rotation parameters for the 360-degree image to obtain a rotated reference picture;

Deriving a reference picture list for the current picture based on the rotated reference picture; And

Generating 360 degrees video information for the current picture, encoding and outputting 360 degree video information for the current picture,

Wherein the rotated reference picture is a picture projected so that a specific position on the 3D space derived from the rotation parameters is mapped to a center of the rotated reference picture. .
12. The method of claim 11,

Processing the 360 degree image projected on the reference picture based on the projection type and the rotation parameters for the 360 degree image to obtain the rotated reference picture,

Deriving the 360-degree image and the rotated 360-degree image based on the rotation parameters; And

And projecting the rotated reference picture by projecting the 360-degree picture onto a 2D picture such that the specific position, which is the center of the rotated 360-degree picture, is mapped to the center of the rotated reference picture based on the projection type Wherein the image encoding method comprises the steps of:
13. The method of claim 12,

The rotation parameters are derived as a specific yaw value, a specific pitch value and a specific roll value,

Wherein the specific position is a position in which the yaw component in the 3D space is the specific yaw value, the pitch component is the specific pitch value, and the roll component is the specific roll value.
14. The method of claim 13,

The projection type is Equirectangular Projection (ERP)

Wherein the specific yaw value is 180, the specific pitch value is 0, and the specific roll value is 90.
14. The method of claim 13,

The projection type is CMP (Cube Map Projection)

Wherein the specific yaw value is 0, the specific pitch value is 0, and the specific roll value is 90.