KR20170059902A - Method and apparatus for encoding/dicoding a multi-perspective video - Google Patents

Abstract

The present invention relates to a method and an apparatus for coding and decoding a multi-view video, and more particularly, to a method and an apparatus for generating a reference image of a multi-view video. To this end, the method for coding an image comprises the steps of: converting a second image into the first point of view when a first image having a first point of view and the second image having a different point of view exist; generating a reference image by adding the second image to one surface of the first image; and storing the reference image to a reference picture list. Therefore, the coding and decoding efficiency of the multi-view video can be improved.

Description

TECHNICAL FIELD [0001] The present invention relates to a method and an apparatus for encoding / decoding multi-view video,

The present disclosure relates to an image encoding / decoding method and apparatus, and more particularly, to a method and apparatus for generating a reference image of multi-view video.

Due to technological advances, high-definition / high-definition broadcast services have become common, and interest in UHD (Ultra High Definition) having resolution over 4 times higher than HD (high definition) resolution is increasing.

In accordance with this tendency, interest in realistic images such as stereoscopic images and omnidirectional videos is increasing in addition to existing two-dimensional images. With the development of new media, devices capable of reproducing realistic images such as a head mount display are being marketed extensively, but realistic media are still being encoded / decoded in the same way as two-dimensional images. Accordingly, in order to reconsider the compression efficiency of the real-sensible media, a compression method suited to the characteristics of real-sensible media should be developed.

SUMMARY OF THE INVENTION The present invention is directed to a method and apparatus for improving the coding / decoding efficiency of multi-view video.

In particular, the technical object of the present invention is to provide a method and an apparatus for performing encoding / decoding using a reference picture in which unified viewpoints are merged into one viewpoint, and then the unified viewpoints are combined.

The technical objects to be achieved by the present disclosure are not limited to the above-mentioned technical subjects, and other technical subjects which are not mentioned are to be clearly understood from the following description to those skilled in the art It will be possible.

According to an aspect of the present disclosure, there is provided a method for converting a first image having a first viewpoint into a first viewpoint when a first image having a first viewpoint and a second image having a different viewpoint exist, Generating a reference image by adding the second image, and storing the reference image in a reference picture list.

According to an aspect of the present disclosure, there is provided a method for converting a first image having a first viewpoint into a first viewpoint when a first image having a first viewpoint and a second image having a different viewpoint exist, Generating a reference image by adding the second image, and storing the reference image in a reference picture list.

The following aspects may be commonly applied to the image encoding method and the image decoding method.

Wherein the transformed image is generated based on a viewpoint difference between the first image and the second image, and the viewpoint difference may include at least one of a distance difference or an angle difference between the first image and the second image have.

The viewpoint difference may be determined based on characteristics of an omnidirectional image including the first image and the second image.

When the omnidirectional image is projected onto the regular polyhedron, the angle difference between the first image and the second image may be determined as the interior angle of the regular polyhedron.

The viewpoint difference may be obtained based on a distance between the first camera used to photograph the first image and a second camera used to photograph the second image and an angle difference.

The position to which the second image is to be added may be determined according to characteristics of the regular polyhedron including the first image and the second image.

The reference picture may be stored in the reference picture list together with the time zone information corresponding to the first video and the second video.

The features briefly summarized above for this disclosure are only exemplary aspects of the detailed description of the disclosure which follow, and are not intended to limit the scope of the disclosure.

According to the present disclosure, a method and apparatus capable of improving the coding / decoding efficiency of multi-view video can be provided.

Specifically, according to the present disclosure, a method and apparatus for performing encoding / decoding using a reference picture in which images with different viewpoints are unified into one viewpoint and then combined with unified viewpoints can be provided.

The effects obtainable from the present disclosure are not limited to the effects mentioned above, and other effects not mentioned can be clearly understood by those skilled in the art from the description below will be.

1 is a diagram illustrating cameras for generating omnidirectional video.
2 is a diagram showing an example in which omnidirectional video is developed in two dimensions.
3 is a diagram illustrating a polyhedron developed view in which omni-directional video can be developed.
4 is a diagram illustrating a polyhedron on which omnidirectional video can be projected.
FIG. 5 is a diagram illustrating an example in which an image at a specific time point during a specific time period is displayed while omnidirectional video is being reproduced.
6 is a diagram for explaining a temporal prediction method.
7 is a diagram showing an example in which nonlinear motion occurs in forward video.
8 is a diagram for explaining an example of generating a reference image by transforming the perspective of the neighboring region.
9 is a flowchart illustrating a process of generating a transformed extension image according to the present invention.
10 is a view illustrating perspective differences of omniazimetric images projected in a polyhedron.
11 is a diagram for explaining an example for calculating a perspective difference between images.
12 is a diagram for explaining an example in which an image is converted in accordance with the perspective of another image.
13 and 14 are diagrams illustrating an example of generating a reference image based on a specific surface of a cube.
15 is a diagram illustrating a diversity image.
16 is a diagram illustrating an example of generating a reference image for a diversity image.
FIG. 17 is a diagram showing an example of generating a reference image by combining a reference image and a transformed image.
18 is a diagram illustrating a convergent image.
19 is a diagram illustrating whether or not a transformed image includes an area essential for temporal prediction.
20 is a diagram illustrating an example of generating a reference image for a convergent image.
FIG. 21 is a diagram showing an example of generating a reference image by combining a perspective image and a transformed image.
22 is a view showing an example in which a reference image list for an omnidirectional image developed in the form of a cube is generated.
23 is a diagram showing an example in which a reference image list for a diversity image is generated.
24 is a diagram showing an example in which a reference image list for a convergent image is generated.
25 is a diagram for comparing a case where a reference image is generated according to the present invention and a case where a reference image is not generated according to the present invention.
26 is a diagram illustrating an encoding aspect according to whether the present invention is applied or not.
27 is a block diagram showing a configuration of an encoder according to the present invention.
28 is a block diagram showing a configuration of a decoder according to the present invention.
29 is a block diagram of a reference image expansion unit according to the present invention.

While the invention is susceptible to various modifications and alternative forms, specific embodiments thereof are shown by way of example in the drawings and will herein be described in detail. It should be understood, however, that the invention is not intended to be limited to the particular embodiments, but includes all modifications, equivalents, and alternatives falling within the spirit and scope of the invention. In the drawings, like reference numerals refer to the same or similar functions throughout the several views. The shape and size of the elements in the figures may be exaggerated for clarity. The following detailed description of exemplary embodiments refers to the accompanying drawings, which illustrate, by way of illustration, specific embodiments. These embodiments are described in sufficient detail to enable those skilled in the art to practice the embodiments. It should be understood that the various embodiments are different, but need not be mutually exclusive. For example, certain features, structures, and characteristics described herein may be implemented in other embodiments without departing from the spirit and scope of the invention in connection with an embodiment. It is also to be understood that the location or arrangement of the individual components within each disclosed embodiment may be varied without departing from the spirit and scope of the embodiments. The following detailed description is, therefore, not to be taken in a limiting sense, and the scope of the exemplary embodiments is to be limited only by the appended claims, along with the full scope of equivalents to which such claims are entitled, if properly explained.

The terms first, second, etc. in the present invention may be used to describe various components, but the components should not be limited by the terms. The terms are used only for the purpose of distinguishing one component from another. For example, without departing from the scope of the present invention, the first component may be referred to as a second component, and similarly, the second component may also be referred to as a first component. And / or < / RTI > includes any combination of a plurality of related listed items or any of a plurality of related listed items.

Whenever an element of the invention is referred to as being "connected" or "connected" to another element, it may be directly connected or connected to the other element, but other elements may be present in between It should be understood. On the other hand, when an element is referred to as being "directly connected" or "directly connected" to another element, it should be understood that there are no other elements in between.

The components shown in the embodiments of the present invention are shown separately to represent different characteristic functions and do not mean that each component is composed of separate hardware or software constituent units. That is, each constituent unit is included in each constituent unit for convenience of explanation, and at least two constituent units of the constituent units may be combined to form one constituent unit, or one constituent unit may be divided into a plurality of constituent units to perform a function. The integrated embodiments and separate embodiments of the components are also included within the scope of the present invention, unless they depart from the essence of the present invention.

The terminology used herein is for the purpose of describing particular embodiments only and is not intended to be limiting of the invention. The singular expressions include plural expressions unless the context clearly dictates otherwise. In the present invention, the term "comprises" or "having ", etc. is intended to specify that there is a feature, number, step, operation, element, But do not preclude the presence or addition of one or more other features, integers, steps, operations, elements, components, or combinations thereof. In other words, the description of "including" a specific configuration in the present invention does not exclude a configuration other than the configuration, and means that additional configurations can be included in the practice of the present invention or the technical scope of the present invention.

Some of the elements of the present invention are not essential elements that perform essential functions in the present invention, but may be optional elements only for improving performance. The present invention can be implemented only with components essential for realizing the essence of the present invention, except for the components used for the performance improvement, and can be implemented by only including the essential components except the optional components used for performance improvement Are also included in the scope of the present invention.

Hereinafter, embodiments of the present invention will be described in detail with reference to the drawings. In the following description of the embodiments of the present invention, a detailed description of known functions and configurations incorporated herein has been omitted for the sake of clarity and conciseness. And redundant descriptions are omitted for the same components.

Conventional two-dimensional video is played back at a fixed point in time, while omnidirectional video (or 360-degree video) can be played back at a desired point in time. To generate omnidirectional video, various types of cameras shown in Fig. 1 may be used. A plurality of cameras shown in Fig. 1 (a), a fisheye lens shown in Fig. 1 (b), or a reflector shown in Fig. 1 (c) Dimensional space (for example, in a 360-degree direction), and project the captured three-dimensional image in the form of a three-dimensional figure such as a polyhedron or a sphere. A two-dimensional image developed on the two-dimensional space can be obtained from the image projected on the three-dimensional figure. For example, FIGS. 2A and 2B show an example in which a two-dimensional image developed in the form of a developed view of a cube is obtained from a three-dimensional image in the form of a sphere.

2, the omnidirectional image can be projected in the form of an expanded view of various polygons. As an example, as in the example shown in Fig. 3, the omnidirectional image can be projected in the form of an expanded view of various stereoscopic shapes such as a tetrahedron, a cube, an octahedron, a dodecahedron, or a regular dodecahedron.

In the following embodiments, 'image' is assumed to mean a certain area of some frame or any frame of 3D video. For example, when an arbitrary frame of the forward video is developed in the form of a cube, as in the example shown in FIG. 2B, the frame may also be referred to as an 'image' (I.e., one side of a cube) may also be referred to as an " image ".

When the video is reproduced, the two-dimensional image can be reconstructed again into a three-dimensional space such as a polyhedron or a sphere. In FIG. 4, an example is shown in which a two-dimensional image is reconstructed into a three-dimensional space in the form of a regular tetrahedron, a cube, an octahedron, a regular tetrahedron, or a regular hexahedron. At this time, a specific point viewed by the user or a region corresponding to a specific point selected by the user may be displayed in a specific time among the entire images projected in three dimensions.

FIG. 5 is a diagram illustrating an example in which an image at a specific time point during a specific time period is displayed while omnidirectional video is being reproduced. 5 (a) and 5 (b) illustrate a concept diagram when viewing an omnidirectional image from the outside, and Fig. 5 (c) to be.

5 (a) and 5 (c) show an example in which a two-dimensional image is reconstructed into a cube shape, and FIG. 5 (b) shows an example in which a two-dimensional image is reconstructed into a sphere shape.

At the time of video reproduction, a part of the entire region of the reconstructed image that is reconstructed in three dimensions can be reproduced at a user's point of view or selected by the user. For example, the hatched area in FIGS. 5A and 5B represents a portion of the reconstructed image in three dimensions, which is directed toward the user viewpoint or selected by the user. In this case, a portion of the omnidirectional image facing the user's viewpoint or a region corresponding to the portion selected by the user may be output. For example, when the user is viewed as a reference, the user can view only a part of the entire area of the omnidirectional image, as in the example shown in Fig. 5 (c).

As described above, since the omnidirectional image is projected in the two-dimensional space and then reconstructed in the three-dimensional form, the encoding / decoding of the omnidirectional video can be performed in the same manner as the encoding / decoding of the 2D video. For example, a temporal prediction (Inter Prediction) method used for encoding / decoding 2D video can be used for encoding / decoding of an omnidirectional image. Here, the temporal prediction method refers to a method of predicting a current image from a reference image based on a high correlation between images having close temporal differences.

For example, FIG. 6 is a diagram for explaining a temporal prediction method.

6A shows a reference frame used to predict a current image, and FIG. 6B shows a current frame including a current block to be encoded / decoded. If the block 'A' contained in the reference image can be used in predicting the current block B 'to be coded / decoded, it is not necessary to encode all the image information for the block B of the current image. Accordingly, when temporal prediction is used, the amount of information to be encoded for each image block can be greatly reduced, and the efficiency of image compression can be increased.

However, in the case of omnidirectional video, since there are a plurality of perspectives in a single image, a nonlinear change is detected at the boundary of each perspective, which may cause a problem of a decrease in temporal prediction efficiency.

For example, FIG. 7 shows an example in which nonlinear motion occurs in forward video.

Assuming that a single image of the omnidirectional video is projected onto the developed view of the cube, the image has six planes P1 to P6, as in the example shown in Fig. 7 (a). In addition, each face has a different perspective.

The shape of the object projected on the image may also vary depending on the perspective. Thus, nonlinear motion can occur at the boundaries of two surfaces having different perspectives. For example, in FIG. 7 (b), nonlinear motion occurs on the P3 surface, the P4 surface, the P4 surface, the P1 surface, and the like that are located adjacent to the boundary line indicated by the dotted line.

Accordingly, if temporal prediction of a block included in a specific plane is performed using a block included in a plane different from the specific plane in the reference image, temporal prediction using a block included in the same plane as the specific plane in the reference image is performed The encoding efficiency will be greatly reduced as compared with the performance.

As described above, if it is not suitable to predict a current image using a temporal reference image, intra-block coding or intra prediction (or spatial prediction) may be used. Instead of temporal prediction, spatial prediction is used Accordingly, there arises a problem that the compression efficiency is lowered.

In the case of general 2D video, when the number of frames per second is satisfied, motion of an object moving linearly in an image can be efficiently encoded and compressed through inter-frame prediction. However, in the case of omnidirectional video, the perspective changes according to the position of the intra-frame area as in the above-described example, so that the linear motion in the real world is distorted non-linearly.

Accordingly, in order to solve the problem of temporal prediction efficiency deterioration of boundary blocks adjacent to the boundary, the present invention linearly corrects nonlinear motion appearing at a boundary where a perspective changes, . For example, in FIG. 7 (c), it is shown that the nonlinear distortion at the boundary between the P1 and P4 planes and the nonlinear distortion at the boundary between P3 and P4 are corrected linearly.

As described above, the present invention is applicable to a moving picture having various perspectives such as an omnidirectional video, a case in which there are one or more successive perspectives on positions in an image in which a plurality of perspectives are combined, The neighboring region (or neighboring image) of the perspective is transformed to improve the prediction efficiency.

Hereinafter, the present invention will be described focusing on an omnidirectional video coding method, but the present invention can be applied to various kinds of realistic media other than omnidirectional video. The realistic media may include divergent video and convergent video as well as omnidirectional video. Divergent video refers to an image generated by multiple cameras capturing images in various directions. Convergent video refers to an image generated by multiple cameras capturing images in specific directions. In addition, although the description will be made mainly on the encoding process of omnidirectional video, it will be apparent that the embodiments described below can be performed in the same or reverse order in the decoding process.

8 is a diagram for explaining an example of generating a reference image by transforming the perspective of the neighboring region.

When there is a continuous perspective on the position, the efficiency of the temporal prediction may deteriorate at the boundary portion where the intra-picture perspective changes. This is because the reference area in the reference image used for the prediction does not include the information of the neighboring area. Even if the reference area includes information on the neighboring area, the current block and the perspective are different, and it is not suitable for use as a reference image.

For example, in the example shown in FIG. 8, when only a region C1 that does not include the information of the neighboring region is used to restore a portion C1 of the current image, the current image C1 is completely restored It is difficult to do. Even if the area including the information of the neighboring area (i.e., the sum of P1 and P2) is used to restore the current image, the perspective between the current image and the newly extended P2 area is different, It is difficult to restore.

Thus, the reference area and the neighboring area having different perspectives are transformed, and then the reference area and the transformed neighboring area are combined to generate a reference image including the transformed extended area, thereby improving coding efficiency in temporal prediction.

Specifically, as in the example shown in Fig. 8A, the image may be distorted (i.e., skewed) by the perspective change at the interface between the reference area and the neighboring area. However, as in the example shown in Fig. 8 (b), when the neighboring area is converted to match the perspective of the reference area, the distortion at the boundary is reduced. Thus, as in the example shown in FIG. 8B, after generating the image TR in which the reference region and the neighboring region having different perspectives are transformed, the reference region P1 and the transformed neighboring region TR are generated, The prediction accuracy can be improved when the current image C1 is predicted using the transformed extension image (R obtained by adding P1 and TR).

In order to obtain the transformed extension image, the following steps can be largely performed.

Step 1. Calculate direction and distance relation between images

Step 2. Generate transformed image in each perspective using relational expression between images

Step 3. The transformed image is extended to the reference image according to the perspective of the existing reference image

Step 4. Insert transformed image into reference image list

In the first step, the feature of the image (for example, the intrinsic property of the cube image) or the additional data other than the image data (for example, in the case of the divergent image, the angle of view of each image, the angle difference between perspectives, In the step of calculating the direction and the distance relation between images, the direction and the distance relation between images are illustrated as? And d in (a) of Fig.

The second step is to generate a transformed image of the neighboring image based on the direction and distance difference between images calculated in the first step. In this step, the neighboring image, which is the extended region of the reference image, can be modified to match the perspective of the reference image. Here, the modification of the neighboring image in accordance with the perspective of the reference image means that the neighboring image is transformed to be in the same perspective space as the reference image.

The third step is to generate a transformed extension image based on the transformed image and the reference image. Specifically, the transformed extension image can be generated by adding the transformed image and the reference image so that the transformed image is continuous with the reference image.

Step 4 is a step of storing the generated transformed extension image in the reference image list. By storing the transformed extension image in the reference image list, it can be used for temporal prediction of the decoded / encoded image.

By combining the transformed image and the reference image as described above to generate an image suitable for temporal prediction, the encoding / decoding efficiency of the forward video can be improved. The above-described steps will be described in more detail through the flow chart of FIG.

9 is a flowchart illustrating a process of generating a transformed extension image according to the present invention.

Referring to FIG. 9, it can be determined whether the perspective images are spatially continuous (S910). Whether or not the perspectives of other images are spatially contiguous can be determined by comparing the characteristics of the data (e.g., metadata) added to the image or the multi-view image (for example, the developed view of the polyhedron has a fixed number of perspectives) . For example, when an image is projected onto a polyhedral development view, it can be judged that the perspective images are spatially continuous.

If the perspectives of other images exist spatially consecutively, Perspective can obtain different perspective differences between images (S920). Here, the perspective difference may include at least one of an angular difference and a position difference. The angle means an Euler angle or a subset thereof for displaying in a three-dimensional space, and the position means a three-dimensional space position coordinate or a subset thereof.

In the case of omni-directional images projected in a polyhedron, the angular difference between the different perspectives has a fixed value. In addition, in the case of omni-directional images projected in a polyhedron, images with different perspectives are continuous at the boundary, so that there is no difference in distance.

For example, FIG. 10 is a diagram illustrating the perspective difference of the omniazimetric image projected in a polyhedron. In the example shown in FIG. 10A, when the omnidirectional image is projected on the cube, the angular difference between the two images P1 and P2 is fixed at 90 degrees, and the two images P1 and P2 are based on the boundary The successive bars, the distance difference of the two images, can be derived as zero.

10 (b), when the omnidirectional image is projected on the dodice, the angular difference between the two images P3 and P4 is fixed at 90 degrees, and the two images P3 and P4 are The distance difference between the two images can be derived as 0, which is continuous with respect to the boundary.

10 (c), when the omni-directional image is projected on the sphere, the angular difference between the two images P5 and P6 can not be calculated by the sphere characteristic. The angle difference 3 between the two images can be calculated by the inter-camera angle difference described later with reference to FIG. 11 (a). However, since the two images P5 and P6 exist continuously, the distance difference between the two images can be derived to be zero.

As described above, when the omni-directional image is projected as a non-polyhedral object such as a sphere, or when there is no rule in the arrangement of each image, the perspective difference between images can be calculated through the positions of the cameras, the viewing angles of the cameras, have.

For example, FIG. 11 is a diagram for explaining an example for calculating a perspective difference between images.

The perspective difference (i.e., the position difference d2 and the angular difference? 4) of the two images is determined by the position of the cameras C1 and C2 and the position of the cameras C1 and C2 when the omnidirectional image is projected as a non- Can be calculated based on at least one of the position difference d1 between the cameras, the angle difference? 3 between the cameras, and the viewing angles? 1 and? 2 of each camera. 11 (b) (x, y, z), and the angle refers to an Euler angle for displaying in the three-dimensional space (for example, (A, b, r)) of the image data. The above information may be encoded or signaled in the form of metadata added to each image.

When perspective differences among different perspectives are calculated, images having different perspectives can be converted to the same perspective based on the calculated perspective differences (S930). As an example, for two different perspective images, a process of converting either of the two images to a different perspective of the other image may be performed.

12 is a diagram for explaining an example in which an image is converted in accordance with the perspective of another image.

FIG. 12A is a view showing an example in which an omnidirectional image is developed into a cube, FIG. 12B is a view showing an example of a cube obtained by looking at a cube from an upper end (for example, P1) This is a plan view when viewed.

Assuming that the omni-directional image is developed into a cube as in the example shown in Fig. 12A, the region included in the P4 plane of the current image is predicted through the reference region included in the P4 plane of the reference image Lt; / RTI > However, when the region included in the P4 plane of the current image is to be predicted using the P3 plane of the reference image, the perspective of the P4 plane and the P3 plane may be different from each other, and the efficiency of prediction may be inferior. Accordingly, when storing the reference image, it is necessary to perform a process of converting the P3 plane to the perspective of the P4 plane.

For example, in the example shown in FIG. 12 (b), by projecting the position x included in the plane P3 to the position y having a perspective such as P4, the plane P3 can be converted into the perspective of the plane P4. At this time, the perspective difference between the P3 plane and the P4 plane may be briefly simplified by the following equation (1).

In Equation (1), a represents the length of one side of the cube. The perspective between the P3 plane of the reference image and the P4 plane of the current image can be unified when the P4 plane of the current image is predicted using the reference image transformed from the P3 plane to the perspective of the P4 plane, .

If another image is spatially neighboring, the neighboring image may be further transformed according to the above principle (S940).

When the neighboring images are transformed based on the perspective of the specific image, the reference image can be generated by combining the specific image and the transformed neighboring images (S950).

For example, when the omnidirectional image has a cube shape, a plurality of neighboring faces neighboring the specific face of the cube are converted to fit the perspective of the specific face, and a reference image is generated by combining the specific face and the plurality of converted neighboring faces .

For example, FIGS. 13 and 14 illustrate examples of generating a reference image based on a specific surface of a cube. Referring to FIG. 13A, the P4 plane is adjacent to the P1, P3, P5, and P6 planes with respect to the P4 plane. Accordingly, P1, P3, P5 and P6 planes are transformed in accordance with the perspective of the P4 plane to generate T1, T3, T5 and T6, and the P4 plane and the generated transformed images T1, T3, T5 and T6 are summed A reference image can be generated.

At this time, the entire area of the neighboring plane neighboring the P4 plane may be set as a conversion target, but only a certain area (e.g., a search range) set by the encoder may be set as a conversion target.

13B is a diagram showing an example in which a reference image is generated when the entire area of the neighboring area is to be transformed. FIG. 13C shows a case where a reference image is generated when a partial area of the neighboring area is to be transformed Fig.

The entire or a part of the converted image is determined in accordance with the image which is the reference of the perspective. In the case of an image projected in a polyhedron, the position of the converted image is determined according to the polyhedron characteristic. The converted image can be combined with the reference image of the perspective according to its position.

For example, in the example shown in FIGS. 14A and 14B, the positions of the images T1, T3, T5, and T6 are determined relative to the image P4 as the reference of the perspective do. That is, each of the transformed images T1, T3, T5, and T6 is combined with P4 at the projected position in accordance with the perspective of P4.

FIGS. 14C and 14D are views illustrating a reference image R4 generated by combining P4 and transformed images T1, T3, T5, and T6. 14C is a diagram illustrating a case where the entire convertible area of the neighboring image is projected in accordance with the perspective of P4, and FIG. 14D is a diagram illustrating a case in which the entire area of the neighboring image is projected in accordance with the perspective of P4 Fig.

Divergent images are images taken when a shooting direction of a plurality of cameras is spread. Accordingly, the diversity image may include a plurality of images having different perspectives. For example, FIG. 15 is a diagram illustrating a diversity image. In the example shown in Fig. 15, the camera C1 photographs the first direction W1, the camera C2 photographs the second direction W2, and the camera C3 photographs the third direction W3. Thus, a diversity image including P1, P2, and P3 is illustrated as being generated.

Like the polyhedron, the divergent image can be transformed based on the angles and positional differences between images. However, unlike a polyhedron, divergent images do not have a certain rule for the arrangement of the respective images. Therefore, it is impossible to know the angular difference and the difference in distance between images having different perspectives only by the characteristics of the image. Thus, the perspective difference of the diversity image may be encoded / signaled through additional data (e.g., metadata) added to the image.

After obtaining the perspective difference between images, a reference image can be generated in the same manner as the above-described polyhedron.

For example, FIG. 16 shows an example of generating a reference image for a diversity image. Referring to FIG. 16 (a), it is shown that a P2 image having a different perspective from the P1 image exists. In the case of generating a reference image with reference to P1, it is possible to convert the entire region or a partial region of P2 to match the perspective of P1, and add the converted image to P1 to generate a reference image. 16B shows an example in which a reference image is generated by combining T2 and P1 generated by converting the entire area of P2, and FIG. 16C shows an example in which a partial area of P2 (for example, (search range)), and a reference image is generated by combining T'2 and P1.

In the case of the diversity image, the position of the transformed image can be determined with respect to the image serving as the reference of the perspective. At this time, the position of the transformed image can be determined based on the position information of each image. When the position of the transformed image is determined, a reference image can be generated by adding the transformed image and the reference image.

For example, FIG. 17 is a diagram illustrating an example of generating a reference image by combining a reference image and a transformed image. In the example shown in Fig. 17, T2 generated by converting the P2 image is illustrated as being located on the right side of the image P1 as the reference of the diversity. Thus, by generating at least a part of the transformed image T2 on the right side of the P1 image, the reference image R1 can be generated.

The convergent image refers to an image that is captured when the photographing directions of a plurality of cameras converge toward one place. The convergent image may also include a plurality of images having different perspectives. For example, FIG. 18 is a diagram illustrating a converter image. In the example shown in Fig. 18, the photographing directions W1, W2, W3 and W4 of the camera, C1, C2, C3 and C4 are illustrated as facing one point. As a result, a convergent image including P1, P2, P3, and P4 is generated.

Convergent images can also be 2D converted based on angle and position differences between images, as in polyhedrons. However, unlike the polyhedron, the convergent image does not have a certain rule for the arrangement of the respective images, so that the difference in distance between the images and the difference in distance between different perspectives can not be known only by the characteristics of the image. Thus, the perspective difference of the converged image may be encoded / signaled through additional data (e.g., metadata) to which the image is appended.

Unlike a polyhedral or divergent image, the viewpoint of a convergent image is oriented in a specific direction. Accordingly, the convergent image may include a plurality of images extended in the same direction while the positions are continuous. Accordingly, when generating a reference image of a convergent image, there are several neighboring images that can be converted to the same position. In this case, a reference image can be generated based on the transformed image generated by transforming at least a part of the neighboring images existing in the same position in the specific image and the specific image.

Unlike a polyhedral or divergent image, when the positional difference between images having different perspectives is not large, such as a convergent image, the generated converted image may have the following pattern.

The first aspect is that a neighboring image converted to a perspective of a specific image includes a region that does not overlap with a specific image. In other words, an extended area added to a specific area (i.e., a part of the converted neighboring image that does not overlap with a specific image) includes essential information (for example, an area necessary for temporal prediction). In this case, the same method as that for generating the reference image in the polyhedron or divergent image may be applied.

The second aspect is that the neighboring image converted to the perspective of the specific image is mostly overlapped with the specific image. In other words, an extended area added to a specific area does not include sufficient additional information (for example, an area required for temporal prediction). It is difficult to use a reference image for temporal prediction if the extended area added to the specific area does not include sufficient additional information.

Referring to Fig. 19, each aspect will be described in more detail.

19 is a diagram illustrating whether or not a transformed image includes an area essential for temporal prediction.

For convenience of explanation, it is assumed that the convergent image includes a plurality of images P1, P2, P3 ... PN, as in the example shown in Fig. 19 (a). FIG. 19B is a diagram showing an example of converting a P2 image neighboring the P1 image to a perspective of the P1 image. As in the example shown in FIG. 19 (b), a reference image for the P1 image can be generated by converting at least one image neighboring the P1 image to a perspective of the P1 image. At this time, the converted neighboring images may completely include the required area or not completely include the required area according to the degree of overlap with the P1 image. For example, in FIG. 19C, the image T2 converted from the P2 image includes an area smaller than a required area (for example, a search range) . ≪ / RTI >

As in the case of the T2 image shown in (c) of FIG. 19, if the transformed image does not include the minimum area required, it may be difficult to use it for temporal prediction. In this case, the reference image can be generated by padding the insufficient area to perform the temporal prediction using the pixel value of the transformed neighboring area. That is, an area that can not be obtained from the conversion of the neighboring image among the areas that are necessarily included for temporal prediction can be padded using the edge sample of the converted neighboring area.

20 is a diagram illustrating an example of generating a reference image for a convergent image. As described with reference to the transformed image T2 shown in FIG. 19 (c), when the residual region excluding the portion overlapping P1 with respect to the P2 image is not sufficiently wide, the image T2 obtained by transforming P2 is also used for temporal prediction It may not include the required area. In this case, as in the example shown in Figs. 20A and 20B, an edge sample of the T2 image can be used to pad an area necessary for prediction.

FIG. 20A shows an example of padding the remaining area using the right edge sample when the P2 image is converted to the right and FIG. 20B shows an example of padding the remaining area using the right edge sample. And the remaining area is padded using the sample.

In FIG. 20, a convergent image has been described as an example. However, the present invention can be applied not only to a convergent image but also to an omnidirectional image or a diversity image if the converted image does not sufficiently include a region necessary for prediction use.

In the case of the convergent image, the position of the transformed image can be determined centering on the image that is the reference of the perspective. At this time, the position of the transformed image can be determined based on the position information of each image. In the case of the convergent image, since a plurality of transformed images extending to the same position can be obtained, at least one reference image can be generated by combining the transformed image and the reference image.

For example, FIG. 21 is a diagram showing an example of generating a reference image by combining a reference image and a transformed image as perspectives. In the example shown in FIG. 21, a reference image R1 is generated by combining T2 and P1 images generated by transforming a P2 image, and a reference image R2 can be generated by combining T3 and P1 images generated by transforming a P3 image. In this manner, N-1 transformed images can be generated for N transformed images based on the P1 image.

When the reference image is generated, the reference image may be stored in the reference image list (S960). If there is no other video in the spatially continuous perspective (S910), the current image can be stored as a reference picture in the reference picture list (S960).

When the generated reference images are stored in the reference image list, the reference images can be grouped and stored based on the same time period.

22 is a view showing an example in which a reference image list for an omnidirectional image developed in the form of a cube is generated.

Assuming that the omni-directional image is developed as a cube, one reference image can be generated for the reference image of the perspective of the specific time period. Since there are a total of six images that can be the reference of a perspective in a specific time period, a maximum of six reference images can be generated in a specific time period. Accordingly, a plurality of reference images at a specific time can be grouped and stored.

For example, with respect to the omni-directional image in the time t0, a total of six reference images (based on the perspective of the reference images R2, ..., P6 generated based on the perspectives of the reference images R1, P2 generated based on the perspective of P1) The generated reference image R6) can be generated. The reference images generated based on the time slot t0 may be grouped into one group and stored. Likewise, reference image lists such as t1, t2, ..., tN can be grouped based on a predetermined time period and stored in a list.

23 is a diagram showing an example in which a reference image list for a diversity image is generated.

In the case of a diversity image, one reference image can be generated for an image that is a reference of a perspective of a specific time period. Since the number of images to be the reference of the perspective is determined by the number of cameras that capture the diversity image, a reference image can be generated by the number of cameras in a specific time period. A plurality of reference images in a specific time zone of the diversity image can be grouped and stored, as in the case of an omni-directional image developed in a polyhedron.

For example, for a diversity image of the time t0, a total of three reference images (generated from the perspective of the reference images R2 and P3 generated based on the perspectives of the reference images R1 and P2 generated based on the perspective of the reference images P1 If the reference image R3 is generated, the three reference images generated based on the time t0 may be grouped into one group and stored. Likewise, reference image lists such as t1, t2, ..., tN can be grouped based on a predetermined time period and stored in a list.

24 is a diagram showing an example in which a reference image list for a convergent image is generated.

In the case of a convergent image, at least one reference image may be generated for an image that is a reference of a perspective of a specific time period. For example, when the reference image is a P1 image, the first reference image R1 may be generated based on the P1 image and the P2 image, and the second reference image may be generated based on the P1 image and the P3 image. The reference image of the convergent image can also be grouped and stored as a plurality of reference images generated in a specific time zone, like the omnidirectional image.

For example, for a convergent image at time t0, N reference images generated based on the perspective of P1, N reference images generated based on the perspective of P2, ..., N generated based on the perspectives of PN Reference images may be present. In this case, a plurality of reference images generated based on the time slot t0 may be grouped into one group and stored. Likewise, reference image lists such as t1, t2, ..., tN can be grouped based on a predetermined time period and stored in a list.

22 to 24 illustrate that a plurality of reference images are grouped and stored based on time zones. Unlike the illustrated example, a plurality of reference images may be grouped and stored based on an image serving as a perspective reference.

To select a reference image, information for selecting a reference image may be encoded / signaled. At this time, the information for selecting a reference image may include at least one of information about a time when the reference image is included or information for identifying at least one of a plurality of reference images included in the corresponding time zone.

Taking FIG. 22 as an example, if the information for selecting a reference image indicates a reference image generated based on the perspective of P1 at time t0, a corresponding reference image in the corresponding time zone may be used for temporal prediction.

As another example, the selection of a plurality of reference images included in a specific time zone can be selected based on the position of an area to be currently encoded / decoded. For example, if the current region to be encoded / decoded is included in the P6 plane of the cube, the encoder and decoder can use the reference image generated based on the perspective of P6 for temporal prediction.

As described above, when the perspective between spatially neighboring images is different, a distortion may occur in which the shape of the projected object varies depending on perspectives based on the boundary of the image. In order to reduce coding / decoding efficiency due to distortion at the boundary between images, a process of converting a neighboring image (or region) in accordance with a perspective of an arbitrary image (or region) By doing so, the encoding / decoding efficiency can be improved.

25 is a diagram for comparing a case where a reference image is generated according to the present invention and a case where a reference image is not generated according to the present invention.

If temporal prediction is used without transforming the neighboring image as in the example shown in Fig. 25A, distortion due to the perspective difference occurs at the boundary of the plane. In Fig. 25 (a), nonlinear distortion occurs at the boundary of the P6 plane. Accordingly, when temporal prediction is to be performed based on a boundary portion of a specific image, temporal prediction efficiency may be reduced due to differences in perspectives.

However, as in the example shown in FIG. 25 (b), if a neighboring image is transformed to generate a reference image and temporal prediction is used based on the generated reference image, the distortion at the boundary of the plane is significantly reduced . In the example shown in FIG. 25 (b), nonlinear changes existing on the boundary of the P6 plane are linearly transformed. Accordingly, even if the temporal prediction is performed based on the boundary of a specific image, the temporal prediction efficiency can be improved due to less distortion due to the perspective difference.

As described above, when the temporal prediction is performed without converting the neighboring image, it is difficult to perform temporal prediction based on the block located at the boundary of the image when encoding / decoding the image. Accordingly, in a state in which the neighboring image is not transformed, boundary blocks adjacent to the boundary where the perspective changes are generally encoded through spatial prediction rather than temporal prediction.

However, as described in the present invention, when temporal prediction is performed based on a reference image generated by transforming a neighboring image, temporal prediction can be performed based on a block located at a boundary of the image during encoding / decoding of the image do. Accordingly, the boundary blocks adjacent to the boundary in which the perspective changes can also be encoded / decoded through temporal prediction, so that the image compression efficiency can be enhanced.

For example, FIG. 26 is a diagram illustrating an encoding aspect according to whether or not the present invention is applied. As in the example shown in Fig. 26 (a), in a state where the present invention is not applied, blocks located at the boundary where the perspective changes are generally encoded through intraprediction. However, as in the example shown in Fig. 26 (b), if the present invention is applied, blocks located at the boundary where the perspective changes can also be coded by temporal prediction.

According to the present invention, whether or not to expand an image to be used for prediction can be encoded with encoding parameters and signaled by a bitstream. As an example, whether to expand the image to be used for prediction can be encoded and signaled by a 1-bit flag. If the flag indicates that the image to be used for prediction is to be extended, a method of generating a reference image by converting a neighboring image according to a perspective of a specific image and then summing a specific image and a neighboring image may be applied. On the other hand, if the flag does not indicate that the image to be used for prediction is to be expanded, a perspective-based conversion and a process of expanding a specific image, etc., will not be performed.

At this time, whether or not to expand the image to be used for prediction can be signaled by a parameter set, a picture unit, a slice unit, or a unit to be coded. Table 1 shows an example in which information indicating whether or not to expand an image to be used for prediction is signaled through the VPS, and Table 2 shows an example in which the information is signaled through the SPS.

seq_parameter_set_rbsp () { Descriptor vps_video_parameter_set_id u (4) vps_reserved_three_2bits u (2) vps_max_layers_minus1 u (6) ... perspective_reference_picture_enabled_flag u (1) ...

seq_parameter_set_rbsp () { Descriptor sps_video_parameter_set_id u (4) sps_max_sub_layers_minus1 u (3) sps_temporal_id_nesting_flag u (1) ... perspective_reference_picture_enabled_flag u (1) ...

In Table 1 and Table 2, 'perspective_reference_picture_enabled_flag' indicates whether to expand the image to be used for prediction. The perspective_reference_picture_enabled_flag 'is set to' 1 'and the perspective_reference_picture_enabled_flag is set to' 0 'when the image to be used for the image is not to be expanded, as suggested in the present invention. Can be set. Alternatively, whether to expand the image to be used for the image may be set to a value opposite to the example described above.

When 'perspective_reference_picture_enabled_flag' is '1', an extended reference image can be generated considering the direction and position of the image when constructing the reference image. In addition, prediction can be performed based on the extended reference image.

The configuration of the encoder and the decoder according to the present invention will be described in detail with reference to FIG. 27 and FIG.

27 is a block diagram showing a configuration of an encoder according to the present invention.

The encoder means a device for encoding the developed view of the omnidirectional image, the converged camera image, and the divergent image. The encoder includes a projection unit 100, an inter picture prediction unit 110, an intra prediction unit 120, a transform unit 130, a quantization unit 140, an entropy coding unit 150, an inverse quantization unit 160, An inverse transform unit 170, a reference image extending unit 180, and a restored picture buffer 190. [

The encoder can perform encoding on an input image in an intra prediction mode (or a spatial mode) and / or an inter prediction mode (or temporal mode). Also, the encoder can generate a bitstream by encoding the input image, and output the generated bitstream. When the intra prediction mode is used in the prediction mode, the switch can be switched to intra prediction, and when the inter prediction mode is used in the prediction mode, the switch can be switched to inter prediction. Here, the intra prediction mode may mean an intra prediction mode (i.e., a spatial prediction mode), and the inter prediction mode may mean an inter prediction mode (i.e., a temporal prediction mode).

The encoder may generate a prediction signal for an input block of the input image. The prediction signal in block units can be referred to as a prediction block. Also, after the prediction block is generated, the encoder can code the residual of the input block and the prediction block. The input image can be referred to as the current image which is the object of the current encoding. The input block may be referred to as the current block or the current block to be coded.

The projection unit 100 serves to project an omnidirectional image or a divergent image in a two-dimensional form such as an expansion plan or an expanded view of a polyhedron. Thus, several images having irregular angles and positions can be converted into two-dimensional images corresponding to the developed view of the polyhedron. The projection unit may convert an omnidirectional image or a divergent image into a two-dimensional image using the positions and angles of the cameras.

When the prediction mode is the intra mode, the intra prediction unit 120 can use the pixel value of the block already coded around the current block as a reference pixel. The intra prediction unit 120 can perform spatial prediction using reference pixels and generate prediction samples for an input block through spatial prediction.

If the prediction mode is the inter mode, the inter-picture prediction unit 110 can search the reference picture for the best match with the input block in the motion estimation process, and derive the motion vector using the searched area have. The reference picture may be stored in the reference picture buffer 190.

The subtractor may generate a residual block using the difference between the input block and the prediction block. The residual block may be referred to as a residual signal.

The transforming unit 130 may perform a transform on the residual block to generate a transform coefficient and output the transform coefficient. Here, the transform coefficient may be a coefficient value generated by performing a transform on the residual block. When the transform skip mode is applied, the transforming unit 130 may skip transforming the residual block.

A quantized transform coefficient level can be generated by applying quantization to the transform coefficients. Hereinafter, in the embodiments, the quantized transform coefficient level may also be referred to as a transform coefficient.

The quantization unit 140 may generate a quantized transform coefficient level by quantizing the transform coefficient according to the quantization parameter, and output the quantized transform coefficient level. At this time, the quantization unit 140 can quantize the transform coefficient using the quantization matrix.

The entropy encoding unit 150 can generate a bitstream by performing entropy encoding based on the values calculated by the quantization unit 140 or the coding parameters calculated in the encoding process according to the probability distribution And can output a bit stream. The entropy encoding unit 150 may perform entropy encoding on information for decoding an image in addition to information on pixels of the image. For example, the information for decoding the image may include a syntax element or the like.

When the encoder performs encoding with inter prediction, the encoded current image can be used as a reference image for another image (s) to be processed later. Accordingly, the encoder can decode the encoded current image again, and store the decoded image as a reference image. The inverse quantization and inverse transform of the current encoded image for decoding can be processed.

The quantized coefficients can be dequantized in the inverse quantization unit 160. And may be inverse transformed by the inverse transform unit 170. The dequantized and inverse transformed coefficients may be summed with the prediction block via an adder 175. A reconstructed block can be generated by adding the residual block generated through the inverse quantization and inverse transform to the prediction block.

Although not shown, the restoration block may be routed through the filter portion. The filter unit may apply at least one of a deblocking filter, a sample adaptive offset (SAO), and an adaptive loop filter (ALF) to a reconstructed block or reconstructed image. The filter portion may also be referred to as an in-loop filter.

The reference image extension unit 180 generates a reference image according to perspectives of the restored forward direction image, diversity image, or image included in the convergent image. The reference images generated through the reference image expanding unit may be grouped by time period or perspective and stored in the reference picture buffer 190. The reference image expanding unit will be described in more detail with reference to FIG.

28 is a block diagram showing a configuration of a decoder according to the present invention.

28, the decoder includes an entropy decoding unit 210, an inverse quantization unit 220, an inverse transform unit 230, an intra prediction unit 240, an inter prediction unit 250, a reference image expansion unit 260 And a reference picture buffer 270. [

The decoder may receive the bit stream output from the encoder. The decoder may perform decoding in an intra mode or an inter mode with respect to the bit stream. Also, the decoder can generate a reconstructed image through decoding and output a reconstructed image.

When the prediction mode used for decoding is the intra mode, the switch can be switched to intra. When the prediction mode used for decoding is the inter mode, the switch can be switched to the inter.

The decoder can obtain a reconstructed residual block from the input bitstream, and can generate a prediction block. Once the restored residual block and the prediction block are obtained, the decoder can generate a restoration block, which is a block to be decoded, by adding the restored residual block and the prediction block. The block to be decoded can be referred to as a current block.

The entropy decoding unit 210 may generate the symbols by performing entropy decoding according to the probability distribution of the bitstream. The generated symbols may include a symbol in the form of a quantized transform coefficient level and information necessary for decoding the image data. Here, the entropy decoding method may be similar to the above-described entropy encoding method. For example, the entropy decoding method may be the inverse of the above-described entropy encoding method.

The entropy decoding unit 210 may change the one-dimensional vector form factor into a two-dimensional block form through a transform coefficient scanning method to decode the transform coefficient level. For example, it can be changed into a two-dimensional block form by scanning the coefficients of the block using up right scanning. Depending on the size of the conversion unit and the intra prediction mode, vertical scanning and horizontal scanning may be used instead of the upright scanning. That is, depending on the size of the conversion unit and the intra prediction mode, it is possible to determine whether any of the up scan, vertical scan and horizontal scan is to be used.

The quantized transform coefficient levels can be inversely quantized in the inverse quantization unit 220 and inversely transformed from the frequency domain into the spatial domain in the inverse transform unit 230. As a result of the dequantized and inverse transformed quantized transform coefficient levels, the reconstructed residual block can be generated. At this time, the inverse quantization unit 220 may apply the quantization matrix to the quantized transform coefficient levels.

When the intra mode is used, the intra prediction unit 240 can generate a prediction block by performing spatial prediction using the pixel value of the already decoded block around the decoding target block in the spatial domain.

When the inter mode is used, the inter-picture prediction unit 250 can generate a prediction block by performing motion compensation using a motion vector and a reference picture stored in the reference picture buffer 270 in the spatial domain. The inter picture prediction unit 250 may generate a prediction block by applying an interpolation filter to a part of the reference image when the value of the motion vector does not have an integer value. A motion compensation method of a prediction unit included in a coding unit based on a coding unit to perform motion compensation includes a skip mode, a merge mode, an AMVP mode, a current picture reference mode It is possible to determine whether there is any method, and motion compensation can be performed according to each mode. Here, the current picture reference mode may refer to a prediction mode using the preexisting reconstructed area in the current picture to which the current block to be decoded belongs. The pre-decompressed area may be an area not adjacent to the decoding target block. A predetermined vector for the current picture reference mode may be used to specify the priori-reconstructed region. A flag or an index indicating whether the current block to be decoded is a block coded in the current picture reference mode may be signaled or may be inferred through a reference picture index of the current block to be decoded. The current picture for the current picture reference mode may be in a fixed position (e.g., the position where refIdx = 0 or the last position) in the reference picture list for the block to be decoded. Alternatively, it may be variably located within the reference picture list, and for this purpose, a separate reference picture index indicating the location of the current picture may be signaled.

The restored residual block and the prediction block may be added through an adder. Although not shown, as the restored residual block and the prediction block are added, the generated block may go through the filter portion. The filter unit may apply at least one of a deblocking filter, a sample adaptive offset, and an adaptive loop filter to a restoration block or restored image.

The reference image extension unit 260 generates a reference image in accordance with perspectives of the restored forward direction image, the diversity image, or the image included in the convergent image. The reference images generated through the reference image expanding unit may be grouped by time period or perspective and stored in the reference picture buffer 270. The reference image expanding unit will be described in more detail with reference to FIG.

29 is a block diagram of a reference image expansion unit according to the present invention.

Referring to FIG. 29, the reference image extension unit may include a transform unit 310, an extension unit 320, and a reference image list generation unit 330.

The conversion unit 310 checks whether or not an image having the same temporal axis and the same positional continuity exists in the image to be used for prediction and converts the image according to the perspective of the image to be used as a prediction if it exists do. For example, for a 2D image in the form of an expanded view of a cube, the transformer may serve to transform other images according to the perspective of at least one of the other spatially neighboring images.

The expansion unit 320 serves to sum images to be used for prediction and images converted by the conversion unit. That is, the size of the image to be used for prediction (that is, the reference image) can be increased by the sum of the converted images by the extension unit. At this time, the position at which the converted image is expanded can be determined based on the characteristics of the image or the position of the image.

The reference image list generator 330 adds the reference image generated by adding the converted image and the image to be used for prediction to the reference image list. The reference image can be input to the reference image list in accordance with the time axis.

The components described in the exemplary embodiments of the present invention may be implemented by a digital signal processor (DSP), a processor, a controller, an application specific integrated circuit (ASIC), a field programmable gate Programmable logic elements, such as an array, other electronic devices, and combinations thereof. At least one of the functions or processes described through the embodiments of the present invention described above may be implemented in software and the software may be recorded in a recording medium. Examples of the recording medium include magnetic media such as a hard disk, a floppy disk and a magnetic tape, optical recording media such as CD-ROM and DVD, magneto-optical media such as a floptical disk, And hardware devices specifically configured to store and execute program instructions such as ROM, RAM, flash memory, and the like. Examples of program instructions include machine language code such as those generated by a compiler, as well as high-level language code that can be executed by a computer using an interpreter or the like. The hardware device may be configured to operate as one or more software modules for performing the processing according to the present invention, and vice versa. The components, functions, processes, and the like described through the embodiments of the present invention may be implemented through a combination of hardware and software.

While the present invention has been particularly shown and described with reference to exemplary embodiments thereof, it is to be understood that the invention is not limited to the disclosed exemplary embodiments, but, on the contrary, Those skilled in the art will appreciate that various modifications, additions and substitutions are possible, without departing from the scope and spirit of the invention as disclosed in the accompanying claims.

Therefore, the spirit of the present invention should not be construed as being limited to the above-described embodiments, and all of the equivalents or equivalents of the claims, as well as the following claims, I will say.

100:
110: inter picture prediction unit
120: intra picture prediction unit
130:
140: Quantization unit
150: Entropy coding unit
160: Inverse quantization unit
170: Inverse transform unit
180: reference image expanding section
190: Restored picture buffer
210: Entropy decoding unit
220: Inverse quantization unit
230: Inverse transform unit
240: intra picture prediction unit
250: inter picture prediction unit
260: reference image expanding section
270: reference picture buffer

Claims (17)

Converting the second image to the first viewpoint when a first image having a first viewpoint and a second viewpoint having a different viewpoint exist;
Generating a reference image by adding the second image to one surface of the first image; And
Storing the reference picture in a reference picture list
/ RTI > The method according to claim 1,
Wherein the converted image is generated based on a viewpoint difference between the first image and the second image,
Wherein the viewpoint difference includes at least one of a distance difference or an angle difference between the first image and the second image. 3. The method of claim 2,
Wherein the viewpoint difference is determined based on characteristics of an omnidirectional image including the first image and the second image. The method of claim 3,
Wherein the angle difference between the first image and the second image is determined as the inner angle of the regular polyhedron when the forward image is projected to the regular polyhedron. 3. The method of claim 2,
Wherein the viewpoint difference is calculated based on a distance and an angle difference between a first camera used to photograph the first image and a second camera used to photograph the second image. The method according to claim 1,
Wherein a position to which the second image is to be added is determined according to characteristics of a regular polyhedron including the first image and the second image. The method according to claim 1,
Wherein the reference picture is stored in the reference picture list together with time-slot information corresponding to the first video and the second video. Converting the second image to the first viewpoint when a first image having a first viewpoint and a second viewpoint having a different viewpoint exist;
Generating a reference image by adding the second image to one surface of the first image; And
Storing the reference picture in a reference picture list
And decodes the decoded image. 9. The method of claim 8,
Wherein the converted image is generated based on a viewpoint difference between the first image and the second image,
Wherein the viewpoint difference includes at least one of a distance difference or an angle difference between the first image and the second image. 10. The method of claim 9,
Wherein the viewpoint difference is determined based on characteristics of an omnidirectional image including the first image and the second image. 11. The method of claim 10,
Wherein the angle difference between the first image and the second image is determined as the inner angle of the regular polyhedron when the forward image is projected to the regular polyhedron. The method according to claim 10,
Wherein the viewpoint difference is calculated based on a distance and an angle difference between a first camera used to photograph the first image and a second camera used to photograph the second image. 9. The method of claim 8,
Wherein the position to which the second image is to be added is determined according to characteristics of a regular polyhedron including the first image and the second image. 9. The method of claim 8,
Wherein the reference picture is stored in the reference picture list together with the time zone information corresponding to the first video and the second video. 15. The method of claim 14,
Selecting at least one reference picture from the reference picture list based on a reference picture index specifying a reference picture of a current block; And
Performing inter-picture prediction on the current block based on the selected reference picture
, ≪ / RTI &
Wherein the reference picture list includes time zone information corresponding to a reference picture of the current block. A conversion unit converting the second image to the first viewpoint when a first image having a first viewpoint and a second viewpoint having a different viewpoint exist;
An extension unit for adding a second image to one side of the first image to generate a reference image; And
A reference picture list generating unit for storing the reference pictures in a reference picture list,
And an image encoding device. A conversion unit converting the second image to the first viewpoint when a first image having a first viewpoint and a second viewpoint having a different viewpoint exist;
An extension unit for adding a second image to one side of the first image to generate a reference image; And
A reference picture list generating unit for storing the reference pictures in a reference picture list,
And decodes the decoded image.

Images