KR20200143287A - Method and apparatus for encoding/decoding image and recording medium for storing bitstream - Google Patents

Abstract

Disclosed are an image encoding/decoding method, an apparatus thereof, and a recording medium storing a bitstream. A method of decoding a multi-view image including a basic view image and at least one additional view image according to an embodiment of the present invention comprises the steps of: obtaining a bitstream including basic view image encoding information on the base view image and residual addition view image encoding information on a plurality of residual additional view images; decoding the base view image and the plurality of residual addition view images based on the bitstream; and restoring the at least one additional view image from the plurality of residual additional view images, based on the base view image encoding information, the residual additional view image encoding information, and the base view image. The residual additional view image encoding information includes packing information of a patch, and the packing information includes information on importance of an image region belonging to the additional view image.

Description

Video encoding/decoding method, apparatus, and recording medium storing the bitstream {METHOD AND APPARATUS FOR ENCODING/DECODING IMAGE AND RECORDING MEDIUM FOR STORING BITSTREAM}

The present invention relates to a video encoding/decoding method, an apparatus, and a recording medium storing a bitstream. Specifically, the present invention provides an image encoding/decoding method, an apparatus, and a recording medium storing a bitstream for providing an omnidirectional video capable of supporting motion parallax in response to a horizontal/vertical movement as well as horizontal/vertical rotation of a viewer. About.

Virtual Reality (VR) service generates 360-degree omni-directional video (or omni-directional video, 360-degree video) in the form of real-life or computer graphics (CG) to create a personal VR terminal (e.g., HMD (Head Mounted Display) or smart Phones, etc.), and is evolving to maximize immersion and realism.

In studies to date, it is known that 6DoF (degrees of freedom) must be reproduced in order to reproduce natural and immersive 360-degree video through HMD. In other words, the video gazed at the viewer's movement in six directions including (1) left and right movement, (2) up and down rotation, (3) up and down movement, and (4) left and right rotation must be played through the HMD. . The omnidirectional DIO playing the live-action image acquired with the camera up to now is 3DoF, and it detects the movement of the vertical rotation and the left and right rotation among the directions and plays the image, and gazes at the viewer's horizontal movement and vertical movement. It is not able to provide video.

The MPEG (Moving Picture Experts Group) standardization group defines media for maximizing immersion as immersive media, and is step-by-step progressing standards for effective encoding and transmission of immersive video necessary for this. Specifically, as the next step of 3DoF, the most basic immersive video, 3DoF+, an immersive video that can reproduce motion parallax in the viewer's seating environment, Omnidirectional 6DoF, which provides motion parallax corresponding to the motion of a few steps of the viewer, and viewers The standardization process will be carried out step by step up to 6DoF, which provides complete parallax according to the free movement of the player. When the immersive video uses multi-view video (e.g., ERP (Equi-Rectangular Projection) format, cubemap format, etc.), windowed-6DoF may be similar to a conventional multi-view video technology having horizontal/vertical parallax. have. Here, Windowed-6DoF is a technology that provides motion parallax through a single viewing window by using flat video (eg, HD, UHD, etc.) from multiple viewpoints.

An object of the present invention is to provide a video encoding/decoding method and apparatus for supporting motion parallax, and a recording medium storing a bitstream.

In addition, an object of the present invention is to provide an immersive video formatting method and apparatus for reproducing natural omnidirectional video through a VR terminal.

The technical problems to be achieved in the present invention are not limited to the technical problems mentioned above, and other technical problems that are not mentioned will be clearly understood by those of ordinary skill in the technical field to which the present invention belongs from the following description. I will be able to.

In the encoding method of a multi-view image including a base view image and at least one additional view image according to an embodiment of the present invention, pruning is performed on the at least one additional view image based on the base view image. Generating one or more patches, packing the at least one patch, generating a plurality of residual-added view images based on the packed at least one or more patches, and adding residuals for the plurality of residual-added view images Including the step of outputting a bitstream including encoding information of a view image, wherein the packing of the at least one patch may be performed based on the importance of an image region belonging to the additional view image.

In the multi-view image encoding method, packing the at least one patch may include adjusting the size of the patch according to the importance of an image region belonging to the additional view image.

In the multi-view image encoding method, packing the at least one patch may include rotating the patch according to the importance of an image region belonging to the additional view image.

In the multi-view image encoding method, the importance of an image region belonging to the additional view image includes a depth value of the image region, a position of a camera used to acquire the image region, whether the image region is an ROI, and the It may be determined based on at least one of the complexity of the image area.

In the multi-view image encoding method, packing the at least one patch may include setting a guard band at a boundary portion of the at least one patch.

In the multi-view image encoding method, the setting of the guard band may include setting the guard band according to an importance of an image region belonging to the additional view image.

In the multi-view image encoding method, the setting of the guard band may include setting the guard band by copying a sample value adjacent to a boundary region of the at least one patch.

In the multi-view image encoding method, the setting of the guard band may include setting the guard band by interpolating a plurality of samples included in a boundary region of the at least one patch.

In the multi-view image encoding method, the packing of the at least one patch may include determining similarity between a plurality of image regions belonging to the additional view image and the at least one patch based on the similarity determination result. It may include packing into a single patch.

In the multi-view image encoding method, the bitstream may further include basic-view image encoding information on the base-view image.

A method for decoding a multi-view image including a base view image and at least one additional view image according to an embodiment of the present invention includes encoding information on a base view image and a plurality of residuals on the base view image. Acquiring a bitstream including additional view image encoding information, decoding the base view image and the plurality of residual additional view images, based on the bitstream, the base view image encoding information, the residual addition view And reconstructing the at least one additional view image from the plurality of residual additional view images based on the encoding information and the base view image, wherein the residual additional view image encoding information includes packing information of a patch, , The packing information may include information on the importance of an image region belonging to the additional view image.

In the multi-view image decoding method, the size of the patch may be determined based on the importance of an image region belonging to the additional view image.

In the multi-view image decoding method, the importance of an image region belonging to the additional viewpoint image is a depth value of the image region, a position of a camera used to acquire the image region, whether the image region is an ROI, and the It may be determined based on at least one of the complexity of the image area.

In the multi-view video decoding method, the packing information may include information on a guard band of the patch.

In the multi-view image decoding method, the guard band may be determined based on the importance of an image region belonging to the additional view image.

In the multi-view image decoding method, the packing information may include information on similarity between a plurality of image regions belonging to the additional view image.

In a non-transitory computer-readable recording medium storing a bitstream related to a method of encoding a multi-view image including a base view image and at least one additional view image according to an embodiment of the present invention, the multi-view image is encoded The method includes generating at least one patch by performing pruning on the at least one additional view image based on the base view image, packing the at least one patch, and in the packed at least one or more patches Generating a plurality of residual-added view images based on the plurality of residual-added view images, including outputting a bitstream including residual-added-view image encoding information on the plurality of residual-added view images, and packing the at least one patch May be performed based on the importance of an image region belonging to the additional view image.

The features briefly summarized above with respect to the present invention are merely exemplary aspects of the detailed description of the present invention to be described later, and do not limit the scope of the present invention.

According to the present invention, a video encoding/decoding method, an apparatus, and a recording medium storing a bitstream for supporting motion parallax can be provided.

In addition, according to the present invention, a method and apparatus for providing a complete and natural stereoscopic image to a VR device by reproducing an image corresponding to the vertical and horizontal movement of the viewer as well as the vertical and horizontal rotation movement of the viewer can be provided.

In addition, according to the present invention, the amount of data transmitted through scaling and rotation for each patch may be reduced, so that the packing efficiency of the patch may be increased.

The effects obtainable in the present invention are not limited to the above-mentioned effects, and other effects not mentioned can be clearly understood by those of ordinary skill in the art from the following description. will be.

1 is a view for explaining the concept of immersive video according to an embodiment of the present invention.
2 is a diagram illustrating an encoder structure of an immersive video according to an embodiment of the present invention.
3 is a view for explaining a patch packing process according to an embodiment of the present invention.
4 is a diagram for explaining a method of adjusting a size of a patch using the importance of an image area to be extracted as a patch according to an embodiment of the present invention.
5 is a diagram illustrating a method of improving encoding efficiency by adjusting a width of a guard band according to an embodiment of the present invention.
6 is a diagram for describing a method in which similar multiple patch areas are packed into a single patch according to an embodiment of the present invention.
7 is a diagram illustrating a patch packing method for each atlas according to an embodiment of the present invention.
8 is a diagram for describing a method of differently encoding video for each atlas according to an embodiment of the present invention.
9 is a diagram illustrating a method of encoding a multi-view image according to an embodiment of the present invention.
10 is a diagram for explaining a method of decoding a multi-view image according to an embodiment of the present invention.

In the present invention, various modifications may be made and various embodiments may be provided, and specific embodiments will be illustrated in the drawings and described in detail in the detailed description. However, this is not intended to limit the present invention to a specific embodiment, it is to be understood to include all changes, equivalents, and substitutes included in the spirit and scope of the present invention. Like reference numerals in the drawings refer to the same or similar functions over several aspects. The shapes and sizes of elements in the drawings may be exaggerated for clearer explanation. For a detailed description of exemplary embodiments described below, reference is made to the accompanying drawings, which illustrate specific embodiments as examples. These embodiments are described in detail sufficient to enable a person skilled in the art to practice the embodiments. It should be understood that the various embodiments are different from each other but need not be mutually exclusive. For example, specific shapes, structures, and characteristics described herein may be implemented in other embodiments without departing from the spirit and scope of the present invention in relation to one embodiment. In addition, it is to be understood that the location or arrangement of individual components within each disclosed embodiment may be changed without departing from the spirit and scope of the embodiment. Accordingly, the detailed description to be described below is not intended to be taken in a limiting sense, and the scope of exemplary embodiments, if properly described, is limited only by the appended claims, along with all scope equivalents to those claimed by the claims.

In the present invention, terms such as first and second may be used to describe various components, but the components should not be limited by the terms. These terms are used only for the purpose of distinguishing one component from another component. For example, without departing from the scope of the present invention, a first element may be referred to as a second element, and similarly, a second element may be referred to as a first element. The term and/or includes a combination of a plurality of related listed items or any of a plurality of related listed items.

When a component of the present invention is referred to as being "connected" or "connected" to another component, it may be directly connected or connected to the other component, but other components exist in the middle. It should be understood that there may be. On the other hand, when a component is referred to as being "directly connected" or "directly connected" to another component, it should be understood that there is no other component in the middle.

Components shown in the embodiments of the present invention are independently shown to represent different characteristic functions, and does not mean that each component is formed of separate hardware or a single software component. That is, each constituent part is listed and included as a constituent part for convenience of explanation, and at least two constituent parts of each constituent part are combined to form one constituent part, or one constituent part may be divided into a plurality of constituent parts to perform a function. Integrated embodiments and separate embodiments of the components are also included in the scope of the present invention unless departing from the essence of the present invention.

The terms used in the present invention are only used to describe specific embodiments, and are not intended to limit the present invention. Singular expressions include plural expressions unless the context clearly indicates otherwise. In the present invention, terms such as "comprises" or "have" are intended to designate the presence of features, numbers, steps, actions, components, parts, or combinations thereof described in the specification, but one or more other features. It is to be understood that the presence or addition of elements or numbers, steps, actions, components, parts, or combinations thereof, does not preclude in advance. That is, in the present invention, the description of "including" a specific configuration does not exclude configurations other than the corresponding configuration, and means that additional configurations may be included in the scope of the implementation of the present invention or the technical idea of the present invention.

Some of the components of the present invention are not essential components that perform essential functions in the present invention, but may be optional components only for improving performance. The present invention can be implemented by including only the components essential to implement the essence of the present invention excluding components used for improving performance, and including only essential components excluding optional components used for improving performance. The structure is also included in the scope of the present invention.

Hereinafter, embodiments of the present invention will be described in detail with reference to the drawings. In describing the embodiments of the present specification, if it is determined that a detailed description of a related known configuration or function may obscure the subject matter of the present specification, the detailed description thereof will be omitted, and the same reference numerals are used for the same elements in the drawings. Used and redundant descriptions for the same components are omitted.

In this specification, an immersive video is a video that allows a user to experience an immersive feeling, and may mean a 3DoF, 3DoF+, or 6DoF type image. Here, immersion may be defined as a phenomenon in which the distinction between the real and the virtual world becomes unclear.

In this specification, a video format may mean a standard for enabling a video to be used in a specific program. In addition, formatting may refer to an act of changing a video format to conform to the standard.

In this specification, video may have the same meaning as video and video.

Hereinafter, embodiments of the present invention will be described with reference to the accompanying drawings.

1 is a view for explaining the concept of immersive video according to an embodiment of the present invention.

Referring to FIG. 1, object 1 (O1) to object 4 (O4) are each a video region in a scene, V _k is an image acquired from the center of the camera (a basic view image or a reference view image), and X _k is The viewpoint position (camera position) and D _k may denote depth information at the camera center position, respectively. Immersive video, to support 6 DOF (Degree Of Freedom, DOF) in accordance with the viewer's motion, the center position (or center position, the reference position) (X _k) primary video (V _k), the viewer shown in the Multi-view position (X _k-2 , X _k-1 , ...) in multi-position video (V _k-2 , V _k-1 , ...), and related spatial information (eg, depth information, camera information), and the like, may be generated and transmitted to the terminal through video compression and packet multiplexing. Here, the basic video and/or multi-position video may be a flat video or an omnidirectional video.

Accordingly, since the immersive media system must be able to acquire, generate, transmit, and reproduce a large amount of immersive video composed of multiple views, large amount of video data must be effectively stored and compressed. When configuring an immersive media system for 3DoF+ or 6DoF, it is necessary to consider maintaining compatibility with the existing immersive video (3DoF).

Meanwhile, the immersive video formatting apparatus may acquire a basic view image, a multi view image, and the like, and a receiver (not shown) may perform the above operations.

2 is a diagram illustrating an encoder structure of an immersive video according to an embodiment of the present invention.

3 is a view for explaining a patch packing process according to an embodiment of the present invention.

Referring to FIG. 2, the encoder 200 of immersive video according to an embodiment of the present invention includes a view optimizer 210, an Atlas Constructor 220, and a texture. ) Information and depth information, an encoder 230 and a metadata composer 240. However, the present invention is not necessarily limited thereto, and a component not shown may be additionally included.

The view optimizer 210 may select at least one basic view image and at least one additional view image from at least one or more source views images captured by a plurality of cameras. In this case, the original view image, the base view image, and the additional view image may each include a texture component and a depth component. In addition, the at least one original view image may mean an immersive video composed of multiple views. Also, the texture component and the depth component may have the same meaning as the texture information and the depth information, respectively.

The atlas configurator 220 may include a pruner 221, an aggregator 222, and a patch packer 223. However, the present invention is not necessarily limited thereto, and a component not shown may be additionally included.

The atlas configurator 220 is based on at least one of depth information of at least one basic view image selected by the view optimizer 210, depth information of at least one additional view image, or camera parameters, It is possible to remove an overlapping area between images.

Specifically, the atlas configurator 220 may perform a pruning process to extract only image information from which redundant regions have been removed from an immersive video composed of multiple views. In this case, the pruning process may be performed by the pruner 221. The pruning process may be applied to a depth component of an image. Alternatively, the pruning process may be applied not only to the depth component of the image but also to the texture component.

The pruning process may refer to a process of extracting an image region that is not included in the basic viewpoint image and other additional viewpoint images and is only visible at a corresponding viewpoint position. As a result of performing the pruning process, a pruning mask image may be generated. The pruning mask image may serve to display only a region that needs to be extracted from an image of a corresponding viewpoint. In addition, the pruning mask image may be binary video data representing a non-overlapping area by comparing a corresponding viewpoint image using a 3D image signal with a basic viewpoint image and other additional viewpoint images.

An area corresponding to valid texture data in the pruning mask image may be defined as a pruning mask.

A patch may be generated based on texture data corresponding to the pruning mask. The patch may be defined as a rectangular area including data of a non-overlapping area. The patch packer 223 extracts a texture component (a non-overlapping image region) corresponding to the pruning mask of each viewpoint position image from the viewpoint position images to generate a patch for each viewpoint position image. In addition, a packed view may be generated by packing the patches of the viewpoint position images into a small number of images. In this case, the packed view may be referred to as an atlas.

The packed view may be a final format input to a general video codec (eg, AVC, HEVC or VVC). That is, the base view image and the packed view may be transmitted to and decoded by a general video codec.

For example, referring to FIGS. 2 and 3, the atlas configurator 220 receives a basic view image (V _k ) and an additional view image (V _k-1 ) and performs pruning. (Pruning) can be performed. In this case, the first mask 1 and the second mask are used as pruning masks to extract the object 1 (O1) and object 4 (O4) regions that do not overlap in the base view image and the additional view image. (mask2) can be created. In this case, the image region corresponding to the first mask and the second mask may be an image region that is not visible in the base view image and is only visible in the corresponding additional view image. Thereafter, the generated first patch (Patch1) and the second patch (Patch2) may be packed to generate a packed view, that is, an atlas. Information on a plurality of residual-added view images including the generated atlas and information on the base view image may be transmitted through a bitstream.

On the other hand, when the patches extracted from each viewpoint image are created as a packed view, they are composed of at least one image frame, so that the packing efficiency is increased in the patch packing process to reduce the size and/or number of the packed view, and it is composed of a rectangle. It is necessary to increase the compression efficiency at the boundary of the patch.

As one of the methods for reducing the amount of transmitted data by efficiently packing the patches according to an embodiment of the present invention, scaling the size of the patch according to the importance of each patch, and rotating the patch in an appropriate direction, There is a way to place it in the empty space of the packed view. In this case, it is important to minimize image quality deterioration due to the adjustment and rotation of the patch when the image is restored by the decoder. In the case of a single view image, it is possible to consider adjusting the size of the patch depending on whether it is a region of interest (ROI), but it is more suitable for immersive media in which a plurality of view images and depth information are additionally transmitted. You need a way to adjust the patch size.

4 is a diagram for explaining a method of adjusting a size of a patch using the importance of an image area to be extracted as a patch according to an embodiment of the present invention.

According to an embodiment of the present invention, in generating a packed view by packing a patch, the sizes of the patches may be differentially adjusted according to the importance of an image area to be extracted for each patch. At this time, factors that can determine the importance of the image region are the depth value of the image region, the camera position used to acquire the image region, the region of interest (ROI), and priority between images. It may include at least one of ranking and pruning order.

In addition, when a packed view is created, some patches may be rotated and packed.

As a result, the amount of data transmitted through scaling and rotation for each patch may be reduced, thereby increasing the packing efficiency of the patch.

If the factor that determines the size of the patch is the depth value, the area in focus and the area to be looked at from the viewer's point of view when acquiring a multi-view image can be mostly determined by the depth value, so that the depth value can determine the importance of the image area. will be.

When the factor determining the size of the patch is the camera position, the importance may be set to be lower from the center of the multiple cameras to the outside. For example, when the camera index is sequentially allocated according to the camera position, since the camera index difference increases as the position moves away from the center camera, the importance of the image may be set to be lower as the index difference with the center camera increases. As another example, the importance of an image may be set according to a predetermined interval (eg, even-numbered or odd-numbered) of multiple cameras. In this case, the predetermined interval may be a predefined value or may be adaptively determined according to the number of all images.

When a factor controlling the size of a patch is priority between images, a patch composed of a video region with high complexity may be set to have higher importance than a patch composed of a region with low complexity. This is because if the size of the image area in the patch is high and the size is adjusted, the image quality deteriorates a lot during restoration.

For example, referring to FIG. 4, the second patch (Patch2) corresponding to the patch regions of the object 2 (O2) and the object 4 (O4) is determined according to the importance of the image region determined by one of the various methods described above. The size can be adjusted (Scaling), rotated (Rotation) and packed.

Information indicating the importance of the patch may be encoded as metadata. For example, a 1-bit flag indicating whether a patch is included in the ROI region may be encoded. Alternatively, information indicating the priority order may be encoded.

According to the importance of the patch, a ratio of adjusting the size of the patch may be determined. For example, the ratio of size adjustment between the critical and non-critical patches may be different.

Alternatively, information indicating at least one of a scaling ratio and rotation of the patch may be encoded as metadata. For example, information indicating at least one of the original size of the patch, the size of the patch after the size is adjusted, or the ratio before and after the size adjustment may be encoded and used to determine the size adjustment ratio of the patch. For example, information indicating at least one of a clockwise rotation, a counterclockwise rotation, or a rotation angle may be encoded and used to determine whether the patch is rotated.

5 is a diagram illustrating a method of improving encoding efficiency by adjusting a width of a guard band according to an embodiment of the present invention.

The packed view generated in the patch packing process may be compressed through the texture component and depth component encoder 230 shown in FIG. 2. In this case, a boundary portion between patches is a high-frequency component, and an image region between the boundaries has a poor correlation, and thus may act as a factor of deteriorating compression efficiency.

Accordingly, according to an embodiment of the present invention, as shown in FIG. 5, a guard band may be set in the boundary area of the patch. The guard band is an area set in the boundary area of the patch, and may refer to an area that separates the boundary in order to prevent a decrease in compression efficiency at the boundary of the patch. Encoding efficiency may be improved through appropriate filtering and filling in the guard band region. In this case, the guard band may be set for both the texture component and the depth component.

For example, a pixel value included in the guard band may be generated by copying a sample adjacent to the boundary area of the patch.

As another example, pixel values included in the guard band may be generated through interpolation of a plurality of samples included in the boundary area of the patch.

As shown on the right side of FIG. 5, in order to indicate whether the text image pixel is included in the guard band, a specific level of the depth information value may be assigned to the guard band. As an example, the lower 32 levels of the 1024 level depth information values may indicate the guard band.

In this case, the width of the guard band may be determined based on the importance of the image area, similar to the description related to FIG. 4. At this time, the importance of the image region is the depth value of the image region, the position of the camera used to acquire the image region (Camera Position), the region of interest (ROI), the priority between images, and the order of pruning. It may be determined based on at least one of. Also, the size of the guard band may be determined based on the size of the patch. Also, the width of the guard band for each patch may be determined based on the priority order between the patches.

According to an embodiment, the width of the guard band may be additionally transmitted through metadata defining a patch. Alternatively, information indicating the size of the horizontal direction guard band and the size of the vertical direction guard band may be individually encoded.

6 is a diagram for describing a method in which similar multiple patch areas are packed into a single patch according to an embodiment of the present invention.

When a patch is extracted from an additional view image using a pruning mask, a similar image region may exist. Accordingly, according to an embodiment of the present invention, it is possible to reduce the amount of data by determining similarity between extracted image regions and packing multiple patch regions having similarity into a single patch.

For example, when pixel values included in the extracted image regions are similar, it may be determined as a similar image region. Accordingly, image regions including similar pixel values may be packed into a single patch.

At this time, if the similarity increases when an appropriate transform (eg, Similarity transform, Affine transform) is applied between the extracted image regions. You can apply the appropriate transformation and pack it into a single patch. Meanwhile, an encoder may additionally transmit a transformation parameter related to the transformation, and an inverse transformation may be performed by receiving a parameter from a decoder, and the transformation may be restored.

For example, referring to FIG. 6, an image region in which the object 0 (O0) and the object 1 (O1) exist in the additional view image V _k-1 may be determined as similar image regions. Accordingly, the amount of data can be reduced by packing the patch regions corresponding to object O and object 1 into the first patch (Patch1) which is a single patch.

7 is a diagram illustrating a patch packing method for each atlas according to an embodiment of the present invention.

According to an embodiment, when a patch is packed in an atlas (Atlas or Packed View), patches are atlas different from each other according to the importance of an image region to be extracted as a patch (e.g., a first atlas 1 and a second atlas 2 )). At this time, factors that can determine the importance of the image region are the depth value of the image region, the camera position used to acquire the image region, the region of interest (ROI), and priority between images. It may include at least one of ranking and pruning order.

Referring to FIG. 7, after the patches are separated into different atlases (eg, a first atlas and a second atlas), an atlas packed with patches of low importance (eg, a second atlas) is again down-scaled. scailing), compressed, and transmitted. According to the above embodiment, there may be an advantage that implementation is simple by adjusting the size of the atlas generated in a general video format in the server and the terminal, respectively.

8 is a diagram for describing a method of differently encoding video for each atlas according to an embodiment of the present invention.

According to an embodiment, when a patch is packed in an atlas (Atlas or Packed View), patches are atlas different from each other according to the importance of an image region to be extracted as a patch (e.g., a first atlas 1 and a second atlas 2 )). At this time, factors that can determine the importance of the image region are the depth value of the image region, the camera position used to acquire the image region, the region of interest (ROI), and priority between images. It may include at least one of ranking and pruning order.

Referring to FIG. 8, after the patches are separated into different atlases (eg, a first atlas and a second atlas), an atlas packed with patches of high importance is compressed at a low compression rate by a first encoder 1 , Atlas packed with patches of low importance may be compressed at a high compression rate in the second encoder (Encoder 2). Accordingly, according to the above embodiment, different decoding quality may be provided while maintaining the same video compression rate as a whole.

In addition, according to the above embodiment, in a service environment with a limited transmission rate, a video region having a high importance can be viewed in high quality, while a video region having a low importance is viewed in a relatively low quality, thereby increasing the overall viewer satisfaction. .

9 is a diagram illustrating a method of encoding a multi-view image according to an embodiment of the present invention.

Referring to FIG. 9, the apparatus for encoding a multi-view image may generate at least one or more patches by performing pruning on at least one additional view image based on a base view image (S901).

In addition, the multi-view image encoding apparatus may pack at least one or more patches (S902). In this case, step S902 may be performed based on the importance of an image region belonging to the additional view image.

On the other hand, the importance of the image region belonging to the additional viewpoint image is the depth value of the image region belonging to the additional viewpoint image, the location of the camera used to acquire the image region belonging to the additional viewpoint image, and the image region belonging to the additional viewpoint image is the region of interest. It may be determined based on at least one of whether or not the image region is complex.

Meanwhile, the step of packing at least one patch (S902) may include adjusting the size of the patch according to the importance of the image region belonging to the additional view image.

In addition, packing the at least one patch (S902) may include rotating the patch according to the importance of an image region belonging to the additional view image.

In addition, packing the at least one patch (S902) may include setting a guard band at a boundary portion of the at least one patch.

Meanwhile, in the step of setting the guard band, the guard band may be set according to the importance of an image area belonging to an additional view image.

In addition, in the setting of the guard band, the guard band may be set by copying sample values adjacent to the boundary region of at least one patch.

In addition, in the setting of the guard band, the guard band may be set by interpolating a plurality of samples included in the boundary area of at least one patch.

On the other hand, the step of packing at least one patch (S902) includes determining the similarity of a plurality of image regions belonging to the additional view image and packing at least one or more patches into a single patch based on the similarity determination result. can do.

In addition, the multi-view image encoding apparatus may generate a plurality of residual-added view images based on at least one packed patch (S903).

In addition, the multi-view image encoding apparatus may output a bitstream including residual-added-view image encoding information about a plurality of residual-added-view images (S904). In this case, the bitstream may further include base view image encoding information on the base view image.

10 is a diagram for explaining a method of decoding a multi-view image according to an embodiment of the present invention.

Referring to FIG. 10, the multi-view image decoding apparatus may obtain a bitstream including basic view image encoding information on a base view image and residual additional view image encoding information on a plurality of residual additional view images (S1001). ). In this case, the residual addition view image encoding information may include packing information of a patch.

In addition, the packing information may include information on the importance of an image region belonging to an additional view image. Also, the packing information may include information on a guard band of the patch.

In addition, the packing information may include information about similarity of a plurality of image regions belonging to an additional view image.

Meanwhile, the guard band may be determined based on the importance of an image region belonging to an additional view image.

On the other hand, the importance of the image region belonging to the additional viewpoint image is the depth value of the image region belonging to the additional viewpoint image, the location of the camera used to acquire the image region belonging to the additional viewpoint image, and the image region belonging to the additional viewpoint image is the region of interest. It may be determined based on at least one of whether or not the image region is complex.

Also, the size of the patch may be determined based on the importance of an image region belonging to the additional view image.

In addition, the multi-view image decoding apparatus may decode a base view image and a plurality of residual additional view images based on the bitstream (S1002).

In addition, the multi-view image decoding apparatus may reconstruct at least one additional view image from a plurality of residual additional view images based on the base view image encoding information, the residual additional view image encoding information, and the base view image (S1003).

The bitstream generated by the image encoding method of the present invention may be temporarily stored in a computer-readable non-transitory recording medium, and may be encoded by the multi-view image encoding method described above.

Specifically, in a non-transitory computer-readable recording medium storing a bitstream related to a method of encoding a multi-view image including a basic view image and at least one additional view image, the method of encoding the multi-view image comprises: Generating at least one patch by performing pruning on the at least one additional view image based on a viewpoint image, packing the at least one patch, a plurality of residuals based on the packed at least one patch Generating an additional view image, and outputting a bitstream including residual additional view image encoding information on the plurality of residual additional view images, wherein packing the at least one patch includes the additional view It may be performed based on the importance of the image region belonging to the image.

The exemplary methods of the present invention are expressed as a series of operations for clarity of explanation, but this is not intended to limit the order in which the steps are performed, and each step may be performed simultaneously or in a different order if necessary. In order to implement the method according to the present invention, it is also possible to include other steps in addition to the illustrated steps, to include other steps excluding some steps, or to include additional other steps excluding some steps.

Various embodiments of the present invention are not listed in all possible combinations, but are intended to describe representative aspects of the present invention, and matters described in the various embodiments may be applied independently or may be applied in combination of two or more.

In addition, various embodiments of the present invention may be implemented by hardware, firmware, software, or a combination thereof. For implementation by hardware, one or more ASICs (Application Specific Integrated Circuits), DSPs (Digital Signal Processors), DSPDs (Digital Signal Processing Devices), PLDs (Programmable Logic Devices), FPGAs (Field Programmable Gate Arrays), general purpose It may be implemented by a general processor, a controller, a microcontroller, a microprocessor, or the like.

The scope of the present invention is software or machine-executable instructions (e.g., operating systems, applications, firmware, programs, etc.) that allow an operation according to the method of various embodiments to be executed on a device or computer, and such software or It includes a non-transitory computer-readable medium (non-transitory computer-readable medium) which stores instructions and the like and is executable on a device or a computer.

Claims (17)

In the encoding method of a multi-view image including a base view image and at least one additional view image,
Generating at least one patch by performing pruning on the at least one additional view image based on the base view image;
Packing the at least one patch;
Generating a plurality of residual view images based on the at least one packed patch; And
Including the step of outputting a bitstream including the residual addition view image encoding information for the plurality of residual addition view images,
Packing the at least one patch,
The encoding method of a multi-view image, characterized in that it is performed based on the importance of an image region belonging to the additional view image.
The method of claim 1,
Packing the at least one patch,
And adjusting the size of the patch according to the importance of an image region belonging to the additional view image.
The method of claim 1,
Packing the at least one patch,
And rotating the patch according to the importance of an image region belonging to the additional view image.
The method of claim 1,
The importance of the video area belonging to the additional view video is,
A method of encoding a multi-view image, characterized in that it is determined based on at least one of a depth value of the image region, a position of a camera used to acquire the image region, whether the image region is an ROI, and a complexity of the region .
The method of claim 1,
Packing the at least one patch,
And setting a guard band at a boundary portion of the at least one patch.
The method of claim 5,
The step of setting the guard band,
And setting the guard band according to an importance of an image region belonging to the additional view image.
The method of claim 5,
The step of setting the guard band,
And setting the guard band by copying a sample value adjacent to a boundary region of the at least one or more patches.
The method of claim 5,
The step of setting the guard band,
And setting the guard band by interpolating a plurality of samples included in a boundary area of the at least one patch.
The method of claim 1,
Packing the at least one patch,
Determining similarity between a plurality of image regions belonging to the additional view image; And
And packing the at least one patch into a single patch based on a result of the similarity determination.
The method of claim 1,
The bitstream,
The encoding method of a multi-view image, further comprising base-view image encoding information on the base-view image.
In the decoding method of a multi-view image including a base view image and at least one additional view image,
Obtaining a bitstream including basic view image encoding information on the base view image and residual additional view image encoding information on a plurality of residual additional view images;
Decoding the base view image and the plurality of residual addition view images based on the bitstream; And
Restoring the at least one additional view image from the plurality of residual addition view images based on the base view image encoding information, the residual addition view image encoding information, and the base view image,
The residual addition view video encoding information includes packing information of a patch,
The packing information includes information on the importance of an image region belonging to the additional view image.
The method of claim 11,
A method of decoding a multi-view image, wherein the size of the patch is determined based on the importance of an image region belonging to the additional view image.
The method of claim 11,
The importance of the video area belonging to the additional view video is,
A method for decoding a multi-view image, characterized in that it is determined based on at least one of a depth value of the image region, a position of a camera used to acquire the image region, whether the image region is an ROI, and a complexity of the region .
The method of claim 11,
The packing information includes information on a guard band of the patch.
The method of claim 14,
The guard band is determined based on the importance of an image region belonging to the additional view image.
The method of claim 11,
The packing information includes information on similarity between a plurality of video regions belonging to the additional view video.
A non-transitory computer-readable recording medium storing a bitstream related to a method for encoding a multi-view image including a basic view image and at least one additional view image,
The encoding method of the multi-view image,
Generating at least one patch by performing pruning on the at least one additional view image based on the base view image;
Packing the at least one patch;
Generating a plurality of residual view images based on the at least one packed patch; And
Including the step of outputting a bitstream including the residual addition view image encoding information for the plurality of residual addition view images,
Packing the at least one patch,
The computer-readable recording medium, characterized in that it is performed based on the importance of an image area belonging to the additional view image.

Images