KR102475334B1 - Video encoding/decoding method and apparatus - Google Patents

Abstract

An image encoding/decoding method and apparatus are provided. An image decoding method according to the present disclosure includes obtaining image data for images of a plurality of viewpoints, determining a basic viewpoint and a plurality of reference viewpoints among the plurality of viewpoints, and a pruning order of the plurality of reference viewpoints. and parsing the image data based on the pruning order, and decoding the image of the base view and the images of the plurality of reference views.

Description

Video encoding/decoding method and apparatus {VIDEO ENCODING/DECODING METHOD AND APPARATUS}

The present disclosure relates to an image encoding/decoding method and apparatus. Specifically, the present disclosure provides an image encoding/decoding method and apparatus for determining a pruning order based on the importance of images of a plurality of reference viewpoints when images of a plurality of reference viewpoints are used, and an image encoding method of the present disclosure. Or, it relates to a recording medium storing a bitstream generated by the device.

Recently, demand for high-resolution and high-quality images such as high definition (HD) images and ultra high definition (UHD) images is increasing in various application fields. As image data becomes higher resolution and higher quality, the amount of data increases relatively compared to existing image data. Therefore, when image data is transmitted using a medium such as an existing wired/wireless broadband line or stored using an existing storage medium, transmission cost and Storage costs increase. In order to solve these problems caused by high-resolution and high-quality video data, high-efficiency video encoding/decoding technology for video with higher resolution and quality is required.

An object of the present disclosure is to provide an image encoding/decoding method and apparatus for determining a pruning order of a plurality of view images.

In addition, an object of the present disclosure is to provide an image encoding/decoding method and apparatus for determining importance of images of a plurality of viewpoints.

In addition, an object of the present disclosure is to provide a video encoding/decoding method and apparatus for efficiently managing video data.

In addition, an object of the present disclosure is to provide an image encoding/decoding method and apparatus for providing a natural omnidirectional image.

In addition, an object of the present disclosure is to provide a video encoding/decoding method and apparatus for improving video compression efficiency and video synthesis quality.

In addition, an object of the present disclosure is to provide a recording medium storing a bitstream generated by a video encoding/decoding method or apparatus of the present disclosure.

According to the present disclosure, obtaining images of a plurality of viewpoints; determining a basic viewpoint and a plurality of reference viewpoints among the plurality of viewpoints; determining a pruning order of the plurality of reference views; and performing pruning between the base view image and the plurality of reference view images based on the pruning order.

The image encoding method may further include determining a protection region mask based on important regions of the images of the plurality of viewpoints.

The region where the protective region mask is determined may not be pruned.

The protection area mask may have an arbitrary shape.

The determining of the pruning order of the plurality of reference viewpoints may include determining importance of the plurality of reference viewpoints; and determining a pruning order of the plurality of reference views based on the importance.

The step of determining the importance of the plurality of reference viewpoints may include assigning a weight to each pixel in an image of the plurality of reference viewpoints; and determining importance of the plurality of reference viewpoints based on the weights.

The assigning of the weight may include assigning the weight to each pixel in the image based on at least one of a location of an object, a distance from a camera, and a width and depth value of an occluded area.

The determining of the importance of the plurality of reference viewpoints may include determining the importance of the plurality of reference viewpoints based on a target position indicating a location of a user's virtual camera.

The determining of a basic viewpoint and a plurality of reference viewpoints among the plurality of viewpoints may include determining the basic viewpoint based on a change in a target position indicating a position of a user's virtual camera.

acquiring image data for images of a plurality of viewpoints; determining a basic viewpoint and a plurality of reference viewpoints among the plurality of viewpoints; determining a pruning order of the plurality of reference viewpoints; and parsing the image data based on the pruning order and decoding the image of the base view and the images of the plurality of reference views.

The image decoding method may further include determining a protection region mask based on important regions of the images of the plurality of viewpoints.

An area where the mask of the protection area is determined may not be pruned.

The protection area mask may have an arbitrary shape.

The determining of the pruning order of the plurality of reference viewpoints may include determining importance of the plurality of reference viewpoints; and determining a pruning order of the plurality of reference views based on the importance.

The step of determining the importance of the plurality of reference viewpoints may include assigning a weight to each pixel in an image of the plurality of reference viewpoints; and determining importance of the plurality of reference viewpoints based on the weights.

The assigning of the weight may include assigning the weight to each pixel based on at least one of a location of an object, a distance from a camera, and a width and depth value of an occluded area.

The determining of the importance of the plurality of reference viewpoints may include determining the importance of the plurality of reference viewpoints based on a target position indicating a location of a user's virtual camera.

The determining of a basic viewpoint and a plurality of reference viewpoints among the plurality of viewpoints may include determining the basic viewpoint based on a change in a target position indicating a position of a user's virtual camera.

Among the plurality of viewpoints, the basic viewpoint may be plural.

A computer-readable recording medium storing a bitstream including video encoding data decoded according to an image decoding method, the image decoding method comprising: obtaining image data for images of a plurality of viewpoints; determining a basic viewpoint and a plurality of reference viewpoints among the plurality of viewpoints; determining a pruning order of the plurality of reference viewpoints; and parsing the image data based on the pruning sequence to decode the image of the base view and the images of the plurality of reference views. .

According to the present disclosure, an image encoding/decoding method and apparatus for determining a pruning order of a plurality of view images may be provided.

In addition, according to the present disclosure, an image encoding/decoding method and apparatus for determining importance of a plurality of viewpoint images may be provided.

In addition, according to the present disclosure, an image encoding/decoding method and apparatus for efficiently managing image data may be provided.

In addition, according to the present disclosure, an image encoding/decoding method and apparatus for providing a natural omnidirectional image may be provided.

In addition, according to the present disclosure, an image encoding/decoding method and apparatus for improving image compression efficiency and image synthesis quality may be provided.

1 illustrates images acquired from a plurality of viewpoints according to an embodiment of the present disclosure.
2 illustrates a process of removing overlapping regions between images of a plurality of viewpoints according to an embodiment of the present disclosure.
3 illustrates a pruning process between images of a plurality of viewpoints according to an embodiment of the present disclosure.
4A and 4B illustrate processes of primary pruning and secondary pruning between images of a plurality of viewpoints according to an embodiment of the present disclosure.
5 illustrates weight factors when calculating the importance of images of a plurality of viewpoints according to an embodiment of the present disclosure.
6 illustrates a change in a target position when calculating importance of images of a plurality of viewpoints according to an embodiment of the present disclosure.
7 illustrates a method of determining a protection region mask in a process of pruning images of a plurality of viewpoints according to an embodiment of the present disclosure.
8 illustrates determining a protection area mask based on an object, such as a person, according to an embodiment of the present disclosure.
9 illustrates an image encoding method according to an embodiment of the present disclosure.
10 illustrates an image decoding method according to an embodiment of the present disclosure.

Since the present disclosure can make various changes and have various embodiments, specific embodiments are illustrated in the drawings and described in detail in the detailed description. However, this is not intended to limit the present disclosure to specific embodiments, and should be understood to include all modifications, equivalents, and substitutes included in the spirit and scope of the present disclosure. Like reference numbers in the drawings indicate the same or similar function throughout the various aspects. The shapes and sizes of elements in the drawings may be exaggerated for clarity. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS For detailed descriptions of exemplary embodiments described below, reference is made to the accompanying drawings, which illustrate specific embodiments by way of example. These embodiments are described in sufficient detail to enable those skilled in the art to practice the embodiments. It should be understood that the various embodiments are different, but need not be mutually exclusive. For example, specific shapes, structures, and characteristics described herein may be implemented in another embodiment without departing from the spirit and scope of the present disclosure in connection with one embodiment. Additionally, it should be understood that the location or arrangement of individual components within each disclosed embodiment may be changed without departing from the spirit and scope of the embodiment. Accordingly, the detailed description set forth below is not to be taken in a limiting sense, and the scope of the exemplary embodiments, if properly described, is limited only by the appended claims, along with all equivalents as claimed by those claims.

In this disclosure, terms such as first and second may be used to describe various components, but the components should not be limited by the terms. These terms are only used for the purpose of distinguishing one component from another. For example, a first element may be termed a second element, and similarly, a second element may be termed a first element, without departing from the scope of the present disclosure. The terms and/or include any combination of a plurality of related recited items or any of a plurality of related recited items.

When a component of the present disclosure is referred to as “connected” or “connected” to another component, it may be directly connected or connected to the other component, but there may be other components in the middle. It should be understood that it may be On the other hand, when an element is referred to as “directly connected” or “directly connected” to another element, it should be understood that no other element exists in the middle.

Components appearing in the embodiments of the present disclosure are shown independently to represent different characteristic functions, and do not mean that each component is composed of separate hardware or a single software component. That is, each component is listed and included as each component for convenience of explanation, and at least two components of each component can be combined to form a single component, or one component can be divided into a plurality of components to perform a function, and each of these components can be divided into a plurality of components. Integrated embodiments and separated embodiments of components are also included in the scope of the present disclosure unless departing from the essence of the present disclosure.

Terms used in the present disclosure are only used to describe specific embodiments, and are not intended to limit the present disclosure. Singular expressions include plural expressions unless the context clearly dictates otherwise. In the present disclosure, terms such as "comprise" or "having" are intended to indicate that there is a feature, number, step, operation, component, part, or combination thereof described in the specification, but one or more other features It should be understood that the presence or addition of numbers, steps, operations, components, parts, or combinations thereof is not precluded. That is, the description of “including” a specific configuration in the present disclosure does not exclude configurations other than the corresponding configuration, and means that additional configurations may be included in the practice of the present disclosure or the scope of the technical idea of the present disclosure.

Some of the components of the present disclosure may be optional components for improving performance rather than essential components that perform essential functions in the present disclosure. The present disclosure may be implemented by including only components essential to implement the essence of the present disclosure, excluding components used for performance improvement, and a structure including only essential components excluding optional components used for performance enhancement. Also included in the scope of the present disclosure.

EMBODIMENT OF THE INVENTION Hereinafter, embodiment of this disclosure is specifically described with reference to drawings. In describing the embodiments of this specification, if it is determined that a detailed description of a related known configuration or function may obscure the gist of the present specification, the detailed description will be omitted, and the same reference numerals will be used for the same components in the drawings. and duplicate descriptions of the same components are omitted.

1 illustrates images acquired from a plurality of viewpoints according to an embodiment of the present disclosure. The viewpoint image may correspond to an image acquired at a corresponding viewpoint. The viewpoint image may be expressed as an image of a corresponding viewpoint.

Referring to FIG. 1 , images may be acquired by a plurality of cameras having different viewpoints. View C1 104 may correspond to an image obtained from a central viewpoint. View L1 102 may correspond to an image acquired from a left viewpoint. View R1 105 may correspond to an image obtained from a right viewpoint. View V 103 may correspond to an image of a virtual viewpoint located in the middle of View L1 102 and View C1 104 using a reference viewpoint. View V 103 may include an occluded area that is not visible in View C1 104 . Since the occluded area is partially visible in View L1 (102), it can be referred to when synthesizing images.

2 illustrates a process of removing overlapping regions between images of a plurality of viewpoints according to an embodiment of the present disclosure. When a basic viewpoint and a reference viewpoint exist, image data may be reduced by removing overlapping parts.

Referring to FIG. 2 , View C1 203 may correspond to an image obtained from a central viewpoint. The remaining View L2 201, View L1 202, and View R1 204 may correspond to images obtained from a reference viewpoint. View C1 203 is an image of each reference viewpoint and may be 3D View Warped using 3D geometric relationships and depth information. Also, View C1 203 may be mapped to an image of each reference viewpoint.

Warped-view C1 onto L2 211 may correspond to an image generated by warping View C1 203 to View L2 201 as a reference view. Warped-view C1 onto L1 212 may correspond to an image generated by warping View C1 203 to View L1 202 as a reference view. In this case, an occluded area in View C1 (203) may be created like a black area in Warped-view C1 onto L2 (211) and Warped-view C1 onto L1 (212). The invisible area may correspond to a hole area without data. The remaining area other than the hole area may correspond to an area visible in View C1 (203). At this time, the overlapping area can be removed through a procedure for checking whether the overlapped area between View L2 (201) and Warped-view C1 onto L2 (211) matches. In addition, the overlapping area may be removed through a procedure for confirming whether the overlapping area between the View L1 202 and the Warped-view C1 onto L1 212 is correct. As a method of removing overlapping regions, texture data and depth information may be compared in units of image pixels mapped to the same coordinates or within a certain range of corresponding coordinates. Then, residual image 1 221 and residual image 2 222 may be generated through the comparison.

Residual image 1 (221) may correspond to an image that is not visible in View C1 (203) and is visible only in View L2 (201), which is a reference viewpoint. Residual image 2 (222) may correspond to an image that is not visible in View C1 (203) and is visible only in View L1 (202), which is a reference viewpoint.

3 illustrates a pruning process between images of a plurality of viewpoints according to an embodiment of the present disclosure. Pruning may correspond to a process of removing residual images between images of a plurality of viewpoints.

Referring to FIG. 3 , images may be obtained from five viewpoints. The viewpoint image v0 and the viewpoint image v1 may correspond to the basic viewpoint image (Basic View). The remaining viewpoint images v2, v3, and v4 may correspond to additional viewpoint images (Additional View). Since the viewpoint image v0 and the viewpoint image v1 are basic viewpoint images, the image information can be used as it is. The remaining additional viewpoint images (v2 to v4) may be compared with the basic viewpoint image and redundancy may be removed. When removing the overlapping region, depth information and camera information may be used. By warping the additional viewpoint image with the basic viewpoint image using depth information and camera information, overlapping pixels may be removed from the additional viewpoint image.

In the viewpoint image v2, overlapping regions based on the basic viewpoint images v0 and v1 may be removed. And, a pruned mask image of v2 representing only the remaining residual image information may be generated. In the following order, overlapping regions of the viewpoint image v3 based on the basic viewpoint images v0 and v1 may be removed. And, a pruned mask image of v3 may be generated. At this time, since an area overlapping with the viewpoint image v2 may additionally exist, the pruning process may also be performed with v2. In the following order, overlapping regions of the viewpoint image v4 based on the basic viewpoint images v0 and v1 may be removed. And, a mask image of the pruned v4 may be generated. At this time, since an area overlapping with the viewpoint image v3 may additionally exist, the pruning process may be performed with v3 as well.

Images remaining in the additional viewpoint image may correspond to information that is not visible due to occlusion in the basic viewpoint image. Such information may exist in overlapping regions between additional viewpoint images. Also, the corresponding information may be removed in the secondary pruning process. In the secondary pruning process, the pruning aspect may vary according to the pruning order of additional viewpoint images. Therefore, the final synthesis quality and compression efficiency may vary depending on the pruning aspect.

4A and 4B illustrate processes of primary pruning and secondary pruning between images of a plurality of viewpoints according to an embodiment of the present disclosure.

Referring to FIGS. 4A and 4B , View 0 (400) and View 1 (401) may correspond to basic viewpoint images. View 2 (402) and View 3 (403) may correspond to the reference viewpoint image. Warping view 0 onto view 2 412 may correspond to an image obtained by warping View 0 400 to the position of View 2 402 using depth. Warping view 0 onto view 3 413 may correspond to an image obtained by warping View 0 400 to the position of View 3 413 using depth. Also, in Warping view 0 onto view 2 412 and Warping view 0 onto view 3 413 , an area not visible in View 0 400 may be detected. After the pruning process, residual image 1 422 and residual image 2 423 may be generated. The pruning process may correspond to a first pruning process.

After the primary pruning process, a secondary pruning process may be performed to remove overlapping regions between reference view images. In the secondary pruning process, among the residual image 1 422 and the residual image 2 423, the pruning pattern may vary according to the pruning order. When the residual image 2 423 is pruned (440) based on the residual image 1 422, a residual image 3 433 may be generated by removing the overlapping region. Also, the pruned images may correspond to residual image 1 422 and residual image 3 433. When the residual image 1 422 is pruned (441) based on the residual image 2 423, a residual image 4 432 may be generated by removing the overlapping region. Also, the pruned images may correspond to residual image 2 423 and residual image 4 432.

When the residual image 2 423 is pruned (440) based on the residual image 1 422, the pruned image may be concentrated in the residual image 1 422, which is one image. On the other hand, when the residual image 1 (422) is pruned (441) based on the residual image 2 (423), the object of the residual image 1 (422) is divided in half, and the residual image 2 (423) and the residual image 4 are respectively divided in half. (432) may exist. Thereafter, the pruned images may be packed in image units. Packed images can be compressed and transmitted to the terminal. The terminal may receive and decode it or perform image synthesis. Here, when video synthesis is performed, it may be advantageous in terms of compression efficiency or image quality that one object exists in one or a small number of patches rather than one object divided into several patches.

에 해당될 수 있다. 여기서 i는 중요도가 산정되는 개별 시점영상의 번호에 해당될 수 있다. x와 y는 개별 시점영상의 화소의 좌표위치에 해당될 수 있다. 각 화소마다 가중치가 계산되면 모든 화소의 가중치를 합산하여 해당 시점영상의 중요도가 산정될 수 있다.5 illustrates weight factors when calculating the importance of images of a plurality of viewpoints according to an embodiment of the present disclosure. In the present disclosure, when a pruning order is determined, a viewpoint image having a high importance among a plurality of viewpoint images may be determined. When the importance of the corresponding view image is high, a high priority may be given. When calculating the importance of the plurality of viewpoint images, the position of the object, the distance from the camera, and the width of the occluded area generated in image synthesis may be considered. In addition, the importance of the plurality of viewpoint images may be calculated by assigning a weight according to each element. The importance calculation formula is

may apply to Here, i may correspond to the number of an individual viewpoint image whose importance is calculated. x and y may correspond to coordinate positions of pixels of individual viewpoint images. When the weight is calculated for each pixel, the importance of the corresponding viewpoint image may be calculated by adding the weights of all pixels.

,

에 해당될 수 있다. 여기서 x와 y는 각 화소의 2차원 좌표에 해당될 수 있다. 장면에서 관심 영역 혹은 관심 객체에 해당하는 화소가 카메라의 중간에 위치할 확률이 높다는 가정에 기반하여, 영상 1(501)에서처럼 각 화소마다 중심으로부터 거리를 고려하여 가중치가 산정될 수 있다. 영상 2(502)는 영상 1(501)의 깊이영상에 해당될 수 있다. 영상 2(502)에서는 개별 전경 혹은 배경의 깊이에 따라 x축과 y축으로 워핑되면서 발생하는 가려짐 영역의 너비 또는 시차(Parallax)의 정도가 달라질 수 있다. 따라서, 시점영상의 중요도 산정 시에 깊이값이 고려될 수 있다. 이는 장면에서 중요한 객체일수록 카메라에 가까울 확률이 높을 수 있기 때문이다. 전경에 속하는 객체가 워핑될 때, 객체에 의해 가려졌던 영역이 발생될 수 있다. 상기 영역은 전경에 의해 가려질 수 있다. 또는, 상기 영역은 인접한 배경 화소의 깊이 값의 차이에 비례할 수 있다. 즉, 전경과 배경의 깊이차이가 클수록 가려짐 영역이 넓거나 길게 생성될 수 있다. 영상 3(503)에서는 전경과 배경간 깊이차이를 나타내기 위해 소벨(Sobel) 연산을 통해 경계부분에서 깊이차이가 산정될 수 있다. 중요도 산정식에서

는 중요도 산정의 각 항마다 적용되는 가중치에 해당될 수 있다.Referring to FIG. 5 , in image 1 501, a higher weight may be assigned as the location of an object is closer to the center of the image. The formula for calculating the weight is

,

may apply to Here, x and y may correspond to 2D coordinates of each pixel. Based on the assumption that there is a high probability that a pixel corresponding to a region of interest or an object of interest in a scene is located in the middle of a camera, a weight may be calculated by considering a distance from the center of each pixel as in image 1 501 . Image 2 (502) may correspond to the depth image of image 1 (501). In the image 2 502, the width of the occluded area or the degree of parallax caused by warping in the x-axis and y-axis may vary according to the depth of each foreground or background. Accordingly, the depth value may be considered when calculating the importance of the viewpoint image. This is because a more important object in a scene may have a higher probability of being closer to the camera. When an object belonging to the foreground is warped, a region that has been occluded by the object may be generated. The area may be obscured by the foreground. Alternatively, the area may be proportional to a difference in depth values of adjacent background pixels. That is, as the depth difference between the foreground and the background increases, the occluded area may be wider or longer. In image 3 (503), a depth difference can be calculated at the boundary through a Sobel operation to indicate a depth difference between the foreground and the background. In the importance calculation formula

may correspond to a weight applied to each term of the importance calculation.

The importance calculation formula can be used in many cases. In the process of pruning a plurality of viewpoint images in FIG. 3 , a viewpoint image determined to be of high importance may correspond to a high priority. Accordingly, a viewpoint image having a high importance among viewpoint images may be pruned less. Small pruning may mean that the number of pixels in which overlapping regions disappear through the pruning process is small. In addition, viewpoint images determined to be of low importance may correspond to low ranks. Accordingly, viewpoint images of low importance among viewpoint images may be pruned a lot. A lot of pruning may mean that there are many pixels in which overlapping regions disappear through the pruning process.

6 illustrates a change in a target position when calculating importance of images of a plurality of viewpoints according to an embodiment of the present disclosure.

Referring to FIG. 6 , nine cameras from camera 0 to camera 8 may capture viewpoint images at respective positions. An image at a target virtual viewpoint may be synthesized using viewpoint images obtained by being simultaneously photographed. In this case, the target position may be changed from position p0 to p15 via p4 and p6 along the time axis. The target location may indicate a location of a virtual camera viewed by the user. Also, the tracking coordinates of the virtual camera moving along the time axis may be defined as a pose trace. The pose trace may correspond to the coordinates of the target virtual viewpoint image that the user arbitrarily moves when the user views the synthesized viewpoint image in the terminal. Further, the coordinates may correspond to coordinates in which the user's movement obtained from the terminal is transmitted to the server-side encoder or the user's expected movement coordinates predefined in advance as intended by the manufacturer.

At this time, the 3D x, y, and z coordinates of the virtual camera at the target location may be translated. Also, a pose that is a rotation angle at which the virtual camera views the scene may be changed. When the target position changes, the number of reference viewpoint images or the importance of each viewpoint image may be different. Here, a high priority may be given to a viewpoint image having a high correlation with the target position. A viewpoint image to which a high priority is assigned may be pruned less often.

Also, when selecting a basic viewpoint image, a viewpoint having a high correlation with the target position may be selected as the basic viewpoint. Also, the information on the basic viewpoint may be transmitted as metadata. The basic viewpoint image may correspond to an image of a viewpoint serving as a criterion for pruning. In the reference viewpoint images, overlapping regions may be removed through a pruning process by referring to pixels of the basic viewpoint image.

7 illustrates a method of determining a protection region mask in a process of pruning images of a plurality of viewpoints according to an embodiment of the present disclosure.

와

를 포함할 수 있다.

가

보다 우선순위가 높으면 702처럼 프루닝이 수행될 수 있다. 반대로,

가

보다 우선순위가 낮으면 703처럼 프루닝이 수행될 수 있다. 701에서

는 숫자면 4에 대한 텍스처 정보가 적고 옆면에 대한 텍스처 정보를 많이 가질 수 있다.

는 옆면에 대한 텍스처 정보가 적고 숫자면 4에 대한 텍스처 정보를 많이 가질 수 있다. 704에서는

와

에서 많이 가진 텍스처 정보는 보존하고 중요도가 떨어지는 영역의 텍스처 정보는 제거될 수 있다. 704와 같이 프루닝이 수행되는 경우, 영상합성에서 좋은 결과가 나타날 수 있다. 704와 같이 프루닝을 수행하기위해서 705처럼 사전에

와

에 중요한 영역이라고 판단되는 곳에 보호 영역 마스크(Protection Mask)가 지정될 수 있다. 705처럼 보호 영역 마스크가 지정된 경우, 해당 영역은 프루닝 과정에서 시점영상의 우선순위에 상관없이 프루닝되지 않고 보존될 수 있다. 즉, 705에서

는 옆면에 보호 영역 마스크가 지정되고

는 숫자면 4에 보호 영역 마스크가 지정되면 704와 같이 프루닝이 수행될 수 있다.Referring to FIG. 7 , 701 is a viewpoint image of an object captured at a different location.

Wow

can include

go

If the priority is higher, pruning may be performed as in 702 . on the other way,

go

If the priority is lower than 703, pruning may be performed. at 701

may have less texture information for the number side 4 and a lot of texture information for the side face.

may have less texture information for the side face and a lot of texture information for the number face 4. in 704

Wow

A lot of texture information can be preserved, and texture information of less important areas can be removed. When pruning is performed as in 704, good results may be obtained in image synthesis. To perform pruning as in 704, as in 705

Wow

A protection mask may be designated where it is determined to be an important area. When a protection region mask is designated as in 705, the corresponding region may be preserved without being pruned regardless of the priority of the viewpoint image in the pruning process. i.e. at 705

is assigned a protection area mask on the side and

pruning can be performed as in step 704 when a protection area mask is designated for the number surface 4.

8 illustrates determining a protection area mask based on an object, such as a person, according to an embodiment of the present disclosure. For example, the protection area mask may have a shape such as a polygon such as a triangle or a quadrangle or a figure such as a circle. For example, a user may designate a protection area mask in a free shape as desired. For example, the protection area mask may be automatically designated by a computer vision algorithm or the like. For example, the corresponding protection area mask may be designated by being transferred to the server as separate metadata after a user watches a synthesized image in a terminal and designates an area that the user pays attention to or arouses interest. For example, the protection area mask may correspond to an object mask image such as a person or an object separated from a background in a scene. In addition, it may correspond to a second mask image generated based on the object mask image. However, the present disclosure is not limited to the above embodiment.

Referring to FIG. 8 , a region of a person in an image may correspond to a region of high interest or an important region. Accordingly, a protection area mask can be assigned to the area for the person. Accordingly, in the pruning process, a region for a person to whom a protection region mask is designated may be excluded from the pruning target. A region synthesized by pixels excluded from pruning may have a relatively improved composite quality compared to other regions.

9 illustrates an image encoding method according to an embodiment of the present disclosure.

Referring to FIG. 9 , images of a plurality of viewpoints may be acquired (S910).

Also, a basic viewpoint and a plurality of reference viewpoints may be determined from among the plurality of viewpoints (S920).

According to an embodiment, a basic viewpoint may be determined based on a change in a target position indicating a position of a user's virtual camera.

According to an embodiment, a plurality of basic viewpoints may correspond to a plurality of viewpoints.

According to an embodiment, information on the basic viewpoint may be transmitted as metadata as separate information.

Then, the pruning order of the plurality of reference viewpoints may be determined (S930).

According to an embodiment, importance of a plurality of reference viewpoints may be determined, and a pruning order of the plurality of reference viewpoints may be determined based on the importance.

According to an embodiment, a weight may be assigned to each pixel in an image of a plurality of reference views, and the importance of the plurality of reference views may be determined based on the weight.

According to an embodiment, a weight may be assigned to each pixel in an image based on at least one of a location of an object, a distance from a camera, and a width and depth value of an occluded area.

According to an embodiment, the importance of a plurality of reference viewpoints may be determined based on a change in a target position representing a position of a user's virtual camera.

Then, pruning may be performed between an image of a basic view and images of a plurality of reference views based on the pruning order (S940).

According to an embodiment, a protection region mask may be determined based on important regions of images of a plurality of viewpoints.

According to one embodiment, the region where the protection region mask is determined may not be pruned.

According to one embodiment, the guard area mask can have any shape.

10 illustrates an image decoding method according to an embodiment of the present disclosure.

Referring to FIG. 10 , image data for images of a plurality of viewpoints may be obtained (S1010).

Also, a basic viewpoint and a plurality of reference viewpoints may be determined from among a plurality of viewpoints (S1020).

According to an embodiment, a basic viewpoint may be determined based on a change in a target position indicating a position of a user's virtual camera.

According to an embodiment, a plurality of basic viewpoints may correspond to a plurality of viewpoints.

And, the pruning order of the plurality of reference viewpoints may be determined (S1030).

According to an embodiment, importance of a plurality of reference viewpoints may be determined, and a pruning order of the plurality of reference viewpoints may be determined based on the importance.

According to an embodiment, a weight may be assigned to each pixel in an image of a plurality of reference views, and the importance of the plurality of reference views may be determined based on the weight.

According to an embodiment, a weight may be assigned to each pixel based on at least one of the location of the object, the distance from the camera, and the width and depth values of the occluded area.

According to an embodiment, the importance of a plurality of reference viewpoints may be determined based on a change in a target position representing a position of a user's virtual camera.

In addition, an image of a basic view and an image of a plurality of reference views may be decoded by parsing the image data based on the pruning sequence (S1040).

According to an embodiment, a protection region mask may be determined based on important regions of images of a plurality of viewpoints.

According to one embodiment, the region where the protection region mask is determined may not be pruned.

According to one embodiment, the guard area mask can have any shape.

According to an embodiment, a computer readable recording medium storing a bitstream including image encoding data decoded according to an image decoding method may be provided.

In the foregoing embodiments, the methods are described on the basis of a flow chart as a series of steps or units, but the present disclosure is not limited to the order of steps, and some steps may occur in a different order or concurrently with other steps as described above. can In addition, those skilled in the art will understand that the steps shown in the flow chart are not exclusive, other steps may be included, or one or more steps of the flow chart may be deleted without affecting the scope of the present disclosure. You will understand.

The foregoing embodiment includes examples of various aspects. It is not possible to describe all possible combinations to represent the various aspects, but those skilled in the art will recognize that other combinations are possible. Accordingly, it is intended that this disclosure embrace all other substitutions, modifications and variations falling within the scope of the following claims.

Embodiments according to the present disclosure described above may be implemented in the form of program instructions that can be executed through various computer components and recorded in a computer readable recording medium. The computer readable recording medium may include program instructions, data files, data structures, etc. alone or in combination. Program instructions recorded on the computer-readable recording medium may be those specially designed and configured for the present disclosure, or may be known and usable to those skilled in the art of computer software. Examples of computer-readable recording media include magnetic media such as hard disks, floppy disks and magnetic tapes, optical recording media such as CD-ROMs and DVDs, and magneto-optical media such as floptical disks. media), and hardware devices specially configured to store and execute program instructions, such as ROM, RAM, flash memory, and the like. Examples of program instructions include high-level language codes that can be executed by a computer using an interpreter or the like as well as machine language codes such as those produced by a compiler. The hardware device may be configured to act as one or more software modules to perform processing according to the present disclosure and vice versa.

In the above, the present disclosure has been described by specific details such as specific components and limited embodiments and drawings, but these are provided to help a more general understanding of the present disclosure, and the present disclosure is not limited to the above embodiments. However, those skilled in the art can make various modifications and variations from these descriptions.

Therefore, the spirit of the present disclosure should not be limited to the above-described embodiments, and not only the claims to be described later, but also all modifications equivalent or equivalent to these claims belong to the scope of the spirit of the present disclosure. will do it

Claims (20)

Acquiring images of a plurality of viewpoints;
determining a basic viewpoint and a plurality of reference viewpoints among the plurality of viewpoints;
determining a pruning order of the plurality of reference views;
determining a protection region mask based on important regions of the images of the plurality of viewpoints; and
Performing pruning between an image of the base view and images of the plurality of reference views based on the pruning order;
An image encoding method according to claim 1 , wherein the region in which the protection region mask is determined is not pruned. delete delete According to claim 1,
The image encoding method, characterized in that the protection region mask has an arbitrary shape. According to claim 1,
Determining the pruning order of the plurality of reference views,
determining importance of the plurality of reference viewpoints; and
and determining a pruning order of the plurality of reference views based on the importance. According to claim 5,
The step of determining the importance of the plurality of reference points,
assigning a weight to each pixel in the images of the plurality of reference viewpoints; and
and determining importance of the plurality of reference views based on the weights. According to claim 6,
In the step of assigning the weight,
An image encoding method characterized in that the weight is assigned to each pixel in the image based on at least one of a location of an object, a distance from a camera, and a width and depth value of an occluded area. According to claim 5,
The step of determining the importance of the plurality of reference points,
An image encoding method characterized in that determining importance of the plurality of reference viewpoints based on a target position indicating a position of a user's virtual camera. According to claim 1,
The step of determining a basic viewpoint and a plurality of reference viewpoints among the plurality of viewpoints,
An image encoding method characterized in that the basic viewpoint is determined based on a change in a target position indicating a position of a user's virtual camera. acquiring image data for images of a plurality of viewpoints;
determining a basic viewpoint and a plurality of reference viewpoints among the plurality of viewpoints;
determining a pruning order of the plurality of reference viewpoints;
determining a protection region mask based on important regions of the images of the plurality of viewpoints; and
Parsing the image data based on the pruning order and decoding the image of the base view and the images of the plurality of reference views;
An image decoding method according to claim 1 , wherein an area in which the mask of the protection area is determined is not pruned. delete delete According to claim 10,
The image decoding method, characterized in that the protection region mask has an arbitrary shape. According to claim 10,
Determining the pruning order of the plurality of reference views,
determining importance of the plurality of reference viewpoints; and
and determining a pruning order of the plurality of reference views based on the importance. According to claim 14,
The step of determining the importance of the plurality of reference points,
assigning a weight to each pixel in the images of the plurality of reference viewpoints; and
and determining importance of the plurality of reference views based on the weights. According to claim 15,
In the step of assigning the weight,
An image decoding method characterized in that the weight is assigned to each pixel based on at least one of a position of an object, a distance from a camera, and a width and depth value of an occluded area. According to claim 14,
The step of determining the importance of the plurality of reference points,
An image decoding method characterized in that determining the importance of the plurality of reference viewpoints based on the variation of the target position indicating the position of the user's virtual camera. According to claim 10,
The step of determining a basic viewpoint and a plurality of reference viewpoints among the plurality of viewpoints,
An image decoding method characterized in that the basic viewpoint is determined based on a target position indicating a position of a user's virtual camera. According to claim 10,
Among the plurality of views, the basic view is a plurality of views. A computer-readable recording medium storing a bitstream including video coded data decoded according to an image decoding method, comprising:
The video decoding method,
acquiring image data for images of a plurality of viewpoints;
determining a basic viewpoint and a plurality of reference viewpoints among the plurality of viewpoints;
determining a pruning order of the plurality of reference viewpoints;
determining a protection region mask based on important regions of the images of the plurality of viewpoints; and
Parsing the image data based on the pruning order and decoding the image of the base view and the images of the plurality of reference views,
A region in which the mask of the protection region is determined is not pruned.

Images