KR102681496B1 - Method and apparatus for converting image - Google Patents

Abstract

An image conversion method according to an embodiment of the present invention includes generating multi-plane 3D data reconstructed into layers in the depth direction based on a plurality of multi-view image data, and dividing the layers of the multi-plane 3D data into at least one layer. It may include combining layers to create a combined layer, and converting multi-plane 3D data including the combined layer into a 2D atlas image.

Description

Image conversion method and device {METHOD AND APPARATUS FOR CONVERTING IMAGE}

The present invention relates to a method and device for converting multi-plane 3D data into a 2D atlas image.

In general, multi-plane 3D data (Multi Plane Image, MPI) is a 3D space expression method that reconstructs 3D space into a depth direction layer and places pixels in the space on a depth direction plane.

The MPI-based spatial expression method can achieve relatively high image quality when freely rendering space from an arbitrary viewpoint, and does not require high-quality depth information, which is the biggest factor in maintaining image quality when expressing photo-realistic spatial information, creating a new 3D It is used in various ways as a method of expressing real-life space.

In order to convert conventional multi-plane 3D data into a 2D atlas image, the patch areas (areas where actual pixels exist) in each layer image of the multi-plane 3D data are collected into a single 2D atlas image, which is the size of the 2D atlas image. becomes very large, and there is a large amount of empty space in the 2D atlas image that does not contain the patch area, which reduces compression efficiency.

The purpose of the present invention is to provide an image conversion method and device for reducing the size of a 2D atlas image when converting multi-plane 3D data into a 2D atlas image.

Additionally, the purpose of the present invention is to provide an image conversion method and device for improving the compression efficiency of 2D atlas images.

The image conversion method according to the present invention for achieving the above object includes generating multi-plane 3D data reconstructed into layers in the depth direction based on a plurality of multi-view image data, and creating layers of the multi-plane 3D data. It may include generating a combined layer by combining at least one layer, and converting multi-plane 3D data including the combined layer into a 2D atlas image.

The step of generating the combining layer includes generating as a first combining layer the pixels that are most visible for the same pixel coordinates when viewed in the direction of the camera origin in the layer with the deepest depth among the layers, and the second combining layer among the layers. Excluding pixels included in the first combining layer when looking at the camera origin direction in a deep layer, it may include generating the pixels that are most visible for each pixel coordinate as a second combining layer.

If there are no pixels in the layer, the step of creating the combined layer can be ended.

The 2D atlas image may include a transparency image and a color image generated based on pixel information of each combined layer of the multi-plane 3D data, and a layer index image having location information of the layer for the pixel.

The step of compressing the 2D atlas image may be further included.

The 2D atlas image can be generated as bitstream data using at least one of HEVC, H.263, and VVC.

The plurality of multi-view image data may include image data in a two-dimensional array.

Multi-plane 3D data can be generated by creating planes for each distance in the depth direction and rearranging the pixel values and transparency of the pixels corresponding to each plane.

In addition, the image conversion device according to the embodiment includes a memory storing a control program for image conversion, and a processor executing the control program stored in the memory, and the processor performs depth direction based on a plurality of multi-view image data. Create multi-plane 3D data reconstructed with layers, combine the layers of the multi-plane 3D data into at least one layer to create a combined layer, and convert the multi-plane 3D data including the combined layer into a 2D atlas image. It can be converted.

The processor generates the first visible pixels for the same pixel coordinates when looking in the direction of the camera origin in the layer with the deepest depth among the layers as a first combining layer, and the camera origin in the layer with the second deepest depth among the layers. Excluding pixels included in the first combining layer when looking at the direction, the pixels that are most visible for each pixel coordinate can be generated as a second combining layer.

The processor may end the process of creating the combined layer if there are no pixels in the layer.

The 2D atlas image may include a transparency image and a color image generated based on pixel information of each combined layer of the multi-plane 3D data, and a layer index image having location information of the layer for the pixel.

The processor may compress the 2D atlas image.

The processor may generate the 2D atlas image as bitstream data using at least one of HEVC, H.263, and VVC.

The plurality of multi-view image data may include image data in a two-dimensional array.

The processor may generate multi-plane 3D data by generating planes for each distance in the depth direction and rearranging the pixel values and transparency of the pixels corresponding to each plane.

According to the present invention, the size of a 2D atlas image can be reduced by combining each layer of multi-plane 3D data.

Additionally, the present invention can improve compression efficiency by minimizing empty space that does not contain patch areas in a 2D atlas image.

Figure 1 is a block diagram showing an image conversion device according to an embodiment.
Figure 2 is a flowchart showing an image conversion method according to an embodiment.
Figure 3 is a diagram showing multi-view image data according to an embodiment.
Figure 4 is a diagram showing conversion into multi-plane 3D data according to an embodiment.
Figure 5 is a diagram showing a comparative example between conventional multi-plane 3D data and multi-plane 3D data according to an embodiment.
Figure 6 is a diagram illustrating a process for creating a combined layer of multi-plane 3D data according to an embodiment.
Figure 7 is a diagram showing conversion of multi-plane 3D data into a 2D atlas image according to an embodiment.
Figure 8 is a block diagram showing the configuration of a computer system according to an embodiment.

The advantages and features of the present invention and methods for achieving them will become clear by referring to the embodiments described in detail below along with the accompanying drawings. However, the present invention is not limited to the embodiments disclosed below and will be implemented in various different forms. The present embodiments only serve to ensure that the disclosure of the present invention is complete and that common knowledge in the technical field to which the present invention pertains is not limited. It is provided to fully inform those who have the scope of the invention, and the present invention is only defined by the scope of the claims. Like reference numerals refer to like elements throughout the specification.

Although terms such as “first” or “second” are used to describe various components, these components are not limited by the above terms. The above terms may be used only to distinguish one component from another component. Accordingly, the first component mentioned below may also be the second component within the technical spirit of the present invention.

The terms used in this specification are for describing embodiments and are not intended to limit the invention. As used herein, singular forms also include plural forms, unless specifically stated otherwise in the context. As used in the specification, “comprises” or “comprising” implies that the mentioned component or step does not exclude the presence or addition of one or more other components or steps.

Unless otherwise defined, all terms used in this specification can be interpreted as meanings commonly understood by those skilled in the art to which the present invention pertains. Additionally, terms defined in commonly used dictionaries are not to be interpreted ideally or excessively unless clearly specifically defined.

In this document, “A or B,” “at least one of A and B,” “at least one of A or B,” “at least one of A, B, or C,” and “at least one of A, B, or C.” Each of the phrases may include any one of the items listed with the corresponding phrase, or any possible combination thereof.

Hereinafter, embodiments of the present invention will be described in detail with reference to the attached drawings. When describing with reference to the drawings, identical or corresponding components will be assigned the same reference numerals and redundant description thereof will be omitted. .

Figure 1 is a block diagram showing an image conversion device according to an embodiment.

Referring to FIG. 1, the image conversion device according to the embodiment includes a multi-plane 3D data generator 100, a multi-plane 3D data layer combining unit 200, a 2D atlas image conversion unit 300, and a 2D image compression unit 400. ) may include.

The multi-plane 3D data generator 100 can receive multi-view image data and generate multi-plane 3D data. Multi-view image data may be M X N two-dimensional array image data or N one-dimensional array image data. In an embodiment, a two-dimensional array of image data may be used.

The multi-plane 3D data generator 100 may generate multi-plane 3D data by reconstructing multi-view image data into layers in the depth direction.

The multi-plane 3D data layer combiner 300 may generate a combined layer by combining multi-plane 3D data layers into at least one layer.

The 2D atlas image converter 300 can convert multi-plane 3D data including a combination layer into a 2D atlas image. The 2D atlas image converter 300 can generate a single image by rearranging areas where pixels exist in all layers of multi-plane 3D data.

The 2D image compression unit 400 can generate a 2D atlas image as bitstream data using one of compression methods such as HEVC, H.263, and VVC.

Figure 2 is a flowchart showing an image conversion method according to an embodiment. Here, the image conversion method according to the embodiment can be performed in an image conversion device.

Referring to FIG. 2, the image conversion device according to the embodiment can generate multi-plane 3D data reconstructed into layers in the depth direction based on a plurality of multi-view image data (S100).

Figure 3 is a diagram showing multi-view image data according to an embodiment.

As shown in FIG. 3, multi-view image data M1 may be captured from a 2D array camera array and may be images obtained from various viewing positions.

Figure 4 is a diagram showing conversion into multi-plane 3D data according to an embodiment.

As shown in FIG. 4, multi-plane 3D data (M2) generates planes for each distance in the depth direction from a specific camera position (reference cam position), and then calculates the corresponding pixel value (RGB) for each plane and the transparency of the pixel. (Alpha) can be rearranged and expressed in 3D space. In other words, multi-plane 3D data can freely generate images from arbitrary camera positions.

Returning to FIG. 2, the image conversion device according to the embodiment may generate a combined layer by combining layers of multi-plane 3D data into at least one layer (S200).

Figure 5 is a diagram showing a comparative example between conventional multi-plane 3D data and multi-plane 3D data according to an embodiment.

As shown in FIG. 5, multi-plane 3D data (M3) from which a combined layer is created can be integrated into fewer layers than multi-plane 3D data. The process of creating a combination layer will be described in detail with reference to FIG. 6.

Figure 6 is a diagram illustrating a process for creating a combined layer of multi-plane 3D data according to an embodiment.

As shown in FIG. 6, among the plurality of layers (MPI Layer #0, MPI Layer #1, MPI Layer #2, etc.) of the multi-plane 3D data (M2), the layer with the deepest depth, that is, the layer furthest from the camera Combining can be performed starting from the layer located in the camera (e.g., MPI Layer #0) and starting from the layer closest to the camera (e.g., MPI Layer #2).

In other words, the first combined layer (Aggregate Layer #0) can be created by combining the pixels that are most visible for the same pixel coordinates when looking toward the origin of the camera in the layer with the deepest depth among the layers (MPI Layer #0). .

r_L) 별로 가장 먼 계층에서 카메라 원점 방향으로 보았을 때 가장 먼저 보이는 픽셀을 하나의 계층으로 모아 결합 계층을 생성할 수 있다.In the hierarchical image of multi-plane 3D data (M2), the same pixel coordinates (x,y) of the images become points passing the same line (ray) from the camera origin, and each line (r _A

r _L ) You can create a combined layer by gathering the pixels that are most visible when viewed in the direction of the camera origin from the furthest layer into one layer.

For example, in r _A , pixel 1 of MPI Layer#2 is the first pixel encountered, in r _B , pixel 2 of MPI Layer#2, in r _D , pixel 1 of MPI Layer#1, and in r _F , pixel 1 is MPI. This is the same order as pixel 1 of Layer #0.

As described above, the result of gathering the first pixels encountered behind each line into one layer image may refer to the first combined layer (Aggregate Layer #0), which is the 0th layer on the right. Here, the multi-plane 3D data (M3) from which the combined layer is created may be composed of a color image plane and a layer index plane. A color image has the color value of the collected pixel, and a layer index image can preserve the layer index that indicates which layer of the multi-plane 3D data the pixel originally came from. Here, the color image includes a transparency image and may have a transparency value of the corresponding pixel.

In the same way, when looking toward the origin from the second deepest layer among the layers, excluding the pixels included in the first combined layer (Aggregate Layer #0), the pixels that are first visible for the same pixel coordinates are placed in the second combined layer (Aggregated Layer #0). It can be created as Aggregated Layer#1).

As described above, combined layers can be created in the order of the short distances in the depth direction for the layers. The creation of the combined layer can be repeated until there are no pixels remaining in the multiplane 3D data (M2).

In Figure 6, if three combination layers are created according to the process of creating a combination layer, all pixels of the multi-plane 3D data will be accepted.

Returning to Figure 2, the image conversion device according to the embodiment can convert multi-plane 3D data with a combination layer into a 2D atlas image (S300).

Figure 7 is a diagram showing conversion of multi-plane 3D data into a 2D atlas image according to an embodiment.

As shown in FIG. 7, the image conversion device according to the embodiment can convert the areas in which pixels exist in all combined layers of multi-plane 3D data into a 2D atlas image by rearranging them into one image.

The transparency image and color image of multi-plane 3D data (M3) with a combined layer can be condensed into one image and converted into a 2D atlas image. At this time, the hierarchical index image can be taken as is. Accordingly, the 2D atlas image may include a transparency image (M4-1), a color image (M4-2), and a hierarchical index image (M4-3).

The embodiment reduces the number of combined layers of multi-plane 3D data by collecting the pixel presence areas (patch areas) of each layer of the initial multi-plane 3D data (M2) into one layer as much as possible within the limit of not overlapping, and reduces the number of combined layers in the combined layer. By increasing the pixel presence areas, the empty area in the 2D atlas image can be greatly reduced when constructing the 2D atlas image.

Returning to Figure 2, the image conversion device according to the embodiment can compress the 2D atlas image (S400).

An image conversion device according to an embodiment can compress a 2D atlas image using one of compression methods such as HEVC, H.263, and VVC, and generate bitstream data as a result.

An image conversion device according to an embodiment can recover a multi-view image using compressed bitstream data.

For example, the image conversion device according to the embodiment can receive compressed bitstream data, restore it into a 2D atlas image, and restore it into multi-plane 3D data using the restored 2D atlas image. Images from arbitrary viewpoints can be freely created using the restored multi-plane 3D data.

The image conversion device according to the embodiment may be implemented in a computer system such as a computer-readable recording medium.

Figure 8 is a block diagram showing the configuration of a computer system according to an embodiment.

Referring to FIG. 8, the computer system 1000 according to the embodiment includes one or more processors 1010, a memory 1030, a user interface input device 1040, and a user interface output device ( 1050) and storage 1060. Additionally, the computer system 1000 may further include a network interface 1070 connected to a network.

The processor 1010 may be a central processing unit or a semiconductor device that executes programs or processing instructions stored in memory or storage. The processor 1010 is a type of central processing device and can control the entire operation of the image conversion device.

The processor 1010 may include any type of device capable of processing data. Here, 'processor' may mean, for example, a data processing device built into hardware that has a physically structured circuit to perform a function expressed by code or instructions included in a program. Examples of data processing devices built into hardware include microprocessor, central processing unit (CPU), processor core, multiprocessor, and application-specific integrated (ASIC). circuit) and FPGA (field programmable gate array), but are not limited thereto.

The memory 1030 may store various data for overall operation, such as a control program for performing an image conversion method according to an embodiment. Specifically, a number of application programs running on the image conversion device, data and commands for operation of the image conversion device may be stored in the memory.

The memory 1030 and storage 1060 may be storage media that includes at least one of volatile media, non-volatile media, removable media, non-removable media, communication media, and information transfer media. For example, memory 1030 may include ROM 1031 or RAM 1032.

According to one embodiment, a computer-readable recording medium storing a computer program, comprising the steps of generating multi-plane 3D data reconstructed into layers in the depth direction based on a plurality of multi-view image data, and the multi-plane 3D For a processor to perform a method comprising the steps of combining layers of data into at least one layer to create a combined layer, and converting multi-plane 3D data including the combined layer into a 2D atlas image. It may contain commands for:

According to one embodiment, there is a computer program stored in a computer-readable recording medium, comprising the steps of generating multi-plane 3D data reconstructed into layers in the depth direction based on a plurality of multi-view image data, and generating the multi-plane 3D data. Contains instructions for causing the processor to perform an operation including generating a combined layer by combining the layers into at least one layer, and converting multi-plane 3D data including the combined layer into a 2D atlas image. can do.

The specific implementations described in the present invention are examples and are not intended to limit the scope of the present invention in any way. For the sake of brevity of the specification, descriptions of conventional electronic components, control systems, software, and other functional aspects of the systems may be omitted. In addition, the connections or connection members of lines between components shown in the drawings exemplify functional connections and/or physical or circuit connections, and in actual devices, various functional connections or physical connections may be replaced or added. Can be represented as connections, or circuit connections. Additionally, if there is no specific mention such as “essential,” “important,” etc., it may not be a necessary component for the application of the present invention.

Therefore, the spirit of the present invention should not be limited to the above-described embodiments, and the scope of the patent claims described below as well as all scopes equivalent to or equivalently changed from the scope of the claims are within the scope of the spirit of the present invention. It will be said to belong to

100: Multi-plane 3D data generation unit
200: Multi-plane 3D data layer joiner
300: 2D atlas image conversion unit
400: 2D image compression unit

Claims (16)

Generating multi-plane 3D data reconstructed into layers in the depth direction based on a plurality of multi-view image data;
generating a combined layer by combining the layers of the multi-plane 3D data into at least one layer; and
Converting multi-plane 3D data including the combined layer into a 2D atlas image;
Including,
The step of creating the combined layer is,
generating the first visible pixels for the same pixel coordinates when looking in the direction of the camera origin in the layer with the deepest depth among the layers as a first combining layer;
Excluding pixels included in the first combining layer when looking at the camera origin direction in the second deepest layer among the layers, generating the pixels that are most visible for the same pixel coordinates as a second combining layer. Video conversion method. delete According to paragraph 1,
An image conversion method that terminates the step of generating the combined layer when there are no pixels in the layer. According to paragraph 2,
The 2D atlas image includes a transparency image and a color image generated based on pixel information of each combined layer of the multi-plane 3D data, and a layer index image having location information of the layer for the pixel. Image conversion method. According to paragraph 1,
An image conversion method further comprising compressing the 2D atlas image. According to clause 5,
An image conversion method for generating the 2D atlas image into bitstream data using HEVC, H.263, or VVC. According to paragraph 1,
An image conversion method wherein the plurality of multi-view image data includes image data in a two-dimensional array. According to paragraph 1,
An image conversion method for generating multi-plane 3D data by generating planes for each distance in the depth direction and rearranging the pixel values corresponding to each plane and the transparency of the pixels. A memory storing a control program for image conversion; and
It includes a processor that executes a control program stored in the memory,
The processor,
Generate multi-plane 3D data reconstructed into layers in the depth direction based on a plurality of multi-view image data, generate a combined layer by combining the layers of the multi-plane 3D data into at least one layer, and create a combined layer. Convert the containing multi-plane 3D data into a 2D atlas image,
The processor,
When looking at the camera origin direction from the layer with the deepest depth among the layers, the first visible pixels for each pixel coordinate are generated as a first combined layer, and when looking at the camera origin direction from the layer with the second deepest depth among the layers, An image conversion device that generates the pixels most visible for each pixel coordinate as a second combining layer, excluding pixels included in the first combining layer. delete According to clause 9,
The processor,
An image conversion device that terminates the process of creating the combined layer if there are no pixels in the layer. According to clause 10,
The 2D atlas image includes a transparency image and a color image generated based on pixel information of each combined layer of the multi-plane 3D data, and a layer index image having location information of the layer for the pixel. Image conversion device. According to clause 9,
The processor,
An image conversion device that compresses the 2D atlas image. According to clause 13,
The processor,
An image conversion device that generates the 2D atlas image into bitstream data using HEVC, H.263, or VVC. According to clause 9,
An image conversion device wherein the plurality of multi-view image data includes image data in a two-dimensional array. According to clause 9,
The processor,
An image conversion device that generates planes for each distance in the depth direction and rearranges the pixel values corresponding to each plane and the transparency of the pixels to generate multi-plane 3D data.

Images