KR20220071126A - Method and apparatus for generating plenoptic point cloud based on multi-view images - Google Patents

Abstract

The present disclosure relates to a method for generating a plenoptic point cloud. The method for generating a plenoptic point cloud according to one embodiment of the present disclosure includes the steps of: obtaining a multi-view texture image and a depth image; generating a truncated signed distance function (TSDF) volume based on the obtained multi-view texture image and the depth image; extracting geometric information of a voxelized point cloud from the generated TSDF volume; and allocating attributes for each viewpoint of the voxelized point cloud based on the extracted geometric information.

Description

Method and apparatus for generating multi-viewpoint image-based plenoptic point clouds

The present disclosure relates to a method of generating a plenoptic point cloud by using a multi-viewpoint image and a depth image and compressing the plenoptic point cloud in an efficient way.

A plenoptic point cloud is a set of plenoptic points, and a plenoptic point refers to an expression method having one piece of geometric information in which one three-dimensional point is located and a plurality of color information obtained from a multi-viewpoint image.

Recently, interest in immersive content has exploded and with the development of broadcasting equipment and image transmission technology, there is an increasing movement to actively utilize immersive content in multimedia industries such as movies and TV. One of the methods of expressing sensory content is a method of expressing using MVD data (Multi-view plus depth data). MVD data refers to data obtained by acquiring images and depth information at a specific point in time by using multiple cameras arranged at short intervals. It is widely used as a raw material for expressing immersive content because it can be modeled by geometrically restoring and synthesizing image color information and depth information as well as camera position information at the time of shooting.

MVD data has the advantage of being able to generate high-quality 3D content as the number of viewpoints increases, but transmission bandwidth may be problematic since that amount of images must be additionally transmitted during transmission. In addition, since it requires a lot of storage space in the case of having a multi-view high-definition image, there is a disadvantage in this regard.

The present disclosure provides a method and apparatus for generating a plenoptic point cloud that minimizes loss of color information in generating a plenoptic point cloud from a multi-viewpoint image.

In addition, the present disclosure provides a method and apparatus for reconstructing a multi-view image close to an original from a plenoptic point cloud.

Another object of the present disclosure is to provide a recording medium storing a bitstream generated by a method and an apparatus for generating a plenoptic point cloud.

A method of generating a plenoptic point cloud according to an embodiment of the present disclosure, the method comprising: acquiring a multi-viewpoint texture image and a depth image; generating a truncated signed distance function (TSDF) volume based on the acquired multi-viewpoint texture image and the depth image; extracting geometric information of the voxelized point cloud from the generated TSDF volume; and allocating properties for each viewpoint of the voxelized point cloud based on the extracted geometric information.

According to the present disclosure, a method and apparatus for minimizing loss of color information by using a plenoptic point cloud generated from a multi-viewpoint image and a depth image may be provided.

According to the present disclosure, there may be provided a recording medium storing a bitstream generated through voxelization of a plenoptic point cloud generated from a multi-viewpoint image and a depth image.

1 is a diagram for explaining a system of a plenoptic point cloud.
FIG. 2 is a diagram for explaining a method of expressing attribute information allocated to a plenoptic point according to a position of a viewpoint.
3 is a diagram for explaining a method of generating a plenoptic point cloud.
4 is a view for explaining a method for generating a multi-viewpoint image according to an embodiment of the present disclosure.
5 is a diagram for explaining an example of allocating attribute values for each viewpoint based on projection results of a plurality of images.
6 is a view for explaining a view-validation usage map.

Since the present invention can have various changes and can have various embodiments, specific embodiments are illustrated in the drawings and described in detail in the detailed description. However, this is not intended to limit the present invention to specific embodiments, and it should be understood to include all modifications, equivalents and substitutes included in the spirit and scope of the present invention. In describing each figure, like reference numerals have been used for like elements.

Terms such as first, second, etc. may be used to describe various elements, but the elements should not be limited by the terms. The above terms are used only for the purpose of distinguishing one component from another. For example, without departing from the scope of the present invention, a first component may be referred to as a second component, and similarly, a second component may also be referred to as a first component. and/or includes a combination of a plurality of related listed items or any of a plurality of related listed items.

When an element is referred to as being “connected” or “connected” to another element, it is understood that it may be directly connected or connected to the other element, but other elements may exist in between. it should be On the other hand, when it is mentioned that a certain element is "directly connected" or "directly connected" to another element, it should be understood that no other element is present in the middle.

The terms used in the present application are only used to describe specific embodiments, and are not intended to limit the present invention. The singular expression includes the plural expression unless the context clearly dictates otherwise. In the present application, terms such as “comprise” or “have” are intended to designate that a feature, number, step, operation, component, part, or combination thereof described in the specification exists, but one or more other features It should be understood that this does not preclude the existence or addition of numbers, steps, operations, components, parts, or combinations thereof.

Hereinafter, embodiments of the present invention will be described in detail with reference to the accompanying drawings. Hereinafter, the same reference numerals are used for the same components in the drawings, and repeated descriptions of the same components are omitted.

Hereinafter, a method for generating and expressing a multi-viewpoint image-based plenoptic point cloud will be described in detail with reference to FIGS. 1 to 6 .

1 is a diagram for explaining a system of a plenoptic point cloud.

When an object is viewed from multiple viewpoints, there may be differences in color even for the same object due to the influence of lighting or the like. This is because, in real situations, most objects have a reflective surface, which can change color depending on the location of the light source. Also, a portion of an overlapping image may exist in the multi-view image. Since the existing 3D point has only one color information per point, it is difficult to reflect these characteristics. The present disclosure presents an embodiment utilizing the concept of a plenoptic point in order to solve this problem.

1 shows an example of generating a plenoptic point cloud. A plenoptic point is one piece of geometry information expressed in three-dimensional coordinates such as X, Y, and Z in a three-dimensional space, and N pieces of RGB, YUV, etc. It is a data type that includes attribute information. A plenoptic point cloud is a set of plenoptic points, and may include at least one plenoptic point.

The plenoptic point cloud may be generated using a 2D image and depth information for each of the N input viewpoints. In this case, the 3D space may be defined through a space including all the generated points.

Here, the 2D image may mean images acquired by one or more cameras, such as a multi-view image and a lightfield image. In addition, the multi-view image may be composed of images simultaneously photographed by a plurality of cameras having different viewpoints in a specific area.

In this case, the defined 3D space may be divided into predetermined unit voxels, and points within the voxels may be merged to have one geometric information value. Also, at this time, all color information of the three-dimensional points is stored, and a plenoptic point cloud may be generated by using the information regarding the point generated from which point.

The color information of a viewpoint in which a 3D point is not generated may be inferred from color information of other viewpoints included in the same voxel. For example, color information of a viewpoint in which a 3D point is not generated may be derived using at least one of an average value, a maximum value, and a minimum value of color information of other viewpoints or points in the voxel. Also, for example, color information of a viewpoint in which a 3D point is not generated may be derived from color information of a viewpoint or point adjacent to the corresponding viewpoint or point.

Meanwhile, when a plurality of 3D points generated at one viewpoint are included in one voxel, a plenoptic point cloud may be generated by storing at least one of the average value, the maximum value, and the minimum value of the color values of the corresponding viewpoint. .

As another example, if multiple 3D points generated at one point are included in one voxel, a plenoptic point cloud can be created by storing the color information of the point with the smallest depth information or the largest depth information. have.

Here, a voxel having one or more attribute information may be referred to as a multi-attribute voxel. That is, the multi-attribute voxel may mean a plenoptic point.

FIG. 2 is a diagram for explaining a method of expressing attribute information allocated to a plenoptic point according to a position of a viewpoint.

As in the example of FIG. 2 , the attribute information assigned to the generated plenoptic point may be expressed in a two-dimensional form using θ and h according to the position of the viewpoint. Here, θ may mean an angle expressed by a real value, an integer value, etc., and h may mean a size expressed by a real value or an integer value. That is, the attribute information assigned to the plenoptic point may have two-dimensional coordinate values expressed by θ and h.

3 is a diagram for explaining a method of generating a plenoptic point cloud.

Referring to FIG. 3 , in step S301, a truncated signed distance field (TSDF) volume may be generated. Specifically, the TSDF volume may be generated based on the scanning data generated by the RGB-D camera. And, in generating the TSDF volume, a rotation matrix and/or a movement matrix for the RGB-D camera may be further reflected.

To generate a TSDF volume, tsdf _i (x) and w _i (x) can be calculated from RGB-D images acquired in units of time. And by accumulating the calculated tsdf _i (x) and w _i (x) on previous calculation results, it is possible to update TSDF _i (x) and W _i (x). The TSDF volume may be generated according to Equation (1).

In Equation 1, depth _i (pic(x)) may indicate a depth value of a location mapped with locations (X, Y, Z) in a 3D space corresponding to the center of a voxel projected on a depth image. . cam _z (x) may mean a distance between a camera and a voxel on an optical axis. t may mean a cutting value. And tsdf _i (x) may mean an sdf _i (x) value truncated to a value between -t and t.

According to an embodiment, in calculating the TSDF value, the cutting value may be the same value in all voxel positions.

According to another embodiment, in calculating the TSDF value, the cutting value may be a different value for each voxel position. Specifically, the cutting value may be different values determined according to the depth _i (pic(x)) value at the voxel position. For example, when depth _i (pic(x)) is smaller than a predefined threshold, the first truncation value may be used in calculating the TSDF value. On the other hand, when depth _i (pic(x)) is greater than a predefined threshold, the second truncation value may be used in calculating the TSDF value. Here, the first cutting value may be greater than or smaller than the second cutting value.

Here, each voxel constituting the TSDF volume may have a TSDF value between -1 and 1.

In step S303, voxelized point cloud geometry information may be extracted. Specifically, geometric information may be extracted from voxels having a W _i (x) value greater than 0 and a TSDF _i (x) value between -N and M in the entire TSDF volume. Here, N and M may be less than or equal to 1. And, N and M may be the same as or different from each other.

According to an embodiment, N and M may be 1. That is, when N and M are 1, geometric information of a voxelized point cloud may be extracted from all voxels having a TSDF _i (x) value between -1 and 1. That is, N and M may be expressed as geometric information extraction range values.

In extracting voxelized point cloud geometry information, different N and M values may be applied according to voxel positions of the TSDF volume to extract geometric information from voxels.

According to an embodiment, different geometric information extraction values may be used according to a depth _i (pic(x)) value at a voxel location. For example, when depth _i (pic(x)) is smaller than a predefined threshold value, the first geometric information extraction values (N ₁ , M ₁ ) may be used, and depth _i (pic(x)) may be When it is greater than the defined threshold value, the second geometric information extraction values N ₂ , M ₂ may be used.

Here, the first geometric information extraction value may be greater than or smaller than the second geometric information extraction value.

An index within a volume of a voxel that satisfies a condition according to a geometric information extraction range value may be extracted as geometric information of a voxelized point cloud. Here, the geometric information may be expressed in the form of (X _index , Y _index , Z _index ). In addition, X _index , Y _index , and Z _index may have positive integer values including 0.

The center position of the voxel that satisfies the condition according to the geometric information extraction range value may be extracted as geometric information of the voxelized point cloud. The center position of the voxel may be expressed as (X _C , Y _C , Z _C ). X _C , Y _C , and Z _C may be expressed in floating-point form. Alternatively, X _C , Y _C , Z _C may be expressed in the form of a scaled integer.

According to another embodiment, specific location information within a voxel that satisfies a condition according to a geometric information extraction range value may be extracted as geometric information of a voxelized point cloud. For example, coordinates in 3D space of a point moved by a specific offset value along the X-axis, Y-axis, and Z-axis based on the central position in the voxel may be extracted as geometric information of the voxelized point cloud.

When the center position of the voxel is (X _C , Y _C , Z _C ), geometric information of the point cloud may be expressed as (X _C + offset, Y _C + offset, Z _C + offset)). Offset values can be the same or different along the X, Y, and Z axes. According to an embodiment, different offset values may be used for each voxel unit. That is, in extracting geometric information of the point cloud for different voxels, offset values used for each voxel may be different from each other.

According to another embodiment, different offset values may be used for units larger than voxels. For example, different offset values may be used for each voxel in units of NxMxK. A voxel of NxMxK unit may mean a unit including N voxels in the X-axis direction, M voxels in the Y-axis direction, and K voxels in the Z-axis direction. Each of N, M, and K is a natural number greater than zero. And each of N, M, and K may be the same as or different from each other.

The offset value may be expressed as an attribute value of a voxelized point cloud. And the offset value may be encoded in the bitstream and decoded from the bitstream. The offset value may be used in the process of rendering the plenoptic point cloud. And the offset value may be used in the process of reconstructing a 2D multi-view image from the plenoptic point cloud.

In step S305, property information for each view of the voxelized point cloud may be allocated. Here, the attribute information for each view may be allocated based on the projection result of the points (voxels) constituting the voxelized point cloud on the texture image of N views. A method of projecting the plenoptic point onto a plurality of viewpoint images may be described below.

4 is a view for explaining a method for generating a multi-viewpoint image according to an embodiment of the present disclosure.

First, a 3D object may be projected as a 2D image by using the geometric information of the plenoptic point cloud (S400). In addition, a 3D object (point) may be projected as a 2D image by using the occlusion pattern information of the plenoptic point cloud. For example, when occlusion pattern information corresponding to an arbitrary viewpoint among occlusion pattern information indicates meaningless attribute information, the corresponding point may not be projected to the corresponding viewpoint. The occlusion pattern information may have a first value of '1' and a second value of '0'. For example, when the occlusion pattern information of an arbitrary viewpoint has a first value, a pixel value of a position projected to the corresponding viewpoint may be determined as attribute information of the corresponding viewpoint of the corresponding point. On the other hand, when the occlusion pattern information has the second value, the point may not be projected to the corresponding viewpoint.

In this case, when only one point is projected on the current pixel, the attribute information of the current pixel may be determined using attribute information of the corresponding viewpoint (specifically, the point) ( S405 and S410 ). On the other hand, when a plurality of points are projected on the current pixel, attribute information of the corresponding pixel may be determined using attribute information of a viewpoint (specifically, a point) closest to the camera ( S405 and S420 ).

On the other hand, with respect to attribute information in which the projection of the plenoptic point is not performed on the current pixel, attribute information of N neighboring pixels of the current pixel may be referred to. In this case, attribute information of the current pixel may be determined using attribute information of a neighboring pixel having the closest distance to the camera (S415 and S430). In this case, N may be a natural number greater than 0, for example, may have a value of 4 or 8.

In addition, if the projection of the plenoptic point is not performed on the current pixel and there is no property information or no projected points of the N neighboring pixels of the current pixel, using an interpolation method using an NxN mask on a two-dimensional image, Hole filling may be performed on the current sample (S415 and S440). In this case, N may be a natural number greater than 0, for example, may have a value of 5.

That is, the voxels of the point cloud are projected onto the N viewpoint texture images by using the geometric information (X, Y, Z) on the 3D space extracted in step S303. In addition, the properties for each viewpoint of the voxelized point cloud may be assigned based on pixel values of the image projected onto the texture image. A method of allocating attribute values for each viewpoint to the point cloud based on the projection result on the viewpoint texture image may be described below.

5 is a diagram for explaining an example of allocating attribute values for each viewpoint based on projection results of a plurality of images.

Referring to FIG. 5 , voxels of the point cloud are projected onto a texture image of N views. A process of projecting voxels of the point cloud may be expressed as Equation (2).

In Equation 2, p indicates a voxel (point) in a three-dimensional space. k indicates an internal parameter of the camera, and [R|t] indicates an external parameter of the camera. And p' indicates a voxel (point) projected on a two-dimensional image space.

Attribute values assigned to voxels of the point cloud may be assigned for each viewpoint from pixel values of an image projected onto a texture image.

A dotted line in FIG. 5 may indicate a case in which geometric information on the 3D space of the voxelized point cloud is not projected onto the image of the corresponding viewpoint.

When geometric information on the 3D space of the voxelized point cloud is not projected on the image of the corresponding viewpoint, a color value from the image of the corresponding viewpoint may not be assigned to the point cloud. When a color value is not allocated from a specific viewpoint image, a predefined value or an average value of the color values of viewpoints to which the color value is allocated may be allocated to the point cloud.

On the other hand, a solid line in FIG. 5 may mean a case in which projection is made on a texture image of a corresponding viewpoint and a color value is obtained from a pixel position of the corresponding image.

As shown in FIG. 5 , the voxelized point cloud may be projected on six viewpoint images. Here, the point cloud may not be projected on the 4th viewpoint image and the 6th viewpoint image. A color value from an image (eg, image 4 and image 6) of the corresponding viewpoint to which the point cloud is not projected may not be assigned to the voxelized point cloud.

Here, the color value may mean a value expressed in a color space such as RGB, YUV, HSV, LAB, and YCoCg. In addition, information about the color space may be expressed as additional attribute information. Information on the color space may be encoded in the bitstream, and may be decoded and used in the process of reproducing the plenoptic point cloud. Alternatively, the information on the color space may be used in the process of reconstructing a 2D multi-view image from the plenoptic point cloud.

Here, the color value may mean a transform coefficient value indicated in the frequency domain. That is, the color value may mean a value expressed by one DC conversion coefficient and N-1 AC conversion coefficients.

Information indicating whether a valid color value has been allocated from a corresponding point of time may be expressed as additional attribute information of a point.

For example, as a result of generating the plenoptic point cloud according to FIG. 3 , after the color value of the point is expressed, the validity of the color value at each viewpoint can be expressed as 1-bit true/false. have. For example, the attribute regarding the validity of the color value may have an additional attribute value form such as a (111010) value. The attribute regarding the validity of the color value may be encoded/decoded in the bitstream and used in the process of reconstructing a 2D multi-view image from the plenoptic point cloud.

For example, after the color values of the corresponding point are expressed, a view number or view index indicating valid views may be provided as an attribute value of an additional point. The corresponding property may be encoded/decoded and used in a process of reconstructing a 2D multi-view image from the plenoptic point cloud.

Information indicating whether an attribute value of a corresponding view is valid may be imaged as an image having a size of WxH. In addition, information indicating whether an attribute value of a corresponding view is valid may be encoded/decoded through a 2D video codec. The image information may be expressed as a view-valid occupancy map. The viewpoint-validation usage map may be expressed as follows.

6 is a diagram for explaining a view-validation usage map.

Referring to FIG. 6 , in a method of projecting 3D spatial information into a 2D image and encoding/decoding image information, the view-validation use map indicates whether the attribute value of a specific pixel position of the view is valid. can

For example, as shown in FIG. 5 , an atlas may be generated by orthogonally projecting a plenoptic point cloud generated from six viewpoints onto a hexahedron of a bounding box. Here, the bounding box means a box including the entire plenoptic point cloud. In addition, the encoder/decoder may encode/decode the atlas with a 2D video codec such as a V-PCC codec.

Here, by encoding/decoding a binary image having a size of WxH for six views as shown in FIG. 6 , it is possible to determine whether an attribute value of a corresponding view is valid.

A portion marked in black in the view-validation usage map shown in FIG. 6 may represent that an attribute value of a corresponding view is invalid at a pixel position of the corresponding portion.

Example methods of the present disclosure are expressed as a series of operations for clarity of description, but this is not intended to limit the order in which the steps are performed, and if necessary, each step may be performed simultaneously or in a different order. In order to implement the method according to the present disclosure, other steps may be included in addition to the illustrated steps, other steps may be excluded from some steps, or additional other steps may be included except some steps.

Various embodiments of the present disclosure do not list all possible combinations, but are intended to describe representative aspects of the present disclosure, and matters described in various embodiments may be applied independently or in combination of two or more.

In addition, various embodiments of the present disclosure may be implemented by hardware, firmware, software, or a combination thereof. For implementation by hardware, one or more Application Specific Integrated Circuits (ASICs), Digital Signal Processors (DSPs), Digital Signal Processing Devices (DSPDs), Programmable Logic Devices (PLDs), Field Programmable Gate Arrays (FPGAs), general purpose It may be implemented by a processor (general processor), a controller, a microcontroller, a microcontroller, or the like.

The scope of the present disclosure includes software or machine-executable instructions (eg, operating system, application, firmware, program, etc.) that cause an operation according to the method of various embodiments to be executed on a device or computer, and such software or and non-transitory computer-readable media in which instructions and the like are stored and executable on a device or computer.

Claims (1)

A method for generating a plenoptic point cloud, comprising:
acquiring a multi-viewpoint texture image and a depth image;
generating a truncated signed distance function (TSDF) volume based on the acquired multi-viewpoint texture image and the depth image;
extracting geometric information of the voxelized point cloud from the generated TSDF volume; and
and allocating properties of the voxelized point cloud for each viewpoint based on the extracted geometric information.

Images