WO2019194522A1

WO2019194522A1 - Method and device for processing duplicate points

Info

Publication number: WO2019194522A1
Application number: PCT/KR2019/003844
Authority: WO
Inventors: 우지환; 유성열; 오영호
Original assignee: 삼성전자 주식회사
Priority date: 2018-04-06
Filing date: 2019-04-02
Publication date: 2019-10-10
Also published as: KR20190117287A

Abstract

Various embodiments of the present disclosure relate to a method and device for processing duplicate points, the device for processing duplicate points compressing and restoring point clouds included in a three-dimensional image, detecting, as duplicate points, from among the restored point clouds, a plurality of point clouds which have the same location value and different color values, measuring similarity between the color values of the duplicate points, comparing the measured similarity and a threshold value, and determining, as a color value corresponding to the location value, the average value of the color values of the duplicate points on the basis the comparison result.

Description

Duplicate Point Processing Method and Device

Various embodiments of the present disclosure relate to a method and an apparatus for processing duplicate points.

A large amount of three-dimensional data is represented by a point cloud. A point cloud is a collection of vast amounts of points. The point cloud is used to represent one point on the 3D, and may be expressed in a vector form having both a position value and a color value. For example, a point cloud may contain a position value indicated by “X, Y, Z” and a color value indicated by “R, G, B” as “(X, Y, Z, R, G, B ) ”.

Since point clouds representing three-dimensional data occupy a considerable amount of memory resources, a point cloud compression method is required. However, since a general point cloud compression method requires a lot of time to generate and spread a new codec, a method of utilizing an existing codec (eg, 2D video compression method) has been proposed.

Meanwhile, when restoring the compressed point cloud, a plurality of points having the same position value and different color values may occur. These multiple points are generally referred to as duplicate points. Since overlapping points have the same position value but different color values, there is a need for a method for which color value should be assigned to the position value.

According to various embodiments of the present disclosure, a method and apparatus for processing overlapping points may be provided.

According to various embodiments of the present disclosure, when duplicate points having the same location value and different color values occur during compression and decompression of a point cloud, a method and an apparatus for determining a color value to be assigned to the location value are provided. can do.

Method according to an embodiment of the present disclosure; In the overlapping point processing method, a process of compressing and restoring point clouds included in a 3D image, and overlapping a plurality of point clouds having the same position value and different color values among the restored point clouds ) Detecting as points, measuring similarity between color values of the overlapping points, comparing the measured similarity with a threshold value, and calculating an average value for the color values of the overlapping points based on the comparison result. And determining the color value corresponding to the position value.

An apparatus according to an embodiment of the present disclosure; In the redundant point processing apparatus, a decoder for compressing and restoring point clouds included in a 3D image, and a plurality of point clouds having the same position value and different color values among the restored point clouds are duplicated. ) Detects as points, measures similarity between color values of the overlapping points, compares the measured similarity with a threshold value, and compares an average value of the color values of the overlapping points to the position value based on the comparison result. It includes a control unit for determining the corresponding color value.

According to various embodiments of the present disclosure, when overlapping points having the same location value and different color values are generated during the compression and reconstruction of the point cloud, it is possible to efficiently determine the color value to be assigned to the location value. In addition, duplicate points may be processed in various ways by the transmitting side or the receiving side.

The effects obtainable in the present disclosure are not limited to the mentioned effects, and other effects not mentioned will be clearly understood by those skilled in the art from the following description.

1 is a diagram illustrating a case where overlapping points are generated according to one embodiment of the present disclosure;

2 is a diagram illustrating a problem according to color selection when overlapping points are generated according to an embodiment of the present disclosure;

3 is a diagram illustrating an example of a method for processing duplicate points in consideration of temporal similarity according to an embodiment of the present disclosure;

4A illustrates an example of tracking similarity of point clouds between a plurality of frames according to an embodiment of the present disclosure.

4B is an exemplary view of tracking the temporally changed position of the point clouds in one frame according to an embodiment of the present disclosure.

5 illustrates point clouds with spatial similarity according to an embodiment of the present disclosure;

6 is a diagram illustrating an example of a method for processing overlapping points in consideration of spatial similarity according to an embodiment of the present disclosure;

7 is a diagram illustrating an example of a method for processing overlapping points in consideration of space-time similarity according to an embodiment of the present disclosure;

8 illustrates a method of selecting color values of overlapping points in consideration of spatiotemporal similarity according to an embodiment of the present disclosure;

9 is a diagram illustrating an example of a method for processing overlapping points in consideration of color similarity between overlapping points according to an embodiment of the present disclosure;

10A is a view briefly illustrating an internal configuration of a receiver according to an embodiment of the present disclosure;

10B illustrates an operation of a receiver according to an embodiment of the present disclosure.

11A is a diagram schematically illustrating an internal configuration of a transmitter according to an embodiment of the present disclosure;

11B illustrates an operation of a transmitter according to an embodiment of the present disclosure.

Hereinafter, various embodiments of the present disclosure will be described in detail with reference to the accompanying drawings. In the following description of various embodiments of the present disclosure, when it is determined that a detailed description of related known functions or configurations may unnecessarily obscure the subject matter of the present disclosure, the detailed description thereof will be omitted. In addition, terms to be described below are terms defined in consideration of functions in various embodiments of the present disclosure, and may vary according to intentions or customs of a user or an operator. Therefore, the definition should be made based on the contents throughout the present disclosure.

1 is a diagram illustrating a case where overlapping points are generated according to an embodiment of the present disclosure.

Referring to FIG. 1, a point cloud compression process includes compressing and restoring a plurality of point clouds into at least one point cloud. When this point cloud compression process is performed,

point clouds

101, 102 and 103 having different position values and different color values before compression are point clouds having the same position value and different color values after compression. It may be restored to (104) 105 (106). As such,

point clouds

104, 105, 106 having the same location value and different color value may be referred to as duplicate points.

The duplicated points may be generated by quantization (for example, the conversion of a real value into an integer value) performed in the point cloud compression and decompression process and signal distortion due to noise generated in the transmission process. For example, position 1 of the point cloud 101 has a value of “100.27”, position 2 of the point cloud 102 has a value of “100.43”, and position 3 of the point cloud 103 has a value of “100.11”. In the compression process, the values of positions 1 to 3 are quantized to “100”. Accordingly, the position values of the

point clouds

101, 102, 103 that were different before compression may all be the same position value (position A) after compression.

However, since the

point clouds

101, 102, 103 have different color values from each other, a problem may arise regarding which color should be assigned to the location A during the restoration process. For example, as shown in FIG. 2, when duplicate

points

104, 105, 106 are restored to one point cloud at location A, any one of color 1, color 2 and color 3 Problems may arise such as whether to be selected as the color of the point cloud to be restored. In consideration of this, the present disclosure proposes various embodiments in which a color to be assigned to a corresponding position can be determined when overlapping points occur. Hereinafter, the above various embodiments will be described in detail as a method and a device for processing duplicate points.

As an example of the duplicate point processing method, there is a duplicate point processing method in consideration of temporal similarity. Duplicate point processing method considering temporal similarity tracks the color values of the point clouds in the frame every hour, and averages the color values of the tracked point clouds (eg, simple average and weighted average). And a color value to be reconstructed for duplicate points (hereinafter, referred to as a duplicate point color value) as a calculation method). In addition, the overlapping point processing method considering the temporal similarity determines a duplicate point color value as a color value for a point cloud at the same position as the position of the duplicate points among the color values of the tracked point clouds for time consistency. It includes a method.

3 is a diagram illustrating an example of a method for processing overlapping points in consideration of temporal similarity according to an embodiment of the present disclosure.

Referring to FIG. 3, a 2D feature tracking process 310 may be performed to determine a duplicate point color value at T (n) representing a frame corresponding to time n. The two-dimensional feature tracking process 310 includes tracking features for a predetermined number k of two-dimensional frames before time n. That is, the two-dimensional feature tracking process 310 tracks color values for point clouds in each of T (n-1), which is a frame corresponding to time n-1, and T (nk), which is a frame corresponding to time nk. Process.

If color values for point clouds included in k frames are tracked, a process 320 of selecting a duplicate point color value in T (n) may be performed using the tracked color values of each frame. . The process 320 of selecting a duplicate point color value may be performed based on various methods.

For example, selecting 320 a duplicate point color value may include selecting 330 a duplicate point color value using a weighted average. If n is 5 and k is 3, the duplicate point color value of T (5) is equal to the position of the duplicate points of T (5) in T (4), T (3), and T (2), respectively. It may be selected using the color value of the point cloud having (or pointing to the same object as the object to which the overlapping points point). Here, a weight value may be applied to the color value of the point cloud having the same position as that of the overlapping points of T (5) in T (4), T (3), and T (2), respectively. A weight value having a larger value may be applied as belonging to a frame that is close in time.

That is, the largest weight value is applied to the color value of the point cloud having the same position as that of the overlapping points of T (5) in T (4), and the position of the overlapping points of T (5) in T (3). The next largest weighting value is applied to the color values of the point cloud with the same position, and the smallest weighting value is applied to the color value of the point cloud with the same position as the position of the overlapping points of T (5) in T (2). Can be applied. The duplicate point color value of T (5) is selected based on the average value (ie, the weighted average value) of the color values of each point cloud of T (4), T (3), and T (2) to which the weight value is applied. Can be. For example, the duplicate point color value of T (5) is determined as a weighted average value, or the color value closest to the weighted average value among the color values of duplicate points of T (5) is the duplicate point color value of T (5). Can be selected.

Selecting a duplicate point color value 320 may include selecting a color value for a duplicate point position based on a previous feature color 340. The previous feature color may indicate a color value included in a frame immediately before or corresponding to a time corresponding to the current frame. For example, a duplicate point color value of T (5) has a point cloud that has the same position as the position of duplicate points of T (5) in T (4) (or points to the same object as the object to which the duplicate points point). A color value close to the color value of the point cloud of T (4) among the color values of the overlapping points of T (5) may be selected as the duplicate point color value of T (5).

Selecting a duplicate point color value 320 may include selecting a duplicate point color value 350 based on a median value of color values of previous frames. For example, the color value for the duplicate point position of T (5) has the same position as the duplicate point position of T (5) in each of T (4), T (3), T (2) (or duplicate point). May be an intermediate value of the color values of the point cloud).

For example, the color value of the point cloud of T (4) is “250”, the color value of the point cloud of T (3) is “230”, and the color value of the point cloud of T (2) is “240”. In this case, “240” may be determined as a duplicate point color value of T (5) as an intermediate value. Alternatively, the color value closest to “240” among the color values of the overlapping points of T (5) may be selected as the overlapping point color value of T (5).

Selecting the duplicate point color value 320 may include selecting the duplicate point color value 360 to have a feature color similar to the surrounding color in the current frame. For example, a color similar to the color value for the point cloud located within a preset distance from the duplicate point in the current frame T (n) may be determined as the duplicate point color value of T (n).

4A and 4B are diagrams illustrating point clouds tracked over time according to an embodiment of the present disclosure.

First, FIG. 4A illustrates an example of tracking similarity of point clouds between a plurality of frames according to an embodiment of the present disclosure.

Similarity of point clouds between frames may be tracked as shown in FIG. 4A to use the overlapping point processing method in consideration of temporal similarity. In FIG. 4A, point clouds included in T (1) and point clouds included in T (2) are tracked to indicate a correspondence relationship between point clouds having high similarity (for example, 410 and 420). have.

Unlike FIG. 4A, in order to use the overlapping point processing method in consideration of temporal similarity, the temporally changed position of the point clouds may be tracked as shown in FIG. 4B.

4B is an exemplary diagram of tracking the temporally changed position of the point clouds within one frame according to an embodiment of the present disclosure.

4B shows a result of tracking how the positions of the point clouds included in T (2) have changed compared to the previous frame T (1). For example, the position indicated by reference numeral 430 indicates the position of the point cloud at T (2), and the position indicated by reference numeral 440 indicates the position of the point cloud at T (1).

Next, a duplicate point processing method in consideration of spatial similarity according to an embodiment of the present disclosure will be described.

5 is a diagram illustrating point clouds having spatial similarity according to an embodiment of the present disclosure.

Referring to FIG. 5, in the method of processing overlapping points considering spatial similarity, when overlapping

points

510, 512, and 514 having different color values at the same location occur, neighboring point clouds 500 and 502 ( 504) 506 (508) 510 includes a method for determining duplicate point color values in view of the color values.

For example, the duplicate point color value may be a color value similar to the color value of at least one of the neighbor point clouds 500, 502, 504, 506, 508, 510 for spatial consistency. For example, it may be determined as a color value having the smallest difference from the color value of at least one neighboring point cloud among the color values of the overlapping points.

At least one of the neighbor point clouds 500, 502, 504, 506, 508, 510 can be selected according to a predetermined criterion, for example,

duplicate points

510, 512, 514. It may be at least one neighbor point cloud that is closest to the location of or within a preset distance. Here, the distance between the point clouds may be measured using a method of calculating an Euclidian distance, a geodesic distance, or the like.

Meanwhile, a method of designating a duplicate point color by calculating a weighted average value considering distances between

neighboring point clouds

500, 502, 504, 506, 508, 510 and overlapping points may also be used. This will be described in detail with reference to FIG. 6.

6 is a diagram illustrating an example of a method for processing overlapping points in consideration of spatial similarity according to an embodiment of the present disclosure.

The method illustrated in FIG. 6 may be performed by a receiver, a controller included in the receiver, at least one processor, or a redundant point processing apparatus included in the receiver or physically separated from the receiver. Hereinafter, it will be described as an example that is performed in the overlapping point processing apparatus for convenience of description.

Referring to FIG. 6, when the overlapping point processing apparatus detects that overlapping points have occurred in step 600, the overlapping point processing apparatus calculates a distance between the overlapping points and at least one neighboring point cloud in step 602. In operation 604, the overlapping point processing apparatus selects nearby N point clouds within a predetermined distance. The number N of point clouds to be selected may be preset, represent the number of all point clouds within a preset distance, or may indicate the number of any point clouds within a preset distance.

The redundant point processing apparatus calculates an average value of color values of the N point clouds in step 606. In operation 608, the overlapping point processing apparatus selects a color value having a high similarity (the smallest difference in the color values) from the calculated average value among the color values of the overlapping points. In operation 610, the overlapping point processing apparatus calculates an average value of the average value calculated in operation 610 and the color value of the overlapping point, and determines the overlapping point color value.

On the other hand, the overlapping point processing apparatus may omit step 608 and proceed from step 606 to step 610. In this case, the overlapping point processing apparatus may determine the overlapping point color value by calculating the average value calculated in operation 606 and the average value of the color values of the overlapping points.

In operation 606, the overlapping point processing apparatus may calculate a weighted average by applying a weight value to each of the N point clouds. In this case, a weight value considering the distance of each of the overlapping points and the neighboring point clouds may be applied. For example, a higher weight value may be applied to the neighboring point cloud where the distances to the overlapping points are closer.

Next, a method of processing duplicate points in consideration of space-time similarity according to an embodiment of the present disclosure will be described.

The overlapping point processing method considering the spatiotemporal similarity refers to a method in which both the overlapping point processing method considering the temporal similarity and the overlapping point processing method considering the spatial similarity are considered.

For example, when determining a duplicate point color value of T (5), the duplicate point processing apparatus first weights the color values of the point cloud included in each of T (4), T (3), and T (2). Apply to determine the weighted average value (temporal similarity). The duplicate point processing apparatus checks the color values of neighboring point clouds adjacent to the duplicate point at T (5) (consides spatial similarity), and selects the color value closest to the previously determined weighted average value among the identified color values. This can be determined by the duplicate point color value of.

Another example of a method of processing overlapping points in consideration of space-time similarity is a method of using auxiliary information. This will be described in detail with reference to FIG. 7.

7 is a diagram illustrating an example of a method for processing overlapping points in consideration of space-time similarity according to an embodiment of the present disclosure.

Referring to FIG. 7, when the overlapping point processing apparatus detects that the overlapping points have occurred in step 700, the overlapping point processing apparatus refers to additional information corresponding to the color values of each of the overlapping points in step 702. The additional information may include at least one of a value of one of R, G, and B, a sum of R, G, and B, a Y value of a corresponding point cloud before quantization, and an intensity value (luminance). Can be.

The overlapping point processing apparatus compares the additional information of the point cloud neighboring the overlapping points with the additional information corresponding to the color value of each of the overlapping points in step 704. In operation 706, the overlapping point processing apparatus determines a color value having additional information having the smallest difference value and additional information of the neighboring point cloud among the color values of the overlapping points as the duplicate point color value.

The method illustrated in FIG. 7 will be described in detail with reference to FIG. 8.

8 is a diagram illustrating a method of selecting color values of overlapping points in consideration of space-time similarity according to an embodiment of the present disclosure.

Referring to FIG. 8, the overlapping point processing apparatus detects additional information corresponding to each of the overlapping

points

810, 820, and 830 having a color value 1, a color value 2, and a color value 3. For example, the overlapping point processing apparatus may add one of R, G, and B values, R, G, and B as additional information corresponding to each of the overlapping

points

810, 820, and 830, before quantization. It may include at least one of the Y value and the brightness value of the point cloud.

(R1, G1, B1) shown in FIG. 8 represents one of R, G, and B, or R, G, and B of the overlapping point 810, respectively, and (R2, G2, B2) represents the overlapping point. 820 represents one of R, G, and B or R, G, and B, and (R3, G3, B3) represents one of R, G, B, or R, G, The value which totaled B is shown. Y1, Y2, and Y3 represent Y values of the overlapping

points

810, 820, and 830, respectively.

The overlapping point processing apparatus also detects additional information about at least one neighboring point cloud adjacent to the overlapping

points

810, 820, and 830. Additional information on at least one neighboring point cloud may also include at least one of a value of one of R, G, and B, a sum of R, G, and B, a Y value, and a brightness value of the corresponding point cloud before quantization. Can be.

The overlapping point processing apparatus compares the additional information of each of the overlapping

points

810, 820, and 830 with the additional information of at least one neighboring point cloud, and determines the overlapping point color based on the comparison result. For example, “250”, “35”, and “80” are detected as the additional information R values of each of the overlapping

points

810, 820, and 830, and the additional information R value of the neighbor point cloud is “200”. If detected as, the overlapping point processing apparatus may determine the color of the overlapping point 810 having a value of “250” closest to “200” as the overlapping point color.

Next, a method of processing duplicate points in consideration of color similarity between duplicate points will be described.

9 is a diagram illustrating an example of a method of processing overlapping points in consideration of color similarity between overlapping points according to an embodiment of the present disclosure.

Referring to FIG. 9, when duplicate points are detected, the duplicate point processing apparatus stores information on the duplicate points detected in operation 900. The overlapping points may be stored in a preset unit (eg, a voxel unit).

The overlapping point processing apparatus measures similarity between colors of the overlapping points based on a preset method in operation 902. The preset method may include a method of calculating a similarity including a covariance of color values R, G, and B of overlapping points.

The overlap point processing apparatus determines whether or not the similarity measured in step 904 is smaller than the threshold. When the measured similarity is less than the threshold, the overlapping point processing apparatus proceeds to step 906 to calculate an average value for the color values of the overlapping points, and determines the calculated average value as the overlapping point color.

In contrast, when the similarity measured in step 904 is greater than or equal to a threshold, the overlapping point processing apparatus selects neighboring points adjacent to the overlapping points in step 910. The selected neighbor point represents a point cloud existing at a position separated by a threshold distance from the position of the overlapping points, and the threshold distance may represent the maximum distance that can be allowed as the neighbor point.

The overlapping point processing apparatus detects point clouds located between the overlapping points and the selected neighbor point, and calculates an average value of color values of the detected point clouds in step 912. In detail, the overlapping point processing apparatus applies a weight value to each of the color values of the detected point clouds, and calculates an average value for the color values to which the weight value is applied as a weighted average value. Here, the weight value applied to each of the detected point clouds may have a larger value as the corresponding point cloud is closer to the overlapping points.

In operation 914, the duplicate point processing apparatus determines the duplicate point color determination mode. When the duplicate point color determination mode is determined as mode 1, the overlap point processing apparatus proceeds to step 908 to determine the calculated weighted average value as the duplicate point color value. When the duplicate point color determination mode is determined as mode 2, the overlap point processing apparatus proceeds to step 916 to determine a color value closest to the calculated weighted average value among the color values of the duplicate points as the duplicate point color value.

Tables 1 and 2 below are syntaxes indicating generally used indirect coding (Table 2 is syntax following Table 1), and Table 3 is syntax representing direct coding. Indirect coding shown in Tables 1 and 2 includes a process of projecting 3D point clouds in 2D (2D), and then restoring them back to 3D. In Table 1 and Table 2, the (x, y, z) values of the three-dimensional point cloud to be restored are the pixel position x of the two-dimensional projected image (image), the pixel position y, and the (x, y of the two-dimensional projected image. Is used as the brightness value Y (x, y) in.

In the direct coding syntax shown in Table 3, unlike the indirect coding syntax shown in Tables 1 and 2, the (x, y, z) values of the three-dimensional point clouds to be restored are Y, U, and It is stored in the V channel. Therefore, the restored x becomes Y (x, y), y becomes U (x, y), and z becomes V (x, y).

When indirect coding is used, large segments are projected and stored in the projected image, and projected using the direct coding method to avoid duplication of points projected at the same position (x, y) of the next projected image. The (x, y, z) values are stored in any empty space of the captured image. At this time, since the position of the stored image itself is not related to the actual three-dimensional space, x, y, z values are recorded in the Y, U, and V channels, respectively.

Tables 4 and 5 below express the method shown in FIG. 9 in syntax, and Tables 6 to 9 show semantics of the method shown in FIG. Table 5 is a syntax following Table 4, and Tables 6 to 9 are merely expressed by dividing one syntax for convenience. The phrases described in Table 4 and Table 5 may be used in addition to the phrases described in Table 3. For example, after the phrases described in Table 3, the phrases of Tables 4 and 5 may be added and used.

Meanwhile, in addition to the methods described above, the method of processing the duplicated point displays information on the duplicated points at the time of decoding when duplicated points are generated, and records that the restored point is a duplicated point to define the weight of another point in the future. Various methods may be performed, such as a method of lowering, using an average value of color values of all overlapping points, and randomly selecting one of the color values of the overlapping points for quick and easy calculation.

Next, an internal configuration of a receiver according to an embodiment of the present disclosure will be described. The receiver may be the same device as the redundant point processing apparatus described above.

10A is a diagram illustrating an internal configuration of a receiver according to an embodiment of the present disclosure.

Referring to FIG. 10A, a receiver includes an uncapsulation processor 1010, a decoder 1020, an unpacking processor 1030, a 3D image projection processor 1040, a duplicate point processor 1050, a 3D update processor 1060, and a display. Section 1070 and the like.

The decapsulation processing unit 1010 performs a decapsulation process on the received data. A location value, a color value, an occupancy map, other attribute information, auxiliary information, and the like for each point cloud may be obtained through an unencapsulation process.

The decoder 1020 performs decoding on each information obtained through the decapsulation process. The packed 2D image may be reconstructed through the operation of the decoder 1020.

The unpacking processor 1030 performs an unpacking operation on the decoded data (ie, the packed 2D image). The 2D image may be reconstructed through the unpacking operation. The unpacking operation may include performing an inverse transformation of the deformation and / or repositioning of the plurality of regions of the projected 2D image at the transmitter. To this end, the transmitter and the receiver may share information about a packing method in advance.

The 3D image projection processing unit 1040 projects the unpacked 2D image into a 3D image.

The 3D image projection processing unit 1040 may use reverse projection of the projection used to project the 2D image into the 3D image from the transmitter to the 2D image, but is not limited thereto. The receiver may generate the 3D image by projecting the unpacked 2D image into the 3D image.

The duplicate point processor 1050 performs an operation of determining a duplicate point color based on at least one of the duplicate point processing methods according to various embodiments of the present disclosure. The 3D image may be converted into a 2D image for overlapping point processing, and the overlapping point processing unit 1050 may output the duplicated point processed 2D image.

The 3D update processor 1060 restores the duplicated point processed 2D image back to the 3D image. The display unit 1070 displays at least a part of the updated 3D image through the display device. For example, the display unit 1070 may extract and render data corresponding to a current field of view (FOV) of the 3D image.

Meanwhile, although the receiver includes a plurality of components in FIG. 10A, other components except for the display unit 1070 may be integrated as one physical component such as a controller or a processor. In addition, although not shown in FIG. 10A, the receiver may further include a transceiver capable of communicating with the transmitter, a memory capable of storing various information related to the operation of the receiver, and the like.

Referring to FIG. 10B, when a signal is received from a transmitter, the receiver demultiplexes (DeMux) the received signal in step 1012 and performs an unencapsulation process. In addition, in

step

1021, 1023, 1025, 1027, and 1031, the receiver acquires a location value, color value, occupancy map, other attribute information, and additional information about each point cloud acquired through the decapsulation process. auxiliary) Decode information and the like.

The receiver may reconstruct the packed 2D image based on the geometry frame 1022, the color frame 1024, the occupied map frame 1026, and other attribute information frame 1028 generated as a result of the decoding. Subsequently, the receiver performs an unpacking operation on the packed 2D image in operation 1032, and generates a 3D image by projecting the unpacked 2D image into a 3D image in operation 1042.

If duplicate points are found in the 3D image generation process, the receiver performs processing on the duplicate points in step 1052. That is, the receiver performs an operation of determining the duplicate point color based on at least one of the duplicate point processing methods according to various embodiments of the present disclosure.

In step 1062, the receiver reconstructs the image converted into 2D in the process of overlapping points to 3D, and displays the point clouds constituting the 3D image in step 1072.

In the above, the method and apparatus for processing duplicate points on the receiver side have been described. However, the method and apparatus for processing duplicate points on the transmitter side may be considered. The transmitter may be a server or other common device for providing data or services related to 3D images. An internal configuration of the transmitter is as shown in FIG. 11A as an example.

11A is a diagram schematically illustrating an internal configuration of a transmitter according to an embodiment of the present disclosure.

Referring to FIG. 11A, a transmitter includes a 3D image generating or receiving unit 1110, a segmentation processing managing unit 1120, a 2D image projection processing unit 1130, a duplicate point processing unit 1140, an encoder 1150, an encapsulation processing unit 1160, and the like. It includes.

The 3D image generating or receiving unit 1110 generates a 3D image, receives a 3D image from a network, or receives a 3D image from a user. The 3D image generating or receiving unit 1110 may generate 3D images by stitching images from a plurality of cameras photographed from various directions. In addition, data about a 3D image that has already been created may be received from an external device or a network. The 3D image may be rendered in the form of a sphere, cube, cylinder, or octahedron, but the above-mentioned 3D image is merely an example, and in the art Of course, various types of 3D images available may be generated or received.

The segmentation processing manager 1120 includes a LoD determiner 1121, a voxelization processor 1123, a segmentation processor 1125, and the like. In FIG. 11A, the LoD determiner 1121 and the voxelization processor 1123 are illustrated to be disposed in front of the segmentation processor 1125, but the order of arrangement may be variously changed.

The LoD determiner 1121 determines a level of detail (LoD) value with respect to the received 3D image, and determines an L value that is a size of a voxel based on the determined LoD value. LoD values can be expressed as "precision". The L value may be a set value. If the LoD value is 0, the segmentation processor 125 may immediately perform the operation without the Voxelization processor 1123.

The LoD determiner 1121 may check the bandwidth in the point cloud capturing situation and determine the LoD value according to the identified bandwidth. In addition, the LoD determiner 1121 may determine the LoD value according to the location of interest of the user (or the space viewed by the user or the location of the gaze of the user (ie, the distance from the location viewed by the user) to the object.

The voxelization processor 1123 performs a voxelization process. For example, the voxelization processor 1123 may divide one cube into a plurality of voxels based on the L value, and perform an operation of reducing the number of point clouds in order to increase compression efficiency.

The segmentation processor 1125 obtains segmentation related information by using one or more points positioned around a center point in one voxel, and performs segmentation using the obtained segmentation related information. The segmentation related information may include at least one of color, texture, position, normal information, and other attribute information. Here, the texture means information related to the pattern, and the normal information means information related to the plane. For example, normal information means information indicating whether or not it is a plane.

The 2D image projection processing unit 1130 projects the 3D image into a 2D image. To project a 3D image into a 2D image, any one of required projection methods (ERP), octahedron projection (OHP), cylinder projection, cube projection, and various projection methods available in the art may be used.

The duplicated point processor 1140 learns the geometric arrangement of point clouds in which duplicated points frequently occur with respect to the projected 2D image by using machine learning in advance. The duplicated point processor 1140 generates, as additional information, information related to point clouds having a high probability of becoming a duplicated point based on the learned data (eg, distribution information of corresponding point clouds). The additional information may be generated in various forms such as set for each point cloud, set for all point clouds, or set for point clouds having a high probability of becoming duplicate points. In addition, additional bits may be allocated for transmitting additional information such that the geometry information for point clouds that are likely to be duplicate points are encoded with high precision.

Encoder 1150 encodes the projected 2D image. The encoder 1150 may encode two or more regions of the plurality of regions of the packed 2D image. As a possible embodiment, the entire packed 2D image may be encoded. The encoder 1150 may encode using a known encoding method for a 2D image.

The encapsulation processing unit 1160 encapsulates the encoded 2D image related data and additional information. Encapsulation may mean processing the encoded data to conform to a predetermined transmission protocol through processing such as dividing the encoded data and adding a header to the divided data. Data encapsulated by the encapsulation processing unit 1160 may be transmitted to the receiver. At this time, along with the encapsulated data or separately from the data, additional data related to the data, data (eg, metadata) necessary for reproducing the data, may be transmitted.

Meanwhile, although FIG. 11A illustrates that the transmitter includes a plurality of components, the plurality of components may be integrated as one physical component such as a controller or a processor. In addition, although not shown in FIG. 11A, the transmitter may further include a transceiver capable of communicating with a receiver, a memory capable of storing various information related to the operation of the transmitter, and the like.

Referring to FIG. 11B, the transmitter acquires a 3D image of point clouds in step 1112, and performs a segmentation processing operation on the 3D image obtained in step 1127. Specifically, the transmitter determines a LoD value for the obtained 3D image, and determines an L value that is the size of the voxel based on the determined LoD value. The transmitter divides one cube into a plurality of voxels based on the L value and reduces the number of point clouds to increase compression efficiency. Subsequently, the transmitter acquires segmentation related information using one or more points located around the center point in one voxel, and performs segmentation using the obtained segmentation related information.

The transmitter projects the 3D image into the 2D image in step 1132. The transmitter learns the geometric arrangement of point clouds in which overlapping points frequently occur with respect to the projected 2D image in operation 1142. The transmitter generates, as additional information, information related to point clouds having a high probability of becoming a duplicate point based on the learned data (for example, distribution information of corresponding point clouds).

The transmitter generates an occupancy map in step 1152 and packs the projected 2D image in step 1162. Packing may mean modifying and / or rearranging at least some of the plurality of regions constituting the projected 2D image to generate a new 2D image (ie, a packed 2D image). Here, the deformation of the region may be resizing, transforming, rotating and / or re-sampling (e.g., upsampling, downsampling, differential sampling based on location within the region). And the like. This packing scheme may be referred to as region-wise packing.

The transmitter encodes the information related to the packed 2D image at 1172, 1174, 1176, and 1178 to the geometric information 1171, the color information 1171, the occupancy map 1175, and the other attribute map 1177, respectively. In operation 1190, the transmitter allocates bits for transmitting the additional information 1180 for the redundant point processing, and multiplexes the additional information with the encoded information and transmits the additional information.

Meanwhile, in the detailed description of the present disclosure, specific embodiments have been described. However, various modifications may be possible without departing from the scope of the present disclosure. Therefore, the scope of the present disclosure should not be limited to the described embodiments, but should be determined not only by the scope of the following claims, but also by the equivalents of the claims.

Claims

In the duplicate point processing method,

Compressing and restoring point clouds included in the 3D image;

Detecting a plurality of point clouds having the same location value and different color values among the reconstructed point clouds as duplicate points,

Measuring similarity between color values of the overlapping points;

Comparing the measured similarity with a threshold value and determining an average value for color values of the overlapping points as a color value corresponding to the position value based on the comparison result.
The method of claim 1,

The process of measuring the similarity,

And calculating a similarity including a covariance of color values of the overlapping points.
The method of claim 1,

The overlapping point processing method characterized in that the detection of the overlapping points in the unit of voxel (voxel).
The method of claim 1,

The average value of the color values of the overlapping points is calculated for each of the red (R), green (G), blue (B :) elements, characterized in that the overlap point processing method.
The method of claim 1,

The average value of the color values of the overlapping points is determined by using vector values based on red (R), green (G), and blue (B) elements corresponding to the color values of the overlapping points. Method for processing duplicate points, characterized in that calculated.
The method of claim 1,

Selecting a neighbor point cloud having a position value separated by a predetermined distance from the position value of the overlapping points based on the comparison result;

Detecting point clouds located between the duplicated points and the neighboring point cloud;

Applying a weight value to each of the color values of the detected point clouds, and calculating an average value for the color values to which the weight value is applied, respectively, as a weighted average value;

And determining the weighted average value as a color value corresponding to the position values of the overlapping points.
The method of claim 6,

The weighted value applied to each of the detected point clouds has a larger value as the corresponding point cloud gets closer to the duplicated points.
The method of claim 1,

Selecting a neighbor point cloud having a position value separated by a predetermined distance from the position value of the overlapping points based on the comparison result;

Detecting point clouds located between the duplicated points and the neighboring point cloud;

Applying a weight value to each of the color values of the detected point clouds, and calculating an average value for the color values to which the weight value is applied, respectively, as a weighted average value;

And determining a color value having the smallest difference from the weighted average values among the color values of the overlapping points as a color value corresponding to the position value of the overlapping points.
The method of claim 8,

The weighted value applied to each of the detected point clouds has a larger value as the corresponding point cloud gets closer to the duplicated points.
The method of claim 1,

The duplicated points are detected based on additional information, and the additional information includes information on point clouds having a high probability of being duplicated points when reconstructed.
In the redundant point processing apparatus,

A decoder for compressing and restoring point clouds included in a 3D image;

Detecting a plurality of point clouds having the same position value and different color values among the reconstructed point clouds as duplicate points, measuring the similarity between the color values of the duplicate points, and comparing the measured similarity and threshold values. And at least one processor configured to determine an average value of color values of the overlapping points as a color value corresponding to the position value based on the comparison result.
The method of claim 11,

A duplicate point processing apparatus configured to perform the method of any one of claims 2 to 10.