KR102537946B1 - Apparatus and method for processing data assoiciated with point cloud - Google Patents

Abstract

A method for processing point cloud data according to the present disclosure includes identifying auxiliary bits in data about a point cloud and projecting data about a point cloud excluding the identified auxiliary bits onto a plurality of planes. The process of identifying missing points in the plurality of projection images corresponding to the plurality of planes, the process of generating a packed image by packing the plurality of projection images, the auxiliary bit second auxiliary bits excluding the first auxiliary bits corresponding to the identified miss points, information about the first auxiliary bits, and a signal generated by encoding the packed image; include

Description

Method and apparatus for processing data about point cloud {APPARATUS AND METHOD FOR PROCESSING DATA ASSOICIATED WITH POINT CLOUD}

The present disclosure relates to methods and apparatus for processing data relating to point clouds.

A point cloud refers to a collection of a large amount of points, and a large amount of 3D data can be expressed as a point cloud. A point cloud is a value compared to a 2D image, and is a method of expressing a point on a 3D image, and is in the form of a vector that can include location coordinates and color at the same time. For example, a point cloud can be expressed as (x, y, z, R, G, B). The point cloud that forms a spatial composition by gathering innumerable color and location data becomes more and more specific data as the density increases, and it has a meaning as a 3D model.

Since a point cloud representing 3D data occupies a large amount of memory resources, a compression method is required to transmit the point cloud. However, since a lot of time is required to create and spread a new codec for compressing a point cloud, a method for efficiently compressing a point cloud by utilizing an existing codec for compressing 2D data is required. Furthermore, a method for processing data about these point clouds is needed.

The present disclosure provides methods and apparatus for efficiently processing data relating to point clouds. Specifically, the present disclosure provides a method and apparatus for efficiently compressing a point cloud, which is 3D data, using a 2D data compression codec, and a method and apparatus for restoring a 3D image by processing the compressed and transmitted data. provides

In the processing method of data related to a point cloud according to an embodiment of the present disclosure, Identifying auxiliary bits in data related to a point cloud; projecting data other than the identified auxiliary bits from the point cloud data onto a plurality of planes; identifying missed points in a plurality of projection images corresponding to the plurality of planes; generating a packed image by packing the plurality of projection images; and second auxiliary bits other than the first auxiliary bits corresponding to the identified miss points among the identified auxiliary bits, information about the first auxiliary bits, and the packed image generated by encoding the second auxiliary bits. and transmitting a signal, wherein the identified miss points represent points for which an interval between points included in each of the plurality of projection images is greater than or equal to a threshold value, and the information on the first auxiliary bits is the identification It is characterized in that it includes information about the positions of the missed points.

delete

delete

delete

delete

delete

delete

delete

delete

delete

In the processing method of point cloud data according to an embodiment of the present disclosure, second auxiliary bits excluding first auxiliary bits corresponding to miss points among auxiliary bits included in data about a point cloud, receiving and decoding a signal including information on the first auxiliary bits and a packed image; identifying a plurality of projection images by unpacking the packed image; identifying the miss points in the plurality of projection images; estimating the first auxiliary bits corresponding to the miss points based on the information on the first auxiliary bits; and restoring the point cloud based on inversely projected data of the plurality of projection images, the estimated first auxiliary bits, and the second auxiliary bits, wherein the miss points are the plurality of projection images. points where the distance between the points included in each of the points is greater than or equal to a threshold value, and the information on the first auxiliary bits includes information on the locations of the miss points.

In the transmission device according to an embodiment of the present disclosure, the transmission and reception unit; And at least one processor, wherein the at least one processor identifies auxiliary bits related to a point cloud, and projects remaining data excluding the identified auxiliary bits from data related to the point cloud onto a plurality of planes ( projection), identifying missed points in a plurality of projection images corresponding to the plurality of planes, packing the plurality of projection images to generate a packed image, and generating a packed image, generating a signal by encoding second auxiliary bits excluding first auxiliary bits corresponding to the identified miss points among auxiliary bits, information about the first auxiliary bits, and the packed image; Controls the transmitter/receiver to transmit a generated signal, wherein the identified miss points represent points at which an interval between points included in each of the plurality of projection images is equal to or greater than a threshold value, and information on the first auxiliary bits is characterized by including information about the locations of the identified miss points.

delete

delete

delete

delete

delete

In the receiving device according to an embodiment of the present disclosure, the transmitting and receiving unit; and at least one processor, wherein the at least one processor, via the transceiver, second auxiliary bits excluding first auxiliary bits corresponding to miss points among auxiliary bits included in data about the point cloud. , information about the first auxiliary bits, and a packed image, decoding the received signal, and unpacking the packed image to identify a plurality of projection images; Identifying the miss points in projection images of, estimating the first auxiliary bits corresponding to the miss points based on the information about the first auxiliary bits, and inversely projecting the plurality of projection images The point cloud is reconstructed based on one data, the estimated first auxiliary bits and the second auxiliary bits, and the miss points are those in which the distance between points included in each of the plurality of projection images is greater than or equal to a threshold value. points, and the information about the first auxiliary bits includes information about the locations of the miss points.

delete

According to the present disclosure, a 3D image can be compressed by using a 2D image codec, and the amount of data transmission can be reduced through efficient compression and transmission. Therefore, cost can be reduced by utilizing the existing network, and storage space, transmission time, and hardware cost can be reduced. According to the present disclosure, a 3D image may be restored by processing the compressed and transmitted data.

In addition, the effects obtainable in the present disclosure are not limited to the mentioned effects, and other effects not mentioned are clearly understood by those skilled in the art from the description below. It can be.

1 is a flowchart schematically illustrating a processing method for transmission of a 3D image expressed as a point cloud according to an embodiment of the present disclosure.
2 is a diagram illustrating a method for projecting a 3D image expressed as a point cloud according to an embodiment of the present disclosure.
3 is a diagram illustrating an example of an image obtained by projecting a 3D image expressed as a point cloud according to an embodiment of the present disclosure.
4 is a diagram illustrating a method of packing a projection image according to an embodiment of the present disclosure.
5 is a diagram schematically illustrating a process of processing data of a 3D image for transmission of a 3D image expressed as a point cloud according to an embodiment of the present disclosure.
6 is a diagram for explaining a method of transmitting information about an offset of a missed point of a projection image according to an embodiment of the present disclosure.
7 is a flowchart illustrating a method of estimating an offset of a miss point of a projection image according to an embodiment of the present disclosure.
8 is a flowchart illustrating a method of transmitting information about an offset of a missed point according to points according to an embodiment of the present disclosure.
9 is a flowchart illustrating a method of estimating an offset of a miss point according to an embodiment of the present disclosure.
10 is a diagram for explaining a method of transmitting information about an offset of a miss point according to an embodiment of the present disclosure.
11 is a diagram for explaining a method of transmitting information about an offset of a miss point according to an embodiment of the present disclosure.
12 is a flowchart illustrating a method of transmitting information about an offset of a miss point according to an embodiment of the present disclosure.
13 is a diagram for explaining a method of transmitting information about an offset of a miss point according to an embodiment of the present disclosure.
14 is a flowchart illustrating a method of estimating information about an offset of a miss point according to an embodiment of the present disclosure.
15 is a diagram schematically showing the configuration of an apparatus for transmitting or receiving data according to an embodiment of the present disclosure.

Hereinafter, embodiments of the present disclosure will be described in detail with reference to the accompanying drawings. In the following description of the present disclosure, if it is determined that a detailed description of a related known function or configuration may unnecessarily obscure the gist of the present disclosure, the detailed description will be omitted. In addition, terms to be described below are terms defined in consideration of functions in the present disclosure, which may vary according to the intention or custom of a user or operator. Therefore, the definition should be made based on the contents throughout this specification.

In an embodiment of the present disclosure, a 'module' or 'unit' performs at least one function or operation, and may be implemented in hardware or software or a combination of hardware and software. In addition, a plurality of 'modules' or a plurality of 'units' may be integrated into at least one module and implemented by at least one processor, except for 'modules' or 'units' that need to be implemented with specific hardware.

Expressions such as “include” or “may include” that may be used in various embodiments of the present disclosure indicate the existence of a corresponding disclosed function, operation, or component, and may include one or more additional functions, operations, or components. components, etc. are not limited. In addition, in various embodiments of the present invention, terms such as "include" or "have" are intended to designate that the features, numbers, steps, operations, components, parts, or combinations thereof described in the specification exist, It should be understood that it does not preclude the possibility of the presence or addition of one or more other features, numbers, steps, operations, components, parts, or combinations thereof.

In various embodiments of this disclosure, expressions such as “or” include any and all combinations of the words listed together. For example, "A or B" may include A, may include B, or may include both A and B.

Expressions such as “first,” “second,” “first,” or “second,” used in various embodiments of the present disclosure may modify various components of various embodiments, but limit the corresponding components. I never do that. For example, the above expressions do not limit the order and/or importance of corresponding components. The above expressions may be used to distinguish one component from another. For example, the first user device and the second user device are both user devices and represent different user devices. For example, a first element may be termed a second element, and similarly, a second element may also be termed a first element, without departing from the scope of the various embodiments of the present invention.

In various embodiments of the present disclosure, when a certain component is referred to as being “connected” or “connected” to another component, the certain component may be directly connected or connected to the other component. However, it should be understood that another new component may exist between the certain component and the other component. On the other hand, when an element is referred to as being “directly connected” or “directly connected” to another element, it will be understood that no new element exists between the element and the other element. should be able to

Terms used in various embodiments of the present disclosure are used only to describe specific embodiments, and are not intended to limit various embodiments of the present disclosure. Singular expressions include plural expressions unless the context clearly dictates otherwise.

Unless defined otherwise in various embodiments of the present disclosure, all terms used herein, including technical or scientific terms, are those commonly understood by one of ordinary skill in the art to which various embodiments of the present disclosure belong. have the same meaning. Terms such as those defined in commonly used dictionaries should be interpreted as having a meaning consistent with the meaning in the context of the related art, and unless explicitly defined in various embodiments, interpreted in an ideal or excessively formal meaning. It doesn't work.

Hereinafter, a processing method of 3D data expressed as a point cloud according to an embodiment of the present disclosure will be described with reference to FIGS. 1 to 6 . This method may be performed by a device for transmitting data.

1 is a flowchart schematically illustrating a processing method for transmission of a 3D image expressed as a point cloud according to an embodiment of the present disclosure. 2 is a diagram illustrating a method for projecting a 3D image expressed as a point cloud according to an embodiment of the present disclosure. 3 is a diagram illustrating an example of an image obtained by projecting a 3D image expressed as a point cloud according to an embodiment of the present disclosure.

4 is a diagram illustrating a method of packing a projection image according to an embodiment of the present disclosure. 5 is a diagram schematically illustrating a process of processing data of a 3D image for transmission of a 3D image expressed as a point cloud according to an embodiment of the present disclosure. 6 is a diagram for explaining a method of transmitting information about an offset of a missed point of a projection image according to an embodiment of the present disclosure.

First, referring to FIG. 1 , a processing method of a 3D image expressed as a point cloud includes a process of estimating a normal vector of each point in the point cloud (101).

Referring to FIG. 2 , an object 200 means an object in a 3D space of 3D data, and may be expressed as a point cloud that is a set of a plurality of points. The object 200 may represent one scene among a plurality of scenes of a 3D image.

Each point of the object 200 may include data about the position (x, y, z) and color (R (red), G (green), B (blue)). In the present disclosure, one point included in a point cloud is position coordinates (x, y, z) having a size of 10 bits for each coordinate and color coordinates (R, G, B) is assumed. That is, the positional coordinates of one point can be expressed with 10 bits of x coordinate, 10 bits of y coordinate, 10 bits of z coordinate, a total of 30 bits, and color is 8 bits of R value, 8 bits of G value, and B value It can be expressed as 8 bits of , a total of 24 bits.

In order to project the object 200, a virtual hexahedral space 201 in which the object 200 is located at the center may be assumed. The virtual hexahedral space 201 includes a first XY plane (XY0), a second XY plane (XY1), a first YZ plane (YZ0), a second YZ plane (YZ1), a first XZ plane (XZ0), and a second It may consist of an XZ plane (XZ1). The normal vector of the first XY plane XY0 may be in the -z direction, and the normal vector of the second XY plane XY1 may be in the +z direction. The normal vector of the first YZ plane YZ0 may be in the -x direction, and the normal vector of the second YZ plane YZ1 may be in the +x direction. The normal vector of the first XZ plane XZ0 may be in the -y direction, and the normal vector of the second XZ plane XZ1 may be in the +y direction.

In order to project the object 200 , first, the normal vector 202 of each point of the object 200 may be estimated. The normal vector 202 of each point may be estimated with respect to a plane formed by the corresponding point and adjacent points within a certain area.

Also, a plurality of segments may be configured by grouping a plurality of points. In this case, points having normal vectors in the same or similar direction may be grouped into one segment. Here, the similar direction may be a case where the degree of similarity between the directions of the normal vectors is equal to or greater than a predetermined threshold value. Alternatively, adjacent points having a distance between points equal to or less than a predetermined value may be grouped into one segment. In addition, when constructing a segment, both the direction of the normal vector and the distance between the points may be considered. Even if the similarity between the directions of the normal vectors is low or the distance between the points is greater than a certain value, if the colors of the points match or are similar, they are classified as the same segment. You can also group them. The representative normal vector of the segment may be an average value of normal vectors of a plurality of points included in the segment.

Referring back to FIG. 1, next, a plurality of points of the point cloud are projected (102).

Referring to FIG. 2 , a plurality of points of an object 200 expressed as a point cloud may be projected onto a plane constituting a virtual hexahedral space 201 . In this case, the direction of the estimated normal vector of the point and the direction of the normal vector of the projection plane may be the same or similar. Here, the similar direction may be a case where the degree of similarity between the directions of the normal vectors is equal to or greater than a predetermined threshold value. Also, points may be projected in units of segments in which points are grouped.

In order to project a 3D image into a 2D image, any one of equirectangular projection (ERP), octahedron projection (OHP), cylinder projection, cube projection, and various projection methods available in the art may be used.

As described above, the present disclosure provides that each point of the point cloud has x coordinates, y coordinates, and z coordinates each having a size of 10 bits, and R (red) values, G (green) values, and B values each having a size of 8 bits. It is assumed that it is expressed as a (blue) value. However, a codec for encoding such data may not support encoding of 10-bit data. Alternatively, a codec for encoding such data may be optimized for encoding of data of less than 10 bits. In this disclosure, it is assumed that a codec used to encode data to be transmitted supports 8-bit data encoding.

In this case, in terms of one point of the point cloud, since the R (red) value, G (green) value, and B (blue) value for color are each 8 bits, a codec that supports 8-bit data encoding is used. encoding is possible. However, since the x-, y-, and z-coordinates of the point position each have 10 bits, 2-bit auxiliary bits must be separately encoded for each coordinate.

That is, the lower 2 bits are distinguished from the 10 bits of the x coordinate, the lower 2 bits are distinguished from the 10 bits of the y coordinate, and the lower 2 bits are distinguished from the 10 bits of the z coordinate, and a total of 6 bits are separately encoded as auxiliary bits. can be sent. Here, the lower 2 bits may be 2 bits of the lowest importance among 10 bits, and may be located at the end of 10 bits. Hereinafter, the term offset may be used in the same meaning as the auxiliary bit.

Excluding the offset, each point in the point cloud has 8 bits each for x, y, and z coordinates, and 8 bits each for R (red) value, G (green) value, and B (blue) value, for a total of 48 It can have bit-sized data. In this way, a point cloud having 48 bits per point can be projected.

Referring to FIG. 3 , an example of an image in which a point cloud is projected onto each surface of a virtual hexahedral space can be confirmed. As an example, the projection is a first XY plane (XY0), a second XY plane (XY1), a first YZ plane (YZ0), a second YZ plane (YZ1), a first XZ plane (XZ0), a second XZ plane ( Projection may be repeatedly performed in the order of XZ1).

3 shows a first projection image (projection #0) projected onto a first XY plane (XY0), a second projection image (projection #1) projected onto a second XY plane (XY1), and a first YZ plane (YZ0). ), a third projection image (projection #2), a fourth projection image (projection #3) projected onto the second YZ plane (YZ1), and a fifth projection image (projection #3) projected onto the first XZ plane (XZ0). #4), a sixth projection image (projection #5) projected onto the second XZ plane (XZ1), a seventh projection image (projection #6) projected onto the first XY plane (XY0), and a second XY plane An example of an eighth projection image (projection #7) secondly projected on (XY1) and a ninth projection image (projection #8) secondly projected on the first YZ plane (YZ0) is shown. However, the projection order and the number of projections are only examples.

For each point included in the projection image, 8 bits each for x, y, and z coordinates per point, and 8 bits each for R (red) value, G (green) value, and B (blue) value, total 48 bits You can have size data.

Although not shown in FIG. 3, the 10th projection image secondly projected onto the second YZ plane (YZ1) according to the number of projections, the 11th projection image secondly projected onto the first XZ plane (XZ0), and the second XZ plane ( XZ1), a 12th projection image secondly projected onto the first XY plane (XY0), a 13th projected image thirdly projected onto the first XY plane (XY0), a 14th projection image thirdly projected onto the second XY plane (XY1)... There may be more.

As the projection is repeatedly performed, the number of points included in the projected image may decrease. Referring to FIG. 3 , the number of points (points=156839) of the first projection image (projection #0) first projected onto the first XY plane (XY0) is greater than the number of points (points = 156839) in the second projection onto the first XY plane (XY0). It can be seen that the number of points (points=34549) of the projection image (projection #6) is smaller. This is, for example, when first projected onto the first XY plane XY0, 80% of points having the same or similar normal vector direction as the normal vector direction of the first XY plane XY0 are projected, and the first XY plane ( This may be because 16% of points having the same or similar normal vector direction as the normal vector direction of the first XY plane XY0 are projected during the second projection to XY0). However, this number may be an example.

In the case of repeated projection to the first XY plane (XY0) for the third, fourth, or more times, close to 100% of the points having normal vectors in the same or similar directions as the normal vector directions of the first XY plane (XY0) are projected. It can be. That is, when viewing the object 200 on the first XY plane XY0 of the virtual hexahedral space 201, points covered by other points may also be projected through repetitive projection.

Similarly, referring to FIG. 3, the number of points (points = 156597) of the second projection image (projection #1) first projected onto the second XY plane (XY1) is the second projected onto the second XY plane (XY1). It can be seen that the number of points (points=14660) of the 8 projection image (projection #7) is smaller. In addition, the number of points (points=170221) of the third projection image (projection #2) first projected onto the first YZ plane (YZ0) is greater than that of the ninth projection image (projection #2) secondly projected onto the first YZ plane (YZ0). It can be seen that the number of points (points = 9470) of #8) is smaller.

If the number of points included in the projection image decreases due to repetitive projection, intervals between points included in the projection image may widen. In this case, when the interval between the points included in the projection image is greater than or equal to the threshold value, these points are referred to as missed points. That is, when a distance between a first point included in the projection image and a second point closest to the first point is greater than or equal to a threshold value, the first point may correspond to a miss point. By indicating these miss points as flags and transmitting them to the receiver, it is possible to provide the receiver with information on whether or not the miss points correspond.

Referring back to FIG. 1 , next, 2D projection images may be packed (103).

Referring to FIG. 4 , a plurality of projection images 400 , 401 , and 402 may be packed into one or more packed 2D images 403 . Packing may mean generating one or more packed 2D images 403 by transforming and/or rearranging at least some of the projected 2D projection images 400 , 401 , and 402 . Here, transformation includes resizing, transforming, rotation, and/or re-sampling (eg, upsampling, downsampling, differential sampling according to a location within a region) of a region, and the like. can mean This packing method may be referred to as region-wise packing. Although FIG. 4 shows only three projection images 400, 401, and 402, three or more projection images may be packed.

Referring back to FIG. 1 , the packed data and offsets are then encoded (104) and transmitted to the receiving device (105). At this time, the offset of at least one miss point is excluded from encoding and transmission. Also, offsets of all miss points may be excluded from encoding and transmission.

FIG. 5 schematically illustrates a process in which point cloud data, which is 3D data, is processed for transmission, based on the location coordinates of one point included in the point cloud. Referring to FIG. 5 , 6-bit auxiliary bits (offset) are separately encoded by identifying the lower 2 bits from a total of 30 bits of data of 10 bits for each of the x-, y-, and z-coordinates. Although not shown, the offset of the miss point may be excluded from encoding.

In addition, a total of 24-bit data of 8 bits each for coordinates, y-coordinates, and z-coordinates excluding offsets is projected and packed to encode the packed 2D image. Although not shown in FIG. 5, data for R, G, and B, each having 8 bits, may also be projected, packed, and encoded along with position data.

The encoded packed 2D image and the encoded offset excluding the offset of the miss point are transmitted to the receiving device.

Encoding includes 2D video such as High Efficiency Video Coding (HEVC), H.264, Future Video Codec, Advanced Video Coding (AVC), Pure Video 9 (VP9), Pure Video 8 (VP8), and Joint Video Exploration Team (JVET). A codec capable of base compression may be used.

The encoded data may be divided or processed according to a transmission protocol determined through processing such as adding a header and then transmitted. Along with the encoded data or separately, additional data related to the encoded data, data necessary for reproducing the data (eg, metadata) may be transmitted.

Data transmission by the transmitting device may follow MPEG Media Transport (MMT) or Dynamic Adaptive Streaming over HTTP (DASH), but is not necessarily limited thereto.

Although not shown, in addition to data related to images, data related to audio may be transmitted to the receiving device.

When the packed 2D image and offset are transmitted, the offset of the miss point may not be transmitted. As described above, as projection is repeated, the projected image may include a miss point whose distance to an adjacent point is greater than or equal to a threshold value. The offset of the miss point may be separately encoded and transmitted as in the offset of FIG. 5, but a method capable of reducing the transmission amount may be needed. At this time, the amount of data transmission can be reduced by transmitting information on the offset of the miss point having smaller data instead of transmitting the offset of the miss point.

Hereinafter, a method of transmitting information about an offset of a miss point will be described with reference to FIG. 6 .

Referring to FIG. 6, for example, the object of the projection image first projected on the first XY (XY0) plane is the first patch 600, the object of the second projection image projected on the first XY (XY0) plane The second patch 601 and the third patch 602 may be an object of a projection image obtained by third projection on the first XY (XY0) plane. It is assumed that the third patch 602 includes miss points 603 , 604 , 605 , 606 , and 607 .

At this time, the offset of each of the miss points 603, 604, 605, 606, and 607 is the first offset value (offset_1) of the first midpoint C1 of the first patch 600, the first midpoint C1 and It can be estimated by considering the distance α1 of the second patch 601, the second offset value offset_2 of the second midpoint C2 of the second patch 601, and the distance β1 from the second midpoint C2. Here, the midpoint may be the center of gravity of the patch.

Specifically, taking the first miss point 603 as an example, the first offset value offset_1 of the first midpoint C1 of the first patch 600, the first miss point 603 and the first midpoint C1 ), the second offset value (offset_2) of the second midpoint C2 of the second patch 601, and the distance between the first miss point 603 and the second midpoint C2 ( β1), a weighted average (est_mp_offset_1) based on a weight that is inversely proportional to the distance can be calculated as shown in [Equation 1], and this can be estimated as the offset of the first miss point 603. .

In this case, the offset of the first miss point 603 does not need to be transmitted, and the data transmission amount can be reduced by the amount of offset data of the first miss point 603 that is not transmitted. The process of estimating the offset of the first miss point 603 may be performed by the device receiving the transmitted data.

In this case, instead of transmitting the offset of the first miss point 603, information on the coordinates of the midpoint C1 of the first patch 600 and the coordinates of the midpoint C2 of the second patch 601 may be transmitted. Alternatively, instead of transmitting the offset of the first miss point 603, the distance α1 between the first miss point 603 and the midpoint C1 of the first patch 600 and the miss point 603 Information on the distance β1 between the y and the midpoint C2 of the second patch 601 may be transmitted. In addition, both the coordinates of the midpoint of the patch and the distance between the miss point and the midpoint of the patch may be transmitted.

In addition, information on the first offset value (offset_1) of the first midpoint (C1) of the first patch (600) and the second offset value (offset_2) of the second midpoint (C2) of the second patch (601) is transmitted. can

The receiving device may estimate the offset of the first miss point 603 by receiving the information on the miss point.

Regarding the method of estimating the offset of the miss point, the estimation mode may be displayed and provided by the transmitting device during data transmission, or the receiving device may select the estimation mode based on received information.

2nd miss point 604, 3rd miss point 605?? Offset estimation of the like can be performed in the same way. That is, taking the second miss point 604 as an example, the distance α2 between the second miss point 604 and the midpoint C1 of the first patch 600 and the second miss point 604 and the second miss point 604 Information on the distance β2 from the midpoint C2 of the two-patch 601 can be additionally transmitted, and the receiving device can estimate the offset of the second miss point 604 by receiving the information.

6 illustrates a case of estimating an offset of a miss point based on two patches, a first patch 600 and a second patch 601, but the number of patches is not limited thereto. When the number of patches adjacent to the first miss point is n, a weighted average (est_mp_offset_1) can be calculated as shown in [Equation 2], and this can be estimated as an offset of the first miss point.

In [Equation 2], offset(i) is the offset of the midpoint of the ith patch, distance(i) is the distance between the midpoint of the ith patch and the first miss point, and distance(k) is the midpoint of the kth patch. is the distance between and the first miss point.

Similarly in this case, the transmitting device may transmit information about the offset of the miss point including at least one of coordinates of the midpoint of the patch, a distance between the miss point and the midpoint of the patch, and an offset of the midpoint of the patch, and the receiving device may transmit Based on this, the offset of the miss point can be estimated.

Hereinafter, a method for processing data about a point cloud according to an embodiment of the present disclosure will be described with reference to FIG. 7 . Descriptions of the same parts as in the above-described embodiment are omitted.

7 is a flowchart illustrating a method of estimating an offset of a miss point of a projection image according to an embodiment of the present disclosure.

The receiving device decodes the received data (700). The decoded data may include a packed 2D image, offsets of points excluding miss points, and information on offsets of miss points.

The receiving device may restore the projection images by unpacking the packed 2D image. Unpacking may be an inverse transformation of a packing method performed by a transmitting device. To this end, the transmitting device and the receiving device may share a packing method in advance.

The receiving device may identify a miss point in the restored projection images. The receiving device checks the flag of the point included in the projection images (701), moves to the next point if it is not a miss point (702), and identifies N patches if it is a miss point (703). In this case, the N patches may be patches adjacent to the miss point.

Next, the receiving device identifies the distance between the midpoint of the N patches included in the decoded data and the miss point, or identifies the coordinates of the midpoint of the N patches included in the received data to determine the midpoint of the N patches and the distance between the miss points. The distance to the miss point can be calculated. Based on this, the receiving device may calculate a weighted average of the offsets of the N patches as shown in [Equation 2] (704).

The receiving device may estimate the calculated weighted average value as an offset of the miss point (705). An offset may be estimated using the same method for each of the identified plurality of miss points.

The receiving device may project the plurality of unpacked projection images into a 3D image. In order to project a 2D projection image into a 3D image, a projection inverse of the projection used in the transmitting device may be used, but is not necessarily limited thereto.

The receiving device may restore 3D data based on data obtained by projecting a plurality of projection images into a 3D image, offsets of points excluding the miss points, and estimated offsets of the miss points.

Hereinafter, a method of transmitting information about an offset of a miss point according to an embodiment of the present disclosure will be described with reference to FIGS. 6 and 8 . Descriptions of the same contents as in the above-described embodiment are omitted.

8 is a flowchart illustrating a method of transmitting information about an offset of a missed point according to points according to an embodiment of the present disclosure.

The transmitting device may transmit information about the offset of the miss point without transmitting the offset of the miss point. First, the transmitting device may determine whether a point included in the projection image corresponds to a miss point (800). As described above, whether a point corresponds to a miss point may be determined by determining whether a distance to another point is equal to or greater than a threshold value.

If it is not a miss point, it moves to the next point (801).

Next, a patch closest to the miss point can be identified (803). Referring to FIG. 6 as an example, the second patch 601 may be identified as the closest patch to the first miss point 603 .

Next, the offset of the nearest patch is identified (804), and the difference between the offset of the nearest patch and the offset of the miss point can be calculated (805). For example, the second offset (offset_2) of the second midpoint (C2) of the second patch (601) is identified, and the offset (mp_offset_1) of the first miss point (603) and the second patch (601) A difference value (delta_1) of the second offset (offset_2) of the second midpoint (C2) may be calculated.

The transmitting device may encode and transmit information about the offset of the miss point including the calculated difference value (delta_1) (806). The information on the offset of the miss point may include the value of the second offset (offset_2) of the second midpoint C2.

Hereinafter, a method of estimating an offset of a miss point according to an embodiment of the present disclosure will be described with reference to FIGS. 6 and 9 . Descriptions of the same contents as in the above-described embodiment are omitted.

9 is a flowchart illustrating a method of estimating an offset of a miss point according to an embodiment of the present disclosure.

The receiving device decodes the received data (900). The received data includes information about the offset of the miss point.

The receiving device may check the flag to determine whether it is a miss point (901). If it is not a miss point, it moves to the next point (902), and if it is a miss point, the nearest patch can be identified (903). Referring to FIG. 6 as an example, the second patch 601 may be identified as the closest patch to the first miss point 603 .

Next, the receiving device may identify an offset of a patch closest to the received data (904). For example, a second offset (offset_2) of the second midpoint C2 of the second patch 601 may be identified from the received data.

The receiving device may identify a difference value corresponding to a miss point in the received data (905). For example, in the case of the first miss point 603, the difference between the offset (mp_offset_1) of the first miss point 603 and the second offset (offset_2) of the second midpoint (C2) of the second patch 601 The value delta_1 may be identified as a difference value corresponding to the first miss point.

Next, the offset of the miss point may be calculated based on the difference between the offset of the nearest patch and the miss point (906). For example, the receiving device determines the first miss based on the difference value delta_1 corresponding to the first miss point 603 and the second offset_2 of the second midpoint C2 of the second patch 601. The offset of the de point 603 can be calculated. The offset thus calculated can be estimated as the offset of the first miss point 603 .

In this case, for example, if the value of the actual offset (mp_offset_1) of the first miss point 602 is in the range of 0 to 255, only the difference value (delta_1) in the range of 0 to 100 is smaller than the value of the actual offset (mp_offset_1). transmission, so the amount of data transmission can be reduced.

Offset estimation of the second miss point 603, the third miss point 604, etc. may be performed in the same way.

Hereinafter, a method of transmitting information about an offset of a miss point according to an embodiment of the present disclosure and a method of estimating an offset of a miss point based thereon will be described with reference to FIG. 10 . Descriptions of the same parts as in the above-described embodiment are omitted.

10 is a diagram for explaining a method of transmitting information about an offset of a miss point according to an embodiment of the present disclosure.

A plurality of groups (Group A, Group B, Group C, and Group D) in FIG. 10 is a grouping of a plurality of miss points. Instead of transmitting offsets of a plurality of miss points, a plurality of miss points may be grouped and a representative offset and group identifier of each group may be transmitted as shown in FIG. 10 .

In this case, the plurality of miss points may be grouped according to various criteria. As an example, based on the offset of one miss point, miss points having the same or similar offset values may be grouped into one group. Here, that the offset values are similar may be a case where a difference between the offsets is equal to or less than a predetermined threshold value.

As an example, if the angle difference value between offset vectors of the miss points is less than or equal to a threshold value, they may be grouped into one group. For example, the offset vector of the first miss point is a = (a1, a2, ..., an) and the offset vector of the second miss point is b = (b1, b2, ..., bn) When the angle difference value (cosθ) between the vector (a) of the offset of the first miss point and the vector (b) of the offset of the second miss point calculated by [Equation 3] is less than or equal to the threshold value, The offset vector of the 1st miss point and the offset vector of the 2nd miss point may be grouped into one group.

Alternatively, based on one miss point, miss points within a certain area may be grouped into one group. Alternatively, the miss points may be grouped considering both the proximity and the similarity of the offsets between the miss points.

In this case, since each group identifier and the representative offset of each group need only be transmitted to the receiver, the amount of data transmission can be reduced compared to the case where offsets of all miss points are transmitted.

The receiving device may estimate that the offsets of the miss points included in the corresponding group are the same as the representative offset of the corresponding group.

Hereinafter, a method of transmitting information about an offset of a miss point according to an embodiment of the present disclosure and a method of estimating an offset of a miss point based thereon will be described with reference to FIG. 11 . Descriptions of the same parts as in the above-described embodiment are omitted.

11 is a diagram for explaining a method of transmitting information about an offset of a miss point according to an embodiment of the present disclosure.

Instead of transmitting the offsets of a plurality of miss points, a plurality of miss points may be grouped as shown in FIG. 11 and a group identifier of a representative group and a representative offset of the representative group may be transmitted to the receiver. In addition, a difference value between an offset of the representative group and a representative offset of a group other than the representative group may be transmitted to the receiver.

For example, a plurality of groups (Group A, Group B, Group C, and Group D) may be arranged in order of offset size, and a group having the largest offset may be determined as a representative group.

Assuming that the first group (Group A) is a representative group, a group identifier of the first group (Group A) and a representative offset of the first group (Group A) may be transmitted. In addition, a difference value (delta AB) between the representative offset of the first group (Group A) and the representative offset of the second group (Group B) is calculated, and the representative offset of the second group (Group B) and the representative offset of the third group (Group B) are calculated. Calculate the difference value (delta BC) between the representative offsets of the group (Group C), and calculate the difference value (delta CD) between the representative offsets of the third group (Group C) and the representative offsets of the fourth group (Group D) It is possible to transmit these difference values by calculating .

In this case, since the group identifier of each group, the representative offset of the representative group, and the difference values between the representative offsets of the groups need only be transmitted, the amount of data transmission can be reduced compared to the case of transmitting the offsets of all miss points. In addition, as the calculation of the difference value between these representative offsets is repeated, the size of the difference values can be reduced, so the data transmission amount can be reduced.

The receiving device may estimate the representative offset of each of the plurality of groups by calculating based on the received representative offset of the representative group and difference values between the representative offsets. The receiving device may estimate that the offsets of the miss points included in the corresponding group are the same as the estimated representative offset of the corresponding group.

Hereinafter, a method of transmitting information about an offset of a miss point according to an embodiment of the present disclosure will be described with reference to FIGS. 12 and 13 . Descriptions of the same parts as in the above-described embodiment are omitted.

12 is a flowchart illustrating a method of transmitting information about an offset of a miss point according to an embodiment of the present disclosure. 13 is a diagram for explaining a method of transmitting information about an offset of a miss point according to an embodiment of the present disclosure.

The transmitting device may transmit information about the offset of the miss point without transmitting the offset of the miss point. First, the transmitting device may determine whether a point corresponds to a miss point (1200). If it is not a miss point, it moves to the next point (1201), and if it is a miss point, it is possible to set a flag indicating that it corresponds to a miss point (1202).

Next, miss points may be grouped (1203), and a representative offset of each of the miss point groups may be determined (1204). 13 as an example, a plurality of miss points can be grouped into a plurality of groups (Group A, Group B, Group C, Group D), and a plurality of groups (Group A, Group B, Group C, Group D). D) Each representative offset can be determined.

Next, a representative group may be determined from among a plurality of groups (1205). For example, a plurality of groups (Group A, Group B, Group C, and Group D) may be arranged in order of offset size, and a group having the largest offset may be determined as a representative group.

Next, a difference value between a representative offset of a representative group and a representative offset of another group may be calculated (1206). Assuming that the first group (Group A) is a representative group, a difference value (delta AB) between a representative offset of the first group (Group A) and a representative offset of the second group (Group B) is calculated, and the first group ( A difference value (delta AC) between the representative offset of Group A) and the representative offset of the third group (Group C) is calculated, and the representative offset of the first group (Group A) and the representative offset of the fourth group (Group D) is calculated. The difference value (delta AD) can be calculated.

The transmitting device may encode and transmit a group identifier of each group, a representative offset of a representative group, and difference values between representative offsets of groups (1207). For example, the transmitting device includes a group ID of each of a plurality of groups (Group A, Group B, Group C, and Group D), a representative offset of the first group (Group A), and a difference between the representative offsets (delta AB, delta AC, delta AD) can be transmitted.

In this case, since the group identifier of each group, the representative offset of the representative group, and the difference values between the representative offsets of the groups need only be transmitted, the amount of data transmission can be reduced compared to the case of transmitting the offsets of all miss points.

Hereinafter, a method of estimating an offset of a miss point according to an embodiment of the present disclosure will be described with reference to FIGS. 13 and 14 . Descriptions of the same parts as in the above-described embodiment are omitted.

14 is a flowchart illustrating a method of estimating information about an offset of a miss point according to an embodiment of the present disclosure.

The receiving device decodes the received data (1400). The received data includes information about the offset of the miss point.

The receiving device may check the flag to determine whether it is a miss point (1401). If it is not a miss point, it moves to the next point (1402), and if it is a miss point, a group identifier of a group including the miss point may be identified (1403). For example, when the miss point is included in the third group (Group C), a group identifier of the third group (Group C) may be identified.

Next, a difference value corresponding to a group including a miss point may be identified. For example, the difference value delta AC may be identified as a difference value corresponding to the third group (Group C).

The receiving device may identify the representative offset of the representative group and calculate the representative offset of the group including the miss point (1405). For example, based on the representative offset of the first group (Group A), which is the representative group, and the difference value (delta AC) corresponding to the third group (Group C) including the miss point, the third group (Group C) A representative offset of can be calculated. The receiving device may estimate the calculated representative offset of the third group (Group C) as an offset of a miss point included in the third group (Group C).

Syntax such as the following [Table 1] to [Table 5] may be used to perform the embodiments according to the present disclosure described above.

In [Table 1] to [Table 5], pick_nearest_neighborhood(id,num) is the number of segments closest to (Point[id].x, Point[id].y, Point[id].z) ( num) and returns the offset. calculate_distance(id, Neighbor[i]) is a function that measures and returns the geometric distance between Point[id] and Neighbor[i]. Delta[id] means a space that stores the difference value (Delta) between Point[id] and the offset of the reference segment. Offset[group_index] means the space where the group_index group's offset is stored.

Hereinafter, the structure of an apparatus for transmitting or receiving data according to an embodiment of the present disclosure will be described with reference to FIG. 15 . Descriptions of the same parts as in the above-described embodiment are omitted.

15 is a diagram schematically showing the configuration of an apparatus for transmitting or receiving data according to an embodiment of the present disclosure.

The device 1500 for transmitting or receiving data includes a transceiver 1501 for transmitting and receiving an encoded signal, a control unit 1502 for controlling all operations of the device for transmitting or receiving data, and storing data related to a point cloud. It may include a memory unit 1503 that does.

At least one or more of all techniques or methods such as projection, packing, encoding, inverse projection, unpacking, and decoding performed by the apparatus 1500 for transmitting or receiving data described above in the present disclosure is performed by the control unit 1502. It can be understood that it is performed by control.

Data related to the point cloud detailed in the present disclosure may be temporarily or semi-permanently stored in whole or in part in the memory unit 1503 . In addition, algorithms for all techniques or methods performed by the apparatus 1500 for transmitting or receiving data described above in this disclosure may be temporarily or semi-permanently stored in whole or in part in the memory unit 1502. .

However, the transceiver 1501, the control unit 1502, and the memory unit 1503 do not necessarily have to be implemented as separate devices, but can be implemented as a single component in the form of a single chip. . Furthermore, the transmission/reception unit 1501, the control unit 1502, and the memory unit 1503 do not necessarily all have to be included, and some may be omitted.

It should be noted that the method diagrams, system configuration diagrams, device configuration diagrams, etc. illustrated in FIGS. 1 to 15 are not intended to limit the scope of the present disclosure. That is, all the configurations or operations described in FIGS. 1 to 15 should not be interpreted as being essential components for the implementation of the present disclosure, and even if only some components are included, they can be implemented within a range that does not impair the essence of the present disclosure. can

The operations described above may be realized by including a memory device storing the corresponding program code in a server or an arbitrary component in the device. That is, the control unit of the server or device may execute the above-described operations by reading and executing program codes stored in the memory device by a processor or a central processing unit (CPU).

Various components, modules, and the like of the server or device described in this specification include hardware circuits, for example, complementary metal oxide semiconductor-based logic circuits, firmware, and software. may be operated using hardware circuitry, such as software and/or a combination of hardware and firmware and/or software embedded in a machine readable medium. As an example, various electrical structures and methods may be implemented using electrical circuits such as transistors, logic gates, and application specific semiconductors.

Meanwhile, in the detailed description of the present disclosure, specific embodiments have been described, but various modifications are possible without departing from the scope of the present disclosure. For example, the present disclosure has been described on the assumption of point data having position coordinates of 10 bits and color coordinates of 8 bits, but the size of the data is not limited thereto. In addition, the present disclosure has been described assuming a codec supporting coding of 8-bit data, but the codec is not limited thereto.

Therefore, the scope of the present disclosure should not be limited to the described embodiments and should not be defined, but should be defined by not only the scope of the claims to be described later, but also those equivalent to the scope of these claims.

1501: transceiver
1502: control unit
1503: memory unit

Claims (21)

In the processing method of data about the point cloud,
identifying auxiliary bits in data about the point cloud;
projecting data other than the identified auxiliary bits from the point cloud data onto a plurality of planes;
identifying missed points in a plurality of projection images corresponding to the plurality of planes;
generating a packed image by packing the plurality of projection images; and
A signal generated by encoding second auxiliary bits excluding the first auxiliary bits corresponding to the identified miss points among the identified auxiliary bits, information about the first auxiliary bits, and the packed image Including the process of transmitting,
The identified miss points represent points for which an interval between points included in each of the plurality of projection images is equal to or greater than a threshold value,
Wherein the information about the first auxiliary bits includes information about the locations of the identified miss points. According to claim 1,
The size of each of the identified auxiliary bits corresponds to a difference between the size of data related to the point cloud and the size of encodeable data supported by a codec used to generate the signal. How to process data about it. delete According to claim 1,
and setting a flag to indicate whether each of the points included in the plurality of projection images is a miss point. According to claim 1,
Information about the location of the identified miss points,
Point characterized in that it includes at least one of coordinate information about the midpoint of each of the patches included in the plurality of projection images, and distance information from the midpoint of each of the patches to each of the identified miss points. How to process data about the cloud. According to claim 1,
Information about the location of the identified miss points,
an auxiliary bit for a middle point of a patch most adjacent to each of the identified miss points among patches included in the plurality of projection images; and
and a difference value between an auxiliary bit for each of the identified miss points and an auxiliary bit for a midpoint of the patch. According to claim 1,
Further comprising grouping the identified miss points into a plurality of groups,
Wherein the information about the locations of the identified miss points includes a group identifier of each of the plurality of groups and a representative auxiliary bit of each of the plurality of groups. According to claim 7,
The process of grouping the identified miss points,
and grouping the identified miss points based on at least one of a similarity between the first auxiliary bits and a distance between the identified miss points. According to claim 1,
grouping the identified miss points into a plurality of groups; and
Calculating a difference between a representative auxiliary bit of a first group included in the plurality of groups and a representative auxiliary bit of a second group included in the plurality of groups;
wherein the information about the locations of the identified miss points includes a group identifier of each of the plurality of groups, a representative auxiliary bit of the first group, and the calculated difference value. method. According to claim 9,
Wherein the first group is a group having the largest representative auxiliary bit among the plurality of groups. In the processing method of data about the point cloud,
A signal including second auxiliary bits excluding first auxiliary bits corresponding to miss points among auxiliary bits included in data about the point cloud, information about the first auxiliary bits, and a packed image receiving and decoding;
identifying a plurality of projection images by unpacking the packed image;
identifying the miss points in the plurality of projection images;
estimating the first auxiliary bits corresponding to the miss points based on the information on the first auxiliary bits; and
Restoring the point cloud based on inversely projected data of the plurality of projection images, the estimated first auxiliary bits, and the second auxiliary bits;
The miss points represent points for which an interval between points included in each of the plurality of projection images is greater than or equal to a threshold value,
The method of processing data about a point cloud, characterized in that the information about the first auxiliary bits includes information about the locations of the miss points. According to claim 11,
The size of each of the auxiliary bits included in the data about the point cloud corresponds to the difference between the size of the data about the point cloud and the size of encodeable data supported by a codec used to generate the signal. A method of processing data about a characterized point cloud. According to claim 11,
The process of identifying the miss points is,
acquiring a flag for each of a plurality of points included in the plurality of projection images from a decoded signal; and
and identifying the miss points based on a flag for each of the plurality of points,
Wherein the flag for each of the plurality of points indicates whether each of the plurality of points is a miss point. According to claim 11,
Information about the location of the miss points,
Point cloud data comprising coordinate information about the midpoint of each of the patches included in the plurality of projection images, and distance information from the midpoint of each of the patches to each of the miss points. processing method. According to claim 14,
The process of estimating the first auxiliary bits,
The auxiliary bits are obtained by using a weight value determined based on distance information from the midpoint of each of the patches to each of the miss points and a weighted average value based on auxiliary bits for the midpoint of each of the patches. A method of processing data about a point cloud, comprising the step of estimating. According to claim 11,
Information about the location of the miss points,
an auxiliary bit for a midpoint of a patch most adjacent to each of the miss points among patches included in the plurality of projection images; and
and a difference value between an auxiliary bit for each of the miss points and an auxiliary bit indicating information on a midpoint of the patch. According to claim 11,
The miss points are grouped into a plurality of groups,
Wherein the information about the location of the miss points includes a group identifier of each of the plurality of groups and a representative auxiliary bit of each of the plurality of groups. According to claim 11,
The miss points are grouped into a plurality of groups,
The information about the location of the miss points includes a group identifier of each of the plurality of groups, a representative auxiliary bit of a first group included in the plurality of groups, and a representative auxiliary bit of the first group and the plurality of groups. A method of processing data about a point cloud, characterized in that it includes difference values of representative auxiliary bits of the second group included in . According to claim 18,
Wherein the first group is a group having the largest representative auxiliary bit among the plurality of groups. In the transmitting device,
transceiver; and
includes at least one processor;
The at least one processor,
identify auxiliary bits for a point cloud;
Projecting data other than the identified auxiliary bits from the data on the point cloud onto a plurality of planes;
identifying missed points in a plurality of projection images corresponding to the plurality of planes;
Packing the plurality of projection images to create a packed image;
A signal is generated by encoding second auxiliary bits excluding the first auxiliary bits corresponding to the identified miss points among the identified auxiliary bits, information about the first auxiliary bits, and the packed image. do,
Controlling the transceiver to transmit the generated signal;
The identified miss points represent points for which an interval between points included in each of the plurality of projection images is equal to or greater than a threshold value,
The transmitting device of claim 1, wherein the information on the first auxiliary bits includes information on locations of the identified miss points. In the receiving device,
transceiver; and
includes at least one processor;
The at least one processor,
Second auxiliary bits excluding first auxiliary bits corresponding to miss points among auxiliary bits included in the data about the point cloud, information about the first auxiliary bits, and a packed image through the transceiver. Receive a signal containing
decoding the received signal;
unpacking the packed image to identify a plurality of projection images;
identify the miss points in the plurality of projection images;
estimating the first auxiliary bits corresponding to the miss points based on the information about the first auxiliary bits;
Restoring the point cloud based on data obtained by inversely projecting the plurality of projection images, the estimated first auxiliary bits, and the second auxiliary bits;
The miss points represent points for which an interval between points included in each of the plurality of projection images is greater than or equal to a threshold value,
The receiving device of claim 1, wherein the information on the first auxiliary bits includes information on locations of the miss points.

Images