TW202234894A - Hybrid-tree coding for inter and intra prediction for geometry coding - Google Patents

Abstract

A device for decoding a bitstream that includes point cloud data is configured to determine an octree that defines an octree-based splitting of a space containing the point cloud, wherein a leaf node of the octree contains one or more points of the point cloud; and directly decode positions of each of the one or more points in the leaf node, wherein to directly decode the positions of each of the one or more points in the leaf node, the one or more processors are further configured to: generate a prediction of the one or more points; and determine the one or more points based on the prediction.

Description

Hybrid tree coding for inter and intra prediction for geometry coding

This application claims the benefit of US Provisional Patent Application 63/131,546, filed on December 29, 2020, the entire contents of which are incorporated herein by reference.

The present disclosure relates to point cloud encoding and decoding.

In summary, this disclosure describes a hybrid tree coding method that combines octree coding and predictive coding for enhanced inter/intra prediction at the block level to achieve point clouds compression.

In one example, the present disclosure describes a method of decoding a point cloud, the method comprising: determining an octree that defines an octree-based split of a space containing the point cloud, wherein: the The leaf nodes of the octree contain one or more points of the point cloud, and the position of each of the one or more points in the leaf node is directly signaled; using intraframe predicting or inter-predicting to generate a prediction for the one or more points; and coding a syntax element indicating whether the one or more points are predicted using intra-prediction or inter-prediction.

According to one example of the present disclosure, an apparatus for decoding a bitstream including point cloud data includes: memory for storing the point cloud data; and coupled to the memory and in a circuit Implementing one or more processors configured to: determine an octree that defines an octree-based split of a space containing a point cloud, wherein the leaves of the octree a node containing one or more points of the point cloud; and directly decoding the position of each of the one or more points in the leaf node, wherein in order to directly decode the point in the leaf node decoding the position of each of the one or more points in the one or more processors, the one or more processors are further configured to: generate a prediction of the one or more points; and based on the prediction to determine the one or more points.

According to another example of the present disclosure, a method of decoding point cloud material includes determining an octree that defines an octree-based split of a space containing the point cloud, wherein the octree's A leaf node contains one or more points of the point cloud; the position of each of the one or more points in the leaf node is directly decoded, wherein the position of each of the one or more points in the leaf node is directly decoded Decoding the location of each of the one or more points includes: generating a prediction of the one or more points; and determining the one or more points based on the prediction.

According to another example of the present disclosure, a computer-readable storage medium stores instructions that, when executed by one or more processors, cause the one or more processors to: determine a definition that includes a point cloud A spatial octree-based split octree, wherein a leaf node of the octree contains one or more points of the point cloud; and a direct response to the one or more points in the leaf node decoding the position of each of the plurality of points, wherein, to directly decode the position of each of the one or more points in the leaf node, the instructions cause the The one or more processors: generate a prediction of the one or more points; and determine the one or more points based on the prediction.

According to another example of the present disclosure, an apparatus includes means for determining an octree-based split of an octree that defines a space containing a point cloud, wherein leaf nodes of the octree contain the one or more points of a point cloud; means for directly decoding the position of each of said one or more points in said leaf node, wherein said means for directly decoding said one or more points in said leaf node The means for decoding the position of each of the one or more points in the leaf node comprises: means for generating a prediction of the one or more points; and determining based on the prediction components of the one or more points.

The details of one or more examples are set forth in the accompanying drawings and the description below. Other features, objects and advantages will be apparent from the description, drawings and claims.

A point cloud is a collection of points in three-dimensional (3D) space. These points may correspond to points on objects in three-dimensional space. Therefore, point clouds can be used to represent solid content in three-dimensional space. Point clouds can be useful in a variety of situations. For example, point clouds can be used in the context of autonomous vehicles to represent the location of objects on the road. In another example, point clouds may be used in the context of physical content representing an environment for the purpose of locating virtual objects in augmented reality (AR) or mixed reality (MR) applications. Point cloud compression is the process used to encode and decode point clouds. Encoding point clouds can reduce the amount of data needed to store and transmit point clouds.

There have previously been two main proposals for signaling the location of points in a point cloud: octree coding and predictive tree coding. As part of encoding the point cloud data using octree decoding, the G-PCC encoder can generate octrees. Each node of the octree corresponds to cube space. A node of an octree can have zero children or eight children. In other examples, nodes may be divided into child nodes according to other tree structures. The child nodes of the parent node correspond to cubes of equal size within the cube corresponding to the parent node. The position of each point of the point cloud can be signaled relative to the origin of the node. A node is said to be unoccupied if it does not contain any points of the point cloud. If the node is not occupied, signaling additional information about the node may not be necessary. Conversely, a node is said to be occupied if it contains one or more points of the point cloud.

When encoding point cloud data using prediction tree coding, the G-PCC encoder determines a prediction mode for each point of the point cloud. The prediction mode for points can be one of the following: (1) No Prediction/Zero Prediction (0) (2) Incremental prediction (p0) (3) Linear prediction (2*p0 – p1) (4) Parallelogram prediction (2*p0 + p1 – p2)

In the case where the prediction mode for a point is "no prediction/zero prediction", the point is treated as a root point (ie, a root vertex), and the coordinates of the point (eg, x, y, z coordinates). Where the prediction mode for a point is "delta prediction", the G-PCC encoder determines the difference (ie, delta) between the coordinates of the point and the coordinates of the parent point (such as the root point or other points). When the prediction mode is "linear prediction", the G-PCC encoder uses linear prediction of the coordinates of the two parent points to determine the predicted coordinates of the point. The G-PCC encoder signals the difference between the predicted coordinates determined using linear prediction and the actual coordinates of the point. When the prediction mode is "parallelogram prediction", the G-PCC encoder uses the three parent points to determine the prediction coordinates. The G-PCC encoder then signals the difference between the predicted and actual coordinates of the point (eg, the "main residual"). Predictive relationships between points essentially define a tree of points.

It has been observed experimentally that octree decoding may be more suitable for dense point clouds than prediction tree decoding. Point clouds obtained using 3D modeling are usually dense enough to make octree decoding better. However, in automotive applications, for example, point clouds obtained using LiDAR tend to be a bit rough, and therefore, predictive coding may work better for these applications.

In some examples, the angle mode may be used to represent the coordinates of a point in a spherical coordinate system. Information may be lost due to imperfect transformations between spherical and Cartesian coordinate systems (eg, x, y, z). However, since the G-PCC encoder can perform the transformation process, the G-PCC encoder can signal a "sub-residual" for a point that indicates the result of applying the transformation process to the spherical coordinates of the point The difference between a point's Cartesian coordinates and the point's original Cartesian coordinates.

The present disclosure relates to a hybrid decoding model in which point clouds are decoded using octree decoding and direct decoding. For example, octree coding may initially be used to divide the space into nodes up to a certain level. Nodes at a particular level (and other occupied nodes of the octree that are not further split) may be referred to as "leaf nodes". Points within the volume of a leaf node can be decoded using a "direct" decoding mode.

When encoding the points of a leaf node in "direct" coding mode, the G-PCC encoder may select an intra-prediction mode for the leaf node or an inter-prediction mode for the leaf node. The G-PCC encoder can signal whether the point of the leaf node is encoded using intra-prediction mode or inter-prediction mode.

If the G-PCC encoder selects an intra prediction mode for a leaf node, the G-PCC encoder can use prediction tree coding to encode the points in the leaf node in much the same way as described above. That is, the G-PCC encoder can select from four prediction modes and signal the coordinates of the points accordingly. However, instead of signaling coordinates relative to the origin of the entire space associated with the octree, the G-PCC encoder may signal coordinates relative to the origin of the leaf nodes. This can improve decoding efficiency, especially for the root node.

If the G-PCC encoder selects an inter prediction mode for a leaf node, the G-PCC encoder may encode the points in the leaf node relative to the set of points in the reference frame. A reference frame may be a previously coded frame, similar to a previous frame of video. The G-PCC encoder may perform motion estimation to identify sets of points in the reference frame that have a similar spatial arrangement to the points in the leaf nodes. The motion vector for the leaf node indicates the displacement between the point of the leaf node and the set of points identified in the reference frame.

The G-PCC encoder may signal the set of parameters for the leaf nodes. Parameters for leaf nodes may include reference indices identifying reference frames. Parameters for leaf nodes may also include a value indicating the number of points in the leaf node.

The parameters of the leaf node may also include residual values for each point in the leaf node. The residual value for the point in the leaf node indicates the difference between the predicted coordinates of the leaf node (as by adding the motion vector of the leaf node to the point in the reference frame corresponding to the point in the leaf node) Sure). In the example using the angle mode, the G-PCC encoder may also signal the sub-residual for the points.

In some examples, the parameters for leaf nodes also include motion vector difference (MVD). The MVD indicates the difference between the motion vector of the leaf node and the predicted motion vector. The predicted motion vector is the motion vector of the adjacent nodes of the octree. Parameters for leaf nodes may include indices that identify neighboring nodes.

In other examples, similar to merge mode in conventional video coding, the parameters for the leaf nodes do not include MVD, and the motion vectors of the leaf nodes may be assumed to be the same as the motion vectors of the identified neighboring nodes.

In some examples, signaling residuals may be skipped. In some such examples using angular mode, signaling of the primary residual may be skipped, while the secondary residual is still signaled.

1 is a block diagram illustrating an example encoding and decoding system 100 in which the techniques of this disclosure may be implemented. In general, the techniques of this disclosure relate to transcoding (encoding and/or decoding) point cloud data, ie, to support point cloud compression. In general, point cloud profiles include any profile used to process point clouds. Decoding may be efficient in compressing and/or decompressing point cloud data.

As shown in FIG. 1 , system 100 includes source device 102 and target device 116 . Source device 102 provides encoded point cloud material to be decoded by target device 116 . Specifically, in the example of FIG. 1 , source device 102 provides point cloud material to target device 116 via computer-readable medium 110 . Source device 102 and target device 116 may include any of a wide variety of devices, including desktop computers, notebook (ie, laptop) computers, tablet computers, set-top boxes, telephone handsets (such as smart phones), televisions , cameras, display devices, digital media players, video game consoles, video streaming devices, land or sea vehicles, spacecraft, aircraft, robots, LIDAR devices, satellites, etc. In some cases, source device 102 and target device 116 may be equipped for wireless communication.

In the example of FIG. 1 , source device 102 includes data source 104 , memory 106 , G-PCC encoder 200 , and output interface 108 . Target device 116 includes input interface 122 , G-PCC decoder 300 , memory 120 , and data consumer 118 . According to the present disclosure, the G-PCC encoder 200 of the source device 102 and the G-PCC decoder 300 of the target device 116 may be configured to apply the present disclosure in combination with octree decoding and predictive decoding to use Techniques related to hybrid tree coding methods for enhanced inter/intra prediction at block level for point cloud compression. Thus, source device 102 represents an example of an encoding device, while target device 116 represents an example of a decoding device. In other examples, source device 102 and target device 116 may include other components or arrangements. For example, source device 102 may receive data (eg, point cloud data) from internal or external sources. Likewise, the target device 116 may interface with external data consumers, rather than including the data consumers in the same device.

The system 100 as shown in FIG. 1 is only one example. In general, other digital encoding and/or decoding devices may perform the present disclosure in combination with octree coding and predictive coding for enhanced inter/intra prediction at the block level for point cloud compression related technologies. Source device 102 and target device 116 are merely examples of such devices in which source device 102 generates decoded material for transmission to target device 116 . This disclosure refers to a "decoding" device as a device that performs the decoding (encoding and/or decoding) of material. Thus, G-PCC encoder 200 and G-PCC decoder 300 represent examples of decoding devices (specifically, encoder and decoder, respectively). In some examples, source device 102 and target device 116 may operate in a substantially symmetrical manner, such that source device 102 and target device 116 each include encoding and decoding components. Thus, system 100 may support one-way or two-way transmission between source device 102 and target device 116, eg, for streaming, playback, broadcast, telephony, navigation, and other applications.

In general, material source 104 represents a source of material (ie, raw, unencoded point cloud material) and may provide a sequential series of "frames" of material to G-PCC encoder 200, which G-PCC encoder 200 has The data used for the frame is encoded. Source 104 of source device 102 may include a point cloud capture device, such as any of a variety of cameras or sensors, eg, a 3D scanner or light detection and ranging (LIDAR) device, one or more cameras, including Archive of previously captured data, and/or a data feed interface for receiving data from data content providers. Alternatively or additionally, point cloud data may be computer-generated or other data from scanners, cameras, sensors. For example, material sources 104 may generate computer graphics-based material as source material, or a combination of live material, archive material, and computer-generated material. In each case, the G-PCC encoder 200 encodes captured, pre-captured or computer-generated material. The G-PCC encoder 200 may rearrange the frames from the received order (sometimes referred to as "display order") to the decoding order for decoding. G-PCC encoder 200 may generate one or more bitstreams that include encoded material. Source device 102 may then output the encoded data onto computer-readable medium 110 via output interface 108 for reception and/or retrieval by input interface 122 of target device 116, for example.

Memory 106 of source device 102 and memory 120 of target device 116 may represent general purpose memory. In some examples, memory 106 and memory 120 may store raw data, eg, raw data from data source 104 and raw decoded data from G-PCC decoder 300 . Additionally or alternatively, memory 106 and memory 120 may store software instructions executable by, for example, G-PCC encoder 200 and G-PCC decoder 300, respectively. Although memory 106 and memory 120 are shown separate from G-PCC encoder 200 and G-PCC decoder 300 in this example, it should be understood that G-PCC encoder 200 and G-PCC decoder The device 300 may also include internal memory for a functionally similar or equivalent purpose. In addition, memory 106 and memory 120 may store encoded data output from G-PCC encoder 200 and input to G-PCC decoder 300, for example. In some examples, portions of memory 106 and memory 120 may be allocated as one or more buffers, eg, to store raw decoded and/or encoded data. For example, memory 106 and memory 120 may store data representing point clouds.

Computer-readable medium 110 may represent any type of media or device capable of transmitting encoded material from source device 102 to target device 116 . In one example, computer-readable medium 110 represents a communication medium that enables source device 102 to send encoded material directly to target device 116 in real-time, such as via a radio frequency network or a computer-based network. The output interface 108 may modulate the transmission signal including the encoded data, and the input interface 122 may demodulate the received transmission information according to a communication standard, such as a wireless communication protocol. Communication media may include any wireless or wired communication media, such as a radio frequency (RF) spectrum or one or more physical transmission lines. The communication medium may form part of a packet-based network such as a local area network, a wide area network, or a global network such as the Internet. Communication media may include routers, switches, base stations, or any other device that may be useful to facilitate communication from source device 102 to target device 116 .

In some examples, source device 102 may output the encoded data from output interface 108 to storage device 112 . Similarly, target device 116 may access encoded data from storage device 112 via input interface 122 . Storage device 112 may include any of a variety of distributed or locally-accessed data storage media, such as hard drives, Blu-ray discs, DVDs, CD-ROMs, flash memory, volatile or non-volatile memory, or any other suitable digital storage medium for storing encoded data.

In some examples, source device 102 may output the encoded data to file server 114 or another intermediate storage device that may store the encoded data generated by source device 102 . The target device 116 may access the stored data from the file server 114 via streaming or downloading. File server 114 may be any type of server device capable of storing encoded data and sending the encoded data to target device 116 . File server 114 may represent a web server (eg, for a website), a file transfer protocol (FTP) server, a content delivery network device, or a network attached storage (NAS) device. Target device 116 may access encoded data from file server 114 over any standard data connection, including an Internet connection. This may include wireless channels (eg, Wi-Fi connections), wired connections (eg, digital subscriber line (DSL), cable modem, etc.) suitable for accessing encoded data stored on file server 114, or a combination of the two. File server 114 and input interface 122 may be configured to operate according to a streaming protocol, a download protocol, or a combination thereof.

Output interface 108 and input interface 122 may represent wireless transmitters/receivers, modems, wired networking components (eg, Ethernet cards), wireless communication components operating in accordance with any of a variety of IEEE 802.11 standards, or other entity components. In examples in which output interface 108 and input interface 122 include wireless components, output interface 108 and input interface 122 may be configured according to cellular communication standards such as 4G, 4G-LTE (Long Term Evolution), LTE-Advanced, 5G, etc. ) to transmit data, such as encoded data. In some examples in which output interface 108 includes a wireless transmitter, output interface 108 and input interface 122 may be configured in accordance with other wireless standards (such as the IEEE 802.11 specification, the IEEE 802.15 specification (eg, ZigBee™), the Bluetooth™ standard, etc.) to transmit data, such as encoded data. In some examples, source device 102 and/or target device 116 may include respective system-on-chip (SoC) devices. For example, source device 102 may include a SoC device for performing functions ascribed to G-PCC encoder 200 and/or output interface 108, and target device 116 may include an SoC device for performing functions ascribed to G-PCC decoder 300 and/or The SoC device that inputs the functions of the interface 122 .

The techniques of this disclosure may be applied to encoding and decoding to support any of a variety of applications, such as communications between autonomous vehicles, in scanners, cameras, sensors, and processing devices such as local or remote servers ), geographic mapping or other applications.

The input interface 122 of the target device 116 receives the encoded bitstream from the computer-readable medium 110 (eg, communication medium, storage device 112, file server 114, etc.). The encoded bitstream may include signaling information defined by G-PCC encoder 200 (which is also used by G-PCC decoder 300 ), such as with descriptions of coded units (eg, slices, pictures, groups of pictures) , sequence, etc.) properties and/or syntax elements of the processed value. The data consumer 118 uses the decoded data. For example, the data consumer 118 may use the decoded data to determine the location of the physical object. In some examples, data consumer 118 may include a display that renders imagery based on a point cloud.

G-PCC encoder 200 and G-PCC decoder 300 may each be implemented as any of a wide variety of suitable encoder and/or decoder circuits, such as one or more microprocessors, digital signal processors (DSP), Application Specific Integrated Circuit (ASIC), Field Programmable Gate Array (FPGA), discrete logic, software, hardware, firmware, or any combination thereof. When the techniques are implemented partially in software, the apparatus may store instructions for the software in a suitable non-transitory computer-readable medium and execute the instructions in hardware using one or more processors to perform the present invention Techniques for Disclosing Content. Each of the G-PCC encoder 200 and the G-PCC decoder 300 may be included in one or more encoders or decoders, either of which may be integrated into a corresponding device part of a combined encoder/decoder (CODEC). Devices including G-PCC encoder 200 and/or G-PCC decoder 300 may include one or more integrated circuits, microprocessors, and/or other types of devices.

The G-PCC encoder 200 and the G-PCC decoder 300 may operate according to a coding standard, such as the Video Point Cloud Compression (V-PCC) standard or the Geometric Point Cloud Compression (G-PCC) standard. In general terms, the present disclosure may relate to the coding (eg, encoding and decoding) of pictures to include the process of encoding or decoding data. The encoded bitstream typically includes a series of values for syntax elements representing coding decisions (eg, coding modes).

In general terms, the present disclosure may involve "signaling" certain information, such as syntax elements. The term "signaling" may generally refer to the transmission of values for syntax elements and/or other data used to decode encoded data. That is, G-PCC encoder 200 may signal values for syntax elements in the bitstream. Generally, signaling refers to generating a value in a bitstream. As described above, source device 102 may transmit the bitstream to target device 116 substantially instantaneously or not, such as may occur when syntax elements are stored to storage device 112 for later retrieval by target device 116 .

ISO/IEC MPEG (JTC 1/SC 29/WG 11) is investigating the potential need for standardization of point cloud decoding techniques whose compression capabilities significantly exceed those of current methods. The group is working together on this exploratory activity in a collaborative collaboration known as the 3D Graphics Group (3DG) to evaluate compression technology designs proposed by their experts in the field.

Point cloud compression activities are classified into two different approaches. The first method is "Video Point Cloud Compression" (V-PCC), which segments 3D objects and projects the segments on multiple 2D planes (which are represented as "patches" in 2D frames) , it is further coded by conventional 2D video codecs such as the High Efficiency Video Coding (HEVC) (ITU-T H.265) codec. The second method is "Geometry-Based Point Cloud Compression" (G-PCC), which directly compresses 3D geometry (i.e., the location of a collection of points in 3D space), and (for each point) associated attribute value. G-PCC solves the problem of compression of point clouds under both category 1 (static point cloud) and category 3 (dynamically acquired point cloud). A recent draft of the G-PCC standard is available in the following document: "Text of ISO/IEC FDIS 23090-9 Geometry-based Point Cloud Compression", ISO/IEC JTC 1/SC29/WG 7 MDS19617, Conference Call, October 2020 month, and a description of the codec is available in the following document: G-PCC Codec Description, ISO/IEC JTC 1/SC29/WG 7 MDS19620, Conference Call, October 2020.

A point cloud contains a collection of points in 3D space and can have attributes associated with the points. Attributes may be color information (such as R, G, B or Y, Cb, Cr), or reflectance information, or other attributes. Point clouds can be captured by various cameras or sensors, such as LIDAR sensors and 3D scanners, and can also be computer-generated. Point cloud data are used in a variety of applications, including but not limited to: construction (modeling), graphics (3D models for visualization and animation), and the automotive industry (LIDAR sensors to aid navigation).

The 3D space occupied by the point cloud data can be enclosed by a virtual bounding box. The location of a point in a bounding box can be represented with some precision; therefore, the location of one or more points can be quantified based on the precision. At the minimum level, the bounding box is split into voxels, which are the smallest spatial units represented by a unit cube. The voxels in the bounding box can be associated with zero, one, or more than one points. A bounding box can be split into cubes/cube regions, which can be called tiles. Each tile can be coded into one or more slices. The division of the bounding box into slices and tiles may be based on the number of points in each partition, or based on other considerations (eg, certain regions may be coded as tiles). Slice regions can be further segmented using split decisions similar to those in video codecs.

FIG. 2 provides an overview of the G-PCC encoder 200 . FIG. 3 provides an overview of G-PCC decoder 300 . The modules shown are logical and not necessarily related to the implementation in the reference implementation of the G-PCC codec (ie, the TMC13 test model software studied by ISO/IEC MPEG (JTC 1/SC 29/WG 11)) The codes correspond one-to-one.

In both the G-PCC encoder 200 and the G-PCC decoder 300, the point cloud positions are first decoded. Attribute coding depends on the decoded geometry. Surface approximation analysis unit 212 and RAHT unit 218 of FIG. 2 and surface approximation synthesis unit 310 and RAHT unit 314 of FIG. 3 are options typically used for Category 1 material. LOD generation unit 220 and lift unit 222 and LOD generation unit 316 and reverse lift unit 318 of FIG. 3 are options typically used for Category 3 material. All other mods are shared between Category 1 and Category 3.

For geometry, there are two different types of coding techniques: octree coding and predictive tree coding. In the following, the present disclosure focuses on octree coding. For category 3 data, the compressed geometry is typically represented as an octree from the root all the way down to the leaf level of individual voxels. For Category 3 data, the compressed geometry is typically represented as an octree from the root all the way down to the leaf level of a single voxel. For category 1 data, the compressed geometry is typically composed of a pruned octree (ie, an octree from the root down to the leaf level of blocks larger than voxels) plus the pair of octrees in the pruned octree The surfaces within each lobe are represented by an approximate model. In this way, both class 1 data and class 3 data share the octree decoding mechanism, while class 1 data can additionally utilize a surface model to approximate the voxels within each leaf. The surface model used is a triangulation, which includes 1-10 triangles per block, resulting in a triangle soup. Therefore, class 1 geometry codecs are called Trisoup geometry codecs, and class 3 geometry codecs are called octree geometry codecs.

4 is a conceptual diagram illustrating an example octree split for geometry coding. Octree 400 includes 8 child nodes. Some of these child nodes, such as node 402, have no child nodes. However, other child nodes, such as node 404, do have child nodes, and some child nodes of node 404 also have child nodes, and so on.

At each node of the octree, occupancy is signaled (when not inferred) for one or more of the child nodes (up to eight nodes). Multiple neighborhoods are specified, including: (a) nodes that share faces with the current octree node, (b) nodes that share faces, edges, or vertices with the current octree node, etc. Within each neighborhood, the occupancy of the node and/or its children can be used to predict the occupancy of the current node or its children. For points that are sparsely populated in certain nodes of the octree, the codec also supports a direct coding mode, where the 3D position of the point is directly encoded. A flag can be signaled to indicate that direct mode is signaled. At the lowest level, the number of points associated with the octree node/leaf node can also be decoded.

Once the geometry is decoded, the attributes corresponding to the geometry points are decoded. When there are multiple attribute points corresponding to one reconstructed/decoded geometry point, an attribute value representing the reconstructed point can be derived.

There are three attribute coding methods in G-PCC: Region Adaptive Hierarchical Transform (RAHT) coding, Interpolation-based Hierarchical Nearest Neighbor Prediction (Predictive Transform), and Interpolation-based Hierarchical with Update/Boost Steps Nearest neighbor prediction (boosting transform). RAHT and boost are usually used for category 1 data, while prediction is usually used for category 3 data. However, either method can be used for any material, and just like the geometry codec in G-PCC, the attribute decoding method used to decode the point cloud is specified in the bitstream.

The decoding of attributes can be done in levels of detail (LoD), where with each level of detail a finer representation of the point cloud attributes can be obtained. Each level of detail can be specified based on a distance metric from neighboring nodes or based on sampling distance.

At the G-PCC encoder 200, the residual obtained as the output of the coding method for attributes is quantized. The quantized residual may be coded using context adaptive arithmetic coding.

In the example of FIG. 2, the G-PCC encoder 200 may include a coordinate transformation unit 202, a color transformation unit 204, a voxelization unit 206, an attribute transfer unit 208, an octree analysis unit 210, a surface approximation analysis unit 212, an arithmetic The encoding unit 214 , the geometry reconstruction unit 216 , the RAHT unit 218 , the LOD generation unit 220 , the lifting unit 222 , the coefficient quantization unit 224 , and the arithmetic encoding unit 226 .

As shown in the example of FIG. 2, the G-PCC encoder 200 may receive a set of locations and a set of attributes. The location may include the coordinates of the point in the point cloud. Attributes may include information about points in the point cloud, such as colors associated with points in the point cloud.

The coordinate transformation unit 202 may apply a transformation to the coordinates of the points to transform the coordinates from the original domain to the transformed domain. This disclosure may refer to the transformed coordinates as transformed coordinates. Color transform unit 204 may apply transforms to transform color information of attributes to different domains. For example, the color transform unit 204 may transform the color information from the RGB color space to the YCbCr color space.

Furthermore, in the example of FIG. 2, voxelization unit 206 may voxelize the transformed coordinates. Voxelization of the transformed coordinates may include quantizing and removing some points in the point cloud. In other words, multiple points in a point cloud may be grouped within a single "voxel", which may thereafter be considered a point in some aspects. Furthermore, octree analysis unit 210 may generate an octree based on the voxelized transformed coordinates. Additionally, in the example of FIG. 2, the surface approximation analysis unit 212 may analyze the points to potentially determine a surface representation of a set of points. Arithmetic encoding unit 214 may entropy encode syntax elements representing information about the octree and/or the surface determined by surface approximation analysis unit 212 . G-PCC encoder 200 may output these syntax elements in a geometry bitstream.

The geometry reconstruction unit 216 may reconstruct the transformed coordinates of the points in the point cloud based on the octree, data indicative of the surface determined by the surface approximation analysis unit 212, and/or other information. Due to voxelization and surface approximation, the number of transformed coordinates reconstructed by the geometry reconstruction unit 216 may differ from the original number of points of the point cloud. The present disclosure may refer to the resulting points as reconstructed points. The attribute transfer unit 208 may transfer the attributes of the original points of the point cloud to the reconstructed points of the point cloud.

Additionally, RAHT unit 218 may apply RAHT coding to the attributes of the reconstructed points. Alternatively or additionally, LOD generation unit 220 and lifting unit 222 may apply LOD processing and lifting, respectively, to attributes of the reconstructed points. RAHT unit 218 and boost unit 222 may generate coefficients based on attributes. Coefficient quantization unit 224 may quantize coefficients generated by RAHT unit 218 or boosting unit 222 . Arithmetic encoding unit 226 may apply arithmetic coding to syntax elements representing quantized coefficients. The G-PCC encoder 200 may output these syntax elements in the attribute bitstream.

In the example of FIG. 3, the G-PCC decoder 300 may include a geometry arithmetic decoding unit 302, an attribute arithmetic decoding unit 304, an octree synthesis unit 306, an inverse quantization unit 308, a surface approximation synthesis unit 310, a geometry reconstruction unit 312 , RAHT unit 314 , LoD generation unit 316 , inverse lift unit 318 , inverse transform coordinate unit 320 , and inverse transform color unit 322 .

The G-PCC decoder 300 can obtain geometry bitstreams and attribute bitstreams. The geometry arithmetic decoding unit 302 of the G-PCC decoder 300 may apply arithmetic decoding (eg, context adaptive binary arithmetic coding (CABAC) or other types of arithmetic decoding) to syntax elements in the geometry bitstream. Similarly, attribute arithmetic decoding unit 304 may apply arithmetic decoding to syntax elements in the attribute bitstream.

Octree synthesis unit 306 may synthesize octrees based on syntax elements parsed from the geometry bitstream. Where surface approximation is used in the geometry bitstream, the surface approximation synthesis unit 310 may determine the surface model based on syntax elements parsed from the geometry bitstream and based on an octree.

Additionally, geometry reconstruction unit 312 may perform reconstruction to determine coordinates of points in the point cloud. Inverse transform coordinates unit 320 may apply an inverse transform to the reconstructed coordinates to transform the reconstructed coordinates (positions) of points in the point cloud from the transformed domain back to the original domain.

In addition, in the example of FIG. 3, the inverse quantization unit 308 may inverse quantize the attribute value. The attribute value may be based on syntax elements obtained from the attribute bitstream (eg, including syntax elements decoded by attribute arithmetic decoding unit 304).

Depending on how the attribute values are encoded, RAHT unit 314 may perform RAHT coding to determine color values for the points of the point cloud based on the inverse quantized attribute values. Alternatively, LOD generation unit 316 and inverse boosting unit 318 may use level of detail based techniques to determine color values for the points of the point cloud.

Furthermore, in the example of FIG. 3, inverse transform color unit 322 may apply an inverse color transform to the color values. The inverse color transform may be the inverse process of the color transform applied by the color transform unit 204 of the G-PCC encoder 200 . For example, the color transform unit 204 may transform the color information from the RGB color space to the YCbCr color space. Therefore, the inverse color transform unit 322 can transform the color information from the YCbCr color space to the RGB color space.

The various elements of FIGS. 2 and 3 are shown to aid in understanding the operations performed by G-PCC encoder 200 and G-PCC decoder 300 . A unit may be implemented as a fixed function circuit, a programmable circuit, or a combination thereof. A fixed function circuit refers to a circuit that provides a specific function, and is preset with respect to operations that can be performed. Programmable circuits refer to circuits that can be programmed to perform various tasks and provide flexible functionality in operations that can be performed. For example, the programmable circuit may execute software or firmware that causes the programmable circuit to operate in a manner defined by the instructions of the software or firmware. Fixed-function circuits may execute software instructions (eg, to receive parameters or output parameters), but the type of operations performed by fixed-function circuits is generally immutable. In some examples, one or more of these units may be distinct circuit blocks (fixed function or programmable), and in some examples, one or more of these units may be integrated circuit.

Predictive geometry coding is introduced as an alternative to octree geometry coding, where nodes are arranged in a tree structure (which defines the prediction structure), and various prediction strategies are used to predict each node in the tree relative to its predictor coordinate.

Figure 5 shows an example of a prediction tree 401, which is represented as a directed graph with arrows pointing in the prediction direction. Node 412 is the root vertex and has no predictor. Nodes 414A and 414B have two child nodes. The dotted node has 3 children. A node filled with white has one child, and nodes 418A-418E are leaf nodes with no children. Each node has only one parent node.

Four prediction strategies are specified for each node, based on its parent (p0), grandfather (p1), and great-grandfather (p2): (1) No prediction/zero prediction (0) (2) Incremental prediction (p0) (3) Linear prediction (2*p0 – p1) (4) Parallelogram prediction (2*p0 + p1 – p2)

The G-PCC encoder 200 can employ any algorithm to generate the prediction tree; the algorithm used can be determined based on the application/use case, and several strategies can be used. Some strategies are described in the following document: G-PCC Codec Description, ISO/IEC JTC 1/SC29/WG 7 MDS19620, Conference Call, October 2020.

For each node, the residual coordinate values are decoded in the bitstream in a depth-first manner starting from the root node.

Predictive geometry decoding is mainly used for category 3 (LIDAR collected) point cloud data, eg, for low-latency applications.

（半徑、方位角和雷射折射率），並且在該域中執行預測（在

域中對殘差進行譯碼）。由於四捨五入的誤差，在

中進行譯碼不是無損的，並且因此對與笛卡爾座標相對應的第二殘差集合進行譯碼。下文複製了來自以下文檔的針對用於預測幾何形狀譯碼的角度模式的編碼和解碼策略的描述：G-PCC Codec Description，ISO/IEC JTC 1/SC29/WG 7 MDS19620，電話會議，2020年10月。 The angle mode can be used in predictive geometry coding, where the properties of the LIDAR sensor can be used to more efficiently code the predictive tree. Convert the coordinates of the location to

(radius, azimuth, and laser refractive index), and perform predictions in this domain (in

The residuals are decoded in the domain). Due to rounding errors, the

Coding in is not lossless, and therefore a second set of residuals corresponding to Cartesian coordinates is coded. The description of encoding and decoding strategies for angular modes for predictive geometry coding is reproduced below from the following document: G-PCC Codec Description, ISO/IEC JTC 1/SC29/WG 7 MDS19620, Conference Call, October 2020 moon.

圍繞Z軸旋轉（參見圖6）。每個雷射器可以具有不同的仰角

和高度

。假設雷射

擊中點

，其具有根據在圖6中描述的座標系定義的笛卡爾整數座標

。 The method focuses on point clouds acquired using a rotating lidar (lidar) model. Here, the lidar has N lasers (eg, N=16, 32, 64), which vary according to the azimuth angle

Rotate around the Z axis (see Figure 6). Each laser can have a different elevation angle

and height

. hypothetical laser

hit point

, which has Cartesian integer coordinates defined according to the coordinate system depicted in Figure 6

.

對M的位置進行建模，這些參數計算如下： (1)

(2)

(3)

The method provides for using three parameters

To model the position of M, these parameters are calculated as follows: (1)

(2)

(3)

的量化版本，其被表示為

，其中，三個整數

、

和

計算如下： (1)

(2)

(3)

其中： (1) (

和

分別是控制

和

的精度的量化參數。 (2)

是函數，其在t為正數的情況下返回1，否則返回(-1)。 (3)

是

的絕對值。 More precisely, the method uses

A quantified version of , which is expressed as

, where three integers

,

and

The calculation is as follows: (1)

(2)

(3)

of which: (1) (

and

respectively control

and

The precision of the quantization parameter. (2)

is a function that returns 1 if t is positive and (-1) otherwise. (3)

Yes

the absolute value of .

和

的值：

其中： (1) (

和

分別是控制

和

的精度的量化參數。 如下獲得經重建的笛卡爾座標： (1)

(2)

(3)

, 其中，

和

是

和

的近似。計算可以使用定點表示、查找表和線性插值。 應注意，由於各種原因，

可以與

不同： (1) 量化 (2) 近似 (3) 模型不精確 (4) 模型參數不精確性 令

為重建殘差，其定義如下： (1)

(2)

(3)

在該方法中，編碼器（例如，G-PCC編碼器200）如下進行： (1) 對模型參數

以及量化參數

、

進行編碼 (2) 將在以下文檔中描述的幾何形狀預測方案應用於表示

：Text   of ISO/IEC FDIS 23090-9 Geometry-based Point Cloud Compression，ISO/IEC JTC 1/SC29/WG 7 MDS19617，電話會議，2020年10月。 ＜1＞  可以引入一種利用雷射雷達的特性的新預測器。例如，雷射雷達掃描器圍繞z軸的旋轉速度通常是恒定的。因此，可以如下預測當前

：

其中 (1)

是編碼器可以從中選擇的潛在速度集合。索引

可以 顯式地寫入位元流，或者可以基於編碼器和解碼器兩者應用的確定性策略從上下文推斷，以及 (2)

是可以顯式地寫入位元流或可以基於編碼器和解碼器兩者應用的確定性策略從上下文推斷的跳過的點的數量。

稍後也可以被稱為“phi乘數”，並且在一些實現中，可僅與增量預測器一起使用。 (3) 將重建殘差

與每個節點一起編碼。 To avoid reconstruction mismatches due to the use of floating point arithmetic, precompute and quantize as follows

and

The value of:

of which: (1) (

and

respectively control

and

The precision of the quantization parameter. The reconstructed Cartesian coordinates are obtained as follows: (1)

(2)

(3)

, in,

and

Yes

and

approximation. Calculations can use fixed-point representation, look-up tables, and linear interpolation. It should be noted that for various reasons,

With

Different: (1) Quantization (2) Approximation (3) Model inaccuracy (4) Model parameter inaccuracy makes

To reconstruct the residual, it is defined as follows: (1)

(2)

(3)

In this method, an encoder (eg, G-PCC encoder 200 ) proceeds as follows: (1) For model parameters

and quantization parameters

,

To encode (2) apply the geometry prediction scheme described in the following document to the representation

: Text of ISO/IEC FDIS 23090-9 Geometry-based Point Cloud Compression, ISO/IEC JTC 1/SC29/WG 7 MDS19617, Conference Call, October 2020. <1> A new predictor that utilizes the properties of lidar can be introduced. For example, the rotational speed of a lidar scanner around the z-axis is usually constant. Therefore, the current can be predicted as follows

:

of which (1)

is the set of potential speeds from which the encoder can choose. index

The bitstream may be written explicitly, or may be inferred from context based on deterministic policies applied by both the encoder and decoder, and (2)

is the number of skipped points that can be explicitly written to the bitstream or inferred from context based on deterministic policies applied by both the encoder and decoder.

It may also be referred to later as the "phi multiplier", and in some implementations, may be used only with the delta predictor. (3) will reconstruct the residual

Coded with each node.

以及量化參數

、

進行編碼 (2) 根據在以下文檔中描述的幾何形狀預測方案來對與節點相關聯的 參數

進行編碼：Text of ISO/IEC FDIS 23090-9 Geometry-based Point Cloud Compression，ISO/IEC JTC 1/SC29/WG 7 MDS19617，電話會議，2020年10月。 (3) 如上所述地計算經重建的座標

(4) 對殘差

進行編碼 ＜1＞如在下一節中討論的，可以通過對重建殘差(r_x,r_y,r_z)進行量 化來支持有損壓縮 (5) 如下计算原始坐标

＜1＞

＜2＞

＜3＞

The G-PCC decoder 300 proceeds as follows: (1) Compare the model parameters

and quantization parameters

,

Encode (2) the parameters associated with the nodes according to the geometry prediction scheme described in the following document

Encoding: Text of ISO/IEC FDIS 23090-9 Geometry-based Point Cloud Compression, ISO/IEC JTC 1/SC29/WG 7 MDS19617, Conference Call, October 2020. (3) Calculate the reconstructed coordinates as above

(4) For the residual

Encoding <1> As discussed in the next section, lossy compression can be supported by quantizing the reconstruction residuals (r_x, r_y, r_z) (5) The original coordinates are calculated as follows

<1>

<2>

<3>

或通過丟棄點來實現。 Lossy compression can be achieved by applying quantization to the reconstructed residual

or by dropping points.

(2)

(3)

其中，

、

和

分別是控制

、

和

的精度的量化參數。 The quantized reconstruction residual is calculated as follows: (1)

(2)

(3)

in,

,

and

respectively control

,

and

The precision of the quantization parameter.

Trellis quantization can be used to further improve RD (Rate Distortion) performance results.

The quantization parameters can be changed at the sequence/frame/slice/block level to achieve region-adaptive quality and for rate control purposes.

The G-PCC encoder 200 may be configured to perform motion estimation for inter prediction. The motion estimation (global and local) process applied in the InterEM software is described below. InterEM is an octree-based coding extension for inter prediction. Although motion estimation applies to octree-based frameworks, a similar process (or at least a portion thereof) can also be applied to predictive geometry coding.

There are two kinds of motion involved in G-PCC InterEM software: global motion matrix and local node motion vector. Global motion parameters include rotation matrices and translation vectors that can be applied to all points in the predicted (reference) frame. A local node motion vector for a node of an octree is a motion vector that applies only to points within the node in the predicted (reference) frame. Details of the motion estimation algorithm in InterEM are described below.

Figure 7 shows a flowchart illustrating the motion estimation process. Inputs to this process include predicted frame 420 and current frame 422 . G-PCC encoder 200 first estimates global motion on a global scale (424). After applying the estimated global motion to the predicted frame 420 (426), the G-PCC encoder 200 estimates the local motion at a finer scale (at the node level in the octree) (428). Finally, G-PCC encoder 200 applies motion compensation to the estimated local node motion and encodes the determined motion vector and points (430).

Aspects of FIG. 7 are explained in greater detail below. G-PCC encoder 200 may perform procedures for estimating global motion matrices and translation vectors. In the InterEM software, a global motion matrix is defined to match feature points between the predicted frame (reference) and the current frame.

FIG. 8 shows an example of a global motion estimation process that may be performed by the G-PCC encoder 200 . In the example of FIG. 8, the G-PCC encoder 200 finds feature points (432), samples the feature points (434), and performs motion estimation (436) using a least mean squares (LMS) algorithm.

In the algorithm shown in Fig. 8, points with large positional changes between the predicted frame and the current frame can be defined as feature points. For each point in the current frame, the closest point in the predicted frame is found, and a point pair is established between the current frame and the predicted frame. If the distance between paired points is greater than a threshold, the paired points are regarded as feature points.

After the feature points are found, sampling is performed on the feature points to reduce the scale of the problem (eg, by selecting a subset of the feature points to reduce the complexity of motion estimation). Then, an LMS algorithm is applied in order to derive motion parameters by trying to reduce the error between the corresponding feature points in the predicted frame and the current frame.

Figure 9 shows an example process for estimating local node motion vectors. In the local node estimation algorithm shown in Fig. 9, motion vectors are estimated recursively. The cost function for selecting the best appropriate motion vector may be based on rate-distortion cost. In Figure 9, path 440 shows the process for the current node that is not split into 8 child nodes, and path 442 shows the process for the current node that is split into 8 child nodes.

If the current node is not split into 8 child nodes (440), then determine the motion vector that can result in the lowest cost between the current node and the predicted node. If the current node is divided into 8 child nodes (442), a motion estimation algorithm is applied and the total cost under split condition is obtained by adding the estimated cost value of each child node. By comparing the cost of splitting and not splitting, decide whether to split or not to split; if split, each child node is assigned its corresponding motion vector (or can be further split into its children), if not split, assign a motion vector to the current node.

Two parameters that affect the performance of motion vector estimation are the block size (BlockSize) and the minimum prediction unit size (MinPUSize). BlockSize defines an upper bound for the node size to which motion vector estimation is to be applied, and MinPUSize defines a lower bound.

The InterEM software, which is essentially an octree decoder, performs occupancy prediction and uses global/local motion and reference point cloud information when doing occupancy prediction. Therefore, the InterEM software does not perform direct motion compensation of points (which may, for example, include applying motion to points in a reference frame to project the points to the current frame). The difference between the actual point and the predicted point can then be decoded, which can be more efficient when performing inter prediction.

One or more of the techniques disclosed in this document may be applied individually or in combination. This disclosure proposes techniques for performing direct motion compensation while still benefiting from flexible octree partitioning based coding structures. In the following, these techniques are mainly shown in the context of octree splitting, but can also be extended to OTQTBT (octree-quadtree-binary tree) splitting scenarios.

G-PCC encoder 200 and/or G-PCC decoder 300 may be configured to perform high-level splitting and process mode flags. In one example of the present disclosure, G-PCC encoder 200 and/or G-PCC decoder 300 may be configured to perform octree-based splitting (for occupancy prediction) on the current point cloud. However, it is possible to stop the octree split at a certain level and then directly code the points within the octree volume (hereafter referred to as "direct prediction") instead of coding the occupancy. A leaf node size or octree depth value may be signaled to specify the level at which octree splitting stops and points are coded as octree volumes.

For each such octree node where octree splitting stops and "direct prediction" is activated, a signal may be signaled indicating whether the set of points within the octree volume is intra-predicted or inter-predicted symbols of. In the geometry parameter set, you can define the maximum and minimum size for octree leaves.

FIG. 10 is a conceptual diagram illustrating high-level octree splitting of octree 444 . Figure 10 is an example of an octree node containing 13 points (O0 to O12) for direct prediction. In special cases, the root node of the octree (without splitting) can be decoded using "direct prediction".

G-PCC encoder 200 and/or G-PCC decoder 300 may be configured to perform intra prediction. When the flag value is set to intra, all points within the volume are intra-predicted. For this, a "local prediction tree" is generated. The generation of such a tree is non-canonical (the points may be traversed in a different order, such as azimuthal, Morton, radial, or some other order). For each point, its prediction mode (0, 1, 2, 3), number of child nodes information, primary and secondary residuals (if angle mode is enabled) are signaled. So, in summary, intra prediction is similar in function to predictive geometry coding.

Alternatively, in addition, a single prediction mode is signaled for all points in the octree volume, which can reduce the associated signaling cost. The radius value (if angular mode is enabled) or the (x,y,z) value (if angular mode is disabled) for the zero predictor can be set to e.g. the left vertex inside the octree volume. Alternatively, a zero predictor may be signaled for the octree volume, or an index may be signaled indicating a point within the octree volume to be used for the zero predictor. Also, clipping can be performed after performing prediction/reconstruction if the value is outside the octree volume.

The syntax table for intra prediction can be similar to the syntax described for prediction trees in the following document: Text of ISO/IEC FDIS 23090-9 Geometry-based Point Cloud Compression, ISO/IEC JTC 1/SC29/WG 7 MDS19617, Conference Call, October 2020, the entire contents of which are incorporated by reference.

G-PCC encoder 200 and/or G-PCC decoder 300 may be configured to perform inter prediction. Assuming that the octree has N points within it: (O(0), ...., O(N-1)), for inter prediction, the current set of points in the octree volume is used at the encoder side to Motion estimation is performed, and the best match is found to a set of similar points in the reference point cloud frame (where the reference point cloud may be motion-uncompensated or globally motion-compensated). For inter prediction of octree leaves, the following are signaled: i. Reference index (if there are multiple reference point cloud frames to predict from) ii. Motion Vector Difference (MVD). (difference between the actual MV and the predicted MV (as mentioned above about performing MV prediction from neighbors)) iii. The number of points in the octopus leaf (N) iv. Primary residual (and secondary residual if angle mode is enabled) for N points (R'i) (tuple, difference between 3D coordinates).

。 In the following, the present disclosure describes the motion compensation process given the signaled reference index (if applicable) and MV for the octree node. a. The current octree has a top-left point at (X0, Y0, Z0) and has dimensions of (Sx, Sy, Sz), and the motion vector is MV = (MVx, MVy, MVz). Therefore, the corresponding reference block in the reference point cloud frame has a point at the upper left of (Xr, Yr, Zr) = (X0 – MVx, Y0 – MVy, Z0 – MVz), and is of size (Sx, Sy, Sz ) b. Extract all points within this reference block and arrange them into a 1D array, the ordering may be predetermined/fixed or signaled (eg for octrees). Say, there are M such points with the following coordinates (in the reference frame): (R0,....R(M-1)), where Ri is a triple providing the 3D coordinates of the ith point. For i = 0...(M-1)), as shown in Figure 12. c. All points are motion compensated by applying the signaled MV, which is used as the predicted geometric position (Pi), i.e., Pi = Ri + MV, as shown in Figure 13. d. If angle mode is enabled, for all M points, derive the corresponding

.

In Figure 12, the current point set is labeled O0 to O12, and the reference point set is labeled R0 to R12, where N=M=13. In FIG. 13 , the current set of points is marked as O0 to O12 and the motion compensated reference point sets are marked as P0 to P12, where N=M=13.

Now, three scenarios are possible: i. N = M (the current octree node and the reference block have the same number of points). ii. N > M. iii. N < M.

The first scenario where N=M will now be described. The residuals (primary and, if applicable, secondary) are directly added to the motion compensated points to generate reconstruction = Pi + R'i.

The second scenario where N>M will now be described. Use the last value P(M-1) to expand the motion compensated points in the 1D array (Pi) (i.e. [P'0,...P'(M-1), P'(M),.... P'(N-1)] = [P0, …P(M-1), P(M-1),…..P(M-1)] ), and then add the residual directly to generate the reconstruction = P'i + R'i. Alternatively, a zero predictor is used for expansion.

A third scenario where N < M will now be described. The residuals are summed directly with the first N points (i.e. [P0, ...P(N-1)]) to generate reconstruction = Pi + Ri.

G-PCC encoder 200 and/or G-PCC decoder 300 may be configured to perform MV prediction based on neighbors. It is possible to predict the MV of the current octree from the MV between spatiotemporally adjacent octrees, and the corresponding MV difference can be signaled. In cases where there are multiple spatiotemporal candidates, the MV prediction index can be signaled. In addition to the spatiotemporal neighbor candidates, it is also possible to add previously used MV candidates to the recent history-based MV candidate list.

It is also possible to merge MV information with spatiotemporal neighbors by specifying a signaled "merge flag". In cases where there are multiple spatiotemporal candidates, the merge index can be signaled.

G-PCC encoder 200 and/or G-PCC decoder 300 may be configured to perform skipping the main residual.

For good inter prediction, the main residual applied when angular mode is enabled is usually small, or even close to zero. In this case it is also possible to skip the main residuals entirely for all points in the octree volume. Therefore, the primary_residual_skip flag can be signaled for the octree volume. In this case, the difference between the original point and the predicted point is completely coded with the secondary residual.

Alternatively, the primary_residual_skip_flag may be signaled at an octree level above the octree level and apply to one or more octree levels associated with that octree level.

The following table is a syntax table for inter-predicted octrees. Octree_leaf(X0, Y0, Z0){ if(number_of_references > 1) ref_idx ae(v) num_points_minus1 ae(v) merge_flag ae(v) if(!merge_flag) { mvp_idx ae(v) for( i = 0 ; i <3; i++){ abs_mvd [i] ae(v) if(!abs_mvd[i]) mvd_sign [i] ae(v) } } else merge_idx ae(v) if(geom_angular_enabled_flag) primary_residual_skip_flag ae(v) for(n = 0; n <= num_points_minus1; n++){ for( i = 0 ; i <3; i++){ If(!primary_residual_skip_flag) { abs_primary_residual [n][i] ae(v) if(!abs_primary_residual[n][i]) primary_residual_sign[n][i] ae(v) } abs_secondary_residual [n][i] ae(v) if(!abs_secondary_residual[n][i]) secondary_residual_sign [n][i] ae(v) } } }

The examples in various aspects of the present disclosure may be used alone or in any combination.

14 is a conceptual diagram illustrating an example ranging system 600 that may be used with one or more techniques of the present disclosure. In the example of FIG. 14 , ranging system 600 includes illuminator 602 and sensor 604 . Illuminator 602 may emit light 606 . In some examples, illuminator 602 may emit light 606 as one or more laser beams. Light 606 may have one or more wavelengths, such as infrared wavelengths or visible light wavelengths. In other examples, the light 606 is not coherent laser light. When light 606 encounters an object, such as object 608 , light 606 creates return light 610 . Return light 610 may include backscattered and/or reflected light. Return light 610 may pass through lens 611 , which directs return light 610 to create image 612 of object 608 on sensor 604 . Sensor 604 generates signal 618 based on image 612 . Image 612 may include a collection of points (eg, as represented by the dots in image 612 of FIG. 14 ).

In some examples, illuminator 602 and sensor 604 may be mounted on a rotating structure such that illuminator 602 and sensor 604 capture a 360-degree view of the environment. In other examples, ranging system 600 may include one or more optical components (eg, mirrors, collimators, diffraction gratings, etc.) that enable illuminator 602 and sensor 604 to detect within a specified range (eg, up to 360 degrees) objects. Although the example of FIG. 14 shows only a single illuminator 602 and sensor 604, ranging system 600 may include multiple sets of illuminators and sensors.

In some examples, illuminator 602 generates a structured light pattern. In such an example, ranging system 600 may include a plurality of sensors 604 on which respective images of the light pattern of the structures are formed. The ranging system 600 can use the difference between the images of the light pattern of the structure to determine the distance to the object 608 from which the light pattern of the structure is backscattered. Structured light based ranging systems can have a high level of accuracy (eg, in the sub-millimeter range) when the object 608 is relatively close to the sensor 604 (eg, 0.2 meters to 2 meters). This high level of accuracy may be useful in facial recognition applications, such as unlocking mobile devices (eg, mobile phones, tablets, etc.) and for security applications.

In some examples, ranging system 600 is a time-of-flight (ToF) based system. In some examples in which ranging system 600 is a ToF-based system, illuminator 602 generates light pulses. In other words, the illuminator 602 can modulate the amplitude of the emitted light 606 . In such an example, sensor 604 detects return light 610 from light pulse 606 generated by illuminator 602 . The ranging system 600 can then determine the distance to the object 608 from which the light 606 is backscattered based on the delay between the light 606 being emitted and being detected and the known speed of the light in air. In some examples, the illuminator 602 may modulate the phase of the emitted light 606 instead of (or in addition to) the amplitude of the emitted light 606 . In such an example, the sensor 604 may detect the phase of the returning light 610 from the object 608 and use the speed of light and based on the time at which the illuminator 602 generates the light 606 at a particular phase and the sensor 604 at the particular phase The time difference between when the returning light 610 is detected determines the distance to the point on the object 608 .

In other examples, the point cloud may be generated without the use of the illuminator 602 . For example, in some examples, sensor 604 of ranging system 600 may include two or more optical cameras. In such an example, ranging system 600 may use an optical camera to capture a stereoscopic image of the environment including object 608 . The ranging system 600 (eg, point cloud generator 620 ) can then calculate the differences between the positions in the stereoscopic image. The ranging system 600 can then use this difference to determine the distance to the location shown in the stereoscopic image. Based on these distances, the point cloud generator 620 may generate a point cloud.

Sensor 604 may also detect other properties of object 608, such as color and reflectivity information. In the example of FIG. 14 , point cloud generator 620 may generate a point cloud based on signal 618 generated by sensor 604 . Ranging system 600 and/or point cloud generator 620 may form part of data source 104 (FIG. 1).

15 is a conceptual diagram illustrating an example vehicle-based scenario in which one or more techniques of the present disclosure may be employed. In the example of FIG. 15, vehicle 700 includes a laser package 702, such as a LIDAR system. Although not shown in the example of FIG. 15 , vehicle 700 may also include a material source and a G-PCC encoder, such as G-PCC encoder 200 ( FIG. 1 ). In the example of FIG. 15, the laser package 702 emits a laser beam 704, which is reflected from a pedestrian 706 or other objects in the road. The data source of the vehicle 700 may generate a point cloud based on the signals generated by the laser package 702 . The G-PCC encoder of vehicle 700 may encode the point cloud to generate bitstream 708 . The bitstream 708 may include far fewer bits than the unencoded point cloud obtained by the G-PCC encoder. An output interface of vehicle 700 (eg, output interface 108 (FIG. 1)) may send bitstream 708 to one or more other devices. Therefore, the vehicle 700 may be able to send the bitstream 708 to other devices faster (compared to the unencoded point cloud material). Additionally, bitstream 708 may require less data storage capacity.

In the example of FIG. 15 , vehicle 700 may send bitstream 708 to another vehicle 710 . Vehicle 710 may include a G-PCC decoder, such as G-PCC decoder 300 (FIG. 1). The G-PCC decoder of the vehicle 710 may decode the bitstream 708 to reconstruct the point cloud. Vehicle 710 may use the reconstructed point cloud for various purposes. For example, vehicle 710 may determine, based on the reconstructed point cloud, that pedestrian 706 is in the road ahead of vehicle 700 and therefore begins to decelerate (eg, even before the driver of vehicle 710 realizes pedestrian 706 is in the road). Thus, in some examples, the vehicle 710 may perform autonomous navigation operations, generate a notification or warning, or perform another action based on the reconstructed point cloud.

Additionally or alternatively, vehicle 700 may send bitstream 708 to server system 712 . The server system 712 may use the bitstream 708 for various purposes. For example, server system 712 may store bitstream 708 for subsequent reconstruction of the point cloud. In this example, the server system 712 may use the point cloud along with other data (eg, vehicle telemetry data generated by the vehicle 700 ) to train the autonomous driving system. In other examples, server system 712 may store bitstream 708 for subsequent reconstruction for forensic accident investigation (eg, if vehicle 700 collides with pedestrian 706 ) or may send a notification for navigating to vehicle 700 or vehicle 710 or instruction.

16 is a conceptual diagram illustrating an example extended reality system in which one or more techniques of the present disclosure may be employed. Extended reality (XR) is a term used to cover a range of technologies including: Augmented Reality (AR), Mixed Reality (MR), and Virtual Reality (VR). In the example of FIG. 16 , user 800 is in first location 802 . User 800 wears XR headset 804 . As an alternative to XR headset 804, user 800 may use a mobile device (eg, mobile phone, tablet, etc.). The XR headset 804 includes a depth detection sensor (such as a LIDAR system) that detects the location of a point on the object 806 at the first location 802 . The data source of the XR headset 804 may use the signals generated by the depth detection sensors to generate a point cloud representation of the object 806 at the location 802 . XR headset 804 may include a G-PCC encoder (eg, G-PCC encoder 200 of FIG. 1 ) configured to encode a point cloud to generate bitstream 808 .

The XR headset 804 may send a bitstream 808 (eg, via a network such as the Internet) to the XR headset 810 worn by the user 812 at the second location 814 . The XR headset 810 can decode the bitstream 808 to reconstruct the point cloud. The XR headset 810 may use the point cloud to generate an XR visualization (eg, AR, MR, VR visualization) representing the object 806 at the first location 802 . Thus, in some examples, such as when the XR headset 810 generates a VR visualization, the user 812 at the location 814 may have a 3D immersive experience at the first location 802 . In some examples, the XR headset 810 may determine the location of the virtual object based on the reconstructed point cloud. For example, the XR headset 810 may determine, based on the reconstructed point cloud, that the environment (eg, the first location 802 ) includes a flat surface, and then determine that the virtual object (eg, a cartoon character) is to be positioned on the flat surface. The XR headset 810 may generate an XR visualization in which the virtual object is located at the determined location. For example, the XR headset 810 may display a cartoon character on a flat surface.

17 is a conceptual diagram illustrating an example mobile device system in which one or more techniques of the present disclosure may be employed. In the example of FIG. 17 , a mobile device 900 (such as a mobile phone or tablet) includes a depth detection sensor (such as a LIDAR system) that detects the location of a point on an object 902 in the environment of the mobile device 900 . The data source of the mobile device 900 may generate a point cloud representation of the object 902 using the signals generated by the depth detection sensors. Mobile device 900 may include a G-PCC encoder (eg, G-PCC encoder 200 of FIG. 1 ) configured to encode a point cloud to generate bitstream 904 . In the example of FIG. 17, the mobile device 900 may send a bitstream to a remote device 906, such as a server system or other mobile device. The remote device 906 can decode the bitstream 904 to reconstruct the point cloud. The remote device 906 may use the point cloud for various purposes. For example, remote device 906 may use the point cloud to generate a map of the environment of mobile device 900 . For example, the remote device 906 may generate a map of the interior of the building based on the reconstructed point cloud. In another example, the remote device 906 may generate imagery (eg, computer graphics) based on the point cloud. For example, the remote device 906 may use the points of the point cloud as the vertices of the polygon, and use the color properties of the points as the basis for coloring the polygon. In some examples, the remote device 906 may use the point cloud to perform facial recognition.

18 is a flowchart illustrating example operations for decoding a bitstream including point cloud material. G-PCC decoder 300 may perform the operations of FIG. 18 as part of decoding the point cloud. In the example of Figure 18, the G-PCC decoder 300 determines an octree (1000) that defines an octree-based split of the space containing the point cloud. The leaf nodes of the octree contain one or more points of the point cloud.

The G-PCC decoder 300 directly decodes the location of each of the one or more points in the leaf node (1002). To directly code the location of each of the one or more points in the leaf node, the G-PCC decoder 300 generates a prediction of the one or more points ( 1004 ), and based on the prediction determines one or more points (1006). In order to directly decode the position of each of the one or more points in the leaf node, the G-PCC decoder 300 may be configured to receive a flag, wherein the first value of the flag indicates the location of the one or more points The prediction was generated by intra prediction and the second value of the flag indicates that the prediction for one or more points was generated by inter prediction, and based on the value of the flag, intra prediction or inter prediction is used to predict the prediction. one or more points to decode.

The G-PCC decoder 300 may be configured to receive, in a bitstream for a point cloud, an octree leaf volume specifying the volume of a leaf node. For example, suppose the entire point cloud is encapsulated in a WxWxW cube. The point cloud can be split recursively, and for a given split depth d, the octree volume is W/2 ^d x W/2 ^d x W/2 ^d . At this level, an occupancy flag (binary) can be signaled, which when equal to 1 indicates that the cube has at least one point. When the occupancy flag is 1, then an additional intra or inter flag indicating whether a point within the cube is intra- or inter-predicted, respectively, may be signaled.

In order to generate predictions of one or more points, the G-PCC decoder 300 may also be configured to generate predictions of one or more points using intra prediction, and to generate predictions of one or more points using intra prediction , the G-PCC decoder 300 may also be configured to determine a local prediction tree for one or more points.

To determine the one or more points based on the prediction, the G-PCC decoder 300 may be configured to receive, in the bitstream for the point cloud, the prediction mode, the primary residual, for each of the one or more points At least one of difference and sub-residual. To generate predictions of one or more points, G-PCC decoder 300 may be configured to generate predictions of one or more points using inter prediction, and to generate predictions of one or more points using inter prediction, The G-PCC decoder 300 may also be configured to perform motion estimation using one or more points to determine a set of similar points in the reference point cloud frame.

To generate predictions of one or more points, G-PCC decoder 300 may also be configured to generate predictions of one or more points using inter prediction, and to generate predictions of one or more points using inter prediction , the G-PCC decoder 300 may also be configured to perform motion compensation to predict one or more points based on the set of points in the reference point cloud frame. To perform motion compensation, the G-PCC decoder 300 may also be configured to apply a motion vector to a set of points in the reference point cloud frame to determine a prediction of one or more points. The G-PCC decoder 300 may be configured to predict motion vectors based on motion vectors between spatiotemporally adjacent octrees.

The G-PCC decoder 300 may also be configured to reconstruct point clouds from point cloud data. As part of reconstructing the point cloud, the G-PCC decoder 300 may also be configured to determine the location of one or more points of the point cloud based on the planar location.

It is recognized that, by way of example, certain acts or events of any of the techniques described herein may be performed in a different order, may be added, combined, or omitted entirely (eg, not all described acts or events may be used for all described acts or events). practice of the techniques described above is necessary). Furthermore, in some examples, actions or events may be performed concurrently rather than sequentially, eg, through multi-threaded processing, interrupt processing, or multiple processors.

In one or more examples, the functions described can be implemented in hardware, software, firmware, or any combination thereof. If implemented in software, the functions may be stored on or transmitted over as one or more instructions or code on a computer-readable medium and executed by a hardware-based processing unit. Computer-readable media may include computer-readable storage media, which correspond to tangible media, such as data storage media, or communication media, including any medium that facilitates transfer of a computer program from one place to another, for example, in accordance with a communication protocol . In this manner, computer-readable media may generally correspond to (1) non-transitory tangible computer-readable storage media, or (2) communication media such as signals or carrier waves. Data storage media can be any available media that can be accessed by one or more computers or one or more processors for instructions, code and/or data structures for implementing the techniques described in this disclosure. The computer program product may include a computer-readable medium.

By way of example and not limitation, such computer-readable storage media may include RAM, ROM, EEPROM, CD-ROM or other optical disk storage, magnetic disk storage or other magnetic storage devices, flash memory, or capable of The desired program code is stored in the form of instructions or data structures and any other medium that can be accessed by the computer. Also, any connection is properly termed a computer-readable medium. For example, if coaxial cable, fiber optic cable, twisted pair, digital subscriber line (DSL), or wireless technologies such as infrared, radio, and microwave are used to send commands from a website, server, or other remote source, coaxial cable, fiber optic cable, or Fiber optic cable, twisted pair, DSL or wireless technologies such as infrared, radio and microwave are included in the definition of media. It should be understood, however, that computer-readable storage media and data storage media do not include connections, carrier waves, signals, or other transitory media, but are instead directed to non-transitory, tangible storage media. As used herein, magnetic and optical discs include compact discs (CDs), laser discs, optical discs, digital video discs (DVDs), floppy discs, and Blu-ray discs, where magnetic discs generally reproduce material magnetically, while optical discs use laser to optically reproduce material. Combinations of the above should also be included within the scope of computer-readable media.

Instructions may be executed by one or more processors, such as one or more digital signal processors (DSPs), general purpose microprocessors, application specific integrated circuits (ASICs), field programmable gate arrays (FPGAs), or other Equivalent integrated or discrete logic circuit. Accordingly, the terms "processor" and "processing circuit" as used herein may refer to any of the foregoing structures or any other structure suitable for implementing the techniques described herein. Additionally, in some aspects, the functionality described herein may be provided within dedicated hardware and/or software modules configured for encoding and decoding, or incorporated in a combined codec. Furthermore, the techniques may well be implemented in one or more circuits or logic elements.

The techniques of this disclosure may be implemented in a wide variety of devices or apparatus, including a wireless handset, an integrated circuit (IC), or a set of ICs (eg, a chip set). Various components, modules, or units are described in this disclosure to emphasize functional aspects of devices configured to perform the disclosed techniques, but do not necessarily require implementation by distinct hardware units. Rather, individual units may be combined in codec hardware units, as described above, or by a collection of interoperable hardware units (including one or more processors as described above) in conjunction with appropriate software and / or firmware.

The following numbered clauses describe one or more aspects of the devices and techniques described in this disclosure.

Clause 1A: A method of decoding a point cloud, the method comprising: determining an octree-based split octree defining a space containing the point cloud, wherein: leaf nodes of the octree contains one or more points of the point cloud, and the position of each of the one or more points in the leaf node is directly signaled; generated using intra prediction or inter prediction prediction of the one or more points; and coding a syntax element indicating whether the one or more points are predicted using intra prediction or inter prediction.

Clause 2A: The method of Clause 1A, wherein the octree volume specifying the volume of the leaf node is signaled in a bitstream.

Clause 3A: The method of clause 1A or 2A, wherein: generating the prediction of the one or more points comprises generating the prediction of the one or more points using intra prediction, and using intra prediction Predicting to generate the prediction for the one or more points includes determining a local prediction tree for the one or more points.

Clause 4A: The method of Clause 3A, wherein, for each of the one or more points, at least one of a prediction mode, a primary residual, and a secondary residual is signaled.

Clause 5A: The method of Clause 1A or 2A, wherein: generating the prediction of the one or more points comprises generating the prediction of the one or more points using inter prediction, and using inter prediction Predicting to generate the prediction of the one or more points includes performing motion estimation using the one or more points to determine a set of similar points in a reference point cloud frame.

Clause 6A: The method of any of Clauses 1A, 2A, or 5A, wherein: generating the prediction of the one or more points comprises using inter prediction to generate the prediction of the one or more points Predicting, and generating the prediction of the one or more points using inter prediction includes performing motion compensation to predict the one or more points based on a set of points in a reference point cloud frame.

Clause 7A: The method of Clause 6A, wherein performing motion compensation comprises applying a motion vector to the set of points in the reference point cloud frame to determine a prediction of the one or more points.

Clause 8A: The method of Clause 7A, further comprising predicting the motion vector based on a motion vector between spatiotemporally adjacent octrees.

Clause 9A: The method of any of Clauses 1A-8A, further comprising generating the point cloud.

Clause 10A: An apparatus for processing a point cloud, the apparatus comprising one or more means for performing the method of any of clauses 1A-9A.

Clause 11A: The apparatus of Clause 10A, wherein the one or more components comprise one or more processors implemented in a circuit.

Clause 12A: The apparatus of any of Clause 10A or Clause 11A, further comprising: memory storing the data representing the point cloud.

Clause 13A: The apparatus of any of clauses 10A-12A, wherein the apparatus comprises a decoder.

Clause 14A: The apparatus of any of clauses 10A-13A, wherein the apparatus comprises an encoder.

Clause 15A: The apparatus of any of clauses 10A-14A, further comprising: apparatus for generating the point cloud.

Clause 16A: The apparatus of any of clauses 10A-15A, further comprising: a display for rendering imagery based on the point cloud.

Clause 17A: A computer-readable storage medium having instructions stored thereon that, when executed, cause one or more processors to perform the method of any of clauses 1A-9A.

Clause 1B: An apparatus for decoding a bitstream including point cloud data, the apparatus comprising: a memory for storing the point cloud data; and a memory coupled to the memory and implemented in a circuit one or more processors configured to: determine an octree that defines an octree-based split of a space containing the point cloud, wherein the leaves of the octree node contains one or more points of the point cloud; and directly decoding the position of each of the one or more points in the leaf node, wherein, in order to directly decode the point in the leaf node decoding the location of each of the one or more points, the one or more processors further configured to: generate a prediction of the one or more points; and determine based on the prediction the one or more points.

Clause 2B: The apparatus of Clause 1B, wherein, to directly decode the location of each of the one or more points in the leaf node, the one or more processors further is configured to: receive a flag, wherein a first value of the flag indicates that the prediction of the one or more points was generated by intra prediction, and a second value of the flag indicates the one or The prediction for a plurality of points is generated by inter prediction; and the one or more points are decoded using intra prediction or inter prediction based on the value of the flag.

Clause 3B: The apparatus of Clause 1B, wherein the one or more processors are further configured to receive, in the bitstream including the point cloud, an octet specifying a volume of the leaf node Leaf volume.

Clause 4B: The apparatus of Clause 1B, wherein: to generate the prediction for the one or more points, the one or more processors are further configured to generate the one or more using intra prediction the prediction of the one or more points, and in order to generate the prediction of the one or more points using intra prediction, the one or more processors are further configured to determine the prediction for the one or more points Local prediction tree.

Clause 5B: The apparatus of Clause 1B, wherein, to determine the one or more points based on the prediction, the one or more processors are further configured to: At least one of a prediction mode, a primary residual, and a secondary residual for each of the one or more points is received in the bitstream.

Clause 6B: The apparatus of Clause 1B, wherein: to generate the prediction for the one or more points, the one or more processors are further configured to generate the one or more using inter prediction the prediction of a number of points, and in order to generate the prediction of the one or more points using inter prediction, the one or more processors are further configured to utilize the one or more points to perform motion estimate to determine the set of similar points in the reference point cloud frame.

Clause 7B: The apparatus of Clause 1B, wherein: to generate the prediction for the one or more points, the one or more processors are further configured to generate the one or more using inter prediction the prediction of the points, and to generate the prediction of the one or more points using inter prediction, the one or more processors are further configured to generate the prediction based on the set of points in the reference point cloud frame Motion compensation is performed to predict the one or more points.

Clause 8B: The apparatus of Clause 7B, wherein, to perform motion compensation, the one or more processors are further configured to apply a motion vector to the set of points in the reference point cloud frame, to determine a prediction for the one or more points.

Clause 9B: The apparatus of Clause 8B, wherein the one or more processors are further configured to predict the motion vector based on a motion vector between spatiotemporally adjacent octrees.

Clause 10B: The apparatus of Clause 1B, wherein the one or more processors are further configured to reconstruct a point cloud from the point cloud data.

Clause 11B: The apparatus of clause 10B, wherein the one or more processors are configured to, as part of reconstructing the point cloud, determine one or more of the point cloud based on the planar position point location.

Clause 11B: The apparatus of clause 10B, wherein the one or more processors are configured to: as part of reconstructing the point cloud, based on each of the one or more points in the leaf node The directly decoded positions of the points determine the position of the one or more points of the point cloud.

Clause 12B: The apparatus of Clause 11B, wherein the one or more processors are further configured to generate a map of the interior of a building based on the point cloud.

Clause 13B: The apparatus of Clause 11B, wherein the one or more processors are further configured to perform autonomous navigation operations based on the point cloud.

Clause 14B: The apparatus of Clause 11B, wherein the one or more processors are further configured to generate computer graphics based on the point cloud.

Clause 15B: The apparatus of Clause 11B, wherein the one or more processors are configured to: determine a location of a virtual object based on the point cloud; and generate an extended reality (XR) visualization, wherein the The virtual object is located at the determined location.

Clause 16B: The apparatus of Clause 11B, further comprising: a display for rendering imagery based on the point cloud.

Clause 17B: The device of Clause 1B, wherein the device is a mobile phone or a tablet.

Clause 18B: The apparatus of Clause 1B, wherein the apparatus is a vehicle.

Clause 19B: The device of Clause 1B, wherein the device is an extended reality device.

Clause 20B: A method of decoding point cloud data, the method comprising: determining an octree-based split octree defining a space containing the point cloud, wherein leaf nodes of the octree including one or more points of the point cloud; directly decoding the position of each of the one or more points in the leaf node, wherein all the points in the leaf node are directly decoded Decoding the location of each of the one or more points includes generating a prediction of the one or more points; and determining the one or more points based on the prediction.

Clause 21B: The method of Clause 20B, wherein directly decoding the location of each of the one or more points in the leaf node further comprises receiving a flag, wherein the a first value of the flag indicates that the prediction for the one or more points was generated by intra prediction, and a second value of the flag indicates that the prediction for the one or more points was generated by inter and the one or more points are decoded using intra prediction or inter prediction based on the value of the flag.

Clause 22B: The method of Clause 20B, further comprising receiving, in a bitstream for the point cloud, an octree leaf volume specifying a volume of the leaf node.

Clause 23B: The method of Clause 20B, wherein: generating the prediction for the one or more points comprises generating the prediction for the one or more points using intra prediction, and using intra prediction to Generating the prediction for the one or more points includes determining a local prediction tree for the one or more points.

Clause 24B: The method of Clause 20B, wherein determining the one or more points based on the prediction comprises receiving in a bitstream for the one or more points in a bitstream for the point cloud At least one of the prediction mode, primary residual, and secondary residual for each point in .

Clause 25B: The method of Clause 20B, wherein: generating the prediction for the one or more points comprises generating the prediction for the one or more points using inter prediction, and using inter prediction to Generating the prediction of the one or more points includes performing motion estimation using the one or more points to determine a set of similar points in a reference point cloud frame.

Clause 26B: The method of Clause 20B, wherein: generating the prediction for the one or more points comprises generating the prediction for the one or more points using inter prediction, and using inter prediction to Generating the prediction of the one or more points includes performing motion compensation to predict the one or more points based on a set of points in a reference point cloud frame.

Clause 27B: The method of Clause 26B, wherein performing motion compensation comprises applying a motion vector to the set of points in the reference point cloud frame to determine a prediction of the one or more points.

Clause 28B: The method of Clause 27B, further comprising predicting the motion vector based on a motion vector between spatiotemporally adjacent octrees.

Clause 29B: A computer-readable storage medium storing instructions that, when executed by one or more processors, cause the one or more processors to: determine an octet-based method that defines a space containing a point cloud A split octree of a tree, wherein a leaf node of the octree contains one or more points of the point cloud; and a direct response to the one or more points in the leaf node decode the position of each point, wherein, in order to directly decode the position of each of the one or more points in the leaf node, the instructions cause the one or more processing The processor operates to: generate a prediction of the one or more points; and determine the one or more points based on the prediction.

Clause 30B: An apparatus comprising: means for determining an octree-based splitting of an octree that defines a space containing a point cloud, wherein a leaf node of the octree contains one or more of the point cloud. a plurality of points; means for directly decoding the position of each of the one or more points in the leaf node, wherein the means for directly decoding all the points in the leaf node the means for decoding the location of each of the one or more points comprises: means for generating a prediction of the one or more points; and determining the one or more points based on the prediction a point component.

Clause 1C: An apparatus for decoding a bitstream comprising point cloud data, the apparatus comprising: a memory for storing the point cloud data; and a memory coupled to the memory and implemented in a circuit one or more processors configured to: determine an octree that defines an octree-based split of a space containing the point cloud, wherein the leaves of the octree a node containing one or more points of the point cloud; and directly decoding the position of each of the one or more points in the leaf node, wherein in order to directly decode the point in the leaf node decoding the position of each of the one or more points in the one or more processors, the one or more processors are further configured to: generate a prediction of the one or more points; and based on the prediction to determine the one or more points.

Clause 2C: The apparatus of Clause 1C, wherein, to directly decode the location of each of the one or more points in the leaf node, the one or more processors is further configured to: receive a flag, wherein a first value of the flag indicates that the prediction of the one or more points was generated by intra prediction, and a second value of the flag indicates the one The prediction of the point or points is generated by inter prediction; and the one or more points are decoded using intra prediction or inter prediction based on the value of the flag.

Clause 3C: The apparatus of Clause 1C or 2C, wherein the one or more processors are further configured to receive, in the bitstream including the point cloud, a volume specifying the leaf node Octopus leaf volume.

Clause 4C: The apparatus of any of clauses 1C-3C, wherein: to generate the prediction for the one or more points, the one or more processors are further configured to use intra prediction to generating the prediction for the one or more points, and in order to generate the prediction for the one or more points using intra prediction, the one or more processors are further configured to determine for the A local prediction tree for one or more points.

Clause 5C: The apparatus of any of Clauses 1C-4C, wherein, to determine the one or more points based on the prediction, the one or more processors are further configured to: At least one of a prediction mode, a primary residual, and a secondary residual for each of the one or more points is received in the bitstream of the point cloud.

Clause 6C: The apparatus of any of clauses 1C-5C, wherein: to generate the prediction of the one or more points, the one or more processors are further configured to use inter prediction to generating the prediction of the one or more points, and in order to generate the prediction of the one or more points using inter prediction, the one or more processors are further configured to utilize the one or more Motion estimation is performed at multiple points to determine a set of similar points in the reference point cloud frame.

Clause 7C: The apparatus of any of clauses 1C-5C, wherein: to generate the prediction of the one or more points, the one or more processors are further configured to use inter prediction to generating the prediction of the one or more points, and in order to generate the prediction of the one or more points using inter prediction, the one or more processors are further configured to generate the prediction of the one or more points based on the reference point cloud A set of points in the frame to perform motion compensation to predict the one or more points.

Clause 8C: The apparatus of Clause 7C, wherein, to perform motion compensation, the one or more processors are further configured to apply a motion vector to the set of points in the reference point cloud frame, to determine a prediction for the one or more points.

Clause 9C: The apparatus of Clause 8C, wherein the one or more processors are further configured to predict the motion vector based on a motion vector between spatiotemporally adjacent octrees.

Clause 10C: The apparatus of any of clauses 1C-9C, wherein the one or more processors are further configured to reconstruct a point cloud from the point cloud data.

Clause 11C: The apparatus of clause 10C, wherein the one or more processors are configured to: as part of reconstructing the point cloud, based on each of the one or more points in the leaf node The directly decoded locations of the points determine the location of the one or more points of the point cloud.

Clause 12C: The apparatus of Clause 11C, wherein the one or more processors are further configured to generate a map of the interior of a building based on the point cloud.

Clause 13C: The apparatus of Clause 11C, wherein the one or more processors are further configured to perform autonomous navigation operations based on the point cloud.

Clause 14C: The apparatus of Clause 11C, wherein the one or more processors are further configured to generate computer graphics based on the point cloud.

Clause 15C: The apparatus of Clause 11C, wherein the one or more processors are configured to: determine a location of a virtual object based on the point cloud; and generate an extended reality (XR) visualization, wherein the The virtual object is located at the determined location.

Clause 16C: The apparatus of any of clauses 11C-15C, further comprising: a display for rendering imagery based on the point cloud.

Clause 17C: The device of any of Clauses 1C-16C, wherein the device is a mobile phone or a tablet.

Clause 18C: The apparatus of any of Clauses 1C-16C, wherein the apparatus is a vehicle.

Clause 19C: The device of any of Clauses 1C-16C, wherein the device is an extended reality device.

Clause 20C: A method of decoding point cloud data, the method comprising: determining an octree-based split octree defining a space containing the point cloud, wherein leaf nodes of the octree are including one or more points of the point cloud; directly decoding the position of each of the one or more points in the leaf node, wherein all the points in the leaf node are directly decoded Decoding the location of each of the one or more points includes generating a prediction of the one or more points; and determining the one or more points based on the prediction.

Clause 21C: The method of Clause 20C, wherein directly decoding the location of each of the one or more points in the leaf node further comprises receiving a flag, wherein the flag The first value of the flag indicates that the prediction of the one or more points was generated by intra prediction, and the second value of the flag indicates that the prediction of the one or more points was generated by inter prediction and the one or more points are decoded using intra prediction or inter prediction based on the value of the flag.

Clause 22C: The method of clause 20C or 21C, further comprising receiving, in a bitstream for the point cloud, an octree volume specifying a volume of the leaf node.

Clause 23C: The method of any of Clauses 20C-22C, wherein: generating the prediction for the one or more points comprises generating the prediction for the one or more points using intra prediction, And using intra prediction to generate the prediction for the one or more points includes determining a local prediction tree for the one or more points.

Clause 24C: The method of any one of Clauses 20C-23C, wherein determining the one or more points based on the prediction comprises receiving in a bitstream for the point cloud for all the points at least one of a prediction mode, a primary residual, and a secondary residual for each of the one or more points.

Clause 25C: The method of any of Clauses 20C-24C, wherein: generating the prediction for the one or more points comprises generating the prediction for the one or more points using inter prediction, And using inter prediction to generate the prediction of the one or more points includes performing motion estimation using the one or more points to determine a set of similar points in a reference point cloud frame.

Clause 26C: The method of any of Clauses 20C-25C, wherein: generating the prediction for the one or more points comprises generating the prediction for the one or more points using inter prediction, And using inter prediction to generate the prediction of the one or more points includes performing motion compensation to predict the one or more points based on a set of points in a reference point cloud frame.

Clause 27C: The method of Clause 26C, wherein performing motion compensation comprises applying a motion vector to the set of points in the reference point cloud frame to determine a prediction of the one or more points.

Clause 28C: The method of Clause 27C, further comprising predicting the motion vector based on a motion vector between spatiotemporally adjacent octrees.

Various examples have been described. These and other examples are within the scope of the following claims.

100: Encoding and Decoding Systems 102: Source Device 104: Sources 106: Memory 108: Output interface 110: Computer-readable media 112: Storage Devices 114: file server 116: target device 118: Data consumer 120: memory 122: Input interface 200: G-PCC encoder 300: G-PCC Decoder 200: G-PCC Encoder 200 202: Coordinate transformation unit 204: Color transformation unit 206: Voxelization unit 208: Attribute Transform Unit 210: Octree Analysis Unit 212: Surface Approximation Analysis Unit 214: Arithmetic coding unit 216: Geometry Reconstruction Unit 218: RAHT unit 220:LOD generation unit LOD 222: Lifting Unit 224: Coefficient quantization unit 226: Arithmetic coding unit 300: G-PCC Decoder 302: Geometry Arithmetic Decoding Unit 304: attribute arithmetic decoding unit 306: Octree Synthesis Unit 308: Inverse Quantization Unit 310: Surface Approximation Synthesis Unit 312: Geometry Reconstruction Unit 314: RAHT unit 316: LOD generation unit 318: Inverse Lifting Unit 320: Inverse coordinate transformation unit 322: Inverse color transform unit 400: Octree 402: Node 404: Node 412: Node 414A, 414B: Nodes 418A, 418B, 418C, 418D, 418E: Nodes 420: Predicted frame 422: previous frame 424: Global Motion Estimation 426: Apply the estimated global motion to the predicted frame 428: Local node motion estimation 430: Motion compensation, encoding motion vectors and points 432: Find feature points 434: Sampling feature point pairs 436: Execute LMS 440: if not split into 8 child nodes 442: If split into 8 child nodes, similar to octree division 444: Octree O0, O1, O2, O3, O4, O5, O6, O7, O8, O9, O10, O11, O12: point or current point set R0, R1, R2, R3, R4, R5, R6, R7, R8, R9, R10, R11, R12: reference point set P0, P1, P2, P3, P4, P5, P6, P7, P8, P9, P10, P11, P12: motion compensated reference point set 600: Ranging System 602: Illuminator 604: Sensor 606: Emit Light 608: Object 610: Return Light 611: Lens 612: Image 618: Signal 620: Point Cloud Generator 700: Vehicle 702: Laser packaging 704: Laser Beam 706: Pedestrian 708: bitstream 710: Vehicles 712: Server System 800: User 802: First position 804:XR Headphones 806: Object 808: bitstream 810: XR Headphones 812: User 814: Second position 900: Mobile Devices 902: Object 904: bitstream 906: Remote Device 1000: Determine the octree that defines the octree-based split of the space containing the point cloud 1002: Steps 1004: Steps 1006: Steps

1 is a block diagram illustrating an example encoding and decoding system that may implement the techniques of this disclosure.

2 is a block diagram illustrating an example Geometric Point Cloud Compression (G-PCC) encoder.

3 is a block diagram illustrating an example G-PCC decoder.

4 is a conceptual diagram illustrating an example octree split for geometry coding.

FIG. 5 is a conceptual diagram illustrating an example of a prediction tree.

6 is a conceptual diagram illustrating an example rotating lidar acquisition model.

7 is a conceptual diagram illustrating an example motion estimation flow diagram for InterEM.

8 is a conceptual diagram illustrating an example algorithm for estimation of global motion.

9 is a conceptual diagram illustrating an example algorithm for estimation of local node motion vectors.

FIG. 10 is a conceptual diagram illustrating an example of advanced octree splitting.

FIG. 11 is a conceptual diagram illustrating an example of local prediction tree generation.

FIG. 12 is a conceptual diagram illustrating example current point sets (O0 to O12) and reference point sets (R0 to R12) (where N=M=13).

13 is a conceptual diagram illustrating an example current point set and motion compensated reference point set (where N=M=13).

14 is a conceptual diagram illustrating an example ranging system that may be used with one or more techniques of the present disclosure.

15 is a conceptual diagram illustrating an example vehicle-based scenario in which one or more techniques of the present disclosure may be employed.

16 is a conceptual diagram illustrating an example extended reality system in which one or more techniques of the present disclosure may be employed.

17 is a conceptual diagram illustrating an example mobile device system in which one or more techniques of the present disclosure may be employed.

18 is a flowchart illustrating example operations for decoding a bitstream including point cloud material.

1000: Determine the octree that defines the octree-based split of the space containing the point cloud

1002: Steps

1004: Steps

1006: Steps

Claims (30)

An apparatus for decoding a bitstream comprising point cloud data, the apparatus comprising: memory for storing the point cloud data; and one or more processors coupled to the memory and implemented in circuitry, the one or more processors configured to: determining an octree-based split octree that defines a space containing a point cloud, wherein leaf nodes of the octree contain one or more points of the point cloud; and directly decoding the position of each of the one or more points in the leaf node, wherein in order to directly decode each of the one or more points in the leaf node to decode the location, the one or more processors are further configured to: generating a prediction for the one or more points; and The one or more points are determined based on the prediction. The apparatus of claim 1, wherein, in order to directly decode the location of each of the one or more points in the leaf node, the one or more processors are further Configured as: receiving a flag, wherein a first value of the flag indicates that the prediction of the one or more points was generated by intra prediction, and a second value of the flag indicates that the prediction of the one or more points was the prediction is generated by inter prediction; and Based on the value of the flag, the one or more points are decoded using intra prediction or inter prediction. The apparatus of claim 1, wherein the one or more processors are further configured to receive, in the bitstream including the point cloud, an octree leaf volume specifying the volume of the leaf node . The apparatus of claim 1, wherein: To generate the prediction for the one or more points, the one or more processors are further configured to generate the prediction for the one or more points using intra prediction, and To generate the prediction for the one or more points using intra prediction, the one or more processors are further configured to determine a local prediction tree for the one or more points. The apparatus of claim 1, wherein, to determine the one or more points based on the prediction, the one or more processors are further configured to: At least one of a prediction mode, a primary residual, and a secondary residual for each of the one or more points is received in the stream. The apparatus of claim 1, wherein: To generate the prediction for the one or more points, the one or more processors are further configured to generate the prediction for the one or more points using inter prediction, and In order to generate the prediction of the one or more points using inter prediction, the one or more processors are further configured to perform motion estimation using the one or more points to determine where in the reference point cloud A collection of similar points in the frame. The apparatus of claim 1, wherein: To generate the prediction for the one or more points, the one or more processors are further configured to generate the prediction for the one or more points using inter prediction, and In order to generate the prediction of the one or more points using inter prediction, the one or more processors are further configured to perform motion compensation to predict the one based on a set of points in a reference point cloud frame or multiple points. The apparatus of claim 7, wherein, to perform motion compensation, the one or more processors are further configured to apply a motion vector to the set of points in the reference point cloud frame to determine Prediction of the one or more points. The apparatus of claim 8, wherein the one or more processors are further configured to predict the motion vector based on a motion vector between spatiotemporally adjacent octrees. The apparatus of claim 1, wherein the one or more processors are further configured to reconstruct the point cloud from the point cloud data. The apparatus of claim 10, wherein the one or more processors are configured to: as part of reconstructing the point cloud, based on each of the one or more points in the leaf node to determine the location of one or more points of the point cloud. The apparatus of claim 11, wherein the one or more processors are further configured to generate a map of the interior of a building based on the point cloud. The apparatus of claim 11, wherein the one or more processors are further configured to perform autonomous navigation operations based on the point cloud. The apparatus of claim 11, wherein the one or more processors are further configured to generate computer graphics based on the point cloud. The apparatus of claim 11, wherein the one or more processors are configured to: determining the location of the virtual object based on the point cloud; and An extended reality (XR) visualization is generated, wherein the virtual object is located at the determined location of the virtual object. The apparatus of claim 11, further comprising a display for rendering imagery based on the point cloud. The device of claim 1, wherein the device is a mobile phone or a tablet. The apparatus of claim 1, wherein the apparatus is a vehicle. The device of claim 1, wherein the device is an extended reality device. A method for decoding point cloud data, the method comprising: determining an octree-based split octree that defines a space containing the point cloud, wherein leaf nodes of the octree contain one or more points of the point cloud; directly decoding the position of each of the one or more points in the leaf node, wherein the location of each of the one or more points in the leaf node is directly decoded Decoding the location includes: generating a prediction for the one or more points; and The one or more points are determined based on the prediction. The method of claim 20, wherein directly decoding the location of each of the one or more points in the leaf node further comprises: receiving a flag, wherein a first value of the flag indicates that the prediction of the one or more points was generated by intra prediction, and a second value of the flag indicates that the prediction of the one or more points was the prediction is generated by inter prediction; and Based on the value of the flag, the one or more points are decoded using intra prediction or inter prediction. The method of claim 20, further comprising receiving, in a bitstream for the point cloud, an octree leaf volume specifying the volume of the leaf node. The method of claim 20, wherein: Generating the prediction for the one or more points includes generating the prediction for the one or more points using intra prediction, and Generating the prediction for the one or more points using intra prediction includes determining a local prediction tree for the one or more points. The method of claim 20, wherein determining the one or more points based on the prediction comprises receiving in a bitstream for the one or more points in a bitstream for the one or more points At least one of prediction mode, primary residual, and secondary residual for each point. The method of claim 20, wherein: Generating the prediction for the one or more points includes generating the prediction for the one or more points using inter prediction, and Using inter prediction to generate the prediction of the one or more points includes performing motion estimation using the one or more points to determine a set of similar points in a reference point cloud frame. The method of claim 20, wherein: Generating the prediction for the one or more points includes generating the prediction for the one or more points using inter prediction, and Using inter prediction to generate the prediction of the one or more points includes performing motion compensation to predict the one or more points based on a set of points in a reference point cloud frame. The method of claim 26, wherein performing motion compensation comprises applying a motion vector to the set of points in the reference point cloud frame to determine a prediction of the one or more points. The method of claim 27, further comprising predicting the motion vector based on a motion vector between spatiotemporally adjacent octrees. A computer-readable storage medium storing instructions that, when executed by one or more processors, cause the one or more processors to: determining an octree-based split octree that defines a space containing a point cloud, wherein leaf nodes of the octree contain one or more points of the point cloud; and directly decoding the position of each of the one or more points in the leaf node, wherein in order to directly decode each of the one or more points in the leaf node , the instruction causes the one or more processors to: generating a prediction for the one or more points; and The one or more points are determined based on the prediction. A device comprising: means for determining an octree-based split octree that defines a space containing a point cloud, wherein leaf nodes of the octree contain one or more points of the point cloud; means for directly decoding the position of each of the one or more points in the leaf node, wherein the means for directly decoding the one or more points in the leaf node The means for decoding the position of each of the points include: means for generating a prediction of the one or more points; and means for determining the one or more points based on the prediction.

Images