JP2024501966A - Hybrid tree coding for inter and intra prediction for geometry coding - Google Patents

Abstract

A device for decoding a bitstream containing point cloud data is to determine an octree that defines an octree-based partitioning of the space containing the point cloud, wherein the leaf nodes of the octree are: the one or more points at the leaf node, the one or more points at the leaf node being configured to include one or more points of the point cloud, and directly decoding the position of each of the one or more points at the leaf node; The one or more processors generate predictions of the one or more points and determine the one or more points based on the predictions to directly decode the positions of each of the points. It is further configured as follows.

Description

This application claims priority to U.S. Patent Application No. 17/562,398, filed on December 27, 2021, and U.S. Provisional Patent Application No. 63/131,546, filed on December 29, 2020. The entire contents are incorporated herein by reference. U.S. Patent Application No. 17/562,398, filed on December 27, 2021, claims the benefit of U.S. Provisional Patent Application No. 63/131,546, filed on December 29, 2020.

TECHNICAL FIELD This disclosure relates to point cloud encoding and decoding.

“Text of ISO/IEC FDIS 23090-9 Geometry-based Point Cloud Compression”, ISO/IEC JTC 1/SC29/WG 7 MDS19617, Teleconference, October 2020 G-PCC Codec Description, ISO/IEC JTC 1/SC29/WG 7 MDS19620, Teleconference, October 2020

In general, this disclosure describes a hybrid tree coding method that combines octree coding and predictive coding for enhanced inter/intra prediction at the block level for point cloud compression.

In one example, this disclosure describes a method of coding a point cloud, the method comprising: determining an octree that defines an octree-based partition of a space that includes the point cloud; A leaf node of contains one or more points of the point cloud, and the position of each of the one or more points in the leaf node is directly signaled, using intra-prediction or inter-prediction. or generating a prediction for multiple points and coding a syntax element indicating whether the one or more points are predicted using intra prediction or inter prediction. including.

According to one example of the present disclosure, a device for decoding a bitstream including point cloud data includes a memory for storing point cloud data, and one or more processors coupled to the memory and implemented in circuitry. and the one or more processors determining an octree defining an octree-based partition of a space containing the point cloud, the leaf nodes of the octree defining an octree-based partition of a space containing the point cloud, one or more points, and configured to directly decode the position of each of the one or more points in a leaf node, In order to directly decode the position, the one or more processors further include generating a prediction of the one or more points and determining the one or more points based on the prediction. configured.

According to another example of the present disclosure, a method for decoding a point cloud includes the steps of: determining an octree that defines an octree-based partition of a space containing the point cloud; the node includes one or more points of the point cloud; directly decoding the position of each of the one or more points at the leaf node; Directly decoding each location includes generating a prediction of the one or more points and determining the one or more points based on the prediction.

According to another example of the present disclosure, a computer-readable storage medium stores instructions that, when executed by the one or more processors, cause the one or more processors to perform a defining an octree-based partition, determining an octree, a leaf node of the octree containing one or more points of the point cloud, and one in the leaf node; or causes the one or more processors to directly decode the position of each of the one or more points in the leaf node, or generating a prediction of a plurality of points and determining one or more points based on the prediction.

According to another example of the present disclosure, the apparatus is a means for determining an octree that defines an octree-based partition of a space that includes a point cloud, wherein leaf nodes of the octree include: each of the one or more points at the leaf node, including one or more points of the point cloud, and means for directly decoding the position of each of the one or more points at the leaf node. Means for directly decoding the position of comprises means for generating a prediction of one or more points and means for determining one or more points based on the prediction.

The details of one or more examples are set forth in the accompanying drawings and the description below. Other features, objects, and advantages will be apparent from the description, drawings, and claims.

FIG. 1 is a block diagram illustrating an example encoding and decoding system that may implement the techniques of this disclosure. FIG. 1 is a block diagram illustrating an example geometry point cloud compression (G-PCC) encoder. FIG. 2 is a block diagram illustrating an example G-PCC decoder. FIG. 2 is a conceptual diagram illustrating an example octree decomposition for geometry coding. FIG. 2 is a conceptual diagram showing an example of a prediction tree. FIG. 2 is a conceptual diagram illustrating an example rotating lidar acquisition model. FIG. 2 is a conceptual diagram illustrating an example motion estimation flowchart for InterEM. FIG. 2 is a conceptual diagram illustrating an example algorithm for global motion estimation. FIG. 2 is a conceptual diagram illustrating an example algorithm for estimating local node motion vectors. FIG. 2 is a conceptual diagram illustrating an example of high-level 8-ary tree partitioning. FIG. 2 is a conceptual diagram showing an example of local prediction tree generation. FIG. 2 is a conceptual diagram illustrating an exemplary current set of points (O0-O12) and reference set of points (R0-R12), where N=M=13. FIG. 3 is a conceptual diagram illustrating an exemplary current set of points and a motion compensated reference set of points, where N=M=13. FIG. 1 is a conceptual diagram illustrating an example distance measurement system that may be used with one or more techniques of this disclosure. 1 is a conceptual diagram illustrating an example vehicle-based scenario in which one or more techniques of this disclosure may be used. FIG. 1 is a conceptual diagram illustrating an example extended reality system in which one or more techniques of this disclosure may be used. FIG. 1 is a conceptual diagram illustrating an example mobile device system in which one or more techniques of this disclosure may be used. 2 is a flowchart illustrating example operations for decoding a bitstream that includes point cloud data.

A point cloud is a collection of points in three-dimensional (3D) space. A point may correspond to a point on an object in three-dimensional space. Therefore, point clouds can be used to represent the physical content of three-dimensional space. Point clouds may have utility in a wide variety of situations. For example, point clouds may be used in the context of autonomous vehicles to represent the location of objects on a road. In another example, point clouds may be used in the context of representing the physical contents of an environment for the purpose of positioning virtual objects in augmented reality (AR) or mixed reality (MR) applications. Point cloud compression is a process for encoding and decoding point clouds. Encoding the point cloud may reduce the amount of data required for storage and transmission of the point cloud.

Two major proposals have previously existed for signaling the location of points in a point cloud: octree coding and prediction tree coding. As part of encoding point cloud data using octree coding, the G-PCC encoder may generate an octree. Each node of the octree corresponds to a rectangular parallelepiped space. A node in an octree may have 0 child nodes or 8 child nodes. In other examples, nodes may be split into child nodes according to other tree structures. A child node of a parent corresponds to a cuboid of equal size within the cuboid corresponding to the parent node. The positions of individual points of the point cloud may be signaled with respect to the origin of the node. A node is said to be unoccupied if it does not contain any points in the point cloud. If a node is not occupied, there may be no need to signal additional data about the node. Conversely, a node is said to be occupied if it contains one or more points of the point cloud.

When encoding point cloud data using predictive tree coding, the G-PCC encoder determines a prediction mode for each point in the point cloud. The prediction mode for a point can be one of the following:

・No prediction/zero prediction (0)
・Delta prediction (p0)
・Line prediction (2*p0-p1)
・Parallelogram prediction (2*p0+p1-p2)
If the prediction mode for a point is "No Prediction/Zero Prediction", the point is considered a root point (i.e., root vertex) and the point's coordinates (e.g., x, y, z coordinates) are stored in the bitstream. signaled in If the prediction mode for a point is "delta prediction," the G-PCC encoder determines the difference (i.e., delta) between the coordinates of the point and the coordinates of a parent point, such as the root point or other points. do. If the prediction mode is "linear prediction", the G-PCC encoder uses a linear prediction of the coordinates of the two parent points to determine the predicted coordinates of the point. The G-PCC encoder signals the difference between the predicted coordinates and the actual coordinates of a point, determined using linear prediction. If the prediction mode is "parallelogram prediction", the G-PCC encoder uses three parent points to determine the predicted coordinates. The G-PCC encoder then signals the difference (eg, the "first-order residual") between the predicted coordinates and the actual coordinates of the point. The predictive relationships between points essentially define a tree of points.

It has been experimentally observed that octree coding may be better suited for dense point clouds than predictive tree coding. Point clouds acquired using 3D modeling are often dense enough for octree coding to work better. However, for example, for automotive applications, point clouds acquired using LiDAR tend to be rather coarse, so predictive coding may work better in these applications.

In some examples, angular modes may be used to represent the coordinates of a point in a spherical coordinate system. Because the conversion process between spherical and Cartesian (eg, x, y, z) coordinate systems is not perfect, information may be lost. However, since the G-PCC encoder can perform a conversion process, the G-PCC encoder can convert the Cartesian coordinates of a point resulting from applying the conversion process to the spherical coordinates of the point to the original Cartesian coordinates of the point. A "quadratic residual" can be signaled for the points, indicating the difference between.

This disclosure relates to a hybrid coding model where both octree coding and direct coding are used to code point clouds. For example, octree coding may be first used to partition the space into nodes to a certain level. Nodes at a particular level (and other occupied nodes of the octree that are not further split) are sometimes called "leaf nodes." Points within the volume of leaf nodes may be coded using the "direct" coding mode.

When encoding the points of a leaf node in the "direct" coding mode, the G-PCC encoder may select an intra-prediction mode for the leaf node or an inter-prediction mode for the leaf node. The G-PCC encoder may signal whether a leaf node point is encoded using an intra-prediction mode or an inter-prediction mode.

If the G-PCC encoder selects intra-prediction mode for a leaf node, the G-PCC encoder encodes the points at the leaf nodes using predictive tree coding in much the same manner as described above. can be converted into That is, the G-PCC encoder may select among four prediction modes and signal the coordinates for the points accordingly. However, rather than signaling the coordinates relative to the origin of the entire space associated with the octree, the G-PCC encoder may signal the coordinates relative to the origin of the leaf nodes. This may improve coding efficiency, especially for the root node.

If the G-PCC encoder selects inter-prediction mode for a leaf node, the G-PCC encoder may encode the points at the leaf node with respect to the set of points in the reference frame. A reference frame may be a previously coded frame that is similar to a previous frame of the video. The G-PCC encoder may perform motion estimation to identify a set of points in the reference frame that have a similar spatial arrangement as the points in the leaf nodes. A motion vector for a leaf node indicates the displacement between a point of the leaf node and an identified set of points in the reference frame.

A G-PCC encoder may signal a set of parameters for leaf nodes. Parameters for leaf nodes may include a reference index that identifies a reference frame. Parameters for leaf nodes may also include a value indicating the number of points in the leaf node.

The leaf node parameters may also include residual values for each of the points in the leaf node. The residual value for a point at a leaf node is the difference between the leaf node's predicted coordinates (determined by adding the leaf node's motion vector to the point in the reference frame that corresponds to the point at the leaf node). Show the difference. In examples where angular mode is used, the G-PCC encoder may also signal the quadratic residual for the points.

In some examples, the parameters for leaf nodes also include motion vector differences (MVD). MVD indicates the difference between the leaf node's motion vector and the predicted motion vector. The predicted motion vector is the motion vector of the neighboring nodes of the octree. Parameters for leaf nodes may include indices that identify neighboring nodes.

In another example, similar to merge mode in traditional video coding, the parameters for leaf nodes do not include MVD and the motion vector of the leaf node is the same as the motion vector of the identified neighbor node. It can be assumed.

In some examples, residual signaling may be skipped. In some such instances where angular mode is used, the signaling of the first-order residual may be skipped while still signaling the second-order residual.

FIG. 1 is a block diagram illustrating an example encoding and decoding system 100 that may implement the techniques of this disclosure. The techniques of this disclosure are generally directed to coding (encoding and/or decoding) point cloud data, i.e., supporting point cloud compression. Generally, point cloud data includes any data for processing point clouds. Coding may be effective in compressing and/or decompressing point cloud data.

As shown in FIG. 1, system 100 includes a source device 102 and a destination device 116. Source device 102 provides encoded point cloud data to be decoded by destination device 116. Specifically, in the example of FIG. 1, source device 102 provides point cloud data to destination device 116 via computer-readable storage medium 110. Source device 102 and destination device 116 may include desktop computers, notebook (i.e., laptop) computers, tablet computers, set-top boxes, telephone handsets such as smartphones, televisions, cameras, display devices, digital media players, video gaming consoles, etc. It may comprise any of a wide range of devices, including video streaming devices, ground or sea vehicles, spacecraft, aircraft, robots, LIDAR devices, satellites, etc. In some cases, source device 102 and destination device 116 may be equipped for wireless communication.

In the example of FIG. 1, source device 102 includes data source 104, memory 106, G-PCC encoder 200, and output interface 108. Destination device 116 includes input interface 122, G-PCC decoder 300, memory 120, and data consumer 118. According to this disclosure, G-PCC encoder 200 of source device 102 and G-PCC decoder 300 of destination device 116 perform 8-tree coding and The techniques of this disclosure may be configured to apply techniques related to hybrid tree coding methods that combine predictive coding. Thus, source device 102 represents an example of an encoding device and destination device 116 represents an example of a decoding device. In other examples, source device 102 and destination device 116 may include other components or arrangements. For example, source device 102 may receive data (eg, point cloud data) from an internal or external source. Similarly, destination device 116 may interface with external data consumers rather than including the data consumers within the same device.

A system 100 as shown in FIG. 1 is only one example. In general, other digital encoding and/or decoding devices are related to hybrid tree coding methods that combine octree coding and predictive coding for enhanced inter/intra prediction at block level for point cloud compression. The disclosed techniques may be implemented. Source device 102 and destination device 116 are only examples of devices for which source device 102 generates encoded data for transmission to destination device 116. This disclosure refers to a device that performs coding (encoding and/or decoding) of data as a "coding" device. Accordingly, G-PCC encoder 200 and G-PCC decoder 300 represent examples of coding devices, specifically encoders and decoders, respectively. In some examples, source device 102 and destination device 116 may operate substantially symmetrically, such that source device 102 and destination device 116 each include encoding and decoding components. Thus, system 100 may support one-way or two-way transmission between source device 102 and destination device 116, for example, for streaming, playback, broadcast, telephony, navigation, and other applications.

Generally, data source 104 represents a source of data (i.e., raw, unencoded point cloud data) and may provide a sequential series of “frames” of data to G-PCC encoder 200; G-PCC encoder 200 encodes data for the frame. The data sources 104 of the source devices 102 may include various cameras or sensors, such as 3D scanners or light detection and ranging (LIDAR) devices, one or more video cameras, archives containing previously captured data, and/or It may include a point cloud capture device, such as any of a data feed interface for receiving data from a data content provider. Alternatively or additionally, point cloud data may be computer generated from scanners, cameras, sensors or other data. For example, data source 104 may generate computer graphics-based data as the source data, or may produce a combination of live, archived, and computer-generated data. In each case, G-PCC encoder 200 encodes captured data, pre-captured data, or computer-generated data. G-PCC encoder 200 may reorder frames from a received order (sometimes referred to as a "display order") to a coding order for coding. G-PCC encoder 200 may generate one or more bitstreams containing encoded data. Source device 102 may then output the encoded data via output interface 108 onto computer-readable storage medium 110, for example, for reception and/or retrieval by input interface 122 of destination device 116.

Memory 106 of source device 102 and memory 120 of destination device 116 may represent general purpose memory. In some examples, memory 106 and memory 120 may store raw data, eg, raw data from data source 104 and raw decoded data from G-PCC decoder 300. Additionally or alternatively, memory 106 and memory 120 may store software instructions executable by G-PCC encoder 200 and G-PCC decoder 300, respectively, for example. Although memory 106 and memory 120 are shown separately from G-PCC encoder 200 and G-PCC decoder 300 in this example, G-PCC encoder 200 and G-PCC decoder 300 may be functionally similar or It should be understood that internal memory may also be included for equivalent purposes. Additionally, memory 106 and memory 120 may store encoded data that is output from G-PCC encoder 200 and input to G-PCC decoder 300, for example. In some examples, a portion of memory 106 and memory 120 may be allocated as one or more buffers, eg, for storing raw decoded and/or encoded data. For example, memory 106 and memory 120 may store data representing point clouds.

Computer-readable storage medium 110 may represent any type of medium or device that can transport encoded data from source device 102 to destination device 116. In one example, computer-readable storage medium 110 is a communication medium that enables source device 102 to transmit encoded data directly to destination device 116 in real-time, such as over a radio frequency network or a computer-based network. represents. In accordance with a communication standard, such as a wireless communication protocol, output interface 108 may modulate transmitted signals containing encoded data, and input interface 122 may demodulate received transmitted signals. A communication medium may comprise any wireless or wired communication medium, such as the radio frequency (RF) spectrum or one or more physical transmission lines. The communication medium may form part of a packet-based network, such as a local area network, wide area network, or global network such as the Internet. Communication media may include routers, switches, base stations, or any other equipment that may be useful for facilitating communication from source device 102 to destination device 116.

In some examples, source device 102 may output encoded data from output interface 108 to storage device 112. Similarly, destination device 116 may access encoded data from storage device 112 via input interface 122. Storage device 112 may be a hard drive, Blu-ray disc, DVD, CD-ROM, flash memory, volatile or non-volatile memory, or any other suitable digital storage medium for storing encoded data. , may include any of a variety of distributed or locally accessed data storage media.

In some examples, source device 102 may output encoded data to file server 114 or another intermediate storage device that may store encoded data generated by source device 102. Destination device 116 may access stored data from file server 114 via streaming or downloading. File server 114 may be any type of server device capable of storing encoded data and transmitting the encoded data to destination device 116. File server 114 may represent a web server (eg, for a website), a file transfer protocol (FTP) server, a content distribution network device, or a network attached storage (NAS) device. Destination device 116 may access encoded data from file server 114 through any standard data connection, including an Internet connection. This can be used to access encoded data stored on a wireless channel (e.g., Wi-Fi connection), a wired connection (e.g., digital subscriber line (DSL), cable modem, etc.), or on a file server 114. may include combinations of both as appropriate. File server 114 and input interface 122 may be configured to operate according to streaming transmission protocols, download transmission protocols, or a combination thereof.

Output interface 108 and input interface 122 may be a wireless transmitter/receiver, modem, wired networking component (e.g., an Ethernet card), a wireless communication component operating in accordance with any of various IEEE 802.11 standards, or other physical configuration. Can represent an element. In examples where output interface 108 and input interface 122 comprise wireless components, output interface 108 and input interface 122 are encoded according to cellular communication standards such as 4G, 4G-LTE (Long Term Evolution), LTE Advanced, 5G, etc. may be configured to transfer data, such as data that has been In some examples where output interface 108 comprises a wireless transmitter, output interface 108 and input interface 122 may be compatible with other wireless transmitters, such as the IEEE 802.11 specification, the IEEE 802.15 specification (e.g., ZigBee(TM)), the Bluetooth(TM) standard, etc. It may be configured to transfer data, such as data encoded according to a standard. In some examples, source device 102 and/or destination device 116 may include respective system-on-chip (SoC) devices. For example, source device 102 may include a SoC device for performing functions due to G-PCC encoder 200 and/or output interface 108, and destination device 116 may include G-PCC decoder 300 and/or input interface 122 may include an SoC device for performing functions attributed to the 122.

The techniques of this disclosure may be used in a variety of applications, such as communication between autonomous vehicles, communication between scanners, cameras, sensors and processing devices such as local or remote servers, geographic mapping, or other applications. Any of the examples may be applied to support encoding and decoding.

Input interface 122 of destination device 116 receives an encoded bitstream from computer-readable storage medium 110 (eg, a communication medium, storage device 112, file server 114, etc.). The encoded bitstream is also used by the G-PCC decoder 300, such as syntax elements having values that describe the characteristics and/or processing of the coding unit (e.g., slice, picture, picture group, sequence, etc.) The G-PCC encoder 200 may include signaling information defined by the G-PCC encoder 200. Data consumer 118 uses the decrypted data. For example, data consumer 118 may use the decoded data to determine the location of a physical object. In some examples, data consumer 118 may include a display to present an image based on the point cloud.

G-PCC encoder 200 and G-PCC decoder 300 each include one or more microprocessors, digital signal processors (DSPs), application specific integrated circuits (ASICs), field programmable gate arrays (FPGAs), discrete logic, software , hardware, firmware, or any combination thereof. When the techniques are partially implemented in software, the device stores instructions for the software on a suitable non-transitory computer-readable storage medium and uses one or more processors to execute the techniques of this disclosure. instructions may be executed in hardware. Each of the G-PCC encoder 200 and the G-PCC decoder 300 may be included within one or more encoders or decoders, any of which may be a combined encoder/decoder (codec) within the respective device. May be integrated as part. A device including G-PCC encoder 200 and/or G-PCC decoder 300 may include one or more integrated circuits, microprocessors, and/or other types of devices.

G-PCC encoder 200 and G-PCC decoder 300 may operate according to a coding standard, such as a video point cloud compression (V-PCC) standard or a geometry point cloud compression (G-PCC) standard. This disclosure may generally refer to coding (eg, encoding and decoding) pictures to include the process of encoding or decoding data. An encoded bitstream typically includes a series of values for syntax elements that represent coding decisions (eg, coding modes).

This disclosure may generally refer to "signaling" some information, such as syntax elements. The term "signaling" may generally refer to the communication of values for syntax elements and/or other data used to decode encoded data. That is, G-PCC encoder 200 may signal values for syntax elements in the bitstream. Generally, signaling refers to producing a value in a bitstream. As mentioned above, source device 102 may generate a bitstream in substantially real-time or in a non-real-time manner, such as when storing syntax elements on storage device 112 for later retrieval by destination device 116. may be transported to destination device 116.

ISO/IEC MPEG (JTC1/SC29/WG11) is investigating the potential need for standardization of point cloud coding techniques with compression capabilities that significantly exceed those of current methods. The group is working together on this research effort in a collaboration known as the 3-Dimensional Graphics Team (3DG) to evaluate compression technology designs proposed by experts in the field. .

Point cloud compression activities are categorized into two different techniques. The first technique is "Video Point Cloud Compression" (V-PCC), which segments a 3D object, projects the segments onto multiple 2D planes (represented as "patches" in a 2D frame), and These are further coded by a legacy 2D video codec, such as the High Efficiency Video Coding (HEVC) (ITU-T H.265) codec. The second technique is "geometry-based point cloud compression" (G-PCC), which uses a 3D geometry, i.e., the location of a set of points in 3D space, and (for each point associated with the 3D geometry) ) and the associated attribute values directly. G-PCC addresses point cloud compression in both category 1 (static point clouds) and category 3 (dynamically acquired point clouds). A recent draft of the G-PCC standard is available in "Text of ISO/IEC FDIS 23090-9 Geometry-based Point Cloud Compression", ISO/IEC JTC 1/SC29/WG 7 MDS19617, Teleconference, October 2020. Yes, a description of the codec is available in G-PCC Codec Description, ISO/IEC JTC 1/SC29/WG 7 MDS19620, Teleconference, October 2020.

A point cloud includes a set of points in 3D space and may have attributes associated with the points. The attributes may be color information such as R, G, B, or Y, Cb, Cr, or reflectance information, or other attributes. Point clouds may be captured by various cameras or sensors, such as LIDAR sensors and 3D scanners, and may be computer generated. Point cloud data has a variety of applications including, but not limited to, construction (modeling), graphics (3D models for visualization and animation), and the automotive industry (LIDAR sensors used to aid navigation). used in

The 3D space occupied by the point cloud data may be surrounded by a virtual bounding box. The position of a point within a bounding box may be represented with a certain precision, and therefore the position of one or more points may be quantized based on the precision. At the minimal level, the bounding box is divided into voxels, which are the smallest units of space represented by unit cubes. Voxels in the bounding box may be associated with zero, one, or multiple points. The bounding box may be divided into multiple cubic/cuboid regions, which may be called tiles. Each tile may be coded into one or more slices. The partitioning of the bounding box into slices and tiles may be based on the number of points in each partition or based on other considerations (eg, particular regions may be coded as tiles). The slice regions may be further partitioned using partitioning decisions similar to those in video codecs.

FIG. 2 provides an overview of G-PCC encoder 200. FIG. 3 provides an overview of G-PCC decoder 300. The illustrated modules are logical and do not necessarily correspond one-to-one to the code implemented in the reference implementation of the G-PCC codec, i.e. the TMC13 test model software studied by ISO/IEC MPEG (JTC1/SC29/WG11). It does not correspond.

In both G-PCC encoder 200 and G-PCC decoder 300, point cloud positions are first coded. Attribute coding depends on the decoded geometry. Surface approximation analysis unit 212 and RAHT unit 218 of FIG. 2 and surface approximation synthesis unit 310 and RAHT unit 314 of FIG. 3 are options typically used for Category 1 data. The lifting units of LOD generation units 220 and 222, as well as LOD generation unit 316 and reverse lifting unit 318 of FIG. 3, are options typically used for Category 3 data. All other modules are common between categories 1 and 3.

For geometry, two different types of coding techniques exist: octree and predictive tree coding. Below, this disclosure focuses on octree coding. For Category 3 data, the compressed geometry is typically represented as an octree from the root of the individual voxel to the leaf level. For Category 3 data, the compressed geometry is typically represented as an octree from the root of the individual voxel to the leaf level. For Category 1 data, the compressed geometry is typically compressed into a pruned octtree (i.e., an octree from the root down to the leaf level for blocks larger than a voxel), within each leaf of the pruned octtree. is expressed by adding a model that approximates the surface of . In this way, both category 1 and category 3 data share an octree coding scheme, and category 1 data may additionally approximate voxels within each leaf using a surface model. The surface model used is a triangulation containing 1 to 10 triangles per block, resulting in a triangle soup. Thus, category 1 geometry codecs are known as Trisoup geometry codecs, and category 3 geometry codecs are known as octree geometry codecs.

FIG. 4 is a conceptual diagram illustrating an example octree decomposition for geometry coding. Octree 400 includes eight child nodes. Some of those child nodes, such as node 402, have no child nodes. However, other of the child nodes, such as node 404, have child nodes, some of the child nodes of node 404 also have child nodes, and so on.

At each node of the octree, occupancy is signaled (when not inferred) for one or more of the child nodes (up to eight nodes). Multiple neighbors are specified, including (a) nodes that share faces with the current octree node, and (b) nodes that share faces, edges, or vertices with the current octree node. Within each neighborhood, the occupancy of the node and/or its children may be used to predict the occupancy of the current node or its children. For sparse points at some nodes of the octree, the codec also supports a direct coding mode in which the 3D positions of the points are directly encoded. A flag may be signaled to indicate that direct mode is signaled. At the lowest level, the number of points associated with an octree node/leaf node may also be coded.

Once the geometry is coded, the attributes corresponding to the geometry points are coded. When there are multiple attribute points corresponding to one reconstructed/decoded geometry point, attribute values representing the reconstructed point can be derived.

There are three attribute coding methods for G-PCC, namely Area Adaptive Hierarchical Transform (RAHT) coding, interpolation-based hierarchical nearest neighbor prediction (prediction transform), and interpolation-based hierarchical nearest neighbor prediction with update/lifting steps (lifting transform). There is. RAHT and lifting are typically used for Category 1 data, and forecasting is typically used for Category 3 data. However, either method may be used for arbitrary data; just like using the geometry codec in G-PCC, the attribute coding method used to code the point cloud is specified in the bitstream. be done.

Coding of attributes may be performed at certain levels of detail (LOD), and each level of detail may be used to obtain a more refined representation of point cloud attributes. Each level of detail may be specified based on a distance metric from neighboring nodes or based on a sampling distance.

In the G-PCC encoder 200, the residual obtained as the output of the coding method for the attributes is quantized. The quantized residual may be coded using context-adaptive arithmetic coding.

In the example of FIG. 2, the G-PCC encoder 200 includes a coordinate transformation unit 202, a color transformation unit 204, a voxelization unit 206, an attribute transfer unit 208, an octree analysis unit 210, a surface approximation analysis unit 212, and an arithmetic coding unit. 214, a geometry reconstruction unit 216, a RAHT unit 218, an LOD generation unit 220, a lifting unit 222, a coefficient quantization unit 224, and an arithmetic encoding unit 226.

As shown in the example of FIG. 2, G-PCC encoder 200 may receive a set of locations and a set of attributes. A location may include the coordinates of a point within a point cloud. Attributes may include information about points in the point cloud, such as colors associated with points in the point cloud.

Coordinate transformation unit 202 may apply a transformation to the coordinates of the points to transform the coordinates from an initial domain to a transformed domain. This disclosure may refer to the transformed coordinates as transformed coordinates. Color transformation unit 204 may apply a transformation to transform the color information of the attributes to different domains. For example, color conversion unit 204 may convert color information from RGB color space to YCbCr color space.

Additionally, in the example of FIG. 2, voxelization unit 206 may voxelize the transformed coordinates. Voxelization of the transformed coordinates may include quantization and removing some points of the point cloud. In other words, multiple points of a point cloud may be contained within a single "voxel," which may then be treated as a single point in some respects. Additionally, octree analysis unit 210 may generate an octree based on the voxelized transformed coordinates. Additionally, in the example of FIG. 2, surface approximation analysis unit 212 may analyze the points to potentially determine a surface representation of the set of points. Arithmetic encoding unit 214 may entropy encode syntax elements representing information of the octree and/or surface determined by surface approximation analysis unit 212. G-PCC encoder 200 may output these syntax elements in a geometry bitstream.

Geometry reconstruction unit 216 may reconstruct transformed coordinates of points in the point cloud based on the octree, data representing the surface determined by surface approximation analysis unit 212, and/or other information. The number of transformed coordinates reconstructed by the geometry reconstruction unit 216 may differ from the original number of points of the point cloud due to voxelization and surface approximation. This disclosure may refer to the obtained points as reconstructed points. Attribute transfer unit 208 may transfer attributes of the original points of the point cloud to the reconstructed points of the point cloud.

Additionally, RAHT unit 218 may apply RAHT coding to the attributes of the reconstructed points. Alternatively or additionally, LOD generation unit 220 and lifting unit 222 may apply LOD processing and lifting, respectively, to the reconstructed point attributes. RAHT unit 218 and lifting unit 222 may generate coefficients based on the attributes. Coefficient quantization unit 224 may quantize the coefficients generated by RAHT unit 218 or lifting unit 222. Arithmetic encoding unit 226 may apply arithmetic coding to syntax elements representing quantized coefficients. G-PCC encoder 200 may output these syntax elements in the attribute bitstream.

In the example of FIG. 3, the G-PCC decoder 300 includes a geometry arithmetic decoding unit 302, an attribute arithmetic decoding unit 304, an octree synthesis unit 306, an inverse quantization unit 308, a surface approximation synthesis unit 310, a geometry reconstruction unit 312, It may include a RAHT unit 314, a LoD generation unit 316, an inverse lifting unit 318, an inverse coordinate transformation unit 320, and an inverse color transformation unit 322.

G-PCC decoder 300 may obtain a geometry bitstream and an attribute bitstream. Geometry arithmetic decoding unit 302 of G-PCC decoder 300 may apply arithmetic decoding (eg, context adaptive binary arithmetic coding (CABAC) or other types of arithmetic decoding) to syntax elements in the geometry bitstream. Similarly, attribute arithmetic decoding unit 304 may apply arithmetic decoding to syntax elements in the attribute bitstream.

Octree synthesis unit 306 may synthesize an octree based on syntax elements parsed from the geometry bitstream. In instances where a surface approximation is used in the geometry bitstream, the surface approximation synthesis unit 310 may determine the surface model based on syntax elements parsed from the geometry bitstream and based on the octree. .

Additionally, geometry reconstruction unit 312 may perform reconstruction to determine coordinates of points in the point cloud. Inverse coordinate transformation unit 320 may apply an inverse transformation to the reconstructed coordinates (positions) of points in the point cloud to convert them from the transformation domain back to the initial domain.

Additionally, in the example of FIG. 3, dequantization unit 308 may dequantize the attribute values. The attribute values may be based on syntax elements obtained from the attribute bitstream (eg, including syntax elements decoded by attribute arithmetic decoding unit 304).

Depending on how the attribute values are encoded, RAHT unit 314 may perform RAHT coding to determine color values for points in the point cloud based on the dequantized attribute values. . Alternatively, LOD generation unit 316 and inverse lifting unit 318 may determine color values for points in the point cloud using detail-based techniques.

Additionally, in the example of FIG. 3, inverse color transform unit 322 may apply an inverse color transform to the color values. The inverse color transform may be the inverse of the color transform applied by color transform unit 204 of G-PCC encoder 200. For example, color conversion unit 204 may convert color information from RGB color space to YCbCr color space. Accordingly, inverse color conversion unit 322 may convert the color information from YCbCr color space to RGB color space.

The various units in FIGS. 2 and 3 are shown to aid in understanding the operations performed by G-PCC encoder 200 and G-PCC decoder 300. A unit may be implemented as a fixed function circuit, a programmable circuit, or a combination thereof. Fixed function circuitry refers to circuitry that provides a specific function and is preset for the operations that may be performed. Programmable circuit refers to a circuit that can be programmed to perform a variety of tasks, providing flexibility in the operations that can be performed. For example, a programmable circuit may execute software or firmware that causes the programmable circuit to operate in a manner defined by the software or firmware instructions. Although fixed function circuits may execute software instructions (eg, to receive parameters or output parameters), the types of operations that fixed function circuits perform generally remain unchanged. In some examples, one or more of the units may be different circuit blocks (fixed function or programmable), and in some examples, one or more of the units may be an integrated circuit. good.

Predictive geometry coding was introduced as an alternative to octree geometry coding, where nodes are placed in a tree structure (defining the prediction structure) and different prediction strategies Used to predict node coordinates.

FIG. 5 shows an example of a prediction tree 410, which is represented as a directed graph with arrows pointing in prediction directions. Node 412 is the root vertex and has no predictors. Nodes 414A and 414B have two children. The dashed node has three children. White nodes have one child, and nodes 418A-418E are leaf nodes with no children. Each node has only one parent node.

Four prediction strategies are specified for each node based on its parent (p0), grandfather (p1) and grandfather's father (p2).

・No prediction/zero prediction (0)
・Delta prediction (p0)
・Line prediction (2*p0-p1)
・Parallelogram prediction (2*p0+p1-p2)
The G-PCC encoder 200 may employ any algorithm to generate the prediction tree, and the algorithm used may be determined based on the application/use case, and several strategies may be used. You can. Some strategies are described in the G-PCC Codec Description, ISO/IEC JTC 1/SC29/WG 7 MDS19620, Teleconference, October 2020.

For each node, residual coordinate values are coded in the bitstream starting from the root node in a depth-first manner.

Predictive geometry coding is primarily useful for category 3 (LIDAR-acquired) point cloud data, eg, for low-latency applications.

Angular mode may be used in predictive geometry coding, where the characteristics of the LIDAR sensor can be exploited in coding the predictive tree more efficiently. The coordinates of the position are converted to (r,φ,i) (radius, azimuth, and laser index) and the prediction is performed in this domain (the residuals are in the r,φ,i domain). ). Due to errors in rounding, the coding in,r,φ,i,is not lossless, so a second set of residuals,corresponding to Cartesian coordinates, is coded. A description of the encoding and decoding strategies used for angular modes for predictive geometry coding is reproduced below from the G-PCC Codec Description, ISO/IEC JTC 1/SC29/WG 7 MDS19620, Teleconference, October 2020. Ru.

The method focuses on point clouds acquired using a rotating lidar model. Here, the lidar has N lasers (eg, N=16, 32, 64) rotating around the Z axis according to the azimuthal angle φ (see Figure 6). Each laser may have a different elevation angle θ(i) _i=1...N and height σ(i) _i=1...N . Assume that laser i hits a point M with Cartesian integer coordinates (x,y,z) defined according to the coordinate system described in FIG.

This method provides modeling of the position of M with three parameters (r, φ, i), which are calculated as follows.

More precisely, the method is

We use a quantized version of (r,φ,i), denoted by where three integers

,

and i are calculated as follows.

however,
・(q _r ,o _r ) and (q _φ ,o _φ ) are

and

is a quantization parameter that controls the precision of each.

・ sign(t) is a function that returns 1 if t is positive, and (-1) otherwise.

- |t| is the absolute value of t.

To avoid reconstruction inconsistencies due to the use of floating point arithmetic, σ(i) _i=1...N and tan(θ(i)) _i=1...N are precomputed as and quantized.

however,
・(q _σ ,o _σ ) and (q _θ ,o _θ ) are

and

is a quantization parameter that controls the precision of each.

The reconstructed Cartesian coordinates are obtained as follows.

However, app_cos(.) and app_sin(.) are approximate values of cos(.) and sin(.). Calculations may be using fixed point representations, lookup tables, and linear interpolation.

For various reasons,
- Quantization
- approximation
- Model inaccuracy
- Note that it may differ from (x,y,z) due to inaccuracies in model parameters.

Let (r _x , r _y , r _z ) be the reconstruction residuals defined as follows.

In this method, the encoder (eg, G-PCC encoder 200) proceeds as follows.

・Model parameters

and

and encode the quantization parameters q _r , q _σ , q _θ , and q _φ Text of ISO/IEC FDIS 23090-9 Geometry-based Point Cloud Compression, ISO/IEC JTC 1/SC29/WG 7 MDS19617, Teleconference , representing the geometry prediction scheme described in October 2020

○ New predictors that exploit lidar characteristics may be introduced. For example, the rotational speed of a lidar scanner about the z-axis is typically constant. Therefore, the current

can be predicted as follows.

however,
○ (δ _φ (k)) _k=1...K is the set of possible velocities from which the encoder can choose. The index k may be written explicitly into the bitstream or may be inferred from the context based on a deterministic strategy applied by both the encoder and decoder.

o n(j) is the number of skipped points, which can be explicitly written to the bitstream or inferred from the context based on a deterministic strategy applied by both the encoder and decoder. n(j) is also later referred to as a "phi multiplier" and may in some implementations be used only with a delta predictor.

・Encode the reconstruction residual (r _x , r _y , r _z ) using each node
G-PCC decoder 300 proceeds as follows.

・Model parameters

and

and decode the quantization parameters q _r , q _σ , q _θ , and q _φ Text of ISO/IEC FDIS 23090-9 Geometry-based Point Cloud Compression, ISO/IEC JTC 1/SC29/WG 7 MDS19617, Teleconference, associated with the node according to the geometry prediction scheme described in October 2020.

Decode the parameters.

- Reconstructed coordinates as explained above

Decode the residual (r _x ,r _y ,r _z ) ○ As explained in the next section, lossy compression quantizes the reconstructed residual (r _x ,r _y ,r _z ) can be supported by calculating the original coordinates (x,y,z) as follows

Lossy compression can be achieved by applying quantization to the reconstruction residuals (r _x , r _y , r _z ) or by dropping points.

The quantized reconstruction residual is calculated as follows.

However, (q _x, o _x ), (q _y, o _y ) and (q _z, o _z ) are

,

and

is a quantization parameter that controls the precision of each.

Trellis quantization may be used to further improve the RD (rate-distortion) performance results.

Quantization parameters may be varied at the sequence/frame/slice/block level to achieve region adaptive quality and for rate control purposes.

G-PCC encoder 200 may be configured to perform motion estimation for inter prediction. In the following, we describe the motion estimation (global and local) process applied in the InterEM software. InterEM is based on an octree-based coding extension for inter prediction. Although motion estimation is applied to an octree-based framework, a similar process (or at least part of it) may also be applicable to predictive geometry coding.

There are two types of motion involved in the G-PCC InterEM software: global motion matrices and local nodal motion vectors. Global motion parameters include rotation matrices and translation vectors that can be applied to all points in the predicted (reference) frame. The local node motion vector of a node in an octree is a motion vector that applies only to points within the node in the predicted (reference) frame. The details of the motion estimation algorithm in InterEM are explained below.

FIG. 7 shows a flowchart illustrating the motion estimation process. Inputs to the process include predicted frame 420 and current frame 422. G-PCC encoder 200 first estimates global motion on a global scale (424). After applying the estimated global motion to the predicted frame 420 (426), the G-PCC encoder 200 estimates the local motion at a finer scale, node level, in the octree (428). Finally, G-PCC encoder 200 applies motion compensation to the estimated local node motion and encodes the determined motion vectors and points (430).

The embodiment of FIG. 7 is described in more detail below. G-PCC encoder 200 may perform a process to estimate global motion matrices and translation vectors. In InterEM software, a global motion matrix is defined to match the feature points between the predicted frame (reference) and the current frame.

FIG. 8 shows an example of a global motion estimation process that may be performed by G-PCC encoder 200. In the example of FIG. 8, G-PCC encoder 200 finds feature points (432), samples the feature points (434), and performs motion estimation using a least mean squares (LMS) algorithm (436). .

In the algorithm illustrated by FIG. 8, points that have a large position change between the predicted frame and the current frame may be defined as feature points. For each point in the current frame, the closest point in the predicted frame is found and point pairs are constructed between the current frame and the predicted frame. If the distance between paired points is greater than a threshold, the paired points are considered feature points.

After finding the feature points, sampling at the feature points is performed to reduce the scale of the problem (eg, by selecting a subset of the feature points to reduce the complexity of motion estimation). The LMS algorithm is then applied to derive motion parameters by attempting to reduce the error between the respective feature points in the predicted frame and the current frame.

FIG. 9 shows an example process for estimating local node motion vectors. In the local node estimation algorithm shown in FIG. 9, motion vectors are estimated in a recursive manner. The cost function used to select the most suitable motion vector may be based on rate-distortion costs. In FIG. 9, path 440 shows the process for the current node that is not split into eight children, and path 442 shows the process for the current node that is split into eight children.

If the current node is not split into eight children (440), the motion vector that can yield the lowest cost between the current node and the predicted node is determined. If the current node is split into eight children (442), a motion estimation algorithm is applied and the total cost under the split condition is obtained by adding the estimated cost values of each child node. The decision to split or not split is reached by comparing the cost between split and no split; if split, each subnode is assigned its respective motion vector (or ), otherwise the current node is assigned a motion vector.

Two parameters that affect the performance of motion vector estimation are block size (BlockSize) and minimum prediction unit size (MinPUSize). BlockSize defines the upper bound of the node size to apply motion vector estimation, and MinPUSize defines the lower bound.

The InterEM software, which is essentially an octree coder, performs occupancy prediction and uses global/local motion and reference point cloud information while making occupancy prediction. Therefore, the InterEM software does not perform direct point motion compensation; direct point motion compensation applies motion to a point in a reference frame, e.g. to project the point to the current frame. may include. The difference between the actual point and the predicted point may then be coded, which may be more effective in performing inter-prediction.

One or more techniques disclosed in this document may be applied independently or combined. This disclosure proposes techniques for performing direct motion compensation while still benefiting from a flexible octree partition-based coding structure. In the following, these techniques are primarily presented in the context of octree partitioning, but can also be extended to OTQTBT (octtree-quadtree-binary tree) partitioning scenarios.

G-PCC encoder 200 and/or G-PCC decoder 300 may be configured to perform high-level segmentation and process mode flags. In one example of this disclosure, G-PCC encoder 200 and/or G-PCC decoder 300 may be configured to perform octree-based segmentation (for occupancy prediction) on the current point cloud. However, rather than stopping the octree split at some level and then coding the occupancy, it is possible to directly code points inside that octree-leaf volume (in what follows, this is referred to as direct prediction) is possible. A leaf node size, or octree depth value, may be signaled to specify the level at which octree splitting is stopped and points are coded as octree leaf volumes.

For each such octree leaf node, if the octree splitting is stopped and "direct prediction" is activated, then the set of points inside the octree leaf volume are either intra-predicted or inter-predicted. A flag may be signaled to indicate whether the In the geometry parameter set, maximum and minimum sizes for the octree leaves may be defined.

FIG. 10 is a conceptual diagram showing high-level octree partitioning of octree 444. FIG. 10 is an example of an octree leaf node for direct prediction that includes 13 points (O0 to O12). In a special case, the root node of an octree (without splitting) may be coded using "direct prediction".

G-PCC encoder 200 and/or G-PCC decoder 300 may be configured to perform intra prediction. When the flag value is set to intra, all points inside the volume are intra predicted. For this purpose, a "local prediction tree" is generated. The generation of such a tree is non-normative (points may be traversed in different orders by azimuthal, Morton, radial, etc., or in some other order). For each point, its prediction mode (0, 1, 2, 3), number of children information, first-order residual, and second-order residual (if angular mode is enabled) are signaled. So, in summary, intra prediction is similar in function to predictive geometry coding.

Alternatively, a single prediction mode may also be signaled for all points within the octree leaf volume, thereby reducing the associated signaling costs. The radius value (if angular mode is enabled) or (x, y, z) value (if angular mode is disabled) for the zero predictor is e.g. Can be set to the top left point inside. Alternatively, a zero predictor may be signaled for the octree leaf volume, or an index indicating a point to use for the zero predictor inside the octree leaf volume may be signaled. Moreover, if the value is outside the octree leaf volume, clipping may be performed after performing prediction/reconstruction.

The syntax table for intra-prediction is incorporated by reference in its entirety, Text of ISO/IEC FDIS 23090-9 Geometry-based Point Cloud Compression, ISO/IEC JTC 1/SC29/WG 7 MDS19617, Teleconference, 2020 The syntax can be similar to that described for prediction trees in October.

G-PCC encoder 200 and/or G-PCC decoder 300 may be configured to perform inter prediction. Assuming that an octree leaf has N points inside, (O(0), ....,O(N-1)), for inter prediction, motion estimation is performed using an octree leaf. is performed on the encoder side using the current set of points in the volume to find the best match with a similar set of points in the reference point cloud frame (if the reference point cloud is not motion compensated or global motion compensation (if applicable). For inter-prediction of octree leaves, the following is signaled:

i. Reference index (if you have multiple reference point cloud frames to predict from)
ii. Motion Vector Difference (MVD). (the difference between the actual MV and the predicted MV (as explained above regarding performing MV prediction from neighbors))
iii.Number of points in the octree leaf (N).

iv. The linear (and also quadratic if angular mode is enabled) residuals (tuplets, differences between 3D coordinates) for N points (R'i).

Below, this disclosure describes a motion compensation process given a signaled reference index (if applicable) and MV for an octree node.

a. The current octree leaf has its top left point at (X0, Y0, Z0) and dimensions as (Sx, Sy, Sz), and the motion vector is MV = (MVx, MVy, MVz) It is. Therefore, the corresponding reference block in the reference point cloud frame has the upper left corner (Xr, Yr, Zr) = (X0 - MVx, Y0 - MVy, Z0 - MVz) and the size of (Sx, Sy, Sz). have

b. Fetch all points inside this reference block and located in a 1D array, the order may be predetermined/fixed or signaled for the octree leaves, for example. For example, suppose there are M such points (R0,....R(M-1)) with coordinates (in the reference frame), where Ri denotes the 3D coordinates of the i-th point. Triplets provided. As shown in Figure 12, when i = 0...(M-1)).

c. All points are motion compensated by applying the signaled MV, which is used as the predicted geometry position (Pi), i.e. Pi = Ri + MV, as shown in Figure 13.

d. If angular mode is enabled, for all M points, the corresponding

is derived.

In Figure 12, the current set of points is labeled O0-O12, the reference set of points is labeled R0-R12, and N = M = 13. In Figure 13, the current set of points is labeled O0-O12, the motion compensated reference set of points is labeled P0-P12, and N = M = 13.

There are three possible scenarios here.

i.N = M (current octree node and reference block have the same number of points).

ii.N>M.

iii.N < M.

The first scenario where N=M will be explained next. The residuals (first order and, if applicable, second order residuals) are added directly to the motion compensated points to produce the reconstruction = Pi + R'i.

The second scenario where N > M is described next. The motion compensated point (Pi) in the 1D array is expanded using the last value P(M-1), i.e. [P'0, ......P'(M-1), P'(M),.....P'(N-1)] = [P0, ......P(M-1), P(M-1),.....P( M-1)] and the residuals are then added directly to produce reconstruction = P'i + R'i. Alternatively, a zero predictor is used for expansion.

A third scenario in which N < M is described next. The residual is added directly to the first N points, ie [P0,...P(N-1)], producing reconstruction = Pi + Ri.

G-PCC encoder 200 and/or G-PCC decoder 300 may be configured to perform MV prediction from neighbors. From the MVs of the spatiotemporally neighboring inter-octree leaves, it is possible to predict the MV of the current octree leaf, and the corresponding MV difference can be signaled. The MV prediction index may be signaled when there are multiple spatiotemporal candidates. In addition to spatiotemporal neighbor candidates, it is also possible to add previously used MV candidates in the MV candidate list based on recent history.

It is also possible to merge MV information with spatiotemporal neighbors by specifying a signaled "merge flag." A merge index may be signaled when there are multiple spatiotemporal candidates.

G-PCC encoder 200 and/or G-PCC decoder 300 may be configured to perform skipping the first order residual.

For good inter-prediction, the first-order residual that is applicable when angular mode is enabled is typically small or even close to zero. In such cases, it is also possible to skip the first-order residuals completely for all points in the octree leaf volume. Therefore, the primary_residual_skip flag may be signaled for octree leaf volumes. In this case, the difference between the original point and the predicted point is fully coded in the quadratic residual.

Alternatively, primary_residual_skip_flag may be signaled at an octree level higher than the octree leaf volume and applied to one or more octree leaves associated with the octree level.

The table below is the syntax table for inter-predicted octree leaves.

Examples of various aspects of this disclosure may be used individually or in any combination.

FIG. 14 is a conceptual diagram illustrating an example distance measurement system 600 that may be used with one or more techniques of this disclosure. In the example of FIG. 14, distance measurement system 600 includes an illuminator 602 and a sensor 604. Illuminator 602 may emit light 606. In some examples, illuminator 602 may emit light 606 as one or more laser beams. Light 606 can be within one or more wavelengths, such as infrared wavelengths or visible light wavelengths. In other examples, light 606 is not coherent laser light. When light 606 encounters an object, such as object 608, light 606 produces returned light 610. Returned light 610 may include backscattered light and/or reflected light. Return light 610 may pass through lens 611 which directs return light 610 to produce an image 612 of object 608 on sensor 604. Sensor 604 generates signal 618 based on image 612. Image 612 may include a set of points (eg, as represented by the dots in image 612 of FIG. 14).

In some examples, illuminator 602 and sensor 604 may be mounted on a rotating structure such that illuminator 602 and sensor 604 capture a 360 degree view of the environment. In other examples, the ranging system 600 includes one or more optical components ( (eg, mirrors, collimators, diffraction gratings, etc.). Although the example of FIG. 14 shows only a single illuminator 602 and sensor 604, the distance measurement system 600 may include multiple sets of illuminators and sensors.

In some examples, illuminator 602 generates a structured light pattern. In such an example, the distance measurement system 600 may include a plurality of sensors 604 on which respective images of the structured light pattern are formed. The distance measurement system 600 may use the disparity between images of the structured light pattern to determine the distance to the object 608 from which the structured light pattern backscatters. Structured light-based distance measurement systems may have a high level of accuracy (eg, accuracy in the sub-millimeter range) when object 608 is relatively close to sensor 604 (eg, 0.2 meters to 2 meters). This high level of accuracy can be useful in facial recognition applications such as unlocking mobile devices (eg, mobile phones, tablet computers, etc.) and for security applications.

In some examples, distance measurement system 600 is a time-of-flight (ToF)-based system. In some examples where distance measurement system 600 is a ToF-based system, illuminator 602 generates pulses of light. In other words, illuminator 602 may modulate the amplitude of emitted light 606. In such an example, sensor 604 detects return light 610 from the pulse of light 606 generated by illuminator 602. The distance measurement system 600 then backscatters the light 606 therefrom based on the delay between when the light 606 is emitted and when it is detected and the known speed of light in air. A distance to object 608 may be determined. In some examples, instead of (or in addition to) modulating the amplitude of the radiation 606, the illuminator 602 may modulate the phase of the radiation 606. In such an example, the sensor 604 detects the phase of the returned light 610 from the object 608 and uses the speed of light and when the illuminator 602 produces the light 606 at a particular phase and the sensor 604 may determine the distance to a point above object 608 based on the time difference between when it detects returned light 610 at that particular phase.

In other examples, the point cloud may be generated without using illuminator 602. For example, in some examples, sensor 604 of distance measurement system 600 may include two or more optical cameras. In such an example, ranging system 600 may use an optical camera to capture a stereoscopic image of the environment including object 608. Distance measurement system 600 (eg, point cloud generator 620) may then calculate disparity between locations in the stereo image. Distance measurement system 600 may then use the parallax to determine the distance to the location shown in the stereo image. From these distances, point cloud generator 620 may generate a point cloud.

Sensor 604 may also detect other attributes of object 608, such as color and reflective information. In the example of FIG. 14, point cloud generator 620 may generate a point cloud based on signal 618 generated by sensor 604. Distance measurement system 600 and/or point cloud generator 620 may form part of data source 104 (FIG. 1).

FIG. 15 is a conceptual diagram illustrating an example vehicle-based scenario in which one or more techniques of this disclosure may be used. In the example of FIG. 15, vehicle 700 includes a laser package 702, such as a LIDAR system. Although not shown in the example of FIG. 15, vehicle 700 may also include a data source and a G-PCC encoder, such as G-PCC encoder 200 (FIG. 1). In the example of FIG. 15, laser package 702 emits a laser beam 704 that reflects off pedestrians 706 or other objects in the road. A data source on vehicle 700 may generate a point cloud based on signals generated by laser package 702. A G-PCC encoder in vehicle 700 may encode the point cloud to generate bitstream 708. Bitstream 708 may include far fewer bits than the uncoded point cloud obtained by the G-PCC encoder. An output interface of vehicle 700 (eg, output interface 108 (FIG. 1)) may transmit bitstream 708 to one or more other devices. Therefore, vehicle 700 may be able to transmit bitstream 708 to other devices more quickly than unencoded point cloud data. Additionally, bitstream 708 may require less data storage capacity.

In the example of FIG. 15, vehicle 700 may transmit bitstream 708 to another vehicle 710. Vehicle 710 may include a G-PCC decoder, such as G-PCC decoder 300 (FIG. 1). A G-PCC decoder in vehicle 710 may decode bitstream 708 to reconstruct the point cloud. Vehicle 710 may use the reconstructed point cloud for various purposes. For example, vehicle 710 determines that pedestrian 706 is on the road in front of vehicle 700 based on the reconstructed point cloud, and thus, for example, It can begin to slow down even before the person is aware of it. Thus, in some examples, vehicle 710 may perform autonomous navigation operations, generate notifications or alerts, or take other actions based on the reconstructed point cloud.

Additionally or alternatively, vehicle 700 may transmit bitstream 708 to server system 712. Server system 712 may use bitstream 708 for various purposes. For example, server system 712 may store bitstream 708 for subsequent reconstruction of the point cloud. In this example, server system 712 may use the point cloud along with other data (eg, vehicle telemetry data generated by vehicle 700) to train the autonomous driving system. In other examples, server system 712 may store bitstream 708 for subsequent reconstruction for a forensic crash investigation (e.g., if vehicle 700 collides with pedestrian 706) or for navigation. A notification or instruction for the purpose may be sent to vehicle 700 or vehicle 710.

FIG. 16 is a conceptual diagram illustrating an example extended reality system in which one or more techniques of this disclosure may be used. Extended reality (XR) is a term used to cover a variety of technologies including augmented reality (AR), mixed reality (MR), and virtual reality (VR). In the example of FIG. 16, user 800 is located at first location 802. In the example of FIG. User 800 is wearing XR headset 804. As an alternative to XR headset 804, user 800 may use a mobile device (eg, mobile phone, tablet computer, etc.). XR headset 804 includes a depth sensing sensor, such as a LIDAR system, that detects the position of a point above object 806 at first location 802. The data source of the XR headset 804 may use the signals generated by the depth sensing sensors to generate a point cloud representation of the object 806 at the first location 802. XR headset 804 may include a G-PCC encoder (eg, G-PCC encoder 200 of FIG. 1) configured to encode the point cloud and generate bitstream 808.

XR headset 804 may transmit bitstream 808 (eg, over a network such as the Internet) to XR headset 810 worn by user 812 at a second location 814. XR headset 810 may decode bitstream 808 and reconstruct the point cloud. XR headset 810 may use the point cloud to generate an XR visualization (eg, AR, MR, VR visualization) representing object 806 at first location 802. Thus, in some examples, such as when XR headset 810 generates a VR visualization, user 812 at location 814 may have a 3D immersive experience of first location 802. In some examples, XR headset 810 may determine the position of the virtual object based on the reconstructed point cloud. For example, the XR headset 810 determines based on the reconstructed point cloud that the environment (e.g., first location 802) includes a flat surface and then determines that the virtual object (e.g., a cartoon character) It may be determined that it should be placed on a flat surface. XR headset 810 may generate an XR visualization with virtual objects at determined positions. For example, the XR headset 810 may show a cartoon character sitting on a flat surface.

FIG. 17 is a conceptual diagram illustrating an example mobile device system in which one or more techniques of this disclosure may be used. In the example of FIG. 17, a mobile device 900, such as a mobile phone or tablet computer, includes a depth sensing sensor, such as a LIDAR system, that detects the location of a point on an object 902 within the environment of the mobile device 900. A data source on mobile device 900 may generate a point cloud representation of object 902 using the signals generated by the depth sensing sensor. Mobile device 900 may include a G-PCC encoder (eg, G-PCC encoder 200 of FIG. 1) configured to encode the point cloud and generate bitstream 904. In the example of FIG. 17, mobile device 900 may send the bitstream to a remote device 906, such as a server system or other mobile device. Remote device 906 may decode bitstream 904 and reconstruct the point cloud. Remote device 906 may use the point cloud for various purposes. For example, remote device 906 may use a point cloud to generate a map of mobile device 900's environment. For example, remote device 906 may generate a map of the interior of a building based on the reconstructed point cloud. In another example, remote device 906 may generate an image (eg, computer graphics) based on a point cloud. For example, remote device 906 may use points in the point cloud as vertices of a polygon and use the color attributes of the points as a basis for shading the polygon. In some examples, remote device 906 may perform facial recognition using a point cloud.

FIG. 18 is a flowchart illustrating example operations for decoding a bitstream that includes point cloud data. G-PCC decoder 300 may perform the operations of FIG. 18 as part of decoding the point cloud. In the example of FIG. 18, G-PCC decoder 300 determines an octree (1000) that defines an octree-based partitioning of the space containing the point cloud. A leaf node of the octree contains one or more points of the point cloud.

G-PCC decoder 300 directly codes the position of each of one or more points in a leaf node (1002). To directly code the position of each of the one or more points at a leaf node, G-PCC decoder 300 generates a prediction of the one or more points (1004) and, based on the prediction, one or more points. Determine multiple points (1006). To directly decode the position of each of the one or more points in a leaf node, the G-PCC decoder 300 receives a flag, the first value for the flag being one or more a second value for the flag indicates that the prediction for one or more points is generated by inter-prediction, and a value for the flag may be configured to decode one or more points using intra-prediction or inter-prediction based on.

G-PCC decoder 300 may be configured to receive octree leaf volumes that specify volumes of leaf nodes in a bitstream for a point cloud. For example, assume that the entire point cloud is encapsulated within a W×W×W cuboid. The point cloud may be partitioned recursively, and for a given partitioning depth d, the octree leaf volume is W/2 ^d ×W/2 ^d ×W/2 ^d . At this level, an occupancy flag (binary) can be signaled, when the occupancy flag is equal to 1, it indicates that the cuboid has at least one point. When the occupancy flag is 1, a further intra or inter flag may then be signaled, indicating whether the points inside the cuboid are intra-predicted or inter-predicted, respectively.

To generate the one or more point predictions, the G-PCC decoder 300 may be further configured to generate the one or more point predictions using intra prediction; In order to generate predictions for the one or more points, G-PCC decoder 300 may be further configured to determine a local prediction tree for the one or more points.

To determine one or more points based on the prediction, G-PCC decoder 300 determines the prediction mode, first-order residual, for each of the one or more points in the bitstream for the point cloud. and a quadratic residual. To generate one or more point predictions, G-PCC decoder 300 may be configured to generate one or more point predictions using inter prediction, and may be configured to generate one or more point predictions using inter prediction. To generate a prediction of one or more points, G-PCC decoder 300 uses the one or more points to determine a similar set of points in the reference point cloud frame. It may be further configured to perform estimation.

To generate the one or more point predictions, the G-PCC decoder 300 may be further configured to generate the one or more point predictions using inter prediction; In order to generate a prediction of one or more points, the G-PCC decoder 300 performs a motion estimation process to predict one or more points based on a set of points in a reference point cloud frame. It may be further configured to perform compensation. To perform motion compensation, G-PCC decoder 300 may be further configured to apply a motion vector to a set of points in a reference point cloud frame to determine a prediction of one or more points. . G-PCC decoder 300 may be configured to predict motion vectors based on motion vectors of spatiotemporally neighboring inter-octtree leaves.

G-PCC decoder 300 may be further configured to reconstruct a point cloud from the point cloud data. As part of reconstructing the point cloud, G-PCC decoder 300 may be further configured to determine the location of one or more points of the point cloud based on the planar position.

Depending on the example, some acts or events of any of the techniques described herein may be performed in different sequences, and may be added, integrated, or excluded entirely ( For example, it is recognized that not all actions or events described may be necessary for the practice of a technique). Moreover, in some examples, acts or events may be performed in parallel, eg, through multi-threaded processing, interrupt processing, or multiple processors, rather than sequentially.

In one or more examples, the described functionality may be implemented as hardware, software, firmware, or any combination thereof. If implemented in software, the functions may be stored on or transmitted over a computer-readable medium as one or more instructions or code for execution by a hardware-based processing unit. can be done. Computer-readable storage media include computer-readable storage media corresponding to tangible media such as data storage media or any medium that facilitates transfer of a computer program from one place to another according to, e.g., a communication protocol. may include communication media. Thus, computer-readable storage media generally may correspond to (1) non-transitory, tangible computer-readable storage media, or (2) a communication medium such as a signal or carrier wave. A data storage medium can be any type of data storage medium that can be accessed by one or more computers or one or more processors to retrieve instructions, code, and/or data structures for implementation of the techniques described in this disclosure. It can be any available medium. A computer program product may include a computer readable storage medium.

By way of example and not limitation, such computer-readable storage medium may include RAM, ROM, EEPROM, CD-ROM or other optical disk storage, magnetic disk storage or other magnetic storage device, flash memory, or a memory containing instructions or data structures. Any other medium that can be used to store desired form of program code and that can be accessed by a computer may be provided. Also, any connection is properly termed a computer-readable storage medium. For example, instructions may be transmitted from a website, server, or other remote source using coaxial cable, fiber optic cable, twisted pair, digital subscriber line (DSL), or wireless technologies such as infrared, radio, and microwave. If so, coaxial cable, fiber optic cable, twisted pair, DSL, or wireless technologies such as infrared, radio, and microwave are included in the definition of medium. It should be understood, however, that computer-readable storage media and data storage media do not include connections, carrier waves, signals or other transitory media, and instead refer to non-transitory tangible storage media. As used herein, "disk" and "disc" refer to compact disc (disc) (CD), laser disc (disc), optical disc (disc), digital versatile disc (disc) (DVD), and floppy disc. (discs) and Blu-ray discs (discs), with discs typically reproducing data magnetically and discs reproducing data optically using lasers. Combinations of the above should also be included within the scope of computer-readable media.

The instructions may be implemented in one or more digital signal processors (DSPs), general purpose microprocessors, application specific integrated circuits (ASICs), field programmable gate arrays (FPGAs), or other equivalent integrated or discrete logic circuits. , may be executed by one or more processors. Accordingly, the terms "processor" and "processing circuitry" as used herein may refer to any of the structures described above, or any other structure suitable for implementing the techniques described herein. . Additionally, in some aspects, the functionality described herein may be provided within dedicated hardware and/or software modules configured for encoding and decoding, or in combination codecs. May be incorporated. Also, the techniques may be implemented entirely in one or more circuits or logic elements.

The techniques of this disclosure may be implemented in a wide variety of devices or apparatuses, including wireless handsets, integrated circuits (ICs), or sets of ICs (eg, chipsets). Although various components, modules, or units are described in this disclosure to emphasize functional aspects of devices configured to perform the disclosed techniques, they do not necessarily require implementation by different hardware units. Not necessarily. Rather, as explained above, the various units may be combined in a codec hardware unit, or include one or more processors as explained above, together with suitable software and/or firmware. It may also be provided by a collection of interoperable hardware units.

The numbered sections below indicate one or more aspects of the devices and techniques described in this disclosure.

Clause 1A: A method for coding a point cloud, the step of determining an octree defining an octree-based partition of the space containing the point cloud, wherein the leaf nodes of the octree are Predicting the one or more points using intra-prediction or inter-prediction, with steps including one or more points and the position of each of the one or more points at a leaf node being directly signaled. and coding a syntax element indicating whether the one or more points are predicted using intra prediction or inter prediction.

Clause 2A: The method of Clause 1A, wherein an octree leaf volume specifying the volume of the leaf node is signaled in the bitstream.

Clause 3A: The step of generating the one or more point predictions includes the step of generating the one or more point predictions using the intra prediction, and the step of generating the one or more point predictions using the intra prediction. The method of clause 1A or 2A, wherein the step of generating a prediction for the points includes the step of determining a local prediction tree for the one or more points.

Clause 4A: The method of Clause 3A, wherein at least one of a prediction mode, a first-order residual, and a second-order residual is signaled for each of the one or more points.

Clause 5A: The step of generating a prediction of one or more points comprises the step of generating a prediction of one or more points using inter prediction; of clause 1A or 2A, wherein the step of generating a prediction of the points in the reference point cloud frame comprises the step of performing motion estimation with the one or more points to determine a similar set of points in the reference point cloud frame. Method.

Clause 6A: Generating a prediction for the one or more points comprises generating a prediction for the one or more points using inter prediction; Clause 1A, 2A, or 5A either way.

Clause 7A: The method of Clause 6A, wherein the step of performing motion compensation comprises applying a motion vector to a set of points in a reference point cloud frame to determine a prediction of the one or more points.

Clause 8A: The method of Clause 7A, further comprising predicting a motion vector based on the motion vectors of the spatiotemporal neighboring inter-octree leaves.

Clause 9A: Any method of Clauses 1A to 8A further comprising the step of generating a point cloud.

Clause 10A: Device for processing point clouds, comprising one or more means for performing any of the methods of Clauses 1A to 9A.

Clause 11A: A device according to Clause 10A, wherein the one or more means comprises one or more processors implemented in a circuit.

Clause 12A: The device of any of Clause 10A or 11A, further comprising a memory for storing data representing a point cloud.

Clause 13A: A device according to clauses 10A to 12A, where the device comprises a decoder.

Clause 14A: A device according to any of Clauses 10A to 13A, where the device comprises an encoder.

Clause 15A: A device according to any of Clauses 10A to 14A, further comprising a device for generating a point cloud.

Clause 16A: A device according to any of Clauses 10A to 15A, further comprising a display for presenting an image based on a point cloud.

Clause 17A: A computer-readable storage medium having instructions stored thereon that, when executed, cause one or more processors to perform the method of any of Clauses 1A to 9A.

Clause 1B: A device for decoding a bitstream containing point cloud data, comprising a memory for storing the point cloud data and one or more processors coupled to the memory and implemented in a circuit. , the one or more processors determining an octree defining an octree-based partition of a space containing the point cloud, wherein a leaf node of the octree is one or more of the point clouds; including a plurality of points, and configured to directly decode the position of each of the one or more points at the leaf node, and to directly decode the position of each of the one or more points at the leaf node. For decoding, the one or more processors are further configured to generate a prediction of the one or more points and, based on the prediction, determine the one or more points. ,device.

Clause 2B: In order to directly decode the position of each of the one or more points in a leaf node, the one or more processors receive a flag, the first value for the flag being: indicating that the predictions for the one or more points are generated by intra-prediction, and a second value for the flag indicates that the predictions for the one or more points are generated by inter-prediction; and the device of clause 1B, further configured to decode the one or more points using intra-prediction or inter-prediction based on the value of the flag.

Clause 3B: The device of Clause 1B, wherein the one or more processors are further configured to receive octree leaf volumes specifying volumes of leaf nodes in the bitstream that includes the point cloud.

Clause 4B: To generate the one or more point predictions, the one or more processors are further configured to generate the one or more point predictions using intra prediction; the one or more processors are further configured to determine a local prediction tree for the one or more points to use the prediction to generate a prediction for the one or more points; Clause 1B Devices.

Clause 5B: In order to determine the one or more points based on the prediction, the one or more processors determine a prediction mode for each of the one or more points in the bitstream comprising the point cloud; The device of clause 1B, further configured to receive at least one of a first-order residual, and a second-order residual.

Clause 6B: To generate the one or more point predictions, the one or more processors are further configured to generate the one or more point predictions using inter prediction; To generate a prediction of one or more points using prediction, one or more processors use one or more processors to determine a similar set of points in a reference point cloud frame. The device of clause 1B, further configured to perform motion estimation using points.

Clause 7B: The one or more processors are further configured to generate the one or more point predictions using inter-prediction to generate the one or more point predictions; Using prediction, one or more processors predict one or more points based on a set of points in a reference point cloud frame to generate a prediction of one or more points. The device of Clause 1B further configured to perform motion compensation in order to perform motion compensation.

Clause 8B: To perform motion compensation, one or more processors apply a motion vector to a set of points in a reference point cloud frame to determine a prediction of the one or more points. Further consisting of the device of clause 7B.

Clause 9B: The device of Clause 8B, wherein the one or more processors are further configured to predict motion vectors based on motion vectors of spatiotemporally neighboring inter-octtree leaves.

Clause 10B: The device of Clause 1B, wherein the one or more processors are further configured to reconstruct a point cloud from point cloud data.

Clause 11B: Clause 10B, wherein the one or more processors are configured to determine the position of one or more points of the point cloud based on the planar position as part of reconstructing the point cloud. device.

Clause 11B: The one or more processors, as part of reconstructing the point cloud, calculate one or more of the point cloud based on the directly decoded position of each of the one or more points at the leaf nodes. Clause 10B device configured to determine the position of a plurality of points.

Clause 12B: The device of Clause 11B, wherein the one or more processors are further configured to generate a map of the interior of a building based on the point cloud.

Clause 13B: The device of Clause 11B, wherein the one or more processors are further configured to perform autonomous navigation operations based on the point cloud.

Clause 14B: The device of Clause 11B, wherein the one or more processors are further configured to generate computer graphics based on the point cloud.

Clause 15B: The one or more processors are configured to determine the position of a virtual object based on a point cloud and to generate an extended reality (XR) visualization in which the virtual object is at the determined position. Clause 11B devices constituted by.

Clause 16B: The device of Clause 11B further comprising a display for presenting an image based on a point cloud.

Clause 17B: A Clause 1B device where the device is a mobile phone or tablet computer.

Clause 18B: A clause 1B device where the device is a vehicle.

Clause 19B: A Clause 1B device where the device is an extended reality device.

Clause 20B: A method for decoding a point cloud, the method comprising: determining an octree that defines an octree-based partition of a space containing the point cloud, wherein leaf nodes of the octree directly decoding the position of each of the one or more points at the leaf node, directly decoding the position of each of the one or more points at the leaf node. A method, wherein the step of decoding includes the steps of generating a prediction of one or more points and determining one or more points based on the prediction.

Clause 21B: Directly decoding the position of each of the one or more points at the leaf node is receiving a flag, the first value for the flag being a prediction of the one or more points. is generated by intra-prediction, and a second value for the flag indicates that the prediction for the one or more points is generated by inter-prediction, based on the value of the flag, and decoding the one or more points using intra-prediction or inter-prediction.

Clause 22B: The method of Clause 20B, further comprising receiving an octree leaf volume specifying a volume of a leaf node in the bitstream for the point cloud.

Clause 23B: The step of generating a prediction of one or more points includes the step of generating a prediction of one or more points using intra-prediction; The method of clause 20B, wherein the step of generating a prediction for the points includes determining a local prediction tree for the one or more points.

Clause 24B: Determining one or more points based on the prediction comprises, in the bitstream for the point cloud, a prediction mode for each of the one or more points, a first-order residual, and two The method of clause 20B, comprising the step of receiving at least one of the following residuals:

Clause 25B: Generating a prediction of the one or more points comprises generating a prediction of the one or more points using inter prediction; The method of clause 20B, wherein generating a prediction of points in the reference point cloud frame includes performing motion estimation with the one or more points to determine a similar set of points in the reference point cloud frame.

Clause 26B: The step of generating one or more point predictions includes the step of generating one or more point predictions using inter prediction; The method of clause 20B, wherein generating a prediction of the points in the reference point cloud frame includes performing motion compensation to predict the one or more points based on the set of points in the reference point cloud frame.

Clause 27B: The method of Clause 26B, wherein the step of performing motion compensation comprises applying a motion vector to a set of points in the reference point cloud frame to determine a prediction of the one or more points.

Clause 28B: The method of Clause 27B, further comprising predicting a motion vector based on the motion vectors of the spatiotemporal neighboring inter-octree leaves.

Clause 29B: A computer-readable storage medium storing instructions that, when executed by one or more processors, cause an octree-based partitioning of a space containing a point cloud into one or more processors. determining an octree, the leaf nodes of the octree containing one or more points of the point cloud, and each of the one or more points in the leaf nodes Instructions cause one or more processors to directly decode the position of each of the one or more points in a leaf node. and determining one or more points based on the prediction.

Clause 30B: Apparatus for determining an octree, defining an octree-based partitioning of a space containing point clouds, wherein a leaf node of the octree is one of the point clouds. or a plurality of points, and means for directly decoding the position of each of the one or more points at the leaf node, directly decoding the position of each of the one or more points at the leaf node. An apparatus, wherein the means for generating a prediction of one or more points and means for determining one or more points based on the prediction.

Clause 1C: A device for decoding a bitstream containing point cloud data, comprising a memory for storing the point cloud data and one or more processors coupled to the memory and implemented in a circuit. , the one or more processors determining an octree defining an octree-based partition of a space containing the point cloud, wherein a leaf node of the octree is one or more of the point clouds; including a plurality of points, and configured to directly decode the position of each of the one or more points at the leaf node, and to directly decode the position of each of the one or more points at the leaf node. For decoding, the one or more processors are further configured to generate a prediction of the one or more points and, based on the prediction, determine the one or more points. ,device.

Clause 2C: In order to directly decode the position of each of the one or more points in a leaf node, the one or more processors receive a flag, the first value for the flag being: indicating that the predictions for the one or more points are generated by intra-prediction, and a second value for the flag indicates that the predictions for the one or more points are generated by inter-prediction; and the device of clause 1C, further configured to decode the one or more points using intra-prediction or inter-prediction based on the value of the flag.

Clause 3C: The device of Clause 1C or 2C, wherein the one or more processors are further configured to receive octree leaf volumes specifying volumes of leaf nodes in a bitstream that includes a point cloud.

Clause 4C: The one or more processors are further configured to generate the one or more point predictions using intra prediction, and the one or more processors are further configured to generate one or more point predictions using intra prediction. the one or more processors are further configured to determine a local prediction tree for the one or more points to use the prediction to generate a prediction for the one or more points; Any device from clauses 1C to 3C.

Clause 5C: In order to determine the one or more points based on the prediction, the one or more processors determine a prediction mode for each of the one or more points in the bitstream comprising the point cloud; The device of any of clauses 1C to 4C, further configured to receive at least one of a first-order residual, and a second-order residual.

Clause 6C: The one or more processors are further configured to generate the one or more point predictions using inter prediction, and the one or more processors are further configured to generate one or more point predictions using To generate a prediction of one or more points using prediction, one or more processors use one or more processors to determine a similar set of points in a reference point cloud frame. A device according to any of clauses 1C to 5C, further configured to perform motion estimation using points.

Clause 7C: The one or more processors are further configured to generate one or more point predictions using inter prediction, and the one or more processors are further configured to generate one or more point predictions using inter prediction; Using prediction, one or more processors predict one or more points based on a set of points in a reference point cloud frame to generate a prediction of one or more points. A device according to any of Clauses 1C to 5C, further configured to perform motion compensation.

Clause 8C: To perform motion compensation, one or more processors apply a motion vector to a set of points in a reference point cloud frame to determine a prediction of the one or more points. Further consisting of the devices of Article 7C.

Clause 9C: The device of Clause 8C, wherein the one or more processors are further configured to predict motion vectors based on motion vectors of spatiotemporally neighboring inter-octtree leaves.

Clause 10C: The device of any of Clauses 1C to 9C, wherein the one or more processors are further configured to reconstruct a point cloud from point cloud data.

Clause 11C: The one or more processors, as part of reconstructing the point cloud, calculate one or more of the point cloud based on the directly decoded position of each of the one or more points at the leaf nodes. Clause 10C device configured to determine the position of a plurality of points.

Clause 12C: The device of Clause 11C, wherein the one or more processors are further configured to generate a map of the interior of a building based on the point cloud.

Clause 13C: The device of Clause 11C, wherein the one or more processors are further configured to perform autonomous navigation operations based on point clouds.

Clause 14C: The device of Clause 11C, wherein the one or more processors are further configured to generate computer graphics based on the point cloud.

Clause 15C: The one or more processors are configured to determine the position of a virtual object based on a point cloud and to generate an extended reality (XR) visualization in which the virtual object is at the determined position. Clause 11C devices constituted by.

Clause 16C: A device according to any of Clauses 11C to 15C, further comprising a display for presenting an image based on a point cloud.

Clause 17C: Any device in clauses 1C to 16C, where the device is a mobile phone or a tablet computer.

Article 18C: A device in any of Articles 1C to 16C, where the device is a vehicle.

Clause 19C: A device in any of Clauses 1C to 16C where the device is an extended reality device.

Clause 20C: A method for decoding a point cloud, the step of determining an octree defining an octree-based partition of a space containing the point cloud, the step of determining an octree, wherein leaf nodes of the octree directly decoding the position of each of the one or more points at the leaf node, directly decoding the position of each of the one or more points at the leaf node. A method, wherein the step of decoding includes the steps of generating a prediction of one or more points and determining one or more points based on the prediction.

Clause 21C: Directly decoding the position of each of the one or more points in a leaf node is receiving a flag, the first value for the flag being a prediction of the one or more points. is generated by intra-prediction, and a second value for the flag indicates that the prediction for the one or more points is generated by inter-prediction, based on the value of the flag, and decoding the one or more points using intra-prediction or inter-prediction.

Clause 22C: The method of Clause 20C or 21C, further comprising receiving an octree leaf volume specifying a volume of a leaf node in the bitstream for the point cloud.

Clause 23C: The step of generating a prediction of one or more points includes the step of generating a prediction of one or more points using intra-prediction; The method of any of clauses 20C to 22C, wherein the step of generating a prediction for the points includes the step of determining a local prediction tree for the one or more points.

Clause 24C: Determining one or more points based on the prediction includes, in the bitstream for the point cloud, a prediction mode for each of the one or more points, a first-order residual, and two The method of any of clauses 20C to 23C, comprising the step of receiving at least one of the following residuals:

Clause 25C: Generating a prediction of the one or more points comprises generating a prediction of the one or more points using inter prediction; of clauses 20C to 24C, wherein the step of generating a prediction of the points in the reference point cloud frame comprises the step of performing motion estimation with the one or more points to determine a similar set of points in the reference point cloud frame. Either way.

Clause 26C: The step of generating one or more point predictions includes the step of generating one or more point predictions using inter prediction; Any of clauses 20C to 25C, wherein the step of generating a prediction of the points in the reference point cloud frame comprises the step of performing motion compensation to predict the one or more points based on the set of points in the reference point cloud frame. That method.

Clause 27C: The method of Clause 26C, wherein the step of performing motion compensation comprises applying a motion vector to a set of points in a reference point cloud frame to determine a prediction of the one or more points.

Clause 28C: The method of Clause 27C, further comprising predicting a motion vector based on the motion vectors of the spatiotemporal neighboring inter-octree leaves.

Various examples have been explained. These and other examples are within the scope of the following claims.

100 Encoding and Decoding Systems, Systems
102 Source device
104 Data Source
106 Memory
108 output interface
110 Computer-readable recording medium
112 Storage devices
114 File server
116 Destination Device
118 Data Consumer
120 memory
122 input interface
200G-PCC encoder
202 Coordinate transformation unit
204 Color conversion unit
206 Voxelization unit
208 Attribute transfer unit
210 Eight tree analysis unit
212 Surface approximation analysis unit
214, 226 arithmetic coding unit
216, 312 geometry reconstruction unit
218, 314 RAHT unit
220 LOD generation unit
222 Lifting unit
224 Coefficient quantization unit
300G-PCC decoder
302 Geometry Arithmetic Decoding Unit
304 Attribute arithmetic decoding unit
306 8-ary tree synthesis unit
308 Inverse quantization unit
310 Surface approximation synthesis unit
316 LoD generation unit, LOD generation unit
318 Reverse lifting unit
320 Inverse coordinate transformation unit
322 Reverse color conversion unit
400, 444 8-fold tree
402, 404 nodes
410 Prediction tree
412, 414A, 414B, 418A-418E nodes
420 predicted frames
422 current frame
440, 442 pass
600 distance measuring system
602 Illuminator
604 sensor
606 Light, synchrotron radiation
608, 806, 902 objects
610 Return light
611 Lens
612 images
618 Signal
620 point cloud generator
700, 710 vehicles
702 laser package
704 Laser Beam
706 Pedestrian
708, 808, 904 bitstream
712 Server System
800, 812 users
802 1st location
804, 810 XR headset
814 Second location
900 mobile devices
902 Object
906 remote device

Claims (30)

A device for decoding a bitstream containing point cloud data, the device comprising:
a memory for storing the point cloud data;
one or more processors coupled to the memory and implemented in circuitry, the one or more processors comprising:
determining an octree that defines an octree-based partition of a space containing a point cloud, a leaf node of the octree containing one or more points of the point cloud; , and configured to directly decode the position of each of the one or more points at the leaf node, for directly decoding the position of each of the one or more points at the leaf node. said one or more processors,
The device is further configured to: generate a prediction of the one or more points; and determine the one or more points based on the prediction. the one or more processors to directly decode the position of each of the one or more points in the leaf node;
receiving a flag, a first value for the flag indicating that the prediction of the one or more points is generated by intra prediction; and a second value for the flag. indicates that the prediction of the one or more points is generated by inter-prediction, and based on the value of the flag, the prediction of the one or more points is generated using intra-prediction or inter-prediction. 2. The device of claim 1, further configured to decode points. 2. The device of claim 1, wherein the one or more processors are further configured to receive octree leaf volumes that specify volumes of the leaf nodes in the bitstream that includes the point cloud. to generate the prediction of the one or more points, the one or more processors are further configured to generate the prediction of the one or more points using intra-prediction. ,
to generate the prediction of the one or more points using intra-prediction, the one or more processors determining a local prediction tree for the one or more points; The device of claim 1, further configured. In order to determine the one or more points based on the prediction, the one or more processors, in the bitstream including the point cloud, for each of the one or more points. 2. The device of claim 1, further configured to receive at least one of a prediction mode, a first-order residual, and a second-order residual. to generate the prediction of the one or more points, the one or more processors are further configured to generate the prediction of the one or more points using inter-prediction. ,
to generate the prediction of the one or more points using inter-prediction, the one or more processors to determine a similar set of points in the reference point cloud frame; 2. The device of claim 1, further configured to perform motion estimation using one or more points. to generate the prediction of the one or more points, the one or more processors are further configured to generate the prediction of the one or more points using inter-prediction. ,
to generate the prediction of the one or more points using inter prediction, the one or more processors generate the one or more predictions based on a set of points in a reference point cloud frame; 2. The device of claim 1, further configured to perform motion compensation to predict points of . to perform motion compensation, the one or more processors are configured to apply a motion vector to the set of points in the reference point cloud frame to determine a prediction of the one or more points. 8. The device of claim 7, further configured to. 9. The device of claim 8, wherein the one or more processors are further configured to predict the motion vector based on motion vectors of spatiotemporal neighboring inter-octree leaves. 2. The device of claim 1, wherein the one or more processors are further configured to reconstruct the point cloud from the point cloud data. The one or more processors, as part of reconstructing the point cloud, calculate the point cloud based on the directly decoded position of each of the one or more points at the leaf nodes. 11. The device of claim 10, configured to determine the position of one or more points. 12. The device of claim 11, wherein the one or more processors are further configured to generate a map of a building interior based on the point cloud. 12. The device of claim 11, wherein the one or more processors are further configured to perform autonomous navigation operations based on the point cloud. 12. The device of claim 11, wherein the one or more processors are further configured to generate computer graphics based on the point cloud. said one or more processors,
configured to: determine a position of a virtual object based on the point cloud; and generate an extended reality (XR) visualization in which the virtual object is at the determined position of the virtual object; 12. The device according to claim 11. 12. The device of claim 11, further comprising a display for presenting an image based on the point cloud. 2. The device of claim 1, wherein the device is a mobile phone or a tablet computer. 2. The device of claim 1, wherein the device is a vehicle. 2. The device of claim 1, wherein the device is an extended reality device. A method for decoding a point cloud, the method comprising:
determining an octree that defines an octree-based partition of a space containing the point cloud, a leaf node of the octree containing one or more points of the point cloud; step and
directly decoding the position of each of the one or more points at the leaf node, directly decoding the position of each of the one or more points at the leaf node,
generating a prediction of the one or more points;
determining the one or more points based on the prediction. directly decoding the position of each of the one or more points in the leaf node,
receiving a flag, a first value for the flag indicating that the prediction of the one or more points is generated by intra-prediction, and a second value for the flag; indicates that the prediction of the one or more points is generated by inter-prediction;
21. The method of claim 20, further comprising decoding the one or more points using intra-prediction or inter-prediction based on the value of the flag. 21. The method of claim 20, further comprising receiving octree leaf volumes specifying volumes of the leaf nodes in a bitstream for the point cloud. generating the prediction of the one or more points includes generating the prediction of the one or more points using intra prediction;
21. The method of claim 20, wherein generating the prediction for the one or more points using intra-prediction comprises determining a local prediction tree for the one or more points. . Determining the one or more points based on the prediction includes, in a bitstream for the point cloud, a prediction mode for each of the one or more points, a first-order residual, and 21. The method of claim 20, comprising receiving at least one of the quadratic residuals. generating the prediction of the one or more points comprises generating the prediction of the one or more points using inter-prediction;
generating the prediction of the one or more points using inter-prediction, using the one or more points to determine a similar set of points in a reference point cloud frame; 21. The method of claim 20, comprising performing motion estimation. generating the prediction of the one or more points comprises generating the prediction of the one or more points using inter-prediction;
generating the prediction of the one or more points using inter-prediction to predict the one or more points based on a set of points in a reference point cloud frame; 21. The method of claim 20, comprising the step of performing compensation. 27. The method of claim 26, wherein performing motion compensation comprises applying a motion vector to the set of points in the reference point cloud frame to determine a prediction of the one or more points. Method. 28. The method of claim 27, further comprising predicting the motion vector based on motion vectors of spatiotemporal neighboring inter-octree leaves. a computer-readable storage medium storing instructions, the instructions, when executed by the one or more processors, transmitting the instructions to the one or more processors;
determining an octree that defines an octree-based partition of a space containing a point cloud, a leaf node of the octree containing one or more points of the point cloud; , and causing direct decoding of the position of each of the one or more points at the leaf node, and for directly decoding the position of each of the one or more points at the leaf node, instructions to the one or more processors;
A computer-readable storage medium that causes: generating a prediction of the one or more points; and determining the one or more points based on the prediction. A device,
means for determining an octree, defining an octree-based partition of a space containing a point cloud, wherein a leaf node of the octree includes one or more points of the point cloud; , means and
and means for directly decoding the position of each of the one or more points at the leaf node, the means for directly decoding the position of each of the one or more points at the leaf node. but,
means for generating a prediction of the one or more points;
and means for determining the one or more points based on the prediction.

Images