WO2025217859A1

WO2025217859A1 - Point cloud encoding and decoding method

Info

Publication number: WO2025217859A1
Application number: PCT/CN2024/088459
Authority: WO
Inventors: Fan LIANG; Dian ZUO; Xiaoyong TANG; Zhou ZHI; Tianyu DONG
Original assignee: Shenzhen TCL New Technology Co Ltd
Current assignee: Shenzhen TCL New Technology Co Ltd
Priority date: 2024-04-17
Filing date: 2024-04-17
Publication date: 2025-10-23
Anticipated expiration: 2026-10-17

Abstract

A point cloud encoding method. An encoder performs: downsampling a current point cloud from an original scale to a first downsampled scale; downsampling a reference point cloud from the original scale to the first downsampled scale; obtaining features and coordinates of the downsampled current point cloud and the downsampled reference point cloud; obtaining a bottleneck encoding result of the downsampled current point cloud; and generating an encoded bitstream based on the features and coordinates of the downsampled current point cloud and the downsampled reference point cloud, and the bottleneck encoding result of the downsampled current point cloud.

Description

POINT CLOUD ENCODING AND DECODING METHOD

BACKGROUND OF DISCLOSURE

1. Field of Disclosure

The present disclosure relates to the field of image processing, and more particularly, to a point cloud encoding and decoding method.

2. Description of Related Art

Point clouds (PC) represent three-dimensional (3D) objects and scenes with flexibility and versatility, encompassing 3D point coordinates and associated attributes. Widely utilized in fields like autonomous driving, virtual reality, and robotics, the vast size of point cloud data poses significant challenges for storage and transmission. Consequently, efficient point cloud compression (PCC) methods are urgently needed.

Technical Problem

A significant challenge lies in the extensive data volume of point clouds and the inherent computational complexity of the models used for their processing. Consequently, current methods are limited to retrieving encoded node information within a relatively confined space and constructing context accordingly. This constraint prevents the full exploitation of large-scale context and temporal context inherent in point clouds.

Hence, a point cloud encoding and decoding method for address the challenge are desirable.

SUMMARY

An object of the present disclosure is to propose a encoding and decoding method.

In a first aspect, an embodiment of the disclosure provides a point cloud encoding method for execution by an encoder, comprising:

downsampling a current point cloud from an original scale to a first downsampled scale;

downsampling a reference point cloud from the original scale to the first downsampled scale;

obtaining features and coordinates of the downsampled current point cloud and the downsampled reference point cloud;

obtaining a bottleneck encoding result of the downsampled current point cloud; and

generating an encoded bitstream based on the features and coordinates of the downsampled current point cloud and the downsampled reference point cloud, and the bottleneck encoding result of the downsampled current point cloud.

In a second aspect, an embodiment of the disclosure provides an encoding device comprising a processor configured to call and run a computer program stored in a memory, to cause a device in which the processor is installed to execute the disclosed method.

In a third aspect, an embodiment of the disclosure provides a point cloud decoding method for execution by a decoder, comprising:

receiving an encoded bitstream;

obtaining, based on the encoded bitstream, features and coordinates of a downsampled reference point cloud of the first downsampled scale;

obtaining, based on the encoded bitstream, a bottleneck encoding result of a downsampled current point cloud; and

generating a reconstructed point cloud corresponding to a current point cloud based on the features and coordinates of the downsampled reference point cloud, and the bottleneck encoding result of the downsampled current point cloud, wherein the downsampled current point cloud is a downsampled representation at first downsampled scale of the current point cloud that has an original scale; and generating a decoded bitstream of the current point cloud of the original scale using the reconstructed point cloud.

In a fourth aspect, an embodiment of the disclosure provides a decoding device comprising a processor configured to call and run a computer program stored in a memory, to cause a device in which the chip is installed to execute the disclosed method and any combination of embodiments of the disclosed method.

The disclosed method may be programmed as computer executable instructions stored in non-transitory computer readable medium. The non-transitory computer readable medium, when loaded to a computer, directs a processor of the computer to execute the disclosed method.

The non-transitory computer readable medium may comprise at least one from a group consisting of: a hard disk, a CD-ROM, an optical storage device, a magnetic storage device, a Read Only Memory, a Programmable Read Only Memory, an Erasable Programmable Read Only Memory, EPROM, an Electrically Erasable Programmable Read Only Memory and a Flash memory.

The disclosed method may be programmed as a computer program product that causes a computer to execute the disclosed method.

The disclosed method may be programmed as a computer program that causes a computer to execute the disclosed method.

BRIEF DESCRIPTION OF DRAWINGS

In order to more clearly illustrate the embodiments of the present disclosure or related art, the following figures will be described in the embodiments are briefly introduced. It is obvious that the drawings are merely some embodiments of the present disclosure, a person having ordinary skill in this field may obtain other figures according to these figures without paying the premise.

FIG. 1 illustrates a schematic view of a G-PCC encoder.

FIG. 2 illustrates a schematic view of a G-PCC decoder.

FIG. 3 illustrates a schematic view showing an example of a system.

FIG. 4 illustrates a schematic view showing another example of a system.

FIG. 5 illustrates a schematic view showing an embodiment of the disclosed encoding method.

FIG. 6 illustrates a schematic view showing an embodiment of the disclosed decoding method.

FIG. 7 illustrates a schematic view showing an overall structure of a system for implementing the disclosed method.

FIG. 8 illustrates a schematic view showing an overall solution of the disclosed encoding method.

FIG. 9 illustrates a schematic view showing an overall solution of the disclosed decoding method.

FIG. 10 illustrates a schematic view showing a module of downsampling.

FIG. 11 illustrates a schematic view showing a module of residual encoder.

FIG. 12 illustrates a schematic view showing a module of lossless decoder.

FIG. 13 illustrates a schematic view showing temporal codec units (TCUs) .

FIG. 14 illustrates a schematic view showing an example of parameters of a coordinate generator shared between the encoder and decoder.

FIG. 15 illustrates a schematic view showing an example of convolution on target coordinates to obtain the fused features.

FIG. 16 illustrates a schematic view showing a chip or executing the disclosed method in an encoder.

FIG. 17 illustrates a schematic view showing a chip or executing the disclosed method in a decoder.

DETAILED DESCRIPTION OF EMBODIMENTS

Embodiments of the disclosure are described in detail with the technical matters, structural features, achieved objects, and effects with reference to the accompanying drawings as follows. Specifically, the terminologies in the embodiments of the present disclosure are merely for describing the purpose of the certain embodiment, but not to limit the disclosure.

The Moving Picture Experts Group (MPEG) has proposed two PCC frameworks: Geometry-based PCC (G-PCC) and Video-based PCC (V-PCC) . The G-PCC scheme handles the geometry and attribute information of points directly in 3D space. FIG. 1 illustrates a schematic view of a G-PCC encoder, and FIG. 2 illustrates a schematic view of a G-PCC decoder.

MPEG G-PCC incorporates various advanced point cloud compression techniques, including direct coding modes, context-based entropy coding, intra-frame prediction, and specialized coding modes for structures like planar surfaces. Notably, G-PCC employs a voxel-based approach that leverages coded voxels to predict the current voxel occupancy code using a hand-crafted model. Some deep learning-based methods further enhance this process by utilizing sparse 3D convolution to model the dependency between coded and coding voxels more effectively.

In the V-PCC scheme, the coordinates and attributes of PC are projected onto 2D images, subsequently compressed using established video coding techniques like HEVC and VVC. This approach employs a chunked multi-directional projection to flatten the 3D point cloud into a 2D image, enabling seamless integration with existing video compression algorithms and hardware. In terms of compression performance, MPEG V-PCC excels at lossy compression of dense point clouds, surpassing G-PCC in this regard. Here's a breakdown of the main steps involved:

Patch generation is the initial step in the V-PCC point cloud projection process. It relies on the normal vector of the point cloud, projecting each point onto one of the six faces of the point cloud envelope. The projection direction of each point is then fine-tuned based on the projection directions of neighbouring points of each point. Once projected, the points are grouped into patches using the connected component algorithm.

Patch packing aims to arrange the projected 2D point cloud patches onto a specified-size 2D image. This process iteratively searches for the most uniform and non-overlapping placement of patches, while striving to make the texture of neighbouring patches as similar as possible to enhance the performance of subsequent video compression.

To capture the loss incurred during the projection process, geometry and occupancy images are generated. The Geometry image records the distance from each original point to the projection plane, while the Occupancy image, a binary image, indicates whether a pixel on the projection plane corresponds to an original point.

Attribute image generation involves creating a 2D image that encapsulates the attribute information of the original point cloud. A reconstructed frame comprises a reconstructed point cloud. In this process, the reconstructed point cloud is initially recoloured using the attributes from the original point cloud. Subsequently, these attributes are projected and block-packed. To refine the attribute map, blank areas are filled through multilayer up and down sampling, minimizing high-frequency information and optimizing the prediction performance of image blocks during video compression. Ultimately, the geometric, placeholder, and attribute maps resulting from this process are compressed using established video encoders. The minimal auxiliary information necessary for reconstruction is compressed using arithmetic encoders, culminating in the final transmission stream.

Learning-based Point Cloud Compression (PCC) methods make use of learnable operators and end-to-end training to fully utilize spatial context. Some modern deep learning-based techniques incorporate sparse 3D convolution to more accurately capture the intricate relationships between different voxels in the point cloud. To address the challenges of lossy dynamic PCC, a new convolution-based approach is provided in the disclosure. This method compensates and compresses the dynamic point cloud through a network that effectively leverages temporal information.

The significance of the multi-scale context network is paramount in learning-based static PCC. Numerous studies have shown that it excels at spatial context modelling and possesses a broad receptive field. However, due to computational constraints, current dynamic point cloud (DPC) compression techniques do not incorporate the multi-scale network for extracting nor integrate temporal context.

On the other hand, hierarchical convolutions are employed in lossless PCC to gradually reduce the point cloud into latent features across multiple scales. These features are then entropy encoded at the final scale. This approach ensures efficient and effective compression of the point cloud while preserving its integrity.

To enhance the extraction of temporal context, prior research draws inspiration from video compression and introduces a single-scale DPC compression strategy based on motion estimation. However, despite adopting complex and well-designed motion estimation modules, these works tend to overlook the rich cross-frame context at different scales within the hierarchical PCC structure.

This disclosure presents a voxel-based dynamic PCC method that differs from the current approaches. Instead of relying solely on a single scale, it leverages multi-scale cross-frame context as a prior in a temporal dimension. Prior here refers to a prior distribution, which is a pre-assumption about unknown quantities. Prior comprises pre-assumptions or knowledge about certain attributes or features of the data during the encoding or decoding process. The prior can be based on statistics, experience, or understanding of data characteristics. Prior is crucial in the encoding and decoding process because it can help improve data compression efficiency and decoding accuracy. Furthermore, this prior in a temporal dimension is utilized to compress both the coordinates and feature residuals of the PC within the latent space, enhancing the overall compression efficiency.

Overall solution:

To facilitate understanding of the technical solutions presented in this application’s embodiments, a concise overview of the video encoding and decoding system will be given in the following.

With reference to FIG. 3, a system including a first image processing device 10a and a second image processing device 10b. The first image processing device 10a may operates as an encoding device comprising a processor 75a. The processor 75a comprises an encoder configured to execute the disclosed encoding method. The encoder encodes image or video frames into bit streams, and the processor 75a transmits the bit steam to the processor 75b. The second image processing device 10b may operates as a decoding device comprising a processor 75b. The processor 75b comprises a decoder configured to execute the disclosed decoding method.

Similarly, the second image processing device 10b may operates as an encoding device comprising a processor 75b. The processor 75b comprises an encoder configured to execute the disclosed encoding method. The encoder encodes image or video frames into bit streams, and the processor 75b transmits the bit steam to the processor 75a. The first image processing device 10a may operates as a decoding device comprising a processor 75b. The processor 75b comprises a decoder configured to execute the disclosed decoding method.

In some embodiments of the disclosure, the encoder and the decoder may be implemented into computer readable programs or instructions. A device when executing the programs or instructions performs embodiments of the disclosed method. An example of the electronic device in the description may include one of the device 10a or device 10b. The device 10a may communicate with the device 10b through a direct connection or through a base station and/or a network device entity.

With reference to FIG. 4, the electronic device 10a may communicate with the electronic device 10b through a wireline or a wireless communication connection. One of the electronic device 10a or electronic device 10b may serve as an image encoder and the other may serve as an image decoder. Each of the electronic device 10a and electronic device 10b may include both an image encoder and an image decoder.

An image encoder may be implemented as an inter-frame prediction unit, an intra-frame prediction unit, a chip, an electronic device, or a video encoding system. An image decoder may be implemented as an inter-frame prediction unit, an intra-frame prediction unit, a chip, an electronic device, or a video decoding system. In an embodiment of the disclosure, the inter-frame prediction unit in an image encoder and/or an image decoder performs encoding and decoding method. In an embodiment of the disclosure, the intra-frame prediction unit in an image encoder and/or an image decoder performs encoding and decoding method.

With reference to FIG. 5, an embodiment of a point cloud encoding method can be performed by an image encoder. The point cloud encoding method comprises:

Step B001: downsampling a current point cloud from an original scale to a first downsampled scale;

Step B002: downsampling a reference point cloud from the original scale to the first downsampled scale;

Step B003: obtaining features and coordinates of the downsampled current point cloud and the downsampled reference point cloud;

Step B004: obtaining a bottleneck encoding result of the downsampled current point cloud; and

Step B005: generating an encoded bitstream based on the features and coordinates of the downsampled current point cloud and the downsampled reference point cloud, and the bottleneck encoding result of the downsampled current point cloud.

In some embodiments of the disclosure, the features and coordinates of the downsampled current point cloud and the downsampled reference point cloud cooperatively represent temporal changes between the downsampled reference point cloud of the first downsampled scale and the downsampled current point cloud.

In some embodiments of the disclosure, encoding of the features and coordinates comprises compression using quantization and arithmetic encoding.

The encoder further performs:

obtaining a reconstructed point cloud based on the features and coordinates of the downsampled current point cloud and the features and coordinates of the downsampled reference point cloud, and the bottleneck encoding result of the downsampled current point cloud; and

obtaining residual information based on the reconstructed point cloud and the current point cloud; wherein the residual information is included in the encoded bitstream.

In some embodiments of the disclosure, the encoded bitstream is for reproducing decoded bitstream of the current point cloud of the original scale.

In some embodiments of the disclosure, the encoded bitstream comprises the residual information.

In some embodiments of the disclosure, the reconstructed point cloud has the same scale with the current point cloud of the original scale.

In some embodiments of the disclosure, the reconstructed point cloud comprises an output of a bottleneck coding component.

In some embodiments of the disclosure, the reconstructed point cloud comprises a bottleneck decoded bitstream output by a bottleneck decoder block; the bottleneck decoded bitstream is output by the bottleneck decoder block using a bottleneck encoding result of the downsampled current point cloud.

In some embodiments of the disclosure, the downsampling is performed by a lossy coding component comprising multiple downsampling blocks, each generating a downsampled point cloud at a distinct scale.

In some embodiments of the disclosure, outputs of the downsampling blocks, corresponding to various scales, are combined into a multi-scale point clouds of the current point cloud.

In some embodiments of the disclosure, point clouds of different scales produced by different ones of the downsampling blocks are aggregated, resulting in multi-scale point clouds that retain geometric information of the current point cloud across diverse resolutions.

In some embodiments of the disclosure, the multi-scale point clouds are fed into a temporal encoder block for encoding; the temporal encoder block encodes the multi-scale point clouds to produce multi-scale bitstreams.

In some embodiments of the disclosure, a coordinate generator in the encoder shares parameters with a coordinate generator in a decoder.

With reference to FIG. 6, an embodiment of a point cloud decoding method can be performed by an image decoder. The point cloud decoding method comprises:

Step C001: receiving an encoded bitstream;

Step C002: obtaining, based on the encoded bitstream, features and coordinates of a downsampled reference point cloud of the first downsampled scale;

Step C003: obtaining, based on the encoded bitstream, a bottleneck encoding result of a downsampled current point cloud; and

Step C004: generating a reconstructed point cloud corresponding to a current point cloud based on the features and coordinates of the downsampled reference point cloud, and the bottleneck encoding result of the downsampled current point cloud, wherein the downsampled current point cloud is a downsampled representation at first downsampled scale of the current point cloud that has an original scale; and

Step C005: generating a decoded bitstream of the current point cloud of the original scale using the reconstructed point cloud.

In some embodiments of the disclosure, the decoder generates the reconstructed point cloud corresponding to a current point cloud based on the features and coordinates of the downsampled current point cloud and the features and coordinates of the downsampled reference point cloud, and the bottleneck encoding result of the downsampled current point cloud. The decoder may receive the features and coordinates of the downsampled current point cloud in the encoded bitstream or generate the features and coordinates of the downsampled current point cloud itself.

In some embodiments of the disclosure, the decoder further performs:

segmenting the encoded bitstream into a first bitstream and a second bitstream;

decoding the first bitstream by a lossless decoder, wherein the lossless decoder performs residual decoding and upsampling on the first bitstream;

decoding the second bitstream by a bottleneck decoder block; and

generating a decoded point cloud of a current point cloud using the decoded first bitstream and the decoded second bitstream.

In some embodiments of the disclosure, the second bitstream comprises multi-scale point clouds; the multi-scale point clouds comprise the downsampled current point cloud at the first downsampled scale associated with the current point cloud at the original scale; and the multi-scale point clouds comprise the downsampled reference point cloud at the first downsampled scale associated with a reference point cloud at the original scale.

The decoder further performs:

obtaining, based on the encoded bitstream, residual information; and

updating the reconstructed point cloud based on the residual information.

In some embodiments of the disclosure, a decoded bitstream of the current point cloud of the original scale is generated based on the residual information and the reconstructed point cloud; wherein the residual information is included in the encoded bitstream.

In some embodiments of the disclosure, the reconstructed point cloud has the same scale with the current point cloud of the original scale..

In some embodiments of the disclosure, a coordinate generator in the decoder shares parameters with a coordinate generator in an encoder.

S1: In some embodiments, the arithmetic encoder depicted in FIG. 13 can be realized using either shared entropy models (Shared EM) or non-shared entropy models (non-Shared EM) .

In the context of FIG. 8, "Shared EM" denotes the utilization of a unified entropy model across diverse components or processes within a system that perform the disclosed method. This shared EM aids in minimizing redundancy and fostering consistency during the encoding process. Conversely, "non-Shared EM" signifies that each component or process may employ its own dedicated entropy model, potentially offering more tailored and optimized compression performance albeit with increased model complexity.

S2: In some embodiments, the Feature Generator offers the choice between utilizing shared parameters or operating with non-shared parameters.

The "Feature Generator" is tasked with extracting and generating features from the data for the compression process. When employing shared parameters, the Feature Generator may leverage common feature extraction techniques or representations that are beneficial for both encoding and decoding stages, resulting in a more streamlined and efficient system. Conversely, enabling the Feature Generator to utilize non-shared parameters affords the flexibility to refine the feature extraction process for specific tasks or data types, potentially enhancing compression performance at the expense of additional computational resources.

S3: In some embodiments, the Feature Generator within the system can employ a variety of convolutional operations for feature aggregation, including sparse convolution, 3D convolution, or graph convolution neural networks.

S4: In some embodiments, FIG. 14 illustrates a network architecture serving as a reference compression approach. Achieving the same objective can be attained by adjusting the network parameters within FIG. 9. For instance, one could utilize convolutional kernels of different sizes, incorporate varying numbers of convolutional or MLP (Multi-Layer Perceptron) layers within a single module, or substitute the PC Union Block with a Concatenate operation.

S5: In some embodiments, the downsampling and upsampling blocks in the lossy coding component may rely on a sequence of sparse convolutions or MLP layers.

S6: In some embodiments, the feature fusion block in the TCU may utilize sparse convolution on target coordinates.

S7: In some embodiments, the arithmetic encoder in the bottleneck coding component may be an Range Variant Asymmetric Numeral Systems (rANS) encoder.

S8: The parameters of the entire architecture are determined through training using training datasets. The cross-entropy loss function is formulated based on the actual input occupancy code or number of points and probability distribution estimated by the network, as expressed by the formula:

Among these elements, L represents the cross-entropy loss;

Q represents the estimated probability that coordinates of the ith point is occupied by a voxel;

i is a positive integer variable used to denote an index of a voxel; and

X_i represents context of voxel i.

For example, a coordinate of index i is occupied means a voxel is at a point of the coordinate in the PC. This model can undergo training utilizing the AdamW optimizer, initialized with a learning rate of 0.0005.

Embodiments of the disclosed method are detailed in the following:

Problem Formation

Sparse tensors are utilized in the embodiments of the disclosure to depict point clouds. Therefore, each frame of a point cloud can be represented as a combination of coordinates and features:

in which x represents the whole PC with a number of points N⁰,

c represents coordinates of the PC; and

f represents features of the PC; and

i is a positive integer variable.

A current frame may contain a current point cloud, while a reference frame may contain a reference point cloud. Additionally, a reconstructed frame may include a reconstructed point cloud. Each of these point clouds can undergo downsampling or upsampling to match a specific scale. Here, a point cloud of a scale corresponds to the resolution of that particular scale.

Geometry-based feature extraction modules, such as sparse convolution network (SparseCNN) and MLP, can aggregate original featuresaccording to geometric informationIn the schematic diagram of the hierarchical compression framework, superscripts indicate scale indices of the PC.For example, the point cloud x⁰ at a scale index 0 represents the unencoded original PC. Through a sequence of geometry-based feature extraction modules, the point cloud is progressively encoded from x⁰ as

xⁱ⁺¹=Encⁱ⁺¹ (xⁱ) 3

Encⁱ⁺¹ (xⁱ) represents an encoding operation for encoded PC xⁱ⁺¹ using encoded PC xⁱ as input. This process maps the initial sparse geometric information into the dense latent feature space, enhancing the independence of variables and rendering them more conducive to entropy coding. On the decoder side, the point cloud is progressively decoded from an output of the bottleneck coding component to the scale consistent with x⁰, denoted asx'⁰. Each upsampling process can be formulated as:

x'^i-1=Dec^i-1 (xⁱ) 4

Dec^i-1 (xⁱ) represents a decoding operation for x′^i-1 using xⁱ as input. A reference frame includes a reference point cloud. In an embodiment of the disclosure, the compression process is divided into two parts: lossy coding and lossless coding. In the lossy coding part, temporal context from the reference frame is employed for compression and decompression, which can be expressed as follows:

represents an encoding operation for point cloud xⁱ⁺¹ using point cloud xⁱ andas input; represents a decoding operation for point cloudx′^i-1 using point cloud xⁱ andas input; represents temporal context from the reference frame.

Through well-designed context aggregation modules, the model can leverage rich cross-frame context, significantly enhancing compression performance.

Framework

The overall structure is shown in FIG. 7. A system 100 executes an embodiment of the point cloud encoding and decoding method. The current point cloud is represented as Cur PC, whereas the reference point cloud is designated as Ref PC. Furthermore, the reconstructed point cloud is also labeled as Rec PC. In particular, m, n, and k are positive integers.

FIG. 7 illustrates an architecture of the disclosed method, which comprises a lossy coding component 101, a TCP network component 102, and a bottleneck coding component 103. A temporal encoder 102b comprises a temporal feature encoder 102b-1 and a temporal coordinate encoder 102b-2. A temporal decoder 102c comprises a temporal feature decoder 102c-1 and a temporal coordinate decoder 102c-2 block. The lossy coding component 101 adopts a hierarchical compression structure, composed of several independent compression modules known as Lossy Codec Units (LCUs) . FIG. 7 displays the structure of the Lossy Codec Unit (LCU) , primarily comprising downsampling blocks and an upsampling block composed of a series of sparse convolution and MLP layers.

The TCP network component 102 comprises a downsampling module 102a, a temporal feature encoder 102b-1, a temporal feature decoder 102c-1, a temporal coordinate encoder 102b-2 and temporal coordinate decoder 102c-2.

The bottleneck coding component 103 comprises a bottleneck encoder block and a bottleneck decoder block.

Encoding Process:

FIG. 8 depicts a comprehensive encoding process of the disclosed method.

Point Cloud Transformation:

Initially, the current point cloud undergoes transformation processing in the Lossy Encoder. The transformation processing in the Lossy Encoder involves the application of various lossy techniques to reduce the size of point clouds while endeavouring to preserve its essential geometric features.

Lossy Downsampling:

After transformation, the point cloud undergoes multiple rounds of lossy downsampling. This process decreases the spatial resolution of the point cloud, further reducing its size while retaining its overall structure.

Bottleneck Encoder Block:

The downsampled point cloud is then directed to the Bottleneck Encoder Block. The Bottleneck Encoder Block performs lossless entropy encoding on the downsampled point cloud to ensure that the compressed data can be accurately reconstructed during decoding by the Bottleneck decoder Block. The encoded downsampled point forms Bitstream_2. This process generates Bitstream_2 at the Bottleneck Encoder Block, containing crucial information of the point cloud. The Bottleneck decoder Block decodes the encoded point cloud to produce decoded point cloud as input to the residual encoder (shown as Res Encoder in FIG. 8) .

Residual Compression:

Once Bitstream_2 is obtained and decoded, it enters the residual encoder (shown as Res Encoder in FIG. 8) . The residual encoder takes the downsampled current point cloud and decoded point cloud (decoded from Bitstream_2) the as input. The residual encoder and calculates differences between downsampled current point cloud (i.e., original point cloud) and decoded point cloud (decoded from Bitstream_2) and encodes the differences as encoded residuals. The residual encoder produces Bitstream_1 that includes the encoded residuals.

A downsampling module (e.g., downsampling module 102a) generates downsampled reference point cloud to the lossless encoder (e.g., temporal encoder 102b) of TCP network component 102. The downsampled reference point cloud provides temporal information for being referenced by the downsampling block (s) and the residual encoder in the lossless encoder of TCP network component 102.

Bitstream Concatenation:

The two segments of the Bitstream_1 and Bitstream_2, are combined to create output bitstream. This output bitstream represents a compressed representation of the original point cloud (the current point cloud) , with Bitstream_1 containing residual information and Bitstream_2 providing essential structural details. The structural details include features and coordinates of the reconstructed point cloud.

Temporal Context Aggregation:

The encoding and decoding method involves temporal context, such as video sequences or sequences of point clouds. The reference point cloud also undergoes multiple downsampling stages. The reference point cloud that is downsampled to the same scale as the current point cloud is then fed into the residual encoder to enhance residual compression, ensuring efficient capture and encoding of temporal changes.

The encoding process aims to balance the trade-off between compression ratio and the quality of reconstructed point clouds. By integrating both lossy and lossless compression techniques, the method guarantees efficient compression and accurate reconstruction of point clouds, even when dealing with high-resolution 3D data.

Decoding Process:

FIG. 9 illustrates the overall decoding process of the disclosed method.

Bitstream Segmentation:

The final output bitstream received by a decoder that performs the decoding process is initially segmented into two parts: Bitstream_11 and Bitstream_21. Bitstream_11 represents Bitstream_1 output by the residual encoder, and Bitstream_21 represents Bitstream_2 output by the Bottleneck decoder Block. These segments represent the outcomes of distinct compression strategies.

Bitstream Decoding:

Bitstream_21 undergoes decoding in the bottleneck decoder block for lossless decompression, reconstructing a representation of the point cloud at the bottleneck layer. As Bitstream_21 is encoded using lossless compression, the original bottleneck layer information of the point cloud can be fully recovered.

Point Cloud Reconstruction:

Bitstream_11 is decoded in the lossless decoder (e.g., temporal decoder 102c) , corresponding to the lossless encoder (e.g., temporal encoder 102b) of TCP network component 102 in the encoding process. The decoding in the lossless decoder comprises residual decoding and upsampling. The lossless decoder performs residual decoding and upsampling on the Bitstream_11. The residual decoder utilizes the decoded bitstream to reconstruct the residual information, depicting the geometric changes from the downsampled point cloud to the original resolution point cloud.

The decoded point cloud output by the bottleneck decoder block (in bottleneck layer) and residual information are jointly utilized by the lossless decoder and the lossy decoder to reconstruct the original point cloud. In the decoding process, temporal context aggregation involves processing the reference point cloud as well. After downsampling the reference point cloud at multiple scales, the downsampled the reference point cloud is integrated with the output of the residual encoder at the same scale to enhance the compression effect of the residual information.

Through these steps, a reconstructed point cloud closely resembling the original input is achieved. While this reconstructed point cloud preserves most of the geometric information of the original point cloud, some information may be lost due to the lossy compression employed during encoding.

FIG. 10, FIG. 11, and FIG. 12 depicts the modules DownSampling, residual encoder, and lossless decoder.

Downsampling Process:

With reference to FIG. 10, an example of the downsampling module102a is detailed. The downsampling module102a comprises several downsampling blocks, each producing a downsampled point cloud at a specific scale. The outputs from downsampling blocks corresponding to different scales are concatenated into a unified representation for output. The unified representation forms a comprehensive multi-scale representation of the point cloud. The individual scale point clouds generated by the different downsampling blocks are aggregated, forming a multi-scale point cloud that preserves the geometric information of the original point cloud across various resolutions. A scale of the point cloud corresponds to a resolution of the point cloud.

Residual Encoder:

With reference to FIG. 11, an example of the residual encoder a is detailed. A current frame comprises a current point cloud. Following the acquisition of multi-scale point cloud of reference frames and multi-scale point cloud of the current frame as input data, the residual encoder divides the input data into multiple point clouds at different scales. The multiple point clouds at different scales are referred to as multi-scale point clouds. These point clouds are then sequentially fed into the Temporal Encoder 102b for encoding. The Temporal Encoder 102b encodes the multi-scale point clouds to produce multi-scale bitstreams. The encoding in the Temporal Encoder 102b comprises capturing temporal changes and compressing the information of the captured temporal changes. The Temporal Encoder 102b captures and compresses temporal changes between multi-scale point clouds of the reference frame and the multi-scale point clouds of the reconstructed frame. The compressed temporal changes are combined with the multi-scale bitstreams to form resulting multi-scale bitstreams. The resulting multi-scale bitstreams are concatenated to form Bitstream_1, representing the compressed data of the temporal context and the multi-scale geometric information of the point clouds.

Lossless Decoder:

With reference to FIG. 12, an example of the lossless decoder is detailed. In the decoding process, the lossless decoder (e.g., temporal decoder 102c) parses the Bitstream_11 to extract the multi-scale bitstreams and separates the compressed information (i.e., the multi-scale bitstreams) into its component parts, each corresponding to a specific scale. That is, each component part of the multi-scale bitstreams comprises multi-scale bitstreams at a scale. The multi-scale bitstreams at a scale represents point clouds at the scale. These multi-scale bitstreams are then combined with reference point clouds of the corresponding scale and the reconstructed point clouds of the corresponding scale. Sequentially, the result of the combining are input into the Temporal Decoder 102c for information decoding. The Temporal Decoder 102c reconstructs the point cloud from the inputs, progressively refining the geometry of the point cloud at each scale. This ensures accurate reflection of temporal changes in the reconstruction. The final output from the Temporal Decoder 102c is a losslessly reconstructed frame, closely resembling the original point cloud while preserving the integrity of its geometric information across all scales.

Lossy Encoder &Decoder:

The lossy coding component 101 adopts a hierarchical compression structure, comprising multiple independent compression modules known as the Lossy Codec Unit (LCU) . As depicted in FIG. 7, the Lossy Codec Unit (LCU) primarily comprises downsampling blocks and upsampling blocks composed of a stack of sparse convolution and MLP layers.

In bottleneck coding, features and coordinates are compressed sequentially through quantization and arithmetic encoding.

Temporal Codec Unit (TCU) 102d comprises the Temporal Encoder 102b and the Temporal Decoder 102c. To efficiently aggregate multi-scale temporal and spatial contexts, the TCU 102d leverages temporal context from the reference frame. The temporal context from the reference frame along with spatial context from the lower scale aids in encoding (i.e., compressing) and decoding (i.e., reconstructing) features and coordinates of the point cloud at a current scale. The TCU 102d performs compression of both features and coordinates of the point cloud. During reconstruction, coordinates are prioritized and reconstructed first, followed by features. The original point cloud undergoes a series of sparse convolutional downsampling and MLP modules to obtain a single feature point. This feature point is then subjected to lossless compression in the bottleneck coding component 103.

This feature point is obtained by down-sampling the original point cloud and extracting features through a series of downsampling blocks. The point cloud (PC) inputted into the bottleneck encoder block consists of feature points obtained through continuous downsampling. Unlike the original points that solely contain geometric information, these downscaled points include feature information. Therefore, they are referred to as feature points. In an embodiment, the PC entering the bottleneck encoder block comprises only one feature point.

The detailed Temporal Codec Unit (TCU) 102d is depicted in FIG. 13. In an embodiment, the TCU 102d uses SparseCNN. With the SparseCNN characteristics, precise PC features are only attainable after acquiring accurate coordinates. In the lossless coordinate encoding process, a trainable coordinate generator module integrate cross-frame context (i.e., temporal context) from the reference frame and spatial context from the lower scale. In the lossless coordinate decoding process, a trainable coordinate generator module to integrate cross-frame context (i.e., temporal context) from the reference frame and spatial context from the lower scale.

For encoding and decoding consistency, the parameters of the coordinate generator are shared between the encoder and decoder, as depicted in FIG. 14. In an embodiment, prior to extracting temporal context, SparseCNN is applied to the reference frame to enhance the receptive field. To leverage correlations between the reference and current point clouds, a Minkowski union block is employed to merge these two frames into a new point cloudFor context fusion, in an embodiment, a convolution is performed on target coordinates to obtain the fused features, which is illustrated in FIG. 15.

For consistency between the coordinates in the encoder and decoder at each scale, it's essential to compress the residuals between the predicted coordinatesand the ground truth coordinates cⁱ. To achieve this, an embodiment of the disclosed method utilizes the mask compression strategy. This mask is then subjected to lossless compression using an arithmetic encoder. In the decoder, the predicted coordinatesare pruned based on the mask to obtain the reconstructed coordinates cⁱ. Regarding the feature encoding module, akin to the coordinate encoding method, a feature generator performs feature reconstruction. Following this, a residual block is employed to compress the residual features.

Training Setup

Training Datasets:

Our model is trained and evaluated on multiple datasets, including dense point clouds (e.g., dense point clouds in MPEG 8i) . In one embodiment, model training may utilize sequences such as Soldier10, Longdress10, Andrew10, David10, and Sarah10. In one embodiment, model training may utilize the official MPEG training dataset, also known as 8i Voxelized Full Bodies (8iVFB) . This dataset is specifically designed to capture the surface characteristics of the human body, featuring four dynamic point cloud sequences: long dress, loot, red&black, and soldier. Each sequence comprises a substantial collection of 300 frames of point clouds, offering a diverse and comprehensive representation of dynamic human movements. This dataset serves as a valuable resource for training models aimed at capturing the nuances of various body motions and shapes, thereby enhancing the robustness and generalization capabilities of our approach.

Training Procedure:

Multiple training iterations are conducted with varying values of λ to cover a range of bitrates effectively. The λ values can be empirically set to 0.6, 0.8, 1.0, 1.4, and 2.0 to evenly distribute the corresponding bitrates as much as possible. All models are trained from scratch for 50 epochs with a batch size of 1. In one embodiment, model training may employ the AdamW optimizer with a learning rate of 0.0005, which decays by a factor of 0.3 every 12 epochs. To ensure training stability, in one embodiment, model training may set a slightly lower initial learning rate of 0.0001 for the learnable parameters in entropy models. Additionally, during the initial 1,000 iterations, λ is manually set to 0.01 to facilitate a warm-up phase.

With reference to FIG. 16, the embodiment of the disclosure also provides a chip 70 that may correspond to an encoder in the embodiments of the disclosure. The chip 70 may implement a corresponding process realized by the encoder in various methods of the embodiments of the disclosure. The chip 70 includes a processor 71, and the processor 71 may call and run a computer program from memory to implement the methods in the embodiments of the present application.

Optionally, the chip 70 may also include a memory 72. In particular, the processor 71 may call and run the computer program from the memory 72 to implement the methods in the embodiments of the present application.

Moreover, the memory 72 may be a separate device from the processor 71 or may be integrated into the processor 71.

Optionally, the chip 70 may further include an input interface 73. Note that the processor 71 may control the input interface 73 to communicate with other devices or chips, specifically, to obtain messages or data sent by other devices or chips.

Optionally, the chip 70 may further include an output interface 74. Note that the processor 71 may control the output interface 74 to communicate with other devices or chips, specifically, to output messages or data to other devices or chips.

With reference to FIG. 17, the embodiment of the disclosure also provides another chip 80 that may correspond to a decoder in the description, and the chip 80 may implement the corresponding processes implemented by the decoder in the various methods of the embodiments of the disclosure. The chip 80 includes a processor 81, and the processor 81 may call and run a computer program from the memory 82 to implement the methods in the embodiments of the present application.

Optionally, the chip 80 may further include a memory 82. In particular, the processor 81 may call and run the computer program from the memory 82 to implement the methods in the embodiments of the present application.

Wherein the memory 82 may be a separate device from the processor 81 or may be integrated into the processor 81.

Optionally, the chip 80 may also include an input interface 83. In particular, the processor 81 may control the input interface 83 to communicate with other devices or chips, specifically, to obtain messages or data sent by other devices or chips.

Optionally, the chip may further include an output interface 84. In particular, the processor 81 may control the output interface 84 to communicate with other devices or chips, specifically, to output messages or data to other devices or chips.

The present invention introduces a voxel-based compression framework designed to capture geometrical and temporal contexts adaptively and independently. For extracting temporal context, at least one embodiment of the disclosure provides a multi-scale temporal context extraction framework utilizing the Temporal Contextual Pyramid (TCP) . This approach leverages cross-frame information from dynamic point clouds to maximize efficiency.

Useful Effect:

At least one embodiment of the disclosure provides a dynamic point cloud compression framework that efficiently extracts cross-frame context through the Temporal Contextual Pyramid (TCP) . Unlike existing convolution-based context models, this framework fully utilizes both the geometric and temporal contexts of dynamic point clouds.

The framework incorporates a TCP network for lossless coding, effectively aggregating multi-scale cross-frame context. To prevent excessive time complexity, a Temporal Codec Unit (TCU) module is introduced, which relies on sparse convolution for cross-frame contextual aggregation. By integrating the TCP Network with a lossy coding module, an efficient end-to-end lossy Point Cloud Compression (PCC) framework is obtained.

The intra-frame prediction model introduced in this invention enables the point cloud encoding system to achieve superior prediction results, ultimately enhancing compression efficiency.

While the present disclosure has been described in connection with what is considered the most practical and preferred embodiments, it is understood that the present disclosure is not limited to the disclosed embodiments but is intended to cover various arrangements made without departing from the scope of the broadest interpretation of the appended claims.

Claims

A point cloud encoding method for execution by an encoder, comprising:

downsampling a current point cloud from an original scale to a first downsampled scale;

downsampling a reference point cloud from the original scale to the first downsampled scale;

obtaining features and coordinates of the downsampled current point cloud and the downsampled reference point cloud;

obtaining a bottleneck encoding result of the downsampled current point cloud; and

generating an encoded bitstream based on the features and coordinates of the downsampled current point cloud and the downsampled reference point cloud, and the bottleneck encoding result of the downsampled current point cloud.
The method of claim 1, wherein the features and coordinates of the downsampled current point cloud and the downsampled reference point cloud cooperatively represent temporal changes between the downsampled reference point cloud of the first downsampled scale and the downsampled current point cloud.
The method of claim 1, wherein encoding of the features and coordinates comprises compression using quantization and arithmetic encoding.
The method of claim 1, further comprising:

obtaining a reconstructed point cloud based on the features and coordinates of the downsampled current point cloud and the features and coordinates of the downsampled reference point cloud, and the bottleneck encoding result of the downsampled current point cloud; and

obtaining residual information based on the reconstructed point cloud and the current point cloud;

wherein the residual information is included in the encoded bitstream.
The method of claim 4, wherein the encoded bitstream is for reproducing decoded bitstream of the current point cloud of the original scale.
The method of claim 4, wherein the reconstructed point cloud has the same scale with the current point cloud of the original scale.
The method of claim 4, wherein the reconstructed point cloud comprises an output of a bottleneck coding component.
The method of claim 7, wherein the reconstructed point cloud comprises a bottleneck decoded bitstream output by a bottleneck decoder block;

the bottleneck decoded bitstream is output by the bottleneck decoder block using a bottleneck encoding result of the downsampled current point cloud.
The method of claim 1, wherein the downsampling is performed by a lossy coding component comprising multiple downsampling blocks, each generating a downsampled point cloud at a distinct scale.
The method of claim 9, wherein outputs of the downsampling blocks, corresponding to various scales, are combined into a multi-scale point clouds of the current point cloud.
The method of claim 9, wherein point clouds of different scales produced by different ones of the downsampling blocks are aggregated, resulting in multi-scale point clouds that retain geometric information of the current point cloud across diverse resolutions.
The method of claim 11, wherein the multi-scale point clouds are fed into a temporal encoder block for encoding;

the temporal encoder block encodes the multi-scale point clouds to produce multi-scale bitstreams.
The method of claim 1, wherein a coordinate generator in the encoder shares parameters with a coordinate generator in a decoder.
A chip, comprising:

a processor, configured to call and run a computer program stored in a memory, to cause a device in which the chip is installed to execute the method of any of claims 1 to 13.
A computer-readable storage medium, in which a computer program is stored, wherein the computer program causes a computer to execute the method of any of claims 1 to 13.
A computer program product, comprising a computer program, wherein the computer program causes a computer to execute the method of any of claims 1 to 13.
A computer program, wherein the computer program causes a computer to execute the method of any of claims 1 to 13.
An encoding device comprising:

a processor configured to execute the method of any of claims 1 to 13.
A point cloud decoding method for execution by a decoder, comprising:

receiving an encoded bitstream;

obtaining, based on the encoded bitstream, features and coordinates of a downsampled reference point cloud of the first downsampled scale;

obtaining, based on the encoded bitstream, a bottleneck encoding result of a downsampled current point cloud; and

generating a reconstructed point cloud corresponding to a current point cloud based on the features and coordinates of the downsampled reference point cloud, and the bottleneck encoding result of the downsampled current point cloud, wherein the downsampled current point cloud is a downsampled representation at first downsampled scale of the current point cloud that has an original scale; and

generating a decoded bitstream of the current point cloud of the original scale using the reconstructed point cloud.
The method of claim 19, further comprising:

segmenting the encoded bitstream into a first bitstream and a second bitstream;

decoding the first bitstream by a lossless decoder, wherein the lossless decoder performs residual decoding and upsampling on the first bitstream;

decoding the second bitstream by a bottleneck decoder block; and

generating a decoded point cloud of a current point cloud using the decoded first bitstream and the decoded second bitstream.
The method of claim 20, wherein the second bitstream comprises multi-scale point clouds;

the multi-scale point clouds comprise the downsampled current point cloud at the first downsampled scale associated with the current point cloud at the original scale; and

the multi-scale point clouds comprise the downsampled reference point cloud at the first downsampled scale associated with a reference point cloud at the original scale.
The method of claim 19, further comprising:

obtaining, based on the encoded bitstream, residual information; and

updating the reconstructed point cloud based on the residual information.
The method of claim 22, wherein a decoded bitstream of the current point cloud of the original scale is generated based on the residual information and the reconstructed point cloud;

wherein the residual information is included in the encoded bitstream.
The method of claim 22, wherein the reconstructed point cloud has the same scale with the current point cloud of the original scale..
The method of claim 22, wherein the reconstructed point cloud comprises an output of a bottleneck coding component.
The method of claim 25, wherein the reconstructed point cloud comprises a bottleneck decoded bitstream output by a bottleneck decoder block; .

the bottleneck decoded bitstream is output by the bottleneck decoder block using a bottleneck encoding result of the downsampled current point cloud.
The method of claim 19, wherein a coordinate generator in the decoder shares parameters with a coordinate generator in an encoder.
A chip, comprising:

a processor, configured to call and run a computer program stored in a memory, to cause a device in which the chip is installed to execute the method of any of claims 20 to 27.
A computer-readable storage medium, in which a computer program is stored, wherein the computer program causes a computer to execute the method of any of claims 20 to 27.
A computer program product, comprising a computer program, wherein the computer program causes a computer to execute the method of any of claims 20 to 27.
A computer program, wherein the computer program causes a computer to execute the method of any of claims 20 to 27.
A decoding device comprising:

a processor configured to execute the method of any of claims 20 to 27.