RU2778377C1 - Method and apparatus for encoding a point cloud - Google Patents

Abstract

FIELD: encoding.

SUBSTANCE: invention relates to a method, a machine-readable storage medium, and an apparatus for encoding a point cloud. In the method, a processor is used to determine one or more source points in the point cloud, linked with the reconstructed position, wherein the positions of one or more source points are reconstructed in accordance with the quantisation of geometry into the reconstructed position; the processor is used to determine the attribute value for the reconstructed position, based on the attribute information, for said one or more source points; and the processor is used to encode the texture of the point cloud using the reconstructed position with the above determined attribute value.

EFFECT: increase in the speed of encoding a point cloud.

12 cl, 13 dwg

Description

[0001] The present application claims priority under U.S. Patent Application No. 17/064,029 "Method and Apparatus for Encoding a Point Cloud", filed Oct. 6, 2020, which claimed priority under Provisional U.S. Patent Application No. 62/942,536 " Fast recoloring for point cloud encoding”, filed December 2, 2019.

FIELD OF TECHNOLOGY

[0002] The present invention describes embodiments of the invention generally related to point cloud coding.

BACKGROUND OF THE INVENTION

[0003] The description of the prior art is given here to represent the overall context of the invention. The work of the authors of the invention, to the extent that it is described in this section, as well as aspects of the description that cannot be qualified as state of the art at the time of filing, neither expressly nor impliedly recognized as the state of the art for the present invention.

[0004] Various technologies have been developed for obtaining a representation of the world, such as objects in the world, the environment in the world, and the like, in three-dimensional (3D) space. 3D representations of the world can facilitate more immersive interaction and connection. Point clouds can be used as a 3D representation of the world. A point cloud is a set of points in 3D space, each of which has attributes associated with it, such as color, material properties, texture information, intensity attributes, reflectivity attributes, motion-related attributes, modality attributes, and various other attributes. Such point clouds can contain large amounts of data, and their storage and transmission can be costly and time consuming.

SUMMARY OF THE INVENTION

[0005] Aspects of the invention provide methods and apparatus for compressing and decompressing a point cloud. In some examples, the device for compressing/decompressing a point cloud contains a processing circuit. In some embodiments, the processing scheme defines one or more seed points in the point cloud that are associated with the reconstructed position. The positions of one or more of the original points may be reconstructed in accordance with the quantization of the geometry into the reconstructed position. The processing circuit then determines an attribute value for the reconstructed position based on the attribute information of one or more source points, and texture-encodes the point cloud using the reconstructed position having said attribute value to be determined.

[0006] In some embodiments of the invention, the device includes a memory storing a data structure that associates one or more original points with a reconstructed position. The data structure is accessed based on the reconstructed position to retrieve one or more seed points associated with the reconstructed position.

[0007] In some embodiments, the processing circuit performs an octree-based partitioning that partitions the point cloud space into voxels in geometric quantization and associates one or more seed points that are located in the voxel with a reconstructed position to represent the voxel. For example, a link relationship is stored using an appropriate data structure.

[0008] In some embodiments of the invention, the processing circuit calculates the average value of the attributes of the set of seed points as said determined attribute value for the reconstructed position in response to the fact that the set of seed points are associated with the reconstructed position. In some examples, the processing circuit calculates a weighted average of the attributes of the set of seed points as said determined attribute value for the reconstructed position. In one example, the processing scheme assigns a weight to an attribute value of a seed point in a plurality of seed points based on the inverse distance between the seed point and the reconstructed position.

[0009] In some embodiments, the processing circuit assigns a particular attribute value of the closest point in a set of seed points as said determined attribute value for a reconstructed point in response to the set of seed points being associated with a reconstructed position. In the example, the processing circuit selects the median attribute value among the attribute values for the nearest points in response to the fact that the nearest points in the set of source points have the same shortest distance to the reconstructed position. In another example, the processing circuit calculates an attribute average of attribute values for the nearest points in response to the fact that the nearest points in the set of source points have the same shortest distance to the reconstructed position.

[0010] Aspects of the invention also provide a computer-readable medium storing instructions that, when executed by a point cloud encoding/decoding computer, causes the computer to perform any or combination of point cloud encoding/decoding methods.

BRIEF DESCRIPTION OF THE DRAWINGS

[0011] Other features, nature and various advantages of the present invention will become apparent from the following detailed description and the accompanying drawings.

[0012] FIG. 1 is a schematic illustration of a simplified block diagram of a communication system in accordance with an embodiment of the invention;

[0013] FIG. 2 is a schematic illustration of a simplified block diagram of a data streaming system in accordance with an embodiment of the invention;

[0014] FIG. 3 shows a block diagram of an encoder for encoding point cloud frames according to some embodiments of the invention;

[0015] FIG. 4 shows a block diagram of a decoder for decoding a compressed bitstream corresponding to point cloud frames, in accordance with some embodiments of the invention;

[0016] FIG. 5 is a schematic illustration of a simplified block diagram of a video decoder in accordance with an embodiment of the invention;

[0017] FIG. 6 is a schematic illustration of a simplified block diagram of a video encoder in accordance with an embodiment of the invention;

[0018] FIG. 7 shows a block diagram of an encoder for encoding point cloud frames according to some embodiments of the invention;

[0019] FIG. 8 shows a block diagram of a decoder for decoding a compressed bitstream corresponding to point cloud frames, according to some embodiments of the invention;

[0020] FIG. 9 shows a diagram illustrating cube partitioning using an octree-based partitioning technique in accordance with some embodiments of the present invention.

[0021] FIG. 10 shows an example of an octree-based partition and an octree structure corresponding to an octree-based partition, according to some embodiments of the invention.

[0022] FIG. 11 shows a diagram illustrating production methods according to an embodiment of the invention.

[0023] FIG. 12 shows a flowchart describing an example of a method in accordance with some embodiments of the invention.

[0024] FIG. 13 is a schematic illustration of a computer system in accordance with an embodiment of the invention.

DETAILED DESCRIPTION OF EMBODIMENTS

[0025] Aspects of the invention provide point cloud coding (PCC) methods. The PCC may be performed according to various schemes such as a geometry based scheme called G-PCC, a video coding based scheme called V-PCC, and the like. According to some aspects of the invention, the G-PCC scheme encodes 3D geometry directly and is a purely geometric approach that has little to do with video coding, while the V-PCC scheme relies heavily on video coding. For example, the V-PCC scheme may convert a 3D cloud point to a 2D grid (image) pixel. The V-PCC scheme can use universal video codecs to compress the point cloud. The Moving Picture Experts Group (MPEG) is working on the G-PCC standard and the V-PCC standard, which respectively use the G-PCC scheme and the V-PCC scheme.

[0026] Aspects of the invention provide recoloring techniques for a PCC scheme such as the G-PCC scheme and the V-PCC scheme. Repainting techniques can increase the speed of point cloud encoding.

[0027] Point clouds can be widely used in many applications. For example, point clouds can be used in autonomous vehicles for object detection and localization, in geographic information systems (GIS) for mapping, and in cultural heritage for visualization and archiving of cultural heritage objects and collections, etc.

[0028] Hereinafter, a point cloud can typically refer to a plurality of points in 3D space, where each point can be defined by position information with associated attributes, e.g., color, material properties, texture information, intensity attributes, reflectivity attributes, attributes , associated with motion, modality attributes, and various other attributes. Point clouds can be used to reconstruct a scene object as a composition of such points. The points can be acquired with multiple cameras, depth sensors and lidar in a variety of setups and can be made up of thousands or billions of points to realistically represent rendered scenes. A patch can usually refer to a continuous subset of a surface described by a point cloud. In one example, the patch includes points with surface normal vectors that deviate from each other by less than a threshold amount.

[0029] Compression techniques can reduce the amount of data required to represent a point for faster data transfer and reduced memory footprint. Thus, techniques are needed to compress lossy point clouds for use in real-time communication and virtual reality with six degrees of freedom (6DoF, 6 Degrees of Freedom). In addition, lossless point cloud compression technology is required in the context of dynamic mapping for autonomous driving, cultural heritage applications, and the like.

[0030] According to an aspect of the invention, the main philosophy of V-PCC is to use existing video codecs to compress the geometry, occupancy, and texture of a dynamic point cloud as three separate video sequences. Additional metadata required to interpret the three video sequences is compressed separately. A small part of the overall bitstream is metadata, which can be efficiently encoded/decoded by software implementation. Most of the information is processed by the video codec.

[0031] FIG. 1 illustrates a simplified block diagram of a communication system (100) according to an embodiment of the present invention. The communication system (100) includes a plurality of terminal devices that can communicate with each other, for example, via a network (150). For example, the communication system (100) includes a pair of terminals (110) and (120) interconnected via a network (150). In FIG. 1, the first pair of terminals (110) and (120) can perform unidirectional transmission of point cloud data. For example, tag (110) may compress a point cloud (eg, points representing structure) that is received by a sensor (105) connected to tag (110). The compressed point cloud may be transmitted, for example, as a bitstream to another terminal (120) via the network (150). The terminal device (120) may receive the compressed point cloud from the network (150), decompress the bitstream to reconstruct the point cloud, and display the reconstructed point cloud appropriately. One-way communication may be inherent in media service applications and the like.

[0032] FIG. 1, terminal devices (110) and (120) can be illustrated as servers and personal computers, but the principles of the present invention are not limited to this. Embodiments of the present invention find application in laptop computers, tablet computers, smartphones, game terminals, media players, and/or specialized 3D equipment. Network (150) represents any number of networks that transmit compressed point cloud between terminals (110) and (120). Network (150) may include, for example, wired and/or wireless communication networks. The network (150) may communicate on circuit-switched and/or packet-switched channels. Examples of networks include telecommunications networks, local area networks, wide area networks and/or the Internet. For the purposes of this discussion, the architecture and topology of the network (150) may not be relevant to the present invention unless specifically noted below.

[0033] FIG. 2 illustrates a simplified block diagram of a streaming system (200) in accordance with an embodiment of the invention. In FIG. 2 shows an example of the application of the invention to a point cloud. The present invention can also be used in other point cloud-enabled applications, such as a 3D telepresence application, a virtual reality application, and the like.

[0034] The streaming system (200) may include a capture subsystem (213). The acquisition subsystem (213) may include a point cloud source (201), such as light detection and ranging (LIDAR) systems, 3D cameras, 3D scanners, a graphics generation component that generates uncompressed point clouds in software, and the like. , for example, clouds (202) of points without compression. In the example, point clouds (202) include points acquired by 3D cameras. The point clouds (202) are shown with a bold line to emphasize the large amount of data compared to the compressed point clouds (204) (compressed point cloud bitstream). The compressed point clouds (204) may be generated by an electronic device (220) that includes an encoder (203) coupled to a point cloud source (201). The encoder (203) may include hardware, software, or a combination thereof to enable aspects of the invention to be implemented, as described in more detail below. Compressed point clouds (204) (compressed point clouds (204) bitstream), depicted as a thin line to emphasize the smaller amount of data compared to the stream of point clouds (202), can be stored on the streaming server (205) for future use . One or more streaming client subsystems, such as client subsystems (206) and (208) in FIG. 2 may access the streaming server (205) to retrieve copies (207) and (209) of the compressed point cloud (204). The client subsystem (206) may include a decoder (210), such as in an electronic device (230). The decoder (210) decodes the incoming copy (207) of the compressed point clouds and creates an outgoing stream of reconstructed point clouds (211) that can be displayed on the display device (212).

[0035] It should be noted that the electronic devices (220) and (230) may include other components (not shown). For example, the electronic device (220) may include a decoder (not shown) and the electronic device (230) may also include an encoder (not shown).

[0036] In some streaming systems, compressed point clouds (204), (207), and (209) (eg, compressed point cloud bitstreams) may be compressed according to certain standards. Some examples use video coding standards when compressing point clouds. Examples of these standards include High Efficiency Video Coding (HEVC), Versatile Video Coding (VVC), and the like.

[0037] FIG. 3 shows a block diagram of a V-PCC encoder (300) for encoding point cloud frames, in accordance with some embodiments of the invention. In some embodiments of the invention, the V-PCC encoder (300) may be used in the communication system (100) and the streaming system (200). For example, the encoder (203) may be configured and may operate similarly to the V-PCC encoder (300).

[0038] The V-PCC encoder (300) receives point cloud frames as uncompressed input and generates a bitstream corresponding to compressed point cloud frames. In some embodiments, the V-PCC encoder (300) may receive point cloud frames from a point cloud source, such as a point cloud source (201) or the like.

[0039] In the example of FIG. 3, the V-PCC encoder (300) comprises a patch generation module (306), a patch packing module (308), a geometry imaging module (310), a texture imaging module (312), a patch information module (304), a module (314) ) occupancy maps, smoothing module (336), image padding modules (316) and (318), group extension module (320), video data compression modules (322), (323) and (332), module (338) section assistance information compression, an entropy compression module (334), and a multiplexer (324).

[0040] According to an aspect of the invention, the V-PCC encoder (300) converts 3D point cloud frames into an image-based representation along with some metadata (e.g., occupancy map and parcel information) that is used to convert the compressed point cloud to a decompressed point cloud. . In some examples, the V-PCC encoder (300) may convert 3D point cloud frames into geometry images, texture images, and occupancy maps, and then use video encoding techniques to encode the geometry images, texture images, and occupancy map into a bitstream. Typically, a geometry image is a 2D image with pixels with geometry values associated with points projected onto the pixels, and a geometry value pixel may be referred to as a geometry sample. A texture image is a two-dimensional image with pixels with texture values associated with dots projected onto the pixels, and a texture value pixel may be referred to as a texture sample. The occupancy map is a two-dimensional image with pixels with values that indicate whether they are occupied by parcels.

[0041] The parcel generation module (306) divides the point cloud into a plurality of parcels (e.g., a parcel is defined as a continuous subset of the surface described by the point cloud) that may or may not overlap, so that each parcel can be described by a depth of field with respect to a plane in a two-dimensional space. In some embodiments of the invention, the section generation module (306) aims to decompose the point cloud into a minimum number of sections with smooth boundaries, as well as to minimize the reconstruction error.

[0042] The parcel information module (304) may collect parcel information that indicates the size and shape of the parcels. In some examples, the region information may be packed into an image frame and then encoded by the region assistance information compression module (338) to generate compressed region assistance information.

[0043] The parcel packing module (308) is configured to map the extracted parcels onto a two-dimensional (2D) grid, minimizing wasted space and ensuring that each MxM (eg, 16×16) grid block is associated with a unique parcel. Efficient parcel packing can directly affect compression efficiency, either by minimizing wasted space or by ensuring consistency over time.

[0044] The geometry imaging module (310) may generate two-dimensional geometry images associated with point cloud geometry at predetermined parcel locations. The texture imaging module (312) may generate two-dimensional texture images associated with a point cloud texture at predetermined patch locations. The geometry imaging module (310) and the texture imaging module (312) use the 3D to 2D transformation computed during the packing process to store the point cloud geometry and texture as images. To better handle the case where multiple points are projected onto the same sample, each area is projected onto two images, called layers. In the example, the geometry image is represented by a W×H monochromatic frame in YUV420-8bit format. To generate a texture image, the texture generation procedure uses the reconstructed/smoothed geometry to calculate the colors to be associated with the resampled points.

[0045] The occupancy map module (314) may generate an occupancy map that describes padding information in each unit. For example, the busy image includes a binary map that indicates, for each grid cell, whether the cell belongs to empty space or a point cloud. In the example, the occupancy map uses binary information describing for each pixel whether the pixel is padded or not. In another example, the occupancy map uses binary information describing for each pixel block whether the pixel block is padded or not.

[0046] The occupancy map generated by the occupancy map module (314) may be compressed using lossless coding or lossy coding. When lossless coding is used, an entropy compression module (334) is used to compress the occupancy map. When lossy coding is used, a video compression module (332) is used to compress the occupancy map.

[0047] It should be noted that the patch packing module (308) may leave some empty spaces between the two-dimensional patches packed into an image frame. The image padding modules (316) and (318) may fill in empty spaces (this is called padding) to form an image frame that may be suitable for 2B video and image codecs. Image padding is also called background padding, which can fill unused space with redundant information. In some examples, good background padding minimally increases the bit rate while at the same time does not introduce significant distortion when encoding around segment boundaries.

[0048] Video compression modules (322), (323), and (332) can encode two-dimensional images such as geometry augmented images, texture augmented images, and occupancy maps based on a suitable video coding standard such as HEVC, VVC, etc. P. In the example, video compression modules (322), (323) and (332) are separate components that operate separately. It should be noted that video data compression modules (322), (323) and (332) may be implemented as a single component in another example.

[0049] In some examples, the anti-aliasing module (336) is configured to create a smooth image of the reconstructed geometry image. A smoothed image may be provided to generate (312) a texture image. The texture imaging module (312) can then control the texture imaging based on the reconstructed geometry images. For example, when the shape of a region (eg, geometry) is slightly distorted during encoding and decoding, the distortion may be taken into account when generating texture images to correct distortion of the region's shape.

[0050] In some embodiments of the invention, group expansion (320) is configured to pad pixels around object boundaries with redundant low frequency content to increase encoding gain as well as improve the visual quality of the reconstructed point cloud.

[0051] The multiplexer (324) may multiplex the compressed geometry image, the compressed texture image, the compressed occupancy map, and the compressed section assistance information into a compressed bitstream.

[0052] FIG. 4 shows a block diagram of a V-PCC decoder (400) for decoding a compressed bitstream corresponding to point cloud frames, in accordance with some embodiments of the invention. In some embodiments of the invention, the V-PCC decoder (400) may be used in the communication system (100) and the streaming system (200). For example, the decoder (210) may be configured to operate similarly to the V-PCC decoder (400). The V-PCC decoder (400) receives the compressed bitstream and generates a reconstructed point cloud based on the compressed bitstream.

[0053] In the example of FIG. 4, the V-PCC decoder (400) comprises a demultiplexer (432), video data decompression modules (434) and (436), an occupancy map decompressor module (438), a segment assistance information decompressor module (442), a geometry reconstruction module (444), a smoothing module (446), a texture reconstruction module (448), and a color smoothing module (452).

[0054] The demultiplexer (432) may receive and separate the compressed bitstream into a compressed texture image, a compressed geometry image, a compressed occupancy map, and compressed patch assistance information.

[0055] Modules (434) and (436) decompress video data can decode compressed images in accordance with the appropriate standard (eg, HEVC, VVC, etc.) and output decompressed images. For example, video decompressor 434 decodes compressed texture images and outputs decompressed texture images, and video decompressor 436 decodes compressed geometry images and outputs decompressed geometry images.

[0056] The occupancy map decompressor module (438) may decode compressed occupancy maps according to an appropriate standard (eg, HEVC, VVC, etc.) and output decompressed occupancy maps.

[0057] The section assistance information decompressor module (442) may decode the compressed section assistance information according to an appropriate standard (eg, HEVC, VVC, etc.) and output the decompressed section assistance information.

[0058] The geometry reconstruction module (444) may receive decompressed geometry images and generate reconstructed point cloud geometry based on the decompressed occupancy map and the decompressed site assistance information.

[0059] The smoothing module (446) can smooth out inconsistencies at the edges of the sections. The smoothing procedure is aimed at mitigating potential inhomogeneities that may occur at the boundaries of the sections due to compression artifacts. In some embodiments of the invention, an anti-aliasing filter may be applied to pixels located at the boundaries of the region to reduce distortion that may be caused by compression/decompression.

[0060] The texture reconstruction module (448) may determine texture information for points in a point cloud based on decompressed texture images and smoothing geometry.

[0061] The color smoothing module (452) can smooth out color mismatches. Non-adjacent sections in 3D space are often packed next to each other into 2 V videos. In some examples, pixel values from non-contiguous areas may be mixed by a block video codec. The purpose of color smoothing is to reduce the visible artifacts that appear at the boundaries of the plots.

[0062] FIG. 5 shows a block diagram of a video decoder (510) according to an embodiment of the present invention. The video decoder (510) may be used in the V-PCC decoder (400). For example, video data decompressor modules (434) and (436) and busy map decompressor module (438) may be configured similarly to video decoder (510).

[0063] The video decoder (510) may include an analyzer (520) for reconstructing symbols (521) from compressed images, such as from an encoded video sequence. The categories of these symbols include information used to control the operation of the video decoder (510). The analyzer (520) may analyze/entropy decode the received encoded video sequence. The encoding of the video sequence may be according to a video coding technology or standard and may follow various principles including variable run length coding, Huffman coding, arithmetic coding with or without context sensitivity, and so on. The analyzer (520) can extract from the encoded video sequence a set of subgroup parameters for at least one of the subgroups of pixels in the video decoder based on at least one parameter corresponding to the group. Subgroups may include groups of pictures (GOP, Group of Pictures), images, tiles, slices, macroblocks, coding units (CU, Coding Unit), blocks, transformation units (TU, Transform Unit), prediction units (PU, Prediction Unit) and etc. The analyzer (520) can also extract information from the encoded video sequence, such as transform coefficients, quantization parameter values, motion vectors, and so on.

[0064] The analyzer (520) may perform an entropy decoding/analysis operation of the video sequence received from the buffer memory to create symbols (521).

[0065] Several different modules may be used for symbol reconstruction (521) depending on the type of encoded video image or parts thereof (eg, intra- and inter-coded images, intra- and inter-coded blocks) and other factors. Which modules are used and how can be determined by the subgroup management information obtained from the encoded video sequence by the analyzer (520). The flow of such subgroup control information between the analyzer (520) and the modules is not shown hereinafter for simplicity.

[0066] In addition to the previously mentioned functional blocks, the video decoder (510) can be fundamentally divided into several functional modules, as described below. In practical implementation under commercial constraints, many of these modules interact closely with each other and can, at least partially, be built into one another. However, for the purpose of describing the present invention, a fundamental division into the following functional modules is appropriate.

[0067] The first module is the scaling/inverse transform module (551). The scaling/inverse transform module (551) receives the quantized transform coefficient as well as control information including the transform used, block size, quantization factor, quantization scaling matrices, and so on. as character(s) (521) from the analyzer (520). The scaling/inverse transform module (551) can output blocks containing sample values that can be input to the aggregator (555).

[0068] In some cases, the output samples of the scaling / inverse transform module (551) may refer to an intra-coded block, that is, a block that does not use predicted information from previously reconstructed images, but can use predicted information from previously reconstructed parts of the current image . Such predicted information may be provided by an intra image prediction module (552). In some cases, the intra image prediction module (552) generates a block of the same size and shape as the block to be reconstructed using already reconstructed environment information retrieved from the current image buffer (558). The current picture buffer (558) buffers, for example, a partially reconstructed current picture and/or a fully reconstructed current picture. The aggregator (555) in some cases adds, for each sample, the prediction information generated by the intra prediction module (552) to the output sample information provided by the scaling/inverse transform module (551).

[0069] In other cases, the output samples of the scaling/inverse transform module (551) may refer to an intra-coded block, possibly with motion compensation. In such a case, the motion compensation prediction module (553) may access the reference picture memory (557) to retrieve samples used for prediction. After motion compensation is applied to the extracted samples according to the symbols (521) associated with the block, these samples can be added by the aggregator (555) to the output signal of the scaling / inverse transform module (551) (in this case called residual samples or residual signal) for formation of information of output readings. The addresses in the reference picture memory (557) from where the motion compensation prediction module (553) retrieves samples for prediction may be controlled by the motion vectors available to the motion compensation prediction module (553) in the form of symbols (521), which may have, for example , X, Y components, and reference image components. Motion compensation may also include interpolation of sample values retrieved from reference picture memory (557) when accurate sub-sample motion vectors, motion vector prediction mechanisms, etc. are used.

[0070] Various loop filtering techniques can be applied to the aggregator (555) output samples in the loop filter module (556). The video data compression technologies may include loop filter technologies controlled by parameters included in the coded video sequence (also referred to as the coded video bitstream) and available to the loop filter module (556) as symbols (521) from the analyzer (520), but may also respond to the meta-information obtained during the decoding of the previous (in the order of decoding) parts of the encoded image or encoded video sequence, as well as to respond to the previously reconstructed and contour filtered sample values.

[0071] The loop filter module (556) may output a stream of samples that may be fed to the renderer and also stored in the reference picture memory (557) for use in future inter-picture prediction.

[0072] Some encoded pictures, when fully reconstructed, can be used as reference pictures for future prediction. For example, when the encoded picture corresponding to the current picture is fully reconstructed and identified as a reference picture (e.g., by the analyzer (520)), the current picture buffer (558) may become part of the reference picture memory (557) and a fresh buffer for the current picture may be re-allocated. image before the next encoded image is reconstructed.

[0073] The video decoder (510) may perform decoding operations according to a predetermined video data compression technology in a standard such as ITU-T Rec. H.265. The encoded video sequence may follow the syntax given by the video compression technology or standard used, in the sense that the encoded video sequence may follow the syntax of the video compression technology or standard and the profiles specified in the video compression technology or standard document. In particular, a profile may select certain tools as the only tools available for use in that profile from among all the tools available in a video compression technology or standard. It may also be necessary for consistency that the complexity of the encoded video sequence remains within the limits set by the level of technology or video compression standard. In some cases, the levels limit the maximum image size, the maximum frame rate, the maximum sampling rate for reconstruction (measured in megasamples per second, for example), the maximum reference image size, and so on. The limits set by the levels, in some cases, may be further limited by the specifications of the Hypothetical Reference Decoder (HRD) and the metadata for managing the HRD buffer signaled in the encoded video sequence.

[0074] FIG. 6 shows a block diagram of a video encoder (603) according to an embodiment of the present invention. A video encoder (603) may be used in the V-PCC encoder (300) to compress point clouds. In the example, video compression modules (322) and (323) and video compression module (332) are configured similarly to encoder (603).

[0075] The video encoder (603) may receive images, such as augmented geometry images, augmented texture images, and the like, and generate compressed images.

[0076] According to an embodiment of the invention, the video encoder (603) can encode and compress images of the original video sequence (images) into an encoded video sequence (compressed images) in real time or with any other time constraints required by the application. Establishing a proper coding rate is one of the functions of the controller (650). In some embodiments of the invention, the controller (650) controls other functional modules, as described below, and is operatively connected to them. The connection is not shown for simplicity. The parameters set by the controller (650) may include parameters related to frequency adjustment (picture skip, quantizer, lambda value used in rate-distortion optimization methods, etc.), image size, grouping of pictures (GOP , group of pictures), maximum motion vector search area, etc. The controller (650) may have other suitable functions related to the video encoder (603) optimized for a particular system design.

[0077] In some embodiments of the invention, the video encoder (603) is configured to operate in a "coding loop". As a very simplified description, for example, an encoding loop may include a source encoder (630) (for example, responsible for creating symbols, for example, a symbol stream based on the input image to be encoded and the reference image(s)) and (local ) decoder (633) built into the video encoder (603). The decoder (633) reconstructs the symbols to create sample data in a similar way as a (remote) decoder would (because any compression of symbols into the encoded video bitstream is lossless in the video compression technologies discussed herein). A stream of reconstructed samples (sample data) enters the reference picture memory (634). Since decoding the symbol stream produces bit-accurate results that are independent of the position of the (local or remote) decoder, the content in the reference picture memory (634) will also be the same bit-accurate for the local encoder and the remote encoder. In other words, the prediction portion of the encoder "sees" as reference picture samples exactly the same sample values as the decoder would "see" if using prediction during decoding. This fundamental principle of reference picture synchronism (and eventually drift if synchronism cannot be maintained due to, for example, channel errors) is also used in some related fields of technology.

[0078] A "local" decoder (633) may operate in the same manner as a "remote" decoder, such as the video decoder (510) detailed above with reference to FIG. 5. However, again, referring to FIG. 5, since the symbols are available and the entropy encoder (645) and parser (520) can be losslessly encoded/decoded into the encoded video sequence, the entropy decoding parts of the video decoder (510), including the parser (520), may not be fully implemented in the local decoder (633).

[0079] In doing so, it can be concluded that any decoding technology present in a decoder, with the exception of analysis/entropy decoding, must also necessarily be present in essentially identical functional form in the corresponding encoder. For this reason, the present description focuses on the operation of the decoder. The description of the encoding technologies can be abbreviated as they are the inverse of the decoding technologies described in detail. Only in some areas a more detailed description is required, which is given below.

[0080] According to some examples, during operation, the source encoder (630) may perform motion-compensated predictive coding, in which an input picture is predictively encoded based on one or more previously encoded pictures from a video sequence, referred to as "reference pictures". Thus, the coding engine (632) encodes the differences between the pixel blocks of the input image and the pixel blocks of the reference image(s) that can be selected as the prediction reference(s) to the input frame.

[0081] The local video decoder (633) may decode the encoded video data of pictures that can be specified as reference pictures based on the symbols generated by the source encoder (630). The operations of the encoding engine (632) may be predominantly lossy methods. When the encoded video data can be decoded in a video decoder (not shown in FIG. 6), the reconstructed video sequence can typically be a copy of the original video sequence with some errors. The local video decoder (633) duplicates the decoding processes that may be performed by the video decoder on the reference pictures and may store the reconstructed reference pictures in the reference picture cache (634). Thus, the video encoder (603) can locally store copies of the reconstructed reference pictures having the same content as the reconstructed reference pictures that would be received by the remote video decoder (in the absence of transmission errors).

[0082] The predictor (635) may perform prediction searches for the encoding engine (632). Thus, for a new picture to be encoded, the predictor (635) may search the reference picture memory (634) for sample data (as pixel reference block candidates) or specific metadata such as reference picture motion vectors, block shapes, etc. .d., which can serve as a proper reference for predicting new images. The predictor (635) may operate on a "blocks of samples×blocks of pixels" basis to find proper references for prediction. In some cases, according to the search results obtained by the predictor (635), the input picture may have prediction references extracted from the plurality of reference pictures stored in the reference picture memory (634).

[0083] The controller (650) can control the encoding operations of the source encoder (630), including, for example, setting the parameters as well as the subgroup parameters used to encode the video data.

[0084] The output signal of all the above functional modules can be entropy encoded in an entropy encoder (645). An entropy encoder (645) translates symbols generated by various functional units into an encoded video sequence by lossless symbol compression according to, for example, techniques such as Huffman coding, variable run length coding, arithmetic coding, and so on.

[0085] The controller (650) may control the operation of the video encoder (603). During encoding, the controller (650) may assign to each encoded picture one or another encoded picture type, which may determine the encoding methods applicable to the corresponding picture. For example, the following image types can often be assigned to images.

[0086] An intra-coded picture (I-picture) that can be encoded and decoded without using any other picture in the sequence as a source for prediction. Some video codecs allow different types of intra-coded images, including, for example, images in the Independent Decoder Refresh (IDR) format. The person skilled in the art will be aware of the varieties of I-images and their respective applications and features.

[0087] A predictive picture (P-picture) that can be encoded and decoded using intra prediction or inter prediction using at most one motion vector and a reference index to predict sample values of each block.

[0088] A bidirectional predictive picture (B-picture) that can be encoded and decoded using intra prediction or inter prediction using no more than two motion vectors and reference indices to predict sample values of each block. Similarly, multi-prediction pictures may use more than two reference pictures and associated metadata to reconstruct a single block.

[0089] Source images typically allow for spatial division into multiple blocks of samples (eg, blocks of 4x4, 8x8, 4x8, or 16x16 samples each) and coding on a block-by-block basis. The blocks may be predictively encoded with reference to other (previously coded) blocks defined by the coding assignment applied to the respective images of those blocks. For example, blocks of 1-pictures may be encoded without prediction or predictively with reference to previously encoded blocks of the same picture (with spatial prediction or intra prediction). P-picture pixel blocks may be predictively encoded by spatial prediction or temporal prediction with reference to one previously encoded reference picture. Blocks of B-pictures may be predictively encoded by spatial prediction or temporal prediction with reference to one or two previously encoded reference pictures.

[0090] The video encoder (603) may perform encoding operations according to a predetermined video coding technology or standard, such as ITU-T Rec. H.265. In operation, the video encoder (603) may perform various compression operations, including predictive coding operations that exploit temporal and spatial redundancy in the input video sequence. Therefore, the encoded video data may follow the syntax given by the technology used or the video coding standard.

[0091] The video may be in the form of a plurality of source images in time sequence. Intra-image prediction (often abbreviated as intra-image prediction) uses the spatial correlation within a given image, while inter-image prediction uses (temporal or otherwise) correlation between images. For example, a specific image to be encoded/decoded, which is called the current image, is divided into blocks. When a block in the current picture is similar to a reference block in a previously encoded and still buffered reference picture in a video, the block in the current picture may be encoded with a vector called a motion vector. The motion vector points to a reference block in the reference picture and may have a third dimension identifying the reference picture if multiple reference pictures are used.

[0092] In some embodiments of the invention, a dual prediction technique in inter-picture prediction may be used. According to the double prediction method, two reference pictures are used, for example, a first reference picture and a second reference picture, both of which precede the current picture in the video in decoding order (but may be in the past and future, respectively, in display order). A block in the current picture may be encoded by a first motion vector that points to a first reference block in the first reference picture and a second motion vector that points to a second reference block in the second reference picture. The block may be predicted with a combination of the first reference block and the second reference block.

[0093] Additionally, a merge mode technique may be used in inter-picture prediction to improve coding efficiency.

[0094] According to some embodiments of the invention, predictions, such as inter-picture prediction and intra-picture prediction, are performed block by block. For example, according to the HEVC standard, an image in a video sequence is divided into coding tree units (CTUs) for compression, while the CTUs in the image have the same size, for example, 64×64, 32×32, or 16×16 pixels. In general, a CTU includes three coding tree blocks (CTB), namely, one luma CTB and two chrominance CTBs. Each CTU may be recursively split on a quadtree basis into one or more coding units (CUs). For example, a 64x64 pixel CTU can be divided into one 64x64 pixel CU, or 4 32x32 pixel CUs, or 16 16x16 pixel CUs. For example, each CU is analyzed to determine a prediction type for that CU, such as an inter prediction type or an intra prediction type. The CU is divided into one or more prediction units (PUs) based on temporal and/or spatial predictability. In general, each PU includes a luminance prediction block (PB) and two chrominance prediction blocks PB. According to an embodiment of the invention, the encoding (encoding/decoding) prediction operation is performed in a prediction block unit. As an example of a prediction block, a luminance prediction block includes a matrix of values (e.g., luminance values) for pixels, e.g., 8x8 pixels, 16x16 pixels, 8x16 pixels, 16x8 pixels, and the like.

[0095] FIG. 7 shows a block diagram of a G-PPC encoder (700) in accordance with an embodiment of the invention. The encoder (700) may be configured to receive point cloud data and compress the point cloud data to create a bit stream carrying the compressed point cloud data. In an embodiment of the invention, the encoder (700) may include a position quantization module (710), a duplicate point removal module (712), an octree encoding module (730), an attribute transmission module (720), a LOD generation module (740), a module (750) attribute prediction, a residual quantization module (760), an arithmetic encoding module (770), a residual dequantization module (780), an addition module (781), and a memory (790) for storing the reconstructed attribute values.

[0096] As shown, the encoder (700) may receive an input point cloud (701). The positions (eg, three-dimensional coordinates) of the point cloud (701) are provided to the quantization module (710). The quantization module (710) is configured to quantize coordinates to generate quantized positions. The duplicate dot removal module (712) is configured to receive the quantized positions and perform a filtering process to identify and remove duplicate dots. The octree encoding module (730) is configured to receive the filtered positions from the duplicate point removal module (712) and perform an octree-based encoding process to generate a sequence of busy codes that describe a three-dimensional grid of voxels. The busy codes are passed to the arithmetic coding module (770).

[0097] The attribute transfer module (720) is configured to receive attributes of the input point cloud and perform an attribute transfer process to determine an attribute value for each voxel when a plurality of attribute values are associated with the corresponding voxel. The attribute transfer process may be performed on the reordered points output from the octree encoding module (730). The attributes after the transfer operations are provided to the attribute prediction module (750). The LOD generation module (740) is configured to operate on the reordered points output from the octree encoding module (730) and reorganize the points into different LOD levels. The LOD information is provided to the attribute prediction module (750).

[0098] The attribute prediction module (750) processes the points in accordance with the order based on the LOD specified in the LOD information from the LOD generation module (740). The attribute prediction module (750) generates an attribute prediction for the current point based on the reconstructed attributes of the plurality of neighboring points for the current point stored in the memory (790). Prediction residuals can subsequently be obtained based on the original attribute values received from the attribute transmission module (720) and the locally generated attribute predictions. When candidate indexes are used in a corresponding attribute prediction process, an index corresponding to the selected prediction candidate may be provided to the arithmetic encoding unit (770).

[0099] The residual quantization module (760) is configured to receive prediction residuals from the attribute prediction module (750) and perform quantization to generate quantized residuals. The quantized residuals are provided to the arithmetic coding module (770).

[0100] The residual dequantization module (780) is configured to receive quantized residuals from the residual quantization module (760) and generate reconstructed prediction residuals by performing the reverse operations of the quantization operations performed in the residual quantization module (760). The adder (781) is configured to receive the reconstructed prediction residuals from the residual dequantizer (780) and corresponding attribute predictions from the attribute predictor (750). By combining the reconstructed prediction residuals and the attribute predictions, the reconstructed attribute values are generated and stored in memory (790).

[0101] The arithmetic encoding module (770) is configured to receive busy codes, candidate indices (if used), quantized residuals (if generated), and other information, and perform entropy encoding to further compress the received values or information. As a result, a compressed bit stream (702) carrying compressed information can be generated. The bitstream (702) may be transmitted to or otherwise provided to a decoder that decodes the compressed bitstream, or may be stored in a storage device.

[0102] FIG. 8 shows a block diagram of a G-PCC decoder (800) according to an embodiment of the invention. The decoder (800) may be configured to receive the compressed bitstream and decompress the point cloud data to decompress the bitstream to generate decoded point cloud data. In an embodiment, the decoder (800) may comprise an arithmetic decoding module (810), a residual dequantization module (820), an octree decoding module (830), a LOD generation module (840), an attribute prediction module (850), and a memory (860) for storing reconstructed attribute values.

[0103] As shown, the compressed bit stream (801) may be received in the arithmetic decoding module (810). The arithmetic decoding module (810) is configured to decode the compressed bitstream (801) to obtain quantized residuals (if any) and point cloud occupancy codes. The octree decoding module (830) is configured to determine reconstructed point cloud positions in accordance with busy codes. The LOD generation module (840) is configured to reorganize the points into different LOD levels based on the reconstructed positions and determine the order based on the LOD. The residual dequantization module (820) is configured to generate reconstructed residuals based on the quantized residuals received from the arithmetic decoding module (810).

[0104] The attribute prediction module (850) is configured to perform an attribute prediction process to determine attribute predictions for points according to LOD-based order. For example, a current point attribute prediction may be determined based on reconstructed neighboring point attribute values for the current point stored in memory (860). An attribute prediction module (850) may combine the attribute prediction with the corresponding reconstructed residual to generate a reconstructed attribute for the current point.

[0105] The sequence of reconstructed attributes generated by the attribute prediction module (850), together with the reconstructed positions generated by the octree decoding module (830), corresponds to the decoded point cloud (802) that is output from the decoder (800) in one example. In addition, the reconstructed attributes are also stored in memory (860) and can later be used to derive attribute predictions for subsequent points.

[0106] In various embodiments, the encoder (300), decoder (400), encoder (700), and/or decoder (800) may be implemented in hardware, software, or a combination thereof. For example, the encoder (300), decoder (400), encoder (700), and/or decoder (800) may be implemented with a processing circuit, such as one or more integrated circuits that operate with or without software, such as a dedicated integrated circuit (ASIC, application specific integrated circuit), field programmable gate array (FPGA), etc. In another example, the encoder (300), decoder (400), encoder (700), and/or decoder (800) may be implemented as software or firmware, including instructions stored on a persistent (or non-volatile) computer-readable storage medium. The instructions, when executed by a processing circuit, such as one or more processors, cause the processing circuit to perform the functions of an encoder (300), decoder (400), encoder (700), and/or decoder (800).

[0107] It should be noted that attribute prediction modules (750) and (850) configured to implement the attribute prediction methods disclosed herein may be included in other decoders or encoders, which may have a structure similar to those shown in FIG. . 7 and FIG. 8, or a structure different from them. In addition, the encoder (700) and decoder (800) may be included in the same device or separate devices in different examples.

[0108] According to some aspects of the invention, in some related examples (for example, in the current version of the TMC13 model), a recoloring process is applied on the encoder side when quantizing the geometry and combining repeated positions. The repainting process is relatively complex. For example, the repainting process relies on a k-d tree data structure (eg, a three-dimensional tree) to find the nearest neighbor(s). Searching the k-d tree can be complex and time consuming.

[0109] In a related example (e.g., in a version of the TMC13 model), the input to the repainting algorithm may include the original positions of the original points in the point cloud, attributes associated with the original positions, and reconstructed positions (for example, the results of geometry quantization and merging of repeated positions) . The repainting algorithm may transfer the attributes associated with the original positions to the attributes associated with the reconstructed positions, eg, minimizing attribute distortion. Attributes associated with rendered positions can form rendered points in the rendered point cloud. It should be noted that the repainting algorithm may convey any suitable attribute such as color, material properties, texture information, intensity attributes, reflectivity attributes, motion related attributes, modality attributes, and various other attributes. In one example, the repainting algorithm includes five steps, which are detailed below.

обозначает реконструированные положения, a N_rec обозначает количество реконструированных положений. Если повторяющиеся точки существуют и объединены, то N_rec<N, в противном случае N_rec=N.[0110] In the first step, the original positions of the points in the point cloud, the attributes associated with the original positions, and the reconstructed positions are received. In the example, (X _i ) _i=0…N-1 denotes the original positions of the seed points in the source point cloud, N denotes the number of points in the source point cloud,

denotes the reconstructed positions, and N _rec denotes the number of reconstructed positions. If duplicate points exist and are merged, then N _rec <N, otherwise N _rec =N.

в реконструированном облаке точек выполняют процесс поиска, например, на основе поиска по дереву k-d, чтобы определить

который является ближайшим соседом в исходном облаке точек, и

обозначает значение атрибута, связанного с ближайшим соседом в исходном облаке точек.[0111] In the second step, for each position

in the reconstructed point cloud, a search process is performed, for example, based on a kd tree search, to determine

which is the nearest neighbor in the original point cloud, and

denotes the value of the attribute associated with the nearest neighbor in the original point cloud.

в реконструированном облаке точек выполняют процессы поиска, например, на основе поиска по дереву k-d, чтобы определить множество исходных положений в исходном облаке точек, которое обозначается

при этом множество исходных положений совместно использует

в качестве ближайшего соседа в реконструированном облаке точек. H(i) обозначает количество элементов в

а

обозначает один из элементов

Следует отметить, что множество

может быть пустым или может иметь один или более элементов.[0112] In the third step, for each position

in the reconstructed point cloud, search processes are performed, for example, based on a kd tree search, to determine a set of initial positions in the original point cloud, which is denoted

in this case, the set of initial positions shares

as the nearest neighbor in the reconstructed point cloud. H(i) denotes the number of elements in

a

denotes one of the elements

It should be noted that many

may be empty or may have one or more elements.

Когда

пусто, значение атрибута

связанного с положением

устанавливают равным

[0113] In the fourth step, check

When

empty, attribute value

associated with the position

set equal to

не пусто, значение атрибута

связанного с положением

может быть вычислено, например, на основе выражения (1):[0114] In the fifth step, if

not empty, attribute value

associated with the position

can be calculated, for example, based on expression (1):

[0115] In a related example, searches such as a k-d tree search (eg, a 3D tree) are used to find the nearest neighbor(s). Searching the k-d tree can be complex and time consuming. The present invention provides repainting techniques to reduce complexity and increase processing speed. For example, a nearest neighbor search can be determined based on the information stored when the geometry is quantized and the repeated points are combined.

[0116] The proposed methods can be used individually or combined in any order. Furthermore, each of the encoder and decoder methods (or embodiments) may be implemented by a processing circuit (eg, one or more processors or one or more integrated circuits). In one example, one or more processors execute a program that is stored on a computer-readable medium.

[0117] According to some aspects of the invention, geometry information and associated point cloud attributes, such as color, reflectance, and the like, can be separately compressed (eg, in a TMC13 model). In some embodiments of the invention, information about the geometry of the point cloud, which includes the three-dimensional coordinates of points in the point cloud, can be processed in accordance with the process of quantization and merging repetitions. In some examples, the repetition quantization and merging process may be based on octree-based partitioning, and the geometry information may be encoded by octree-based partitioning with partition occupancy information. The attributes can be compressed based on the reconstructed geometry using, for example, prediction, lifting, and region adaptive hierarchical transform (RAHT) techniques. For example, the encoded geometry information may be dequantized to reconstruct the geometry information for the point cloud, such as reconstructed positions. The original point attributes are transferred to the reconstructed positions and then encoded.

[0118] According to some aspects of the invention, three-dimensional space can be partitioned using octree-based partitioning. Octrees are the 3D analog of quadtrees in 2D. The octree-based partitioning method refers to a partitioning method that recursively partitions a three-dimensional space into eight octants, and the octree structure refers to a tree structure that represents partitions. In one example, each node in the octree structure corresponds to a 3D space, and the node can be a leaf node (no split node, also called a leaf node in some examples) or a further split node. A division at a node can divide the three-dimensional space represented by the node into eight octants. In some examples, nodes corresponding to partitions of a particular node may be referred to as child nodes of the particular node.

[0119] FIG. 9 shows a diagram illustrating partitioning of a 3D cube (900) (corresponding to a node) based on an octree based partitioning method according to some embodiments of the present invention. The division may divide the 3D cube (900) into smaller equal sized cubes 0-7 as shown in FIG. 9.

[0120] The octree partitioning method (eg, in TMC13) may recursively divide the original 3D space into smaller units, and the occupancy information of each subspace may be encoded to represent geometric positions.

[0121] In some embodiments of the invention (eg, TMC13), an octree geometry codec is used. An octree geometry codec can perform geometry encoding. In some examples, geometry encoding is performed on a cubic box. For example, a cube box could be an axis-aligned bounding box B, which is defined by two points (0,0,0) and (2m ^-1 , ^2m-1 , ^2m-1 ), where ^2m-1 defines the size of the bounding box. box B, and M may be specified in the bitstream.

[0122] The octree structure is then built by recursively splitting the cubic box. For example, a cubic box defined by two points (0,0,0) and (2m- ¹ , 2m- ¹ , 2m ^-1 ) is divided into 8 sub-cube boxes, and then an 8-bit code is generated, called the busy code . Each bit of the busy code is associated with a subcube box, and the value of this bit is used to indicate whether the corresponding subcube box contains any point cloud points. For example, a bit value of 1 indicates that the subcube box associated with the bit contains one or more point cloud points, and a bit value of 0 indicates that the subcube box associated with the bit contains no points. point clouds.

[0123] An empty subcube box (eg, the value of the bit associated with the subcube box is 0) is no longer partitioned. When a subcube box has one or more point cloud points (for example, the value of the bit associated with the subcube box is 1), the subcube box is further divided into 8 smaller subcube boxes, and an occupancy code for the subcube box can be generated to indicate the occupancy of the smaller ones. subcubic boxes. In some examples, partitioning operations may be repeated on non-empty subcube boxes until the size of the subcube boxes is equal to a predetermined threshold, such as 1. In some examples, subcube boxes of size 1 (one size) are called voxels, and subcube boxes that have larger sizes than voxels may be referred to as non-voxels.

[0124] FIG. 10 shows an example of an octree based partition (1010) and an octree structure (1020) corresponding to an octree based partition (1010), according to some embodiments of the invention. Fig. 10 shows two levels of partitioning in partitioning (1010) based on the octree. The octree structure (1020) includes a node (N0) corresponding to a cubic box for partitioning (1010) based on the octree. At the first level, the cube box is divided into 8 sub-cube boxes, which are numbered from 0 to 7 according to the numbering method shown in FIG. 9. The occupancy code for splitting node N0 is "10000001" in binary, indicating that the first subcube box represented by node N0-0 and the eighth subcube box represented by node N0-7 contain point cloud points and the others subcubic boxes are empty.

[0125] Then, in the second division level, the first subcube box (represented by node N0-0) and the eighth subcube box (represented by node N0-7) are further divided into eight octants, respectively. For example, the first subcube box (represented by node N0-0) is divided into 8 smaller subcube boxes, which are numbered 0-7 according to the numbering method shown in FIG. 9. The busy code for splitting node N0-0 is "00011000" in binary, indicating that the fourth smaller subcube box (represented by node N0-0-3) and the fifth smaller subcube box (represented by node N0-0-4) contain point cloud points, and the other smaller subcubic boxes are empty. At the second level, the seventh subcube box (represented by node N0-7) is likewise divided into 8 smaller subcube boxes, as shown in FIG. ten.

[0126] In FIG. 10, nodes corresponding to a non-empty cubic space (such as a cubic box, subcubic boxes, smaller subcubic boxes, etc.) are colored gray and are referred to as shaded nodes.

[0127] In some embodiments of the invention, for a unit size cube box voxel, the points in the unit size cube box may be quantized, for example, to the center position of the unit size cube box. If there are multiple points in a one-size cube box, the points are merged into the center position of the one-size cube box. The central positions of the voxels can form reconstructed positions for the reconstructed points in the reconstructed point cloud.

[0128] According to some aspects of the invention, a suitable data structure (such as a lookup table, list, etc.) can be used to associate seed points in a voxel with a reconstructed position (eg, center position) of the voxel. For example, an entry in a lookup table may associate a voxel index with indexes of seed points in the voxel. The data structure can be saved. Thus, for each position in the reconstructed point cloud that is the center point of the voxel, the original points in the voxel that are combined with the center position can be determined based on the data structure.

каждого положения в реконструированном облаке точек не пусто. Следовательно, четвертый этап в алгоритме перекрашивания не нужен.[0129] According to an aspect of the invention, point cloud geometry is quantized by scalar quantization, and scalar quantization is the only source of geometric distortion in some examples. In some embodiments of the invention, when octree-based partitioning is used for geometry quantization, the voxels are a unit size cubic box, each containing one or more point cloud seed points, and one or more seed points quantized into the voxel center point. Thus, one or more original points in a voxel have the same center point (in the reconstructed point cloud) of the voxel as the nearest neighbor and the set

each position in the reconstructed point cloud is not empty. Therefore, the fourth stage in the repainting algorithm is not needed.

при квантовании геометрии. Ромбовидные точки 1-8 представляют собой исходные точки, а круглые черные точки R1-R3 представляют собой реконструированные положения реконструированных точек (также называемые квантованными точками в некоторых примерах).[0130] FIG. 11 shows a one-dimensional diagram illustrating acquisition methods that can be used in three-dimensional (3D) space to determine a set of points sharing a reconstructed position.

in the quantization of geometry. The diamond dots 1-8 are the original dots and the round black dots R1-R3 are the reconstructed positions of the reconstructed dots (also called quantized dots in some examples).

могут быть квантованы в положение nq, где n=0, ±1, ±2, ….[0131] In the example, the quantization step is q, so all points in the range

can be quantized to position nq, where n=0, ±1, ±2, ….

каждого реконструированного положения - это исходные точки, которые квантуются в реконструированное положение. Таким образом, в некоторых вариантах осуществления изобретения подходящая структура данных используется для хранения множества

связанного с реконструированным положением. Затем на основе структуры данных для реконструированного положения можно быстро определить множество

без дополнительного поиска по дереву k-d.For example, seed points 1-3 are in the range [-3q/2, -q/2) and are quantized to the reconstructed position R1, seed points 4-5 are in the range [-q/2, q/2) and are quantized to the reconstructed position position R2, the original points 6-8 are in the range [q/2, 3q/2) and are quantized to the reconstructed position R3. Therefore, the nearest neighbor of the original point in the reconstructed point cloud is the reconstructed position. In other words, many

of each reconstructed position are the original points that are quantized into the reconstructed position. Thus, in some embodiments of the invention, a suitable data structure is used to store a set of

associated with the reconstructed position. Then, based on the data structure for the reconstructed position, you can quickly determine the set

without additional search in the kd tree.

[0132] Based on the quantization results stored in a suitable data structure, the repainting algorithm can be simplified. For example, an attribute value of each quantized position is assigned based on the attribute values in the source positions that are quantized into the quantized position. In particular, in some examples, the repainting algorithm may include 3 steps, which are detailed below.

обозначает реконструированные положения реконструированного облака точек, a N_rec обозначает количество реконструированных положений в реконструированном облаке точек. Если повторяющиеся точки существуют и объединены, то N_rec<N, в противном случае N_rec=N.[0133] In the first step, the original positions of the points in the point cloud, the attributes associated with the original positions, and the reconstructed positions are received. In the example (X _i ) _i=0…N-1 denotes the initial positions of the points in the original point cloud, N denotes the number of points in the original point cloud,

denotes the reconstructed positions of the reconstructed point cloud, and N _rec denotes the number of reconstructed positions in the reconstructed point cloud. If duplicate points exist and are merged, then N _rec <N, otherwise N _rec =N.

в реконструированном облаке точек может осуществляться доступ к структуре данных, в которой хранится множество исходных положений в связи с каждым реконструированным положением, на основе реконструированного положения для определения множества исходных положений

при этом исходные положения в указанном множестве квантованы и деквантованы в

H(i) обозначает количество элементов в

a X_i(h) обозначает один из элементов

Множество

может быть получено в процессе квантования геометрии.[0134] In the second step, for each reconstructed position

in the reconstructed point cloud, a data structure that stores a plurality of initial positions in connection with each reconstructed position can be accessed, based on the reconstructed position, to determine the plurality of initial positions

in this case, the initial positions in the specified set are quantized and dequantized in

H(i) denotes the number of elements in

a X _i (h) denotes one of the elements

Lots of

can be obtained in the process of geometry quantization.

связанное с реконструированным положением

может быть получено из значений атрибутов, связанных с исходными положениями в множестве

[0135] In the third step, the attribute value

associated with reconstructed position

can be obtained from the attribute values associated with the initial positions in the set

могут использоваться различные методы.[0136] To calculate an attribute value

different methods can be used.

для реконструированного положения. В примере a _i(h) обозначает значение атрибута, связанного с X_i(h), при этом значение атрибута

связанного с реконструированной точкой

может быть вычислено с использованием выражения (2):[0137] In an embodiment of the invention, the average value of the attributes is calculated as the value of the attribute

for the reconstructed position. In the example, a _i (h) denotes the value of the attribute associated with X _i (h), with the attribute value

associated with the reconstructed point

can be calculated using expression (2):

вычисляют с помощью средневзвешенного значения атрибутов, например, используя выражение (3):[0138] In another embodiment of the invention

calculated using the weighted average of the attributes, for example, using expression (3):

w_i(h)≥0 обозначает вес атрибута в соответствующем положении, который может быть обратно пропорционален расстоянию между исходным положением X_i(h) и реконструированным положением

Расстояние между исходным положением X_i(h) и реконструированным положением

можно оценить с помощью любой подходящей меры расстояния. Например, расстояние это пространственное расстояние, такое как евклидово расстояние.where

w _i (h)≥0 denotes the weight of the attribute at the corresponding position, which may be inversely proportional to the distance between the original position X _i (h) and the reconstructed position

Distance between the original position X _i (h) and the reconstructed position

can be estimated using any suitable distance measure. For example, distance is a spatial distance, such as Euclidean distance.

назначают значение атрибута

ближайшего исходного положения в

Когда множество исходных положений имеют одинаковое ближайшее расстояние до реконструированного положения

медианное значение атрибута или среднее значение атрибута значений атрибутов множества исходных положений может быть присвоено

Расстояние между исходным положением X_i(h) и реконструированным положением

можно оценить с помощью любой другой меры расстояния. Например, расстояние - это пространственное расстояние, такое как евклидово расстояние.[0139] In another embodiment of the invention, the reconstructed position

assign an attribute value

closest starting position in

When many original positions have the same closest distance to the reconstructed position

the median attribute value or mean attribute value of the attribute values of a set of reference positions can be assigned

Distance between the original position X _i (h) and the reconstructed position

can be estimated using any other measure of distance. For example, distance is a spatial distance such as Euclidean distance.

[0140] FIG. 12 shows a flowchart describing a method (1200) according to an embodiment of the invention. Method (1200) may be used in a point cloud encoding process. In various embodiments of the invention, the method (1200) is performed by a processing circuit, such as a processing circuit in terminal devices (110), a processing circuit that performs the functions of an encoder (203) and/or a decoder (201), a processing circuit that performs the functions of an encoder ( 300), a decoder (400), an encoder (700), and/or a decoder (800), and the like. In some embodiments of the invention, the method (1200) is implemented in program instructions, when executed by the processing circuit, the processing circuit executes the method (1200). The method starts from step (S1201) and proceeds to step (S1210).

[0141] In step (S1210), one or more seed points in the point cloud that are associated with the reconstructed position are determined. The positions of one or more of the original points are reconstructed according to the quantization of the geometry into the reconstructed position.

[0142] In some examples, octree-based partitioning may be performed to encode geometry information. Octree-based partitioning can partition point cloud space into voxels. One or more seed points are in the voxel and are associated with a reconstructed position representing the voxel. The geometry information of one or more seed points may be reconstructed (eg, quantized and dequantized) into the reconstructed position. In an embodiment of the invention, a data structure is stored that associates one or more original points with a reconstructed position. The data structure may be accessed based on the reconstructed position to retrieve one or more seed points associated with the reconstructed position.

[0143] In step (S1220), an attribute value for the reconstructed position is determined based on the attribute information of one or more seed points.

[0144] In an embodiment of the invention, the average value of the seed point set attributes is calculated as said determined attribute value for the reconstructed position when the seed point set is associated with the reconstructed position.

[0145] In some examples, an attribute value of a seed point in a plurality of seed points may be weighted based on the inverse distance between the seed point and the reconstructed position.

[0146] In another embodiment of the invention, a weighted average of the attributes of a plurality of seed points is calculated as said determined attribute value for the reconstructed position. In the example, an attribute value of a seed point in a set of seed points is assigned a weight based on the inverse distance between the seed point and the reconstructed position.

[0147] In another embodiment of the invention, a particular attribute value of the nearest point in a set of seed points is assigned as said determined attribute value for the reconstructed position in response to the set of seed points being associated with the reconstructed position. In the example, the median attribute value among the attribute values for the nearest points is chosen in response to the existence of a plurality of nearest points having the same shortest distance to the reconstructed position. In another example, an attribute average of attribute values for nearest points is computed in response to the existence of a plurality of nearest points having the same shortest distance to the reconstructed position.

[0148] In step (S1230), the point cloud texture is encoded using the reconstructed position having the determined attribute value. In the example, the attribute values for the reconstructed positions are compressed and included in the encoded bitstream for the point cloud as point cloud texture information. Then, the method proceeds to step (S1299) and ends.

[0149] The above methods may be implemented as computer software using computer-readable instructions and physically stored on one or more computer-readable media. For example, in FIG. 13 shows a computer system (1300) suitable for practicing some embodiments of the invention.

[0150] Computer software may be coded using any suitable machine code or computer language that may be assembled, compiled, linked, or similarly processed to produce code containing instructions that can be executed directly or through interpretation, execution of microcode, etc. etc., one or more computer central processing units (CPU, central processing units), graphic processors (GPU, Graphics Processing Units), etc.

[0151] The instructions may be executed on various types of computers or components thereof, including, for example, personal computers, tablet computers, servers, smartphones, gaming devices, IoT devices, and the like.

[0152] The computer system components (1300) shown in FIG. 13 are illustrative in nature and are not intended to impose any limitation on the scope of application or functionality of computer software implementing embodiments of the present invention. The component configuration should also not be interpreted as having any dependency or requirement in connection with any component or combination of components shown in the exemplary computer system (1300).

[0153] The computer system (1300) may include some human-machine interface input devices. Such an input device may be responsible for input by one or more users through, for example, tactile input (e.g., keystrokes, waving, glove movements), audio input (e.g., voice, claps), visual input (e.g., gestures), olfactory input (not shown). Interface devices may also be used to capture certain media not necessarily directly related to conscious human input, such as sound (e.g. speech, music, external sound), images (e.g. scanned images, photographic images received from a still camera), video (eg 2D video, 3D video including stereoscopic video).

[0154] The input human-machine interface of the device may include one or more of the following (shown one at a time): keyboard (1301), mouse (1302), touch pad (1303), touch screen, information glove (not shown), joystick ( 1305), microphone (1306), scanner (1307) and camera (1308).

[0155] The computer system (1300) may also include some human-machine interface output devices. Such output devices may stimulate the senses of one or more users through, for example, tactile output, sound, light, and smell/taste. Such output devices may include tactile output devices (e.g., tactile feedback via a touch screen, data glove (not shown), or joystick (1305), as well as tactile feedback devices that do not serve as input devices), audio output devices (e.g.: speakers (1309), headphones (not shown)), visual output devices (e.g., screens (1310), including CRT screens, LCD screens, plasma screens, OLED screens, each with or without touch screen input capabilities , each with or without haptic feedback capabilities, some of which are capable of 2D visual output or more than 3D output via, for example, stereographic output, virtual reality goggles (not shown), holographic displays, smoke tanks (not shown), and printers (not shown).

[0156] The computer system (1300) may also include human accessible storage devices and associated media, such as optical media including CD/DVD ROM/RW (1320) with CD/DVD media (1321) and the like, flash memory cards (1322), removable hard drive or solid state drive (1323), conventional magnetic media such as tape and magnetic disk (not shown), dedicated ROM/ASIC/PLD based devices such as security dongles (not shown). shown), etc.

[0157] Those skilled in the art should also understand that the term "computer-readable media" as used in connection with the present invention does not encompass transmission media, carrier waves, or other time signals.

[0158] The computer system (1300) may also include an interface to one or more communication networks. Networks can be, for example, wireless, wired, optical. Networks can also be local, global, urban, transport and industrial, real-time, latency, etc. Examples of networks include local area networks, such as Ethernet, wireless LANs, cellular networks, including GSM, 3G, 4G, 5G, LTE, etc., global digital networks of wired or wireless television, including cable television, satellite television and terrestrial broadcasting, transport and industrial networks, including CANBus, etc. Some networks typically require external network interface adapters that are connected to some general purpose data ports or peripheral buses (1349) (for example, computer system USB ports (1300)); others are typically built into the core of the computer system (1300) by connecting to a system bus as described below (eg, an Ethernet interface to a PC computer system or a cellular network interface to a smartphone computer system). Using any of these networks, the computer system (1300) can communicate with other entities. Such communications may be unidirectional with receive-only capability (eg broadcast), unidirectional with transmit-only capability (eg CANbus to some CANbus devices), or bidirectional, eg to other computer systems using a local area network or a wide area digital network. Several protocols and protocol stacks may be used on each of these networks and the network interfaces described above.

[0159] The aforementioned human-machine interface devices, human-accessible storage devices, and network interfaces can be connected to the core (1340) of the computer system (1300).

[0160] The core (1340) may include one or more central processing units (CPUs) (1341), graphics processing units (GPUs) (1342), specialized programmable processing modules in the form of Field Programmable Gate Arrays (FPGAs) ( 1343), hardware accelerators (1344) for some tasks, etc. These devices, together with read-only memory (ROM) (1345), random access memory (1346), high-capacity internal storage such as user-inaccessible internal hard drives, SSD, etc. (1347) can be connected via the system bus (1348). In some computer systems, the system bus (1348) may be available in the form of one or more physical connectors to allow for expansions with additional CPUs, GPUs, or the like. Peripherals can be connected either directly to the system bus (1348) of the kernel, or via a peripheral bus (1349). Peripheral bus architectures include PCI, USB, and the like.

[0161] CPU (1341), GPU (1342), FPGA (1343), and accelerator (1344) devices may execute some instructions that together may constitute the aforementioned computer code. This computer code may be stored in ROM (1345) or RAM (1346). Temporary data may also be stored in the RAM memory (1346), while permanent data may be stored, for example, in an internal mass storage (1347). Fast storage and retrieval from any storage device can be achieved through the use of cache memory, which can be closely associated with one or more processors CPU (1341), GPU (1342), mass storage (1347), ROM memory (1345) , RAM (1346), etc.

[0162] Computer-readable media may store computer code for performing various computer-executable operations. The media and computer code may be specially created for the purposes of the present invention or may be well known and available to those skilled in the art of computer software.

[0163] By way of example, and not limitation of the invention, a computer system having an architecture (1300), and in particular, a core (1340) can provide functionality due to the execution by the processor(s) (including(and) CPU, GPU , FPGAs, accelerators, etc.) software embodied in one or more tangible computer-readable media. Such computer-readable media may be media associated with the user-accessible mass storage described above, as well as some persistent kernel storage (1340), such as internal mass storage (1347) or ROM (1345). Software implementing various embodiments of the present invention may be stored on such devices and executed by the kernel (1340). The computer-readable medium may include one or more memory devices or microchips, as appropriate. The software may direct the core (1340), and in particular its processors (including CPUs, GPUs, FPGAs, and the like), to execute specific processes or specific portions of the specific processes described herein, including setting up data structures stored in RAM memory (1346 ), and modification of such data structures according to the processes specified by the software. Additionally or alternatively, the computer system may provide functionality through logic hardwired or otherwise embodied in circuitry (e.g., an accelerator (1344)), which may operate in place of, or in conjunction with software, to perform specific processes or specific portions of the specific processes described herein. processes. A reference to software may cover logic, and vice versa when appropriate. Reference to computer-readable media may cover a circuit (eg, an integrated circuit (IC, integrated circuit)) where software is stored for execution, a circuit embodying logic for execution, or both circuits, when appropriate. The present invention encompasses any suitable combination of hardware and software.

[0164] Although some embodiments of the invention have been described here, changes, permutations, and various equivalent substitutions are possible within the scope of the invention.

Claims (26)

1. A method for encoding a point cloud, including: determining (1210), by the processor, one or more seed points in the point cloud that are associated with the reconstructed position, wherein the positions of the one or more seed points are reconstructed in accordance with the quantization of the geometry into the reconstructed position; determining (1220), by the processor, an attribute value for the reconstructed position based on the attribute information of said one or more seed points, and encoding (1230), by the processor, a point cloud texture using a reconstructed position having said attribute value to be determined. 2. The method according to p. 1, also including: storing a data structure that associates one or more original points with a reconstructed position in accordance with the quantization of the geometry. 3. The method according to p. 1, also including: performing, by the processor, an octree-based partitioning that partitions the point cloud space into voxels; and linking, by the processor, one or more seed points, which are located in the voxel of said voxels, with a reconstructed position to represent the voxel. 4. The method according to p. 2, also including: accessing the data structure based on the reconstructed position to retrieve one or more seed points associated with the reconstructed position. 5. The method according to p. 1, also including: calculating, by the processor, an average value of the attributes of the set of seed points as said determined attribute value for the reconstructed position in response to the fact that the set of seed points are associated with the reconstructed position. 6. The method according to p. 1, also including: calculating, by the processor, a weighted average of the attributes of the set of seed points as said determined attribute value for the reconstructed position. 7. The method according to p. 6, also including: assigning a weight, by the processor, to the seed point attribute value in the seed point set based on the inverse distance between the seed point and the reconstructed position. 8. The method according to p. 1, also including: assigning, by the processor, a particular attribute value of the closest point in the seed point set as said attribute value to be determined for the reconstructed position in response to the fact that the seed point set is associated with the reconstructed position. 9. The method according to p. 8, also including: selecting, by the processor, a median attribute value among the attribute values for the nearest points in response to the fact that the nearest points in the set of source points have the same shortest distance to the reconstructed position. 10. The method according to p. 8, also including: calculating, by the processor, an attribute average of the attribute values for the nearest points in response to the fact that the nearest points in the set of source points have the same shortest distance to the reconstructed position. 11. A point cloud encoding device, comprising: processing circuit configured to implement the method according to any one of paragraphs. 1-10. 12. A computer-readable medium on which a program is stored, the program being configured to implement the method according to any one of paragraphs. 1-10 when executed by one or more processors.

Images