EP4602559A1 - Parameterization-guided packing of displacements for dynamic mesh coding - Google Patents

Abstract

An example method includes: receiving, with an encoder for a frame of an input volumetric video mesh: a simplified base mesh with texture coordinates, a set of pre-computed displacements for a chosen subdivision method, and a texture frame; generating a static base mesh encoding that outputs a base mesh substream including encoded and quantized texture coordinates, and that generates an encoded static base mesh; decoding the encoded static base mesh, and generating a reconstructed dequantized base mesh and reconstructed texture coordinates; adapting the pre-computed displacements to the reconstructed dequantized base mesh with applying subdivision to the reconstructed dequantized base mesh; computing a remapping of the reconstructed texture coordinates to a regular grid based on a remapping technique to generate a displacement packing image; signaling metadata describing the remapping technique; and signaling a presence of collided vertices in the displacement packing image along the bitstream.

Description

Parameterization-guided Packing Of Displacements For Dynamic Mesh Coding

TECHNICAL FIELD

[0001] The examples and non-limiting embodiments relate generally to volumetric video coding, and more particularly, to parameterization-guided packing of displacements for dynamic mesh coding.

BACKGROUND

[0002] It is known to perform encoding and decoding of images and video.

BRIEF DESCRIPTION OF THE DRAWINGS

[0003] The foregoing aspects and other features are explained in the following description, taken in connection with the accompanying drawings, wherein:

[0004] FIG. 1 A is a diagram showing volumetric media conversion at an encoder side.

[0005] FIG. IB is a diagram showing volumetric media reconstruction at a decoder side.

[0006] FIG. 2 shows an example of block to patch mapping.

[0007] FIG. 3A shows an example of an atlas coordinate system.

[0008] FIG. 3B shows an example of a local 3D patch coordinate system.

[0009] FIG. 3C shows an example of a final target 3D coordinate system.

[0010] FIG. 4 shows elements of a mesh.

[0011] FIG. 5 shows an example V-PCC extension for mesh encoding, based on the embodiments described herein.

[0012] FIG. 6 shows an example V-PCC extension for mesh decoding, based on the embodiments described herein.

[0013] FIG. 7 shows one subdivision step of a triangle into four triangles by connecting mid points of the initial triangle edges.

[0014] FIG. 8 depicts a multi-resolution analysis of a mesh. [0015] FIG. 9 is block diagram of an encoder composed of a pre-processing module.

[0016] FIG. 10 depicts pre-processing steps at the encoder.

[0017] FIG. 11 is a block diagram of an intra frame encoder scheme.

[0018] FIG. 12 is a block diagram of an inter frame encoder scheme.

[0019] FIG. 13 depicts a decoder scheme composed of a decoder module that demultiplexes and decodes all sub-streams and a post-processing module that reconstructs the dynamic mesh sequence.

[0020] FIG. 14 shows a decoding process in intra mode.

[0021] FIG. 15 shows a decoding process in inter mode.

[0022] FIG. 16 shows a base mesh encoder in a VDMC encoder.

[0023] FIG. 17 shows an example base mesh encoder.

[0024] FIG. 18 shows an example base mesh decoder.

[0025] FIG. 19 illustrates an overview of a base mesh data substream structure.

[0026] FIG. 20 depicts segmentation of a mesh into sub-meshes.

[0027] FIG. 21 illustrates a 2 submesh example.

[0028] FIG. 22A is a visualization of a UV -parameterization of base mesh vertices and triangles.

[0029] FIG. 22B is a visualization of a texture coordinates map after mid-point subdivision.

[0030] FIG. 23 depicts modified encoder scheme embodiments, including an image packing module.

[0031] FIG. 24 depicts modified decoder scheme embodiments, including an image unpacking module.

[0032] FIG. 25 depicts example remapping techniques.

[0033] FIG. 26 illustrates different tiling strategies for different LODs.

[0034] FIG. 27 is an example apparatus to implement the examples described herein. [0035] FIG. 28 shows a representation of an example of non-volatile memory media.

[0036] FIG. 29 is an example method to implement the examples described herein.

[0037] FIG. 30 is example method to implement the examples described herein.

DETAILED DESCRIPTION OF EXAMPLE EMBODIMENTS

[0038] The examples described herein relate to the new standardization activity called Videobased Dynamic Mesh Coding (V-DMC) ISO/IEC 23090-29, which is a new application of the Visual Volumetric Video Coding (V3C) standard family ISO/IEC 23090-5.

[0039] Volumetric video data

[0040] Volumetric video data represents a three-dimensional scene or object and can be used as input for AR, VR and MR applications. Such data describes geometry (shape, size, position in 3D- space) and respective attributes (e.g. color, opacity, reflectance, ...), plus any possible temporal transformations of the geometry and attributes at given time instances (like frames in 2D video). Volumetric video is either generated from 3D models, e.g., CGI, or captured from real-world scenes using a variety of capture solutions, e.g. multi-camera, laser scan, combination of video and dedicated depth sensors, and more. Also, a combination of CGI and real-world data is possible. Typical representation formats for such volumetric data are triangle meshes, point clouds, or voxels. Temporal information about the scene can be included in the form of individual capture instances, e.g., “frames” in 2D video, or other means, e.g. position of an object as a function of time.

[0041] Because volumetric video describes a 3D scene (or object), such data can be viewed from any viewpoint. Therefore, volumetric video is an important format for AR, VR, or MR applications, especially for providing 6DOF viewing capabilities.

[0042] Increasing computational resources and advances in 3D data acquisition devices have enabled reconstruction of highly detailed volumetric video representations of natural scenes. Infrared, lasers, time-of-flight and structured light are all examples of devices that can be used to construct 3D video data. Representation of the 3D data depends on how the 3D data is used. Dense voxel arrays have been used to represent volumetric medical data. In 3D graphics, polygonal meshes are extensively used. Point clouds on the other hand are well suited for applications such as capturing real world 3D scenes where the topology is not necessarily a 2D manifold. Another way to represent 3D data is coding this 3D data as a set of textures and a depth map as is the case in the multi-view plus depth framework. Closely related to the techniques used in multi- view plus depth is the use of elevation maps, and multi- level surface maps.

[0043] MPEG visual volumetric video-based coding (V C)

[0044] Selected excerpts from the ISO/IEC 23090-5 Visual Volumetric Video-based Coding and Video-based Point Cloud Compression 2nd Edition standard are referred to herein.

[0045] Visual volumetric video, a sequence of visual volumetric frames, when uncompressed, may be represented by a large amount of data, which can be costly in terms of storage and transmission. This has led to the need for a high coding efficiency standard for the compression of visual volumetric data.

[0046] The V3C specification enables the encoding and decoding processes of a variety of volumetric media by using video and image coding technologies. This is achieved through first a conversion of such media from their corresponding 3D representation to multiple 2D representations, also referred to as V3C components, before coding such information. Such representations may include occupancy, geometry, and attribute components. The occupancy component can inform a V3C decoding and/or rendering system of which samples in the 2D components are associated with data in the final 3D representation. The geometry component includes information about the precise location of 3D data in space, while attribute components can provide additional properties, e.g. texture or material information, of such 3D data. An example is shown in FIG. 1A and FIG. IB.

[0047] FIG. 1A shows volumetric media conversion at the encoder, and FIG. IB shows volumetric media conversion at the decoder side. The 3D media 102 is converted to a series of 2D representations: occupancy 118, geometry 120, and attribute 122. Additional atlas information 108 is also included in the bitstream to enable inverse reconstruction. Refer to ISO/IEC 23090-5.

[0048] As further shown in FIG. 1A, a volumetric capture operation 104 generates a projection 106 from the input 3D media 102. In some examples, the projection 106 is a projection operation. From the projection 106, an occupancy operation 110 generates the occupancy 2D representation 118, a geometry operation 112 generates the geometry 2D representation 120, and an attribute operation 114 generates the attribute 2D representation 122. The additional atlas information 108 is included in the bitstream 116. The atlas information 108, the occupancy 2D representation 118, the geometry 2D representation 120, and the attribute 2D representation 122 are encoded into the V3C bitstream 124 to encode a compressed version of the 3D media 102. Based on the examples described herein, V-DMC packing signaling 129 may also be signaled in the V3C bitstream 124 or directly to a decoder. The V- DMC packing signaling 129 may be used on the decoder side, as shown in FIG. IB. [0049] As shown in FIG. IB, a decoder using the V3C bitstream 124 derives 2D representations using an occupancy operation 128, a geometry operation 130 and an attribute operation 132. The atlas information operation 126 provides atlas information into a bitstream 134. The occupancy operation 128 derives the occupancy 2D representation 136, the geometry operation 130 derives the geometry 2D representation 138, and the attribute operation 132 derives the attribute 2D representation 140. The 3D reconstruction operation 142 generates a decompressed reconstruction 144 of the 3D media 102, using the atlas information 126/134, the occupancy 2D representation 136, the geometry 2D representation 138, and the attribute 2D representation 140.

[0050] Additional information that allows associating all these subcomponents and enables the inverse reconstruction, from a 2D representation back to a 3D representation is also included in a special component, referred to herein as the atlas. An atlas comprises multiple elements, namely patches. Each patch identifies a region in all available 2D components and includes information necessary to perform the appropriate inverse projection of this region back to the 3D space. The shape of such regions is determined through a 2D bounding box associated with each patch as well as their coding order. The shape of these regions is also further refined after the consideration of the occupancy information.

[0051] Atlases are partitioned into patch packing blocks of equal size. Refer for example to block

202 in FIG. 2, where FIG. 2 shows an example of block to patch mapping. The 2D bounding boxes of patches and their coding order determine the mapping between the blocks of the atlas image and the patch indices. FIG. 2 shows an example of block to patch mapping with 4 projected patches (204, 204- 2, 204-3, 204-4) onto an atlas 201 when asps_patch_precedence_order_flag is equal to 0. Projected points are represented with dark gray. The area that does not include any projected points is represented with light grey. Patch packing blocks 202 are represented with dashed lines. The number inside each patch packing block 202 represents the patch index of the patch (204, 204-2, 204-3, 204-4) to which it is mapped.

[0052] Axes orientations are specified for internal operations. For instance, the origin of the atlas coordinates is located on the top-left corner of the atlas frame. For the reconstruction step, an intermediate axes definition for a local 3D patch coordinate system is used. The 3D local patch coordinate system is then converted to the final target 3D coordinate system using appropriate transformation steps.

[0053] FIG. 3A shows an example of an atlas coordinate system, FIG. 3B shows an example of a local 3D patch coordinate system, and FIG. 3C shows an example of a final target 3D coordinate system. Refer to ISO/IEC 23090-5. [0054] FIG. 3A shows an example of a single patch 302 packed onto an atlas image 304. This patch 302 is then converted, with reference to FIG. 3B, to a local 3D patch coordinate system (U, V, D) defined by the projection plane with origin O’, tangent (U), bi-tangent (V), and normal (D) axes. For an orthographic projection, the projection plane is equal to the sides of an axis-aligned 3D bounding box 306, as shown in FIG. 3B. The location of the bounding box 306 in the 3D model coordinate system, defined by a left-handed system with axes (X, Y, Z), can be obtained by adding offsets TilePatch3dOffsetU 308, TilePatch3DOffsetV 310, and TilePatch3DOffsetD 312, as illustrated in FIG. 3C.

[0055] V3C High Level Syntax

[0056] Coded V3C video components are referred to herein as video bitstreams, while an atlas component is referred to as the atlas bitstream. Video bitstreams and atlas bitstreams may be further split into smaller units, referred to herein as video and atlas sub-bitstreams, respectively, and may be interleaved together, after the addition of appropriate delimiters, to construct a V3C bitstream.

[0057] V3C patch information is included in an atlas bitstream, atlas_sub_bitstream(), which includes a sequence of NAL units. A NAL unit is specified to format data and provide header information in a manner appropriate for conveyance on a variety of communication channels or storage media. All data are included in NAL units, each of which includes an integer number of bytes. A NAL unit specifies a generic format for use in both packet-oriented and bitstream systems. The format of NAL units for both packet-oriented transport and sample streams is identical except that in the sample stream format specified in Annex D of ISO/IEC 23090-5 each NAL unit can be preceded by an additional element that specifies the size of the NAL unit.

[0058] NAL units in an atlas bitstream can be divided into atlas coding layer (ACL) and nonatlas coding layer (non-ACL) units. The former is dedicated to carry patch data, while the latter is dedicated to carry data necessary to properly parse the ACL units or any additional auxiliary data.

[0059] In the nal_unit_header() syntax nal_unit_type specifies the type of the RBSP data structure included in the NAL unit as specified in Table 4 of ISO/IEC 23090-5. nal_layer_id specifies the identifier of the layer to which an ACL NAL unit belongs or the identifier of a layer to which a non- ACL NAL unit applies. The value of nal_layer_id shall be in the range of 0 to 62, inclusive. The value of 63 may be specified in the future by ISO/IEC. Decoders conforming to a profile specified in Annex A of ISO/IEC 23090-5 shall ignore (e.g., remove from the bitstream and discard) all NAL units with values of nal_layer_id not equal to 0.

[0060] V3C extension mechanisms [0061] While designing the V3C specification it was envisaged that amendments or new editions can be created in the future. In order to ensure that the first implementations of V3C decoders are compatible with any future extension, a number of fields for future extensions to parameter sets were reserved.

[0062] For example, the second edition of V3C introduced extensions in VPS related to MIV and the packed video component.

[0063] Rendering and meshes

[0064] A polygon mesh is a collection of vertices, edges and faces that defines the shape of a polyhedral object in 3D computer graphics and solid modeling. The faces usually comprise triangles (triangle mesh), quadrilaterals (quads), or other simple convex polygons (n-gons), since this simplifies rendering, but may also be more generally composed of concave polygons, or even polygons with holes. [0065] With reference to FIG. 4, objects 400 created with polygon meshes are represented by different types of elements. These include vertices 402, edges 404, faces 406, polygons 408 and surfaces 410 as shown in FIG. 4. Thus, FIG. 4 illustrates elements of a mesh.

[0066] Polygon meshes are defined by the following elements:

[0067] Vertex (402): a position in 3D space defined as (x,y,z) along with other information such as color (r,g,b), normal vector and texture coordinates.

[0068] Edge (404): a connection between two vertices.

[0069] Face (406): a closed set of edges 404, in which a triangle face has three edges, and a quad face has four edges. A polygon 408 is a coplanar set of faces 406. In systems that support multi-sided faces, polygons and faces are equivalent. Mathematically a polygonal mesh may be considered an unstructured grid, or undirected graph, with additional properties of geometry, shape and topology.

[0070] Surfaces (410): or smoothing groups, are useful, but not required to group smooth regions.

[0071] Groups: some mesh formats include groups, which define separate elements of the mesh, and are useful for determining separate sub-objects for skeletal animation or separate actors for non- skeletal animation.

[0072] Materials: defined to allow different portions of the mesh to use different shaders when rendered.

[0073] UV coordinates: most mesh formats also support some form of UV coordinates which are a separate 2D representation of the mesh "unfolded" to show what portion of a 2-dimensional texture map to apply to different polygons of the mesh. It is also possible for meshes to include other such vertex attribute information such as color, tangent vectors, weight maps to control animation, etc. (sometimes also called channels).

[0074] V-PCC mesh coding extension (MPEG M49588)

[0075] FIG. 5 and FIG. 6 show the extensions to the V-PCC encoder and decoder to support mesh encoding and mesh decoding, respectively, as proposed in MPEG input document [MPEG M47608].

[0076] In the encoder extension 500, the input mesh data 502 is demultiplexed with demultiplexer 504 into vertex coordinates+attributes 506 and vertex connectivity 508. The vertex coordinates+attributes data 506 is coded using MPEG-I V-PCC (such as with MPEG-I VPCC encoder 510), whereas the vertex connectivity data 508 is coded (using vertex connectivity encoder 516) as auxiliary data 518. Both of these (encoded vertex coordinates and vertex attributes 517 and auxiliary data 518) are multiplexed using multiplexer 520 to create the final compressed output bitstream 522. Vertex ordering 514 is carried out on the reconstructed vertex coordinates 512 at the output of MPEG- I V-PCC 510 to reorder the vertices for optimal vertex connectivity encoding 516.

[0077] Based on the examples described herein, as shown in FIG. 5, the encoding process/apparatus 500 of FIG. 5 may be extended such that the encoding process/apparatus 500 signals packing signaling 530 (e.g. V-DMC packing signaling) within the output bitstream 522. Alternatively, packing signaling 530 may be provided and signaled separately from the output bitstream 522.

[0078] As shown in FIG. 6, in the decoder 600, the input bitstream 602 is demultiplexed with demultiplexer 604 to generate the compressed bitstreams for vertex coordinates+attributes 605 and vertex connectivity 606. The input/compressed bitstream 602 may comprise or may be the output from the encoder 500, namely the output bitstream 522 of FIG. 5. The vertex coordinates+attributes data 605 is decompressed using MPEG-I V-PCC decoder 608 to generate vertex attributes 612. Vertex ordering 616 is carried out on the reconstructed vertex coordinates 614 at the output of MPEG-I V-PCC decoder 608 to match the vertex order at the encoder 500. The vertex connectivity data 606 is also decompressed using vertex connectivity decoder 610 to generate vertex connectivity information 618, and everything (including vertex attributes 612, the output of vertex reordering 616, and vertex connectivity information 618) is multiplexed with multiplexer 620 to generate the reconstructed mesh 622.

[0079] Based on the examples described herein, as shown in FIG. 6, the decoding process/apparatus 600 of FIG. 6 may be extended such that the decoding process/apparatus 600 receives and decodes packing signaling 630 (e.g. V-DMC packing signaling), which may be part of the compressed bitstream 602. The packing signaling 630 of FIG. 6 may comprise or correspond to the packing signaling 530 of FIG. 5. Alternatively, packing signaling 630 may be received and signaled separately from the compressed bitstream 602 or output bitstream 522 (e.g. signaled to the demultiplexer 604 separately from the compressed bitstream 602).

[0080] Generic mesh compression

[0081] Mesh data may be compressed directly without projecting it into 2D-planes, like in V- PCC based mesh coding. In fact, the anchor for V-PCC mesh compression call for proposals (CfP) utilizes off-the shelf mesh compression technology, Draco (https://google.github.io/draco/), for compressing mesh data excluding textures. Draco is used to compress vertex positions in 3D, connectivity data (faces) as well as UV coordinates. Additional per- vertex attributes may be also compressed using Draco. The actual UV texture may be compressed using traditional video compression technologies, such as H.265 or H.264.

[0082] Draco uses the edgebreaker algorithm at its core to compress 3D mesh information. Draco offers a good balance between simplicity and efficiency, and is part of Khronos endorsed extensions for the glTF specification. The main idea of the algorithm is to traverse mesh triangles in a deterministic way so that each new triangle is encoded next to an already encoded triangle. This enables prediction of vertex specific information from the previously encoded data by simply adding delta to the previous data. Edgebreaker utilizes symbols to signal how each new triangle is connected to the previously encoded part of the mesh. Connecting triangles in such a way results on average in 1 to 2 bits per triangle when combined with existing binary encoding techniques.

[0083] V-DMC

[0084] The V-DMC standardization works have started after the completion of the call for proposal (CfP) issued by MPEG 3DG (ISO/IEC SC29 WG 2) on integration of MESH compression into the V3C family of standards (ISO/IEC 23090-5). The retained technology after the CfP result analysis is based on multiresolution mesh analysis and coding. This approach includes:

[0085] 1. Generating a base mesh that is a simplified (low resolution) mesh approximation of the original mesh, called a base mesh (this is done for all frames of the dynamic mesh sequence) m,

[0086] 2. Performing several mesh subdivision iterative steps (e.g., each triangle 700 is converted into four triangles (701, 702, 703, 704) by connecting the triangle edge midpoints as illustrated on Fig. 7) on the generated base mesh, generating other approximation meshes m", where n stands for the number of iterations with m, = m°,

[0087] 3. Defining displacement vectors di, also named error vectors, for each vertex of each mesh approximation m", with n > 0, noted d"i

[0088] 4. For each subdivision level, the deformed mesh, obtained by m", + d"i, e.g., by adding the displacement vectors to the subdivided mesh vertices generates the best approximation of the original mesh at that resolution, given the base mesh and prior subdivision levels.

[0089] 5. The displacement vectors may undergo a lazy wavelet transform prior to compression.

[0090] 6. The attribute map of the original mesh is transferred to the deformed mesh at the highest resolution (e.g., subdivision level) such that texture coordinates are obtained for the deformed mesh and a new attribute map is generated.

[0091] The scheme is illustrated on Fig. 8. FIG. 7 shows one subdivision step of a triangle 700 into four triangles (701, 702, 703, 704) by connecting mid points of the initial triangle edges. FIG. 8 shows a multi -resolution analysis of a mesh. Abase mesh (left, 802) undergoes a first step of subdivision and error vectors are added to each vertex (arrows of 804), after a series of iterative subdivision and displacements, the highest resolution mesh is generated (right, 808 from 806). The connectivity of the highest resolution deformed mesh 808 is generally different from the original mesh 802, however, the geometry of the deformed mesh 808 is a good approximation of the original mesh geometry.

[0092] The encoding process 900 can be separated into two main modules: the pre-processing module 902 and the actual encoder module 904 as illustrated on Fig. 9. FIG. 9 shows that the encoder 901 is composed of a pre-processing module 902 that generates a base mesh 906 and the displacement vectors 908, given the input mesh sequence 903 and its attribute maps 905. The encoder module 904 generates the compressed bitstream 910 by ingesting the inputs and outputs of the pre-processing module 902. The encoder 904 provides feedback 912 to pre-processing 902.

[0093] The pre-processing 902 includes mainly three steps: decimation 1002 (reducing the original mesh resolution to produce a base mesh 906 and a decimated mesh 1004), uv-atlas isocharting 1006 (creating a parameterization of the base mesh and generating a parameterized decimated mesh 1008) and the subdivision surface fitting 1010 as illustrated on FIG. 10. Thus, FIG. 10 shows the preprocessing steps 902 at the encoder 901.

[0094] The encoder is illustrated on FIG. 11 and FIG. 12 for the INTRA case (1101) and INTER case (1201) respectively. In the latter 1201, the base mesh connectivity of the first frame of a group of frames is imposed to the subsequent frame’s base meshes to improve compression performance. FIG. 11 shows an intra frame encoder scheme 1100.

[0095] FIG. 11 shows the encoder process 1100 for INTRA frame encoding. Inputs to this module are the base mesh 906 (that is an approximation of the input mesh 903 but that includes less faces and vertices), the patch information 1102 related to the input base mesh 906, the displacements 908, the static/dynamic input mesh frame 903 and the attribute map 905. Outputs of this module is a compressed bitstream 910 that includes a V3C extended signaling sub-bitstream including patch data information 1102, compressed base mesh substream 1104, a compressed displacement video component substream 1106 and a compressed attribute video component sub-bitstream 1108. The module 1101 takes the input base mesh 906 and first quantizes its data in the quantization module 1110, which can be dynamically tuned by a control module 1112. The quantized base mesh is then encoded with the static mesh encoder module 1114, which outputs a compressed base mesh sub-bitstream 1104 that is muxed 1116 in the output bitstream 910. The encoded base mesh is decoded in the static mesh decoder module 1118 that generates a reconstructed quantized base mesh 1120. The update displacements module 1122 takes as input the reconstructed quantized base mesh 1120, the pristine base mesh 906 and the input displacements 908 to generate new updated displacements 1124 that are remapped to the reconstructed base mesh data in order to avoid precision errors due to the static mesh encoding and decoding process. The updated displacements 1124 are filtered with a wavelet transform in the wavelet transform module 1126 (that also takes as input the reconstructed base mesh 1120) and then quantized in the quantization module 1128. The quantized wavelet coefficients 1130 produced from the updated displacements 1124 are then packed into a video component in the image packing module 1132. This video component 1133 is then encoded with a 2D video encoder such as HEVC, VVC, etc., in the video encoder module 1134, and the output compressed displacement video component sub-bitstream 1106 is muxed 1116 along with the V3C signaling information sub-bitstream 1102 into the output compressed bitstream 910. Then the compressed displacement video component is first decoded and reconstructed and then unpacked into encoded and quantized wavelet coefficients 1138 in the image unpacking module 1136. These wavelet coefficients 1138 are then unquantized in the inverse quantization module 1140 and reconstructed with the inverse wavelet transform module 1142 that generates reconstructed displacements 1144. The reconstructed base mesh 1146 is unquantized in the inverse quantization module 1148 and the unquantized base mesh 1146 is combined with the reconstructed displacements 1144 in the reconstruct deformed mesh module 1150 to obtain the reconstructed deformed mesh 1152. This reconstructed deformed mesh 1152 is then fed into the attribute transfer module 1154 together with the attribute map 905 produced by the pre-processing 902 and the input static/dynamic mesh frame 903. The output of the attribute transfer module is an updated attribute map 1156 that now corresponds to the reconstructed deformed mesh frame 1152. The updated attribute map 1154 is then padded 1156, undergoes color conversion 1158, and is encoded as a video component with a 2D video codec such as HEVC or VVC, in the padding 1156, color conversion 1158 and video encoder 1160 modules respectively. The output compressed attribute map bitstream 1108 is multiplexed 1116 into the encoder output bitstream 910.

[0096] The V3C signaling information sub-bitstream corresponds to the long horizontal arrow at the top of the FIG. 11 that corresponds to the input patch information 1102 (derived in the preprocessing step). Additional metadata is added to this patch data information during the encoding but that is not represented in FIG. 11.

[0097] FIG. 12 shows an inter frame encoder scheme 1200, similar to the intra case 1100, but with the base mesh connectivity being constrained for all frames of a group of frames. A motion encoder 1202 is used to efficiently encode displacements between base meshes compared to the base mesh of the first frame of the group of frames.

[0098] The inter encoding process 1200 is similar to the intra encoding process with the following changes. The reconstructed reference base mesh 1146 is an input of the inter coding process. A new module called motion encoder 1202 takes as input the quantized input base mesh 906 and the reconstructed quantized reference base mesh 1146 to produce compressed motion information encoded as a compressed motion bitstream 1204, which is multiplexed 1206 into the encoder output compressed bitstream 910. All other modules and processes are similar to the intra encoding case (1100, 1101).

[0099] The compressed bitstream 910 generated by the encoder 1201 multiplexes (1206): a subbitstream with the encoded base mesh using a static mesh codec, a sub-bitstream 1204 with the encoded motion data using an animation codec for base meshes in case INTER coding is enabled, a sub-bitstream 1208 with the wavelet coefficients of the displacement vectors packed in an image and encoded using a video codec 1209, a sub-bitstream 1210 with the attribute map encoded using a video codec 1212, and a sub-bitstream that includes all metadata required to decode and reconstruct the mesh sequence based on the aforementioned sub-bitstreams. The signaling of the metadata is based on the V3C syntax and includes necessary extensions that are specific to meshes.

[0100] The decoding process 1300 is illustrated on FIG. 13. First the compressed bitstream 910 is demultiplexed with decoder 1301 into sub-bitstreams that are reconstructed, e.g., metadata 1302, reconstructed base mesh 1304, reconstructed displacements 1306 and the reconstructed attribute map data 1308. The reconstruction of the mesh sequence is performed based on that data in the postprocessing module 1310.

[0101] Thus, FIG. 13 depicts the decoder scheme 1300 composed of a decoder module 1201 that demultiplexes (demuxes) and decodes all sub-streams and a post-processing module 1310 that reconstructs the dynamic mesh sequence to generate output mesh 1312 and output attributes 1314.

[0102] FIG. 14 and FIG. 15 illustrate the decoding process in INTRA mode 1400 and INTER mode 1500 respectively.

[0103] FIG. 14 depicts the decoding process 1400 in intra mode using intra frame decoding 1401. The intra frame decoding process includes the following modules and processes. First the input compressed bitstream is de-multiplexed 1402 into V3C extended atlas data information (or patch information) 1403, a compressed static mesh bitstream 1405, a compressed displacement video component 1407 and a compressed attribute map bitstream 1409, respectively. The static mesh decoding module 1404 converts the compressed static mesh bitstream 1405 into a reconstructed quantized static mesh 1411, which represents a base mesh. This reconstructed quantized base mesh 1411 undergoes inverse quantization in the inverse quantization module 1406 to produce a decoded reconstructed base mesh 1413. The compressed displacement video component bitstream 1407 is decoded in the video decoding module 1408 to generate a reconstructed displacement video component 1415. This displacement video component 1415 is unpacked into reconstructed quantized wavelet coefficients in the image unpacking module 1410. Reconstructed quantized wavelet coefficients are inverse quantized in the inverse quantization module 1412 and then undergo an inverse wavelet transform in the inverse wavelet transform module 1414, that produces decoded displacement vectors 1416. The reconstruct deformed mesh module 1418 takes into account the patch information and takes as input the decoded reconstructed base mesh 1413 and decoded displacement vectors 1416 to produce the output decoded mesh frame 1312. The compressed attribute map video component 1409 is decoded with video decoder 1420, and possibly undergoes color conversion 1422 to produce a decoded attribute map frame 1314 that corresponds to the decoded mesh frame 1312.

[0104] FIG. 15 depicts the decoding process 1500 in inter mode using inter frame decoding 1501. The inter decoding process 1500 is similar to the intra decoding process module 1401 with the following changes. The decoder also demultiplexes a compressed information bitstream 1503. A decoded reference base mesh 1502 is taken as input of a motion decoder module 1504 together with the compressed motion information sub-bitstream 1503. This decoded reference base mesh 1502 is selected from a buffer of previously decoded base mesh frames (by the intra decoder process 1401 for the first frame of a group of frames). The reconstruction of base mesh module 1506 takes the decoded reference base mesh 1502 and the decoded motion information 1505 as input to produce a decoded reconstructed quantized base mesh 1507. All other processes are similar to the intra decoding process 1401.

[0105] The signaling of the metadata and substreams produced by the encoder 901 and ingested by the decoder 1301 was proposed as an extension of V3C in the technical submission to the dynamic mesh coding CfP, and should be considered as purely indicative. It is as follows and mainly includes additional V3C unit header syntax, additional V3C unit payload syntax, and a mesh intra patch data unit.

[0106] V3C unit header syntax

[0107] V3C unit payload syntax

[0108] Mesh Intra patch data unit

[0109] A refinement of the metadata and substreams signaling is being discussed and is as follows.

[0110] Base meshes are the output of the base mesh substream decoder.

[0111] A submesh is a set of vertices, their connectivity and the associated attributes which can be decoded completely independently in a mesh frame. Each base mesh can have one or more submeshes.

[0112] Resampled base meshes are the output of the mesh subdivision process. The inputs to the process is the base meshes(or sets of submeshes) as well the information from the atlas data substream on how to subdivide/resample the meshes(submeshes).

[0113] A displacement video is the output of the displacement decoder. The inputs to the process is the decoded geometry video as well the information from the atlas data substream on how to interpret/process this video. The displacement video includes displacement values to be added to the corresponding vertices.

[0114] . facegroupld is one of the attribute types assigned to each triangle face of the resampled base meshes. Facegroupld can be compared with the ids of the subparts in a patch to determine the corresponding facegroups to the patch. If facegrould is not conveyed through the base mesh substream decoder, it is derived by the information in the atlas data substream.

[0115] V3C unit

[0116] Compressed base meshes are signaled in a new substream, named as the Base Mesh data substream (unit type V3C_MD). As with other v3c units, the unit type, and its associated v3c parameter set id and atlas id are signaled in the v3c_unit_header().

[0117] V3c parameter set extension

[0118] A new extension needs to be introduced in the v3c_parameter_set syntax structure to handle V-DMC. Several new parameters are introduced in this extension including the following:

[0119] vps_ext_mesh_data_facegroup_id_attribute_present_flag equals 1 indicates that one of the attribute types present in the base mesh data stream is the facegroup Id. [0120] vps_ext_mesh_data_attribute_count indicates the number of total attributes in the base mesh including both the attributes signaled through the base mesh data substream and the attributes signaled in the video sub streams (using ai_attribute_count). When vps_ext_mesh_data_facegroup_id_attribute_present_flag equals 1, it shall be greater or equal to ai_attribute_count+l. This can be constrained by profile/levels.

[0121] The types of attributes that are signaled through the base mesh substream and not through the video substreams are signaled are signaled as vps_ext_mesh_attribute_type data types. When vps_ext_mesh_data_facegroup_id_attribute_present_flag equals 1, one of the vps_ext_mesh_attribute_type must be a facegroup_id.

[0122] vps_ext_mesh_data_substream_codec_id indicates the identifier of the codec used to compress the base mesh data. This codec may be identified through the profiles a component codec mapping SEI message, or through means outside this document.

[0123] vps_ext_attribute_frame_width[i] and vps_ext_attribute_frame_height[i] indicate the corresponding width and height of the video data corresponding to the i-th attribute among the attributes signaled in the video substreams.

[0124] Atlas sequence parameter set extension

[0125] The information included in this extension can be overwritten by the same information in the AFPS extension or the patch data units. The following parameters are introduced:

[0126] asps_vmc_ext_prevent_geometry_video_conversion_flag prevents the outputs of the geometry video substream decoder from being converted. When the flag is true, the outputs are used as they are without any conversion process from Annex B of ISO/IEC 23090-5. When the flag is true, the size of geometry video shall be same as nominal video sizes indicated in the bitstream.

[0127] asps_vmc_ext_prevent_attribute_video_conversion_flag prevents the outputs of attribute video substream decoder from being converted. When the flag is true, the outputs are used as they are without any conversion process from Annex B of ISO/IEC 23090-5. When the flag is true, the size of attribute video shall be same as nominal video sizes indicated in the bitstream.

[0128] asps_vmc_ext_subdivision_method and asps_vmc_ext_subdivision_iteration_count signal information about the subdivision method.

[0129] asps_vmc_ext_transform_index indicates the transform applied to the displacement. The transform index can indicate any transform is not applied. When the transform is LINEAR_LIFTING, the necessary parameters are signaled as vmc_lifting_transform_parameters. [0130] asps_vmc_ext_patch_mapping_method indicates how to map a subpart of a submesh to a patch. When asps_vmc_ext_patch_mapping_method is equal to 0, all the triangles in the corresponding submesh are associated with the current patch. In this case, there is only one patch associated with the submesh. When asps_vmc_ext_patch_mapping_method is equal to 1, the subpart_ids are explicitly signaled in the mesh patch data unit to indicate the associated subparts. In other cases, the triangle faces in the corresponding submesh are divided into subparts by the method indicated by asps_vmc_ext_patch_mapping_method.

[0131] asps_vmc_ext_tjunction_removing_method indicates the method to remove t- junctions created by different subdivision methods or by different subdivision iterations of two triangles sharing an edge.

[0132] asps_vmc_ext_num_attribute indicates the total number of attributes that the corresponding mesh carries. Its value shall be less or equal to vps_ext_mesh_data_attribute_count.

[0133] asps_vmc_ext_attribute_type is the type of the i-th attribute, and it shall be one of ai_attribute_type_ids or vps_ext_mesh_attribute_types.

[0134] asps_vmc_ext_direct_atrribute_projection_enabled_flag indicates that the 2d locations where attributes are projected are explicitly signaled in the mesh patch data units. Therefore, the projection id and orientation index in V3C V-PCC ISO/IEC 23090-5:2021 can be also used as in ISO/IEC 23090-5:2021.

[0135] The asps_vmc_extension() may be as follows:

[0136] The vmc_lifting_transform_parameters syntax element may be as follows:

[0137] Atlas Frame Parameter set extension

[0138] afps_vmc_ext_single_submesh_in_frame_flag indicates there is only one submesh for the mesh frame

[0139] When afps_vmc_ext_overriden_flag in afps_vmc_extension() is tree, the subdivision method, displacement coordinate system, transform index, transform parameters, and attribute transform parameters can be signaled again and the information overrides the one signaled in asps_vmc_extension() .

[0140] afps_vmc_ext_single_attribute_tile_in_frame_flag indicates there is only one tile for each attribute signaled in the video streams.

[0141] The afps_vmc_extension syntax element may be as follows:

[0142] afps_ext_vmc_attribute_tile_information() includes the tile information for the attributes signaled through the video substreams.

[0143] Atlas Tile Header [0144] A tile can be associated with one or more submeshes whose id is ath_submesh_id.

[0145] Patch data unit

[0146] As with the V-PCC Patch data units, Mesh patch data units are signaled in the Atlas data substream. Mesh Intra patch data unit, Mesh Inter patch data unit, Mesh Merge patch data unit, and Mesh Skip patch data unit can be used.

[0147] The patch_information_data syntax element may be as follows:

[0148] mdu_submesh_id indicate which submesh the patch is associated with among those indicated in the atlas tile header.

[0149] mdu_vertex_count_minusl and mdu_triangle_count_minusl indicate the number of vertices and triangles associated with the current patch.

[0150] When asps_vmc_ext_patch_mapping_method is not 0, the syntax elements mdu_num_subparts and mdu_subpart_id are signaled. When asps_vmc_ext_patch_mapping_method is 1, the associated triangle faces are the union of the triangle faces whose facegroupld is equal to mdu_subpart_id.

[0151] When mdu_patch_parameters_enable_flag is true, the subdivision method, displacement coordinate system, transform index, transform parameters, and attribute transform parameters can be signaled again and the information overrides the corresponding information signaled in in asps_vmc_extension().

[0152] The mesh_inter_data_unit syntax element may be as follows:

[0153] The mesh_merge_data_unit may be as follows:

[0154] The mesh_skip_data_unit syntax element may be as follows:

[0155] The mesh_raw_data_unit syntax element may be as follows:

[0156] The signaling of the base mesh substream is also under investigation and may be as shown in FIG. 16 (output bitstream 1602), FIG. 17 (base-mesh bitstream 1702), and FIG. 18 (input base-mesh bitstream 1802).

[0157] One of the main features of the current V-DMC specification design is the support for a base mesh signal that can be encoded using any currently or future specified static mesh codec. For example, such information could be coded using Draco 3D Graphics Compression. This representation could provide the basis for applying other decoded information to reconstruct the output mesh frame within the context of V-DMC.

[0158] Furthermore, for coding dynamic mesh frames, it is highly desirable to be able to exploit any temporal correlation that may exist with previously coded base mesh frames. In a design (see FIG. 16, FIG. 17 and FIG. 18), this was accomplished by encoding (with 1604, 1704) a mesh motion field instead of directly encoding the base mesh (1601, 1701), and using this information and a previously encoded base mesh to reconstruct the base mesh of the current frame (1606, 1608, 1706, 1708, 1806, 1808). This approach could be seen as the equivalent of inter prediction in video coding.

[0159] It is highly desirable also to associate all coded base mesh frames or motion fields with information that could help determine their decoding output order as well as their referencing relationships. It is possible, for example, that better coding efficiency could be achieved if the coding order of all frames does not follow the display order or by using as reference for generating a motion field for frame N an arbitrary previously coded motion field or base mesh instead of the immediately previous coded one. Also highly desirable is the ability to instantly detect random access points and independently decode multiple sub-meshes that together can form a single mesh, much like subpictures in video compression.

[0160] For the above reasons, a new Base Mesh Substream format is introduced. This new format is very similar to a video coding format such as HEVC or the atlas sub-bitstream used in V3C, with the base mesh sub-bitstream also constructed using NAL units. Referring to FIG. 19, high level syntax (HLS) structures such as base mesh sequence parameter sets 1902, base mesh frame parameter sets 1904, and submesh layer 1906 are also specified. An overview of this bitstream 1900 with its different subcomponents is shown in FIG. 19. Thus, FIG. 19 is an overview of a base mesh data substream structure 1900.

[0161] One of the desirable features of this design is the ability to segment a mesh into multiple smaller partitions, referred to in this document as submeshes (FIG. 20). FIG. 20 shows segmentation of a mesh 2002 into sub-meshes (2004 and 2006). These submeshes (2004, 2006) can be decoded completely independently, which can help with partial decoding and spatial random access. Although it may not be a requirement for all applications, some applications may require that the segmentation in submeshes remains consistent and fixed in time. The submeshes do not need to use the same coding type, e.g., for one frame one submesh may use intra coding while for another inter coding could be used at the same decoding instance, but it is commonly a requirement that the same coding order is used and the same references are available for all submeshes corresponding at a particular time instance. Such restrictions can help guarantee proper random access capabilities for the entire stream. An example where two submeshes are used is shown in FIG. 21.

[0162] FIG. 21 shows picture order count 0 having submesh 2102 and submesh 2104, picture order count 1 having submesh 2112 and submesh 2114, and picture order count 2 having submesh 2120 and submesh 2122. Shown also is base mesh frame parameter set 2130 and base mesh sequence parameter set 2140.

[0163] NAL unit syntax

[0164] As discussed earlier, the new bitstream is also based on NAL units, and it is similar to those of the atlas substream in V3C. The syntax is provided below.

[0165] General NAL unit syntax

[0166] The bmesh_nal_unit_header syntax element may be as follows:

[0167] NAL unit header syntax

[0168] The bmesh_nal_unit_header syntax element may be as follows:

[0169] NAL unit semantics

[0170] This section includes some of the semantics that correspond to the above syntax structures. More details would provided for syntax elements that have not been defined in complete detail.

[0171] 1. General NAL unit semantics

[0172] NumBytesInNalUnit specifies the size of the NAL unit in bytes. This value is required for decoding of the NAL unit. Some form of demarcation of NAL unit boundaries is necessary to enable inference of NumBytesInNalUnit. One such demarcation method is specified in Annex TBD for the sample stream format. Other methods of demarcation can be specified outside this document.

[0173] The mesh coding layer (MCL) is specified to efficiently represent the content of the mesh data. The NAL is specified to format that data and provide header information in a manner appropriate for conveyance on a variety of communication channels or storage media. All data are included in NAL units, each of which includes an integer number of bytes. A NAL unit specifies a generic format for use in both packet-oriented and bitstream systems. The format of NAL units for both packet-oriented transport and sample streams is identical except that in the sample stream format specified in Annex TBD each NAL unit can be preceded by an additional element that specifies the size of the NAL unit.

[0174] rbsp_byte[ i ] is the i-th byte of an RBSP. An RBSP is specified as an ordered sequence of bytes as follows:

[0175] The RBSP includes a string of data bits (SODB) as follows: If the SODB is empty (e.g., zero bits in length), the RBSP is also empty; Otherwise, the RBSP includes the SODB as follows:

[0176] The first byte of the RBSP includes the first (most significant, left-most) eight bits of the SODB; the next byte of the RBSP includes the next eight bits of the SODB, etc., until fewer than eight bits of the SODB remain.

[0177] The rbsp_trailing_bits( ) syntax structure is present after the SODB as follows: i)The first (most significant, left-most) bits of the final RBSP byte include the remaining bits of the SODB (if any); ii)The next bit comprises a single bit equal to 1 (e.g., rbsp_stop_one_bit); iii)When the rbsp_stop_one_bit is not the last bit of a byte-aligned byte, one or more bits equal to 0 (e.g., instances of rbsp_alignment_zero_bit) are present to result in byte alignment.

[0178] Syntax structures having these RBSP properties are denoted in the syntax tables using an "_rbsp" suffix. These structures are carried within NAL units as the content of the rbsp_byte[ i ] data bytes. The association of the RBSP syntax structures to the NAL units is as specified in Table 4.

[0179] When the boundaries of the RBSP are known, the decoder can extract the SODB from the RBSP by concatenating the bits of the bytes of the RBSP and discarding the rbsp_stop_one_bit, which is the last (least significant, right-most) bit equal to 1, and discarding any following (less significant, farther to the right) bits that follow it, which are equal to 0. The data necessary for the decoding process is included in the SODB part of the RBSP.

[0180] 2. NAL unit header semantics

[0181] Similar NAL unit types, as for the atlas case, were defined for the base mesh enabling similar functionalities for random access and segmentation of the mesh. Unlike the atlas that is split into tiles, in this document we define the concept of a sub-mesh and define specific nal units that correspond to coded mesh data. In addition, NAL units that can include metadata such as SEI messages are also defined.

[0182] In particular, the base mesh NAL unit types supported are specified as follows:

[0183] The names for the types could be changed to avoid confusion with the atlas ones.

[0184] Raw byte sequence payloads, trailing bits, and byte alignment syntax

[0185] Base mesh sequence parameter set RBSP syntax

[0186] As with similar bitstreams, the primary syntax structure that is defined for a base mesh bitstream is a sequence parameter set. This syntax structure includes basic information about the bitstream, identifying features for the codecs supported for either the intra coded and inter coded meshes, as well as information about references

[0187] 1.1 General base mesh sequence parameter set RBSP syntax

[0188] bmsps_log2_max_mesh_frame_order_cnt_lsb_minus4, bmsps_max_dec_mesh_frame_ bufferin g_minus 1 , bmsps_long_term_ref_mesh_frames_flag, bmsps_num_ref_mesh_frame_lists_in_ bmsps, bmesh_ref_list_struct( i ) are equivalent to those in ASPS.

[0189] bmsps_intra_mesh_codec_id indicates the static mesh codec used to encode the base meshes in this base mesh substream. It could be associated with a specific mesh or motion mesh codec through the profiles specified in the corresponding specification, or could be explicitly indicated with an SEI message as is done in the V3C specification for the video sub-bitstreams.

[0190] bmsps_intra_mesh_data_size_precision_bytes_minusl (+1) specifies the precision, in bytes, of the size of the coded mesh data.

[0191] bmsps_inter_mesh_codec_present_flag indicates when a specific codec indicated by bmsps_inter_mesh_codec_id is used to encode the inter predicted submeshes

[0192] bmsps_inter_mesh_data_size_precision_bytes_minusl(+l) specifies the precision, in bytes, of the size of the inter predicted mesh data. This precision is signaled considering the size of the coded mesh data and the inter predicted mesh data(e.g. motion field) can be significantly different.

[0193] bmsps_facegroup_segmentation_method indicates how facegroups could be derived for a mesh. A facegroup is a set of triangle faces in a submesh. Each triangle face is associated with a Facegroupld indicating the facegroup it belongs to. When bmsps_facegroup_segmentation_method is 0, then Facegroupld is present directly in the coded submesh. Other values indicate that the facegroup can be derived using different methodologies based on the characteristics of the stream. For example, value 1 means that there is no Facegroupld associated with any face. A value 2 means that all faces are identified with a single ID, a 3 that facegroups are identified based on the connected component method, while a value of 4 indicates that each individual face has its own unique ID. Currently ue(v) is used to indicate bmsps_facegroup_segmentation_method, but fixed length coding or partitioning to more elements could have been used instead.

[0194] 1.2 Base Mesh Profile, tier, and level syntax

[0195] bmptl_extended_sub_profile_flag providing support for sub profiles can be quite useful for further restricting the base mesh profiles depending on usage and applications.

[0196] 1.3 Base mesh frame parameter set RBSP syntax

[0197] The base mesh frame parameter set has the frame level information such as number of submeshes in the frames corresponding to one mfh_mesh_frm_order_cnt_lsb. A submesh is coded in one mesh_data_submesh_layer() and is independently decodable from other submeshes. In the case of inter frame prediction, a submesh can refer only to the submeshes with the same smh_id in its associated reference frames. The mechanism is equivalent to what is specified in 8.3.6.2.2 in V3C.

[0198] The mechanism is equivalent to the Atlas frame tile information syntax (8.3.6.2.2 in

V3C).

[0199] The bmesh_sub_mesh_information syntax element may be as follows:

[0200] 1.4 Base mesh submesh layer rbsp syntax

[0201] 1.4.1 bmesh submesh layer rbsp syntax

[0202] A bmesh_submesh_layer includes a submesh information. One or more bmesh_submesh_layer_rbsp can correspond to one mesh frame indicated by mfh_mesh_frm_order_cnt_l sb .

[0203] 1.4.2 submesh header syntax

[0204] The mechanism is equivalent to the atlas tile header (8.3.6.11 in the standard)

[0205] smh_id is the id of the current submesh included in the mesh data submesh data.

[0206] smh_type indicates how the mesh is coded. When smh_type is I_SUBMESH, the mesh data is coded with the indicated static mesh codec. When sinh lype is P_SUBMESH, inter prediction is used to code the mesh data.

[0207] 1.4.3 Submesh data unit

[0208] smdu_intra_sub_mesh_unit( unitSize ) includes a sub mesh unit stream of size unitSize, in bytes, as an ordered stream of bytes or bits within which the locations of unit boundaries are identifiable from patterns in the data. The format of such sub mesh unit stream is identified by a 4CC code as defined by bmptl_profile_codec_group_idc or by a component codec mapping SEI message.

[0209] smdu_inter_sub_mesh_unit( unitSize ) includes a sub mesh unit stream of size unitSize, in bytes, as an ordered stream of bytes or bits within which the locations of unit boundaries are identifiable from patterns in the data. The format of such sub mesh unit stream is identified by a 4CC code as defined by bmptl_profile_codec_group_idc or by a component codec mapping SEI message.

[0210] The V-DMC approach uses an image-based packing of displacement vectors that is suboptimal. While displacements are arranged in a Morton order (see background section), this does not ensure that neighboring pixels of the packed displacements correspond to displacements of neighboring vertices at any level-of-detail (LOD). As video codecs use blocks of connected pixels (macroblocks for AVC, Coding Units for HEVC etc.) to perform intra or inter predictions and DCT- transform, the fact that neighboring pixels may not correspond to displacements that are similar to each other, leads to poor compression performance.

[0211] It can be observed that there is redundancy between neighboring displacements on a surface with or without wavelet transform (the residual after wavelet transform still shows local smoothness as it derives from the local curvature). It is therefore important to design a packing strategy that leads to the smallest possible image resolution, while maintaining the correspondence between displacement neighborhoods (globally or at LOD level) and the packed pixels neighborhoods. This is also true for the temporal aspect, the location of pixels neighborhoods from frame to frame should be similar to enable an efficient compression performance.

[0212] Suboptimal packing of displacement vectors not only have negative implications on the bitrate as described above but also increase the reconstruction artefacts after lossy video decoding operations.

[0213] In the V-DMC framework, the uv-coordinates of the base mesh vertices are encoded in the base mesh substream and their reconstructed values are available at both the encoder and the decoder (e.g., in closed loop).

[0214] FIG. 22A and FIG. 22B show parameterization and UV texture coordinates. FIG. 22A is a visualization 2201 of the UV -parameterization of the base mesh vertices and triangles. FIG. 22B is a visualization 2202 of the texture coordinates map after mid-point subdivision (three iterations). Displacement vectors are assigned to each vertex (position in the UV-map) of each LOD, starting from the base mesh to the highest LOD level. UV-coordinates of vertices of a given LOD-level are obtained by mid-point subdivision (or another subdivision as flagged in the V-DMC bitstream).

[0215] As depicted on FIG. 22, the uv-coordinates represent a parameterization in 2D of the base mesh, which is a mapping in 2D of the mesh connectivity that preserves the areas and shapes of the 3D triangles. [0216] This mapping has the interesting advantage that it preserves local neighborhoods inside submeshes or patches. However, it does not directly map vertices/vertex positions to a regular grid that can be compressed with video codecs. We define ways to exploit the uv-map to define the displacement vectors packing into images by a process that is performed in the same way at the decoder and encoder, and which maps the irregular sampling parameterization to a regular pixel grid.

[0217] Several embodiments are described herein that cover the following: an encoding process for generating the displacement image packing based on the reconstructed parameterization, for each LOD, a decoding process for assigning the decoded packed displacement vectors to their corresponding vertices, for each LOD, and signaling required for unpacking the displacement vectors including two classes of pixels: empty pixels (when no displacement vector is attached to the pixel) and duplicated pixels (when more than one displacement vector is attached to the pixel).

[0218] The following embodiments are based on the fact that the static mesh decoder included in both the V-DMC encoder and V-DMC decoder output a reconstructed base mesh including reconstructed texture coordinates.

[0219] Any parameterization method can be used to generate the texture coordinates in the V- DMC framework and for the examples described herein. This being said, locally isometric maps (e.g. iso-charts) are preferred as they minimize distortions once the texture charts (or patches) are mapped on the mesh based on the uv-coordinates. Other possible parameterization techniques include harmonic maps, conformal maps or signal-specialized parameterizations, for example.

[0220] The encoder modules and decoder modules that are modified are illustrated in FIG. 23 and FIG. 24, respectively. Intra frame encoder 2301 of FIG. 23 modifies the encoder module 1101 shown in FIG. 11, and intra frame decoder 2401 of FIG. 24 modifies the decoder module 1401 shown in FIG. 14. While illustrated for the intra coding case, the figures remain valid for the inter case (encoder 1201 of FIG. 12 and decoder 1501 of FIG. 15), mutatis mutandis. The modified image packing module at the encoder and modified image unpacking at the decoder are described hereafter.

[0221] FIG. 23 shows modified encoder scheme 2300 embodiments, including intra frame encoder 2301. The image packing module 2332 takes reconstructed texture coordinates (2320) from the static mesh decoder 2318 and performs a reparameterization to compute displacement pixel positions. The same applies for the inter case. Signaling is provided to instruct the decoder on how to deal with unused pixels or vertex collisions in the packed image.

[0222] FIG. 24 depicts modified decoder scheme 2400 embodiments, including intra frame decoding 2401. The image unpacking module (2410) takes reconstructed texture coordinates (2411) from the static mesh decoder 2404, extracts the corresponding signaling information regarding unused pixels and collisions, and performs a remapping to compute the vertex indices corresponding to the decoded pixel positions for each displacement vector. The same applies for the inter case.

[0223] The principles behind the remapping techniques applied in both the encoder and decoder are illustrated on FIG. 25 with a simple example.

[0224] FIG. 25 shows example remapping techniques. On the top left, an example original parameterization 2502 with an overlaid grid that represents a possible image packing (3x3 pixels 2503) that is suboptimal as many vertices 2501 (and their displacement vectors) are mapped to the same pixels 2503. On the top right (2504), the pixel-distance-based approach uses the shortest distance between two vertices in the optimal to define a regular grid (10x8 pixels 2503 in this example), which avoids the problem of having multiple vertices 2501 being mapped to the same vertex but that generates many unused vertices. On the bottom left (2506), a relaxation with boundary mapping technique is illustrated, which performs a harmonic reparameterization constrained by a 3x4 pixels grid (each pixel as 2503), whose size was chosen based on the submesh boundary. This approach still generates collisions (multiple vertices mapped to the same pixel illustrated in pixel 2503-1). On the bottom right (2508), a variation of the relaxation-based approach is illustrated, where the boundary constraint is relaxed to allow a grid with a sufficiently high resolution to avoid vertex collisions at the cost of generating unused pixels (shown are unused pixels 2503-2 and 2503-3.

[0225] Encoder embodiments:

[0226] An encoder receiving for each frame of an input volumetric video mesh: a simplified base mesh with texture coordinates, a set of pre-computed displacements for a chosen subdivision method, a texture frame, generating a static base mesh encoding that outputs a base mesh substream including encoded and quantized texture coordinates, decoding the encoded static base mesh and generating reconstructed dequantized base mesh and texture coordinates, adapting the pre-computed displacements to the reconstructed base mesh by applying subdivision to the reconstructed base mesh, optionally filtering the adapted displacements with a wavelet filter, computing a remapping of the reconstructed texture coordinates to a regular grid according to a mapping technique to generate a displacement packing image, signaling the metadata describing the remapping technique, and the presence of collided vertices in the packed displacement image along the bitstream, encoding the displacement video frame to generate a displacement sub-stream, remapping the texture map to generate an attribute video frame, encoding the attribute video frame as a substream, signaling the V -DMC metadata along a substream, muxing the substreams into an output stream. [0227] In another embodiment for the encoder: as above, but where the static mesh encoder and decoder generate a reconstructed reference base mesh for one or more of the frames of the sequence, and where a motion encoder is used to compress the static base mesh with respect to the reference base mesh.

[0228] Decoder embodiments

[0229] A decoder receiving and de-muxing an extended V-DMC bitstream into substreams including: a base mesh substream, an attribute displacement substream, an attribute texture substream, a V-DMC signaling substream

[0230] The decoder performs operations including extracting the V-DMC metadata, decoding and dequantizing the base mesh substream into a reconstructed base mesh frame including reconstructed texture coordinates, decoding the attribute texture substream into a texture frame, decoding the displacement substream into a displacement image, perform the subdivision of the reconstructed base mesh according to the V-DMC metadata, extracting subset metadata required for remapping the displacement pixel positions to the corresponding vertex indices and optionally handling collisions as signaled by the metadata, apply decoded displacements, optionally including wavelet inverse transform as specified by the V-DMC metadata to the subdivided mesh, mapping the reconstructed and color converted texture to the subdivided mesh, Store or render the subdivided and deformed subdivision mesh as the output mesh, Signaling the metadata describing the remapping technique, and the presence of collided vertices in the packed displacement image along the bitstream,

[0231] In another embodiment for the decoder: as above, but where the static mesh decoder generates a reconstructed reference base mesh for one or more of the frames of the sequence, and where a motion decoder is used to decode and reconstruct the static base mesh with respect to the reconstructed reference base mesh.

[0232] Remapping algorithms used in the encoding and decoding embodiments are illustrated on FIG. 25 and are as follows.

[0233] Edge Distance based regular grid size

[0234] In the remapping technique, each submesh of the texture coordinates is analyzed to obtain the minimal edge length. This minimal edge length can be set per submesh or for the entire mesh. Once the minimal edge length is obtained for each patch, a regular grid is set so that the pixel distance is equal to the edge length. UV coordinates are then quantized, and vertices are mapped to their closest pixel. This approach does not generate collisions but introduces possibly many unoccupied pixels. These unused pixels can be inpainted by a push-pull blending algorithm for example so as to optimize encoding by reducing edges. The decoder can identify unused pixels by recomputing the remapping in the exact same way.

[0235] In another embodiment, a regular grid is set per submesh, the regular grids are stitched together in an atlas by respecting the submeshes relative positions in the UV-map. This remapping is performed in the same way by the decoder that does not require offsets to convert pixel positions into vertex indices. Submeshes positions in the fused regular grid (considered as an atlas) can be flagged as patches with their coordinates, width and height in one embodiment.

[0236] Harmonic reparameterization

[0237] In the harmonic reparameterization approach, the encoder and decoder first extract the borders of each submesh and map it to a regular grid border. Given a submesh of M vertices with a boundary of N vertices, define Nf as the first larger even integer and define integers W and H such that 2*(W+H)=Nf. These are the weight W and height H of the regular grid set for this submesh that includes W*H pixels. Depending on the submesh connectivity, W*H can be different from M.

[0238] The encoder and decoder perform following:

[0239] mapping the boundary vertices to the regular grid boundary. This can be done by starting with the vertex with the minimal index and map it to the top left pixel of the regular grid, then selecting its first boundary vertex neighbor and map it to the next regular grid boundary pixel in clockwise manner.

[0240] mapping the interior vertices. Once all boundary vertices are mapped, interior vertices (non-boundary vertices) are iteratively positioned at the center of gravity of their neighbors in subpixel coordinates (the boundary vertex positions being kept unchanged) and quantized to the nearest 1- connected available pixel position (e.g., an increment of -1, 0 or 1 in both pixel coordinates). When there is no available pixel for a vertex, a collision score is incremented by one.

[0241] This operation is repeated until there are no more unoccupied pixels in case M is larger than W*H or until the collision score is not decreased after an additional iteration when M is smaller than W*H.

[0242] At the end of this step, all vertices are mapped to pixels, with a list of collisions or unused pixels. The list of collisions per submesh can be signaled in atlas frame parameter set or at patch level when patches are defined to correspond to submeshes. [0243] Harmonic relaxed reparameterization

[0244] The harmonic relaxed reparameterization is similar to the harmonic parameterization, but instead of selecting a boundary of 2*(W+H) = Nf, W and H are increased such that W*H is larger than M and 2*(W+H) is larger than Nf. This enables to avoid collisions. The iterative process is also modified for the boundary. While it is set initially as in the harmonic reparameterization step 1) (leaving several regular grid boundary pixels unoccupied because of the larger grid), the iterative process of mapping interior vertices includes a shifting of the boundary mapping whenever the collision score cannot be reduced. The closest boundary pixel to the pixel with most collisions (or the one with smallest coordinates in case of equality) is freed by shifting neighboring boundary vertices to their neighboring position in clockwise manner. Once this operation is performed, the iterative process is restarted until no collisions remain.

[0245] Collision Avoidance Context Padding

[0246] In this remapping approach, any of the above approaches can be used, but instead of listing and signaling collisions, the collided vertices with their pixel neighborhood (3x3 pixels) is copied and padded for example at the bottom of the regular grid.

[0247] The collision avoidance remapping is motivated by the following. Collision of (u, v) positions can occur because more than one base vertex maps to the same pixel position after remapping or in case of the subdivision of the base mesh triangles, some vertex obtained by subdivision maps to a pixel position already in use by some other vertex. The decoder can detect the exact collisions the encoder produces based on the metadata and the remapping algorithm being the same at encoder and decoder.

[0248] Some collisions are not critical in case the displacement values are similar up to a signaled threshold, which remains an encoder choice. The decoder can use the stored displacement values on pixels with vertex collisions and map the available values to the corresponding collided vertex indices. Otherwise, when values are too different for vertices that collide on the same pixel, these values are signaled for example in the atlas frame parameter set (see later in the document).

[0249] In another embodiment, instead of signaling the values in the asps metadata, an extra padding of the displacement attribute frame can be set to paste collided values with their pixel neighbors, which can be seen as a rectangular context, for example at the bottom of the frame.

[0250] For storing those replicated context with the proper displacement values of the collision points the displacement atlas resolution must be made extended at the encoder side. [0251] In an encoder embodiment, the replication of the context can be seen as an encoding mode like the other remapping approaches. The encoder may select one approach or the other by performing rate distortion optimization and flag the remapping mode in the afps metadata.

[0252] If the generated displacement image includes empty pixels, they are inpainted for example using the same push-pull algorithm used in V3C V-PCC and MIV to inpaint attribute or geometry component atlases.

[0253] It should be noted that in V-DMC, when using inter prediction, a reference base mesh, including its uv-coordinates is used for the full Group of Frames, by encoding motion parameters for example with the MPEG FAMC standard. In one embodiment, when the remapping is based on the base mesh, the parameterization of the reference base mesh is used instead of recomputing it at every frame.

[0254] As remapping techniques may generate collisions or unused pixels, it is necessary to flag them (if the values are not padded in the displacement video), for example in the atlas frame parameter set.

[0255] Several image packings are possible, as the remapping can be applied on the reconstructed base mesh texture coordinates (left side of FIG. 22) or on the subdivided reconstructed base mesh texture coordinates (right side of FIG. 22). In the first case, the mapping obtained for the base mesh texture coordinates can be re-used for each subdivision step, for example by mapping each new vertex obtained by a subdivided edge to the pixel position of the edge vertex with the smallest index. The copied remappings for each LOD can be stacked into a single image in several ways, for example by using vertical stacking, which could enable random access to LODs.

[0256] Alternatively, the remapping can be performed on the full resolution of the reconstructed subdivided base mesh, which maximizes the preservation of vertex neighborhoods but reduces the ease of extracting LODs.

[0257] FIG. 26 illustrates different tiling strategies for the different LODs. On the left a simple example of base mesh 2602 before subdivision and its image packing for the displacement data. On the right, the same base mesh after one iteration of subdivision (LODI) with three examples of tiling: INTERLEAVED 2610 where the packing is based on the subdivided resolution, HIERARCHICAL 2620 where the result of the base mesh packing is hierarchically tiled to create a 2 times larger and higher image at each LOD level, VERTICAL_STACKING 2630, which stacks the packing realized for the base mesh packing for each LOD, therefore preserving the width of the image but multiplying by four the height. Unused pixels are represented as 2604. It should be noted that on large meshes, after each subdivision step, the total number of vertices is multiplied by 4.

[0258] Signaling embodiments.

[0259] The remapping technique can be signaled in the atlas sequence parameter set asps for the V-DMC extension.

[0260] asps_vmc_ext_attribute_packing_method indicates the attribute (displacement or wavelet coefficient) packing method, with values defined in the following table. TRAVERSAL corresponds to the raster and Morton order used in the V-DMC test model, which converts a ID traversal of the attributes or wavelet coefficients into an image. PARAMETERIZATION_GUIDED corresponds to the packing methods, which are based on the reconstructed base mesh 2D parameterization (texture coordinates).

[0261] asps_vmc_ext_ap_remapping_method indicates the remapping method to be used when asps_vmc_ext_attribute_packing_method is equal to PARAMETERIZATION_GUIDED. The following methods are defined in the following table. EDGE_DISTANCE describes the remapping method where the minimal edge length is used to compute the packing grid resolution. HARMONIC describes the relaxation method based on the boundary constraint, which is a harmonic reparameterization. HARMONIC_RELAXED indicates the remapping method using harmonic reparameterization with a relaxed boundary constraint.

[0262] The width and height of the attribute corresponding to the displacements or wavelet coefficients is provided by asps_vmc_ext_attribute_frame_width[ i ] and asps_vmc_ext_attribute_frame_height[ i ], respectively, where i stands for the index of the attribute index corresponding to the displacements or wavelet coefficients.

[0263] asps_vmc_ext_ap_tiling_method indicates the tiling approached used after packing when asps_vmc_ext_attribute_packing_method is equal to PARAMETERIZATION_GUIDED. The values are described in the following table. VERTICAL_STACKING indicates that the packing is obtained from base mesh vertices remapping and copied and stacked at the bottom maintaining the width of the first mapping, for the other LOD iterations. HIERARCHICAL indicates that each LOD mapping is copied from the one derived for the base mesh and LOD maps are tiled on the right, bottom and bottom right of the previous LOD level, in a similar way as done for the 2D discrete wavelet transform for images. INTERLEAVED indicates the remapping is performed on the full resolution of the subdivided mesh, leading to an interleaved packing of displacements for all LODs.

[0264] The asps_vmc_ext_ap_tiling_method syntax element may be as follows:

[0265] Collisions or unused pixels can be signaled to the decoder in different ways. For the case of unused pixels, the decoder can identify such pixels and avoid taking them into account in remapping displacements to the reconstructed mesh vertices, therefore, no specific signaling is required but a number of unused pixels can be transmitted for detecting errors at the decoder.

[0266] Collisions require to flag displacements in the following way. By convention, the displacement corresponding to the vertex with the smallest index is encoded and decoded in the packed image. For the other values, the following signaling is used. They can be signaled in the atlas frame parameter set.

[0267] afps_vmc_ext_ap_remapping_collisions_number_minusl indicates the number of pixels with multiple values mapped and requiring additional signaling of their non-packed values.

[0268] afps_vmc_ext_ap_collision_first_index indicates the index in the attribute frame of the first collision.

[0269] afps_vmc_ext_ap_collision_attribute_number_minusl[i] indicates the number of values encoded for that attribute frame pixel position.

[0270] afps_vmc_ext_ap_collision_attribute_vector[i][j] indicates the value of the vector.

[0271] afps_vmc_ext_ap_collision_attribute_scalar[i][j] indicates the value of the scalar attribute in case afps_vmc_ext_displacement_coordinate_system_enable_flag is set to true.

[0272] afps_vmc_ext_ap_collision_offset[i] indicates the pixel offset towards the next collision.

[0273] The afps_vmc_extension syntax element may be as follows:

[0274] FIG. 27 is an apparatus 2700 which may be implemented in hardware, configured to implement parameterization-guided packing of displacements for dynamic mesh coding, based on any of the examples described herein. The apparatus comprises a processor 2702, at least one memory 2704 (memory 2704 may be non-transitory, transitory, non-volatile, or volatile) including computer program code 2705, wherein the at least one memory 1404 and the computer program code 2705 are configured to, with the at least one processor 2702, cause the apparatus to implement circuitry, a process, component, module, function, coding, and/or decoding (collectively 2706) to implement hierarchical coding of texture and geometry data for dynamic meshes, based on the examples described herein. The apparatus 2700 is further configured to provide or receive signaling 2707, based on the signaling embodiments described herein. The apparatus 2700 optionally includes a display and/or I/O interface 2708 that may be used to display an output (e.g., an image or volumetric video) of a result of coding/decoding 2706. The display and/or I/O interface 2708 may also be configured to receive input such as user input (e.g. with a keypad, touchscreen, touch area, microphone, biometric recognition, one or more sensors etc.). The apparatus 2700 also includes one or more communication interfaces (I/F(s)) 2710, such as a network (N/W) interface. The communication I/F(s) 2710 may be wired and/or wireless and communicate over a channel or the Internet/other network(s) via any communication technique. The communication I/F(s) 2710 may comprise one or more transmitters and one or more receivers. The communication I/F(s) 2710 may comprise standard well-known components such as an amplifier, filter, frequency-converter, (de)modulator, and encoder/decoder circuitry(ies) and one or more antennas. In some examples, the processor 2702 is configured to implement item 2706 and/or item T1Q7 without use of memory 2704.

[0275] The apparatus 2700 may be a remote, virtual or cloud apparatus. The apparatus 2700 may be either a writer or a reader (e.g. parser), or both a writer and a reader (e.g. parser). The apparatus 2700 may be either a coder or a decoder, or both a coder and a decoder (codec). The apparatus 2700 may be a user equipment (UE), a head mounted display (HMD), or any other fixed or mobile device.

[0276] The memory 2704 may be implemented using any suitable data storage technology, such as semiconductor based memory devices, flash memory, magnetic memory devices and systems, optical memory devices and systems, fixed memory and removable memory. The memory 2704 may comprise a database for storing data. Interface 2712 enables data communication between the various items of apparatus 2700, as shown in FIG. 27. Interface 2712 may be one or more buses. For example, the interface 2712 may be one or more buses such as address, data, or control buses, and may include any interconnection mechanism, such as a series of lines on a motherboard or integrated circuit, fiber optics or other optical communication equipment, and the like. Computer program code 2705 may comprise object-oriented software. The apparatus 2700 need not comprise each of the features mentioned, or may comprise other features as well. The apparatus 2700 may be an embodiment of and have the features of any of the apparatuses shown in FIG. 1A, FIG. IB, FIG. 5, FIG. 6, FIG. 11, FIG. 14, FIG. 23, FIG. 24, or any of the other figures described and shown herein.

[0277] FIG. 28 shows a schematic representation of non-volatile memory media 2800a (e.g. computer/compact disc (CD) or digital versatile disc (DVD)) and 2800b (e.g. universal serial bus (USB) memory stick) storing instructions and/or parameters 2802 which when executed by a processor allows the processor to perform one or more of the steps of the methods described herein.

[0278] FIG. 29 is a method 2900 to implement the examples described herein. At 2910, the method includes receiving, with an encoder for a frame of an input volumetric video mesh: a simplified base mesh with texture coordinates, a set of pre-computed displacements for a chosen subdivision method, and a texture frame. At 2920, the method includes generating a static base mesh encoding that outputs a base mesh substream including encoded and quantized texture coordinates, and that generates an encoded static base mesh. At 2930, the method includes decoding the encoded static base mesh, and generating a reconstructed dequantized base mesh and reconstructed texture coordinates. At 2940, the method includes adapting the pre-computed displacements to the reconstructed dequantized base mesh with applying subdivision to the reconstructed dequantized base mesh. At 2950, the method includes computing a remapping of the reconstructed texture coordinates to a regular grid based on a remapping technique to generate a displacement packing image. At 2960, the method includes signaling metadata describing the remapping technique. At 2970, the method includes signaling a presence of collided vertices in the displacement packing image along the bitstream. Method 1500 may be performed with an encoder apparatus (e.g. 1101, 1201, 2301, 2700).

[0279] FIG. 30 is a method 3000 to implement the examples described herein. At 3010, the method includes receiving and de-multiplexing, with a decoder, an extended video-based dynamic mesh coding bitstream into substreams including: a base mesh substream, a displacement substream, an attribute substream, and a video-based dynamic mesh coding signaling substream. At 3020, the method includes extracting video-based dynamic mesh coding metadata. At 3030, the method includes decoding and dequantizing the base mesh substream into a reconstructed base mesh frame including reconstructed texture coordinates. At 3040, the method includes decoding the attribute substream into a texture frame. At 3050, the method includes decoding the displacement substream into a displacement image. At 3060, the method includes performing subdivision of the reconstructed base mesh frame based on the video-based dynamic mesh coding metadata to generate a subdivided mesh. At 3070, the method includes extracting subset metadata for remapping displacement pixel positions of the displacement image to corresponding vertex indices. At 3080, the method includes applying decoded displacements to the subdivided mesh as specified with the video-based dynamic mesh coding metadata. Method 3000 may be performed with a decoder apparatus (e.g. 1401, 1501, 2401, 2700).

[0280] The following examples are provided and described herein.

[0281] Example 1. An apparatus including: at least one processor; and at least one memory storing instructions that, when executed by the at least one processor, cause the apparatus at least to: receive, with an encoder for a frame of an input volumetric video mesh: a simplified base mesh with texture coordinates, a set of pre-computed displacements for a chosen subdivision method, and a texture frame; generate a static base mesh encoding that outputs a base mesh substream including encoded and quantized texture coordinates, and that generates an encoded static base mesh; decode the encoded static base mesh, and generate a reconstructed dequantized base mesh and reconstructed texture coordinates; adapt the pre-computed displacements to the reconstructed dequantized base mesh with applying subdivision to the reconstructed dequantized base mesh; compute a remapping of the reconstructed texture coordinates to a regular grid based on a remapping technique to generate a displacement packing image; signal metadata describing the remapping technique; and signal a presence of collided vertices in the displacement packing image along the bitstream.

[0282] Example 2. The apparatus of example 1, wherein the apparatus is caused to: filter the adapted pre-computed displacements with a wavelet filter.

[0283] Example 3. The apparatus of any of examples 1 to 2, wherein the displacement packing image comprises a displacement video frame.

[0284] Example 4. The apparatus of example 3, wherein the apparatus is caused to: encode the displacement video frame to generate a displacement sub-stream.

[0285] Example 5. The apparatus of any of examples 1 to 4, wherein the apparatus is caused to: remap a texture map to generate an attribute video frame.

[0286] Example 6. The apparatus of example 5, wherein the apparatus is caused to: encode the attribute video frame as a first substream. [0287] Example 7. The apparatus of example 6, wherein the apparatus is caused to: signal videobased dynamic mesh coding (V-DMC) metadata along a second substream.

[0288] Example 8. The apparatus of example 7, wherein the apparatus is caused to: multiplex the first substream and the second substream into an output stream.

[0289] Example 9. The apparatus of any of examples 1 to 8, wherein the static base mesh encoding is performed with a static mesh encoder, and the decoding of the encoded static base mesh is performed with a static mesh decoder.

[0290] Example 10. The apparatus of example 9, wherein the apparatus is caused to: generate, with the static mesh encoder and the static mesh decoder, a reconstructed reference base mesh for one or more frames of a sequence of the input volumetric video mesh; and compress, with a motion encoder, the static base mesh with respect to the reconstructed reference base mesh.

[0291] Example 11. An apparatus including at least one processor; and at least one memory storing instructions that, when executed by the at least one processor, cause the apparatus at least to: receive and de-multiplex, with a decoder, an extended video-based dynamic mesh coding bitstream into substreams including: a base mesh substream, a displacement substream, an attribute substream, and a video-based dynamic mesh coding signaling substream; extract video-based dynamic mesh coding metadata; decode and dequantize the base mesh substream into a reconstructed base mesh frame including reconstructed texture coordinates; decode the attribute substream into a texture frame; decode the displacement substream into a displacement image; perform subdivision of the reconstructed base mesh frame based on the video-based dynamic mesh coding metadata to generate a subdivided mesh; extract subset metadata for remapping displacement pixel positions of the displacement image to corresponding vertex indices; and apply decoded displacements to the subdivided mesh as specified with the video-based dynamic mesh coding metadata.

[0292] Example 12. The apparatus of example 11, wherein the apparatus is caused to: handle collisions as signaled with the subset metadata.

[0293] Example 13. The apparatus of any of examples 11 to 12, wherein the apparatus is caused to: apply a wavelet inverse transform to the subdivided mesh as specified with the video-based dynamic mesh coding (V-DMC) metadata.

[0294] Example 14. The apparatus of any of examples 11 to 13, wherein the apparatus is caused to: generate reconstructed and color converted texture; and map the reconstructed and color converted texture to the subdivided mesh. [0295] Example 15. The apparatus of any of examples 11 to 14, wherein the apparatus is caused to: store or render a subdivided deformed mesh as an output mesh.

[0296] Example 16. The apparatus of any of examples 11 to 15, wherein the apparatus is caused to: signal metadata describing a remapping technique for the remapping of the displacement pixel positions.

[0297] Example 17. The apparatus of any of examples 11 to 16, wherein the apparatus is caused to signal a presence of collided vertices in a packed displacement image along a bitstream.

[0298] Example 18. The apparatus of any of examples 11 to 17, wherein the apparatus is caused to: generate, with a static mesh decoder, a reconstructed reference base mesh for one or more frames of a sequence of a volumetric video mesh; and decode and reconstruct, with a motion decoder, a static base mesh with respect to the reconstructed reference base mesh.

[0299] Example 19. A method including: receiving, with an encoder for a frame of an input volumetric video mesh: a simplified base mesh with texture coordinates, a set of pre-computed displacements for a chosen subdivision method, and a texture frame; generating a static base mesh encoding that outputs a base mesh substream including encoded and quantized texture coordinates, and that generates an encoded static base mesh; decoding the encoded static base mesh, and generating a reconstructed dequantized base mesh and reconstructed texture coordinates; adapting the pre-computed displacements to the reconstructed dequantized base mesh with applying subdivision to the reconstructed dequantized base mesh; computing a remapping of the reconstructed texture coordinates to a regular grid based on a remapping technique to generate a displacement packing image; signaling metadata describing the remapping technique; and signaling a presence of collided vertices in the displacement packing image along the bitstream.

[0300] Example 20. The method of example 19 further comprising filtering the adapted precomputed displacements with a wavelet filter.

[0301] Example 21. The method of any of examples 19 to 20, wherein the displacement packing image comprises a displacement video frame.

[0302] Example 22. The method of example 21further comprising encoding the displacement video frame to generate a displacement sub-stream.

[0303] Example 23. The method of any of examples 19 to 22 further comprising remapping a texture map to generate an attribute video frame. [0304] Example 24. The method of example 23 further comprising encoding the attribute video frame as a first substream.

[0305] Example 25. The method of example 24 further comprising signaling video-based dynamic mesh coding (V -DMC) metadata along a second substream.

[0306] Example 26. The method of example 25 further comprising multiplexing the first substream and the second substream into an output stream.

[0307] Example 27. The method of any of example 19 to 26, wherein the static base mesh encoding is performed with a static mesh encoder, and the decoding of the encoded static base mesh is performed with a static mesh decoder.

[0308] Example 28. The method of example 27 further comprising generating, with the static mesh encoder and the static mesh decoder, a reconstructed reference base mesh for one or more frames of a sequence of the input volumetric video mesh; and compressing, with a motion encoder, the static base mesh with respect to the reconstructed reference base mesh.

[0309] Example 29. A method including: receiving and de-multiplexing, with a decoder, an extended video-based dynamic mesh coding bitstream into substreams including: a base mesh substream, a displacement substream, an attribute substream, and a video-based dynamic mesh coding signaling substream; extracting video-based dynamic mesh coding metadata; decoding and dequantizing the base mesh substream into a reconstructed base mesh frame including reconstructed texture coordinates; decoding the attribute substream into a texture frame; decoding the displacement substream into a displacement image; performing subdivision of the reconstructed base mesh frame based on the video-based dynamic mesh coding metadata to generate a subdivided mesh; extracting subset metadata for remapping displacement pixel positions of the displacement image to corresponding vertex indices; and applying decoded displacements to the subdivided mesh as specified with the videobased dynamic mesh coding metadata.

[0310] Example 30. The method of example 29 further comprising handle collisions as signaled with the subset metadata.

[0311] Example 31. The method of any of examples 29 to 31 further comprising applying a wavelet inverse transform to the subdivided mesh as specified with the video-based dynamic mesh coding (V-DMC) metadata.

[0312] Example 32. The method of any of examples 29 to 31 further comprising generating reconstructed and color converted texture; and mapping the reconstructed and color converted texture to the subdivided mesh.

[0313] Example 33. The method of any of examples 29 to 32 further comprising storing or rendering a subdivided deformed mesh as an output mesh.

[0314] Example 34. The method of any of examples 29 to 33 further comprising signaling metadata describing a remapping technique for the remapping of the displacement pixel positions.

[0315] Example 35. The method of any of examples 29 to 34 further comprising signaling a presence of collided vertices in a packed displacement image along a bitstream.

[0316] Example 36. The method of any of examples 29 to 35 further comprising generating, with a static mesh decoder, a reconstructed reference base mesh for one or more frames of a sequence of a volumetric video mesh; and decoding and reconstruct, with a motion decoder, a static base mesh with respect to the reconstructed reference base mesh.

[0317] Example 37. An apparatus including: means for receiving, with an encoder for a frame of an input volumetric video mesh: a simplified base mesh with texture coordinates, a set of pre-computed displacements for a chosen subdivision method, and a texture frame; means for generating a static base mesh encoding that outputs a base mesh substream including encoded and quantized texture coordinates, and that generates an encoded static base mesh; means for decoding the encoded static base mesh, and generating a reconstructed dequantized base mesh and reconstructed texture coordinates; means for adapting the pre-computed displacements to the reconstructed dequantized base mesh with applying subdivision to the reconstructed dequantized base mesh; means for computing a remapping of the reconstructed texture coordinates to a regular grid based on a remapping technique to generate a displacement packing image; means for signaling metadata describing the remapping technique; and means for signaling a presence of collided vertices in the displacement packing image along the bitstream.

[0318] Example 38. An apparatus including: means for receiving and de-multiplexing, with a decoder, an extended video-based dynamic mesh coding bitstream into substreams including: a base mesh substream, a displacement substream, an attribute substream, and a video-based dynamic mesh coding signaling substream; means for extracting video-based dynamic mesh coding metadata; means for decoding and dequantizing the base mesh substream into a reconstructed base mesh frame including reconstructed texture coordinates; means for decoding the attribute substream into a texture frame; means for decoding the displacement substream into a displacement image; means for performing subdivision of the reconstructed base mesh frame based on the video-based dynamic mesh coding metadata to generate a subdivided mesh; means for extracting subset metadata for remapping displacement pixel positions of the displacement image to corresponding vertex indices; and means for applying decoded displacements to the subdivided mesh as specified with the video-based dynamic mesh coding metadata.

[0319] Example 39. A non-transitory program storage device readable by a machine, tangibly embodying a program of instructions executable with the machine for performing operations is provided, the operations including: receiving, with an encoder for a frame of an input volumetric video mesh: a simplified base mesh with texture coordinates, a set of pre-computed displacements for a chosen subdivision method, and a texture frame; generating a static base mesh encoding that outputs a base mesh substream including encoded and quantized texture coordinates, and that generates an encoded static base mesh; decoding the encoded static base mesh, and generating a reconstructed dequantized base mesh and reconstructed texture coordinates; adapting the pre-computed displacements to the reconstructed dequantized base mesh with applying subdivision to the reconstructed dequantized base mesh; computing a remapping of the reconstructed texture coordinates to a regular grid based on a remapping technique to generate a displacement packing image; signaling metadata describing the remapping technique; and signaling a presence of collided vertices in the displacement packing image along the bitstream.

[0320] Example 40. A non-transitory program storage device readable by a machine, tangibly embodying a program of instructions executable with the machine for performing operations is provided, the operations including: receiving and de-multiplexing, with a decoder, an extended videobased dynamic mesh coding bitstream into substreams including: a base mesh substream, a displacement substream, an attribute substream, and a video-based dynamic mesh coding signaling substream; extracting video-based dynamic mesh coding metadata; decoding and dequantizing the base mesh substream into a reconstructed base mesh frame including reconstructed texture coordinates; decoding the attribute substream into a texture frame; decoding the displacement substream into a displacement image; performing subdivision of the reconstructed base mesh frame based on the videobased dynamic mesh coding metadata to generate a subdivided mesh; extracting subset metadata for remapping displacement pixel positions of the displacement image to corresponding vertex indices; and applying decoded displacements to the subdivided mesh as specified with the video-based dynamic mesh coding metadata.

[0321] References to a ‘computer’, ‘processor’, etc. should be understood to encompass not only computers having different architectures such as single/multi-processor architectures and sequential (Von Neumann)/parallel architectures but also specialized circuits such as field-programmable gate arrays (FPGA), application specific circuits (ASIC), signal processing devices and other processing circuitry. References to computer program, instructions, code etc. should be understood to encompass software for a programmable processor or firmware such as, for example, the programmable content of a hardware device such as instructions for a processor, or configuration settings for a fixed-function device, gate array or programmable logic device, etc.

[0322] As used herein, the term ‘circuitry’ may refer to any of the following: (a) hardware circuit implementations, such as implementations in analog and/or digital circuitry, and (b) combinations of circuits and software (and/or firmware), such as (as applicable): (i) a combination of processor(s) or (ii) portions of processor(s)/software including digital signal processor(s), software, and memory(ies) that work together to cause an apparatus to perform various functions, and (c) circuits, such as a microprocessor(s) or a portion of a microprocessor(s), that require software or firmware for operation, even when the software or firmware is not physically present. As a further example, as used herein, the term ‘circuitry’ would also cover an implementation of merely a processor (or multiple processors) or a portion of a processor and its (or their) accompanying software and/or firmware. The term ‘circuitry’ would also cover, for example and when applicable to the particular element, a baseband integrated circuit or applications processor integrated circuit for a mobile phone or a similar integrated circuit in a server, a cellular network device, or another network device. Circuitry may also be used to mean a function or a process, such as one implemented by an encoder or decoder, or a codec.

[0323] In the figures, arrows between individual blocks represent operational couplings therebetween as well as the direction of data flows on those couplings.

[0324] It should be understood that the foregoing description is only illustrative. Various alternatives and modifications may be devised by those skilled in the art. For example, features recited in the various dependent claims could be combined with each other in any suitable combination(s). In addition, features from different embodiments described above could be selectively combined into a new embodiment. Accordingly, the description is intended to embrace all such alternatives, modifications and variances which fall within the scope of the appended claims.

[0325] The following acronyms and abbreviations that may be found in the specification and/or the drawing figures are defined as follows:

2D and variants two-dimensional

3D and variants three-dimensional

3DG 3D graphics coding group

4CC four character code 6DOF six degrees of freedom

ACL atlas coding layer

AD atlas data afoc atlas frame order count

AFPS and variants atlas frame parameter set ai attribute index ap attribute_packing

AR augmented reality

ASIC application-specific integrated circuit

ASPS and variants atlas sequence parameter set ath atlas tile header

AUD access unit delimiter

Aux auxiliary

AVD attribute video data

BFPS and variants base mesh frame parameter set

BLA broken link access

BMCL base mesh coding layer bmesh base mesh

BMFPS base mesh frame parameter set bmptl base mesh profile, tier, and level

BMSPS and variants base mesh sequence parameter set b(n) n bits bsmi base mesh sub mesh information

BSPS base mesh sequence parameter set

CD compact disc

CfP call for proposal

CGI computer-generated imagery ent count

CRA clean random access

DCT discrete cosine transform

DVD digital versatile disc

EOB end of bitstream

EOS end of sequence

ESEI essential supplemental enhancement information

Exp exponential ext extension

FAMC frame -based animated mesh compression

FD filler data

FDIS final draft international standard f(n) float having n bits e.g. f( 1) glTF graphics language transmission format

GVD geometry video data

H.264 advanced video coding video compression standard

H.265 high efficiency video coding video compression standard

HEVC high efficiency video coding

HLS high level syntax

HMD head mounted display

ID and variants identifier ide indication

Idx index

IDR instantaneous decoding refresh

IEC International Electrotechnical Commission

I/F interface

VO input/output

IRAP intra random access point

ISO International Organization for Standardization len length

LOD and variants level of detail

LP leading picture

Isb least significant bit

It lifting transform

Itp lifting transform parameters

MCL mesh coding layer

MD mesh data mdu mesh intra patch data unit mfoc mesh_frame_order_count midu mesh inter data unit miv and variants MPEG immersive video mmdu mesh merge data unit mpdu mesh patch data unit MPEG moving picture experts group

MPEG-I MPEG immersive

MR mixed reality mrdu mesh raw data unit msh mesh

MUX multiplex

NAL and variants network abstraction layer

NBMCL non-BMCL

NSEI non-essential supplemental enhancement information

N/W network

OVD occupancy video data poc picture order count

Pos position

Quant quantization

RADL random access decodable leading

RASL random access skipped leading rbsp and variants raw byte sequence payload ref reference

RSV reserved

SC subcommittee

SEI supplemental enhancement information se(v) signed integer O-th order Exp-Golomb coding with left bit first (e.g., most significant bit first)) smdu submesh data unit smh and variants submesh

SODB string of data bits

STSA step- wise temporal sublayer access

TBD to be determined

TSA temporal sublayer access ti tile information u(n) unsigned integer using n bits, e.g. u(l), u(2)

UE user equipment ue(v) unsigned integer exponential Golomb coded syntax element with the left bit first

UNSPEC unspecified USB universal serial bus uv and variants coordinate texture, where “U” and “V” are axes of a 2D texture

V3C visual volumetric video-based coding

V-DMC or VDMC video-based dynamic mesh coding vmc volumetric mesh compression

VPCC or V-PCC video-based point cloud coding/compression

VPS V 3C parameter set

VR virtual reality vuh volumetric unit header

VVC versatile video coding

WG working group

Claims

CLAIMS What is claimed is:

1. An apparatus comprising: at least one processor; and at least one memory storing instructions that, when executed by the at least one processor, cause the apparatus at least to: receive, with an encoder for a frame of an input volumetric video mesh: a simplified base mesh with texture coordinates, a set of pre-computed displacements for a chosen subdivision method, and a texture frame; generate a static base mesh encoding that outputs a base mesh substream including encoded and quantized texture coordinates, and that generates an encoded static base mesh; decode the encoded static base mesh, and generate a reconstructed dequantized base mesh and reconstructed texture coordinates; adapt the pre-computed displacements to the reconstructed dequantized base mesh with applying subdivision to the reconstructed dequantized base mesh; compute a remapping of the reconstructed texture coordinates to a regular grid based on a remapping technique to generate a displacement packing image; signal metadata describing the remapping technique; and signal a presence of collided vertices in the displacement packing image along the bitstream.

2. The apparatus of claim 1, wherein the apparatus is caused to: filter the adapted pre-computed displacements with a wavelet filter.

3. The apparatus of any of claims 1 to 2, wherein the displacement packing image comprises a displacement video frame.

4. The apparatus of claim 3, wherein the apparatus is caused to: encode the displacement video frame to generate a displacement sub-stream.

5. The apparatus of any of claims 1 to 4, wherein the apparatus is caused to: remap a texture map to generate an attribute video frame.

6. The apparatus of claim 5, wherein the apparatus is caused to: encode the attribute video frame as a first substream.

7. The apparatus of claim 6, wherein the apparatus is caused to: signal video-based dynamic mesh coding (V-DMC) metadata along a second substream.

8. The apparatus of claim 7, wherein the apparatus is caused to: multiplex the first substream and the second substream into an output stream.

9. The apparatus of any of claims 1 to 8, wherein the static base mesh encoding is performed with a static mesh encoder, and the decoding of the encoded static base mesh is performed with a static mesh decoder.

10. The apparatus of claim 9, wherein the apparatus is caused to: generate, with the static mesh encoder and the static mesh decoder, a reconstructed reference base mesh for one or more frames of a sequence of the input volumetric video mesh; and compress, with a motion encoder, the static base mesh with respect to the reconstructed reference base mesh.

11. An apparatus comprising: at least one processor; and at least one memory storing instructions that, when executed by the at least one processor, cause the apparatus at least to: receive and de-multiplex, with a decoder, an extended video-based dynamic mesh coding bitstream into substreams including: a base mesh substream, a displacement substream, an attribute substream, and a video-based dynamic mesh coding signaling substream; extract video-based dynamic mesh coding metadata; decode and dequantize the base mesh substream into a reconstructed base mesh frame including reconstructed texture coordinates; decode the attribute substream into a texture frame; decode the displacement substream into a displacement image; perform subdivision of the reconstructed base mesh frame based on the video-based dynamic mesh coding metadata to generate a subdivided mesh; extract subset metadata for remapping displacement pixel positions of the displacement image to corresponding vertex indices; and apply decoded displacements to the subdivided mesh as specified with the video-based dynamic mesh coding metadata.

12. The apparatus of claim 11, wherein the apparatus is caused to: handle collisions as signaled with the subset metadata.

13. The apparatus of any of claims 11 to 12, wherein the apparatus is caused to: apply a wavelet inverse transform to the subdivided mesh as specified with the videobased dynamic mesh coding (V -DMC) metadata.

14. The apparatus of any of claims 11 to 13, wherein the apparatus is caused to: generate reconstructed and color converted texture; and map the reconstructed and color converted texture to the subdivided mesh.

15. The apparatus of any of claims 11 to 14, wherein the apparatus is caused to: store or render a subdivided deformed mesh as an output mesh.

16. The apparatus of any of claims 11 to 15, wherein the apparatus is caused to: signal metadata describing a remapping technique for the remapping of the displacement pixel positions.

17. The apparatus of any of claims 11 to 16, wherein the apparatus is caused to signal a presence of collided vertices in a packed displacement image along a bitstream.

18. The apparatus of any of claims 11 to 17, wherein the apparatus is caused to: generate, with a static mesh decoder, a reconstructed reference base mesh for one or more frames of a sequence of a volumetric video mesh; and decode and reconstruct, with a motion decoder, a static base mesh with respect to the reconstructed reference base mesh.

19. A method comprising: receiving, with an encoder for a frame of an input volumetric video mesh: a simplified base mesh with texture coordinates, a set of pre-computed displacements for a chosen subdivision method, and a texture frame; generating a static base mesh encoding that outputs a base mesh substream including encoded and quantized texture coordinates, and that generates an encoded static base mesh; decoding the encoded static base mesh, and generating a reconstructed dequantized base mesh and reconstructed texture coordinates; adapting the pre-computed displacements to the reconstructed dequantized base mesh with applying subdivision to the reconstructed dequantized base mesh; computing a remapping of the reconstructed texture coordinates to a regular grid based on a remapping technique to generate a displacement packing image; signaling metadata describing the remapping technique; and signaling a presence of collided vertices in the displacement packing image along the bitstream.

20. A method comprising: receiving and de-multiplexing, with a decoder, an extended video-based dynamic mesh coding bitstream into substreams including: a base mesh substream, a displacement substream, an attribute substream, and a video-based dynamic mesh coding signaling substream; extracting video-based dynamic mesh coding metadata; decoding and dequantizing the base mesh substream into a reconstructed base mesh frame including reconstructed texture coordinates; decoding the attribute substream into a texture frame; decoding the displacement substream into a displacement image; performing subdivision of the reconstructed base mesh frame based on the video-based dynamic mesh coding metadata to generate a subdivided mesh; extracting subset metadata for remapping displacement pixel positions of the displacement image to corresponding vertex indices; and applying decoded displacements to the subdivided mesh as specified with the videobased dynamic mesh coding metadata.

21. An apparatus comprising: means for receiving, with an encoder for a frame of an input volumetric video mesh: a simplified base mesh with texture coordinates, a set of pre-computed displacements for a chosen subdivision method, and a texture frame; means for generating a static base mesh encoding that outputs a base mesh substream including encoded and quantized texture coordinates, and that generates an encoded static base mesh; means for decoding the encoded static base mesh, and generating a reconstructed dequantized base mesh and reconstructed texture coordinates; means for adapting the pre-computed displacements to the reconstructed dequantized base mesh with applying subdivision to the reconstructed dequantized base mesh; means for computing a remapping of the reconstructed texture coordinates to a regular grid based on a remapping technique to generate a displacement packing image; means for signaling metadata describing the remapping technique; and means for signaling a presence of collided vertices in the displacement packing image along the bitstream.

22. An apparatus comprising: means for receiving and de-multiplexing, with a decoder, an extended video-based dynamic mesh coding bitstream into substreams including: a base mesh substream, a displacement substream, an attribute substream, and a video-based dynamic mesh coding signaling substream; means for extracting video-based dynamic mesh coding metadata; means for decoding and dequantizing the base mesh substream into a reconstructed base mesh frame including reconstructed texture coordinates; means for decoding the attribute substream into a texture frame; means for decoding the displacement substream into a displacement image; means for performing subdivision of the reconstructed base mesh frame based on the video-based dynamic mesh coding metadata to generate a subdivided mesh; means for extracting subset metadata for remapping displacement pixel positions of the displacement image to corresponding vertex indices; and means for applying decoded displacements to the subdivided mesh as specified with the video-based dynamic mesh coding metadata.

23. A non-transitory program storage device readable by a machine, tangibly embodying a program of instructions executable with the machine for performing operations, the operations comprising: receiving, with an encoder for a frame of an input volumetric video mesh: a simplified base mesh with texture coordinates, a set of pre-computed displacements for a chosen subdivision method, and a texture frame; generating a static base mesh encoding that outputs a base mesh substream including encoded and quantized texture coordinates, and that generates an encoded static base mesh; decoding the encoded static base mesh, and generating a reconstructed dequantized base mesh and reconstructed texture coordinates; adapting the pre-computed displacements to the reconstructed dequantized base mesh with applying subdivision to the reconstructed dequantized base mesh; computing a remapping of the reconstructed texture coordinates to a regular grid based on a remapping technique to generate a displacement packing image; signaling metadata describing the remapping technique; and signaling a presence of collided vertices in the displacement packing image along the bitstream.

24. A non-transitory program storage device readable by a machine, tangibly embodying a program of instructions executable with the machine for performing operations, the operations comprising: receiving and de-multiplexing, with a decoder, an extended video-based dynamic mesh coding bitstream into substreams including: a base mesh substream, a displacement substream, an attribute substream, and a video-based dynamic mesh coding signaling substream; extracting video-based dynamic mesh coding metadata; decoding and dequantizing the base mesh substream into a reconstructed base mesh frame including reconstructed texture coordinates; decoding the attribute substream into a texture frame; decoding the displacement substream into a displacement image; performing subdivision of the reconstructed base mesh frame based on the video-based dynamic mesh coding metadata to generate a subdivided mesh; extracting subset metadata for remapping displacement pixel positions of the displacement image to corresponding vertex indices; and applying decoded displacements to the subdivided mesh as specified with the videobased dynamic mesh coding metadata.