EP4690806A1

EP4690806A1 - A method, an apparatus and a computer program product for video encoding and video decoding

Info

Publication number: EP4690806A1
Application number: EP24784468.1A
Authority: EP
Inventors: Patrice Rondao Alface; Lukasz Kondrad; Lauri Aleksi ILOLA
Original assignee: Nokia Technologies Oy
Current assignee: Nokia Technologies Oy
Priority date: 2023-04-06
Filing date: 2024-02-27
Publication date: 2026-02-11
Also published as: WO2024209129A1

Abstract

The embodiments relate to a method for encoding, comprising receiving a dynamic three-dimensional mesh sequence, wherein the three-dimensional mesh represents a three-dimensional object with texture (1905); determining which texture coordinates are not encoded into a bitstream (1910); determining a derivation mode for the texture coordinates not to be encoded (1915); encoding into a bitstream an indication of the determined derivation mode (1920); encoding into a bitstream an information on a reference frame comprising texture coordinates for the current mesh frame (1925); encoding a base mesh into a bitstream with the texture coordinates which are to be encoded (1930); and transmitting the one or more bitstreams to a decoder. In addition, the embodiments relate to a method for decoding, and technical equipment for implementing the methods (1935).

Description

A METHOD, AN APPARATUS AND A COMPUTER PROGRAM PRODUCT FOR VIDEO ENCODING AND VIDEO DECODING

Technical Field

The present solution generally relates to encoding and decoding of volumetric video.

Background

Volumetric video data represents a three-dimensional (3D) scene or object and can be used as input for AR (Augmented Reality), VR (Virtual Reality), and MR (Mixed Reality) applications. Such data describes geometry (Shape, size, position in 3D space) and respective attributes (e.g., color, opacity, reflectance, ...), and any possible temporal transformations of the geometry and attributes at given time instances (like frames in two-dimensional (2D) video). Volumetric video can be generated from 3D models, also referred to as volumetric visual objects, i.e., CGI (Computer Generated Imagery), or captured from real-world scenes using a variety of capture solutions, e.g., multi-camera, laser scan, combination of video and dedicated depth sensors, and more. Also, a combination of CGI and real-world data is possible. Examples of representation formats for volumetric data comprise triangle meshes, point clouds, or voxels. Temporal information about the scene can be included in the form of individual capture instances, i.e., “frames” in 2D video, or other means, e.g., position of an object as a function of time.

Because volumetric video describes a 3D scene (or object), such data can be viewed from any viewpoint. Therefore, volumetric video is an important format for any AR, VR or MR applications, especially for providing 6DOF viewing capabilities.

Increasing computational resources and advances in 3D data acquisition devices have enabled reconstruction of highly detailed volumetric video representations of natural scenes. Infrared, lasers, time-of-flight, and structured light are examples of devices that can be used to construct 3D video data. Representation of the 3D data depends on how the 3D data is used. Dense Voxel arrays have been used to represent volumetric medical data. In 3D graphics, polygonal meshes are extensively used. Point clouds on the other hand are well suited for applications such as capturing real world 3D scenes where the topology is not necessarily a 2D manifold. Another way to represent 3D data is coding, this 3D data as set of texture and depth map as is the case in the multi-view plus depth. Closely related to the techniques used in multiview plus depth is the use of elevation maps, and multi-level surface maps.

Summary

The scope of protection sought for various embodiments of the invention is set out by the independent claims. The embodiments and features, if any, described in this specification that do not fall under the scope of the independent claims are to be interpreted as examples useful for understanding various embodiments of the invention.

Various aspects include a method, an apparatus and a computer readable medium comprising a computer program stored therein, which are characterized by what is stated in the independent claims. Various embodiments are disclosed in the dependent claims.

According to a first aspect, there is provided an apparatus for encoding comprising means for receiving a dynamic three-dimensional mesh sequence, wherein the three-dimensional mesh represents a three-dimensional object with texture; means for determining which texture coordinates are not encoded into a bitstream; means for determining a derivation mode for the texture coordinates not to be encoded; means for encoding into a bitstream an indication of the determined derivation mode; means for encoding into a bitstream an information on a reference frame comprising texture coordinates for the current mesh frame; means for encoding a base mesh into a bitstream with the texture coordinates which are to be encoded; and means for transmitting the one or more bitstreams to a decoder.

According to a second aspect, there is provided an apparatus for decoding comprising means for receiving one or more bitstreams from an encoder; means for decoding a base mesh from a bitstream texture coordinates; means for determining texture coordinates missing from a bitstream; means for decoding an indication on a derivation mode from a bitstream; means for decoding information on a reference frame comprising texture coordinates missing from the bitstream; means for reconstructing missing texture coordinates from the reference frame according to the derivation mode; and means for reconstructing a dynamic three-dimensional mesh sequence according to the texture coordinates.

According to a third aspect, there is provided a method for encoding, comprising: receiving a dynamic three-dimensional mesh sequence, wherein the three-dimensional mesh represents a three-dimensional object with texture; determining which texture coordinates are not encoded into a bitstream; determining a derivation mode for the texture coordinates not to be encoded; encoding into a bitstream an indication of the determined derivation mode; encoding into a bitstream an information on a reference frame comprising texture coordinates for the current mesh frame; encoding a base mesh into a bitstream with the texture coordinates which are to be encoded; and transmitting the one or more bitstreams to a decoder.

According to a fourth aspect, there is provided a method for decoding, comprising: receiving one or more bitstreams from an encoder; decoding a base mesh from a bitstream texture coordinates; determining texture coordinates missing from a bitstream; decoding an indication on a derivation mode from a bitstream; decoding information on a reference frame comprising texture coordinates missing from the bitstream; reconstructing missing texture coordinates from the reference frame according to the derivation mode; and reconstructing a dynamic three-dimensional mesh sequence according to the texture coordinates.

According to a fifth aspect, there is provided an apparatus for encoding, the apparatus comprising at least one processor, memory including computer program code, the memory and the computer program code configured to, with the at least one processor, cause the apparatus to perform at least the following: receive a dynamic three-dimensional mesh sequence, wherein the three-dimensional mesh represents a three-dimensional object with texture; determine which texture coordinates are not encoded into a bitstream; determine a derivation mode for the texture coordinates not to be encoded; encode into a bitstream an indication of the determined derivation mode; encode into a bitstream an information on a reference frame comprising texture coordinates for the current mesh frame; encode a base mesh into a bitstream with the texture coordinates which are to be encoded; and transmit the one or more bitstreams to a decoder.

According to a sixth aspect, there is provided an apparatus for decoding, the apparatus comprising at least one processor, memory including computer program code, the memory and the computer program code configured to, with the at least one processor, cause the apparatus to perform at least the following: receive one or more bitstreams from an encoder; decode a base mesh from a bitstream texture coordinates; determine texture coordinates missing from a bitstream; decode an indication on a derivation mode from a bitstream; decode information on a reference frame comprising texture coordinates missing from the bitstream; reconstruct missing texture coordinates from the reference frame according to the derivation mode; and reconstruct a dynamic three-dimensional mesh sequence according to the texture coordinates.

According to a seventh aspect, there is provided computer program product for encoding comprising computer program code configured to, when executed on at least one processor, cause an apparatus or a system to: receive a dynamic three-dimensional mesh sequence, wherein the three-dimensional mesh represents a three-dimensional object with texture; determine which texture coordinates are not encoded into a bitstream; determine a derivation mode for the texture coordinates not to be encoded; encode into a bitstream an indication of the determined derivation mode; encode into a bitstream an information on a reference frame comprising texture coordinates for the current mesh frame; encode a base mesh into a bitstream with the texture coordinates which are to be encoded; and transmit the one or more bitstreams to a decoder.

According to an eighth aspect, there is provided computer program product for decoding comprising computer program code configured to, when executed on at least one processor, cause an apparatus or a system to: receive one or more bitstreams from an encoder; decode a base mesh from a bitstream texture coordinates; determine texture coordinates missing from a bitstream; decode an indication on a derivation mode from a bitstream; decode information on a reference frame comprising texture coordinates missing from the bitstream; reconstruct missing texture coordinates from the reference frame according to the derivation mode; and reconstruct a dynamic three- dimensional mesh sequence according to the texture coordinates.

According to an embodiment for encoding, in order to determine which texture coordinates are not encoded, the texture coordinates are tested whether they are recovered with any of the derivation mode with a quality exceeding a threshold.

According to an embodiment for encoding, the base mesh is generated from the dynamic three-dimensional mesh sequence, wherein the current mesh is the base mesh frame.

According to an embodiment for encoding, the base mesh is subdivided into submeshes, wherein the current mesh is a submesh frame.

According to an embodiment for encoding, a derivation mode of texture coordinates is indicated by a certain network abstraction layer unit type.

According to an embodiment for encoding, the derivation mode and the reference frame is signaled for each texture type of a mesh individually, the texture type being one of the following: color, reflectance, normal.

According to an embodiment for encoding, the type of mapping is signaled at an atlas sequence parameter set or an atlas frame parameter set.

According to an embodiment, the reference frame is one of the following: a previous reconstructed base mesh frame, an intra frame or any frame stored in a reconstructed base mesh frame buffer or mesh buffer. According to an embodiment, the derivation mode for texture coordinates is one of the following: when connectivity of the reference frame is the same as the current mesh frame, copying extracted texture coordinates from the reference frame to the current reconstructed base mesh frame; when connectivity of the reference frame is not the same as the current mesh frame, mapping extracted texture coordinates from the reference frame to the current reconstructed base mesh frame; when connectivity of the reference frame is not the same as the current mesh frame, mapping extracted texture coordinates from the reference frame to the current reconstructed base mesh frame with anchor points; when connectivity of the reference frame is not the same as the current mesh frame, mapping extracted texture coordinates from the reference frame to the current reconstructed base mesh frame with anchor points and refinable vertices.

According to an embodiment, the computer program product is embodied on a non-transitory computer readable medium.

Description of the Drawings

In the following, various embodiments will be described in more detail with reference to the appended drawings, in which

Fig. 1 a shows an example of a volumetric media conversion at an encoder;

Fig. 1 b shows an example of a volumetric media reconstruction at a decoder;

Fig. 2 shows an example of block to patch mapping; Fig. 3a shows an example of an atlas coordinate system;

Fig. 3b shows an example of a local 3D patch coordinate system;

Fig. 3c shows an example of a final target 3D coordinate system;

Fig. 4 shows a simplified example of a subdivision step of a triangle into four triangles;

Fig. 5 shows an example of a multiresolution analysis of a mesh;

Fig. 6 shows an example of an encoder comprising a pre-processing module for generating a mesh;

Fig. 7 shows an example of pre-processing steps at an encoder;

Fig. 8 shows an example of an intra frame encoder for mesh compression;

Fig. 9 shows an example of an inter frame encoder for mesh compression;

Fig. 10 shows an example of a decoder comprising a post-processing module for reconstruing a dynamic mesh sequence;

Fig. 11 shows an example of a decoding process in intra mode;

Fig. 12 shows an example of a decoding process in inter mode;

Fig. 13 shows an example of a base-mesh encoder;

Fig. 14 shows another example of a base-mesh encoder;

Fig. 15 shows an example of a base-mesh decoder;

Fig. 16 shows an example of segmentation of a mesh into sub-meshes; Fig. 17 shows an example with two submeshes;

Fig. 18 shows an example of texture coordinates mapping;

Fig. 19 is a flowchart illustrating a method for encoding according to an embodiment;

Fig. 20 is a flowchart illustrating a method for decoding according to another embodiment; and

Fig. 21 shown an example of an apparatus. Embodiments

The present embodiments relate to encoding, signalling, and rendering a volumetric video based on mesh coding. The present embodiments are applicable with a Video-based Dynamic Mesh Coding (V-DMC) ISO/IEC 23090-29, which is an application of the Visual Volumetric Video Coding (V3C) standard family ISO/IEC 23090-5.

The following description and drawings are illustrative and are not to be construed as unnecessarily limiting. The specific details are provided for a thorough understanding of the disclosure. However, in certain instances, well- known or conventional details are not described in order to avoid obscuring the description. References to one or an embodiment in the present disclosure can be, but not necessarily are, reference to the same embodiment and such references mean at least one of the embodiments.

Reference in this specification to “one embodiment” or “an embodiment” means that a particular feature, structure, or characteristic described in connection with the embodiment in included in at least one embodiment of the disclosure. Volumetric video data represents a three-dimensional scene or object and can be used as input for AR, VR and MR applications. Such data describes geometry (shape, size, position in 3D space) and respective attributes (e.g., color, opacity, reflectance, ...), plus any possible temporal transformations of the geometry and attributes at given time instances (like frames in 2D video). Volumetric video is either generated from 3D models, i.e., CGI, or captured from real-world scenes using a variety of capture solutions, e.g., multi-camera, laser scan, combination of video and dedicated depth sensors, and more. Also, a combination of CGI and real-world data is possible. Representation formats for such volumetric data are triangle meshes, point clouds, or voxels. Temporal information about the scene can be included in the form of individual capture instances, i.e., “frames” in 2D video, or other means, e.g., position of an object as a function of time.

Increasing computational resources and advances in 3D data acquisition devices have enabled reconstruction of highly detailed volumetric video representations or natural scenes. Infrared, lasers, time-of-flight, and structured light are all examples of devices that can be used to construct 3D video data. Representation of the 3D data depends on how the 3D data is used. Dense Voxel arrays have been used to represent volumetric medical data. In 3D graphics, polygonal meshes are extensively used. Point clouds on the other hand are well suited for applications such as capturing real world 3D scenes where the topology is not necessarily a 2D manifold. Another way to represent 3D data is coding this 3D data as set of texture and depth map as is the case in the multi-view plus depth. Closely related to the techniques used in multi-view plus depth is the use of elevation maps, and multi-level surface maps.

In the following, a short reference of ISO/IEC DIS 23090-5 Visual Volumetric Video-based Coding (V3C) and Video-based Point Cloud Compression (V- PCC) 2nd Edition is given. Visual volumetric video comprising a sequence of visual volumetric frames, if uncompressed, may be represented by a large amount of data, which can be costly in terms of storage and transmission. This has led to the need for a high coding efficiency standard for the compression of visual volumetric data.

V3C enables the encoding and decoding processes of a variety of volumetric media by using video and image coding technologies. This is achieved through first a conversion of such media from their corresponding 3D representation to multiple 2D representations, also referred to as V3C video components, before coding such information. Such representations may include occupancy, geometry, and attribute components. The occupancy component can inform a V3C decoding and/or rendering system of which samples in the 2D components are associated with data in the final 3D representation. The geometry component contains information about the precise location of 3D data in space, while attribute components can provide additional properties, e.g., texture or material information, of such 3D data. An example is shown in Figures 1 a and 1 b, where Figure 1a presents volumetric media conversion at an encoder, and where Figure 1 b presents volumetric media reconstruction at a decoder side. The 3D media is converted to a series of 2D representations: occupancy 101 , geometry 102, and attributes 103. Additional information may also be included in the bitstream to enable inverse reconstruction.

Additional information that allows associating all these V3C video components and enables the inverse reconstruction from a 2D representation back to a 3D representation is also included in a special component, referred to in this document as the atlas 104. An atlas 104 consists of multiple elements, named as patches. Each patch identifies a region in all available 2D components and contains information necessary to perform the appropriate inverse projection of this region back to the 3D space. The shape of such regions is determined through a 2D bounding volume associated with each patch as well as their coding order. The shape of these regions is also further refined after the consideration of the occupancy information.

Atlases may be partitioned into patch packing blocks of equal size. The 2D bounding volumes of patches and their coding order determine the mapping between the blocks of the atlas image and the patch indices. Figure 2 shows an example of block to patch mapping with 4 projected patches onto an atlas when asps_patch_precedence_order_flag is equal to 0. Projected points are represented with dark grey. The area that does not contain any projected points is represented with light grey. Patch packing blocks are represented with dashed lines. The number inside each patch packing block represents the patch index of the patch to which it is mapped.

Axes orientations are specified for internal operations. For instance, the origin of the atlas coordinates is located on the top-left corner of the atlas frame. For the reconstruction step, an intermediate axes definition for a local 3D patch coordinate system is used. The 3D local patch coordinate system is then converted to the final target 3D coordinate system using appropriate transformation steps.

Figure 3a shows an example of a single patch 320 packed onto an atlas image 310. This patch 520 is then converted to a local 3D patch coordinate system (U, V, D) defined by the projection plane with origin O’, tangent (U), bi-tangent (V), and normal (D) axes. For an orthographic projection, the projection plane is equal to the sides of an axis-aligned 3D bounding volume 330, as shown in Figure 3b. The location of the bounding volume 330 in the 3D model coordinate system, defined by a left-handed system with axes (X, Y, Z), can be obtained by adding offsets TilePatch3dOffsetU, TilePatch3DOffsetV, and TilePatch3DOffsetD, as illustrated in Figure 3c.

Coded V3C video components are referred to in this disclosure as video bitstreams, while a coded atlas is referred to as the atlas bitstream. Video bitstreams and atlas bitstreams may be further split into smaller units, referred to here as video and atlas sub-bitstreams, respectively, and may be interleaved together, after the addition of appropriate delimiters, to construct a V3C bitstream.

V3C patch information is contained in atlas bitstream, atlas_sub_bitstream(), which contains a sequence of NAL units. NAL unit is specified to format data and provide header information in a manner appropriate for conveyance on a variety of communication channels or storage media. All data are contained in NAL units, each of which contains an integer number of bytes. A NAL unit specifies a generic format for use in both packet-oriented and bitstream systems. The format of NAL units for both packet-oriented transport and sample streams is identical except that in the sample stream format specified in Annex D of ISO/IEC 23090-5 each NAL unit can be preceded by an additional element that specifies the size of the NAL unit.

NAL units in atlas bitstream can be divided to atlas coding layer (ACL) and non-atlas coding layer (non-ACL) units. The former dedicated to carry patch data while the later to carry data necessary to properly parse the ACL units or any additional auxiliary data.

In the nal_unit_header() syntax nal_unit_type specifies the type of the RBSP (Raw Byte Sequence Payload) data structure contained in the NAL unit as specified in Table 4 of ISO/IEC 23090-5. nal_layer_id specifies the identifier of the layer to which an ACL NAL unit belongs or the identifier of a layer to which a non-ACL NAL unit applies. The value of nal_layer_id shall be in the range of 0 to 62, inclusive. The value of 63 may be specified in the future by ISO/IEC. Decoders conforming to a profile specified in Annex A of ISO/IEC 23090-5 shall ignore (i.e., remove from the bitstream and discard) all NAL units with values of nal_layer_id not equal to 0.

While designing V3C specification it was envisaged that amendments or new editions can be created in the future. In order to ensure that the first implementations of V3C decoders are compatible with any future extension, a number of fields for future extensions to parameter sets were reserved.

For example, second edition of V3C introduced an extension in VPS related to MIV and packed video component.

A polygon mesh is a collection of vertices, edges and faces that defines the shape of a polyhedral object in 3D computer graphics and solid modelling. The faces usually consist of triangles (triangle mesh), quadrilaterals (quads), or other simple convex polygons (n-gons), since this simplifies rendering, but may also be more generally composed of concave polygons, or even polygons with holes. Objects created with polygon meshes are represented by different types of elements. These include vertices, edges, faces, polygons, and surfaces. In many applications, only vertices, edges and either faces or polygons are stored.

Polygon meshes are defined by the following elements: • Vertex: A position in 3D space defined as (x, y, z) along with other information such as color (r, g, b), normal vector and texture coordinates.

• Edge: A connection between two vertices.

• Face: A closed set of edges, in which a triangle face has three edges, and a quad face has four edges. A polygon is a coplanar set of faces. In systems that support multi-sided faces, polygons and faces are equivalent. Mathematically a polygonal mesh may be considered an unstructured grid, or undirected graph, with additional properties of geometry, shape, and topology.

• Surfaces: or smoothing groups, are useful, but not required to group smooth regions.

• Groups: Some mesh formats contain groups, which define separate elements of the mesh, and are useful for determining separate subobjects for skeletal animation or separate actors for non-skeletal animation.

• Materials: defined to allow different portions of the mesh to use different shaders when rendered.

• UV coordinates: Most mesh formats also support some form of UV coordinates which are a separate 2D representation of the mesh "unfolded" to show what portion of a 2-dimensional texture map applies to different polygons of the mesh. It is also possible for meshes to contain other vertex attribute information such as color, tangent vectors, weight maps to control animation, etc. (sometimes also called channels).

Mesh data may be compressed directly without projecting it into 2D-planes, like in V-PCC based mesh coding. In fact, the anchor for V-PCC mesh compression call for proposals (CfP) utilizes off-the shelf mesh compression technology, Draco (https://google.github.io/draco/), for compressing mesh data excluding textures. Draco is used to compress vertex positions in 3D, connectivity data (faces) as well as UV coordinates. Additional per-vertex attributes may be also compressed using Draco. The actual UV texture may be compressed using traditional video compression technologies, such as H.265 or H.264. Draco uses an edgebreaker algorithm at its core to compress 3D mesh information. It offers a good balance between simplicity and efficiency, and is part of Khronos endorsed extensions for the gITF specification. The main idea of the algorithm is to traverse mesh triangles in a deterministic way so that each new triangle is encoded next to an already encoded triangle. This enables prediction of vertex specific information from the previously encoded data by simply adding delta to the previous data. Edgebreaker utilizes symbols to signal how each new triangle is connected to the previously encoded part of the mesh. Connecting triangles in such a way results on average in 1 to 2 bits per triangle when combined with existing binary encoding techniques.

The V-DMC standardization works have started after the completion of the call for proposal (CfP) issued by MPEG 3DG (ISO/IEC SC29 WG 2) on integration of MESH compression into the V3C family of standards (ISO/IEC 23090-5). The retained technology after the CfP result analysis is based on multiresolution mesh analysis and coding. This approach comprises

• generating a base mesh that is a simplified (low resolution) mesh approximation of the original mesh, called base mesh (this is done for all frames of the dynamic mesh sequence) mi

• performing several mesh subdivision iterative steps (e.g., each triangle is converted into four triangles by connecting the triangle edge midpoints as illustrated in Figure 4) on the generated base mesh, generating other approximation meshes mⁿi where n stands for the number of iterations with im = m°i

• defining displacement vectors di, also named error vectors, for each vertex of each mesh approximation mⁿ/with n > 0, noted dⁿi

• For each subdivision level, the deformed mesh, obtained by mⁿi + d i.e., by adding the displacement vectors to the subdivided mesh vertices generates the best approximation of the original mesh at that resolution, given the base mesh and prior subdivision levels.

• The displacement vectors may undergo a lazy wavelet transform prior to compression.

• The attribute map of the original mesh is transferred to the deformed mesh at the highest resolution (i.e., subdivision level) such that texture coordinates are obtained for the deformed mesh and a new attribute map is generated. An example of a multi-resolution analysis of a mesh is illustrated on Fig. 5. A base mesh 510 undergoes a first step of subdivision 515, and error vectors 520 are added to each vertex. After a series of iterative subdivision 525 and displacements, the higher resolution mesh 530 is generated. The connectivity of the highest resolution deformed mesh 535 is generally different from the original mesh, however, the geometry of the deformed mesh is a good approximation of the original mesh geometry.

The encoding process can be separated into two main modules, as shown in Figure 6. Figure 6 illustrates an example of an encoder comprising a preprocessing module 610 and an actual encoder module 620. Figure 7 illustrates the steps of the pre-processing module comprising: decimation (reducing the original mesh resolution to produce a base mesh) 710, UV-atlas isocharting (creating a parameterization of the base mesh) 720 and the subdivision surface fitting 730.

The examples of an encoder are illustrated on Figure 8 and Figure 9, where Figure 8 illustrates an INTRA frame encoder and Figure 9 illustrates an INTER frame encoder. In the INTER frame encoder of Figure 9, the base mesh connectivity of the first frame of a group of frames is imposed to the subsequent frame’s base meshes to improve compression performance.

In the INTRA frame encoding of Figure 8, inputs to this module are the base mesh (that is an approximation of the input mesh but that contains less faces and vertices) 802, the patch information 801 related to the input base mesh, the displacements 803, the static/dynamic input mesh frame 804 and the attribute map 805. Outputs of this module is a compressed bitstream 895 that contains a V3C extended signalling sub-bitstream including patch data information, compressed base mesh substream, a compressed displacement video component substream and a compressed Attribute video component sub-bitstream. The module takes the input base mesh and first quantize its data in the Quantization module, which can be dynamically tuned by a Control Module. The quantized base mesh is then encoded with the static mesh encoder module, which outputs a compressed base mesh sub-bitstream that is multiplexed in the output bitstream. The encoded base mesh is decoded in the Static Mesh Decoder module that generates a reconstructed quantized base mesh. The Update Displacements module takes as input the reconstructed quantizes base mesh, the pristine base mesh, and the input displacements to generate new updated displacements that are remapped to the reconstructed base mesh data in order to avoid precision errors due to the static mesh encoding and decoding process. The updated displacements are filtered with a wavelet transform in the Wavelet Transform module (that also takes as input the reconstructed base mesh) and then quantized in the Quantization module. The quantized wavelet coefficients produced from the updated displacements are then packed into a video component in the Image Packing module. This video component is then encoded with a 2D video encoder such as HEVC, WC, etc., in the Video Encoder module, and the output compressed displacement video component sub-bitstream is multiplexed along with the V3C signalling information sub-bitstream into the output compressed bitstream. Then the compressed displacement video component is first decoded and reconstructed and then unpacked into encoded and quantized wavelet coefficients in the Image Unpacking module. These wavelet coefficients are then unquantized in the inverse quantization module and reconstructed with the inverse wavelet transform module that generates reconstructed displacements. The reconstructed base mesh is unquantized in the inverse quantization module and the unquantized base mesh is combined with the reconstructed displacements in the Reconstruct Deformed Mesh module to obtain the reconstructed deformed mesh. This reconstructed deformed mesh is then fed into the Attribute Transfer module together with the Attribute map produced by the pre-processing and the input static/dynamic mesh frame. The output of the Attribute Transfer module is an updated attribute map that now corresponds to the reconstructed deformed mesh frame. The updated attribute map is then padded, after which it undergoes a color conversion. Finally, the attribute map is encoded as a video component with a 2D video codec such as HEVC or WC, in the Padding, Color Conversion and Video encoder modules, respectively. The output compressed attribute map bitstream is multiplexed into the encoder output bitstream.

The inter encoding process, shown in Figure 9, is similar to the intra encoding process of Figure 8 with the following changes. The reconstructed reference base mesh 910 is an input of the inter coding process. A new module called Motion Encoder 950 takes as input the quantized input base mesh and the reconstructed quantized reference base mesh 910 to produce compressed motion information encoded as a compressed motion bitstream, which is multiplexed into the encoder output compressed bitstream. All other modules and processes are similar to the intra encoding case.

The compressed bitstream generated by the encoder multiplexes:

• A sub-bitstream with the encoded base mesh using a static mesh codec

• A sub-bitstream with the encoded motion data using an animation codec for base meshes in case INTER coding is enabled

• A sub-bitstream with the wavelet coefficients of the displacement vectors packed in an image and encoded using a video codec

• A sub-bitstream with the attribute map encoded using a video codec.

• A sub-bitstream that contains all metadata required to decode and reconstruct the mesh sequence based on the aforementioned subbitstreams. The signalling of the metadata is based on the V3C syntax and includes necessary extensions that are specific to meshes.

Figure 10 illustrates a decoding process according to an embodiment. First the compressed bitstream 1010 is demultiplexed into sub-bitstreams that are reconstructed, i.e., metadata, reconstructed base mesh, reconstructed displacements, and the reconstructed attribute map data. The reconstruction of the mesh sequence is performed based on that data in the post-processing module 1060.

Figure 11 and Figure 12 illustrate the decoding process in INTRA and INTER mode, respectively.

The intra frame decoding process as shown in example of Figure 11 comprises the following modules and processes. First the input compressed bitstream is de-multiplexed 1110 into V3C extended atlas data information (or patch information), a compressed static mesh bitstream, a compressed displacement video component and a compressed attribute map bitstream, respectively. The static mesh decoding module 1120 converts the compressed static mesh bitstream into a reconstructed quantized static mesh, which represents a base mesh. This reconstructed quantized base mesh undergoes inverse quantization in the inverse quantization module 1125 to produce a decoded reconstructed base mesh. The compressed displacement video component bitstream is decoded in the video decoding module 1130 to generate a reconstructed displacement video component. This displacement video component is unpacked into reconstructed quantized wavelet coefficients in the image unpacking module 1135. Reconstructed quantized wavelet coefficients are inverse quantized in the inverse quantization module 1140 and then undergo an inverse wavelet transform in the inverse wavelet transform module 1145, that produces decoded displacement vectors. The reconstruct deformed mesh module 1150 takes into account the patch information and takes as input the decoded reconstructed base mesh and decoded displacement vectors to produce the output decoded mesh frame. The compressed attribute map video component is decoded in the video decoding module 1130, and possibly undergoes color conversion 1160 to produce a decoded attribute map frame that corresponds to the decoded mesh frame.

The inter decoding process, shown in Figure 12, is similar to the intra decoding process module with the following changes. The decoder also demultiplexes a compressed information bitstream. A decoded reference base mesh 1200 is taken as input of a motion decoder module 1210 together with the compressed motion information sub-bitstream. This decoded reference base mesh is selected from a buffer of previously decoded base mesh frames (by the intra decoder process for the first frame of a group of frames). The reconstruction of base mesh module 1250 takes the decoded reference base mesh and the decoded motion information as input to produce a decoded reconstructed quantized base mesh. All other processes are similar to the intra decoding process as shown in Figure 11 .

The signalling of the metadata and substreams produced by the encoder and ingested by the decoder was proposed as an extension of V3C in the technical submission to the dynamic mesh coding CfP, and should be considered as purely indicative for the moment. It is as follows and mainly consists in additional V3C unit header syntax, additional V3C unit payload syntax, and Mesh Intra patch data unit. V3C unit header syntax

V3C unit payload syntax

Mesh Intra patch data unit

A refinement of the metadata and substreams signaling are as follows:

- The output of the base mesh substream decoder is called “base meshes”;

- Each base mesh can have one or more submeshes. A “submesh” is a set of vertices, their connectivity and the associated attributes which can be decoded completely independently in a mesh frame.

- Term “resampled base meshes” refers to an output of the mesh subdivision process. The inputs to the process are the base meshes (or sets of submeshes) and the information from the atlas data substream on how to subdivide/resample the meshes(submeshes).

- “A displacement video” is the output of the displacement decoder. The inputs to the process are the decoded geometry video and the information from the atlas data substream on how to interpret/process this video. The displacement video contains displacement values to be added to the corresponding vertices.

- “A facegroupld” is one of the attribute types assigned to each triangle face of the resampled base meshes. Facegroupld can be compared with the identifications of the subparts in a patch to determine the corresponding facegroups to the patch. If facegrould is not conveyed through the base mesh substream decoder, it can be derived by the information in the atlas data substream.

V3C unit Compressed base meshes are signalled in a new substream, named as the Base Mesh data substream (unit type V3C_MD). As with other v3c units, the unit type, and its associated v3c parameter set id and atlas id are signalled in the v3c_unit_header().

V3C parameter set extension

A new extension needs to be introduced in the v3c_parameter_set syntax structure to handle V-DMC. Several new parameters are introduced in this extension including the following:

• vps_ext_mesh_data_facegroup_id_attribute_present_flag equals

1 indicates that one of the attribute types present in the base mesh data stream is the facegroup Id.

• vps_ext_mesh_data_attribute_count indicates the number of total attributes in the base mesh including both the attributes signalled through the base mesh data substream and the attributes signalled in the video sub streams (using ai_attribute_count). When vps_ext_mesh_data_facegroup_id_attribute_present_flag equals 1 , it shall be greater or equal to ai_attribute_count+1 . This can be constrained by profile/levels.

• The types of attributes that are signalled through the base mesh substream and not through the video substreams are signalled are signaled as vps_ext_mesh_attribute_type data types.

When vps_ext_mesh_data_facegroup_id_attribute_present_flag equals 1 , one of the vps_ext_mesh_attribute_type must be a facegroup d.

• vps_ext_mesh_data_substream_codec_id indicates the identifier of the codec used to compress the base mesh data. This codec may be identified through the profiles a component codec mapping SEI message, or through means outside this document.

• vps_ext_attribute_frame_width[i] and vps_ext_attribute_frame_height[i] indicate the corresponding with and height of the video data corresponding to the i-th attribute among the attributes signalled in the video substreams.

Atlas sequence parameter set extension

The information contained in this extension can be overwritten by the same information in the AFPS extension or the patch data units. The following parameters are introduced:

• asps_vmc_ext_prevent_geometry_video_conversion_flag prevents the outputs of the geometry video substream decoder from being converted. When the flag is true, the outputs are used as they are without any conversion process from Annex B in [2], When the flag is true, the size of geometry video shall be same as nominal video sizes indicated in the bitstream.

• asps_vmc_ext_prevent_attribute_video_conversion_flag prevents the outputs of attribute video substream decoder from being converted. When the flag is true, the outputs are used as they are without any conversion process from Annex B in [2], When the flag is true, the size of attribute video shall be same as nominal video sizes indicated in the bitstream.

• asps_vmc_ext_subdivision_method and asps_vmc_ext_subdivision_iteration_count signal information about the subdivision method.

• asps_vmc_ext_transform_index indicates the transform applied to the displacement. The transform index can indicate any transform is not applied. When the transform is LINEAR LIFTING, the necessary parameters are signalled as vmc_lifting_transform_parameters. • asps_vmc_ext_patch_mapping_method indicates how to map a subpart of a submesh to a patch. o When asps_vmc_ext_patch_mapping_method is equal to 0, all the triangles in the corresponding submesh are associated with the current patch. In this case, there is only one patch associated with the submesh. o When asps_vmc_ext_patch_mapping_method is equal to 1 , the subpart ds are explicitly signalled in the mesh patch data unit to indicate the associated subparts. o In other cases, the triangle faces in the corresponding submesh are divided into subparts by the method indicated by asps_vmc_ext_patch_mapping_method.

• asps_vmc_ext_tjunction_removing_method indicates the method to remove t-junctions created by different subdivision methods or by different subdivision iterations of two triangles sharing an edge. • asps_vmc_ext_num_attribute indicates the total number of attributes that the corresponding mesh carries. Its value shall be less or equal to vps_ext_mesh_data_attribute_count.

• asps_vmc_ext_attribute_type is the type of the i-th attribute and it shall be one of ai_attribute_type_ids or vps_ext_mesh_attribute_types.

• asps_vmc_ext_direct_atrribute_projection_enabled_flag indicates that the 2d locations where attributes are projected are explicitly signalled in the mesh patch data units. Therefore, the projection id and orientation index in V3C V-PCC ISO/IEC 23090-5:2021 can be also used as in ISO/IEC 23090-5:2021 .

Atlas Frame Parameter set extension

• afps_vmc_ext_single_submesh_in_frame_flag indicates there is only one submesh for the mesh frame • When afps_vmc_ext_overriden_flag in afps_vmc_extension() is true, the subdivision method, displacement coordinate system, transform index, transform parameters, and attribute transform parameters can be signalled again and the information ioverrides the one signalled in asps_vmc_extension(). • afps_vmc_ext_single_attribute_tile_in_frame_flag indicates there is only one tile for each attribute signalled in the video streams. • afps_ext_vmc_attribute_tile_information() contains the tile information for the attributes signalled through the video substreams.

Atlas Tile Header

A tile can be associated with one or more submeshes whose identification is ath_submesh_id.

Patch data unit

As with the V-PCC Patch data units, Mesh patch data units are signalled in the Atlas data substream. Mesh Intra patch data unit, Mesh Inter patch data unit, Mesh Merge patch data unit, and Mesh Skip patch data unit can be used.

• mdu_submesh_id indicate which submesh the patch is associated with among those indicated in the atlas tile header.

• mdu_vertex_count_minus1 and mdu_triangle_count_minus1 indicate the number of vertices and triangles associated with the current patch.

• When asps_vmc_ext_patch_mapping_method is not 0, the syntax elements mdu_num_subparts and mdu_subpart_id are signalled. When asps_vmc_ext_patch_mapping_method is 1 , the associated triangle faces are the union of the triangle faces whose facegroupld is equal to mdu_subpart_id.

• When mdu_patch_parameters_enable_flag is true, the subdivision method, displacement coordinate system, transform index, transform parameters, and attribute transform parameters can be signalled again and the information overrides the corresponding information signalled in in asps_vmc_extension().

The signaling of the base mesh substream is also under investigation and is illustrated in Figure 13 in simplified and tentative manner.

One of the key features of the current V-DMC specification design is the support for a base mesh signal that can be encoded using any currently or future specified static mesh codec. For example, such information could be coded using Draco 3D Graphics Compression. This representation could provide the basis for applying other decoded information to reconstruct the output mesh frame within the context of V-DMC.

Furthermore, for coding dynamic mesh frames, it is highly desirable to be able to exploit any temporal correlation that may exist with previously coded base mesh frames. In Figures 13, 14, 15, where Figure 14 illustrates another example of a base-mesh encoder, and Figure 15 illustrates an example of a base-mesh decoder, this has been accomplished by encoding a mesh motion field instead of directly encoding the base mesh, and using this information and a previously encoded base mesh to reconstruct the base mesh of the current frame. This approach could be seen as the equivalent of inter prediction in video coding.

It is desirable also to associate all coded base mesh frames or motion fields with information that could help determine their decoding output order as well as their referencing relationships. It is possible, for example, that better coding efficiency could be achieved if the coding order of all frames does not follow the display order or by using as reference for generating a motion field for frame N an arbitrary previously coded motion field or base mesh instead of the immediately previous coded one. Also highly desirable is the ability to instantly detect random access points and independently decode multiple sub-meshes that together can form a single mesh, much like subpictures in video compression.

For all the above reasons, a new Base Mesh Substream format is introduced. This new format is very similar to a video coding format such as HEVC or the atlas sub-bitstream used in V3C, with the base mesh sub-bitstream also constructed using NAL units. High Level Syntax (HLS) structures such as base mesh sequence parameter sets, base mesh frame parameter sets, submesh layer are also specified.

One of the desirable features of this design is the ability to segment a mesh into multiple smaller partitions, referred to in this document as submeshes. An example of this is illustrated in Figure 16. The submeshes shown in (b) can be decoded completely independently, which can help with partial decoding and spatial random access. Although it may not be a requirement for all applications, some applications may require that the segmentation in submeshes remains consistent and fixed in time. The submeshes do not need to use the same coding type, i.e., for one frame one submesh may use intra coding while for another inter coding could be used at the same decoding instance, but it is commonly a requirement that the same coding order is used and the same references are available for all submeshes corresponding at a particular time instance. Such restrictions can help guarantee proper random access capabilities for the entire stream. An example where two submeshes are used is shown in the Figure 17.

NAL unit syntax

As discussed earlier, the new bitstream is also based on NAL units, and it is similar to those of the atlas substream in V3C. The syntax is provided below.

General NAL unit syntax

NAL unit header syntax

NAL unit semantics

This section contains some of the semantics that correspond to the above syntax structures. More details would be provided for syntax elements that have not been defined in complete detail.

General NAL unit semantics

NumBytesInNalllnit specifies the size of the NAL unit in bytes. This value is required for decoding of the NAL unit. Some form of demarcation of NAL unit boundaries is necessary to enable inference of NumBytesInNalUnit. One such demarcation method is specified in Annex TBD for the sample stream format. Other methods of demarcation can be specified outside this document.

It is to be appreciated that the mesh coding layer (MCL) is specified to efficiently represent the content of the mesh data. The NAL is specified to format that data and provide header information in a manner appropriate for conveyance on a variety of communication channels or storage media. All data are contained in NAL units, each of which contains an integer number of bytes. A NAL unit specifies a generic format for use in both packet-oriented and bitstream systems. The format of NAL units for both packet-oriented transport and sample streams is identical except that in the sample stream format specified in Annex TBD each NAL unit can be preceded by an additional element that specifies the size of the NAL unit. rbsp_byte[ i ] is the i-th byte of an RBSP. An RBSP is specified as an ordered sequence of bytes as follows:

The RBSP contains a string of data bits (SODB) as follows: • If the SODB is empty (i.e., zero bits in length), the RBSP is also empty.

• Otherwise, the RBSP contains the SODB as follows:

1 )The first byte of the RBSP contains the first (most significant, leftmost) eight bits of the SODB; the next byte of the RBSP contains the next eight bits of the SODB, etc., until fewer than eight bits of the SODB remain.

2)The rbsp_trailing_bits( ) syntax structure is present after the SODB as follows: i)The first (most significant, left-most) bits of the final RBSP byte contain the remaining bits of the SODB (if any). ii)The next bit consists of a single bit equal to 1 (i.e., rbsp_stop_one_bit). iii)When the rbsp_stop_one_bit is not the last bit of a byte-aligned byte, one or more bits equal to 0 (i.e. instances of rbsp_alignment_zero_bit) are present to result in byte alignment.

Syntax structures having these RBSP properties are denoted in the syntax tables using an "_rbsp" suffix. These structures are carried within NAL units as the content of the rbsp_byte[ i ] data bytes. The association of the RBSP syntax structures to the NAL units is as specified in Table 4 of ISO/IEC 23090- 5.

It is to be noticed that when the boundaries of the RBSP are known, the decoder can extract the SODB from the RBSP by concatenating the bits of the bytes of the RBSP and discarding the rbsp_stop_one_bit, which is the last (least significant, right-most) bit equal to 1 , and discarding any following (less significant, farther to the right) bits that follow it, which are equal to 0. The data necessary for the decoding process is contained in the SODB part of the RBSP.

NAL unit header semantics

Similar NAL unit types, as for the atlas case, were defined for the base mesh enabling similar functionalities for random access and segmentation of the mesh. Unlike the atlas that is split into tiles, in this document we define the concept of a sub-mesh and define specific nal units that correspond to coded mesh data. In addition, NAL units that can include metadata such as SEI messages are also defined. In particular, the base mesh NAL unit types supported are specified as follows:

Raw byte sequence payloads, trailing bits, and byte alignment syntax Base mesh sequence parameter set RBSP syntax

As with similar bitstreams, the primary syntax structure that is defined for a base mesh bitstream is a sequence parameter set. This syntax structure contains basic information about the bitstream, identifying features for the codecs supported for either the intra coded and inter coded meshes, as well as information about references

General base mesh sequence parameter set RBSP syntax

• bmsps_intra_mesh_codec_id indicates the static mesh codec used to encode the base meshes in this base mesh substream. It could be associated with a specific mesh or motion mesh codec through the profiles specified in the corresponding specification, or could be explicitly indicated with an SEI message as is done in the V3C specification for the video sub-bitstreams.

• bmsps_intra_mesh_data_size_precision_bytes_minus1 (+1) specifies the precision, in bytes, of the size of the coded mesh data. • bmsps_inter_mesh_codec_present_flag indicates if a specific codec indicated by bmsps_inter_mesh_codec_id is used to encode the inter predicted submeshes

• bmsps_inter_mesh_data_size_precision_bytes_minus1 (+1 ) specifies the precision, in bytes, of the size of the inter predicted mesh data. This precision is signalled considering the size of the coded mesh data and the inter predicted mesh data(e.g. motion field) can be significantly different. • bmsps_facegroup_segmentation_method indicates how facegroups could be derived for a mesh. A facegroup is a set of triangle faces in a submesh. Each triangle face is associated with a Facegroupld indicating the facegroup it belongs to. When bmsps_facegroup_segmentation_method is 0, then Facegroupld is present directly in the coded submesh. Other values indicate that the facegroup can be derived using different methodologies based on the characteristics of the stream. For example, value 1 means that there is no Facegroupld associated with any face. A value 2 means that all faces are identified with a single ID, a 3 that facegroups are identified based on the connected component method, while a value of 4 indicates that each individual face has its own unique ID. Currently ue(v) is used to indicate bmsps_facegroup_segmentation_method, but fixed length coding or partitioning to more elements could have been used instead. Base Mesh Profile, tier, and level syntax

• bmptl_extended_sub_profile_flag providing support for sub profiles can be quite useful for further restricting the base mesh profiles depending on usage and applications. Base mesh frame parameter set RBSP syntax

The base mesh frame parameter set has the frame level information such as number of submeshes in the frames corresponding to one mfh_mesh_frm_order_cnt_lsb. A submesh is coded in one mesh_data_submesh_layer() and is independently decodable from other submeshes. In the case of inter frame prediction, a submesh can refer only to the submeshes with the same smh_id in its associated reference frames. The mechanism is equivalent to what is specified in 8.3.6.2.2 in V3C.

Base mesh submesh layer rbsp syntax

A bmesh_submesh_layer contains a submesh information. One or more bmesh_submesh_layer_rbsp can correspond to one mesh frame indicated by mfh_mesh_frm_order_cnt_lsb.

Submesh header syntax

• smh_id is the id of the current submesh contained in the mesh data submesh data.

• smh_type indicates how the mesh is coded. If smh_type is l_SUBMESH, the mesh data is coded with the indicated static mesh codec. If smh_type is P_SUBMESH, inter prediction is used to code the mesh data.

Submesh data unit smdu_intra_sub_mesh_unit( unitsize ) contains a sub mesh unit stream of size unitsize, in bytes, as an ordered stream of bytes or bits within which the locations of unit boundaries are identifiable from patterns in the data. The format of such sub mesh unit stream is identified by a 4CC code as defined by bmptl_profile_codec_group_idc or by a component codec mapping SEI message. smdu_inter_sub_mesh_unit( unitsize ) contains a sub mesh unit stream of size unitsize, in bytes, as an ordered stream of bytes or bits within which the locations of unit boundaries are identifiable from patterns in the data. The format of such sub mesh unit stream is identified by a 4CC code as defined by bmptl_profile_codec_group_idc or by a component codec mapping SEI message. In V-DMC, texture coordinates are transmitted for the base mesh and subdivided for the reconstructed displaced subdivided mesh. It follows that texture coordinates are only compressed in the base mesh substream as vertex attributes, by the chosen base mesh codec (for example based on edgebreaker and parallelogram geometry prediction).

The cost of encoding texture coordinates is significant, for example, for a mesh without texture coordinates encoded with edgebreaker, the file size is equal to 38883 bytes, while with texture coordinates encoded as vertex attributes, the file size increases to 55768 bytes (that is 43% more bitrate).

Two use cases have been identified, where V-DMC does not exploit the possible redundancy of texture coordinates from frame to frame, and where the cost of encoding the texture coordinates in the base mesh codec is suboptimal:

• T racked mesh sequences with a single texture frame: The first use case relates to tracked mesh sequences that have different geometry frames but a single texture frame. For such sequences, the vertex number and the connectivity typically are the same for every frame, and therefore, in the original format texture coordinates from the first frame as well as the texture map are reused for all consecutive frames. In V-DMC, this is not possible, and typically the texture coordinates are recomputed after the base mesh generation through simplification in the preprocessing module of the encoder. This leads the V-DMC bitstream to encode an attribute map for every frame, and these attribute maps are not stable temporally, leading to poor compression by a video codec.

• Dynamic mesh sequences with stable topology: The second use case relates to general dynamic meshes that are not necessarily tracked and for which the texture coordinates and texture maps are available for every frame of the sequence. In that case, V-DMC does not offer mechanisms to enable temporally stable texture coordinates and texture maps without sending the texture coordinates. However, if the topology is stable (i.e., the mesh geometry may have changed, but the number of holes, handles, and the total mesh area remains the same over time), then the mesh texture coordinates could be mapped from mesh frame to mesh frame, but V-DMC does not support this unless the mesh is tracked.

For the more general case, with V-DMC, temporally unstable attribute maps are generated and lead to suboptimal encoding results as for the first case or require encoding temporally stable texture coordinates for each base mesh and therefore comes with a 40% bitrate overhead for the base mesh substream.

There is no tool in the V-DMC framework that enables temporal skipping or predicting small changes of the texture coordinates for the two aforementioned example use cases.

By the present embodiments, instead of encoding all the texture coordinates as vertex “attributes”, some of the texture coordinates are not encoded as before. To determine which texture coordinates are not encoded, the encoder may use testing. For that, the encoder tests for each base mesh (submesh) frame if texture coordinates can be recovered with any of the derivation mode (SKIP, MAP, ASSITED MAP, REFINED MAP) with a correct quality as expressed by a cost function. As an example, the cost function can be a sum of squared differences. In a video encoder, the V-DCM encoder may text the derivation modes, and evaluates the rate distortion cost of selecting the mode: bitrate_for_signalling_mode + distortion(mode), where mode is one of the derivation modes. If no mode results in an acceptable cost compared to a threshold, the texture coordinates are encoded in the base mesh stream.

Sometimes there is no need to determine which texture coordinates are not encoded by testing. In such a scenario, the original sequence has only one texture frame for all mesh frames. The texture coordinates are then the same from frame to frame, but would be encoded at each frame. This is an example of a situation where the solution according to the present embodiments is not used.

After determining which texture coordinates are not encoded, the encoder will instruct the decoder how to reconstruct (i.e., which derivation mode should be used) the texture coordinates that were not encoded. The disclosure use terms “base mesh” and “submeshes”, when discussing the present embodiments. The base mesh is an output of an base mesh (subbitstream) decoder. The submesh is a partition of a base mesh, i.e., a set of vertices, their connectivity and the associated attributes signalled in the basemesh sub-bitstream. Submeshes can be decoded completely independently. Each base mesh consists of one or more submeshes. The present embodiments have the following aspects:

- signalling; and

- texture coordinate derivation modes at the decoder and encoder.

Since the present embodiments is about encoding some of the texture coordinates and not encoding other texture coordinates, this feature is taken into account in signalling. For example, it is signaled that the texture coordinates are present in a reference base mesh, the reference base mesh being a reference base mesh frame or submesh frame. On the other hand, it is also signaled that the texture coordinates are not present in a base mesh or submesh frame, whereupon they need to be derived by a decoder. The texture coordinate derivation mode to be used is signaled per a submesh frame at the decoder such as: SKIP, MAP, ASSISTED_MAP, REFINED_MAP. The present embodiment also provide high-level syntax for signalling the presence of anchor/feature vertices as vertex attribute data in the base mesh.

The texture coordinate derivation mode to be selected may depend on the connectivity of a mesh. Connectivity is defined as a list of primitives that connect the vertices of the mesh. Examples of the primitives are edges connecting two vertices, or faces connecting three or more vertices. In the texture coordinate derivation modes, the texture coordinates are copied from a frame to the next frame if connectivity is the same. The texture coordinates are mapped from a frame to the next frame if connectivity changes, if feature vertices are not available, if feature vertices are available, or if refinable vertices are available.

Mechanisms or modes for copying and/or mapping vertices are as follows. The difference between copying and mapping, is that in copying the vertex indices have to be exactly the same. In mapping, the vertex indices may be different, for example when some of them has been reordered or the number of indices is not the same. The following mechanisms and modes may be applied at the encoder and decoder in the same way:

• SKIP: in SKIP mode, the texture coordinates extracted by the decoder from the reference base mesh frame are copied to the current reconstructed base mesh frame. This addresses the use case with a single texture frame for the whole mesh sequence

• MAP: in MAP mode, the texture coordinates extracted by the decoder from the reference base mesh frame are mapped to the current reconstructed base mesh frame.

• ASSISTED_MAP: in ASSISTED_MAP mode, the texture coordinates extracted by the decoder from the reference base mesh frame are mapped to the current reconstructed base mesh frame by using anchor/feature points.

• REFINED_MAP: in REFINED_MAP mode, the texture coordinates extracted by the decoder from the reference base mesh frame are mapped to the current reconstructed base mesh frame by using anchor/feature points and refinable vertices, for example by specifying a number of smoothing iterations to be performed in the parameterization domain for those vertices.

These four modes (i.e., SKIP, MAP, ASSISTED_MAP, REFINED_MAP) require signaling the reference frame index, which indicates where to find the texture coordinate data to apply to the current mesh frame. This reference frame may be the previous reconstructed base mesh frame, an Intra (I) frame, or any frame referenced by a variable refFrameldx that is stored in a reconstructed base mesh frame buffer as illustrated on Figure 12 relating to a decoder, or mesh buffer as illustrated on Figure 15. The encoder and decoder can handle reference frames in the mesh buffer with a referenceFramelndexList, in a similar way as video encoders such as H.264 Advanced Video Coding (AVC) or H.265 High Efficiency Video Coding (HEVC) for example.

The modes, MAP, ASSISTED_MAP, REFINED_MAP, can be applied to a base mesh and/or to each available submesh present in the V-DMC stream. Applying and signalling these modes at submesh or base mesh level require the same mechanisms described hereafter. The SKIP mode addresses tracked meshes that have only one texture frame (or a small set of texture frames, called key frames) that should be re-used for every frame of the dynamic sequence. A tracked mesh means that the original mesh has been designed such that the order of vertices and their number is the same for all frames. This enables faster processing of the mesh sequence. This mode enables to ensure that V-DMC re-uses the same texture coordinates and texture frame at every mesh frame, such that the video codec used to compress the texture map will mostly use temporal skip prediction modes.

According to an embodiment relating to encoding, in case of such tracked sequence, with a single frame, the simplification module of the preprocessing module is applied to the first frame. The original indices of the vertices that are kept in the generated base mesh are also kept for the base meshes of the next frames as well as their connectivity in the base mesh. This ensures a tracking of the base meshes that is not offered by the current version of V-DMC framework. Other possibilities include a multi-frame optimization for the simplification, e.g., evaluating the cost of removing edges, not only for the current frame, but for a group of frames or the entire mesh sequence, in case the sequence is tracked.

More generally, the SKIP mode may be used for any tracked mesh sequence, not necessarily having a single texture frame.

The MAP mode addresses mesh sequences that are not tracked. In contrast, MAP mode addresses mesh sequences for which the encoder has determined that the mapping can be achieved through inter-mesh-frame fitting without feature points, e.g., by k-NN nearest neighbors between the reconstructed reference frame and the current reconstructed base mesh frame. This is the case for base mesh frames that do not have the exact same number of vertices or triangles, but that do not exhibit significant motion from one frame to the next one. For vertices of the current frame that do not have a mapping in the reference frame, the texture coordinates are obtained by averaging the texture coordinates of the neighboring vertices that were mapped to vertices of the reference base mesh. Figure 18 illustrates a simplified example of texture coordinates mapping using the ASSISTED_MAP from the reference base mesh to the current base mesh using feature vertices. The ASSISTED_MAP mode addresses mesh sequences that are not tracked. In addition, the ASSISTED_MAP mode addresses mesh sequences for which the encoder detects that the inter-mesh- frame fitting requires feature points matching as the motion is more complex or the amplitude of the motion does not enable a k-NN search to provide a correct mapping. In this case, feature vertices are signaled in the reference base mesh as vertex attributes, for example with a flag (reference/not reference). The order of these reference feature points is provided by the order of vertex indices in the reference base mesh frame. The encoder may encode motion vectors for the feature vertices using a motion encoder as in the V- DMC framework, but instead of encoding motion for all vertices, which is expensive in bitrate, the encoding is only made for the sparser feature vertices.

A k-NN search is performed to identify the feature vertices in the current base mesh frame based on the encoded motion vector. Once feature vertices are detected on the current base mesh frame, the mapping between non feature vertices of the reference base mesh frame and the current frame is performed by k-NN search using connected vertices of the feature vertices. The mapping may then be performed by propagating through connected vertices of already mapped vertices in a recursive manner. That is, for all not-yet-mapped vertices v_i in the neighborhood (i.e., connected by an edge) of an already mapped vertex v_l, with rv_l being the reference vertex that was mapped to v_l from the reference mesh frame, and let rv_j be the connected neighbors of rv_j in the reference mesh frame, a mapping is obtained by nearest neighbors (k-NN) to perform a one to one mapping of each v_i to each rv_j vertex. If the number of v_i vertices is larger than the number of rv_j vertices, the unmapped vertices v_j are added to a unmapped pool that is visited once the number of vj vertices is smaller than the number of rv_j vertices. If after this process, the pool is nonempty, the texture coordinates of the unmapped v_j vertices are obtained by linear interpolation of the texture coordinates of their neighboring and already mapped vertices. Feature vertices can be identified by the encoder through user assistance, or automatically by using known mesh vertex features such as protrusion extrema, curvature, salient points, or for example simply by detecting the vertices that after N smoothing iterations still have a normal displacement above a given threshold compared to their vertex neighborhood.

The REFINED_MAP is a refinement of the ASSISTED_MAP, where the encoder may identify vertices that would get a better mapping if their geometry was smoothed by a signaled number of smoothing iterations prior to performing nearest neighbor search and mapping. Alternatively, the encoder may entropy encode (e.g., with arithmetic coding) the residuals of the texture coordinates for these vertices. The residuals are 2D vectors obtained at the encoder side by the vector difference between the ASSISTED_MAP generated 2D texture coordinates and the actual 2D texture coordinates of the vertices identified for refinement.

SIGNALING:

Signaling for the base mesh codec, i.e., signaling that texture coordinates are available or not in the coded base mesh stream can be achieved by a xxxx_texture_coordinates_available_flag indicates “true” when texture coordinates are encoded in the base mesh stream and “false” when texture coordinates are not present in the base mesh stream. xxxx_tc_skip_reference_frame_index indicates the mesh reference frame index used for the texture coordinates skipping. xxxx_tc_skip_method indicates a skipping method as specified in the following table

According to an embodiment, additional syntax structure is present for ASSISTED_MAP and REFINED_MAP: bmesh_tsmi_number_of_feature_points_minus1[submesh_id] indicates the number of feature points present in the current submesh with index submesh d. bmesh_tsmi_number_of_refined_points_minus1 [submeshjd] indicates the number of refined points present in the current submesh with index submeshjd. feature_point_index[submesh_id,i] indicates the i:th index of the feature points of submesh with index submeshjd. refined_point_index[submesh id,i] indicates the i:th index of the refined points of submesh with index submesh d. residual_texture_coordinate_u[submesh id,i] indicates the residual of the texture coordinate component U of the i:th refined point of submesh with index submeshjd. residual_texture_coordinate_v[submesh id,i] indicates the residual of the texture coordinate component V of the i:th refined point of submesh with index submeshjd.

According to another embodiment the syntax can be signaled in a base frame parameter set (BFAS). In such embodiment the BFAS indicates that all base mesh frames other than intra frame that reference this parameter set use a given mode for texture coordinate estimation. bfps_texture_coordinates_available_flag indicates “true” when texture coordinates are encoded in the base mesh stream, and “false” when texture coordinates are not present in the base mesh stream. bfps_tc_skip_reference_frame_index indicates that the mesh reference frame index is used for the texture coordinates skipping. bfps_tc_skip_method indicates the skipping method as specified in the following table:

According to an embodiment, the syntax can be signaled in base sequence parameter set (BSAS). In that embodiment the BSAS indicates that all base mesh frames other than intra frame that reference this parameter set use a given mode for texture coordinate estimation.

The following example shows this information at the submesh level in bmesh_sub_mesh_information syntax structure. bsmi_texture_coordinates_available_flag[ i ] indicates “true” if submesh identifier equal to i contains encoded texture coordinates. bsmi_tc_skip_reference_frame_index[i] indicates the mesh reference frame that is used for texture coordinate skipping for submesh index i. bsmi_tc_skip_method[i] indicates that the texture coordinate skipping method is used for submesh index i.

According to an embodiment, the signalling information is provided on submesh level in sdu_inter_sub_mesh_unit syntax structure

sismu_texture_coordinates_available_flag[ subMeshlD ] indicates “true” when submesh with identifier equal to subMeshlD contains encoded texture coordinates. sismu_tc_skip_reference_frame_index[ subMeshlD ] indicates the mesh reference frame used for texture coordinate skipping for submesh with identifier equal to subMeshlD. sismu_tc_skip_method[ subMeshlD ] indicates the texture coordinate skipping method used for submesh with identifier equal to subMeshlD.

According to an embodiment the derivation mode of texture coordinates can be provided through base mesh NAL unity type. For example, one NAL unit type indicates that SKIP mode for texture coordinates is used in one base mesh or submesh, and another NAL unit type indicates the MAP mode that is used for texture coordinates is used in another base mesh submesh.

According to an embodiment, the signaling of is applied individually for each given texture of a mesh. For example, one texture containing color information would use one mode while other texture containing reflectance/normal information would use another mode.

According to an embodiment, the type of mapping can also be signaled for example at atlas sequence parameter set (ASPS) or atlas frame parameter set (AFPS). The method for encoding according to an embodiment is shown in Figure 19. The method generally comprises receiving 1905 a dynamic three-dimensional mesh sequence, wherein the three-dimensional mesh represents a three- dimensional object with texture; determining 1910 which texture coordinates are not encoded into a bitstream; determining 1915 a derivation mode for the texture coordinates not to be encoded; encoding 1920 into a bitstream an indication of the determined derivation mode; encoding 1925 into a bitstream an information on a reference frame comprising texture coordinates for the current mesh frame; encoding 1930 a base mesh into a bitstream with the texture coordinates which are to be encoded; and transmitting 1935 the one or more bitstreams to a decoder. Each of the steps can be implemented by a respective module of a computer system.

An apparatus according to an embodiment comprises means for receiving a dynamic three-dimensional mesh sequence, wherein the three-dimensional mesh represents a three-dimensional object with texture; means for determining which texture coordinates are not encoded into a bitstream; means for determining a derivation mode for the texture coordinates not to be encoded; means for encoding into a bitstream an indication of the determined derivation mode; means for encoding into a bitstream an information on a reference frame comprising texture coordinates for the current mesh frame; means for encoding a base mesh into a bitstream with the texture coordinates which are to be encoded; and means for transmitting the one or more bitstreams to a decoder. The means comprises at least one processor, and a memory including a computer program code, wherein the processor may further comprise processor circuitry. The memory and the computer program code are configured to, with the at least one processor, cause the apparatus to perform the method of Figure 19 according to various embodiments.

The method for decoding according to an embodiment is shown in Figure 20. The method generally comprises receiving 2010 one or more bitstreams from an encoder; decoding 2015 a base mesh from a bitstream texture coordinates; determining 2020 texture coordinates missing from a bitstream; decoding 2030 an indication on a derivation mode from a bitstream; decoding 2040 information on a reference frame comprising texture coordinates missing from the bitstream; reconstructing 2050 missing texture coordinates from the reference frame according to the derivation mode; and reconstructing 2060 a dynamic three-dimensional mesh sequence according to the texture coordinates. Each of the steps can be implemented by a respective module of a computer system.

An apparatus according to an embodiment comprises means for receiving one or more bitstreams from an encoder; means for decoding a base mesh from a bitstream texture coordinates; means for determining texture coordinates missing from a bitstream; means for decoding an indication on a derivation mode from a bitstream; means for decoding information on a reference frame comprising texture coordinates missing from the bitstream; means for reconstructing missing texture coordinates from the reference frame according to the derivation mode; and means for reconstructing a dynamic three- dimensional mesh sequence according to the texture coordinates. The means comprises at least one processor, and a memory including a computer program code, wherein the processor may further comprise processor circuitry. The memory and the computer program code are configured to, with the at least one processor, cause the apparatus to perform the method of Figure 20 according to various embodiments.

An example of an apparatus is disclosed with reference to Figure 21. Figure 21 shows a block diagram of a video coding system according to an example embodiment as a schematic block diagram of an electronic device 50, which may incorporate a codec. In some embodiments the electronic device may comprise an encoder or a decoder. The electronic device 50 may for example be a mobile terminal or a user equipment of a wireless communication system or a camera device. The electronic device 50 may be also comprised at a local or a remote server or a graphics processing unit of a computer. The device may be also comprised as part of a head-mounted display device. The apparatus 50 may comprise a display 32 in the form of a liquid crystal display. In other embodiments of the invention the display may be any suitable display technology suitable to display an image or video. The apparatus 50 may further comprise a keypad 34. In other embodiments of the invention any suitable data or user interface mechanism may be employed. For example, the user interface may be implemented as a virtual keyboard or data entry system as part of a touch-sensitive display. The apparatus may comprise a microphone 36 or any suitable audio input which may be a digital or analogue signal input. The apparatus 50 may further comprise an audio output device which in embodiments of the invention may be any one of: an earpiece 38, speaker, or an analogue audio or digital audio output connection. The apparatus 50 may also comprise a battery (or in other embodiments of the invention the device may be powered by any suitable mobile energy device such as solar cell, fuel cell or clockwork generator). The apparatus may further comprise a camera 42 capable of recording or capturing images and/or video. The camera 42 may be a multi-lens camera system having at least two camera sensors. The camera is capable of recording or detecting individual frames which are then passed to the codec 54 or the controller for processing. The apparatus may receive the video and/or image data for processing from another device prior to transmission and/or storage.

The apparatus 50 may comprise a controller 56 or processor for controlling the apparatus 50. The apparatus or the controller 56 may comprise one or more processors or processor circuitry and be connected to memory 58 which may store data in the form of image, video and/or audio data, and/or may also store instructions for implementation on the controller 56 or to be executed by the processors or the processor circuitry. The controller 56 may further be connected to codec circuitry 54 suitable for carrying out coding and decoding of image, video and/or audio data or assisting in coding and decoding carried out by the controller.

The apparatus 50 may further comprise a card reader 48 and a smart card 46, for example a IIICC (Universal Integrated Circuit Card) and UICC reader for providing user information and being suitable for providing authentication information for authentication and authorization of the user at a network. The apparatus 50 may comprise radio interface circuitry 52 connected to the controller and suitable for generating wireless communication signals for example for communication with a cellular communications network, a wireless communications system, or a wireless local area network. The apparatus 50 may further comprise an antenna 44 connected to the radio interface circuitry 52 for transmitting radio frequency signals generated at the radio interface circuitry 52 to other apparatus(es) and for receiving radio frequency signals from other apparatus(es). The apparatus may comprise one or more wired interfaces configured to transmit and/or receive data over a wired connection, for example an electrical cable or an optical fiber connection.

The various embodiments may provide advantages. For example, the present embodiments allow a smaller bitrate thanks to the fact that texture coordinates are not encoded in the base mesh frame that are not reference (or key) frames. The present embodiments provide temporally stable texture maps that enable better attribute map compression. The present embodiments also provide faster encoding that using independent parameterization for each frame. Since a motion encoder is not used to map base meshes in the present embodiments, enables avoiding long sequence of motion vectors that are entropy encoded, and which limit the throughput of the decoder. Furthermore, motion codecs may require the connectivity to be the same for the reference mesh frames and the current mesh frames, which is too restrictive. The proposed mapping techniques enable the encoder to efficiently skip texture coordinate data, while handling different connectivities. The present embodiments further enables reduced decoding time for the base mesh frames at the decoder side.

The various embodiments can be implemented with the help of computer program code that resides in a memory and causes the relevant apparatuses to carry out the method. For example, a device may comprise circuitry and electronics for handling, receiving, and transmitting data, computer program code in a memory, and a processor that, when running the computer program code, causes the device to carry out the features of an embodiment. Yet further, a network device like a server may comprise circuitry and electronics for handling, receiving, and transmitting data, computer program code in a memory, and a processor that, when running the computer program code, causes the network device to carry out the features of various embodiments.

If desired, the different functions discussed herein may be performed in a different order and/or concurrently with other. Furthermore, if desired, one or more of the above-described functions and embodiments may be optional or may be combined. Although various aspects of the embodiments are set out in the independent claims, other aspects comprise other combinations of features from the described embodiments and/or the dependent claims with the features of the independent claims, and not solely the combinations explicitly set out in the claims.

It is also noted herein that while the above describes example embodiments, these descriptions should not be viewed in a limiting sense. Rather, there are several variations and modifications, which may be made without departing from the scope of the present disclosure as, defined in the appended claims.

Claims

Claims:

1 . An apparatus for encoding comprising means for receiving a dynamic three-dimensional mesh sequence, wherein the three-dimensional mesh represents a three-dimensional object with texture; means for determining which texture coordinates are not encoded into a bitstream; means for determining a derivation mode for the texture coordinates not to be encoded; means for encoding into a bitstream an indication of the determined derivation mode; means for encoding into a bitstream an information on a reference frame comprising texture coordinates for the current mesh frame; means for encoding a base mesh into a bitstream with the texture coordinates which are to be encoded; and means for transmitting the one or more bitstreams to a decoder.

2. The apparatus according to claim 1 , in order to determine which texture coordinates are not encoded, the apparatus further comprises means for testing the texture coordinates whether they are recovered with any of the derivation mode with a quality exceeding a threshold.

3. The apparatus according to claim 1 , further comprising means for generating the base mesh from the dynamic three-dimensional mesh sequence, wherein the current mesh is the base mesh frame.

4. The apparatus according to claim 2, further comprising means for iteratively subdividing the base mesh into submeshes, wherein the current mesh is a submesh frame.

5. The apparatus according to claim 1 or 2 or 3, wherein the reference frame is one of the following: a previous reconstructed base mesh frame, an intra frame or any frame stored in a reconstructed base mesh frame buffer or mesh buffer.

6. The apparatus according to any of the claims 1 to 4, wherein the derivation mode for texture coordinates is one of the following: when connectivity of the reference frame is the same as the current mesh frame, copying extracted texture coordinates from the reference frame to the current reconstructed base mesh frame; when connectivity of the reference frame is not the same as the current mesh frame, mapping extracted texture coordinates from the reference frame to the current reconstructed base mesh frame; when connectivity of the reference frame is not the same as the current mesh frame, mapping extracted texture coordinates from the reference frame to the current reconstructed base mesh frame with anchor points; when connectivity of the reference frame is not the same as the current mesh frame, mapping extracted texture coordinates from the reference frame to the current reconstructed base mesh frame with anchor points and refinable vertices.

7. The apparatus according to any of the claims 1 to 6, further comprising means for indicating a derivation mode of texture coordinates by a certain network abstraction layer unit type.

8. The apparatus according to any of the claims 1 to 7, further comprising means for signaling the derivation mode and the reference frame for each texture type of a mesh individually, the texture type being one of the following: color, reflectance, normal.

9. The apparatus according to any of the claims 6 to 8, further comprising means for signaling the type of mapping at an atlas sequence parameter set or an atlas frame parameter set.

10. An apparatus for decoding comprising means for receiving one or more bitstreams from an encoder; means for decoding a base mesh from a bitstream texture coordinates; means for determining texture coordinates missing from a bitstream; means for decoding an indication on a derivation mode from a bitstream; means for decoding information on a reference frame comprising texture coordinates missing from the bitstream; means for reconstructing missing texture coordinates from the reference frame according to the derivation mode; and means for reconstructing a dynamic three-dimensional mesh sequence according to the texture coordinates.

11 .A method for encoding comprising receiving a dynamic three-dimensional mesh sequence, wherein the three-dimensional mesh represents a three-dimensional object with texture; determining which texture coordinates are not encoded into a bitstream; determining a derivation mode for the texture coordinates not to be encoded; encoding into a bitstream an indication of the determined derivation mode; encoding into a bitstream an information on a reference frame comprising texture coordinates for the current mesh frame; encoding a base mesh into a bitstream with the texture coordinates which are to be encoded; and transmitting the one or more bitstreams to a decoder.

12. A method for decoding comprising receiving one or more bitstreams from an encoder; decoding a base mesh from a bitstream texture coordinates; determining texture coordinates missing from a bitstream; decoding an indication on a derivation mode from a bitstream; decoding information on a reference frame comprising texture coordinates missing from the bitstream; reconstructing missing texture coordinates from the reference frame according to the derivation mode; and reconstructing a dynamic three-dimensional mesh sequence according to the texture coordinates.

13. An apparatus for encoding, the apparatus comprising at least one processor, memory including computer program code, the memory and the computer program code configured to, with the at least one processor, cause the apparatus to perform at least the following: receive a dynamic three-dimensional mesh sequence, wherein the three-dimensional mesh represents a three-dimensional object with texture; determine which texture coordinates are not encoded into a bitstream; determine a derivation mode for the texture coordinates not to be encoded; encode into a bitstream an indication of the determined derivation mode; encode into a bitstream an information on a reference frame comprising texture coordinates for the current mesh frame; encode a base mesh into a bitstream with the texture coordinates which are to be encoded; and transmit the one or more bitstreams to a decoder.

14. An apparatus for decoding, the apparatus comprising at least one processor, memory including computer program code, the memory and the computer program code configured to, with the at least one processor, cause the apparatus to perform at least the following: receive one or more bitstreams from an encoder; decode a base mesh from a bitstream texture coordinates; determine texture coordinates missing from a bitstream; decode an indication on a derivation mode from a bitstream; decode information on a reference frame comprising texture coordinates missing from the bitstream; reconstruct missing texture coordinates from the reference frame according to the derivation mode; and reconstruct a dynamic three-dimensional mesh sequence according to the texture coordinates.