Classifications

• • H—ELECTRICITY
  • H04—ELECTRIC COMMUNICATION TECHNIQUE
  • H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
  • H04N13/00—Stereoscopic video systems; Multi-view video systems; Details thereof
  • H04N13/10—Processing, recording or transmission of stereoscopic or multi-view image signals
  • H04N13/106—Processing image signals
  • H04N13/161—Encoding, multiplexing or demultiplexing different image signal components
• • G—PHYSICS
  • G06—COMPUTING OR CALCULATING; COUNTING
  • G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
  • G06T9/00—Image coding
  • G06T9/001—Model-based coding, e.g. wire frame
• • H—ELECTRICITY
  • H04—ELECTRIC COMMUNICATION TECHNIQUE
  • H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
  • H04N13/00—Stereoscopic video systems; Multi-view video systems; Details thereof
  • H04N13/10—Processing, recording or transmission of stereoscopic or multi-view image signals
  • H04N13/106—Processing image signals
  • H04N13/172—Processing image signals image signals comprising non-image signal components, e.g. headers or format information
  • H04N13/178—Metadata, e.g. disparity information
• • H—ELECTRICITY
  • H04—ELECTRIC COMMUNICATION TECHNIQUE
  • H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
  • H04N19/00—Methods or arrangements for coding, decoding, compressing or decompressing digital video signals
  • H04N19/50—Methods or arrangements for coding, decoding, compressing or decompressing digital video signals using predictive coding
  • H04N19/597—Methods or arrangements for coding, decoding, compressing or decompressing digital video signals using predictive coding specially adapted for multi-view video sequence encoding
• • H—ELECTRICITY
  • H04—ELECTRIC COMMUNICATION TECHNIQUE
  • H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
  • H04N19/00—Methods or arrangements for coding, decoding, compressing or decompressing digital video signals
  • H04N19/70—Methods or arrangements for coding, decoding, compressing or decompressing digital video signals characterised by syntax aspects related to video coding, e.g. related to compression standards
• • H—ELECTRICITY
  • H04—ELECTRIC COMMUNICATION TECHNIQUE
  • H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
  • H04N19/00—Methods or arrangements for coding, decoding, compressing or decompressing digital video signals
  • H04N19/50—Methods or arrangements for coding, decoding, compressing or decompressing digital video signals using predictive coding
  • H04N19/503—Methods or arrangements for coding, decoding, compressing or decompressing digital video signals using predictive coding involving temporal prediction
  • H04N19/51—Motion estimation or motion compensation
  • H04N19/537—Motion estimation other than block-based
  • H04N19/54—Motion estimation other than block-based using feature points or meshes

Definitions

• the present solution generally relates to encoding and decoding of volumetric video.
• Volumetric video data represents a three-dimensional (3D) scene or object and can be used as input for AR (Augmented Reality), VR (Virtual Reality), and MR (Mixed Reality) applications.
• Such data describes geometry (Shape, size, position in 3D space) and respective attributes (e.g., color, opacity, reflectance, ...), and any possible temporal transformations of the geometry and attributes at given time instances (like frames in two-dimensional (2D) video).
• Volumetric video can be generated from 3D models, also referred to as volumetric visual objects, i.e., CGI (Computer Generated Imagery), or captured from real-world scenes using a variety of capture solutions, e.g., multi-camera, laser scan, combination of video and dedicated depth sensors, and more. Also, a combination of CGI and real-world data is possible. Examples of representation formats for volumetric data comprise triangle meshes, point clouds, or voxels.
• Temporal information about the scene can be included in the form of individual capture instances, i.e., “frames” in 2D video, or other means, e.g., position of an object as a function of time.
• volumetric video describes a 3D scene (or object), such data can be viewed from any viewpoint. Therefore, volumetric video is an important format for any AR, VR or MR applications, especially for providing 6DOF viewing capabilities.
• 3D data acquisition devices have enabled reconstruction of highly detailed volumetric video representations of natural scenes.
• Infrared, lasers, time-of-flight, and structured light are examples of devices that can be used to construct 3D video data.
• Representation of the 3D data depends on how the 3D data is used.
• Dense Voxel arrays have been used to represent volumetric medical data.
• polygonal meshes are extensively used.
• Point clouds on the other hand are well suited for applications such as capturing real world 3D scenes where the topology is not necessarily a 2D manifold.
• Another way to represent 3D data is coding, this 3D data as set of texture and depth map as is the case in the multi-view plus depth. Closely related to the techniques used in multiview plus depth is the use of elevation maps, and multi-level surface maps.
• an apparatus for encoding comprising means for receiving a dynamic three-dimensional mesh sequence, wherein the three-dimensional mesh represents a three-dimensional object with texture; means for determining which texture coordinates are not encoded into a bitstream; means for determining a derivation mode for the texture coordinates not to be encoded; means for encoding into a bitstream an indication of the determined derivation mode; means for encoding into a bitstream an information on a reference frame comprising texture coordinates for the current mesh frame; means for encoding a base mesh into a bitstream with the texture coordinates which are to be encoded; and means for transmitting the one or more bitstreams to a decoder.
• an apparatus for decoding comprising means for receiving one or more bitstreams from an encoder; means for decoding a base mesh from a bitstream texture coordinates; means for determining texture coordinates missing from a bitstream; means for decoding an indication on a derivation mode from a bitstream; means for decoding information on a reference frame comprising texture coordinates missing from the bitstream; means for reconstructing missing texture coordinates from the reference frame according to the derivation mode; and means for reconstructing a dynamic three-dimensional mesh sequence according to the texture coordinates.
• a method for encoding comprising: receiving a dynamic three-dimensional mesh sequence, wherein the three-dimensional mesh represents a three-dimensional object with texture; determining which texture coordinates are not encoded into a bitstream; determining a derivation mode for the texture coordinates not to be encoded; encoding into a bitstream an indication of the determined derivation mode; encoding into a bitstream an information on a reference frame comprising texture coordinates for the current mesh frame; encoding a base mesh into a bitstream with the texture coordinates which are to be encoded; and transmitting the one or more bitstreams to a decoder.
• a method for decoding comprising: receiving one or more bitstreams from an encoder; decoding a base mesh from a bitstream texture coordinates; determining texture coordinates missing from a bitstream; decoding an indication on a derivation mode from a bitstream; decoding information on a reference frame comprising texture coordinates missing from the bitstream; reconstructing missing texture coordinates from the reference frame according to the derivation mode; and reconstructing a dynamic three-dimensional mesh sequence according to the texture coordinates.
• an apparatus for encoding comprising at least one processor, memory including computer program code, the memory and the computer program code configured to, with the at least one processor, cause the apparatus to perform at least the following: receive a dynamic three-dimensional mesh sequence, wherein the three-dimensional mesh represents a three-dimensional object with texture; determine which texture coordinates are not encoded into a bitstream; determine a derivation mode for the texture coordinates not to be encoded; encode into a bitstream an indication of the determined derivation mode; encode into a bitstream an information on a reference frame comprising texture coordinates for the current mesh frame; encode a base mesh into a bitstream with the texture coordinates which are to be encoded; and transmit the one or more bitstreams to a decoder.
• an apparatus for decoding comprising at least one processor, memory including computer program code, the memory and the computer program code configured to, with the at least one processor, cause the apparatus to perform at least the following: receive one or more bitstreams from an encoder; decode a base mesh from a bitstream texture coordinates; determine texture coordinates missing from a bitstream; decode an indication on a derivation mode from a bitstream; decode information on a reference frame comprising texture coordinates missing from the bitstream; reconstruct missing texture coordinates from the reference frame according to the derivation mode; and reconstruct a dynamic three-dimensional mesh sequence according to the texture coordinates.
• a seventh aspect there is provided computer program product for encoding comprising computer program code configured to, when executed on at least one processor, cause an apparatus or a system to: receive a dynamic three-dimensional mesh sequence, wherein the three-dimensional mesh represents a three-dimensional object with texture; determine which texture coordinates are not encoded into a bitstream; determine a derivation mode for the texture coordinates not to be encoded; encode into a bitstream an indication of the determined derivation mode; encode into a bitstream an information on a reference frame comprising texture coordinates for the current mesh frame; encode a base mesh into a bitstream with the texture coordinates which are to be encoded; and transmit the one or more bitstreams to a decoder.
• computer program product for decoding comprising computer program code configured to, when executed on at least one processor, cause an apparatus or a system to: receive one or more bitstreams from an encoder; decode a base mesh from a bitstream texture coordinates; determine texture coordinates missing from a bitstream; decode an indication on a derivation mode from a bitstream; decode information on a reference frame comprising texture coordinates missing from the bitstream; reconstruct missing texture coordinates from the reference frame according to the derivation mode; and reconstruct a dynamic three- dimensional mesh sequence according to the texture coordinates.
• the texture coordinates are tested whether they are recovered with any of the derivation mode with a quality exceeding a threshold.
• the base mesh is generated from the dynamic three-dimensional mesh sequence, wherein the current mesh is the base mesh frame.
• the base mesh is subdivided into submeshes, wherein the current mesh is a submesh frame.
• a derivation mode of texture coordinates is indicated by a certain network abstraction layer unit type.
• the derivation mode and the reference frame is signaled for each texture type of a mesh individually, the texture type being one of the following: color, reflectance, normal.
• the type of mapping is signaled at an atlas sequence parameter set or an atlas frame parameter set.
• the reference frame is one of the following: a previous reconstructed base mesh frame, an intra frame or any frame stored in a reconstructed base mesh frame buffer or mesh buffer.
• the derivation mode for texture coordinates is one of the following: when connectivity of the reference frame is the same as the current mesh frame, copying extracted texture coordinates from the reference frame to the current reconstructed base mesh frame; when connectivity of the reference frame is not the same as the current mesh frame, mapping extracted texture coordinates from the reference frame to the current reconstructed base mesh frame; when connectivity of the reference frame is not the same as the current mesh frame, mapping extracted texture coordinates from the reference frame to the current reconstructed base mesh frame with anchor points; when connectivity of the reference frame is not the same as the current mesh frame, mapping extracted texture coordinates from the reference frame to the current reconstructed base mesh frame with anchor points and refinable vertices.
• the computer program product is embodied on a non-transitory computer readable medium.
• Fig. 1 a shows an example of a volumetric media conversion at an encoder
• Fig. 1 b shows an example of a volumetric media reconstruction at a decoder
• Fig. 2 shows an example of block to patch mapping
• Fig. 3a shows an example of an atlas coordinate system
• Fig. 3b shows an example of a local 3D patch coordinate system
• Fig. 3c shows an example of a final target 3D coordinate system
• Fig. 4 shows a simplified example of a subdivision step of a triangle into four triangles
• Fig. 5 shows an example of a multiresolution analysis of a mesh
• Fig. 6 shows an example of an encoder comprising a pre-processing module for generating a mesh
• Fig. 7 shows an example of pre-processing steps at an encoder
• Fig. 8 shows an example of an intra frame encoder for mesh compression
• Fig. 9 shows an example of an inter frame encoder for mesh compression
• Fig. 10 shows an example of a decoder comprising a post-processing module for reconstruing a dynamic mesh sequence
• Fig. 11 shows an example of a decoding process in intra mode
• Fig. 12 shows an example of a decoding process in inter mode
• Fig. 13 shows an example of a base-mesh encoder
• Fig. 14 shows another example of a base-mesh encoder
• Fig. 15 shows an example of a base-mesh decoder
• Fig. 16 shows an example of segmentation of a mesh into sub-meshes
• Fig. 17 shows an example with two submeshes
• Fig. 18 shows an example of texture coordinates mapping
• Fig. 19 is a flowchart illustrating a method for encoding according to an embodiment
• Fig. 20 is a flowchart illustrating a method for decoding according to another embodiment.
• Fig. 21 shown an example of an apparatus.
• the present embodiments relate to encoding, signalling, and rendering a volumetric video based on mesh coding.
• the present embodiments are applicable with a Video-based Dynamic Mesh Coding (V-DMC) ISO/IEC 23090-29, which is an application of the Visual Volumetric Video Coding (V3C) standard family ISO/IEC 23090-5.
• V-DMC Video-based Dynamic Mesh Coding
• V3C Visual Volumetric Video Coding
• Volumetric video data represents a three-dimensional scene or object and can be used as input for AR, VR and MR applications.
• Such data describes geometry (shape, size, position in 3D space) and respective attributes (e.g., color, opacity, reflectance, ...), plus any possible temporal transformations of the geometry and attributes at given time instances (like frames in 2D video).
• Volumetric video is either generated from 3D models, i.e., CGI, or captured from real-world scenes using a variety of capture solutions, e.g., multi-camera, laser scan, combination of video and dedicated depth sensors, and more. Also, a combination of CGI and real-world data is possible. Representation formats for such volumetric data are triangle meshes, point clouds, or voxels.
• Temporal information about the scene can be included in the form of individual capture instances, i.e., “frames” in 2D video, or other means, e.g., position of an object as a function of time.
• volumetric video describes a 3D scene (or object), such data can be viewed from any viewpoint. Therefore, volumetric video is an important format for any AR, VR or MR applications, especially for providing 6DOF viewing capabilities.
• 3D data acquisition devices have enabled reconstruction of highly detailed volumetric video representations or natural scenes.
• Infrared, lasers, time-of-flight, and structured light are all examples of devices that can be used to construct 3D video data.
• Representation of the 3D data depends on how the 3D data is used.
• Dense Voxel arrays have been used to represent volumetric medical data.
• polygonal meshes are extensively used.
• Point clouds on the other hand are well suited for applications such as capturing real world 3D scenes where the topology is not necessarily a 2D manifold.
• Another way to represent 3D data is coding this 3D data as set of texture and depth map as is the case in the multi-view plus depth. Closely related to the techniques used in multi-view plus depth is the use of elevation maps, and multi-level surface maps.
• Visual volumetric video comprising a sequence of visual volumetric frames, if uncompressed, may be represented by a large amount of data, which can be costly in terms of storage and transmission. This has led to the need for a high coding efficiency standard for the compression of visual volumetric data.
• V3C enables the encoding and decoding processes of a variety of volumetric media by using video and image coding technologies. This is achieved through first a conversion of such media from their corresponding 3D representation to multiple 2D representations, also referred to as V3C video components, before coding such information.
• Such representations may include occupancy, geometry, and attribute components.
• the occupancy component can inform a V3C decoding and/or rendering system of which samples in the 2D components are associated with data in the final 3D representation.
• the geometry component contains information about the precise location of 3D data in space, while attribute components can provide additional properties, e.g., texture or material information, of such 3D data.
• Figures 1 a and 1 b An example is shown in Figures 1 a and 1 b, where Figure 1a presents volumetric media conversion at an encoder, and where Figure 1 b presents volumetric media reconstruction at a decoder side.
• the 3D media is converted to a series of 2D representations: occupancy 101 , geometry 102, and attributes 103. Additional information may also be included in the bitstream to enable inverse reconstruction.
• An atlas 104 consists of multiple elements, named as patches. Each patch identifies a region in all available 2D components and contains information necessary to perform the appropriate inverse projection of this region back to the 3D space. The shape of such regions is determined through a 2D bounding volume associated with each patch as well as their coding order. The shape of these regions is also further refined after the consideration of the occupancy information.
• Atlases may be partitioned into patch packing blocks of equal size.
• the 2D bounding volumes of patches and their coding order determine the mapping between the blocks of the atlas image and the patch indices.
• Figure 2 shows an example of block to patch mapping with 4 projected patches onto an atlas when asps_patch_precedence_order_flag is equal to 0. Projected points are represented with dark grey. The area that does not contain any projected points is represented with light grey.
• Patch packing blocks are represented with dashed lines. The number inside each patch packing block represents the patch index of the patch to which it is mapped.
• Axes orientations are specified for internal operations. For instance, the origin of the atlas coordinates is located on the top-left corner of the atlas frame. For the reconstruction step, an intermediate axes definition for a local 3D patch coordinate system is used. The 3D local patch coordinate system is then converted to the final target 3D coordinate system using appropriate transformation steps.
• Figure 3a shows an example of a single patch 320 packed onto an atlas image 310.
• This patch 520 is then converted to a local 3D patch coordinate system (U, V, D) defined by the projection plane with origin O’, tangent (U), bi-tangent (V), and normal (D) axes.
• the projection plane is equal to the sides of an axis-aligned 3D bounding volume 330, as shown in Figure 3b.
• the location of the bounding volume 330 in the 3D model coordinate system can be obtained by adding offsets TilePatch3dOffsetU, TilePatch3DOffsetV, and TilePatch3DOffsetD, as illustrated in Figure 3c.
• Coded V3C video components are referred to in this disclosure as video bitstreams, while a coded atlas is referred to as the atlas bitstream.
• Video bitstreams and atlas bitstreams may be further split into smaller units, referred to here as video and atlas sub-bitstreams, respectively, and may be interleaved together, after the addition of appropriate delimiters, to construct a V3C bitstream.
• V3C patch information is contained in atlas bitstream, atlas_sub_bitstream(), which contains a sequence of NAL units.
• NAL unit is specified to format data and provide header information in a manner appropriate for conveyance on a variety of communication channels or storage media. All data are contained in NAL units, each of which contains an integer number of bytes.
• a NAL unit specifies a generic format for use in both packet-oriented and bitstream systems. The format of NAL units for both packet-oriented transport and sample streams is identical except that in the sample stream format specified in Annex D of ISO/IEC 23090-5 each NAL unit can be preceded by an additional element that specifies the size of the NAL unit.
• NAL units in atlas bitstream can be divided to atlas coding layer (ACL) and non-atlas coding layer (non-ACL) units.
• ACL atlas coding layer
• non-ACL non-atlas coding layer
• nal_unit_type specifies the type of the RBSP (Raw Byte Sequence Payload) data structure contained in the NAL unit as specified in Table 4 of ISO/IEC 23090-5.
• nal_layer_id specifies the identifier of the layer to which an ACL NAL unit belongs or the identifier of a layer to which a non-ACL NAL unit applies.
• the value of nal_layer_id shall be in the range of 0 to 62, inclusive.
• the value of 63 may be specified in the future by ISO/IEC. Decoders conforming to a profile specified in Annex A of ISO/IEC 23090-5 shall ignore (i.e., remove from the bitstream and discard) all NAL units with values of nal_layer_id not equal to 0.
• V3C second edition of V3C introduced an extension in VPS related to MIV and packed video component.
• a polygon mesh is a collection of vertices, edges and faces that defines the shape of a polyhedral object in 3D computer graphics and solid modelling.
• the faces usually consist of triangles (triangle mesh), quadrilaterals (quads), or other simple convex polygons (n-gons), since this simplifies rendering, but may also be more generally composed of concave polygons, or even polygons with holes.
• Objects created with polygon meshes are represented by different types of elements. These include vertices, edges, faces, polygons, and surfaces. In many applications, only vertices, edges and either faces or polygons are stored.
• Polygon meshes are defined by the following elements: • Vertex: A position in 3D space defined as (x, y, z) along with other information such as color (r, g, b), normal vector and texture coordinates.
• Edge A connection between two vertices.
• a polygon A closed set of edges, in which a triangle face has three edges, and a quad face has four edges.
• a polygon is a coplanar set of faces. In systems that support multi-sided faces, polygons and faces are equivalent.
• Mathematically a polygonal mesh may be considered an unstructured grid, or undirected graph, with additional properties of geometry, shape, and topology.
• Groups Some mesh formats contain groups, which define separate elements of the mesh, and are useful for determining separate subobjects for skeletal animation or separate actors for non-skeletal animation.
• UV coordinates Most mesh formats also support some form of UV coordinates which are a separate 2D representation of the mesh "unfolded" to show what portion of a 2-dimensional texture map applies to different polygons of the mesh. It is also possible for meshes to contain other vertex attribute information such as color, tangent vectors, weight maps to control animation, etc. (sometimes also called channels).
• Mesh data may be compressed directly without projecting it into 2D-planes, like in V-PCC based mesh coding.
• the anchor for V-PCC mesh compression call for proposals utilizes off-the shelf mesh compression technology, Draco (https://google.github.io/draco/), for compressing mesh data excluding textures.
• Draco is used to compress vertex positions in 3D, connectivity data (faces) as well as UV coordinates. Additional per-vertex attributes may be also compressed using Draco.
• the actual UV texture may be compressed using traditional video compression technologies, such as H.265 or H.264.
• Draco uses an edgebreaker algorithm at its core to compress 3D mesh information.
• V-DMC standardization works have started after the completion of the call for proposal (CfP) issued by MPEG 3DG (ISO/IEC SC29 WG 2) on integration of MESH compression into the V3C family of standards (ISO/IEC 23090-5).
• CfP call for proposal
• MPEG 3DG ISO/IEC SC29 WG 2
• MESH compression into the V3C family of standards
• the deformed mesh obtained by m n i + d i.e., by adding the displacement vectors to the subdivided mesh vertices generates the best approximation of the original mesh at that resolution, given the base mesh and prior subdivision levels.
• the displacement vectors may undergo a lazy wavelet transform prior to compression.
• the attribute map of the original mesh is transferred to the deformed mesh at the highest resolution (i.e., subdivision level) such that texture coordinates are obtained for the deformed mesh and a new attribute map is generated.
• An example of a multi-resolution analysis of a mesh is illustrated on Fig. 5.
• a base mesh 510 undergoes a first step of subdivision 515, and error vectors 520 are added to each vertex. After a series of iterative subdivision 525 and displacements, the higher resolution mesh 530 is generated.
• the connectivity of the highest resolution deformed mesh 535 is generally different from the original mesh, however, the geometry of the deformed mesh is a good approximation of the original mesh geometry.
• Figure 6 illustrates an example of an encoder comprising a preprocessing module 610 and an actual encoder module 620.
• Figure 7 illustrates the steps of the pre-processing module comprising: decimation (reducing the original mesh resolution to produce a base mesh) 710, UV-atlas isocharting (creating a parameterization of the base mesh) 720 and the subdivision surface fitting 730.
• Figure 8 illustrates an INTRA frame encoder
• Figure 9 illustrates an INTER frame encoder.
• the base mesh connectivity of the first frame of a group of frames is imposed to the subsequent frame’s base meshes to improve compression performance.
• inputs to this module are the base mesh (that is an approximation of the input mesh but that contains less faces and vertices) 802, the patch information 801 related to the input base mesh, the displacements 803, the static/dynamic input mesh frame 804 and the attribute map 805.
• Outputs of this module is a compressed bitstream 895 that contains a V3C extended signalling sub-bitstream including patch data information, compressed base mesh substream, a compressed displacement video component substream and a compressed Attribute video component sub-bitstream.
• the module takes the input base mesh and first quantize its data in the Quantization module, which can be dynamically tuned by a Control Module.
• the quantized base mesh is then encoded with the static mesh encoder module, which outputs a compressed base mesh sub-bitstream that is multiplexed in the output bitstream.
• the encoded base mesh is decoded in the Static Mesh Decoder module that generates a reconstructed quantized base mesh.
• the Update Displacements module takes as input the reconstructed quantizes base mesh, the pristine base mesh, and the input displacements to generate new updated displacements that are remapped to the reconstructed base mesh data in order to avoid precision errors due to the static mesh encoding and decoding process.
• the updated displacements are filtered with a wavelet transform in the Wavelet Transform module (that also takes as input the reconstructed base mesh) and then quantized in the Quantization module.
• the quantized wavelet coefficients produced from the updated displacements are then packed into a video component in the Image Packing module.
• This video component is then encoded with a 2D video encoder such as HEVC, WC, etc., in the Video Encoder module, and the output compressed displacement video component sub-bitstream is multiplexed along with the V3C signalling information sub-bitstream into the output compressed bitstream.
• the compressed displacement video component is first decoded and reconstructed and then unpacked into encoded and quantized wavelet coefficients in the Image Unpacking module. These wavelet coefficients are then unquantized in the inverse quantization module and reconstructed with the inverse wavelet transform module that generates reconstructed displacements.
• the reconstructed base mesh is unquantized in the inverse quantization module and the unquantized base mesh is combined with the reconstructed displacements in the Reconstruct Deformed Mesh module to obtain the reconstructed deformed mesh.
• This reconstructed deformed mesh is then fed into the Attribute Transfer module together with the Attribute map produced by the pre-processing and the input static/dynamic mesh frame.
• the output of the Attribute Transfer module is an updated attribute map that now corresponds to the reconstructed deformed mesh frame.
• the updated attribute map is then padded, after which it undergoes a color conversion.
• the attribute map is encoded as a video component with a 2D video codec such as HEVC or WC, in the Padding, Color Conversion and Video encoder modules, respectively.
• the output compressed attribute map bitstream is multiplexed into the encoder output bitstream.
• the inter encoding process shown in Figure 9, is similar to the intra encoding process of Figure 8 with the following changes.
• the reconstructed reference base mesh 910 is an input of the inter coding process.
• a new module called Motion Encoder 950 takes as input the quantized input base mesh and the reconstructed quantized reference base mesh 910 to produce compressed motion information encoded as a compressed motion bitstream, which is multiplexed into the encoder output compressed bitstream. All other modules and processes are similar to the intra encoding case.
• a sub-bitstream that contains all metadata required to decode and reconstruct the mesh sequence based on the aforementioned subbitstreams.
• the signalling of the metadata is based on the V3C syntax and includes necessary extensions that are specific to meshes.
• Figure 10 illustrates a decoding process according to an embodiment.
• First the compressed bitstream 1010 is demultiplexed into sub-bitstreams that are reconstructed, i.e., metadata, reconstructed base mesh, reconstructed displacements, and the reconstructed attribute map data.
• the reconstruction of the mesh sequence is performed based on that data in the post-processing module 1060.
• FIG. 11 and Figure 12 illustrate the decoding process in INTRA and INTER mode, respectively.
• the intra frame decoding process as shown in example of Figure 11 comprises the following modules and processes.
• First the input compressed bitstream is de-multiplexed 1110 into V3C extended atlas data information (or patch information), a compressed static mesh bitstream, a compressed displacement video component and a compressed attribute map bitstream, respectively.
• the static mesh decoding module 1120 converts the compressed static mesh bitstream into a reconstructed quantized static mesh, which represents a base mesh. This reconstructed quantized base mesh undergoes inverse quantization in the inverse quantization module 1125 to produce a decoded reconstructed base mesh.
• the compressed displacement video component bitstream is decoded in the video decoding module 1130 to generate a reconstructed displacement video component.
• This displacement video component is unpacked into reconstructed quantized wavelet coefficients in the image unpacking module 1135.
• Reconstructed quantized wavelet coefficients are inverse quantized in the inverse quantization module 1140 and then undergo an inverse wavelet transform in the inverse wavelet transform module 1145, that produces decoded displacement vectors.
• the reconstruct deformed mesh module 1150 takes into account the patch information and takes as input the decoded reconstructed base mesh and decoded displacement vectors to produce the output decoded mesh frame.
• the compressed attribute map video component is decoded in the video decoding module 1130, and possibly undergoes color conversion 1160 to produce a decoded attribute map frame that corresponds to the decoded mesh frame.
• the inter decoding process is similar to the intra decoding process module with the following changes.
• the decoder also demultiplexes a compressed information bitstream.
• a decoded reference base mesh 1200 is taken as input of a motion decoder module 1210 together with the compressed motion information sub-bitstream.
• This decoded reference base mesh is selected from a buffer of previously decoded base mesh frames (by the intra decoder process for the first frame of a group of frames).
• the reconstruction of base mesh module 1250 takes the decoded reference base mesh and the decoded motion information as input to produce a decoded reconstructed quantized base mesh. All other processes are similar to the intra decoding process as shown in Figure 11 .
• V3C unit header syntax V3C unit header syntax
• base meshes The output of the base mesh substream decoder is called “base meshes”
• Each base mesh can have one or more submeshes.
• a “submesh” is a set of vertices, their connectivity and the associated attributes which can be decoded completely independently in a mesh frame.
• “resampled base meshes” refers to an output of the mesh subdivision process.
• the inputs to the process are the base meshes (or sets of submeshes) and the information from the atlas data substream on how to subdivide/resample the meshes(submeshes).
• a displacement video is the output of the displacement decoder.
• the inputs to the process are the decoded geometry video and the information from the atlas data substream on how to interpret/process this video.
• the displacement video contains displacement values to be added to the corresponding vertices.
• a facegroupld is one of the attribute types assigned to each triangle face of the resampled base meshes. Facegroupld can be compared with the identifications of the subparts in a patch to determine the corresponding facegroups to the patch. If facegrould is not conveyed through the base mesh substream decoder, it can be derived by the information in the atlas data substream.
• V3C unit Compressed base meshes are signalled in a new substream, named as the Base Mesh data substream (unit type V3C_MD).
• unit type V3C_MD Base Mesh data substream
• the unit type, and its associated v3c parameter set id and atlas id are signalled in the v3c_unit_header().
• vps_ext_mesh_data_attribute_count indicates the number of total attributes in the base mesh including both the attributes signalled through the base mesh data substream and the attributes signalled in the video sub streams (using ai_attribute_count).
• vps_ext_mesh_data_facegroup_id_attribute_present_flag 1 , it shall be greater or equal to ai_attribute_count+1 . This can be constrained by profile/levels.
• vps_ext_mesh_data_facegroup_id_attribute_present_flag 1
• one of the vps_ext_mesh_attribute_type must be a facegroup d.
• vps_ext_mesh_data_substream_codec_id indicates the identifier of the codec used to compress the base mesh data. This codec may be identified through the profiles a component codec mapping SEI message, or through means outside this document.
• vps_ext_attribute_frame_width[i] and vps_ext_attribute_frame_height[i] indicate the corresponding with and height of the video data corresponding to the i-th attribute among the attributes signalled in the video substreams.
• • asps_vmc_ext_prevent_geometry_video_conversion_flag prevents the outputs of the geometry video substream decoder from being converted. When the flag is true, the outputs are used as they are without any conversion process from Annex B in [2], When the flag is true, the size of geometry video shall be same as nominal video sizes indicated in the bitstream.
• • asps_vmc_ext_prevent_attribute_video_conversion_flag prevents the outputs of attribute video substream decoder from being converted.
• the outputs are used as they are without any conversion process from Annex B in [2]
• the size of attribute video shall be same as nominal video sizes indicated in the bitstream.
• • asps_vmc_ext_transform_index indicates the transform applied to the displacement.
• the transform index can indicate any transform is not applied.
• the necessary parameters are signalled as vmc_lifting_transform_parameters.
• • asps_vmc_ext_patch_mapping_method indicates how to map a subpart of a submesh to a patch. o When asps_vmc_ext_patch_mapping_method is equal to 0, all the triangles in the corresponding submesh are associated with the current patch. In this case, there is only one patch associated with the submesh.
• asps_vmc_ext_patch_mapping_method When asps_vmc_ext_patch_mapping_method is equal to 1 , the subpart ds are explicitly signalled in the mesh patch data unit to indicate the associated subparts. o In other cases, the triangle faces in the corresponding submesh are divided into subparts by the method indicated by asps_vmc_ext_patch_mapping_method.
• • asps_vmc_ext_tjunction_removing_method indicates the method to remove t-junctions created by different subdivision methods or by different subdivision iterations of two triangles sharing an edge.
• • asps_vmc_ext_num_attribute indicates the total number of attributes that the corresponding mesh carries. Its value shall be less or equal to vps_ext_mesh_data_attribute_count.
• asps_vmc_ext_attribute_type is the type of the i-th attribute and it shall be one of ai_attribute_type_ids or vps_ext_mesh_attribute_types.
• • afps_vmc_ext_single_submesh_in_frame_flag indicates there is only one submesh for the mesh frame •
• afps_vmc_ext_overriden_flag in afps_vmc_extension() is true, the subdivision method, displacement coordinate system, transform index, transform parameters, and attribute transform parameters can be signalled again and the information ioverrides the one signalled in asps_vmc_extension().
• • afps_vmc_ext_single_attribute_tile_in_frame_flag indicates there is only one tile for each attribute signalled in the video streams.
• • afps_ext_vmc_attribute_tile_information() contains the tile information for the attributes signalled through the video substreams.
• a tile can be associated with one or more submeshes whose identification is ath_submesh_id.
• Mesh patch data units are signalled in the Atlas data substream.
• Mesh Intra patch data unit Mesh Inter patch data unit
• Mesh Merge patch data unit Mesh Skip patch data unit
• mdu_vertex_count_minus1 and mdu_triangle_count_minus1 indicate the number of vertices and triangles associated with the current patch.
• V-DMC V-DMC specification design
• a base mesh signal that can be encoded using any currently or future specified static mesh codec.
• static mesh codec For example, such information could be coded using Draco 3D Graphics Compression. This representation could provide the basis for applying other decoded information to reconstruct the output mesh frame within the context of V-DMC.
• HLS High Level Syntax
• submeshes One of the desirable features of this design is the ability to segment a mesh into multiple smaller partitions, referred to in this document as submeshes.
• An example of this is illustrated in Figure 16.
• the submeshes shown in (b) can be decoded completely independently, which can help with partial decoding and spatial random access. Although it may not be a requirement for all applications, some applications may require that the segmentation in submeshes remains consistent and fixed in time.
• the submeshes do not need to use the same coding type, i.e., for one frame one submesh may use intra coding while for another inter coding could be used at the same decoding instance, but it is commonly a requirement that the same coding order is used and the same references are available for all submeshes corresponding at a particular time instance. Such restrictions can help guarantee proper random access capabilities for the entire stream.
• An example where two submeshes are used is shown in the Figure 17.
• the new bitstream is also based on NAL units, and it is similar to those of the atlas substream in V3C.
• the syntax is provided below.
• NumBytesInNalllnit specifies the size of the NAL unit in bytes. This value is required for decoding of the NAL unit. Some form of demarcation of NAL unit boundaries is necessary to enable inference of NumBytesInNalUnit. One such demarcation method is specified in Annex TBD for the sample stream format. Other methods of demarcation can be specified outside this document.
• the mesh coding layer is specified to efficiently represent the content of the mesh data.
• the NAL is specified to format that data and provide header information in a manner appropriate for conveyance on a variety of communication channels or storage media. All data are contained in NAL units, each of which contains an integer number of bytes.
• a NAL unit specifies a generic format for use in both packet-oriented and bitstream systems.
• the format of NAL units for both packet-oriented transport and sample streams is identical except that in the sample stream format specified in Annex TBD each NAL unit can be preceded by an additional element that specifies the size of the NAL unit.
• rbsp_byte[ i ] is the i-th byte of an RBSP.
• An RBSP is specified as an ordered sequence of bytes as follows:
• the RBSP contains a string of data bits (SODB) as follows: • If the SODB is empty (i.e., zero bits in length), the RBSP is also empty.
• SODB data bits
• the RBSP contains the SODB as follows:
• the first byte of the RBSP contains the first (most significant, leftmost) eight bits of the SODB; the next byte of the RBSP contains the next eight bits of the SODB, etc., until fewer than eight bits of the SODB remain.
• the rbsp_trailing_bits( ) syntax structure is present after the SODB as follows: i)The first (most significant, left-most) bits of the final RBSP byte contain the remaining bits of the SODB (if any). ii)The next bit consists of a single bit equal to 1 (i.e., rbsp_stop_one_bit). iii)When the rbsp_stop_one_bit is not the last bit of a byte-aligned byte, one or more bits equal to 0 (i.e. instances of rbsp_alignment_zero_bit) are present to result in byte alignment.
• Syntax structures having these RBSP properties are denoted in the syntax tables using an "_rbsp" suffix. These structures are carried within NAL units as the content of the rbsp_byte[ i ] data bytes.
• the association of the RBSP syntax structures to the NAL units is as specified in Table 4 of ISO/IEC 23090- 5.
• the decoder can extract the SODB from the RBSP by concatenating the bits of the bytes of the RBSP and discarding the rbsp_stop_one_bit, which is the last (least significant, right-most) bit equal to 1 , and discarding any following (less significant, farther to the right) bits that follow it, which are equal to 0.
• the data necessary for the decoding process is contained in the SODB part of the RBSP.
• NAL unit types as for the atlas case, were defined for the base mesh enabling similar functionalities for random access and segmentation of the mesh. Unlike the atlas that is split into tiles, in this document we define the concept of a sub-mesh and define specific nal units that correspond to coded mesh data. In addition, NAL units that can include metadata such as SEI messages are also defined. In particular, the base mesh NAL unit types supported are specified as follows:
• the primary syntax structure that is defined for a base mesh bitstream is a sequence parameter set.
• This syntax structure contains basic information about the bitstream, identifying features for the codecs supported for either the intra coded and inter coded meshes, as well as information about references
• bmsps_intra_mesh_codec_id indicates the static mesh codec used to encode the base meshes in this base mesh substream. It could be associated with a specific mesh or motion mesh codec through the profiles specified in the corresponding specification, or could be explicitly indicated with an SEI message as is done in the V3C specification for the video sub-bitstreams.
• bmsps_intra_mesh_data_size_precision_bytes_minus1 (+1) specifies the precision, in bytes, of the size of the coded mesh data.
• bmsps_inter_mesh_codec_present_flag indicates if a specific codec indicated by bmsps_inter_mesh_codec_id is used to encode the inter predicted submeshes
• • bmsps_inter_mesh_data_size_precision_bytes_minus1 (+1 ) specifies the precision, in bytes, of the size of the inter predicted mesh data. This precision is signalled considering the size of the coded mesh data and the inter predicted mesh data(e.g. motion field) can be significantly different.
• • bmsps_facegroup_segmentation_method indicates how facegroups could be derived for a mesh.
• a facegroup is a set of triangle faces in a submesh. Each triangle face is associated with a Facegroupld indicating the facegroup it belongs to. When bmsps_facegroup_segmentation_method is 0, then Facegroupld is present directly in the coded submesh.
• value 1 means that there is no Facegroupld associated with any face.
• value 2 means that all faces are identified with a single ID, a 3 that facegroups are identified based on the connected component method, while a value of 4 indicates that each individual face has its own unique ID.
• ue(v) is used to indicate bmsps_facegroup_segmentation_method, but fixed length coding or partitioning to more elements could have been used instead.
• the base mesh frame parameter set has the frame level information such as number of submeshes in the frames corresponding to one mfh_mesh_frm_order_cnt_lsb.
• a submesh is coded in one mesh_data_submesh_layer() and is independently decodable from other submeshes.
• a submesh can refer only to the submeshes with the same smh_id in its associated reference frames. The mechanism is equivalent to what is specified in 8.3.6.2.2 in V3C.
• a bmesh_submesh_layer contains a submesh information.
• One or more bmesh_submesh_layer_rbsp can correspond to one mesh frame indicated by mfh_mesh_frm_order_cnt_lsb.
• smh_id is the id of the current submesh contained in the mesh data submesh data.
• smh_type indicates how the mesh is coded. If smh_type is l_SUBMESH, the mesh data is coded with the indicated static mesh codec. If smh_type is P_SUBMESH, inter prediction is used to code the mesh data.
• Submesh data unit smdu_intra_sub_mesh_unit( unitsize ) contains a sub mesh unit stream of size unitsize, in bytes, as an ordered stream of bytes or bits within which the locations of unit boundaries are identifiable from patterns in the data.
• the format of such sub mesh unit stream is identified by a 4CC code as defined by bmptl_profile_codec_group_idc or by a component codec mapping SEI message.
• smdu_inter_sub_mesh_unit( unitsize ) contains a sub mesh unit stream of size unitsize, in bytes, as an ordered stream of bytes or bits within which the locations of unit boundaries are identifiable from patterns in the data.
• V-DMC texture coordinates are transmitted for the base mesh and subdivided for the reconstructed displaced subdivided mesh. It follows that texture coordinates are only compressed in the base mesh substream as vertex attributes, by the chosen base mesh codec (for example based on edgebreaker and parallelogram geometry prediction).
• the cost of encoding texture coordinates is significant, for example, for a mesh without texture coordinates encoded with edgebreaker, the file size is equal to 38883 bytes, while with texture coordinates encoded as vertex attributes, the file size increases to 55768 bytes (that is 43% more bitrate).
• T racked mesh sequences with a single texture frame The first use case relates to tracked mesh sequences that have different geometry frames but a single texture frame. For such sequences, the vertex number and the connectivity typically are the same for every frame, and therefore, in the original format texture coordinates from the first frame as well as the texture map are reused for all consecutive frames. In V-DMC, this is not possible, and typically the texture coordinates are recomputed after the base mesh generation through simplification in the preprocessing module of the encoder. This leads the V-DMC bitstream to encode an attribute map for every frame, and these attribute maps are not stable temporally, leading to poor compression by a video codec.
• Dynamic mesh sequences with stable topology The second use case relates to general dynamic meshes that are not necessarily tracked and for which the texture coordinates and texture maps are available for every frame of the sequence.
• V-DMC does not offer mechanisms to enable temporally stable texture coordinates and texture maps without sending the texture coordinates.
• the topology is stable (i.e., the mesh geometry may have changed, but the number of holes, handles, and the total mesh area remains the same over time)
• the mesh texture coordinates could be mapped from mesh frame to mesh frame, but V-DMC does not support this unless the mesh is tracked.
• temporally unstable attribute maps are generated and lead to suboptimal encoding results as for the first case or require encoding temporally stable texture coordinates for each base mesh and therefore comes with a 40% bitrate overhead for the base mesh substream.
• the encoder may use testing. For that, the encoder tests for each base mesh (submesh) frame if texture coordinates can be recovered with any of the derivation mode (SKIP, MAP, ASSITED MAP, REFINED MAP) with a correct quality as expressed by a cost function.
• the cost function can be a sum of squared differences.
• the V-DCM encoder may text the derivation modes, and evaluates the rate distortion cost of selecting the mode: bitrate_for_signalling_mode + distortion(mode), where mode is one of the derivation modes. If no mode results in an acceptable cost compared to a threshold, the texture coordinates are encoded in the base mesh stream.
• the original sequence has only one texture frame for all mesh frames.
• the texture coordinates are then the same from frame to frame, but would be encoded at each frame. This is an example of a situation where the solution according to the present embodiments is not used.
• the disclosure use terms “base mesh” and “submeshes”, when discussing the present embodiments.
• the base mesh is an output of an base mesh (subbitstream) decoder.
• the submesh is a partition of a base mesh, i.e., a set of vertices, their connectivity and the associated attributes signalled in the basemesh sub-bitstream. Submeshes can be decoded completely independently. Each base mesh consists of one or more submeshes.
• the present embodiments have the following aspects:
• the present embodiments is about encoding some of the texture coordinates and not encoding other texture coordinates, this feature is taken into account in signalling. For example, it is signaled that the texture coordinates are present in a reference base mesh, the reference base mesh being a reference base mesh frame or submesh frame. On the other hand, it is also signaled that the texture coordinates are not present in a base mesh or submesh frame, whereupon they need to be derived by a decoder.
• the texture coordinate derivation mode to be used is signaled per a submesh frame at the decoder such as: SKIP, MAP, ASSISTED_MAP, REFINED_MAP.
• the present embodiment also provide high-level syntax for signalling the presence of anchor/feature vertices as vertex attribute data in the base mesh.
• the texture coordinate derivation mode to be selected may depend on the connectivity of a mesh. Connectivity is defined as a list of primitives that connect the vertices of the mesh. Examples of the primitives are edges connecting two vertices, or faces connecting three or more vertices.
• the texture coordinates are copied from a frame to the next frame if connectivity is the same.
• the texture coordinates are mapped from a frame to the next frame if connectivity changes, if feature vertices are not available, if feature vertices are available, or if refinable vertices are available.
• Mechanisms or modes for copying and/or mapping vertices are as follows. The difference between copying and mapping, is that in copying the vertex indices have to be exactly the same. In mapping, the vertex indices may be different, for example when some of them has been reordered or the number of indices is not the same. The following mechanisms and modes may be applied at the encoder and decoder in the same way:
• MAP in MAP mode, the texture coordinates extracted by the decoder from the reference base mesh frame are mapped to the current reconstructed base mesh frame.
• ASSISTED_MAP in ASSISTED_MAP mode, the texture coordinates extracted by the decoder from the reference base mesh frame are mapped to the current reconstructed base mesh frame by using anchor/feature points.
• REFINED_MAP in REFINED_MAP mode, the texture coordinates extracted by the decoder from the reference base mesh frame are mapped to the current reconstructed base mesh frame by using anchor/feature points and refinable vertices, for example by specifying a number of smoothing iterations to be performed in the parameterization domain for those vertices.
• This reference frame may be the previous reconstructed base mesh frame, an Intra (I) frame, or any frame referenced by a variable refFrameldx that is stored in a reconstructed base mesh frame buffer as illustrated on Figure 12 relating to a decoder, or mesh buffer as illustrated on Figure 15.
• the encoder and decoder can handle reference frames in the mesh buffer with a referenceFramelndexList, in a similar way as video encoders such as H.264 Advanced Video Coding (AVC) or H.265 High Efficiency Video Coding (HEVC) for example.
• AVC H.264 Advanced Video Coding
• HEVC High Efficiency Video Coding
• the modes, MAP, ASSISTED_MAP, REFINED_MAP, can be applied to a base mesh and/or to each available submesh present in the V-DMC stream. Applying and signalling these modes at submesh or base mesh level require the same mechanisms described hereafter.
• the SKIP mode addresses tracked meshes that have only one texture frame (or a small set of texture frames, called key frames) that should be re-used for every frame of the dynamic sequence.
• a tracked mesh means that the original mesh has been designed such that the order of vertices and their number is the same for all frames. This enables faster processing of the mesh sequence.
• This mode enables to ensure that V-DMC re-uses the same texture coordinates and texture frame at every mesh frame, such that the video codec used to compress the texture map will mostly use temporal skip prediction modes.
• the simplification module of the preprocessing module is applied to the first frame.
• the original indices of the vertices that are kept in the generated base mesh are also kept for the base meshes of the next frames as well as their connectivity in the base mesh. This ensures a tracking of the base meshes that is not offered by the current version of V-DMC framework.
• Other possibilities include a multi-frame optimization for the simplification, e.g., evaluating the cost of removing edges, not only for the current frame, but for a group of frames or the entire mesh sequence, in case the sequence is tracked.
• the SKIP mode may be used for any tracked mesh sequence, not necessarily having a single texture frame.
• the MAP mode addresses mesh sequences that are not tracked.
• MAP mode addresses mesh sequences for which the encoder has determined that the mapping can be achieved through inter-mesh-frame fitting without feature points, e.g., by k-NN nearest neighbors between the reconstructed reference frame and the current reconstructed base mesh frame. This is the case for base mesh frames that do not have the exact same number of vertices or triangles, but that do not exhibit significant motion from one frame to the next one.
• the texture coordinates are obtained by averaging the texture coordinates of the neighboring vertices that were mapped to vertices of the reference base mesh.
• Figure 18 illustrates a simplified example of texture coordinates mapping using the ASSISTED_MAP from the reference base mesh to the current base mesh using feature vertices.
• the ASSISTED_MAP mode addresses mesh sequences that are not tracked.
• the ASSISTED_MAP mode addresses mesh sequences for which the encoder detects that the inter-mesh- frame fitting requires feature points matching as the motion is more complex or the amplitude of the motion does not enable a k-NN search to provide a correct mapping.
• feature vertices are signaled in the reference base mesh as vertex attributes, for example with a flag (reference/not reference). The order of these reference feature points is provided by the order of vertex indices in the reference base mesh frame.
• the encoder may encode motion vectors for the feature vertices using a motion encoder as in the V- DMC framework, but instead of encoding motion for all vertices, which is expensive in bitrate, the encoding is only made for the sparser feature vertices.
• a k-NN search is performed to identify the feature vertices in the current base mesh frame based on the encoded motion vector. Once feature vertices are detected on the current base mesh frame, the mapping between non feature vertices of the reference base mesh frame and the current frame is performed by k-NN search using connected vertices of the feature vertices. The mapping may then be performed by propagating through connected vertices of already mapped vertices in a recursive manner.
• a mapping is obtained by nearest neighbors (k-NN) to perform a one to one mapping of each v_i to each rv_j vertex.
• the unmapped vertices v_j are added to a unmapped pool that is visited once the number of vj vertices is smaller than the number of rv_j vertices. If after this process, the pool is nonempty, the texture coordinates of the unmapped v_j vertices are obtained by linear interpolation of the texture coordinates of their neighboring and already mapped vertices.
• Feature vertices can be identified by the encoder through user assistance, or automatically by using known mesh vertex features such as protrusion extrema, curvature, salient points, or for example simply by detecting the vertices that after N smoothing iterations still have a normal displacement above a given threshold compared to their vertex neighborhood.
• the REFINED_MAP is a refinement of the ASSISTED_MAP, where the encoder may identify vertices that would get a better mapping if their geometry was smoothed by a signaled number of smoothing iterations prior to performing nearest neighbor search and mapping. Alternatively, the encoder may entropy encode (e.g., with arithmetic coding) the residuals of the texture coordinates for these vertices. The residuals are 2D vectors obtained at the encoder side by the vector difference between the ASSISTED_MAP generated 2D texture coordinates and the actual 2D texture coordinates of the vertices identified for refinement.
• xxxx_texture_coordinates_available_flag indicates “true” when texture coordinates are encoded in the base mesh stream and “false” when texture coordinates are not present in the base mesh stream.
• xxxx_tc_skip_reference_frame_index indicates the mesh reference frame index used for the texture coordinates skipping.
• xxxx_tc_skip_method indicates a skipping method as specified in the following table
• additional syntax structure is present for ASSISTED_MAP and REFINED_MAP: bmesh_tsmi_number_of_feature_points_minus1[submesh_id] indicates the number of feature points present in the current submesh with index submesh d. bmesh_tsmi_number_of_refined_points_minus1 [submeshjd] indicates the number of refined points present in the current submesh with index submeshjd. feature_point_index[submesh_id,i] indicates the i:th index of the feature points of submesh with index submeshjd.
• refined_point_index[submesh id,i] indicates the i:th index of the refined points of submesh with index submesh d.
• residual_texture_coordinate_u[submesh id,i] indicates the residual of the texture coordinate component U of the i:th refined point of submesh with index submeshjd.
• residual_texture_coordinate_v[submesh id,i] indicates the residual of the texture coordinate component V of the i:th refined point of submesh with index submeshjd.
• the syntax can be signaled in a base frame parameter set (BFAS).
• BFAS base frame parameter set
• the BFAS indicates that all base mesh frames other than intra frame that reference this parameter set use a given mode for texture coordinate estimation.
• bfps_texture_coordinates_available_flag indicates “true” when texture coordinates are encoded in the base mesh stream, and “false” when texture coordinates are not present in the base mesh stream.
• bfps_tc_skip_reference_frame_index indicates that the mesh reference frame index is used for the texture coordinates skipping.
• bfps_tc_skip_method indicates the skipping method as specified in the following table:
• the syntax can be signaled in base sequence parameter set (BSAS).
• BSAS base sequence parameter set
• the BSAS indicates that all base mesh frames other than intra frame that reference this parameter set use a given mode for texture coordinate estimation.
• bsmi_texture_coordinates_available_flag[ i ] indicates “true” if submesh identifier equal to i contains encoded texture coordinates.
• bsmi_tc_skip_reference_frame_index[i] indicates the mesh reference frame that is used for texture coordinate skipping for submesh index i.
• bsmi_tc_skip_method[i] indicates that the texture coordinate skipping method is used for submesh index i.
• the signalling information is provided on submesh level in sdu_inter_sub_mesh_unit syntax structure
• sismu_texture_coordinates_available_flag[ subMeshlD ] indicates “true” when submesh with identifier equal to subMeshlD contains encoded texture coordinates.
• sismu_tc_skip_reference_frame_index[ subMeshlD ] indicates the mesh reference frame used for texture coordinate skipping for submesh with identifier equal to subMeshlD.
• sismu_tc_skip_method[ subMeshlD ] indicates the texture coordinate skipping method used for submesh with identifier equal to subMeshlD.
• the derivation mode of texture coordinates can be provided through base mesh NAL unity type.
• one NAL unit type indicates that SKIP mode for texture coordinates is used in one base mesh or submesh
• another NAL unit type indicates the MAP mode that is used for texture coordinates is used in another base mesh submesh.
• the signaling of is applied individually for each given texture of a mesh. For example, one texture containing color information would use one mode while other texture containing reflectance/normal information would use another mode.
• the type of mapping can also be signaled for example at atlas sequence parameter set (ASPS) or atlas frame parameter set (AFPS).
• the method for encoding is shown in Figure 19.
• the method generally comprises receiving 1905 a dynamic three-dimensional mesh sequence, wherein the three-dimensional mesh represents a three- dimensional object with texture; determining 1910 which texture coordinates are not encoded into a bitstream; determining 1915 a derivation mode for the texture coordinates not to be encoded; encoding 1920 into a bitstream an indication of the determined derivation mode; encoding 1925 into a bitstream an information on a reference frame comprising texture coordinates for the current mesh frame; encoding 1930 a base mesh into a bitstream with the texture coordinates which are to be encoded; and transmitting 1935 the one or more bitstreams to a decoder.
• Each of the steps can be implemented by a respective module of a computer system.
• An apparatus comprises means for receiving a dynamic three-dimensional mesh sequence, wherein the three-dimensional mesh represents a three-dimensional object with texture; means for determining which texture coordinates are not encoded into a bitstream; means for determining a derivation mode for the texture coordinates not to be encoded; means for encoding into a bitstream an indication of the determined derivation mode; means for encoding into a bitstream an information on a reference frame comprising texture coordinates for the current mesh frame; means for encoding a base mesh into a bitstream with the texture coordinates which are to be encoded; and means for transmitting the one or more bitstreams to a decoder.
• the means comprises at least one processor, and a memory including a computer program code, wherein the processor may further comprise processor circuitry.
• the memory and the computer program code are configured to, with the at least one processor, cause the apparatus to perform the method of Figure 19 according to various embodiments.
• the method for decoding generally comprises receiving 2010 one or more bitstreams from an encoder; decoding 2015 a base mesh from a bitstream texture coordinates; determining 2020 texture coordinates missing from a bitstream; decoding 2030 an indication on a derivation mode from a bitstream; decoding 2040 information on a reference frame comprising texture coordinates missing from the bitstream; reconstructing 2050 missing texture coordinates from the reference frame according to the derivation mode; and reconstructing 2060 a dynamic three-dimensional mesh sequence according to the texture coordinates.
• Each of the steps can be implemented by a respective module of a computer system.
• An apparatus comprises means for receiving one or more bitstreams from an encoder; means for decoding a base mesh from a bitstream texture coordinates; means for determining texture coordinates missing from a bitstream; means for decoding an indication on a derivation mode from a bitstream; means for decoding information on a reference frame comprising texture coordinates missing from the bitstream; means for reconstructing missing texture coordinates from the reference frame according to the derivation mode; and means for reconstructing a dynamic three- dimensional mesh sequence according to the texture coordinates.
• the means comprises at least one processor, and a memory including a computer program code, wherein the processor may further comprise processor circuitry.
• the memory and the computer program code are configured to, with the at least one processor, cause the apparatus to perform the method of Figure 20 according to various embodiments.
• Figure 21 shows a block diagram of a video coding system according to an example embodiment as a schematic block diagram of an electronic device 50, which may incorporate a codec.
• the electronic device may comprise an encoder or a decoder.
• the electronic device 50 may for example be a mobile terminal or a user equipment of a wireless communication system or a camera device.
• the electronic device 50 may be also comprised at a local or a remote server or a graphics processing unit of a computer.
• the device may be also comprised as part of a head-mounted display device.
• the apparatus 50 may comprise a display 32 in the form of a liquid crystal display. In other embodiments of the invention the display may be any suitable display technology suitable to display an image or video.
• the apparatus 50 may further comprise a keypad 34.
• any suitable data or user interface mechanism may be employed.
• the user interface may be implemented as a virtual keyboard or data entry system as part of a touch-sensitive display.
• the apparatus may comprise a microphone 36 or any suitable audio input which may be a digital or analogue signal input.
• the apparatus 50 may further comprise an audio output device which in embodiments of the invention may be any one of: an earpiece 38, speaker, or an analogue audio or digital audio output connection.
• the apparatus 50 may also comprise a battery (or in other embodiments of the invention the device may be powered by any suitable mobile energy device such as solar cell, fuel cell or clockwork generator).
• the apparatus may further comprise a camera 42 capable of recording or capturing images and/or video.
• the camera 42 may be a multi-lens camera system having at least two camera sensors.
• the camera is capable of recording or detecting individual frames which are then passed to the codec 54 or the controller for processing.
• the apparatus may receive the video and/or image data for processing from another device prior to transmission and/or storage.
• the apparatus 50 may comprise a controller 56 or processor for controlling the apparatus 50.
• the apparatus or the controller 56 may comprise one or more processors or processor circuitry and be connected to memory 58 which may store data in the form of image, video and/or audio data, and/or may also store instructions for implementation on the controller 56 or to be executed by the processors or the processor circuitry.
• the controller 56 may further be connected to codec circuitry 54 suitable for carrying out coding and decoding of image, video and/or audio data or assisting in coding and decoding carried out by the controller.
• the apparatus 50 may further comprise a card reader 48 and a smart card 46, for example a IIICC (Universal Integrated Circuit Card) and UICC reader for providing user information and being suitable for providing authentication information for authentication and authorization of the user at a network.
• the apparatus 50 may comprise radio interface circuitry 52 connected to the controller and suitable for generating wireless communication signals for example for communication with a cellular communications network, a wireless communications system, or a wireless local area network.
• the apparatus 50 may further comprise an antenna 44 connected to the radio interface circuitry 52 for transmitting radio frequency signals generated at the radio interface circuitry 52 to other apparatus(es) and for receiving radio frequency signals from other apparatus(es).
• the apparatus may comprise one or more wired interfaces configured to transmit and/or receive data over a wired connection, for example an electrical cable or an optical fiber connection.
• the various embodiments may provide advantages. For example, the present embodiments allow a smaller bitrate thanks to the fact that texture coordinates are not encoded in the base mesh frame that are not reference (or key) frames.
• the present embodiments provide temporally stable texture maps that enable better attribute map compression.
• the present embodiments also provide faster encoding that using independent parameterization for each frame. Since a motion encoder is not used to map base meshes in the present embodiments, enables avoiding long sequence of motion vectors that are entropy encoded, and which limit the throughput of the decoder. Furthermore, motion codecs may require the connectivity to be the same for the reference mesh frames and the current mesh frames, which is too restrictive.
• the proposed mapping techniques enable the encoder to efficiently skip texture coordinate data, while handling different connectivities.
• the present embodiments further enables reduced decoding time for the base mesh frames at the decoder side.
• a device may comprise circuitry and electronics for handling, receiving, and transmitting data, computer program code in a memory, and a processor that, when running the computer program code, causes the device to carry out the features of an embodiment.
• a network device like a server may comprise circuitry and electronics for handling, receiving, and transmitting data, computer program code in a memory, and a processor that, when running the computer program code, causes the network device to carry out the features of various embodiments.

Landscapes

• Engineering & Computer Science (AREA)
• Multimedia (AREA)
• Signal Processing (AREA)
• Physics & Mathematics (AREA)
• General Physics & Mathematics (AREA)
• Theoretical Computer Science (AREA)
• Library & Information Science (AREA)
• Compression Or Coding Systems Of Tv Signals (AREA)

Applications Claiming Priority (2)

Publications (1)

Family

ID=92971372

Family Applications (1)

Country Status (2)

Family Cites Families (2)

• 2024
  • 2024-02-27 WO PCT/FI2024/050069 patent/WO2024209129A1/en not_active Ceased
  • 2024-02-27 EP EP24784468.1A patent/EP4690806A1/de active Pending

Also Published As

Similar Documents

Legal Events