WO2023180840A1 - Codage d'informations de connectivité de correctif de surface de maillage - Google Patents

Codage d'informations de connectivité de correctif de surface de maillage Download PDF

Info

Publication number
WO2023180840A1
WO2023180840A1 PCT/IB2023/052103 IB2023052103W WO2023180840A1 WO 2023180840 A1 WO2023180840 A1 WO 2023180840A1 IB 2023052103 W IB2023052103 W IB 2023052103W WO 2023180840 A1 WO2023180840 A1 WO 2023180840A1
Authority
WO
WIPO (PCT)
Prior art keywords
encoding
information
vertices
patch
connectivity
Prior art date
Application number
PCT/IB2023/052103
Other languages
English (en)
Inventor
Danillo GRAZIOSI
Alexandre ZAGHETTO
Ali Tabatabai
Original Assignee
Sony Group Corporation
Sony Corporation Of America
Priority date (The priority date is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the date listed.)
Filing date
Publication date
Priority claimed from US17/987,833 external-priority patent/US20230306642A1/en
Application filed by Sony Group Corporation, Sony Corporation Of America filed Critical Sony Group Corporation
Priority to CN202380013125.5A priority Critical patent/CN117751387A/zh
Publication of WO2023180840A1 publication Critical patent/WO2023180840A1/fr

Links

Classifications

    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
    • G06T9/00—Image coding
    • G06T9/001—Model-based coding, e.g. wire frame
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
    • G06T9/00—Image coding
    • G06T9/004—Predictors, e.g. intraframe, interframe coding

Definitions

  • the method also known as V3C (visual volumetric video-based compression) maps the 3D volumetric data into several 2D patches, and then further arranges the patches into an atlas image, which is subsequently encoded with a video encoder.
  • the atlas images correspond to the geometry of the points, the respective texture, and an occupancy map that indicates which of the positions are to be considered for the point cloud reconstruction.
  • CfP call for proposal
  • MPEG is using a test model software (TMC2) based on patch surface modeling, projection of patches from 3D to 2D image, and coding the 2D image with video encoders such as HEVC.
  • TMC2 test model software
  • This method has proven to be more efficient than native 3D coding, and is able to achieve competitive bitrates at acceptable quality.
  • Due to the success for coding 3D point clouds of the projection-based method also known as the video-based method, or V-PCC
  • the standard is expected to include in future versions further 3D data, such as 3D meshes.
  • current version of the standard is only suitable for the transmission of an unconnected set of points, so there is nomechanism to send the connectivity of points, as it is required in 3D mesh compression.
  • V-PCC V-PCC
  • a mesh compression approach like TFAN or Edgebreaker.
  • the limitation of this method is that the original mesh has to be dense, so that the point cloud generated from the vertices is not sparse and can be efficiently encoded after projection.
  • the order of the vertices affect the coding of connectivity, and different method to reorganize the mesh connectivity have been proposed.
  • An alternative way to encode a sparse mesh is to use the RAW patch data to encode the vertices position in 3D.
  • RAW patches encode (x,y,z) directly
  • all the vertices are encoded as RAW data
  • the connectivity is encoded by a similar mesh compression method, as mentioned before.
  • the vertices may be sent in any preferred order, so the order generated from connectivity encoding can be used.
  • the method can encode sparse point clouds, however, RAW patches are not efficient to encode 3D data, and further data such as the attributes of the triangle faces may be missing from this approach.
  • the connectivity information and mapping information of a mesh surface patch are able to be encoded after projection to 2D.
  • the projection operation does not change the connection between vertices, so the same list of connected vertices are able to be carried in the atlas data.
  • the mapping information does not change after projection and is able to be carried in the atlas data.
  • Two methods are disclosed for encoding the connectivity and mapping information.
  • a video-based method uses neighboring color coding.
  • mapping coordinates a method uses the projected vertex position.
  • the connectivity and mapping are also able to be processed by an external mesh encoder. Newly proposed mapping information is able to be taken advantage of to perform temporal compression.
  • a method of encoding connectivity information and mapping information comprises encoding vertex mapping information including delta information for geometry correction and encoding patch connectivity information, including implementing mesh simplification by fixing positions of vertices in time.
  • the method further comprises sending a flag indicating whether the vertex mapping information is implicitly or explicitly sent.
  • Encoding the vertex information and encoding the patch connectivity information are performed by an external encoder.
  • Encoding the patch connectivity information comprises using color coding in an occupancy map. Using color coding in the occupancy map is limited to a maximum of 4 colors.
  • Encoding vertex mapping information further comprises using rate distortion face transmission. Implementing mesh simplification includes sending only boundary vertices and not sending inner vertices.
  • an apparatus comprises a non-transitory memory for storing an application, the application for: encoding vertex mapping information including delta information for geometry correction and encoding patch connectivity information, including implementing mesh simplification by fixing positions of vertices in time and a processor coupled to the memory, the processor configured for processing the application.
  • the application is further configured for sending a flag indicating whether the vertex mapping information is implicitly or explicitly sent.
  • Encoding the vertex information and encoding the patch connectivity information are performed by an external encoder.
  • Encoding the patch connectivity information comprises using color coding in an occupancy map. Using color coding in the occupancy map is limited to a maximum of 4 colors.
  • Encoding vertex mapping information further comprises using rate distortion face transmission.
  • Implementing mesh simplification includes sending only boundary vertices and not sending inner vertices. The inner vertices are determined based on a previous set of inner vertices from a previous frame.
  • a system comprises one or more cameras for acquiring three dimensional content, an encoder for encoding the three dimensional content: encoding vertex mapping information including delta information for geometry correction and encoding patch connectivity information, including implementing mesh simplification by fixing positions of vertices in time and a decoder for decoding the encoded three dimensional content including: using the delta information for adjusting a mesh, determining inner vertices of the patch connectivity information from previous inner vertices from a previous frame.
  • the encoder is further configured for sending a flag indicating whether the vertex mapping information is implicitly or explicitly sent.
  • Encoding the vertex information and encoding the patch connectivity information are performed by an external encoder.
  • Encoding the patch connectivity information comprises using color coding in an occupancy map. Using color coding in the occupancy map is limited to a maximum of 4 colors.
  • Encoding vertex mapping information further comprises using rate distortion face transmission. Implementing mesh simplification includes sending only boundary vertices and not sending inner vertices.
  • Figure 4 illustrates a diagram of temporal stability in mesh connectivity according to some embodiments.
  • Figure 5 illustrates a flowchart of a method of patch mesh connectivity coding according to some embodiments.
  • Figure 6 illustrates a block diagram of an exemplary computing device configured to implement the patch mesh connectivity coding method according to some embodiments.
  • DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENT The connectivity information and mapping information of a mesh surface patch are able to be encoded after projection to 2D. Regarding the connectivity information, the projection operation does not change the connection between vertices, so the same list of connected vertices are able to be carried in the atlas data. Similarly, the mapping information does not change after projection and is able to be carried in the atlas data.
  • the connectivity information indicates which pixels are connected. There are sets of information for a triangle. One set of information is the position of the triangle on the texture map. For each triangle on the texture map there are two sets of information - 1) how the vertices are connected in 3D (e.g., in a connectivity list) and 2) vertex mapping information. There are three ways of encoding vertex mapping information - implicit, explicit, binary.
  • the projection is the same as the mapping.
  • the place hit when projecting on the projection surface is the UV coordinate.
  • the explicit information is encoded with an external encoder (e.g., Draco or AFX).
  • AtlasPatch2dPosX mpdu_vertex_pos_x[tileID][patchIdx][I]
  • AtlasPatch2dPosY mpdu_vertex_pos_y[tileID][patchIdx][I] if (asps_mesh_uv_coordinates_present_flag)
  • mappingU (mpdu_vertex_u_coord[tileID][patchIdx][i]) / 2 asps_mesh_coordinates_bit_depth_minus1+1 -1
  • mappingV (mpdu_vertex_v_coord[tileID][patchIdx][i]) / 2 asps_mesh_coordinates_bit_depth_minus1+1 -1 else
  • the value is scaled by a bit depth.
  • an external mesh encoder is able to be used to encode the patch mesh information. U and V are added on the ply, and the vertex mapping information is encoded with the ply. In some embodiments, delta information for the z coordinate is added. The delta information is able to be used for geometry correction.
  • patch connectivity information For the explicit implementation, it is indicated in the syntax which pixels are connected, so the list of pixel connectivity is sent in the patch.
  • an external encoder is able to be utilized.
  • mesh simplification is able to be performed by fixing the position of the vertices across time.
  • FIG. 1 illustrates a diagram of binary encoding according to some embodiments.
  • the geometry image is used to generate the map 100.
  • the binary image is able to be generated while correcting errors from video transmission.
  • the UV coordinates are also sent which helps with the video compression.
  • an external mesh encoder is used to encode the patch mesh information.
  • Figure 2 illustrates a diagram of color coding using an occupancy map according to some embodiments.
  • FIG. 3 illustrates a diagram of RD face transmission according to some embodiments.
  • a mesh is received/acquired 300.
  • the mesh connectivity points are encoded by the encoder 302. On the decoder side, the mesh is reconstructed 304, but the positions of the points may be slightly different than the original mesh. Correction information (e.g., delta information) is sent to the decoder, so the decoder is able to adjust the mesh 306 so that it is more accurate compared to the original mesh.
  • the vertices of an input mesh are V-PCC encoded and locally decoded.
  • the encoder generates a mesh from a decoded point cloud.
  • the encoder compares the generated face/connectivity information versus the original information.
  • the encoder signals the non- matching faces incorporating rate/distortion tradeoffs.
  • the decoder uses V-PCC to decode the mesh vertices and generate a mesh from the decoded vertices.
  • the decoder uses the signaled non-matched faces to modify the mesh.
  • Figure 4 illustrates a diagram of temporal stability in mesh connectivity according to some embodiments. When a patch is sent, only the boundary vertices are sent. The inner vertices are not sent. The decoder determines the inner vertices (e.g., based on previous or subsequent frames).
  • the inner vertices from patch 400 in frame 1 are used as the inner vertices for patch 402 in frame 2.
  • a first set of inner vertices are sent (e.g., for frame 0 or frame 1), so that future frames have inner vertices from a prior frame to use.
  • the decoder is able to re-generated the triangles from the boundary vertices and the inner vertices. Even if the patch is rotating from frame to frame, the same inner points are able to be used.
  • the inner triangles may be slightly different because of the rotation, but that is acceptable. By not sending the inner vertices, fewer bits are sent.
  • Figure 5 illustrates a flowchart of a method of patch mesh connectivity coding according to some embodiments.
  • vertex mapping information is encoded. Delta information is able to be included with the vertex mapping information for geometry correction. A flag is able to be sent indicating whether the vertex mapping information is implicitly or explicitly sent.
  • encoding vertex mapping information uses RD face transmission.
  • patch connectivity information is encoded. Encoding patch connectivity information includes implementing mesh simplification by fixing positions of vertices in time. In some embodiments, mesh simplification includes sending only boundary vertices and not sending inner vertices. The patch connectivity information is able to include color coding in an occupancy map. The color coding is limited to a maximum of 4 colors. In some embodiments, the vertex information and the patch connectivity information are performed by an external encoder.
  • the order of the steps is modified. In some embodiments, fewer or additional steps are implemented.
  • the encoded information is able to use the delta information to adjust a mesh.
  • inner vertices of the patch connectivity are able to be determined from previous inner vertices from a previous frame.
  • Figure 6 illustrates a block diagram of an exemplary computing device configured to implement the patch mesh connectivity coding method according to some embodiments.
  • the computing device 600 is able to be used to acquire, store, compute, process, communicate and/or display information such as images and videos including 3D content.
  • the computing device 600 is able to implement any of the encoding/decoding aspects.
  • a hardware structure suitable for implementing the computing device 600 includes a network interface 602, a memory 604, a processor 606, I/O device(s) 608, a bus 610 and a storage device 612.
  • the choice of processor is not critical as long as a suitable processor with sufficient speed is chosen.
  • the memory 604 is able to be any conventional computer memory known in the art.
  • the storage device 612 is able to include a hard drive, CDROM, CDRW, DVD, DVDRW, High Definition disc/drive, ultra-HD drive, flash memory card or any other storage device.
  • the computing device 600 is able to include one or more network interfaces 602.
  • An example of a network interface includes a network card connected to an Ethernet or other type of LAN.
  • the I/O device(s) 608 are able to include one or more of the following: keyboard, mouse, monitor, screen, printer, modem, touchscreen, button interface and other devices.
  • Patch mesh connectivity coding application(s) 630 used to implement the patch mesh connectivity coding implementation are likely to be stored in the storage device 612 and memory 604 and processed as applications are typically processed. More or fewer components shown in Figure 6 are able to be included in the computing device 600. In some embodiments, patch mesh connectivity coding hardware 620 is included. Although the computing device 600 in Figure 6 includes applications 630 and hardware 620 for the patch mesh connectivity coding implementation, the patch mesh connectivity coding method is able to be implemented on a computing device in hardware, firmware, software or any combination thereof.
  • the patch mesh connectivity coding applications 630 are programmed in a memory and executed using a processor.
  • the patch mesh connectivity coding hardware 620 is programmed hardware logic including gates specifically designed to implement the patch mesh connectivity coding method.
  • the patch mesh connectivity coding application(s) 630 include several applications and/or modules. In some embodiments, modules include one or more sub- modules as well. In some embodiments, fewer or additional modules are able to be included.
  • suitable computing devices include a personal computer, a laptop computer, a computer workstation, a server, a mainframe computer, a handheld computer, a personal digital assistant, a cellular/mobile telephone, a smart appliance, a gaming console, a digital camera, a digital camcorder, a camera phone, a smart phone, a portable music player, a tablet computer, a mobile device, a video player, a video disc writer/player (e.g., DVD writer/player, high definition disc writer/player, ultra high definition disc writer/player), a television, a home entertainment system, an augmented reality device, a virtual reality device, smart jewelry (e.g., smart watch), a vehicle (e.g., a self-driving vehicle) or any other suitable computing device.
  • a personal computer e.g., a laptop computer, a computer workstation, a server, a mainframe computer, a handheld computer, a personal digital assistant, a cellular/mobile telephone, a smart appliance, a gaming console
  • a device acquires or receives 3D content (e.g., point cloud content).
  • 3D content e.g., point cloud content.
  • the patch mesh connectivity coding method is able to be implemented with user assistance or automatically without user involvement.
  • the patch mesh connectivity coding method enables more efficient and more accurate 3D content encoding compared to previous implementations.
  • SOME EMBODIMENTS OF PATCH MESH CONNECTIVITY CODING 1.
  • a method of encoding connectivity information and mapping information comprising: encoding vertex mapping information including delta information for geometry correction; and encoding patch connectivity information, including implementing mesh simplification by fixing positions of vertices in time. 2.
  • the method of clause 1 further comprising sending a flag indicating whether the vertex mapping information is implicitly or explicitly sent. 3.
  • encoding the vertex information and encoding the patch connectivity information are performed by an external encoder. 4.
  • encoding the patch connectivity information comprises using color coding in an occupancy map. 5.
  • using color coding in the occupancy map is limited to a maximum of 4 colors.
  • encoding vertex mapping information further comprises using rate distortion face transmission.
  • implementing mesh simplification includes sending only boundary vertices and not sending inner vertices.
  • the inner vertices are determined based on a previous set of inner vertices from a previous frame. 9.
  • An apparatus comprising: a non-transitory memory for storing an application, the application for: encoding vertex mapping information including delta information for geometry correction; and encoding patch connectivity information, including implementing mesh simplification by fixing positions of vertices in time; and a processor coupled to the memory, the processor configured for processing the application. 10.
  • the application is further configured for sending a flag indicating whether the vertex mapping information is implicitly or explicitly sent.
  • encoding the vertex information and encoding the patch connectivity information are performed by an external encoder.
  • encoding the patch connectivity information comprises using color coding in an occupancy map. 13.
  • a system comprising: one or more cameras for acquiring three dimensional content; an encoder for encoding the three dimensional content: encoding vertex mapping information including delta information for geometry correction; and encoding patch connectivity information, including implementing mesh simplification by fixing positions of vertices in time; and a decoder for decoding the encoded three dimensional content including: using the delta information for adjusting a mesh; determining inner vertices of the patch connectivity information from previous inner vertices from a previous frame. 18.
  • the encoder is further configured for sending a flag indicating whether the vertex mapping information is implicitly or explicitly sent.
  • encoding the vertex information and encoding the patch connectivity information are performed by an external encoder. 20.
  • encoding the patch connectivity information comprises using color coding in an occupancy map. 21.
  • using color coding in the occupancy map is limited to a maximum of 4 colors.
  • encoding vertex mapping information further comprises using rate distortion face transmission.
  • implementing mesh simplification includes sending only boundary vertices and not sending inner vertices.

Landscapes

  • Engineering & Computer Science (AREA)
  • Multimedia (AREA)
  • Physics & Mathematics (AREA)
  • General Physics & Mathematics (AREA)
  • Theoretical Computer Science (AREA)
  • Compression Or Coding Systems Of Tv Signals (AREA)

Abstract

Les informations de connectivité et les informations de mappage d'un correctif de surface de maillage peuvent être codées après une projection en 2D. Concernant les informations de connectivité, l'opération de projection ne change pas la connexion entre des sommets, de sorte que la même liste de sommets connectés peut être transportée dans les données d'atlas. De façon similaire, les informations de mappage ne changent pas après la projection, et peuvent être transportées dans les données d'atlas. Deux procédés sont décrits pour coder les informations de connectivité et de mappage. Pour les informations de connectivité, un procédé basé sur la vidéo utilise un codage de couleur voisin. Pour mapper des coordonnées, un procédé utilise la position de sommet projeté. Les informations de connectivité et de mappage peuvent également être traitées par un codeur de maillage externe. Des informations de mappage nouvellement proposées peuvent être mises à profit pour réaliser une compression temporelle.
PCT/IB2023/052103 2022-03-25 2023-03-06 Codage d'informations de connectivité de correctif de surface de maillage WO2023180840A1 (fr)

Priority Applications (1)

Application Number Priority Date Filing Date Title
CN202380013125.5A CN117751387A (zh) 2022-03-25 2023-03-06 面元网格连通性编码

Applications Claiming Priority (4)

Application Number Priority Date Filing Date Title
US202263269905P 2022-03-25 2022-03-25
US63/269,905 2022-03-25
US17/987,833 US20230306642A1 (en) 2022-03-25 2022-11-15 Patch mesh connectivity coding
US17/987,833 2022-11-15

Publications (1)

Publication Number Publication Date
WO2023180840A1 true WO2023180840A1 (fr) 2023-09-28

Family

ID=85724760

Family Applications (1)

Application Number Title Priority Date Filing Date
PCT/IB2023/052103 WO2023180840A1 (fr) 2022-03-25 2023-03-06 Codage d'informations de connectivité de correctif de surface de maillage

Country Status (1)

Country Link
WO (1) WO2023180840A1 (fr)

Citations (1)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
WO2021188238A1 (fr) * 2020-03-18 2021-09-23 Sony Group Corporation Compression de maillage sur la base d'une projection

Patent Citations (1)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
WO2021188238A1 (fr) * 2020-03-18 2021-09-23 Sony Group Corporation Compression de maillage sur la base d'une projection

Non-Patent Citations (4)

* Cited by examiner, † Cited by third party
Title
DANILLO GRAZIOSI (SONY) ET AL: "[V-PCC][EE2.6-related] Mesh Patch Data", no. m55368, 7 October 2020 (2020-10-07), XP030292889, Retrieved from the Internet <URL:https://dms.mpeg.expert/doc_end_user/documents/132_OnLine/wg11/m55368-v1-m55368_mesh_patch_data.zip m55368_mesh_patch_data.docx> [retrieved on 20201007] *
DANILLO GRAZIOSI (SONY) ET AL: "[V-PCC][EE2.6-related] Mesh Patch Data", no. m55368, 7 October 2020 (2020-10-07), XP030292890, Retrieved from the Internet <URL:https://dms.mpeg.expert/doc_end_user/documents/132_OnLine/wg11/m55368-v1-m55368_mesh_patch_data.zip m55368_mesh_patch_data.pdf> [retrieved on 20201007] *
DANILLO GRAZIOSI (SONY) ET AL: "[V-PCC][EE2.6-related] Tracked Patch Data", no. m55372, 7 October 2020 (2020-10-07), XP030292895, Retrieved from the Internet <URL:https://dms.mpeg.expert/doc_end_user/documents/132_OnLine/wg11/m55372-v1-m55372_tracked_mesh_patch_data.zip m55372_tracked_mesh_patch_data.docx> [retrieved on 20201007] *
YANG S ET AL: "View-dependent progressive mesh coding based on partitioning", VISUAL COMMUNICATIONS AND IMAGE PROCESSING; 21-1-2002 - 23-1-2002; SAN JOSE,, 21 January 2002 (2002-01-21), XP030080523 *

Similar Documents

Publication Publication Date Title
US11348285B2 (en) Mesh compression via point cloud representation
US11010955B2 (en) Point cloud mapping
CN107454468B (zh) 对沉浸式视频进行格式化的方法、装置和流
US11190803B2 (en) Point cloud coding using homography transform
US11321878B2 (en) Decoded tile hash SEI message for V3C/V-PCC
US10735766B2 (en) Point cloud auxiliary information coding
US11908169B2 (en) Dense mesh compression
US20200105023A1 (en) Packing strategy signaling
US11196977B2 (en) Unified coding of 3D objects and scenes
US20230306642A1 (en) Patch mesh connectivity coding
WO2023180840A1 (fr) Codage d&#39;informations de connectivité de correctif de surface de maillage
US20240127489A1 (en) Efficient mapping coordinate creation and transmission
US20230306641A1 (en) Mesh geometry coding
US20230306644A1 (en) Mesh patch syntax
US20230306683A1 (en) Mesh patch sub-division
US20230306687A1 (en) Mesh zippering
US20240233189A1 (en) V3c syntax extension for mesh compression using sub-patches
US20230306643A1 (en) Mesh patch simplification
CN117751387A (zh) 面元网格连通性编码
US20240177355A1 (en) Sub-mesh zippering
US20240153147A1 (en) V3c syntax extension for mesh compression
WO2023180839A1 (fr) Codage de géométrie de maillage
WO2023180845A1 (fr) Syntaxe de correctif de maillage
WO2023180841A1 (fr) Sous-division d&#39;éléments de maillage
WO2023180844A1 (fr) Zippering de maillage

Legal Events

Date Code Title Description
121 Ep: the epo has been informed by wipo that ep was designated in this application

Ref document number: 23712603

Country of ref document: EP

Kind code of ref document: A1

WWE Wipo information: entry into national phase

Ref document number: 202380013125.5

Country of ref document: CN