WO2024148752A1 - 场景描述文件的解析方法及装置 - Google Patents

场景描述文件的解析方法及装置 Download PDF

Info

Publication number
WO2024148752A1
WO2024148752A1 PCT/CN2023/097877 CN2023097877W WO2024148752A1 WO 2024148752 A1 WO2024148752 A1 WO 2024148752A1 CN 2023097877 W CN2023097877 W CN 2023097877W WO 2024148752 A1 WO2024148752 A1 WO 2024148752A1
Authority
WO
WIPO (PCT)
Prior art keywords
tactile
media
description
buffer
file
Prior art date
Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
Ceased
Application number
PCT/CN2023/097877
Other languages
English (en)
French (fr)
Inventor
杨付正
张伟
曹伊茉
王俊杰
王之奎
李斌
张雯
邢芳
Current Assignee (The listed assignees may be inaccurate. Google has not performed a legal analysis and makes no representation or warranty as to the accuracy of the list.)
Hisense Visual Technology Co Ltd
Original Assignee
Hisense Visual Technology Co Ltd
Priority date (The priority date is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the date listed.)
Filing date
Publication date
Application filed by Hisense Visual Technology Co Ltd filed Critical Hisense Visual Technology Co Ltd
Publication of WO2024148752A1 publication Critical patent/WO2024148752A1/zh
Anticipated expiration legal-status Critical
Ceased legal-status Critical Current

Links

Classifications

    • G—PHYSICS
    • G06—COMPUTING OR CALCULATING; COUNTING
    • G06F—ELECTRIC DIGITAL DATA PROCESSING
    • G06F3/00—Input arrangements for transferring data to be processed into a form capable of being handled by the computer; Output arrangements for transferring data from processing unit to output unit, e.g. interface arrangements
    • G06F3/01—Input arrangements or combined input and output arrangements for interaction between user and computer
    • G—PHYSICS
    • G06—COMPUTING OR CALCULATING; COUNTING
    • G06F—ELECTRIC DIGITAL DATA PROCESSING
    • G06F3/00—Input arrangements for transferring data to be processed into a form capable of being handled by the computer; Output arrangements for transferring data from processing unit to output unit, e.g. interface arrangements
    • G06F3/01—Input arrangements or combined input and output arrangements for interaction between user and computer
    • G06F3/016—Input arrangements with force or tactile feedback as computer generated output to the user
    • G—PHYSICS
    • G06—COMPUTING OR CALCULATING; COUNTING
    • G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
    • G06T1/00—General purpose image data processing
    • G06T1/60—Memory management
    • G—PHYSICS
    • G06—COMPUTING OR CALCULATING; COUNTING
    • G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
    • G06T13/00—Animation
    • G—PHYSICS
    • G06—COMPUTING OR CALCULATING; COUNTING
    • G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
    • G06T13/00—Animation
    • G06T13/20—Three-dimensional [3D] animation
    • G—PHYSICS
    • G06—COMPUTING OR CALCULATING; COUNTING
    • G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
    • G06T13/00—Animation
    • G06T13/20—Three-dimensional [3D] animation
    • G06T13/40—Three-dimensional [3D] animation of characters, e.g. humans, animals or virtual beings
    • G—PHYSICS
    • G06—COMPUTING OR CALCULATING; COUNTING
    • G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
    • G06T15/00—Three-dimensional [3D] image rendering
    • G—PHYSICS
    • G06—COMPUTING OR CALCULATING; COUNTING
    • G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
    • G06T15/00—Three-dimensional [3D] image rendering
    • G06T15/005—General purpose rendering architectures
    • G—PHYSICS
    • G06—COMPUTING OR CALCULATING; COUNTING
    • G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
    • G06T15/00—Three-dimensional [3D] image rendering
    • G06T15/04—Texture mapping
    • G—PHYSICS
    • G06—COMPUTING OR CALCULATING; COUNTING
    • G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
    • G06T17/00—Three-dimensional [3D] modelling for computer graphics
    • G—PHYSICS
    • G06—COMPUTING OR CALCULATING; COUNTING
    • G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
    • G06T17/00—Three-dimensional [3D] modelling for computer graphics
    • G06T17/20—Finite element generation, e.g. wire-frame surface description, tesselation
    • G—PHYSICS
    • G06—COMPUTING OR CALCULATING; COUNTING
    • G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
    • G06T19/00—Manipulating three-dimensional [3D] models or images for computer graphics
    • H—ELECTRICITY
    • H04—ELECTRIC COMMUNICATION TECHNIQUE
    • H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
    • H04N19/00—Methods or arrangements for coding, decoding, compressing or decompressing digital video signals
    • H04N19/50—Methods or arrangements for coding, decoding, compressing or decompressing digital video signals using predictive coding
    • H04N19/597—Methods or arrangements for coding, decoding, compressing or decompressing digital video signals using predictive coding specially adapted for multi-view video sequence encoding
    • H—ELECTRICITY
    • H04—ELECTRIC COMMUNICATION TECHNIQUE
    • H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
    • H04N19/00—Methods or arrangements for coding, decoding, compressing or decompressing digital video signals
    • H04N19/70—Methods or arrangements for coding, decoding, compressing or decompressing digital video signals characterised by syntax aspects related to video coding, e.g. related to compression standards
    • H—ELECTRICITY
    • H04—ELECTRIC COMMUNICATION TECHNIQUE
    • H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
    • H04N21/00—Selective content distribution, e.g. interactive television or video on demand [VOD]
    • H04N21/20—Servers specifically adapted for the distribution of content, e.g. VOD servers; Operations thereof
    • H04N21/23—Processing of content or additional data; Elementary server operations; Server middleware
    • H04N21/234—Processing of video elementary streams, e.g. splicing of video streams or manipulating encoded video stream scene graphs
    • H04N21/23412—Processing of video elementary streams, e.g. splicing of video streams or manipulating encoded video stream scene graphs for generating or manipulating the scene composition of objects, e.g. MPEG-4 objects
    • H—ELECTRICITY
    • H04—ELECTRIC COMMUNICATION TECHNIQUE
    • H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
    • H04N21/00—Selective content distribution, e.g. interactive television or video on demand [VOD]
    • H04N21/20—Servers specifically adapted for the distribution of content, e.g. VOD servers; Operations thereof
    • H04N21/23—Processing of content or additional data; Elementary server operations; Server middleware
    • H04N21/234—Processing of video elementary streams, e.g. splicing of video streams or manipulating encoded video stream scene graphs
    • H04N21/2343—Processing of video elementary streams, e.g. splicing of video streams or manipulating encoded video stream scene graphs involving reformatting operations of video signals for distribution or compliance with end-user requests or end-user device requirements
    • H—ELECTRICITY
    • H04—ELECTRIC COMMUNICATION TECHNIQUE
    • H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
    • H04N21/00—Selective content distribution, e.g. interactive television or video on demand [VOD]
    • H04N21/80—Generation or processing of content or additional data by content creator independently of the distribution process; Content per se
    • H04N21/81—Monomedia components thereof
    • H04N21/816—Monomedia components thereof involving special video data, e.g 3D video
    • H—ELECTRICITY
    • H04—ELECTRIC COMMUNICATION TECHNIQUE
    • H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
    • H04N21/00—Selective content distribution, e.g. interactive television or video on demand [VOD]
    • H04N21/80—Generation or processing of content or additional data by content creator independently of the distribution process; Content per se
    • H04N21/85—Assembly of content; Generation of multimedia applications
    • H04N21/854—Content authoring
    • H04N21/85406—Content authoring involving a specific file format, e.g. MP4 format
    • G—PHYSICS
    • G06—COMPUTING OR CALCULATING; COUNTING
    • G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
    • G06T2210/00—Indexing scheme for image generation or computer graphics
    • G06T2210/61—Scene description

Definitions

  • Some embodiments of the present application relate to the field of video processing technology, and in particular, to a method and device for parsing a scene description file.
  • each tactile material attribute is represented by a texture array, and texture arrays of different attributes are combined to form a specific tactile perception.
  • Each array element contains a tactile texture and a tactile material attribute type represented as an enumeration, and the tactile material attribute types include: high resolution (High_resolution), low resolution (Low_resolution), reference (Reference), and other (Other).
  • the tactile material attribute type is high resolution or low resolution
  • the texture map or tactile texture material data packet of the texture index stores not color values, but tactile values. These tactile values can be directly acquired by the display engine and tactile rendering is performed.
  • each pixel in the texture map or tactile texture material data packet does not directly store the tactile value, but stores the index value of the tactile media file to be rendered.
  • some embodiments of the present application provide a method for parsing a scene description file, including:
  • the target node description module is a node description module corresponding to a node including the target three-dimensional mesh
  • the target media reference array is a media reference array corresponding to a target tactile media file to which the first tactile material attribute belongs;
  • the description information of the haptic media accessor for accessing the decoded data of the target haptic media file is obtained according to the index value declared by the accessor index syntax element.
  • some embodiments of the present application provide a scene description file parsing device, including:
  • a memory configured to store a computer program
  • the processor is configured to, when calling a computer program, enable the three-dimensional scene rendering device to implement the scene description file parsing method described in the first aspect.
  • FIG1 is a schematic diagram showing the structure of an immersive media description framework in some embodiments of the present application.
  • FIG2 is a schematic diagram showing the structure of a scene description file in some embodiments of the present application.
  • FIG3 is a schematic diagram showing the structure of a scene description file in some other embodiments of the present application.
  • FIG4 is a schematic diagram showing a data model of tactile data in some other embodiments of the present application.
  • FIG5 is a schematic diagram showing a layered structure of a human body in some other embodiments of the present application.
  • FIG6 is a schematic diagram showing the structure of a tactile signal encoder in some other embodiments of the present application.
  • FIG7 is a schematic diagram showing the structure of a tactile signal decoder in other embodiments of the present application.
  • FIG8 is a schematic diagram showing the structure of a scene description file in some other embodiments of the present application.
  • FIG9 is a schematic diagram showing the structure of a scene description file in some other embodiments of the present application.
  • FIG10 is a flowchart showing the steps of a method for generating a scene description file in some embodiments of the present application.
  • FIG11 is a flowchart showing the steps of a scene description file parsing method in some embodiments of the present application.
  • FIG12 is a flowchart showing the steps of a scene description file parsing method in some other embodiments of the present application.
  • FIG13 is a flowchart showing a method for processing a tactile media file in some embodiments of the present application.
  • FIG14 is a flowchart showing the steps of a three-dimensional scene rendering method in some embodiments of the present application.
  • FIG15 is a flowchart showing the steps of a cache management method in some embodiments of the present application.
  • FIG. 16 shows an interactive flow chart of a three-dimensional scene rendering method in some embodiments of the present application.
  • Some embodiments of the present application involve scene description of immersive media.
  • the scene description of immersive media decouples the access and processing of media files from the rendering of media files, and designs a media access function (Media Access Function, MAF) 12 to be responsible for the access and processing of media files.
  • MAF Media Access Function
  • a media access function application programming interface is designed. (Application Programming Interface, API), the display engine 11 and the media access function 12 exchange commands through the media access function API.
  • the display engine 11 can issue commands to the media access function 12 through the media access function API, and the media access function 12 can also request commands from the display engine 11 through the media access function API.
  • the general workflow of the scene description framework of immersive media includes: 1) The display engine 11 obtains the scene description file (Scene Description Documents) provided by the immersive media service provider. 2) The display engine 11 parses the scene description file, obtains the access address of the media file, the attribute information of the media file (media type and encoding and decoding parameters, etc.) and the format requirements of the processed media file and other parameters or information, and calls the media access function API to pass all or part of the information obtained by parsing the scene description file to the media access function 12.
  • the display engine 11 obtains the scene description file (Scene Description Documents) provided by the immersive media service provider. 2) The display engine 11 parses the scene description file, obtains the access address of the media file, the attribute information of the media file (media type and encoding and decoding parameters, etc.) and the format requirements of the processed media file and other parameters or information, and calls the media access function API to pass all or part of the information obtained by parsing the scene description file to the media access function 12.
  • the media access function 12 requests to download the specified media file from the media resource server or obtains the specified media file from the local, and establishes a corresponding pipeline for the media file, and then decapsulates, decrypts, decodes and other processes on the media file in the pipeline to convert the media file from the encapsulation format to the format specified by the display engine 11. 4)
  • the pipeline stores the output data obtained after all processing in the specified cache. 5)
  • the display engine 11 reads the fully processed data in the specified cache and renders the media file according to the data read from the cache.
  • the scene description file is used to describe the structure of the three-dimensional scene (its features can be described by a three-dimensional mesh), texture (such as texture mapping, etc.), animation (rotation, translation), camera viewpoint position (rendering perspective), and other contents.
  • GL Transmission Format 2.0 (glTF2.0) has been identified as a candidate format for a scene description file that can meet the requirements of MPEG-Immersive (MPEG-I) and 6-degrees of freedom (6DoF) applications.
  • MPEG-I MPEG-Immersive
  • 6DoF 6-degrees of freedom
  • glTF2.0 is described in the GL Transmission Format (glTF) version 2.0 of the Khronos Group available at github.com/KhronosGroup/glTF/tree/master/specification/2.0#specifying-extensions.
  • FIG. 2 is a schematic diagram of the structure of a scene description file in the glTF2.0 scene description standard (ISO/IEC 12113).
  • the scene description file in the glTF2.0 scene description standard includes but is not limited to: scene description module (scene) 201, node description module (node) 202, mesh description module (mesh) 203, accessor description module (accessor) 204, cache slice description module (bufferView) 205, buffer description module (buffer) 206, camera description module (camera) 207, lighting description module (light) 208, material description module (material) 209, texture description module (texture) 210, sampler description module (sampler) 211 and texture map description module (image) 212, animation description module (animation) 213, skin description module (skin) 214.
  • the scene description module (scene) 201 in the scene description file shown in FIG2 is used to describe the three-dimensional scene contained in the scene description file.
  • a scene description file may contain any number of three-dimensional scenes, and each three-dimensional scene is represented by a scene description module 201.
  • the scene description modules 201 are in parallel with each other, that is, the three-dimensional scenes are in parallel with each other.
  • the node description module (node) 202 in the scene description file shown in FIG2 is a description module at the next level of the scene description module 201, and is used to describe the objects contained in the three-dimensional scene described by the scene description module 201. There may be many specific objects in each three-dimensional scene, such as virtual digital people, three-dimensional objects in the near distance, and background images in the far distance. The scene description file will describe these specific objects through the node description module 201.
  • Each node description module 202 can represent An object or a group of objects consisting of several objects. The relationship between the node description modules 202 reflects the relationship between the various components in the three-dimensional scene described by the scene description module 201.
  • a scene described by a scene description module 201 can contain one or more nodes.
  • a parallel relationship or a hierarchical relationship between multiple nodes can be a parallel relationship or a hierarchical relationship between multiple nodes, that is, there is a relationship of inclusion and being included between the node description modules 202, which allows multiple specific objects to be described together, or multiple specific objects can be described separately. If a node is included by another node, the included node is called a child node (children), and the child node is represented by "children" instead of "node”. By flexibly using nodes and child nodes to combine, a hierarchical node structure can be formed to express rich scene content.
  • the mesh description module (mesh) 203 in the scene description file shown in FIG2 is a next-level description module of the node description module 202, and is used to describe the characteristics of the object represented by the node description module 202.
  • the mesh description module 203 is a set of one or more primitives, each of which may include an attribute, and the attribute of the primitive defines the attribute required for the graphics processing unit (GPU) to render.
  • the attributes may include: position (three-dimensional coordinates), normal (normal vector), tangent (tangent vector), texcoord_n (texture coordinates), color_n (color: RGB or RGBA), joints_n (attributes related to the skin description module 210), and weights_n (attributes related to the skin description module 210), etc.
  • the mesh description module 203 Since the number of vertices contained in the mesh description module 203 is very large, and each vertex contains multiple attribute information, it is inconvenient to directly store a large amount of media data contained in the media file in the mesh description module 203 of the scene description file. Instead, the access address (URI) of the media file is pointed out in the scene description file, and the data in the media file can be downloaded when it is needed, thereby realizing the separation of the scene description file and the media file. Therefore, in general, the mesh description module 203 does not store media data, but stores the index value of the accessor description module (accessor) 204 corresponding to each attribute, and points to the corresponding data in the cache slice (bufferView) of the buffer (buffer) through the accessor description module 204.
  • accessor accessor
  • the scene description file and the media file may be merged to form a binary file, thereby reducing the types and number of files.
  • mode there may be a syntax element "mode” in the primitives of the mesh description module 203.
  • the value of "position” is 1, which points to the accessor description module with index 1, and finally points to the vertex coordinate data stored in the buffer;
  • the value of "color_0" is 2, which points to the accessor description module with index 2. The module is described above and ultimately points to the color data stored in the buffer.
  • the buffer description module (buffer) 206 is used to implement the layer-by-layer refined indexing of the data of the media file by the grid description module 203.
  • the grid description module 203 does not store specific media data, but stores the index value of the corresponding accessor description module 204, and accesses the specific media data through the accessor description module 204 indexed by the index value.
  • the indexing process of the media data by the grid description module 203 includes: first, the index value declared by the syntax element in the attribute of the primitive of the grid description module 203 will point to the corresponding accessor description module 204; then, the accessor description module 204 will point to the corresponding cache slice description module 205; finally, the cache slice description module 205 will point to the corresponding buffer description module 206.
  • the buffer description module 206 in the scene description file shown in FIG. 2 is mainly responsible for pointing to the corresponding media file, including the URI of the media file, the byte length of the media file, etc., and is used to describe the buffer of the media data of the cached media file.
  • a cache can be divided into multiple cache slices.
  • the cache slice description module 205 is mainly responsible for partial access to the media data in the cache, including the starting byte offset of the accessed data and the byte length of the accessed data. Through the cache slice description module 205 and the cache description module 206, partial access to the data of the media file can be achieved.
  • the accessor description module 204 is mainly responsible for adding additional information to the partial data defined in the cache slice description module 205, such as the data type, the number of data of this type, the numerical range of data of this type, etc. Such a three-layer structure can realize the function of retrieving partial data from a media file, which is conducive to the accurate retrieval of data and also convenient for reducing the number of media files.
  • the camera description module (camera) 207 in the scene description file shown in FIG2 is a next-level description module of the node description module 202, and is used to describe the viewpoint, viewing angle, and other information related to visual viewing when the user views the object described by the node description module 202.
  • the node description module 202 can also point to the camera description module 207, and the camera description module 207 describes the viewpoint, viewing angle, and other information related to visual viewing when the user views the object described by the node description module 202.
  • the light description module (light) 208 in the scene description file shown in FIG. 2 is a next-level description module of the node description module 202 , and is used to describe light intensity, ambient light color, light direction, light source position and other light-related information of the object described by the node description module 202 .
  • the material description module (material) 209 in the scene description file shown in FIG2 is a description module of the next level of the mesh description module 203, and is used to describe the material information of the three-dimensional object described by the mesh description module 203.
  • this process can also be referred to as texture mapping or adding textures.
  • the scene description file in the glTF2.0 scene description standard also uses this description module.
  • the material description module 209 uses a set of general parameters to define the material to describe the material information of the geometric objects appearing in the three-dimensional scene.
  • the material description module 209 generally uses the metal-roughness model to describe the material of the virtual object, and the material characteristic parameters based on the metal-roughness model are represented by the widely used physically based rendering (PBR) material. Based on this, the material description module 209 makes a detailed description of the metal-roughness material attribute of the object.
  • PBR physically based rendering
  • the syntax elements in PbrMetarialRoughness (material.PbrMetarialRoughness) in the material description module 209 are defined as shown in Table 5 below:
  • each attribute in PbrMetarialRoughness of the material description module 209 can be defined using factors and/or textures (e.g., baseColorTexture and baseColorFactor). If no texture is given, then all corresponding texture components in this material model must be assumed to have a value of 1.0. If both factors and textures are present, the factor value acts as a linear multiplier for the corresponding texture value. Texture bindings are defined by the index of a texture object and an optional texture coordinate index.
  • factors and/or textures e.g., baseColorTexture and baseColorFactor
  • the material description module 209 By parsing the material description module 209, it is possible to determine that the current material is named “gold” through the material name syntax element and its value ("name”:”gold"), and then determine that the base color value of the current material is [1.000,0.766,0.336,1.0] through the color syntax element under the pbrMetallicRoughness array and its value ("basecolorFactor”:[1.000,0.766,0.336,1.0]), and that the metallic value of the current material is "1.0” through the metallic syntax element under the pbrMetallicRoughness array and its value (“metalnessFactor”:1.0), and that the roughness value of the current material is "0.0” through the roughness syntax element under the pbrMetallicRoughness array and its value (“roughnessFactor”:0.0).
  • the texture description module (texture) 210 in the scene description file shown in FIG2 is a description module at the next level of the material description module 209, and is used to describe the color of the three-dimensional object described by the material description module 209 and other characteristics used in the material definition. Texture is an important aspect of giving an object a realistic appearance.
  • the main color and material of the object can be defined through texture. Other features used in the definition to accurately describe the appearance of the rendered object.
  • the material itself can define multiple texture objects, which can be used as textures for virtual objects during rendering and can be used to encode different material properties.
  • the texture description module 210 uses the sampler syntax element and the texture map syntax element index to reference a sampler description module (sampler) 211 and a texture map description module (image) 212.
  • the texture map description module 212 contains a uniform resource identifier (URI) that links to the texture map or binary file package actually used by the texture description module 210.
  • the sampler description module 211 is used to describe the filtering and packaging mode of the texture.
  • the respective responsibilities and collaborative relationships of the material description module 209, the texture description module 210, the sampler description module 211 and the texture map description module 212 include: the material description module 209 and the texture description module 210 together define the color and physical information of the object surface.
  • the sampler description module 211 defines how to apply the texture map to the object surface.
  • the texture description module 210 specifies the sampler description module 212 and the texture map description module 212.
  • the texture is added through the texture map description module 212, and the texture map description module 212 uses URI for identification and indexing, and uses the accessor description module 204 to access data.
  • the sampler description module 211 implements the specific adjustment and packaging of the texture.
  • the definition of the syntax elements in the texture description module 210 is shown in Table 6 below:
  • the definition of the syntax elements in sample (texture.sample) of the texture description module 210 is shown in Table 7 below:
  • a material description module 209 For example, the following is a JSON example of a material description module 209, a texture description module 210, a sampler description module 211, and a texture map description module 212:
  • the animation description module (animation) 213 in the scene description file shown in FIG. 2 is the next layer of the node description module 202.
  • the node description module 202 is a level description module, which is used to describe the animation information added to the object described by the node description module 202.
  • animation can be added to the object described by the node description module 202. Therefore, the description level of the animation description module 213 in the scene description file is specified by the node description module 202, that is, the animation description module 213 is the next level description module of the node description module 202, and the animation description module 213 also has a corresponding relationship with the grid description module 203.
  • the animation description module 213 can describe the animation in three ways: position movement, angle rotation, and size scaling, and can also specify the start and end time of the animation and the implementation method of the animation. For example, if an animation is added to a grid description module 203 representing a three-dimensional object, the three-dimensional object represented by the grid description module 203 can complete the specified animation process within the specified time window through the fusion of position movement, angle rotation, and size scaling.
  • the skin description module (skin) 214 in the scene description file shown in FIG2 is a description module of the next level of the node description module 202, and is used to describe the motion cooperation relationship between the skeleton added by the mesh description module 203 and the mesh representing the surface information of the object.
  • the mesh description module 203 pointed to by the node description module 202 represents an object with a large degree of freedom of movement such as a person, an animal, or a machine, in order to optimize the motion performance of these objects, the skeleton can be filled into the interior of the object, and the mesh description module 203 representing the surface information of the object becomes a skin in concept at this time.
  • the description level of the skin description module 214 is specified by the node description module 202, that is, the skin description module 214 is a description module of the next level of the node description module 202, and the skin description module 214 has a corresponding relationship with the mesh description module 203.
  • Each description module of the scene description file in the above glTF2.0 scene description standard only has the most basic ability to describe three-dimensional objects. There are problems such as not supporting dynamic three-dimensional immersive media, not supporting audio files, and not supporting scene updates.
  • glTF also declares that each of its object attributes has an optional extended object attribute (extensions), allowing any part of it to be extended with extensions to achieve more complete functions. Including scene description module (scene), node description module (node), mesh description module (mesh), accessor description module (accessor), cache description module (buffer), animation description module (animation), etc. and their internally defined syntax elements all have optional extended object attributes to support certain functional extensions based on glTF2.0.
  • the mainstream immersive media mainly include point cloud, 3D mesh, 6DoF panoramic video, MIV-based, etc.
  • multiple types of immersive media often exist at the same time.
  • Different types of rendering engines are generated according to the types and numbers of supported codecs.
  • the rendering engines designed by different vendors support different types of media.
  • the Moving Picture Experts Group (MPEG) initiated the formulation of the MPEG scene description standard, with the standard number ISO/IEC 23090-14. This standard mainly solves the cross-platform description problem of MPEG media (including codecs formulated by MPEG, MPEG file formats, and MPEG transmission mechanisms) in 3D scenes.
  • the MPEG#128 meeting decided to develop the MPEG-I Scene Description standard based on glTF2.0 (ISO/IEC 12113).
  • the first version of the MPEG scene description standard has been developed and is in the FDIS voting stage.
  • the MPEG scene description standard adds corresponding extensions to address the unmet needs in the cross-platform description of 3D scenes, including interactivity, AR anchoring, user and avatar representation, tactile support, and extended support for immersive media. Codec support, etc.
  • the first version of the MPEG scene description standard has been formulated mainly to define the following contents:
  • the MPEG scene description standard defines a scene description file format for describing immersive 3D scenes. This format combines the original glTF2.0 (ISO/IEC 12113) content and makes a series of extensions based on it.
  • MPEG scene description defines a scene description framework and an application program interface (API) for collaboration between modules, which decouples the acquisition and processing of immersive media from the media rendering process, and is beneficial to the optimization of immersive media adaptation to different network conditions, partial acquisition of immersive media files, access to different levels of detail of immersive media, and content quality adjustment. Decoupling the acquisition and processing of immersive media from the immersive media rendering process is the key to achieving cross-platform description of 3D scenes.
  • API application program interface
  • the extensions of the scene description file in the MPEG scene description standard can be divided into two groups:
  • the first group of extensions includes: MPEG media (MPEG_media) 301, MPEG time-varying accessor (MPEG_accessor_timed) 302, and MPEG circular buffer (MPEG_buffer_circular) 303.
  • MPEG media 301 is an independent extension used to reference external media sources
  • MPEG time-varying accessor 302 is an extension of the accessor level, used to access time-varying media
  • MPEG circular buffer is an extension of the buffer level to support circular buffers.
  • the first group of extensions provides a basic description and format of the media in the scene, meeting the basic requirements of describing time-varying immersive media in the scene description file.
  • the second group of extensions includes: MPEG dynamic scene (MPEG_scene_dynamic) 304, MPEG texture (MPEG_texture_video) 305, MPEG audio space (MPEG_audio_spatial) 306, MPEG viewport recommendation (MPEG_viewport_recommended) 307, MPEG mesh mapping (MPEG_mesh_linking) 308 and MPEG animation time (MPEG_animation_timing) 309.
  • MPEG dynamic scene MPEG_scene_dynamic
  • MPEG texture MPEG_texture_video
  • MPEG audio space MPEG_audio_spatial
  • MPEG viewport recommendation MPEG_viewport_recommended
  • MPEG mesh mapping MPEG_mesh_linking
  • MPEG animation time MPEG_animation_timing
  • MPEG_scene_dynamic 304 is a scene level extension to support dynamic scene updates
  • MPEG_texture_video 305 is a texture level extension to support textures in video form
  • MPEG_audio_spatial 306 is a node level and camera level extension to support spatial 3D audio
  • MPEG_viewport_recommended 307 is a scene level extension to support the description of recommended viewing angles in two-dimensional display
  • MPEG_mesh_linking 308 is a mesh level extension to support linking two meshes and providing mapping information
  • MPEG_animation_timing 309 is a scene level extension to support controlling the animation timeline.
  • the MPEG media in the MPEG scene description file is used to describe the type of media files and to provide necessary instructions for MPEG type media files so that these MPEG type media files can be used later.
  • the definition of the first level syntax elements of MPEG media is shown in Table 8 below:
  • ISO/IEC 23090-14 also defines the transmission format for the delivery of scene description files and data related to the glTF 2.0 extension.
  • ISO/IEC 23090-14 defines how to encapsulate glTF files and related data as non-time-varying and time-varying data (for example, as track samples) in ISOBMFF files.
  • MPEG_scene_dynamic, MPEG_mesh_linking, and MPEG_animation_timing provide a specific form of time-varying data to the display engine, and the display engine 11 should perform corresponding operations based on this changing information.
  • ISO/IEC 23090-14 also defines the format of each extended time-varying data and how to encapsulate it in the ISOBMFF file.
  • MPEF media allows reference to external media streams delivered via protocols such as RTP/SRTP, MPEG-DASH, etc.
  • ISO/IEC 23090-14 defines a new URL scheme that requires the presence of a stream identifier in the query part, but does not specify a specific type of identifier, allowing the use of a Media Stream Identification scheme (RFC5888), a labeling scheme (RFC4575), or a zero-based indexing scheme.
  • Haptics provide users with additional entertainment and sensory immersion. By properly adding haptics to audio/video content, the user's experience and enjoyment of media content can be significantly enhanced.
  • Haptic data representation If there is a standard and unified data model, and the data model includes all the information required by a specified synthesizer to render haptic effects, standard decoding of haptic data can be achieved.
  • FIG. 4 is a schematic diagram of the structure of the tactile data model.
  • the tactile data model includes: a description part 41 and an encoding part 42, and a hierarchical structure is used to describe the tactile data layer by layer.
  • the description part 41 uses the perception and tactile channel information as the main description content in the data structure
  • the encoding part 42 uses the tactile frequency band, tactile effect and key frame as the main description content in the data structure.
  • the highest layer of the description part 41 contains an overall description of the content contained in the entire data structure, which is usually used to describe the entire tactile experience defined in the file or media stream.
  • the tactile experience contains some high-level metadata, and a pre-defined avatar list (i.e., a representation of a body part) is provided to facilitate the subsequent specification of the implementation position of the tactile stimulus on the body.
  • the tactile data is described by a perception list (perception 1 to perception a).
  • Each tactile perception in the tactile perception list corresponds to a tactile signal related to a specific perception modality (such as vibration, force, position, speed, temperature, etc.).
  • Each perception in the perception list contains some metadata information such as the characteristics of the tactile implementation device and the avatar related to the perception.
  • the perception signal itself is a mixture of multiple tactile channels (tactile channel 1 to tactile channel b).
  • Each tactile channel includes not only metadata about the channel configuration (such as mixing gain, etc.), but also a list of tactile frequency bands (tactile frequency band 1 to tactile frequency band c) where the data is decomposed by frequency band.
  • the frequency band data of each tactile frequency band is a partial signal within a given frequency range.
  • the tactile frequency band is described using frequency band data and a list of tactile effects (tactile effect 1 to tactile effect d).
  • Each tactile effect in the tactile effect list contains a list of key frames (key frame 1 to key frame e). Different frequency bands contain different signals, and the tactile signal in the channel can be reconstructed by combining data from different frequency bands.
  • the hierarchical structure of the human body includes: the first level 51 includes all parts of the human body; the second level 52 includes: the upper body and the lower body; the third level 53 includes: the arms, chest, head of the upper body and the legs and waist of the lower body; the fourth level 54 includes: the upper arms, forearms and hands of the arms, the neck and nape of the head and the thighs, calves and feet of the legs; the fifth level 55 includes: the palms and fingers of the hands, the soles and toes of the feet; the sixth level 56 includes: the thumb, index finger, middle finger, ring finger and little finger of the fingers, the big toe, index toe, middle toe, ring toe and little toe of the feet, and the seventh level 57 includes: the proximal phalanx of the thumb, the second knuckle and the
  • Standard tactile coding formats will help incorporate tactile into ISOBMFF, MPEG-DASH and MPEG-I standards, making it easier for content creators and media/streaming content providers to integrate tactile in a standardized way and improve the overall user experience.
  • the codec architecture proposed by Working Group 7 of the ISO/IEC Moving Picture Experts Group (MPEG) in developing the ISO/IEC 23090-31 tactile codec standard can process both pulse code modulation (PCM) files (WAV files/.wav files) and descriptive tactile files (AHAP files/.ahap files or IVS files/.ivs files or HJIF files/.hjif files). Metadata information is provided to the codec in the format of OHM files.
  • the tactile coding representation allows the encoding of descriptive and quantitative data in a human-readable JSON format. The encoding process is shown in Figure 6.
  • FIG. 6 is a schematic diagram of a coding process for encoding a tactile signal.
  • the encoder 600 processes PCM tactile files and descriptive tactile files in different ways. For descriptive tactile files, the tactile signal encoder 600 first identifies the file format of the tactile file. If the tactile file to be encoded is a .hjif file, there is no need for transcoding, it can be directly compressed into a binary format and finally packaged into a MIHS stream. If the tactile file to be encoded is a .ahap file or a .ivs file, transcoding is required. The tactile signal encoder 600 first semantically analyzes the tactile file to be encoded and transcodes it to format it into the data model shown in Figure 4.
  • the data can be exported to an exchange file format (.hjif) encoded in a JSON readable form, a distribution format (.hmpg) encoded as a compressed binary file, or a stream format (MIHS) that defines a packetized bitstream.
  • exchange file format .hjif
  • distribution format .hmpg
  • MIHS stream format
  • the signal analysis process of the tactile signal encoder 600 is divided into two sub-processes. One is: after performing frequency band decomposition on the original signal, the low-frequency signal obtained by frequency band decomposition is encoded based on key frame extraction.
  • the second is to reconstruct the low-frequency signal, calculate the residual between the reconstructed low-frequency signal and the original low-frequency signal, obtain the residual signal, and then add the residual signal to the original high-frequency signal and encode it using wavelet transform, and output the encoding result in at least one of the formats of .hjif, .hmpg or MIHS.
  • FIG. 7 is a schematic diagram of a decoding process for decoding a tactile signal.
  • the tactile signal decoder 700 can either input a stream format (MIHS file) packaged into a data packet, decompress it after unpacking to obtain a .hjif file, or select a .hmpg file as input and output a .hjif file after decompression. Then the tactile data contained in the .hjif file can be presented directly on the tactile device, or an intermediate synthesizer that generates PCM data can be used to output a .wav file for rendering on the tactile device.
  • MIHS file stream format
  • the .hjif format is a human-readable format based on JSON and is not optimized for memory usage. It can be easily parsed and manually edited, which facilitates the design and creation of content, making it an ideal exchange format.
  • the .hjif data can be compressed into a more memory-efficient binary .hmpg bitstream. This compression is lossy, and different parameters affect the encoding depth of the amplitude and frequency that constitute the bitstream.
  • the data can be compressed and packaged into an MPEG-I tactile stream (MIHS). This approach can meet the expectations of related technologies for descriptive and quantitative format compatibility of tactile media files, as well as interoperability between tactile terminal devices to achieve 3D immersive experience and distribution purposes. MIHS tactile stream is also the basis for encapsulating tactile media files into ISOBMFF.
  • MPEG_haptic MPEG haptics
  • MPEG_material_haptic MPEG haptic materials
  • MPEG_media what is indexed from the MPEG media (MPEG_media) is the haptic media that has completed haptic encoding and packaging.
  • MPEG haptic material not only stores the haptic material attribute values, but also supports direct indexing of haptic media files.
  • MPEG haptic 801 is a next-level description module of the node description module, used to support indexing of haptic-related media files in MPEG_media to add haptic information.
  • the definition of syntax elements in MPEG haptic 801 is shown in Table 13 below:
  • the MPEG haptic material (MPEG_material_haptic) 402 is a next-level description module of the mesh description module, and is used to support the user to obtain corresponding haptic feedback through active and/or passive interaction in the virtual scene.
  • Texture mapping refers to the technology of efficiently and realistically displaying the details of a three-dimensional model without the need for a high-resolution mesh. This method was originally developed in computer graphics and is now widely used in tactile rendering. The existence of texture mapping enables virtual objects to enhance the additional tactile material properties distributed on their surfaces and enhance the realism of virtual objects. On the basis of texture mapping, related technologies further propose the concept of tactile materials. In computer graphics, a material is an easy-to-use data package that contains all the data required for the visual rendering of a virtual object. By analogy, tactile materials should provide all the necessary elements for tactile rendering, and different tactile materials contain different tactile characteristics. These different tactile material properties are stored in different tactile material packages, which facilitates the related tasks of tactile design and implementation. The definition of tactile material properties is shown in Table 14 below:
  • Each tactile material attribute in Table 14 above is represented by a texture array, and texture arrays of different attributes are combined to form a specific tactile perception.
  • Tactile material attributes can be used as traditional two-dimensional textures to provide relevant tactile texture values, or as tactile element maps.
  • Each pixel in the tactile element map contains multiple tactile unit elements, which are associated with an index in the media reference array (Media_reference) corresponding to MPEG tactile (MPEG_haptic).
  • Media_reference media reference array
  • MPEG_haptic MPEG tactile
  • the rendering engine can select the most appropriate texture array for rendering. In this way, each array element contains a tactile texture and a texture type represented as an enumeration.
  • the enumerated tactile material texture types include:
  • the tactile texture is a high-resolution 2D texture that directly stores tactile values.
  • the tactile texture is a low-resolution 2D texture that directly stores tactile values.
  • a haptic texture is a two-dimensional map of haptic elements containing references to haptic signals. Each pixel of the haptic texture corresponds to an index in the media reference array (Media_reference) of MPEG_haptic.
  • each texture map pixel in the high-resolution vibration texture is divided into two: the first byte is used to describe the amplitude value, and the second byte is used to describe the frequency value.
  • MPEG_material_haptic 402 The definition of the syntax elements in the MPEG haptic material (MPEG_material_haptic) 402 is shown in Table 17 below:
  • Table 17 above shows the semantics of the syntax elements included in the MPEG haptic material (MPEG_material_haptic) 402, and the enumerated values in Table 17 are the haptic material attributes included in Table 14.
  • the MPEG haptic material (MPEG_material_haptic) 402 in the MPEG scene description file can associate objects in the scene with haptic materials, so that when the user interacts with the object, the corresponding haptic perception can be obtained.
  • Each haptic material attribute contains different haptic features, but the internal storage method is consistent.
  • the following Table 18 takes the stiffness in the haptic material attribute as an example to further explain the stiffness enumerated in the MPEG haptic material (MPEG_material_haptic.stiffness).
  • the main functions of the display engine 11 include obtaining a scene description file, parsing the obtained scene description file, obtaining the composition structure of the three-dimensional scene to be rendered and the detailed information in the three-dimensional scene to be rendered, and rendering and displaying the three-dimensional scene to be rendered according to the information obtained by parsing the scene description file.
  • the specific workflow and principle of the display engine 11 are not limited in the embodiment of the present application, and the display engine 11 is capable of parsing the scene description document, issuing instructions to the media access function 12 through the media access function API, issuing instructions to the cache management module 13 through the cache API, and taking the processed data from the cache and completing the rendering and display of the three-dimensional scene and the objects therein.
  • the media access function 12 can receive instructions from the display engine 11, and complete the access and processing functions of the media files according to the instructions sent by the display engine 11. Specifically, it includes: after obtaining the media file, the media file is processed. There are large differences in the processing process of different types of media files. In order to achieve a wide range of media type support and considering the work efficiency of the media access function, a variety of pipelines are designed in the media access function. During the processing process, only the pipeline that matches the media type is enabled.
  • the input of the pipeline is the media files downloaded from the server or the media files read from the local storage control. These media files often have a more complex structure and cannot be directly used by the display engine 11. Therefore, the main function of the pipeline is to process the data of such media files so that the data of the media files meets the requirements of the display engine 11.
  • the media data processed by the pipeline needs to be delivered to the display engine 11 for use in a standardized arrangement structure, which requires the participation of the cache API and the cache management module 13.
  • the cache API and cache management realize the creation of corresponding caches according to the format of the processed media data, and are responsible for the subsequent management of the cache, such as update, release and other operations.
  • the cache management module 13 can communicate with the media access function 12 through the cache API, and can also communicate with the display engine 11. The goal of communicating with the display engine 11 and/or the media access function 12 is to achieve cache management.
  • the display engine 11 needs to send the relevant instructions of cache management to the media access function 12 through the media access function API first, and the media access function 12 then sends the relevant instructions of cache management to the cache management module 13 through the cache API.
  • the display engine 11 only needs to send the cache management description information parsed from the scene description document directly to the cache management module 13 through the cache API.
  • the current MPEG#140 meeting has confirmed the proposal to add MPEG haptic (MPEG_haptic) 801 and MPEG haptic material (MPEG_material_haptic) 802, and allows MPEG haptic 801 to index into haptic media files in MPEG media (MPEG_media), which mostly describe the overall haptic experience related to an event or environment.
  • MPEG_media MPEG media
  • the indexed haptic media files are regarded as haptic signals, which can essentially directly drive the haptic device to generate corresponding haptic feedback to the human body.
  • each array element includes a tactile texture and a tactile material attribute type represented as an enumeration
  • the tactile material attribute type includes: high resolution (High_resolution), low resolution (Low_resolution), reference (Reference) and other (Other).
  • the texture map or tactile texture material data packet of the texture index stores not color values, but tactile values.
  • the tactile value only represents the size of a specific attribute (such as the size of the friction coefficient, the size of the stiffness coefficient), and can be directly obtained by the display engine and tactile rendering is performed.
  • each pixel in the texture map or tactile texture material data packet does not directly store the tactile value, but stores the index value of the tactile media file to be rendered. Therefore, when the tactile material attribute type is reference, only the encoded and encapsulated tactile media file can be obtained according to the index value, and the decoded tactile value that can be used for rendering cannot be obtained, which causes the display engine to fail to render normally. That is, the related art cannot obtain the tactile value of the tactile material attribute of the reference type.
  • Some embodiments of the present application provide a scene description framework that supports obtaining tactile values of tactile material attributes of a reference type, and specific contents include: support of scene description files for obtaining tactile values of tactile material attributes of a reference type, support of media access function APIs for obtaining tactile values of tactile material attributes of a reference type, support of media access functions for obtaining tactile values of tactile material attributes of a reference type, support of cache APIs for obtaining tactile values of tactile material attributes of a reference type, support of cache management for obtaining tactile values of tactile material attributes of a reference type, and other contents.
  • the process of indexing, rendering and displaying tactile media files in a three-dimensional scene based on a scene description framework includes: first, the display engine obtains a scene description file from the three-dimensional scene provider by downloading it through a network protocol or reading it from local storage.
  • the scene description file obtained by the display engine includes description information of the entire three-dimensional scene and the tactile media files in the scene.
  • the description information of the tactile media files includes the access address of the tactile media files, the attribute information of the access to the tactile media files (media type, the encoding and decoding parameters used, etc.). It also includes the description information of the tactile media files.
  • the format requirements of the data obtained by file decoding processing include the format requirements of the description part and the encoding part of the tactile data obtained after the tactile media file processing (i.e., the description part 41 and the encoding part 42 in FIG. 4, collectively referred to as the tactile media file in the following text) and other information data (data type, data length, data access method, etc.) and the format requirements and addresses of other media data, including the data such as the texture map and the description information of the tactile material attributes to be used later.
  • the display engine After the display engine completes the parsing of the scene description file, it calls the media access function API to pass the access address of the tactile media file, the attribute information of the tactile media file (media type, the codec used, etc.) and other information to the media access function.
  • the display engine also passes the format requirements of the decoded tactile media data and other media data specified in the parsed scene description file to the media access function by calling the media access function API. Then, the media access function requests the server to download the specified tactile media file or obtain the specified tactile media file from the local according to the access address of the tactile media file transmitted by the display engine. After that, the media access function establishes a corresponding pipeline for the media file according to information such as the media type and encoding and decoding parameters to convert the acquired media data from the encapsulation format to the format required in the scene description file, that is, the tactile media file should be converted in the pipeline into the format specified in the scene description file (data type, data length and access method, etc.).
  • the media file can be decapsulated, decrypted, decoded and post-processed in the pipeline.
  • the pipeline stores the information data of the decoded tactile media and other media data processed into the format specified in the scene description file into the specified cache.
  • the cache can be created by the display engine through the cache API calling the cache management module. If the display engine does not create the corresponding cache, the media access function can create the corresponding cache through the cache API calling the cache management module. The purpose is to achieve the creation, update, release, etc. of the cache.
  • the display engine reads the tactile media file in the specified cache in the format specified by the scene description file to complete the rendering of the tactile part in the three-dimensional scene.
  • the following describes the scene description file, media access function API, media access function, cache API, and cache management that support obtaining the tactile value of the tactile material attribute of the reference type.
  • Fig. 9 is a schematic diagram of the structure of a scene description file supporting the acquisition of tactile values of tactile material attributes of a reference type provided by an embodiment of the present application.
  • the scene description file supporting the acquisition of tactile values of tactile material attributes of a reference type includes but is not limited to the following modules: MPEG media (MPEG_media) 2601, scene description module (scene) 2602, node description module (node) 2603, mesh description module (mesh) 2604, accessor description module (accessor) 2605, buffer slice description module (bufferView) 2606, buffer description module (buffer) 2607, material description module 2608, MPEG tactile material (MPEG_material_haptic) 2609, texture description module (texture) 2610, MPEG tactile (MPEG_haptic) 2611, camera description module (camera) 2612, skin description module (skin) 2613 and animation description module (animation) 2614.
  • MPEG media MPEG_media
  • scene scene
  • node description module node description module
  • mesh description module mesh
  • some embodiments of the present application extend the values of the syntax elements in the MPEG media (MPEG_media) 2601 of the scene description file, and the specific extension includes at least one of the following:
  • Extension 1 Extend the value of the media type syntax element (MPEG_media.media.alternatives.mimeType) used to describe the media resource type of the media in the optional items (MPEG_media.media.alternatives) of the media list of the MPEG media in the scene description file.
  • the extension of the media type syntax element (mimeType) includes: adding a value associated with the tactile media file to the media type syntax element (mimeType). For example: in When the media list of the MPEG media in the scene description file includes tactile media, the value of the media type syntax element (MPEG_media.media.alternatives.mimeType) used to describe the media resource type of the media complies with the relevant provisions of the ISO/IEC 23090-32 standard.
  • the value of "mimeType” is "video/mp4"; when the media resource is a media resource including only audio and tactile, the value of "mimeType” is "video/mp4"; when the media resource is a media resource including only tactile, the value of "mimeType” is "haptics/mp4".
  • Extension 2 The value of the track syntax element (MPEG_media.media.alternatives.tracks.track) used to declare the track information of the media file in the optional track array (MPEG_media.media.alternatives.tracks) of the media list of the MPEG media in the scene description file is extended.
  • the extension of the track syntax element (track) includes: when the encoded tactile media file is referenced by the scene description file as an optional track (MPEG_media.media.alternatives.tracks) in the media list of the MPEG media, the reference track selection method in the ISO/IEC23090-32 standard should be followed.
  • Extension 3 The value of the codec parameter syntax element (MPEG_media.media.alternatives.tracks.codecs) used to describe the codec type of the media data contained in the codestream track in the optional track array (MPEG_media.media.alternatives.tracks) of the media list of the MPEG media in the scene description file is extended. Since the codec parameters of the media files contained in the codestream track are defined in IETF RFC 6381. When the codestream track contains different types of codecs (for example, AdaptationSet contains representations with different codecs), the codec parameter syntax element (codecs) can be represented by a comma-separated list of codec values.
  • codec parameter syntax element codecs
  • the extension of the value of the codec parameter syntax element (codecs) includes: when the media data is haptic data and the encapsulation rules follow the 23090-32 standard, the value of the codec parameter syntax element (codecs) should be set in accordance with the provisions of the ISO/IEC 23090-32 Carriage of haptics data standard. For example, when the tactile data is encapsulated in ISMBOFF, the first group of elements of the codec parameter syntax element (codecs) is the code representing the codec in ISO/IEC 23090-31.
  • codecs are in the following form: "codecs”:"mih1.oo", where "mih1" is a data format description of the tactile data encapsulated in ISOBMFF. This value is not unique and allows other sample format names that conform to ISOBMFF encapsulation to be replaced. Where “oo” is the object type indicator value defined on the "Object Type” page of the MP4 registration authority website.
  • some embodiments of the present application extend the values of the syntax elements in the MPEG media in the scene description file, and the specific extension includes extending one or more of the following items as shown in Table 19:
  • the MPEG media in the scene description file can realize the description of the tactile media file.
  • the encapsulation of the tactile media file is based on the data structure shown in Figure 4.
  • the tactile media file not only has an information part, but also allows a metadata configuration part.
  • the embodiment of the present application does not limit the encapsulation method of the tactile media file, and the encoded and encapsulated tactile media file can be obtained according to the access address of the tactile media file in the MPEG media.
  • the description method of the scene description module (scene) 2602 and the node description module (node) 2603 in the scene description file supporting the acquisition of the tactile value of the tactile material attribute of the reference type includes: when the three-dimensional scene includes a tactile media file, the overall structure of the three-dimensional scene and the structural level and position of the tactile media file in the three-dimensional scene are described using the description method of scene and node.
  • MPEG haptic MPEG_haptic
  • the MPEG haptic (MPEG_haptic) 2611 of the node description module (node) of the scene description file shown in FIG9 includes: media reference (Media_reference), and the media reference (Media_reference) includes: haptic media identification syntax element (haptic_id) 26111, the value (haptic_id) of the tactile media identification syntax element 26111, the accessor index syntax element (accessor) 26112, and the value of the accessor index syntax element (accessor).
  • the description method of MPEG haptic (MPEG_haptic) 2611 in a scene description file supporting obtaining tactile values of tactile material attributes of reference type includes: realizing that a texture coordinate point can be indexed to a tactile media file after encoding and encapsulation through a tactile media identification syntax element (haptic_id) and its value in MPEG haptic (MPEG_haptic) 2611, and realizing that a texture coordinate point can be indexed to a set of tactile values in decoded data of a corresponding tactile media file through an accessor index syntax element (accessor) and its value in MPEG haptic (MPEG_haptic) 2611.
  • haptic_id tactile media identification syntax element
  • MPEG_haptic MPEG_haptic
  • haptic_id tactile media identification syntax element
  • accessor index syntax element accessor
  • the relevant supplementary explanation specifically includes: the haptic media identification syntax element (accessor) 26112 and its value in the media reference (Media_reference) of MPEG haptic (MPEG_haptic) 2611 clarify the media description module corresponding to the haptic media file to which the texture coordinate point needs to be indexed, and the accessor index syntax element in the media reference (Media_reference) of MPEG haptic (MPEG_haptic) 2611
  • the element (accessor) 26111 and its value specify the accessor description module corresponding to the accessor that the texture coordinate point can actually access the decoded data of the tactile media file that needs to be indexed.
  • This function corresponds to the description of the buffer description module (buffer) and the buffer slice description module (bufferview) in the scene description file, and together realizes the direct indexing of the decoded data of the tactile media file by the texture coordinate point.
  • each 3D scene is described using an independent scene description module.
  • a 3D scene there may be many specific objects, such as virtual digital people, nearby 3D objects, distant background images, virtual cameras representing the user's viewpoint and perspective, etc.
  • the scene description file will use these specific objects as nodes in the 3D scene.
  • Each node can represent an object or a group of objects composed of several objects.
  • the relationship between nodes reflects the relationship between the various components in the 3D scene.
  • Each scene description file can describe one or more 3D scenes.
  • the 3D scenes can only be in parallel relationship, not hierarchical relationship, that is, there is no relationship of inclusion and being included between scenes.
  • the nodes can be in parallel relationship or hierarchical relationship, that is, there is a relationship of inclusion and being included between nodes, which allows multiple specific objects to be grouped together for representation. If a node is included by another node, the included node is called a child node, and the child node is represented by "children" instead of "node”. By flexibly combining nodes and child nodes, a hierarchical node structure can be formed.
  • a description method of a mesh description module (mesh) 2604 in a scene description file that supports obtaining a tactile value of a tactile material attribute of a reference type includes: for a node representing a three-dimensional object, the scene description file will be described using a data format of a three-dimensional mesh, that is, described using a mesh description module (mesh) 2604.
  • the mesh description module has a variety of specific information, such as three-dimensional coordinates, color, and other information, all of which belong to the syntax elements in the attributes (mesh.primitives.attribute) in the primitives of the mesh description module 2604.
  • the most basic mesh description module includes the three-dimensional coordinates of vertices, the connection relationship between vertices, the color information of vertices, and the three-dimensional coordinates of the required texture values.
  • the texture coordinates (Texcoord) of the three-dimensional coordinate set of the required texture value needs to be enabled so that the tactile value obtained from the texture later is attached to the corresponding three-dimensional coordinates of the three-dimensional object.
  • the texture coordinates (Texcoord) and color data and other information are all attributes (mesh.primitives.attribute) in the primitives of the mesh description module.
  • the tactile media does not need to refer to attributes such as the number of colors for definition at the grid description module level for the time being. It only needs to enable the MPEG tactile material (MPEG_material_haptic) 2609 as a content supplement parallel to the material description module (material) 2604 to support further definition of the tactile material.
  • the description method of the material description module (material) 2608 and the texture description module (texture) 2610 in the scene description file that supports obtaining the tactile value of the tactile material attribute of the reference type includes: describing additional information of the surface of the three-dimensional object through the material description module (material) 2608 and the texture description module (texture) 2610.
  • describing a conventional three-dimensional object only using the mesh description module to describe the geometric information of the three-dimensional object, or monotonically defining the color of the vertices in the three-dimensional mesh, is far from enough to improve the realism of the virtual object, which requires adding more information to the surface of the three-dimensional object.
  • this process can also be referred to as mapping or adding textures.
  • the scene description file that supports obtaining the tactile value of the tactile material attribute of the reference type is based on glTF2.0, and also uses the material description module (material) 2608 and the texture description module (texture) 2610.
  • the collaborative relationship between the material description module (material) 2608 and the texture description module (texture) 2610 includes: the material description module (material) 2608 and the texture description module (texture) 2610 jointly define the color and physical information of the object surface.
  • the texture description module (texture) 2610 specifies the sampler description module (sampler) and the texture map description module (image).
  • the sampler description module (sampler) defines how to map the texture map to the object surface, and realizes the specific adjustment and packaging of the texture.
  • Texture map description module (image) uses ULR to identify and index texture maps.
  • a description method of MPEG tactile material (MPEG_material_haptic) 2609 in a scene description file supporting acquisition of tactile values of tactile material attributes of a reference type includes: further describing the tactile material attributes through MPEG tactile material (MPEG_material_haptic) 2609.
  • MPEG tactile material (MPEG_material_haptic) 2609 includes tactile material attribute description modules corresponding to various tactile material attributes (stiffness, friction, tactile texture, temperature, vibration, and customization), and the tactile material attribute description module includes attribute type syntax elements, and the values of the attribute type syntax elements can be: high resolution (High_resolution), low resolution (Low_resolution), reference (Reference).
  • An object containing tactile feedback in a three-dimensional scene may include different types of tactile material attributes. Different tactile material attributes on the same object can obtain tactile values or tactile media that are felt by the user in different ways, so as to achieve associating tactile media in the scene description file to obtain tactile feedback.
  • MPEG haptic material (MPEG_material_haptic) 2509 includes haptic material attribute description modules corresponding to multiple haptic material attributes (stiffness, friction, haptic texture, temperature, vibration and custom), and the haptic material attribute description module corresponding to each haptic material attribute includes the type of the haptic material attribute, specifically including: low resolution (Low_resolution), high resolution (High_resolution), reference (Reference), and other (Other). If the haptic material texture category of a haptic material attribute is low resolution or high resolution, it means that the haptic value at the texture coordinate point can be obtained from a texture map or other haptic texture material data packet.
  • the value contained in each pixel point on the texture map or haptic texture material data packet represents the haptic value, and as shown in Tables 15 and 16, the bit depth of the haptic value of different haptic material texture types is different, such as the bit depth of the haptic value of the haptic material attribute of high resolution (High_resolution) is 16 bits, while the bit depth of the attribute of the haptic material attribute of low resolution (Low_resolution) is 8 bits.
  • the category of the tactile material attribute is reference (Reference)
  • the tactile value of the texture coordinate point or a group of texture coordinate points can be indexed to a group of tactile values in time or space.
  • MPEG tactile material (MPEG_materia_haptic) 2609 contains the type of each tactile material attribute.
  • the haptic_id specifies the media description module of the tactile media file that the texture coordinate point needs to index
  • the accessor specifies the accessor description module corresponding to the accessor that can actually access the decoded data of the tactile media file that the texture coordinate point needs to index.
  • This function corresponds to the description of the buffer description module (buffer) and the buffer slice description module (bufferview) in the scene description file, and together realizes the direct indexing of the tactile media file by the texture coordinate point.
  • the definition of the syntax elements in the media reference (Media_reference) of MPEG haptic (MPEG_haptic) 2611 is shown in Table 20 below:
  • a description method of an accessor description module (accessor) 2605, a buffer slice description module (bufferView) 2606, and a buffer description module (buffer) 2607 in a scene description file supporting obtaining a tactile value of a tactile material attribute of a reference type includes: pointing to an index of a media description module corresponding to a tactile media file in an MPEG media (MPEG_media) through a value of a media index syntax element (buffer.MPEG_buffer_circular.media) of an MPEG circular buffer of the buffer description module.
  • the tactile media file needs to be specified in the buffer description module (buffer), but rather than directly adding a uniform resource locator (URL) of the tactile media file in the buffer description module (buffer), the index value of the media description module corresponding to the corresponding tactile media file in the MPEG media (MPEG_media) is declared through a media index syntax element (MPEG_buffer_circular.media) of an MPEG circular buffer of the buffer description module (buffer).
  • MPEG_media media index syntax element
  • the value of the media index syntax element (buffer.MPEG_buffer_circular.media) in the MPEG circular buffer of the buffer description module can be set to "2", so as to index the access address in the third media description module in the MPEG media in the MPEG circular buffer of the buffer description module, so as to index the tactile media file through the (MPEG_buffer_circular) media index syntax element in the MPEG circular buffer of the buffer description module.
  • the description method of the accessor description module (accessor) 2605, the buffer slice description module (bufferView) 2606, and the buffer description module (buffer) 2607 in the scene description file supporting the acquisition of the tactile value of the tactile material attribute of the reference type includes: indicating the track information when accessing the encapsulated file of the tactile media file through the track array (buffer.MPEG_buffer_circular.tracks.track) of the MPEG circular buffer of the buffer description module. Specifically, it includes: when the track array (buffer.MPEG_buffer_circular.tracks) of the MPEG circular buffer of the buffer description module contains multiple tracks, the pipeline should perform necessary processing on all referenced tracks to generate the data format requested by the buffer description module.
  • each alternative should record the selection of the track.
  • the pipeline should perform necessary processing on all tracks in the MPEG media (MPEG_media) to generate the data format requested by the buffer description module (buffer), that is, to instruct the buffer description module (buffer) how to correctly store the data in the MPEG media (MPEG_media).
  • MPEG_buffer_circular The syntax elements contained in the MPEG circular buffer (MPEG_buffer_circular) are shown in Table 21:
  • the value of the media index syntax element (media) in Table 21 is the index value of the media description module corresponding to the corresponding tactile media file in the MPEG media (MPEG_media), so that the tactile media file is indexed in the buffer description module.
  • the value of the syntax element "tracks” in Table 21 is the track index containing the tactile media file in the MPEG media, so that the track of the tactile media file is indexed in the buffer description module.
  • a description method of a camera description module (camera) 2612 of a scene description file that supports obtaining tactile values of tactile material attributes of a reference type includes: defining the viewpoint, viewing angle, and other viewing-related visual information of a node description module (node) through the camera description module (camera) 2612.
  • the description method of the animation description module (animation) 2613 of the scene description file that supports obtaining the tactile value of the tactile material attribute of the reference type includes: defining the animation added to the node description module (node) through the animation description module (animation) 2613.
  • the animation description module can describe the animation added to the node description module (node) through one or more of position movement, angle rotation, and size scaling.
  • the animation description module may also indicate at least one of the start time, end time, and implementation method of the animation added to the node description module (node).
  • animation can also be added to the node description module (node) representing the object in the three-dimensional object.
  • the animation description module (animation) describes the animation added to the node in three ways: position movement, angle rotation, and size scaling.
  • start and end time of the animation and the implementation method of the animation can also be specified.
  • a description method of a skin description module (skin) 2614 of a scene description file that supports obtaining tactile values of tactile material attributes of a reference type includes: defining the movement and deformation relationship between a mesh in a node description module (node) and the corresponding bone through the skin description module (skin) 2614.
  • the scene description file can correctly describe the tactile media file.
  • the scene description file shown in FIG. 9 is described below by taking a scene description file of a tactile medium in a single track package including stiffness, temperature, and vibration tactile material attributes as an example.
  • the pair of curly brackets between line 1 and line 241 in the above example contains the main contents of the scene description file shown in FIG9 .
  • the scene description file that supports obtaining the tactile value of the tactile material attribute of the reference type includes: digital asset description module (asset), use of extension description module (extensionUsed), MPEG media (MPEG_media), scene declaration (scene), scene list (scenes), node list (nodes), mesh list (meshes), tactile material Array list (MPEG_material_Haptic), texture list (textures), texture map list (images), sampler list (samplers), accessor list (accessors), buffer slice list (bufferView), buffer list (buffers).
  • digital asset description module assert
  • extension description module extensionUsed
  • MPEG_media MPEG media
  • scene declaration scene
  • scene list scenes
  • node list node list
  • mesh list meshes
  • tactile material Array list MPEG_material_Haptic
  • texture list textures
  • texture map list images
  • sampler list
  • Digital asset description module (asset): The digital asset description module is lines 2 to 4. From the "version”: “2.0" in line 3 of the digital asset description module, it can be determined that the scene description file is written based on glTF 2.0, which is also the reference version of the scene description standard. From the parsing perspective, the display engine can determine which parser should be selected to parse the scene description file based on the digital asset description module.
  • Extension description module used is from line 6 to line 12. Since the extension description module used includes five extensions: MPEG media (MPEG_media), MPEG circular buffer (MPEG_buffer_circular), MPEG time-varying accessor (MPEG_accessor_timed), MPEG haptic (MPEG_Haptic) and MPEG haptic material (MPEG_material_Haptic), it can be determined that the scene description file uses five MPEG extensions: MPEG media, MPEG circular buffer, MPEG time-varying accessor, MPEG haptic, and MPEG haptic material.
  • MPEG media MPEG_media
  • MPEG_buffer_circular MPEG time-varying accessor
  • MPEG_haptic MPEG_Haptic
  • MPEG_material_Haptic MPEG haptic material
  • the display engine can know in advance that the extension items involved in the subsequent parsing include: MPEG media, MPEG circular buffer, MPEG time-varying accessor, MPEG haptic, and MPEG haptic material based on the content of the extension description module used (extensionUsed).
  • MPEG media is lines 14 to 53.
  • MPEG media implements the declaration of tactile media files contained in the 3D scene.
  • the media list (media) of MPEG media includes two curly brackets (two media description modules, the media description modules of lines 17 to 33 and the media description modules of lines 34 to 50), indicating that two media files are declared in the MPEG media of the scene description file.
  • Scene declaration The scene declaration is line 55. Since a scene description file can theoretically include multiple 3D scenes, the above scene description file first indicates through "scene":0 on line 55 that the scene described in the scene description file is the 3D scene corresponding to the first scene description module in the scene list, that is, the scene described in the scene description file enclosed by the curly brackets on lines 57 to 63.
  • Scene list The scene list is from line 57 to line 63.
  • the scene list contains only one curly bracket, indicating
  • the scene description file contains only one scene description module (scene), and only one 3D scene.
  • "nodes":[0] in lines 58 to 61 indicates that the 3D scene only includes one node with index 0.
  • the scene list clarifies that the entire scene description framework should select the first 3D scene in the scene list for subsequent processing and rendering, clarifies the overall structure of the 3D scene, and points to a more detailed next-level node description module (node).
  • Node list (nodes): The node list is from line 65 to line 83.
  • the node list contains only one curly bracket, indicating that the scene description file includes only one node description module, the three-dimensional scene has only one node, and the node is the same as the node with an index value of 0 in the node in the scene description module, and the two are associated through indexing.
  • the name of the node is "Hapticexample_node” through “name”:”Hapticexample_node” in line 67
  • the content mounted on the node is the first three-dimensional mesh in the mesh list (meshes) through “mesh”:"0" in line 68, which corresponds to the mesh description module of the next layer.
  • MPEG haptics (MPEG_Haptic) is also included in lines 69 to 81 of the node description module, and the media reference list (Media_reference) of MPEG haptics includes two curly brackets, indicating that the node description module includes two media reference arrays (media reference arrays in lines 72 to 75 and media reference arrays in lines 76 to 79).
  • the media reference array of lines 72 to 75 indicates through "haptic_id”:0 in line 73 that the media file corresponding to the media reference array of lines 72 to 75 is the media file described by the first media description module in the media list of MPEG media, and indicates through "accessor”:1 in line 74 that the accessor used to access the parsed data of the corresponding media file is the accessor described by the second accessor description module in the accessor list (accessors).
  • the media reference array of lines 76 to 79 indicates through "haptic_id”:1 in line 77 that the media file corresponding to the media reference array of lines 76 to 79 is the media file described by the second media description module in the media list of MPEG media, and indicates through "accessor”:2 in line 78 that the accessor used to access the parsed data of the corresponding media file is the accessor described by the third accessor description module in the accessor list (accessors).
  • the node description module in the node list indicates that the content mounted on the node is a three-dimensional grid, and that the index of the grid description module of the three-dimensional grid is 0.
  • the tactile media files that can be indexed from the MPEG media are the first media file and the second media file in the MPEG media. It also indicates that the accessor description modules of the accessor for accessing the first media file and the accessor for accessing the second media file are the accessors described by the second accessor description module and the third accessor description module in the accessor list (accessors), respectively.
  • the mesh list is from line 85 to line 102.
  • the mesh list contains only one curly bracket, indicating that the scene description file includes only one mesh description module (mesh), the 3D scene has only one 3D mesh, and the 3D mesh is the same mesh as the 3D mesh with index value 0 in the node description module.
  • the name of the 3D mesh is "Hapticexample_mesh” indicated by "name”:"Hapticexample_mesh” in line 87, which is only used as an identification mark, and the "primitives” in line 88 indicates that the 3D mesh has primitives.
  • the "attributes” in line 90, the “mode” in line 93, and the “MPEG_material_Haptic” in line 95 indicate that there are three types of information in the primitives: attribute, mode, and MPEG material haptics (MPEG_material_Haptic).
  • the "TEXCOORD_0":0 in line 91 indicates that the accessor description module corresponding to the texture coordinates is the first accessor description module in the accessor list.
  • the "mode”:0 in line 93 indicates that the topology of the 3D mesh is a scattered point structure.
  • Haptic material array list (MPEG_material_Haptic): The haptic material array list is lines 104 to 130.
  • the haptic material array list (MPEG_material_Haptic) contains only one curly bracket, indicating that the scene description file contains only one haptic material array (the content enclosed by the curly brackets on lines 106 to 128).
  • the haptic material array defines three haptic material properties, stiffness, temperature, and vibration, through lines 107 to 113, 114 to 120, and 121 to 127, respectively.
  • Each haptic material attribute includes two types of information: the texture coordinate index of the texture array.
  • the description module corresponding to stiffness is lines 107 to 113.
  • the "index” and its value in line 109 define that stiffness points to the first texture description module in the texture list.
  • the "texCoord” in line 110 defines that the tactile material attribute of stiffness will use the first texture coordinate set data packet as the coordinate reference for texture mapping of the tactile material attribute.
  • the "type”: Low_resolution in line 112 defines that the type of the tactile material attribute of stiffness is low resolution (Low_resolution).
  • the length of the tactile value of the stiffness attribute obtained subsequently is 8 bits.
  • the description module corresponding to temperature is lines 114 to 120.
  • the "index" and its value in line 116 define temperature.
  • the texture type and source of the tactile material attribute contained in the object can be determined according to the tactile material array list (MPEG_material_Haptic) and point to the more detailed next-level description module texture description module.
  • Texture list The texture list is lines 132 to 145.
  • the texture list contains three curly brackets, indicating that the scene description file includes three texture description modules, and the three texture description modules are used to describe the texture information of stiffness, temperature, and vibration respectively.
  • the stiffness and temperature texture description content includes the sampler syntax element (sampler) and the texture map syntax element (source). According to the values of the sampler syntax element (sampler) and the texture map syntax element (source), the access address of the sampler description module and the texture map of stiffness and temperature can be determined.
  • the source of the texture map corresponds to the texture map list (images) of the next level.
  • Texture map list (images): The texture map list is from line 147 to 157.
  • the texture map list contains three curly brackets, which correspond to the access addresses of the texture maps referenced in the texture description modules of stiffness, temperature, and vibration.
  • ⁇ "uri :"stiffnessTexture.png" ⁇ from line 148 to 150
  • the access address of the texture map of stiffness can be determined as stiffnessTexture.png.
  • ⁇ "uri”:"temperatureTexture.png” ⁇ from line 151 to 153
  • the access address of the texture map of temperature can be determined as temperatureTexture.png.
  • ⁇ "uri”:"Texture.png” ⁇ from line 154 to 156 the access address of the texture map of vibration can be determined as vibrationTexture.png.
  • Sampler list (samplers), the sampler list is lines 159 to 164.
  • the sampler list contains a curly bracket, indicating that the scene description file includes a sampler description module, the display of the 3D scene only needs to use one sampler, and the description information of the corresponding sampler can be determined by "magFilter”:9729, "minFilter”:9987, "wrapS”:33648, "wrapT”:33648 in lines 160 to 163.
  • Buffer list The buffer list is from line 215 to line 240.
  • the buffer list contains three brackets: The number indicates that the scene description file includes three buffer description modules, and the display of the three-dimensional scene only requires the use of three buffers. According to the content in the first curly bracket (the first buffer description module) of lines 216 to 219, it can be determined that the size of the first buffer is 1000, the address of the media file to be stored is "AnimatedBody.bin", and the media file includes the coordinate content that needs to add texture information to specific three-dimensional coordinates.
  • the second curly bracket (the second buffer description module) of lines 220 to 229, which includes an MPEG circular buffer (MPEG_buffer_circular)
  • MPEG_buffer_circular an MPEG circular buffer
  • the "media:0" in line 225 indicates that the data source in the ring buffer is the first media file declared in the MPEG media (MPEG_media) in the previous text.
  • the track with index 1 is not limited here. It can be the only track of the tactile file packaged by a single track, or it can be the reference track of the tactile file packaged by multiple tracks.
  • the buffer list realizes the correspondence of the tactile media files declared in the MPEG media (MPEG_media) to the buffer, or in other words, it realizes the reference of the tactile media files that were previously only declared but not used by the buffer.
  • the tactile media file referenced here is an encapsulated file of an unprocessed tactile media file, and the encapsulated file of the tactile media file needs to be processed by the media access function to extract information that can be directly used for rendering.
  • Cache slice list (bufferViews): The cache slice list is lines 197 to 213.
  • the cache slice list contains three parallel curly braces, indicating that the scene description file includes three cache slice description modules.
  • the cache slice corresponding to the first cache slice description module is the cache slice of the buffer corresponding to the first cache description module in the cache list.
  • the "byteLength":1000 on line 200 and the "byteOffset":0 on line 201 it is declared that the data slice range of the corresponding cache slice is the first 1000 bytes of the first buffer.
  • the cache slice corresponding to the second cache slice description module is the cache slice of the cache corresponding to the second cache description module in the cache list.
  • the "byteLength”:2000 in line 205 and the "byteOffset":0 in line 206 it is declared that the data slice range of the corresponding cache slice is the first 2000 bytes of the second cache.
  • the "buffer”:2 in line 209 of the third curly bracket (third cache slice description module) from 208 to 212 it can be determined that the cache slice corresponding to the third cache slice description module is the cache slice of the cache corresponding to the third cache description module in the cache list.
  • the cache slice list groups the data of the media file, which is conducive to the detailed definition of the subsequent accessor description module.
  • Accessor list (accessors): The accessor list is lines 166 to 195. The structure of the accessor list is similar to that of the cache slice list, and both contain three parallel curly brackets, indicating that the 3D scene description file includes three accessor description modules (accessor), and the display of the 3D scene requires access to media data through three accessors.
  • the accessor corresponding to the first accessor description module can be used to access the first cache slice in the cache slice list. It can also be determined through the "componentType”:5121" on line 169 and the "type”:"VEC2" on line 170 that the data format stored in the corresponding accessor is a two-dimensional vector composed of 8-bit unsigned numbers.
  • the "count”:500 on line 171 indicates that there are 500 data that need to be accessed through this accessor, and each 8-bit unsigned number occupies 1 byte, so this accessor It contains 1000 bytes of data, which is the same as the provisions in the cache slice description module with index value 0.
  • the second curly bracket (the second accessor description module) and the third curly bracket (the third accessor description module) both have MPEG time-varying accessor parts, indicating that these two accessors point to time-varying media defined by MPEG. From the "bufferView":1 in line 179, it can be determined that the content in the first MPEG time-varying accessor points to the cache slice with index value 1, and it is explained that the values of the syntax elements in the corresponding accessor change with time.
  • the accessor list completes the complete definition of the data required for rendering. For example, the data types missing in the cache slice and the buffer are defined in the corresponding accessor description module.
  • the main functions of the display engine include: parsing the scene description file to obtain the method of rendering the three-dimensional scene; sending media access instructions or media data processing instructions to the media access function through the media access function API; sending cache management instructions to the cache management module through the cache API; taking the processed data from the cache, and completing the rendering and display of the three-dimensional scene and the objects in the three-dimensional scene according to the read data. Therefore, the functions of the display engine that supports tactile media include: 1. being able to parse the scene description file containing tactile media and obtain the corresponding three-dimensional scene rendering method; 2.
  • the display engine can obtain the method of rendering the three-dimensional scene by parsing the scene description file, and needs to pass the method of rendering the three-dimensional scene to the media access function or send instructions to the media access function based on the method of rendering the three-dimensional scene.
  • the process of passing the method of rendering the three-dimensional scene to the media access function or sending instructions to the media access function based on the method of rendering the three-dimensional scene is implemented through the media access function API.
  • the display engine may send a media access instruction or a media data processing instruction to the media access function through a media access function API.
  • the instructions issued by the display engine come from the parsing result of the scene description file containing the attribute information of the tactile media, and these instructions may include the address of the media file, the attribute information of the media file (media type, the codec used, etc.), the format requirements for the processed tactile media data and other media data, etc.
  • the media access function may also request a media access instruction or a media data processing instruction from the display engine through a media access function API.
  • the media access function After the media access function receives the media access instruction or media data processing instruction issued by the display engine through the media access function API, it will execute the media access instruction or media data processing instruction issued by the display engine through the media access function API. For example: obtaining tactile media files, establishing appropriate pipelines for tactile media files to convert the data of tactile media files into the format specified in the instruction, and allocating appropriate caches for processing tactile media data and other media data in the format specified in the scene description file.
  • the media access function obtains the haptic media file, including: downloading the haptic media file from a server using a network transmission service.
  • the media access function obtains the haptic media file, including: reading the haptic media file from a local storage space.
  • the media access function After the media access function obtains the tactile media file, it needs to process the tactile media file.
  • the processing process of different types of media files is quite different. In order to achieve wide support for media types and also consider the working efficiency of the media access function, multiple pipelines are designed in the media access function. In the process of processing media files, only the pipeline matching the media type is enabled.
  • the media access function After the media access function processes the data of the tactile media file, the media access function also needs to deliver the processed data to the display engine in a standardized arrangement structure. This requires the processed data to be correctly stored in the cache. This work is completed by the cache management module, but the cache management module needs to obtain cache management instructions from the media access function or the display engine through the cache API.
  • the media access function may send a cache management instruction to the cache management module via a cache API, wherein the cache management instruction is a cache management instruction sent by the display engine to the media access function via the media access function API.
  • the display engine may send cache management instructions to the cache management module via a cache API.
  • the cache management module can communicate with the media access function through the cache API, and can also communicate with the display engine through the cache API, and the purpose of communicating with the media access function or the display engine is to achieve cache management.
  • the display engine needs to send the cache management instruction to the media access function through the media access function API first, and the media access function then sends the cache management instruction to the cache management module through the cache API;
  • the display engine only needs to generate the cache management instruction based on the cache management information parsed from the scene description file, and send it to the cache management module through the cache API.
  • the cache management instruction may include one or more of an instruction to create a cache, an instruction to update a cache, and an instruction to release a cache.
  • a cache management module that supports obtaining the tactile values of tactile material properties of reference types
  • the processed data needs to be delivered to the display engine in a standardized arrangement structure. This requires the processed data to be correctly stored in the cache, which is the responsibility of the cache management module.
  • the cache management module implements management operations such as cache creation, update, and release, and the operation instructions are received through the cache API.
  • the cache management rules are recorded in the scene description document, parsed by the display engine, and finally issued to the cache management module by the display engine or the media access function. After the media file is processed by the media access function, it needs to be stored in a suitable cache and then accessed by the display engine.
  • the role of cache management is to manage these caches well so that they match the format of the processed media data without disrupting the processed media data.
  • the specific design method of the media management module should be based on the design of the display engine and the media access function.
  • some embodiments of the present application provide a method for generating a scene description file.
  • the method for generating a scene description file includes the following steps: S271 to S275:
  • the tactile material attributes in the embodiment of the present application may include: stiffness, friction, tactile texture, temperature, vibration, and one or more of custom attributes.
  • the types of tactile material attributes include: high resolution, low resolution, and reference.
  • the types of different tactile material attributes in the tactile media file may be different, so the tactile media file may include multiple different types of tactile material attributes, and when determining the types of each tactile material attribute of the tactile media file, it is necessary to determine the type of each tactile material attribute one by one.
  • step S271 if the type of the first tactile material attribute of the tactile media file is reference, the following steps S272 and S273 are performed:
  • S272 Add an accessor index syntax element (accessor) to the target node description module of the scene description file of the three-dimensional scene to be rendered.
  • the target node description module is a node description module corresponding to the node of the target three-dimensional grid that carries the first tactile material attribute.
  • the above step S272 (adding an accessor index syntax element in a target node description module of the scene description file of the three-dimensional scene to be rendered) includes:
  • An accessor index syntax element is added to the MPEG_haptic of the target node description module of the scene description file of the three-dimensional scene to be rendered.
  • the implementation method of adding the accessor index syntax element in the MPEG touch of the target node description module of the scene description file of the three-dimensional scene to be rendered includes the following steps 2721 to 2723:
  • Step 2721 Add a media reference list (Media_reference) to the MPEG touch of the target node description module.
  • Step 2722 Add a media reference array corresponding to the tactile media file to the media reference list.
  • Step 2723 Add the accessor index syntax element to the media reference array corresponding to the tactile media file.
  • the target node description module can be as follows:
  • the content of lines n+6 to n+9 is the media reference list added in the MPEG tactile of the target node description module, and the content of lines n+7 to n+9 is the media reference array corresponding to the tactile media file added in the media reference list.
  • the content of line n+8 is the accessor index syntax element added in the media reference array corresponding to the tactile media file.
  • the three-dimensional scene to be rendered may include multiple tactile media files
  • the node corresponding to the target node description module may include a three-dimensional grid carrying tactile material attributes of reference types of multiple tactile media files.
  • the node corresponding to the target node description module includes a three-dimensional grid carrying tactile material attributes of reference types of multiple tactile media files, it is necessary to add the corresponding media reference array of each tactile media file in the MPEG tactile media reference list.
  • the tactile media accessor is an accessor for accessing decoded data of the tactile media file.
  • the target node description module may be as follows:
  • accessor in the n+8th line is the accessor index syntax element added in the media reference array and the value of the accessor index syntax element.
  • “2” is also the index value of the accessor description module corresponding to the tactile media accessor used to access the tactile media file.
  • the method for generating a scene description file includes a tactile media file in a three-dimensional scene to be rendered, and when the type of the first tactile material attribute of the tactile media file is a reference, firstly, an accessor index syntax element is added to a target node description module corresponding to a node of a target three-dimensional mesh that carries the first tactile material attribute, and the value of the accessor index syntax element is set to the index value of the accessor description module corresponding to the tactile media accessor used to access the decoded data of the tactile media file.
  • the method for generating a scene description file includes a tactile media file in the three-dimensional scene to be rendered, and the type of the first tactile material attribute of the tactile media file is a reference, an accessor index syntax element is added to the target node description module, and the value of the accessor index syntax element is set to the index value of the accessor description module corresponding to the tactile media accessor used to access the decoded data of the tactile media file.
  • the tactile media accessor used to access the decoded data of the tactile media file can be determined according to the accessor index syntax element and its value, and then the tactile value of the first tactile material attribute is read from the decoded data of the tactile media file according to the tactile media accessor. Therefore, the embodiment of the present application can support the acquisition of the tactile value of the tactile material attribute of the reference type, thereby solving the problem that the tactile value of the tactile material attribute of the reference type cannot be obtained in the related art.
  • the method for generating a scene description file further includes the following steps 1 and 2:
  • Step 1 Add a haptic media identification syntax element (haptic_id) to the media reference array corresponding to the haptic media file.
  • haptic_id a haptic media identification syntax element
  • Step 2 Set the value of the tactile media identification syntax element as the index value of the target media description module.
  • the target media description module is a media description module corresponding to the tactile media file in the media list of the MPEG media of the scene description file.
  • the above embodiment also adds a tactile media identification syntax element in the media reference array, and sets the value of the tactile media identification syntax element to the index value of the target media description module corresponding to the tactile media file, the above embodiment can index different tactile media files through the tactile media identification syntax element and its value when multiple tactile media files are included in the three-dimensional scene, thereby supporting the addition of multiple tactile media files in the three-dimensional scene.
  • the media reference array corresponding to the tactile media file may be as follows:
  • the method provided by the embodiment of the present application further includes the following steps a to c:
  • Step a adding a buffer description module corresponding to the tactile media buffer to the buffer list (buffers) of the scene description file.
  • the tactile media buffer is a buffer for buffering decoded data of the tactile media file.
  • the above step a (adding the buffer description module corresponding to the tactile media buffer in the buffer list of the scene description file) includes at least one of the following steps a1 to a5:
  • Step a1 adding a byte length syntax element (byteLength) in a buffer description module corresponding to the tactile media buffer, and setting the value of the byte length syntax element to the byte length of the tactile media file.
  • Step a2 adding an MPEG ring buffer to the buffer description module corresponding to the tactile media buffer.
  • Step a3 adding a quantity syntax element (count) in the MPEG circular buffer, and setting the value of the quantity syntax element according to the preconfigured number of storage links of the MPEG circular buffer.
  • Step a4 adding a media index syntax element (media) in the MPEG ring buffer, and setting the value of the media index syntax element according to the index value of the media description module corresponding to the tactile media file.
  • media media
  • Step a5 Add a track index syntax element (tracks) in the MPEG circular buffer, and set the value of the track index syntax element according to the track index value of the data stored in the MPEG circular buffer.
  • the buffer description module corresponding to the tactile media buffer added in the buffer list of the scene description file can be as follows:
  • Step b adding a cache slice description module corresponding to the cache slice of the tactile media buffer to the cache slice list (bufferViews) of the scene description file.
  • the above step b (adding the cache slice description module corresponding to the cache slice of the tactile media cache in the cache slice list of the scene description file) includes at least one of the following steps b1 to b3:
  • Step b1 adding a buffer index syntax element (buffer) in a buffer slice description module corresponding to a buffer slice of the tactile media buffer, and setting a value of the buffer index syntax element according to an index value of the buffer description module corresponding to the tactile media buffer.
  • buffer buffer index syntax element
  • Step b2 adding a byte length syntax element (byteLength) in a cache slice description module corresponding to the cache slice of the tactile media buffer, and setting a value of the byte length syntax element according to the capacity of the corresponding cache slice.
  • byteLength a byte length syntax element
  • Step b3 adding an offset syntax element to a cache slice description module corresponding to the cache slice of the tactile media buffer, and setting a value of the offset syntax element according to an offset of storage data of the corresponding cache slice.
  • the cache slice description module corresponding to the cache slice of the tactile media buffer added in the cache slice list (bufferViews) of the scene description file includes each item in the above steps b1 to b3, the index value of the buffer description module corresponding to the tactile media buffer is 1, the capacity of the tactile media buffer is 8000, and the tactile media buffer includes only one cache slice, then the cache slice description module corresponding to the cache slice of the tactile media buffer added in the cache slice list of the scene description file can be as follows:
  • Step c adding an accessor description module corresponding to the tactile media accessor for accessing data in a cache slice of the tactile media cache to an accessor list (accessors) of the scene description file.
  • the accessor for accessing the data in the cache slice of the tactile media buffer is the accessor for accessing the decoded data of the tactile media file.
  • the above step c (adding a user to the accessor list (accessors) of the scene description file)
  • the accessor description module corresponding to the tactile media accessor for accessing the data in the cache slice of the tactile media cache) comprises at least one of the following steps c1 to c6:
  • Step c1 adding a data type syntax element (componentType) in the accessor description module, and setting the value of the data type syntax element according to the data type of the tactile value of the first tactile material attribute.
  • componentType a data type syntax element
  • Step c2 Add an accessor type syntax element (type) in the accessor description module, and set the value of the accessor type syntax element according to the preconfigured accessor type.
  • Step c3 Add a count syntax element (count) in the accessor description module, and set the value of the count syntax element according to the number of tactile values of the first tactile material attribute.
  • Step c4 Add an MPEG time-varying accessor (MPEG_accessor_timed) to the accessor description module.
  • MPEG_accessor_timed MPEG time-varying accessor
  • Step c5 Add a buffer slice index syntax element (bufferView) in the MPEG time-varying accessor, and set the value of the buffer slice index syntax element according to the index value of the buffer slice description module corresponding to the buffer slice of the tactile media buffer.
  • bufferView a buffer slice index syntax element
  • Step c6 Add a time-varying syntax element to the MPEG time-varying accessor, and set the value of the time-varying syntax element according to whether the value of the syntax element in the corresponding accessor changes with time.
  • the accessor description module corresponding to the tactile media accessor for accessing the data in the cache slice of the tactile media buffer added in the accessor list (accessors) of the scene description file includes each item in the above steps c1 to c6, the data type of the tactile value of the first tactile material attribute is 5123, the preconfigured accessor type is SCALAR, the number of tactile values of the first tactile material attribute is 3000, the index value of the cache slice description module corresponding to the cache slice of the tactile media buffer is 1, and the value of the syntax element in the corresponding accessor changes with time, then the accessor description module corresponding to the tactile media accessor for accessing the data in the cache slice of the tactile media buffer added in the accessor list (accessors) of the scene description file can be as follows:
  • the method for generating a scene description file before setting the value of the tactile media identification syntax element as the index value of the target media description module, the method for generating a scene description file further includes:
  • a target media description module corresponding to the tactile media file is added to the media list (media) of the MPEG media (MPEG_media) of the scene description file.
  • the above (adding the target media description module corresponding to the tactile media file in the media list of the MPEG media in the scene description file) includes at least one of the following steps 11 to 19:
  • Step 11 Add a media name syntax element (name) in the target media description module, and set the value of the media name syntax element according to the name of the tactile media file.
  • Step 12 automatically play a syntax element (autoplay) in the target media description module, and set a value of the autoplay syntax element according to whether the tactile media file needs to be automatically played.
  • Step 13 loop-play syntax element (loop) in the target media description module, and set the value of the loop-play syntax element according to whether the tactile media file needs to be loop-played.
  • Step 14 Add alternatives in the target media description module.
  • Step 15 Add a media type syntax element (mimeType) to the optional items, and set the value of the media type syntax element according to the data contained in the tactile media file.
  • a media type syntax element miType
  • setting the value of the media type syntax element according to the data included in the haptic media file comprises:
  • the value of the media type syntax element in the optional items of the target media description module is set to a second preset value.
  • the first preset value may be video/mp4, and the second preset value may be haptics/mp4. That is, when the tactile media file includes only tactile data, the value of the media type syntax element is set to haptics/mp4, and when the tactile media file includes not only tactile data but also audio and/or video data, the value of the media type syntax element is set to video/mp4.
  • setting the value of the media type syntax element to video/mp4 may be backward compatible with existing .mp4 media files.
  • Step 16 Add an access address syntax element (uri) to the optional items, and set the value of the access address syntax element to the access address of the tactile media file.
  • uri access address syntax element
  • Step 17 Add a track array (tracks) to the optional items.
  • Step 18 Add a track index syntax element (track) to the track array, and set the value of the track index syntax element in the track array of the optional items of the target media description module to the index value of each code stream track of the tactile media file.
  • the encoded tactile media is referenced by the scene description file as an item in MPEG_media.alternative.tracks, and the referenced item complies with the provisions of the ISO/IEC 23090-32 standard on tracks in ISOBMFF: for a tactile media file encapsulated in a single track, the track referenced in MPEG_media is the single track of the tactile encapsulation. For example, if the tactile media is encapsulated into an MIHS track by ISOBMFF, the track referenced in MPEG_media is this MIHS track. For a tactile media file encapsulated in multiple tracks, the track referenced in MPEG_media is the multiple tracks into which the tactile media is encapsulated.
  • a non-zero value is defined in advance in the TrackHeaderBox of the configuration information alternate_group of the encapsulation file, indicating that the tactile media is encapsulated into multiple MIHS tracks at this time, and these tracks are also the tracks indexed by MPEG_media.
  • Step 19 Add a codec parameter syntax element (codec) to the track array, and set the value of the codec parameter syntax element according to the encoding parameters of the data in each bitstream track of the tactile media file and the provisions of the ISO/IEC 23090-32 tactile data transmission standard.
  • codec codec parameter syntax element
  • Codec parameters for media contained in an array of tracks as defined in IETF RFC 6381.
  • the array of tracks contains codecs of different types (e.g. an AdaptationSet contains representations with different codecs)
  • the CodecParameters syntax element can be represented by a comma-separated list of codec values.
  • the target media description module can be as follows:
  • the method for generating a scene description file further includes:
  • the target scene description module corresponding to the three-dimensional scene to be rendered is added to the scene list (scenes) of the scene description file, and the index value of the node description module corresponding to each node in the scene to be rendered is added to the node index list (nodes) of the target scene description module.
  • the target scene description module may be as follows:
  • the method for generating a scene description file further includes:
  • the node description modules corresponding to the nodes in the scene to be rendered to the node list of the scene description file it is necessary to determine whether the three-dimensional grid carrying the tactile material attributes and the tactile material attributes carried are included.
  • the type of material attribute divides the nodes in the scene to be rendered into three categories. For nodes that do not include a three-dimensional mesh that carries a tactile material attribute, the corresponding node description module only needs to add the index value of the mesh description module corresponding to the three-dimensional mesh in each node in the mesh index list of each node description module.
  • MPEG_haptic MPEG haptic
  • the corresponding node description module is generated in a manner similar to the target node description module in the above embodiment, and will not be repeated here.
  • a node does not include a three-dimensional grid carrying tactile material attributes
  • the node is named Hapticexample_node1
  • the node only includes a three-dimensional grid with an index value of 2 of a corresponding grid description module
  • the node description module corresponding to the node can be as follows:
  • the method for generating the scene description file further includes: adding a mesh description module corresponding to the target three-dimensional mesh to a mesh list of the scene description file.
  • adding a corresponding mesh description module in the mesh list of the scene description file includes at least one of the following steps d1 to d5:
  • Step d1 Add a grid name syntax element (name) in the grid description module corresponding to the three-dimensional grid, and set the value of the grid name syntax element according to the name of the three-dimensional grid.
  • Step d2 adding attributes (primitives) to the mesh description module corresponding to the three-dimensional mesh.
  • Step d3 adding primitives (attributes) to the attributes.
  • Step d4 adding a texture coordinate syntax element (TEXCOORD_n) to the primitive, and setting the value of the texture coordinate syntax element to the index value of the accessor description module corresponding to the texture coordinate accessor.
  • TEXCOORD_n texture coordinate syntax element
  • the texture coordinate accessor is an accessor for accessing the texture coordinates of the three-dimensional grid.
  • Step d5 Add a mode syntax element (mode) in the mesh description module corresponding to the three-dimensional mesh, and set the value of the mode syntax element according to the topological structure type of the three-dimensional mesh.
  • mode mode syntax element
  • the mesh description module corresponding to the target three-dimensional mesh in the mesh list of the scene description file includes each item in the above steps d1 to d5, a certain three-dimensional mesh does not carry tactile material attributes, the name of the three-dimensional mesh is example_mesh, and the topological structure of the three-dimensional mesh is scattered points, then the mesh description module corresponding to the three-dimensional mesh added in the mesh list of the scene description file can be as follows:
  • adding a corresponding mesh description module in the mesh list of the scene description file includes at least one of the following steps d1 to d5, and also includes the following steps d6 and d7:
  • Step d6 Add MPEG tactile material (MPEG_material_Haptic) to the mesh description module corresponding to the three-dimensional mesh.
  • Step d7 Add an array index syntax element (index) to the MPEG tactile material, and set the value of the array index syntax element to the index value of the tactile material attribute array of the three-dimensional grid.
  • the mesh description module corresponding to the target three-dimensional mesh in the mesh list of the scene description file includes each item in the above steps d1 to d7, a certain three-dimensional mesh carries tactile material attributes, the name of the three-dimensional mesh is Hapticexample_mesh, the topological structure of the three-dimensional mesh is scattered points, and the index value of the tactile material attribute array of the three-dimensional mesh is 2, then the mesh description module corresponding to the three-dimensional mesh added in the mesh list of the scene description file can be as follows:
  • the method provided by the embodiment of the present application further includes the following steps e to g:
  • Step e adding a buffer description module corresponding to the texture coordinate buffer to the buffer list (buffers) of the scene description file.
  • the texture coordinate buffer is a buffer used to cache the texture coordinates of the target three-dimensional grid.
  • the above step e (adding a buffer description module corresponding to the texture coordinate buffer in the buffer list of the scene description file) includes at least one of the following steps e1 and e2:
  • Step e1 Add a byte length syntax element (byteLength) in the buffer description module corresponding to the texture coordinate buffer, and set the value of the byte length syntax element to the byte length of the texture coordinates of the target three-dimensional mesh.
  • Step e2 Add a texture coordinate access address syntax element (uri) in the buffer description module corresponding to the texture coordinate buffer, and set the value of the texture coordinate access address syntax element to the access address of the texture coordinates of the target three-dimensional mesh. Ask for the address.
  • uri texture coordinate access address syntax element
  • the buffer description module corresponding to the texture coordinate buffer added in the buffer list of the scene description file may be as follows:
  • the biggest difference between the buffer description module corresponding to the texture coordinate buffer and the buffer description module corresponding to the tactile media buffer is that the buffer description module corresponding to the texture coordinate buffer no longer indexes the media file declared in the MPEG media, but directly adds the access address of the texture coordinates, and accesses the texture coordinates through the access address of the texture coordinates.
  • Step f adding a cache slice description module corresponding to the cache slice of the texture coordinate buffer to the cache slice list (bufferViews) of the scene description file.
  • the above step f (adding the cache slice description module corresponding to the cache slice of the texture coordinate cache in the cache slice list of the scene description file) includes at least one of the following steps f1 to f3:
  • Step f1 adding a buffer index syntax element (buffer) in the buffer slice description module corresponding to the buffer slice of the texture coordinate buffer, and setting the value of the buffer index syntax element according to the index value of the buffer description module corresponding to the texture coordinate buffer.
  • buffer buffer index syntax element
  • Step f2 adding a byte length syntax element (byteLength) in a cache slice description module corresponding to the cache slice of the texture coordinate buffer, and setting a value of the byte length syntax element according to the capacity of the corresponding cache slice.
  • Step f3 adding an offset syntax element in a cache slice description module corresponding to the cache slice of the texture coordinate buffer, and setting a value of the offset syntax element according to the offset of the storage data of the corresponding cache slice.
  • the cache slice description module corresponding to the cache slice of the texture coordinate buffer added in the cache slice list (bufferViews) of the scene description file includes each item in the above steps f1 to f3, the index value of the cache description module corresponding to the texture coordinate buffer is 0, the capacity of the texture coordinate buffer is 4000, and the texture coordinate buffer includes only one cache slice, then the cache slice description module corresponding to the cache slice of the texture coordinate buffer added in the cache slice list of the scene description file can be as follows:
  • Step g adding, in the accessor list (accessors) of the scene description file, an accessor description module corresponding to the accessor for accessing the tactile value of the first tactile material attribute from the cache slice of the texture coordinate buffer.
  • the above step g (adding an accessor description module corresponding to an accessor for accessing data in a cache slice of the texture coordinate buffer in the accessor list (accessors) of the scene description file) includes at least one of the following steps g1 to g4:
  • Step g1 Add a cache slice index syntax element (bufferView) in the accessor description module, and The value of the cache slice index syntax element is set according to the index value of the cache slice description module corresponding to the cache slice of the texture coordinate buffer.
  • bufferView a cache slice index syntax element
  • Step g2 Add a data type syntax element (componentType) in the accessor description module, and set the value of the data type syntax element according to the data type of the tactile value of the first tactile material attribute.
  • componentType a data type syntax element
  • Step g3 Add an accessor type syntax element (type) in the accessor description module, and set the value of the accessor type syntax element according to the preconfigured accessor type.
  • Step g4 Add a count syntax element (count) in the accessor description module, and set the value of the count syntax element according to the number of tactile values of the first tactile material attribute.
  • the accessor description module corresponding to the accessor for accessing the tactile value of the first tactile material attribute from the cache slice of the texture coordinate buffer added in the accessor list of the scene description file includes each item in the above steps g1 to g4, the data type of the texture coordinates of the target three-dimensional mesh is 5121, the preconfigured accessor type is VEC2, the number of texture coordinates of the target three-dimensional mesh is 800, and the index value of the cache slice description module corresponding to the cache slice of the texture coordinate buffer is 0, then the accessor description module corresponding to the accessor for accessing the tactile value of the first tactile material attribute from the cache slice of the texture coordinate buffer added in the accessor list of the scene description file can be as follows:
  • the method provided by the embodiment of the present application further includes the following steps h to k:
  • Step h Add a haptic material array list (MPEG_material_Haptic) to the scene description file.
  • Step j adding the tactile material attribute array of each three-dimensional grid carrying the tactile material attribute (including the target three-dimensional grid carrying the first tactile material attribute) to the tactile material array list.
  • the tactile material attribute array of each three-dimensional grid includes a tactile material attribute description module corresponding to each tactile material attribute carried by each three-dimensional grid, and the tactile material attribute description module corresponding to any tactile material attribute includes: a texture array (texture) and an attribute type syntax element (type); the texture array includes: a texture index syntax element (index) and a texture coordinate index syntax element (texCoord).
  • the tactile material attribute array of each three-dimensional grid includes a tactile material attribute description module corresponding to each tactile material attribute carried by each three-dimensional grid
  • the tactile material attribute description module corresponding to any tactile material attribute includes: a texture array and an attribute type syntax element
  • the texture array includes: a texture index syntax element and a texture coordinate index syntax element
  • Step j1 adding a tactile material attribute description module corresponding to each tactile material attribute carried by each three-dimensional grid to the tactile material attribute array of each three-dimensional grid.
  • Step j2 Add texture array and attribute type syntax elements in each tactile material attribute description module (type).
  • Step j3 Add texture index syntax elements and texture coordinate index syntax elements (texCoord) to each texture array.
  • Step i setting the value of the texture index syntax element (index) in the texture array of the tactile material attribute description module corresponding to each tactile material attribute to the index value of the texture description module corresponding to each tactile material attribute.
  • Step j Set the value of the texture coordinate index syntax element (texCoord) in the texture array of the tactile material attribute description module corresponding to each tactile material attribute to the index value of the texture coordinate syntax element of the corresponding three-dimensional grid.
  • Step k setting the value of the attribute type syntax element in the tactile material attribute description module corresponding to each tactile material attribute to the type of each tactile material attribute.
  • the tactile material attribute array of the three-dimensional mesh may be as follows:
  • the tactile material array list added in the scene description file can be as follows:
  • the method for generating a scene description file further includes: before setting the value of the texture index syntax element in the texture array of the tactile material attribute description module corresponding to each tactile material attribute to the index value of the texture description module corresponding to each tactile material attribute, the method further includes performing the following steps l to n to generate a texture list (textures) of the scene description file:
  • Step 1 Add the texture description modules corresponding to each tactile material attribute to the texture list of the scene description file.
  • the texture description module corresponding to any tactile material attribute includes: a texture map syntax element (source) and a sampler syntax element (sampler).
  • Step m setting the value of the texture map syntax element in the texture description module corresponding to each tactile material attribute to the index value of the access address of the texture map of each tactile material attribute.
  • Step n setting the value of the sampler syntax element in the texture description module corresponding to each tactile material attribute to the index value of the sampler description module corresponding to the sampler of each tactile material attribute.
  • the texture description module corresponding to the tactile material attribute added to the texture list of the scene description file may be as follows:
  • the texture list of the scene description file may be as follows:
  • the method for generating a scene description file further includes: setting the value of the texture map syntax element in the texture description module corresponding to each tactile material attribute to the access address of the texture map of each tactile material attribute; Before the index value is obtained, the access address of the texture map of each tactile material attribute of the tactile media file is added to the texture map list (images) of the scene description file.
  • the texture map of the tactile material attribute stores the tactile value of the tactile material attribute, and if the type of a tactile material attribute is reference, the texture map of the tactile material attribute stores the index value of the tactile media file corresponding to the tactile material attribute.
  • the access address of a texture map of a certain tactile material attribute is temperatureTexture.png
  • the access address of the texture map of the tactile material attribute added in the texture list of the scene description file may be as follows:
  • the texture map list of the scene description file may be as follows:
  • the method for generating the scene description file also includes: before setting the value of the sampler syntax element in the texture description module corresponding to each tactile material attribute to the index value of the sampler description module corresponding to the sampler of each tactile material attribute, adding the sampler description module corresponding to the sampler of each tactile material attribute of the tactile media file in the sampler list (samplers) of the scene description file.
  • the description module of the sampler of the tactile material attribute can be as follows:
  • the scene description file includes only one sampler description module, and the sampler list in the scene description file may be as follows:
  • the method for generating a scene description file further includes:
  • a digital asset description module (asset) is added to the scene description file, a version syntax element (version) is added to the digital asset description module, and when the scene description file is a scene description document written based on glTF 2.0, the value of the version syntax element is set to 2.0.
  • the digital asset description module added to the scene description file may be as follows:
  • the method for generating a scene description file further includes:
  • extension usage description module (extensionsUsed) is added to the scene description file, and each MPGE extension of the glTF2.0 version of the scene description file used by the scene description file is added to the extension usage description module.
  • the MPEG extensions used in the scene description file include: MPEG media (MPEG_media), MPEG circular buffer (MPEG_buffer_circular), MPEG time-varying accessor (MPEG_accessor_timed), MPEG haptic (MPEG_Haptic) and MPEG haptic material (MPEG_material_Haptic), then the extended usage description module added in the scene description file may be as follows:
  • the method for generating a scene description file further includes:
  • a scene declaration (scene) is added to the scene description file, and the value of the scene declaration is set to the index value of the scene description module corresponding to the scene to be rendered.
  • the index value of the scene description module corresponding to the scene to be rendered is 1, and the scene declaration added to the scene description file can be as follows:
  • Some embodiments of the present application also provide a method for parsing a scene description file. As shown in FIG. 11 , the method for generating a scene description file includes the following steps: S281 to S283:
  • the tactile material attributes in the embodiment of the present application may include: stiffness, friction, tactile texture, temperature, vibration, and one or more of a custom type.
  • the types of tactile material attributes include: high resolution, low resolution, and reference.
  • a three-dimensional mesh can carry multiple different types of tactile material attributes. Therefore, when determining the types of tactile material attributes carried by each three-dimensional mesh in the three-dimensional scene to be rendered according to the scene description file, it is necessary to determine the type of each tactile material attribute one by one.
  • step S281 if the type of the first tactile material attribute carried by the target three-dimensional mesh in the three-dimensional scene to be rendered is reference, the following steps S282 and S283 are performed:
  • the target node description module is a node description module corresponding to the nodes of the target three-dimensional grid
  • the target media reference array is a media reference array corresponding to the tactile media file to which the first tactile material attribute belongs.
  • the target node description module may include multiple media reference arrays, before obtaining the accessor index syntax element in the target media reference array, it is necessary to first determine the tactile media file to which the first tactile material attribute belongs, and then determine the media reference array corresponding to the tactile media file to which the first tactile material attribute belongs as the target media reference array.
  • the scene description file parsing method first determines the types of tactile material attributes carried by each three-dimensional grid in the three-dimensional scene to be rendered according to the scene description file after obtaining the scene description file, and when the type of the first tactile material attribute carried by the target three-dimensional grid in the three-dimensional scene to be rendered is a reference, obtains a node description module corresponding to a node including the target three-dimensional grid, obtains the index value declared by the accessor index syntax element in the target media reference array corresponding to the target tactile media file to which the first tactile material attribute in the MPEG tactile of the target node description module belongs, and obtains the description information of the tactile media accessor used to access the decoded data of the target tactile media file according to the index value declared by the accessor index syntax element.
  • the embodiment of the present application can obtain the index value declared by the accessor index syntax element in the target media reference array of the MPEG tactile media reference list of the target node description module when the type of the first tactile material attribute carried by the target three-dimensional mesh is a reference, and obtain the description information of the tactile media accessor used to access the decoded data of the target tactile media file according to the index value declared by the accessor index syntax element, the embodiment of the present application can construct the tactile media accessor according to the description information of the tactile media accessor, and then access the decoded data of the target tactile media file through the tactile media accessor to obtain the tactile value of the first tactile material attribute. Therefore, the embodiment of the present application can support the acquisition of the tactile value of the tactile material attribute of the reference type, thereby solving the problem that the tactile value of the tactile material attribute of the reference type cannot be obtained in the related art.
  • the present application provides another scene description file parsing method, as shown in FIG. 12 , the scene description file parsing method includes:
  • a scene declaration (senc) and its declared index value can be obtained from the scene description file, and a target scene description module corresponding to the three-dimensional scene to be rendered can be obtained from the scene list of the scene description file based on the scene declaration and its declared index value.
  • the second scene description module can be obtained from the scene list of the scene description file according to the scene declaration and its declared index value as the target scene description module corresponding to the three-dimensional scene to be rendered.
  • node index list of the target scene description module is as follows: 2902. Obtain the index value declared in the node index list (nodes) of the target scene description module.
  • the index value declared in the node index list of the target scene description module that can be obtained is 0.
  • the first node description module is obtained from the node list of the scene description file as the node description module corresponding to each node in the three-dimensional scene to be rendered.
  • S2904 Obtain a mesh index list (mesh) of a node description module corresponding to each node in the three-dimensional scene to be rendered.
  • a node description module corresponding to a certain node is as follows:
  • the obtained mesh index list of the node description module corresponding to the node is "mesh":0,1.
  • the grid index list declaration index values of the node description module corresponding to node 1 include: 0 and 1
  • the grid index list declaration index values of the node description module corresponding to node 2 include: 1, 2, 3, then the first grid description module, the second grid description module, the third grid description module and the fourth grid description module can be obtained from the grid list of the scene description file.
  • the mesh description module in the three-dimensional scene to be rendered may be as follows:
  • the mesh description module in the three-dimensional scene to be rendered may also be as follows:
  • the index value declared by the array index syntax element in the MPEG tactile material of the mesh description module corresponding to a certain three-dimensional mesh is 1, and the tactile material array list of the scene description file is as follows:
  • the second tactile material array can be obtained from the tactile material array list of the scene description file according to the index value declared by the array index syntax element in the MPEG tactile material of the mesh description module corresponding to the three-dimensional mesh. Therefore, the tactile material attribute array of the three-dimensional mesh obtained from the tactile material array list of the scene description file is as follows:
  • step S2908 determining the type of each tactile material attribute carried by each three-dimensional grid according to the tactile material attribute array corresponding to each three-dimensional grid
  • steps 29081 to 29083 may include the following steps 29081 to 29083:
  • Step 29081 Obtain a tactile material attribute description module corresponding to each tactile material attribute carried by each three-dimensional grid from the tactile material attribute array corresponding to each three-dimensional grid.
  • the tactile material attribute array of the three-dimensional grid is as follows:
  • the tactile material attribute description module corresponding to the temperature is as follows:
  • Step 29082 Obtain the value of the attribute type syntax element (type) in the tactile material attribute description module corresponding to each tactile material attribute.
  • the value of the attribute type syntax element in the tactile material attribute description module corresponding to the temperature can be determined to be Reference according to "type": Reference in the n+18th line.
  • Step 29083 Attribute type syntax elements in the tactile material attribute description module corresponding to each tactile material attribute The value of determines the type of each tactile material attribute carried by each three-dimensional mesh.
  • the value of the attribute type syntax element in the tactile material attribute description module corresponding to the temperature is Reference. Therefore, the type of temperature carried by the three-dimensional grid can be determined as a reference according to the value of the attribute type syntax element in the tactile material attribute description module corresponding to each tactile material attribute.
  • step S2908 if the type of the first tactile material attribute carried by the target three-dimensional mesh in the three-dimensional scene to be rendered is a reference, the following steps are performed:
  • S2909 Determine the node description module corresponding to the node including the target three-dimensional grid as the target node description module.
  • each node description module determines whether the index value declared in the grid index list of each node description module includes the index value of the grid description module corresponding to the target three-dimensional grid; if included, the node description module is determined as the target node description module.
  • the tactile material attribute description module corresponding to a certain tactile material attribute is as follows:
  • the index value declared by the texture index syntax element (index) in the texture array (texture) of the tactile material attribute description module corresponding to the tactile material attribute is 2.
  • the third texture description module is obtained from the texture list (textures) of the scene description file as the texture description module corresponding to the tactile material attribute.
  • the texture description module corresponding to the tactile material attribute is as follows:
  • the index value declared by the texture map syntax element (source) in the texture description module corresponding to the tactile material attribute may be obtained as 1.
  • the reference value is used to obtain the access address of the texture map of the first tactile material attribute from the texture map list (images) of the scene description file.
  • the access address of the second texture map is obtained from the texture map list (images) of the scene description file as the access address of the texture map of the tactile material attribute.
  • S2914 Acquire the texture map of the first tactile material attribute according to the access address of the texture map of the first tactile material attribute.
  • obtaining the texture map of the first tactile material attribute according to the access address of the texture map of the first tactile material attribute includes: downloading a tactile media file from a server using a network transmission service according to the access address of the texture map of the first tactile material attribute.
  • acquiring the texture map of the first tactile material attribute according to the access address of the texture map of the first tactile material attribute includes: reading a tactile media file from a local storage space according to the access address of the texture map of the first tactile material attribute.
  • S2915 Obtain an index value of a target media description module corresponding to the target tactile media file according to the texture map of the first tactile material attribute.
  • the embodiment of the present application can obtain the index value of the target media description module corresponding to the target tactile media file according to the texture map of the first tactile material attribute.
  • S2916 Determine a media reference array whose value of the tactile media identification syntax element in the target node description module is an index value of the target media description module as the target media reference array.
  • the index value of the target media description module is 0, and the media reference list (Media_reference) in the MPEG haptic of the target node description module is as follows:
  • haptic_id the value of the haptic media identification syntax element (haptic_id) in the first media reference array in the media reference list (Media_reference) is the same as the index value of the target media description module, both are 0, the first media reference array in the media reference list (Media_reference) is determined as the target media reference array.
  • S2917 Get the index value declared by the accessor index syntax element in the target media reference array of the MPEG tactile media reference list of the target node description module.
  • the accessor index syntax element in the target media reference array of the MPEG tactile media reference list of the target node description module and its value is "accessor":1
  • the index value declared by the accessor index syntax element in the target media reference array of the MPEG tactile media reference list of the target node description module is 1.
  • the first accessor description module in the accessor list of the scene description file is used as the accessor description module corresponding to the tactile media accessor.
  • the third accessor description module in the accessor list of the scene description file is used as the accessor description module corresponding to the tactile media accessor.
  • the accessor description module corresponding to the tactile media accessor is as follows:
  • the information that can be obtained by parsing the accessor description module corresponding to the tactile media accessor includes: the data type is 5123; the accessor type is a scalar; the data quantity is 1000; the accessor is a time-varying accessor based on MPEG extension modification; the access object is the cache slice corresponding to the second cache slice description module in the cache slice list; the parameters of the accessor will change over time.
  • the description information of the tactile media accessor capable of accessing the tactile value of the first tactile material attribute can be obtained, and then the tactile media accessor is constructed according to the description information of the tactile media accessor to read the tactile value of the first tactile material attribute.
  • the scene description file parsing method provided in the embodiments of the present application further includes:
  • the sampler description module corresponding to the sampler of the first tactile material attribute obtains the sampler description module corresponding to the sampler of the first tactile material attribute from the sampler list (samplers) of the scene description file; and according to the sampler description module corresponding to the sampler of the first tactile material attribute, obtain the description information of the sampler of the first tactile material attribute.
  • the first sampler description module is obtained from the sampler list of the scene description file, and the description information of the sampler of the first tactile material attribute is obtained according to the first sampler description module.
  • the scene description file parsing method provided in the embodiments of the present application further includes the following steps I to III:
  • Step I According to the index value declared by the texture coordinate index syntax element (texCoord) in the texture array (texture) of the tactile material attribute description module corresponding to the first tactile material attribute, obtain the texture coordinate syntax element (TEXCOORD_n) corresponding to the first tactile material attribute from the attributes (attributes) of the primitives (primitives) of the mesh description module corresponding to the target three-dimensional mesh.
  • TEXCOORD_n texture coordinate syntax element
  • the texture coordinate index syntax element (texCoord) in the texture array (texture) of the tactile material attribute description module corresponding to the first tactile material attribute and its value are: "texCoord":0
  • the first texture coordinate syntax element is obtained from the attributes (attributes) of the primitives (primitives) of the mesh description module corresponding to the target three-dimensional mesh.
  • TEXCOORD_0 is the texture coordinate syntax element corresponding to the first tactile material attribute.
  • Step II according to the index value declared by the texture coordinate syntax element corresponding to the first tactile material attribute, obtain the accessor description module corresponding to the texture coordinate accessor of the first tactile material attribute.
  • the texture coordinate syntax element corresponding to the first tactile material attribute and its value are: "TEXCOORD_0":0, then the first accessor description module is obtained from the accessor list (accessors) of the scene description file as the accessor description module corresponding to the texture coordinate accessor of the first tactile material attribute.
  • the accessor description module corresponding to the texture coordinate accessor of the first tactile material attribute is as follows:
  • the information that can be obtained by the accessor description module corresponding to the texture coordinate accessor that parses the first tactile material attribute includes: the access object is the cache slice corresponding to the first cache slice description module in the cache slice list, the data type is 5121; the accessor type is a two-dimensional vector; and the number of data is 500.
  • the description information of the texture coordinate accessor that can access the texture coordinates corresponding to the first tactile material attribute can be obtained, and then the texture coordinate accessor is constructed according to the description information of the texture coordinate accessor to read the texture coordinates corresponding to the first tactile material attribute.
  • the scene description file parsing method further includes:
  • the index value declared by the texture index syntax element (index) in the texture array (texture) of the tactile material attribute description module corresponding to the second tactile material attribute obtains the texture description module corresponding to the second tactile material attribute from the texture list (textures) of the scene description file according to the index value declared by the texture index syntax element in the texture array of the tactile material attribute description module corresponding to the second tactile material attribute; obtain the index value declared by the texture map syntax element (source) and the index value declared by the sampler syntax element (sampler) in the texture description module corresponding to the second tactile material attribute; obtain the access address of the texture map of the second tactile material attribute from the texture map list (images) of the scene description file according to the index value declared by the texture map syntax element (source) in the texture description module corresponding to the second tactile material attribute, and obtain the description information of the sampler of the second tactile material attribute from the sampler list (samplers) of the scene description file according to the index value declared by the sampler syntax
  • the access address of the texture map of the tactile material attribute can be directly obtained from the texture map of the tactile material attribute through the above embodiment, and then the texture map of the tactile material attribute is obtained, and the tactile value of the tactile material attribute is obtained.
  • the scene description file parsing method further includes:
  • the description information of the cache includes at least one of the following:
  • the capacity of the buffer the access address of the data cached in the buffer, whether it is an MPEG circular buffer, the number of storage links in the circular buffer, the index value of the media description module corresponding to the media file cached in the circular buffer, and the track index value of the data of the media file cached in the circular buffer.
  • a buffer description module corresponding to a certain buffer is as follows:
  • the description information of the buffer can be obtained, including: the capacity of the buffer is 2000 bytes; the buffer is a circular buffer based on MPEG extension modification, the number of storage links of the circular buffer is 3, the media file stored in the circular buffer is the first media file declared in the MPEG media, and the track index value of the data of the media file cached by the circular buffer is 1.
  • a buffer description module corresponding to a certain buffer is as follows:
  • the description information of the buffer that can be obtained according to the buffer description module corresponding to the buffer includes: the capacity of the buffer is 1000 bytes; and the access address of the data cached by the buffer is AnimatedBody.bin.
  • the scene description file parsing method further includes:
  • the description information of the cache slice includes at least one of the following:
  • the cache to which the cache slice belongs the capacity of the cache slice, and the offset of the cache slice.
  • the cache slice description module corresponding to a cache slice is as follows:
  • the description information of the cache slice can be obtained, including: the cache slice is the cache slice of the cache corresponding to the third cache description module in the cache list, the capacity of the cache slice is 4000 bytes, and the offset of the cache slice is 0.
  • the scene description file parsing method further includes: acquiring an accessor description module corresponding to each accessor of the to-be-rendered three-dimensional scene from an accessor list of the scene description file; acquiring description information of each accessor according to the accessor description module corresponding to each accessor;
  • the description information of the accessor includes at least one of the following:
  • the cache slice description module corresponding to a cache slice is as follows:
  • the description information of the accessor obtained by parsing the accessor description module corresponding to the accessor includes: the data type of the data accessed by the accessor is 5123, and the type of the accessor is scalar; the number of the accessors is 1000, and the accessor is a time-varying accessor based on MPEG extension modification; the cache slice accessed by the accessor is the cache slice corresponding to the cache slice description module with an index value of 1; the parameters of the accessor will change over time.
  • the cache slice description module corresponding to a cache slice is as follows:
  • the description information of the accessor obtained by parsing the accessor description module corresponding to the accessor includes: the cache slice accessed by the accessor is the cache slice corresponding to the cache slice description module with an index value of 0; the data type of the data accessed by the accessor is 5121; the type of the accessor is a two-dimensional vector; and the number of accessors is 500.
  • the scene description file parsing method further includes: obtaining a media description module corresponding to each media file in the three-dimensional scene to be rendered from a media list of an MPEG media extension of the scene description file; and obtaining description information of each media file according to the media description module corresponding to each media file.
  • the description information of the media file includes at least one of the following:
  • the name of the media file whether the media file is automatically played, whether the media file is played in a loop, the type of the media file, the access address of the media file, the track information of the encapsulated file of the media file, and the encoding and decoding parameters of the media file.
  • a media description module corresponding to a certain media file is shown below:
  • the description information of the media file obtained according to the media description module corresponding to the media file includes: the name of the media file is Hapticexample; the media file is played automatically; the media file is played in a loop, and the type of the media file is a tactile media file that only includes tactile data; the access address of the media file is http://www.example.com/Hapticexample1.mp4, the index value of the code stream track of the encapsulation file of the media file is 1, and the encoding and decoding parameters of the media file are mih1.oo.
  • the scene description file parsing method also includes: obtaining a scene declaration (scene) from the scene description file, and obtaining a target scene description module corresponding to the three-dimensional scene to be rendered from a scene list (scenes) in the scene description file according to an index value declared in the scene declaration.
  • the first scene description module is obtained from the scene list (scenes) of the scene description file as the target scene description module corresponding to the three-dimensional scene to be rendered.
  • the scene description file parsing method further includes: obtaining a used extension description module (extensionsUsed) in the scene description file, and obtaining the MPEG extension used in the scene description file according to the used extension description module.
  • a used extension description module extensionsUsed
  • the MPEG extensions used in the scene description file can be obtained, including: MPEG media (MPEG_media), MPEG circular buffer (MPEG_buffer_circular), MPEG time-varying accessor (MPEG_accessor_timed), MPEG tactile (MPEG_Haptic) and MPEG tactile material (MPEG_material_Haptic); a total of 5 MPEG extensions.
  • MPEG media MPEG_media
  • MPEG circular buffer MPEG_buffer_circular
  • MPEG time-varying accessor MPEG_accessor_timed
  • MPEG tactile MPEG_Haptic
  • MPEG tactile material MPEG_material_Haptic
  • the scene description file parsing method further includes: obtaining a digital asset description module (version) in the scene description file, and determining version information of the scene description file according to the digital asset description module.
  • version digital asset description module
  • a corresponding scene description file parser is selected based on the version information of the scene description file to parse the scene description file, thereby ensuring that the scene description file is correctly parsed.
  • Some embodiments of the present application also provide a method for rendering a three-dimensional scene.
  • the execution subject of the method for rendering a three-dimensional scene is For the display engine in the immersive media description framework, as shown in FIG13 , the rendering method of the three-dimensional scene includes the following steps:
  • S3001. Determine, according to a scene description file, the types of tactile material attributes carried by each three-dimensional mesh in a three-dimensional scene to be rendered.
  • the tactile material attributes in the embodiments of the present application may include: one or more of: stiffness, friction, vibrotactile texture, temperature, vibration, or customization.
  • the types of tactile material attributes include: high resolution, low resolution, and reference.
  • a three-dimensional mesh can carry multiple different types of tactile material attributes. Therefore, when determining the types of tactile material attributes carried by each three-dimensional mesh in the three-dimensional scene to be rendered according to the scene description file, it is necessary to determine the type of each tactile material attribute one by one.
  • step S3001 determining the tactile material attributes carried by each three-dimensional mesh in the three-dimensional scene to be rendered according to the scene description file
  • step S3004 if it is determined that the type of the first tactile material attribute carried by the target three-dimensional mesh in the three-dimensional scene to be rendered is reference, the following steps S3002 to S3004 are executed:
  • S3002 Determine a haptic media accessor according to an index value declared by an accessor index syntax element (accessor) in a target media reference array of an MPEG haptic media reference list (Media_reference) of a target node description module.
  • the target node description module is a node description module corresponding to the nodes of the target three-dimensional grid
  • the target media reference array is a media reference array corresponding to the tactile media file to which the first tactile material attribute belongs.
  • the implementation method of obtaining the index value declared by the accessor index syntax element (accessor) in the target media reference array of the MPEG tactile media reference list (Media_reference) of the target node description module determined by the display engine in the above step S3002 can refer to the embodiment shown in Figure 12, which will not be repeated here.
  • S3003 Access data in a cache slice of a tactile media buffer through the tactile media accessor to obtain a tactile value of the first tactile material attribute.
  • the tactile media buffer is a buffer for buffering decoded data of the target tactile media file.
  • the decoded tactile value of the first tactile material attribute that can be used for rendering can be directly obtained by accessing the data in the cache slice of the tactile media buffer through the tactile media accessor.
  • S3004 Render the to-be-rendered three-dimensional scene according to the tactile value of the first tactile material attribute.
  • the rendering method of a three-dimensional scene first determines the types of tactile material attributes carried by each three-dimensional grid in the three-dimensional scene to be rendered according to the scene description file after obtaining the scene description file, and when the type of the first tactile material attribute carried by the target three-dimensional grid in the three-dimensional scene to be rendered is a reference, obtains a node description module corresponding to a node including the target three-dimensional grid, obtains the index value declared by the accessor index syntax element in the target media reference array corresponding to the target tactile media file to which the first tactile material attribute in MPEG tactile of the target node description module belongs, and determines a tactile media accessor according to the index value declared by the accessor index syntax element, and then accesses data in a cache slice of a tactile media cache for caching the target tactile media file through the tactile media accessor to obtain the tactile value of the first tactile material attribute, and finally renders the three-dimensional scene to be rendered according to the tactile value of the first tactile material attribute.
  • the embodiment of the present application can obtain the index value declared by the accessor index syntax element in the target media reference array of the MPEG tactile media reference list of the target node description module when the type of the first tactile material attribute carried by the target three-dimensional mesh is a reference, and determine the tactile media according to the index value declared by the accessor index syntax element.
  • the body accessor obtains the tactile value of the first tactile material attribute through the tactile media accessor, and renders the three-dimensional scene to be rendered according to the tactile value of the first tactile material attribute.
  • the embodiment of the present application can support the acquisition of the tactile value of the tactile material attribute of the reference type, thereby solving the problem that the tactile value of the tactile material attribute of the reference type cannot be acquired in the related technology.
  • Some embodiments of the present application further provide a method for processing a tactile media file.
  • the execution subject of the method for processing a tactile media file is a media access function in an immersive media description framework. As shown in FIG. 14 , the method for processing a tactile media file includes the following steps:
  • S3101 Receive description information of a tactile media file, description information of a tactile media buffer, and description information of a cache slice of the tactile media buffer sent by a display engine.
  • the tactile media file includes a tactile value of a first tactile material attribute of a reference type.
  • the description information of the haptic media file may include at least one of the following:
  • the name of the media file whether the media file is automatically played, whether the media file is played in a loop, the type of the media file, the access address of the media file, the track information of the encapsulated file of the media file, and the encoding and decoding parameters of the media file.
  • the description information of the tactile media buffer may include at least one of the following:
  • the capacity of the buffer whether it is an MPEG circular buffer, the number of storage links in the circular buffer, the index value of the media description module corresponding to the media file cached in the circular buffer, and the track index value of the data of the media file cached in the circular buffer.
  • the description information of the cache slice of the tactile media buffer may include at least one of the following:
  • the cache to which the cache slice belongs the capacity of the cache slice, and the offset of the cache slice.
  • S3102 Acquire decoded data corresponding to the tactile media file according to the description information of the tactile media file.
  • the media access function obtains the decoded data corresponding to the tactile media file according to the description information of the tactile media file, including:
  • a tactile media pipeline for processing the tactile media file is created according to the description information of the tactile media file, the tactile media file is obtained through the tactile media pipeline, and the tactile media file is decapsulated and decoded to obtain decoded data corresponding to the tactile media file.
  • the display engine can determine the tactile media accessor according to the index value declared by the accessor index syntax element in the target media reference array of the MPEG tactile media reference list of the target node description module, and obtain the tactile value of the first tactile material attribute by accessing the data in the cache slice of the tactile media buffer through the tactile media accessor.
  • the target node description module is a node description module corresponding to the node of the target three-dimensional grid carrying the first tactile material attribute
  • the target media reference array is a media reference array corresponding to the tactile media file.
  • the processing method of the tactile media file receives the description information of the tactile media file, the description information of the tactile media buffer and the description information of the cache slice of the tactile media buffer sent by the display engine, obtains the decoded data corresponding to the tactile media file according to the description information of the tactile media file, and writes the decoded data into the cache slice of the tactile media buffer according to the description information of the tactile media buffer and the description information of the cache slice of the tactile media buffer. Therefore, the display engine can determine the tactile media access according to the index value declared by the accessor index syntax element in the target media reference array of the MPEG tactile media reference list of the target node description module.
  • the tactile media accessor is used to access the data in the cache slice of the tactile media buffer through the tactile media accessor to obtain the tactile value of the first tactile material attribute. Therefore, the embodiment of the present application can support the acquisition of the tactile value of the tactile material attribute of the reference type, thereby solving the problem that the tactile value of the tactile material attribute of the reference type cannot be acquired in the related art.
  • Some embodiments of the present application further provide a cache management method.
  • the execution subject of the cache management method is a cache management module in the immersive media description framework. As shown in FIG. 15 , the cache management method includes the following steps:
  • S3201 Receive description information of a tactile media buffer and description information of a cache slice of the tactile media buffer.
  • the tactile media buffer is a buffer for caching a target tactile media file, and the target tactile media file includes a first tactile material attribute of a reference type.
  • the description information of the tactile media buffer may include at least one of the following:
  • the capacity of the buffer whether it is an MPEG circular buffer, the number of storage links in the circular buffer, the index value of the media description module corresponding to the media file cached in the circular buffer, and the track index value of the data of the media file cached in the circular buffer.
  • the description information of the cache slice of the tactile media buffer may include at least one of the following:
  • the cache to which the cache slice belongs the capacity of the cache slice, and the offset of the cache slice.
  • S3202 Create the tactile media buffer according to the description information of the tactile media buffer.
  • the description information of the tactile media buffer includes: the capacity of the tactile media buffer is 2000 bytes; the tactile media buffer is a circular buffer based on MPEG extension modification, the number of storage links of the circular buffer is 3, the media file stored in the circular buffer is the first media file declared in the MPEG media, and the track index value of the data of the media file cached by the circular buffer is 1, then the cache management module creates a circular buffer with a capacity of 2000 bytes and 3 storage links as the tactile media buffer.
  • S3203 Divide the tactile media buffer into cache slices according to the description information of the cache slices of the tactile media buffer.
  • the track index value of the data of the media file cached by the circular buffer is 1.
  • the cache management module creates a circular buffer with a capacity of 2000 bytes and 3 storage links as the tactile media buffer, if the description information of the cache slice of the tactile media buffer includes: the capacity is 2000 bytes and the offset is 0, the tactile media buffer is divided into 1 cache slice.
  • the media access function can write the decoded data of the target tactile media file into the cache slices of the tactile media buffer
  • the display engine can determine the tactile media accessor according to the index value declared by the accessor index syntax element in the target media reference array of the MPEG tactile media reference list of the target node description module, and obtain the tactile value of the first tactile material attribute by accessing the data in the cache slice of the tactile media buffer through the tactile media accessor.
  • the target node description module is a node description module corresponding to the node of the target three-dimensional grid carrying the first tactile material attribute
  • the target media reference array is a media reference array corresponding to the target tactile media file.
  • the cache management method After receiving the description information of the tactile media buffer and the description information of the cache slice of the tactile media buffer, the cache management method provided in the embodiment of the present application can create the tactile media buffer according to the description information of the tactile media buffer, and divide the tactile media buffer into cache slices according to the description information of the cache slice of the tactile media buffer, so that the media access function can include the tactile value of the first tactile material attribute of the reference type.
  • the decoded data of the tactile media file is written into the cache slice of the tactile media buffer, and the display engine can determine the tactile media accessor according to the index value declared by the accessor index syntax element in the target media reference array of the MPEG tactile media reference list of the target node description module, and obtain the tactile value of the first tactile material attribute by accessing the data in the cache slice of the tactile media buffer through the tactile media accessor. Therefore, the embodiment of the present application can support the acquisition of the tactile value of the tactile material attribute of the reference type, solving the problem that the tactile value of the tactile material attribute of the reference type cannot be obtained in the related art.
  • Some embodiments of the present application also provide a method for rendering a three-dimensional scene, which includes: a scene description file parsing method and a three-dimensional scene rendering method executed by a display engine, a tactile media file processing method executed by a media access function, and a cache management method executed by a cache management module.
  • the target tactile media file is included, and the type of the first tactile material attribute of the target tactile media file is used as an example, and the buffer used to cache the target tactile media file is a tactile media buffer, the buffer used to access the target tactile media file is a tactile media accessor, the grid carrying the first tactile material attribute is a target three-dimensional grid, the buffer used to cache the texture coordinates of the target three-dimensional grid is a texture coordinate buffer, and the accessor used to access the texture coordinates of the target three-dimensional grid is a texture coordinate accessor to illustrate the solution provided by the embodiments of the present application.
  • the method includes the following steps:
  • the method includes the following steps:
  • the display engine obtains a scene description file.
  • the display engine obtains a scene description file of a scene to be rendered, including: the display engine downloads the scene description file from a server using a network transmission service.
  • the display engine obtains a scene description file of a scene to be rendered, including: reading the scene description file from a local storage space.
  • the display engine obtains the media description module corresponding to each media file from the media list (media) of the MPEG media of the scene description file (including: obtaining the media description module corresponding to the target tactile media file from the media list of the MPEG media of the scene description file).
  • the display engine obtains description information of each media file according to the media description module corresponding to each media file (including: obtaining description information of the target tactile media file according to the media description module corresponding to the target tactile media file).
  • the description information of the media file includes at least one of the following:
  • the name of the media file whether the media file is played automatically, whether the media file is played in a loop, the type of the body file, the access address of the media file, the track information of the encapsulated file of the media file, and the encoding and decoding parameters of the media file.
  • the display engine sends description information of each media file to the media access function (sends description information of the target tactile media file to the media access function).
  • the media access function receives the description information of each media file sent by the display engine.
  • the display engine sends the description information of each media file to the media access function, including: the display engine sends the description information of each media file to the media access function through the media access function API.
  • the media access function receives the description information of each media file sent by the display engine, including: the media access function receives the description information of each media file sent by the display engine through the media access function API.
  • the media access function creates a management function for processing each media file according to the description information of each media file.
  • a haptic media pipeline for processing the target haptic media file is created according to the target haptic media file description information.
  • the tactile media pipeline includes: an input module, a decapsulation module, a decoding module and a post-processing module; the input module is used to obtain the target tactile media file (encapsulated file), the decapsulation module is used to decapsulate the target tactile media file, the decoding module is used to decode the decapsulated code stream, and the post-processing module is used to perform post-processing such as format conversion on the decoded data to obtain the decoded data of the target tactile media file.
  • the input module is used to obtain the target tactile media file (encapsulated file)
  • the decapsulation module is used to decapsulate the target tactile media file
  • the decoding module is used to decode the decapsulated code stream
  • the post-processing module is used to perform post-processing such as format conversion on the decoded data to obtain the decoded data of the target tactile media file.
  • the media access function obtains each media file through pipeline processing corresponding to each media file, and decapsulates and decodes each media file to obtain decoded data corresponding to each media file. (including obtaining the target tactile media file through the tactile media pipeline, and decapsulating and decoding the target tactile media file to obtain decoded data corresponding to the target tactile media file).
  • the description information of the target tactile media file includes an access address of the target tactile media file
  • the above step S3306 (the media access function obtains each media file through pipeline processing corresponding to each media file, and decapsulates and decodes each media file to obtain decoded data corresponding to the media file) includes:
  • the media access function obtains the target tactile media file according to the access address of the target tactile media file.
  • the media access function obtains the target tactile media file according to the access address of the target tactile media file, including: the media access function sends a media resource request to a media resource server according to the access address of the target tactile media file, and receives a media resource response sent by the media server carrying the target tactile media file.
  • the media access function obtains the target tactile media file according to the access address of the target tactile media file, including: the media access function reads the target tactile media file from a preset storage space according to the access address of the target tactile media file.
  • the description information of the target tactile media file also includes index values of each code stream track of the target tactile media file, and the above step S3306 (the media access function obtains each media file through pipeline processing corresponding to each media file, and decapsulates and decodes each media file to obtain decoded data corresponding to the media file) includes:
  • the media access function decapsulates the target tactile media file according to the index values of each bitstream track of the target tactile media file to obtain the bitstream of each bitstream track of the target tactile media file.
  • the description information of the target tactile media file also includes encoding and decoding parameters of each stream track of the target tactile media file;
  • the above step S3306 (the media access function obtains each media file through pipeline processing corresponding to each media file, and decapsulates and decodes each media file to obtain decoded data corresponding to the media file) includes:
  • the media access function decodes the code streams of each code stream track of the target tactile media file according to the encoding and decoding parameters of each code stream track of the target tactile media file, and obtains the corresponding decoded data of each code stream track of the target tactile media file, so as to obtain the decoded data corresponding to the target tactile media file.
  • the display engine obtains a buffer description module corresponding to each buffer from the buffer list (buffers) of the scene description file (including: obtaining a buffer description module corresponding to the tactile media buffer for caching the target tactile media file from the buffer list of the scene description file, and a buffer description module for caching the first tactile material attribute; A buffer description module corresponding to the texture coordinate buffer of the texture coordinates of the three-dimensional mesh).
  • the display engine obtains description information of each buffer according to the buffer description module corresponding to each buffer (including: obtaining description information of the tactile media buffer according to the buffer description module corresponding to the tactile media buffer, and obtaining description information of the texture coordinate buffer according to the buffer description module corresponding to the texture coordinate buffer).
  • the description information of the cache may include at least one of the following:
  • the capacity of the buffer the access address of the data cached in the buffer, whether it is an MPEG circular buffer, the number of storage links in the circular buffer, the index value of the media description module corresponding to the media file cached in the circular buffer, and the track index value of the data of the media file cached in the circular buffer.
  • the display engine obtains the cache slice description modules corresponding to the cache slices of each buffer from the cache slice list (bufferViews) of the scene description file (including: obtaining the cache slice description modules corresponding to the cache slices of the tactile media buffer and the cache slice description modules corresponding to the cache slices of the texture coordinate buffer from the cache slice list of the scene description file).
  • the display engine obtains description information of the cache slices of each buffer according to the cache slice description module corresponding to the cache slices of each buffer (including: obtaining description information of the cache slices of the tactile media buffer according to the cache slice description module corresponding to the cache slices of the tactile media buffer, and obtaining description information of the cache slices of the texture coordinate buffer according to the cache slice description module corresponding to the cache slices of the texture coordinate buffer).
  • the cache to which the cache slice belongs the capacity of the cache slice, and the offset of the cache slice.
  • the embodiment of the present application may choose to execute the following scheme 1 or scheme 2 to send the description information of each cache and the description information of the cache slices of each cache to the media access function and the cache management module.
  • Solution 1 includes the following steps S3311 and S3312:
  • the display engine sends the description information of each buffer and the description information of the cache slices of each buffer to the media access function (including: the display engine sends the description information of the tactile media buffer and the description information of the cache slices of the tactile media buffer, as well as the description information of the texture coordinate buffer and the description information of the cache slices of the texture coordinate buffer to the media access function).
  • the media access function receives the description information of each buffer and the description information of the cache slices of each buffer sent by the display engine (including: the media access function receives the description information of the tactile media buffer and the description information of the cache slices of the tactile media buffer, as well as the description information of the texture coordinate buffer and the description information of the cache slices of the texture coordinate buffer sent by the display engine).
  • the implementation of the above step S3311 (the display engine sends the description information of each buffer and the description information of the cache slice of each buffer to the media access function) can be: the display engine sends the description information of each buffer and the description information of the cache slice of each buffer to the media access function through the media access function API.
  • the implementation of the media access function receiving the description information of each buffer sent by the display engine can be: the media access function receives the description information of each buffer and the description information of the cache slice of each buffer sent by the display engine through the media access function API.
  • the media access function sends the description information of each buffer and the description information of the buffer slice of each buffer to the buffer management module (including: the media access function sends the description information of the tactile media buffer to the buffer management module, description information of a cache slice of the tactile media buffer, description information of the texture coordinate buffer, and description information of a cache slice of the texture coordinate buffer).
  • the cache management module receives description information of each buffer sent by the media access function (including: the media access function receives description information of the tactile media buffer sent by the display engine, description information of the cache slice of the tactile media buffer, description information of the texture coordinate buffer, and description information of the cache slice of the texture coordinate buffer).
  • the implementation of the above step S3311 (the media access function sends the description information of each buffer and the description information of the cache slice of each buffer to the cache management module) can be: the media access function sends the description information of each buffer and the description information of the cache slice of each buffer to the cache management module through the cache API.
  • the implementation of the cache management module receiving the description information of each buffer and the description information of the cache slice of each buffer sent by the media access function can be: the cache management module receives the description information of each buffer and the description information of the cache slice of each buffer sent by the media access function through the cache API.
  • Solution 2 includes the following steps S3313 and S3314:
  • the display engine sends description information of each buffer and description information of the cache slices of each buffer to the media access function.
  • the media access function receives description information of each buffer and description information of the buffer slice of each buffer sent by the display engine.
  • the display engine sends description information of each buffer and description information of the cache slices of each buffer to the cache management module.
  • step S3314 the display engine sends the description information of each buffer and the description information of the buffer slice of each buffer and the description information of the buffer slice of the tactile media buffer to the buffer management module
  • the display engine sends the description information of each buffer and the description information of the buffer slice of each buffer to the buffer management module through the buffer API
  • the implementation of the buffer management module receiving the description information of each buffer sent by the display engine can be: the buffer management module receives the description information of each buffer and the description information of the buffer slice of each buffer sent by the display engine through the buffer API.
  • the cache management module constructs each cache according to the description information of each cache (including constructing the tactile media cache according to the description information of the tactile media cache, and constructing the texture coordinate cache according to the description information of the texture coordinate cache).
  • the cache management module divides each cache into cache slices according to the description information of the cache slices of each cache (including dividing the cache slices of the tactile media cache according to the description information of the cache slices of the tactile media cache, and dividing the cache slices of the texture coordinate cache according to the description information of the cache slices of the texture coordinate buffer).
  • the media access function writes the decoded data corresponding to each media file into the cache slice of the buffer corresponding to each media file according to the description information of each buffer and the description information of the cache slice of each buffer (including: writing the decoded data of the target tactile media file into the cache slice of the tactile media buffer).
  • the media access function obtains the texture coordinates of the target three-dimensional mesh according to the access address of the texture coordinates of the target three-dimensional mesh.
  • the media access function writes the texture coordinates of the target three-dimensional mesh into the cache slice of the texture coordinate buffer according to the description information of the texture coordinate buffer and the description information of the cache slice of the texture coordinate buffer.
  • the display engine obtains the accessor description module corresponding to each accessor from the accessor list (accessors) of the scene description file (including: the display engine obtains the accessor description module corresponding to the tactile media accessor for accessing the target tactile media file from the accessor list of the scene description file, and obtains the accessor description module corresponding to the texture coordinate accessor for accessing the texture coordinates).
  • the display engine obtains description information of each accessor according to the accessor description module corresponding to each accessor (including: the display engine obtains description information of the tactile media accessor according to the accessor description module corresponding to the tactile media accessor, and obtains description information of the texture coordinate accessor according to the accessor description module corresponding to the texture coordinate accessor).
  • the description information of the accessor includes at least one of the following:
  • the display engine creates each accessor according to the description information of each accessor (including: the display engine creates the tactile media accessor according to the description information of the tactile media accessor, and creates the texture coordinate accessor according to the description information of the texture coordinate accessor).
  • the display engine determines, according to the scene description file, the types of tactile material attributes carried by each three-dimensional mesh in the three-dimensional scene to be rendered.
  • step S3323 if the display engine determines that the type of the first tactile material attribute carried by the target three-dimensional mesh in the three-dimensional scene to be rendered is a reference, the following steps are performed:
  • the display engine determines the tactile media accessor according to the index value declared by the accessor index syntax element in the target media reference array of the MPEG tactile media reference list of the target node description module.
  • the target node description module is a node description module corresponding to the nodes of the target three-dimensional grid
  • the target media reference array is a media reference array corresponding to the target tactile media file to which the first tactile material attribute belongs.
  • S3325 The display engine accesses the data in the cache slice of the tactile media buffer through the tactile media accessor to obtain the tactile value of the first tactile material attribute.
  • the display engine obtains the description module of the sampler of each tactile material attribute from the sampler list (samplers) of the scene description file (including: the display engine obtains the description module of the sampler of the first tactile material attribute from the sampler list of the scene description file).
  • the display engine obtains description information of the samplers of each tactile material attribute according to the description module of the samplers of each tactile material attribute (including: the display engine obtains description information of the sampler of the first tactile material attribute according to the description module of the sampler of the first tactile material attribute).
  • the display engine constructs the samplers of each tactile material attribute according to the description information of the samplers of each tactile material attribute (including: the display engine constructs the sampler of the first tactile material attribute according to the description information of the sampler of the first tactile material attribute).
  • S3329 Render the to-be-rendered three-dimensional scene according to the tactile values of each tactile material attribute, the texture coordinates corresponding to the first tactile material attribute, and the sampler of the first tactile material attribute.
  • some embodiments of the present application provide a scene description file parsing device, and the scene description file generation device includes:
  • a memory configured to store a computer program
  • the processor is configured to, when calling the computer program, enable the scene description file parsing device to implement the scene description file parsing method described in any of the above embodiments.
  • some embodiments of the present application provide a computer-readable storage medium, on which a computer program is stored.
  • the computer program When executed by a computing device, the computing device implements the scene description file parsing method described in any of the above embodiments.
  • some embodiments of the present application provide a computer program product, which, when executed on a computer, enables the computer to implement the scene description file parsing method described in any of the above embodiments.

Landscapes

  • Engineering & Computer Science (AREA)
  • Theoretical Computer Science (AREA)
  • Physics & Mathematics (AREA)
  • General Physics & Mathematics (AREA)
  • Computer Graphics (AREA)
  • General Engineering & Computer Science (AREA)
  • Multimedia (AREA)
  • Signal Processing (AREA)
  • Software Systems (AREA)
  • Human Computer Interaction (AREA)
  • Geometry (AREA)
  • Computer Security & Cryptography (AREA)
  • Computer Hardware Design (AREA)
  • Processing Or Creating Images (AREA)
  • User Interface Of Digital Computer (AREA)
  • Information Retrieval, Db Structures And Fs Structures Therefor (AREA)
  • Management, Administration, Business Operations System, And Electronic Commerce (AREA)

Abstract

一种场景描述文件的解析方法及装置，包括：根据场景描述文件确定待渲染三维场景中的各个三维网格承载的各个触觉材质属性的类型；当待渲染三维场景中的目标三维网格承载的第一触觉材质属性的类型为参考时，获取目标节点描述模块的MPEG触觉的媒体参考列表的目标媒体参考数组中的访问器索引语法元素声明的索引值；目标节点描述模块为包括目标三维网格的节点对应的节点描述模块，目标媒体参考数组为目标触觉媒体文件对应的媒体参考数组；根据访问器索引语法元素声明的索引值获取触觉媒体访问器的描述信息。

Description

场景描述文件的解析方法及装置
本申请要求在2023年01月10日提交、申请号为202310036790.8的；在2023年03月20日提交、申请号为202310275300.X的；在2023年4月6日提交、申请号为202310362181.1专利申请的优先权，该申请的内容通过引用结合在本申请中。
技术领域
本申请一些实施例涉及视频处理技术领域。尤其，涉及一种场景描述文件的解析方法及装置。
背景技术
随着三维媒体内容的逐渐丰富，用户对虚拟场景的真实感与丰富度也有了更高的要求。触觉作为感官的一部分，成为在沉浸式媒体中需要考虑和添加的对象体验。例如，用户可以在虚拟场景中感受一场爆炸效果，这不仅需要视觉、听觉的参与，也需要触觉作为一种感官体验参与其中，以提高此虚拟爆炸效果带给人的真实感和临场感。
相关技术标准中，每个触觉材质属性使用一个纹理数组来表示，不同属性的纹理数组结合起来组成一种特定触觉感知。每个数组元素都包含一个触觉纹理和一个表示为枚举的触觉材质属性类型，且触觉材质属性类型包括：高分辨率(High_resolution)、低分辨率(Low_resolution)、参考(Reference)以及其他(Other)。在触觉材质属性类型为高分辨率或低分辨率的情况下，纹理索引的纹理贴图或触觉纹理材质数据包中存储的不是颜色值，而是的触觉值。这些触觉值能够直接被显示引擎获取并进行触觉渲染。而当触觉材质属性类型为参考的时，纹理贴图或触觉纹理材质数据包中的每一个像素并不直接存储的触觉值，而是存储着待渲染的触觉媒体文件的索引值。如前所述，当触觉材质属性类型为参考时，从纹理贴图或触觉纹理材质数据包获取到的数据为触觉媒体的索引值，且根据该索引值仅能只能获取到编码并封装后的触觉媒体文件，无法获取解码后的可用于渲染的触觉值，进而导致显示引擎无法正常进行渲染。
发明内容
第一方面，本申请一些实施例提供了一种场景描述文件的解析方法，包括：
根据场景描述文件确定待渲染三维场景中的各个三维网格承载的各个触觉材质属性的类型；
当所述待渲染三维场景中的目标三维网格承载的第一触觉材质属性的类型为参考时，获取目标节点描述模块的动态图像专家组MPEG触觉的媒体参考列表的目标媒体参考数组中的访问器索引语法元素声明的索引值；所述目标节点描述模块为包括所述目标三维网格的节点对应的节点描述模块，所述目标媒体参考数组为所述第一触觉材质属性所属的目标触觉媒体文件对应的媒体参考数组；
根据所述访问器索引语法元素声明的索引值获取用于访问所述目标触觉媒体文件的解码数据的触觉媒体访问器的描述信息。
第二方面，本申请一些实施例提供了一种场景描述文件的解析装置，包括：
存储器，被配置为存储计算机程序；
处理器，被配置为用于在调用计算机程序时，使得所述三维场景的渲染装置实现第一方面所述的场景描述文件的解析方法。
附图说明
图1示出了本申请一些实施例中的沉浸式媒体描述框架的结构示意图；
图2示出了本申请一些实施例中的场景描述文件的结构示意图；
图3示出了本申请另一些实施例中的场景描述文件的结构示意图；
图4示出了本申请另一些实施例中的触觉数据的数据模型示意图；
图5示出了本申请另一些实施例中的人体分层结构的示意图；
图6示出了本申请另一些实施例中的触觉信号编码器的结构示意图；
图7示出了本申请另一些实施例中的触觉信号解码器的结构示意图；
图8示出了本申请另一些实施例中的场景描述文件的结构示意图；
图9示出了本申请另一些实施例中的场景描述文件的结构示意图；
图10示出了本申请一些实施例中的场景描述文件的生成方法的步骤流程图；
图11示出了本申请一些实施例中的场景描述文件解析方法的步骤流程图；
图12示出了本申请另一些实施例中的场景描述文件解析方法的步骤流程图；
图13示出了本申请一些实施例中的触觉媒体文件处理方法的步骤流程图；
图14示出了本申请一些实施例中的三维场景渲染方法的步骤流程图；
图15示出了本申请一些实施例中的缓存管理方法的步骤流程图；
图16示出了本申请一些实施例中的三维场景渲染方法的交互流程图。
具体实施方式
为使本申请的目的和实施方式更加清楚，下面将结合本申请示例性实施例中的附图，对本申请示例性实施方式进行清楚、完整地描述，显然，描述的示例性实施例仅是本申请一部分实施例，而不是全部的实施例。
需要说明的是，本申请中对于术语的简要说明，仅是为了方便理解接下来描述的实施方式，而不是意图限定本申请的实施方式。除非另有说明，这些术语应当按照其普通和通常的含义理解。
术语“包括”和“具有”以及他们的任何变形，意图在于覆盖但不排他的包含，例如，包含了一系列组件的产品或设备不必限于清楚地列出的所有组件，而是可包括没有清楚地列出的或对于这些产品或设备固有的其它组件。
说明书中提及“一些实现方式”、“一些实施例”等是表明所描述的实现方式或实施例可包括特定的特征、结构或特性，但可能不一定每个实施例都包括该特定特征、结构或特性。此外，这种短语不一定指同一实现方式。另外，当联系一实施例来描述特定的特征、结构或特性时，认为联系其他实现方式(无论本文是否明确描述)来实现这种特征、结构或特性，是在本领域技术人员的知识范围内的。
本申请一些实施例涉及沉浸式媒体的场景描述。参照图1所示的沉浸式媒体的场景描述框架，为了使显示引擎11能够专注于媒体渲染，沉浸式媒体的场景描述将媒体文件的访问与处理和媒体文件的渲染进行了解耦，且设计了媒体接入函数(Media Access Function，MAF)12来负责媒体文件的访问与处理功能。同时设计了媒体接入函数应用程序编程接口 (Application Programming Interface，API)，显示引擎11与媒体接入函数12之间通过媒体接入函数API进行指令交互。显示引擎11可以通过媒体接入函数API向媒体接入函数12下达指令，媒体接入函数12也可以通过媒体接入函数API向显示引擎11请求指令。
沉浸式媒体的场景描述框架的一般工作流程包括：1)、显示引擎11获取沉浸式媒体服务方提供的场景描述文件(Scene Description Documents)。2)、显示引擎11解析场景描述文件，获取媒体文件的访问地址、媒体文件的属性信息(媒体类型及编解码参数等)和经过处理的媒体文件的格式要求等参数或信息，并调用媒体接入函数API将解析场景描述文件得到的全部或部分信息传递给媒体接入函数12。3)、媒体接入函数12根据显示引擎11传递的信息，从媒体资源服务器请求下载指定的媒体文件或者从本地获取指定的媒体文件，并为该媒体文件建立相应的管线(pipeline)，随后在管线中对媒体文件进行解封装、解密、解码等处理，以将媒体文件从封装格式转换为显示引擎11规定的格式。4)、管线将完成所有处理得到的输出数据存放到指定的缓存中。5)、最后，显示引擎11在指定的缓存中读取经过完备处理的数据，并根据从缓存中读取的数据进行媒体文件的渲染。
以下进一步对沉浸式媒体的场景描述框架所涉及的文件和功能模块进行说明。
一、场景描述文件
在沉浸式媒体的场景描述框架的工作流程中，场景描述文件用于描述三维场景的结构(可用三维网格描述其特征)、纹理(如纹理贴图等)、动画(旋转、平移)、相机视点位置(渲染视角)等内容。
相关技术领域中，GL传输格式2.0(glTF2.0)已被确定为一种场景描述文件的候选格式，可以满足动态图像专家组-浸入式(MPEG-Immersive，MPEG-I)和六自由度(Degrees of Freedom，6DoF)应用的需求。例如，在github.com/KhronosGroup/glTF/tree/master/specification/2.0#specifying-extensions可获得的Khronos Group的GL传输格式(glTF)版本2.0中，描述了glTF2.0。参照图2所示，图2为glTF2.0场景描述标准(ISO/IEC 12113)中的场景描述文件的结构示意图。如图2所示，glTF2.0场景描述标准中的场景描述文件包括但不限于：场景描述模块(scene)201、节点描述模块(node)202、网格描述模块(mesh)203、访问器描述模块(accessor)204、缓存切片描述模块(bufferView)205、缓存器描述模块(buffer)206、相机描述模块(camera)207、光照描述模块(light)208、材质描述模块(material)209、纹理描述模块(texture)210、采样器描述模块(sampler)211以及纹理贴图描述模块(image)212、动画描述模块(animation)213、蒙皮描述模块(skin)214。
图2所示场景描述文件中的场景描述模块(scene)201用于描述场景描述文件中包含的三维场景。一个场景描述文件中可能包含任意数量的三维场景，每一个三维场景分别使用一个场景描述模块201进行表示。场景描述模块201与场景描述模块201之间为并列关系，即三维场景和三维场景之间为并列关系。
图2所示场景描述文件中的节点描述模块(node)202为场景描述模块201的下一层级描述模块，用于描述场景描述模块201所描述的三维场景中所包含的物体。每一个三维场景中都可能存在许多具体物体，比如虚拟数字人、近处的三维物体、远处的背景图片等，场景描述文件会通过节点描述模块201对这些具体物体进行描述。每个节点描述模块202可以表示 一个物体或者是数个物体组成的一组物体集合，节点描述模块202之间的关系反映了场景描述模块201所描述的三维场景中各个组成部分之间的关系，一个场景描述模块201描述的场景中可以包含一个或多个节点。多个节点之间可以是并列关系或是层级关系，即节点描述模块202之间存在包含与被包含的关系，这使得多个具体物体可以集合在一起进行描述，也可以分别对多个具体物体进行描述。如果一个节点被另一个节点包含，则被包含的节点称为子节点(children)，对于子节点使用"children"替换"node"进行表示。灵活使用节点与子节点进行组合，就可以构成层级化的节点结构，从而表达丰富的场景内容。
图2所示场景描述文件中的网格描述模块(mesh)203为节点描述模块202的下一层级描述模块，用于描述节点描述模块202所代表的物体的特征。网格描述模块203为由一个或多个基元(primitives)组成的集合，每基元又可以包括一个属性(attributes)，基元的属性定义了图形处理器(Graphics Processing Unit，GPU)渲染时需要使用的属性。属性可以包括：position(三维坐标)，normal(法向量)，tangent(切向量)，texcoord_n(纹理坐标)，color_n(颜色：RGB或RGBA)，joints_n(与蒙皮描述模块210相关的属性)，和weights_n(与蒙皮描述模块210相关的属性)等。由于网格描述模块203中包含的顶点数量非常大，每个顶点又包含了多种属性信息，因此不便将媒体文件中包含的大量媒体数据直接存储在场景描述文件的网格描述模块203中，而是在场景描述文件中指出了媒体文件的访问地址(URI)，当需要取用媒体文件中的数据时再进行下载即可，从而实现场景描述文件与媒体文件的分离。所以一般情况下网格描述模块203中不存储媒体数据，而是存储每一个属性对应的访问器描述模块(accessor)204的索引值，并通过访问器描述模块204指向缓存器(buffer)的缓存切片(bufferView)中相应的数据。
在一些实施例中，还可以将场景描述文件与媒体文件融合起来，形成一个二进制文件，从而减少文件的种类与数量。
此外，网格描述模块203的基元的中还可能有模式(mode)这一语法元素。模式语法元素用于描述图形处理器(Graphics Processing Unit，GPU)绘制三维网格时的拓扑结构，比如mode＝0代表散点，mode＝1代表线，mode＝4代表三角形等。
示例性的，以下为一个网格描述模块203的JSON示例：
在上述网格描述模块203中"position"的值为1，指向了索引为1的访问器描述模块，并最终指向的缓存器中存储的顶点坐标数据；"color_0"的值为2，指向了索引为2的访问器描 述模块，并最终指向的缓存器中存储的颜色数据。
网格描述模块203的基元的属性(mash.primitives.attributes)中的语法元素的定义如下表1所示：
表1
网格描述模块203的基元的属性(mash.primitives.attributes)中索引的访问器类型的定义如下表2所示：
表2
网格描述模块203的基元的属性(mash.primitives.attributes)中的分量类型的定义如下表3所示：
表3
图2所示场景描述文件中的访问器描述模块(accessor)204、缓存切片描述模块(bufferView) 205、缓存器描述模块(buffer)206用于实现网格描述模块203对媒体文件的数据的逐层精细化的索引。如上所述，网格描述模块203中不存储具体的媒体数据，而是存储对应的访问器描述模块204的索引值，并通过索引值所索引的访问器描述模块204来访问具体的媒体数据。网格描述模块203对媒体数据的索引过程包括：首先网格描述模块203的基元的属性中的语法元素声明的索引值会指向对应的访问器描述模块204；然后，访问器描述模块204又会指向对应的缓存切片描述模块205；最后，缓存切片描述模块205再指向对应的缓存器描述模块206。图2所示场景描述文件中的缓存器描述模块206主要负责指向对应的媒体文件，包含了媒体文件的URI，媒体文件的字节长度等，用于描述缓存媒体文件的媒体数据的缓存器。一个缓存器可以被划分为多个缓存切片，缓存切片描述模块205主要负责对缓存器中的媒体数据进行部分访问，包含了访问数据的起始字节偏移与访问数据的字节长度等，通过这缓存切片描述模块205以及缓存器描述模块206可以实现对媒体文件的数据的部分访问。访问器描述模块204主要负责为缓存切片描述模块205中划定的部分数据增添附加信息，比如数据类型，该类型数据的数量，该类型数据的数值范围等。这样的三层结构可以实现从一个媒体文件中取用部分数据的功能，有利于数据的精准取用，也便于减少媒体文件的数量。
图2所示场景描述文件中的相机描述模块(camera)207为节点描述模块202的下一层级描述模块，用于描述用户观看节点描述模块202所描述的物体时的视点、视角等与视觉观看相关的信息。为了使用户能够置身于三维场景之中，且能够观看三维场景，节点描述模块202还可以指向相机描述模块207，并通过相机描述模块207描述用户观看节点描述模块202所描述的物体时的视点、视角等与视觉观看相关的信息。
图2所示场景描述文件中的光照描述模块(light)208为节点描述模块202的下一层级描述模块，用于描述节点描述模块202所描述的物体的光照强度、环境光颜色、光照方向、光源位置等与光照相关的信息。
图2所示场景描述文件中的材质描述模块(material)209为网格描述模块203的下一层级描述模块，用于描述网格描述模块203所描述的三维物体的材质信息。在描述的三维物体时，仅借助网格描述模块203描述三维物体的几何信息，或单调地定义三维物体颜色和/或位置，是无法提高三维物体的真实感的，这就需要在三维物体的表面附加更多的信息。对于常规的三维模型技术，这一过程也可以简称为纹理贴图或添加纹理等。glTF2.0场景描述标准中的场景描述文件也沿用了这一描述模块。材质描述模块209使用一组通用参数定义材质，以描述三维场景中出现的几何对象的材质信息。该材质描述模块209普遍使用金属-粗糙度模型(metallic-roughness)描述虚拟物体的材质，基于金属-粗糙度模型的材质特性参数采用广泛使用的基于物理的渲染(Physically Based-Rendering，PBR)的材质表示。基于此，材质描述模块209对物体的金属-粗糙度材质属性做了详细的说明，材质描述模块209中的语法元素的定义如表4所示：
表4
在一些实施例中，材质描述模块209中的PbrMetarialRoughness(material.PbrMetarialRoughness)中的语法元素的定义如下表5所示：
表5
材质描述模块209的PbrMetarialRoughness中的每个属性的值可以使用因子和/或纹理(例如baseColorTexture和baseColorFactor)定义。如果未给定纹理，则必须假设此材质模型中的所有相应纹理组件的值均为1.0。如果同时存在因子和纹理，则因子值充当相应纹理值的线性乘数。纹理绑定由纹理对象的索引和可选的纹理坐标索引定义。
示例性的，以下为一个材质描述模块209的JSON示例：
解析上述材质描述模块209，可以通过材质名称语法元素及其值("name":"gold")，确定当前材质命名为"gold"，再通过pbrMetallicRoughness数组下的颜色语法元素及其值("basecolorFactor":[1.000,0.766,0.336,1.0])，确定当前材质的基础颜色的值为[1.000,0.766,0.336,1.0]，通过pbrMetallicRoughness数组下金属性语法元素及其值("metalnessFactor":1.0)，确定当前材质的金属性值为"1.0"，通过pbrMetallicRoughness数组下的粗糙度语法元素及其值("roughnessFactor":0.0)，确定当前材质的粗糙度值为"0.0"。
图2所示场景描述文件中的纹理描述模块(texture)210为材质描述模块209的下一层级描述模块，用于描述材质描述模块209所描述的三维物体的颜色以及材质定义中使用的其他特性。纹理是赋予对象真实外观的一个重要方面。通过纹理可以定义对象的主颜色以及材质 定义中使用的其他特性，以便精确描述渲染对象的外观。材质本身可以定义多个纹理对象，这些纹理对象可以在渲染期间用作虚拟物体的纹理，并且可以用于编码不同的材质属性。纹理描述模块210使用采样器语法元素和纹理贴图语法元素索引来引用一个采样器描述模块(sampler)211和一个纹理贴图描述模块(image)212。纹理贴图描述模块212包含一个统一资源标识符(Uniform Resource Identifier，URI)，该URI链接到纹理描述模块210实际使用的纹理贴图或二进制文件包。而采样器描述模块211则是用于描述纹理的过滤和包装模式。材质描述模块209、纹理描述模块210、采样器描述模块211以及纹理贴图描述模块212各自的职责与协作关系包括：材质描述模块209与纹理描述模块210一起，定义了物体表面的颜色和物理信息。采样器描述模块211定义了如何将纹理贴图贴在物体表面。纹理描述模块210指定了采样器描述模块212和纹理贴图描述模块212，通过纹理贴图描述模块212实现了添加纹理，而纹理贴图描述模块212则使用URI进行标识和索引，并使用访问器描述模块204进行数据的访问。采样器描述模块211则实现了纹理的具体调整和包装。纹理描述模块210中的语法元素的定义如下表6所示：
表6
在一些实施例中，纹理描述模块210的sample(texture.sample)中的语法元素的定义如下表7所示：
示例性的，以下为一个材质描述模块209、纹理描述模块210、采样器描述模块211以及纹理贴图描述模块212的JSON示例：
图2所示场景描述文件中的动画描述模块(animation)213为节点描述模块202的下一层 级描述模块，用于描述为节点描述模块202所描述的物体添加的动画信息。为了让节点描述模块202所代表的物体不被限制在静止的状态，可以为节点描述模块202所描述的物体增添动画，因此场景描述文件中的动画描述模块213这一描述层级是被节点描述模块202所指定的，即动画描述模块213是节点描述模块202的下一层级描述模块，动画描述模块213同样与网格描述模块203具有对应关系。动画描述模块213可通过位置移动、角度旋转、大小缩放三种方式进行动画的描述，同时可以规定动画的开始、结束时间以及动画的实现方式。比如为一个代表三维物体的网格描述模块203增添一个动画，则该网格描述模块203所代表的三维物体就可以在指定的时间窗口内，通过位置移动、角度旋转、大小缩放的融合，完成规定的动画过程。
图2所示场景描述文件中的蒙皮描述模块(skin)214为节点描述模块202的下一层级描述模块，用于描述为网格描述模块203所添加的骨骼与表物体表面信息的网格之间的运动协作关系。当节点描述模块202指向的网格描述模块203代表了人、动物、机械等运动自由度较大的物体时，为了优化这些物体的运动表现效果，可以向物体的内部填充骨骼，代表物体表面信息的网格描述模块203此时在概念上就成为了蒙皮。蒙皮描述模块214这一描述层级是被节点描述模块202所指定的，即蒙皮描述模块214是节点描述模块202的下一层级描述模块，蒙皮描述模块214与网格描述模块203具有对应关系。以骨骼的运动带动物体表面的网格进行运动，再结合仿真仿生的设计，可以实现较为逼真的运动效果，比如当人的手进行握拳动作时，表面的皮肤会随着内部的骨骼发生拉伸、遮盖等变化，此时在手部模型内预先填充的骨骼，再定义骨骼与蒙皮之间的协作关系就可以实现对这一动作的逼真模拟。
上述glTF2.0场景描述标准中的场景描述文件各个描述模块只是具备了最基本的描述三维物体的能力，存在不能支持动态的三维沉浸式媒体，不支持音频文件，不支持场景更新等问题。glTF同时也声明了其每个对象属性下都有一个可选的扩展对象属性(extensions)，允许在其任何部分使用extensions进行扩展以达到更完善的功能。包括场景描述模块(scene)、节点描述模块(node)、网格描述模块(mesh)、访问器描述模块(accessor)、缓存描述模块(buffer)、动画描述模块(animation)等及其这些内部定义的语法元素都有可选的扩展对象属性，以支持在glTF2.0的基础上进行一定的功能扩展。
目前主流的沉浸式媒体主要有点云(Point Cloud)、三维网格(Mesh)、6DoF全景视频、基于MIV等。在三维场景中，多种类型的沉浸式媒体往往同时存在。这就要求渲染引擎支持多种不同类型的沉浸式媒体的编解码，根据所支持编解码器的种类和数量不同，产生了不同种类的渲染引擎。不同供应商设计的渲染引擎支持的媒体类型不同，为了实现对不同种类媒体所组成的三维场景的跨平台描述，动态图像专家组(Moving Picture Experts Group，MPEG)启动了MPEG场景描述标准的制订，标准号为ISO/IEC 23090-14。该标准主要解决MPEG media(包括MPEG制订的编解码器、MPEG文件格式、MPEG传输机制)在三维场景中的跨平台描述问题。
MPEG#128次会议决议以glTF2.0(ISO/IEC 12113)作为基础，制订MPEG-I Scene Description标准。目前已制订出第一版的MPEG场景描述标准，处于FDIS投票阶段。MPEG场景描述标准在第一版标准的基础上，通过添加相应扩展，以解决三维场景跨平台描述中仍未实现的需求，包括交互性、AR锚定、用户和化身表示、触觉支持，以及扩展对沉浸式媒体 编解码器的支持等。
已制订出的第一版MPEG场景描述标准主要制订了以下内容：
1)MPEG场景描述标准定义了一种用于描述沉浸式三维场景的场景描述文件格式，此格式结合了原来glTF2.0(ISO/IEC 12113)的内容并在其基础上进行了一系列扩展。
2)MPEG场景描述定义了一个场景描述框架以及其中用于模块间协作的应用程序接口(Application Program Interface，API)，实现了沉浸式媒体的获取与处理过程同媒体渲染过程的解耦，有益于实现进行沉浸式媒体对不同网络条件的适应、部分获取沉浸式媒体文件、针对沉浸式媒体的不同细节层次的访问以及内容质量的调整等方面的优化。沉浸式媒体的获取与处理过程同沉浸式媒体渲染过程的解耦是实现三维场景跨平台描述的关键。
3)MPEG场景描述提出了一系列基于基于国际标准化组织基本媒体文件格式(International Standardization Organization Base Media File Format，ISOBMFF)(ISO/IEC 14496-12)的扩展用于传输沉浸式媒体内容。
参照图3所示，相比于glTF2.0场景描述标准中的场景描述文件，MPEG场景描述标准中的场景描述文件的扩展可以分为两组：
第一组扩展包括：MPEG媒体(MPEG_media)301、MPEG时变访问器(MPEG_accessor_timed)302以及MPEG环形缓存器(MPEG_buffer_circular)303。其中，MPEG媒体301为独立扩展，用于引用外部媒体源；MPEG时变访问器302为访问器层级的扩展，用于访问时变媒体；MPEG环形缓存器为缓存器层级的扩展用于支持循环缓存器。第一组扩展提供了对场景中媒体的基本描述和格式，满足了在场景描述文件中描述时变的沉浸式媒体的基本需求。
第二组扩展包括：MPEG动态场景(MPEG_scene_dynamic)304、MPEG纹理(MPEG_texture_video)305、MPEG音频空间(MPEG_audio_spatial)306、MPEG视角推荐(MPEG_viewport_recommended)307、MPEG网格映射(MPEG_mesh_linking)308以及MPEG动画时间(MPEG_animation_timing)309。其中，MPEG_scene_dynamic 304为场景层级扩展，用于支持动态场景更新；MPEG_texture_video 305为纹理层级扩展，用于支持视频形式的纹理；MPEG_audio_spatial 306，为节点层级和相机层级扩展，用于支持空间3D音频，MPEG_viewport_recommended 307为场景层级扩展，用于支持在二维显示时描述推荐视角，MPEG_mesh_linking 308为网格层级扩展，用于支持链接两个网格并提供映射信息；MPEG_animation_timing 309为场景层级扩展，用于支持控制动画时间线。
下面将对上述各个扩展部分进行详细展开说明：
MPEG场景描述文件中的MPEG媒体用于对媒体文件的类型进行描述，并对MPEG类型的媒体文件进行必要的说明，以便后续取用这些MPEG类型的媒体文件。其中，MPEG媒体的第一层级的语法元素的定义如下表8所示：
表8
MPEG媒体的媒体列表(MPEG_media.media)中的语法元素的定义如下表9所示：
表9
MPEG媒体的媒体列表中的可选项(MPEG_media.alternatives)中的语法元素的定义如下表10所示：
表10
MPEG媒体的媒体列表的可选项中的轨道数组(MPEG_media.alternatives.tracks)中的语法元素的定义如下表11所示：
表11
此外，基于ISOBMFF(ISO/IEC 14496-12)，ISO/IEC 23090-14还定义了与场景描述文件的交付以及与glTF 2.0扩展相关的数据交付的传输格式。为了便于将场景描述文件交付给客户端，ISO/IEC 23090-14定义了如何将glTF文件和相关数据作为非时变和时变的数据(例如，作为轨道样本)封装在ISOBMFF文件中。MPEG_scene_dynamic、MPEG_mesh_linking、MPEG_animation_timing向显示引擎提供了特定形式的时变数据，显示引擎11应根据这些变化的信息进行相应的操作。ISO/IEC 23090-14还定义了每个扩展的时变数据的格式，以及将其封装在ISOBMFF文件中的方式。MPEF媒体(MPEG_media)允许引用通过RTP/SRTP、MPEG-DASH等协议传递的外部媒体流。为了允许在不知道实际协议方案、主机名或端口值的情况下对媒体流进行寻址，ISO/IEC 23090-14定义了一个新的URL方案。该方案要求在查询部分中存在一个流标识符，但没有指定特定类型的标识符，允许使用媒体流标识方案(Media Stream Identification scheme，RFC5888)、标记方案(labeling scheme，RFC4575)或基于0的索引方案。
触觉为用户提供了额外的娱乐和感官沉浸感，通过在音频/视频内容中正确地添加触觉，可以显着增强媒体内容带给用户的体验和享受。触觉数据表示如果具有标准统一的数据模型，且数据模型包括指定合成器呈现触觉效果所需的全部信息，则能够实现触觉数据的标准译码。
参照图4所示，图4为触觉数据模型的结构示意图。如图4所示，触觉数据模型包括：描述部分41和编码部分42，且采用分层结构逐层描述触觉数据。其中，描述部分41以感知和触觉通道信息作为数据结构中主要的描述内容，编码部分42以触觉频段、触觉效果和关键帧作为数据结构中主要的描述内容。描述部分41的最高层包含了对整个数据结构中所含内容的整体描述，通常用来描述文件或媒体流中定义的整个触觉体验。触觉体验包含一些高级元数据，同时提供了预先定义好的化身列表(即身体部分的表示)，便于后续指定触觉刺激在身体上的实现位置。触觉数据通过感知列表(感知1～感知a)进行描述。触觉感知列表中的各个触觉感知分别对应于与特定感知模态(例如振动、力、位置、速度、温度等)相关的触觉信号。感知列表中的各个感知分别包含一些有关触觉实现设备的特征以及与该感知相关的化身等元数据信息。除元数据外，感知信号本身是由多个触觉通道(触觉通道1～触觉通道b)混合而成。每条触觉通道不仅包括通道配置(如混合增益等)方面的元数据，也包含数据按频带分解的触觉频段列表(触觉频段1～触觉频段c)，每个触觉频段的频段数据给定频率范围内的部分信号。触觉频段使用频段数据和触觉效果列表(触觉效果1～触觉效果d)进行描述，触觉效果列表中的每个触觉效果都包含一个关键帧列表(关键帧1～关键帧e)。不同频段包含不同信号，通过组合不同频段的数据就能重建通道中的触觉信号。
在一些实施例中，为了实现通道级别的触觉效果在指定身体部位实现，也可以采用分层 结构逐层描述人体的部位。参照图5所示，人体的分层结构包括：第一层级51包括人体全部部分；第二层级52包括：上身和下身；第三层级53包括：上身的胳膊、胸部、头部以及下身的腿部和腰部；第四层级54包括：胳膊的上臂、前臂手、头部的颈部和项部以及腿部的大腿、小腿和脚部；第五层级55包括：手部的手掌和手指、脚部的脚掌和脚趾；第六层级56包括：手指的拇指、食指、中指、无名指以及小拇指，脚部的拇趾、食趾、中趾、无名趾以及小拇趾，第七层级57包括：拇指的近位指骨、第二指关节以及第三指关节。
触觉数据模型中触觉感知的全部感知模态如下表12所示：
表12
数字编码对于允许数字触觉设备运行至关重要，作为一种类似于音频和视频的一阶媒体类型，触觉媒体文件也是以数字格式存储触觉数据。当前触觉媒体文件缺乏统一的编码格式，各种技术制造商开发和使用的编码格式不统一，导致了触觉媒体文件在解码器端格式的不兼容问题，阻碍了触觉在市场上的广泛采用。因此，ISO/IEC运动图像专家组(MPEG)中的第七工作组制定ISO/IEC 23090-31触觉标准化工作突出了标准化触觉编码表示的必要性。标准的触觉编码格式(以及相关的标准化解码器)将有助于将触觉纳入ISOBMFF，MPEG-DASH和MPEG-I标准，使内容创建者以及媒体/流媒体内容提供商更容易以标准化方式整合触觉并改善整体用户体验。
ISO/IEC运动图像专家组(MPEG)中的第七工作组在制定ISO/IEC 23090-31触觉编译码标准中所提出的编解码器架构，既可以为脉冲编码调制(Pulse Code Modulation，PCM)文件(WAV文件/.wav文件)，也可以处理描述性触觉文件(AHAP文件/.ahap文件或IVS文件/.ivs文件或HJIF文件/.hjif文件)。元数据信息以OHM文件的格式提供给编解码器。触觉编码表示允许以人类可读的JSON格式对描述性和量化数据进行编码，编码流程如图6所示。
参照图6所示，图6为对触觉信号进行编码的编码流程示意图。如图6所示，触觉信号 编码器600以不同的方式处理PCM触觉文件和描述性触觉文件。针对描述性触觉文件，触觉信号编码器600首先识别触觉文件的文件格式。如果待编码触觉文件是.hjif文件，则无需转码，可以直接压缩为二进制格式并最终打包成MIHS流。如果待编码触觉文件是.ahap文件或.ivs文件，则需要转码。触觉信号编码器600首先在语义上分析待编码触觉文件，并将其转码以格式化为图4所示的数据模型。转码后，数据可以导出为以JSON可读形式编码的交换文件格式(.hjif)、编码为压缩二进制文件的分发格式(.hmpg)或定义数据包化比特流的流格式(MIHS)。针对PCM文件，触觉信号编码器600的信号分析过程分为两个子过程。其一为：对原始信号执行频带分解后，基于关键帧提取对频带分解得到的低频信号进行编码。其二为：重建低频信号，并计算重建的低频信号与原始低频信号之间的残差，获取残差信号，然后将该残差信号添加到原始高频信号后使用小波变换进行编码，输出.hjif或.hmpg或MIHS中的至少一种格式的编码结果
参照图7所示，图7为对触觉信号进行解码的解码流程示意图。如图7所示，触觉信号解码器700既可以将打包成数据包的流格式输入(MIHS文件)，拆包之后进行解压缩得到.hjif文件，也可以选择将.hmpg文件作为输入，解压缩后输出.hjif文件。然后.hjif文件中包含的触觉数据可以直接在触觉设备上呈现，或使用生成PCM数据的中间合成器，输出.wav文件在触觉设备上渲染。其中.hjif格式是基于JSON的人类可读格式，未针对内存使用进行优化。它可以轻松解析和手动编辑，方便内容的设计与创建，这使其成为理想的交换格式。出于分发目的，可以将.hjif数据压缩为内存效率更高的二进制.hmpg比特流。这种压缩是有损的，不同的参数会影响构成比特流的振幅和频率的编码深度。出于流式传输目的，可以将数据压缩并打包成MPEG-I触觉流(MIHS)。这种方法可以满足相关技术对触觉媒体文件的描述性和量化格式兼容性的期望，以及触觉终端设备之间的互操作性，以实现3D沉浸式体验和分发目的。MIHS触觉流也是将触觉媒体文件封装成ISOBMFF的基础。
在一些实施例中，参照图8所示，为了将触觉媒体文件与场景中的对象相关联，将触觉作为沉浸式媒体体验的一部分，在MPEG#140次会议中，通过了在MPEG场景描述文件中添加MPEG触觉(MPEG_haptic)801和MPEG触觉材质(MPEG_material_haptic)802的提案。基于该glTF的触觉扩展的新版本，可在场景描述中启用触觉支持。其中，MPEG触觉801通过索引MPEG媒体(MPEG_media)的媒体列表中的媒体描述模块得到触觉媒体文件的访问地址。需要说明的是，从MPEG媒体(MPEG_media)中索引到的是完成触觉编码且封装完毕的触觉媒体。MPEG触觉材质作为对原有虚拟对象的材质属性的补充说明，存储触觉材质属性值的同时也支持对触觉媒体文件的直接索引。
MPEG触觉(MPEG_haptic)801为节点描述模块的下一层级描述模块，用于支持索引MPEG_media中与触觉相关的媒体文件以添加触觉信息。MPEG触觉801中的语法元素的定义如下表13所示：
表13
MPEG触觉材质(MPEG_material_haptic)402为网格描述模块的下一层级描述模块，用于支持用户在虚拟场景中通过主动和/或被动交互得到相应触觉反馈。
纹理映射是指在不需要高分辨率网格的情况下，高效、逼真地显示三维模型细节的技术。这种方法最初是在计算机图形学中开发的，如今已被广泛地应用于触觉渲染。纹理映射的存在，能够使虚拟对象增强其表面分布的附加触觉材质属性，增强虚拟对象的真实性。在纹理映射的基础上，相关技术进一步提出了触觉材质的概念。在计算机图形学中，材质是一个易于使用的数据包，包含了虚拟对象的可视化渲染所需的所有数据。类比而言，触觉材质应该为触觉渲染提供所有必要的元素，且不同的触觉材质蕴含不同的触觉特征，这些不同的触觉材质属性存储在不同的触觉材质包中，方便了触觉设计和实现的相关任务。触觉材质属性的定义如下表14所示：
表14
上表14中的每个触觉材质属性使用纹理数组表示，不同属性的纹理数组结合起来组成一种特定触觉感知。触觉材质属性既可以作为传统的二维纹理提供相关触觉纹理值，也可以作为触觉元素贴图。触觉元素贴图中每个像素点包含多个触觉单位元素，关联到对应于MPEG触觉(MPEG_haptic)的媒体参考数组(Media_reference)中的一个索引。渲染引擎可以选择最合适的纹理数组进行渲染。这样，每个数组元素都包含一个触觉纹理和一个表示为枚举的纹理类型。枚举的触觉材质纹理类型包括：
高分辨率(High_resolution)：触觉纹理是直接存储触觉值的高分辨率二维纹理。
低分辨率(Low_resolution)：触觉纹理是直接存储触觉值的低分辨率二维纹理。
参考(Reference)：触觉纹理是包含触觉信号参考的二维触觉元素图。触觉纹理的每个像素对应于MPEG触觉(MPEG_haptic)的媒体参考数组(Media_reference)中的一个索引。
其他(Other)：未知的专有纹理格式。
为了解释二维纹理中包含的数据，需要指定这些纹理的位深和范围，下表15、16分别示出了低分辨率触觉纹理和高分辨率触觉纹理的每个触觉材质属性的位深和范围：
表15
表16
如上表16所示，高分辨率振动纹理中的每个纹理映射像素的值一分为二：第一个字节用于描述幅度值，第二个字节用于描述频率值。
MPEG触觉材质(MPEG_material_haptic)402中的语法元素的定义如下表17所示：
表17
上表17为MPEG触觉材质(MPEG_material_haptic)402中包括的语法元素的语义，表17中的枚举值就是表14中所包括的触觉材质属性。MPEG场景描述文件中的MPEG触觉材质(MPEG_material_haptic)402，能够将场景中的对象与触觉材质相关联，在用户与对象进行交互时，就能得到相应的触觉感知。
每个触觉材质属性包含不同的触觉特征，但内部存储方式都是一致的，下表18以触觉材质属性中的刚度(stiffness)作为示例，对MPEG触觉材质中枚举的刚度(MPEG_material_haptic.stiffness)进行进一步说明。
表18
二、显示引擎
参照图1所示，在沉浸式媒体的场景描述框架的工作流程中，显示引擎11主要作用包括获取场景描述文件，并对获取到的场景描述文件进行解析，以获取待渲染三维场景的组成结构和待渲染三维场景中的细节信息，以及根据解析场景描述文件得到的信息进行待渲染三维场景的渲染与展示。本申请实施例中不对显示引擎11具体工作流程和原理进行限制，以显示引擎11能够解析场景描述文档，并通过媒体接入函数API向媒体接入函数12下达指令，通过缓存API向缓存管理模块13下达指令，以及从缓存中取用经过处理的数据并完成三维场景及其中物体的渲染与展示为准。
三、媒体接入函数
在沉浸式媒体的场景描述框架的工作流程中，媒体接入函数12可以接收来自显示引擎11的指令，并根据显示引擎11发送的指令完成媒体文件的访问与处理功能。具体包括：获取到媒体文件后，对媒体文件进行处理。不同类型的媒体文件的处理过程存在较大差异，为了实现广泛的媒体类型支持，也考虑到媒体接入函数的工作效率，于是在媒体接入函数中设计了多种管线，在处理过程中只启用匹配于媒体类型的管线即可。
管线的输入是从服务器下载的媒体文件或本地存储控件读取的媒体文件，这些媒体文件往往具有较为复杂的结构，无法直接被显示引擎11使用，所以管线的主要功能就是对这种媒体文件的数据进行处理，使媒体文件的数据符合显示引擎11的要求。
在沉浸式媒体的场景描述框架的工作流程中，管线处理完成的媒体数据需要以规范的排列结构交付给显示引擎11使用，这就需要缓存API与缓存管理模块13的参与，缓存API与缓存管理实现了根据经过处理的媒体数据的格式，创建相应的缓存，并负责对缓存的后续管理，比如更新、释放等操作。缓存管理模块13可以通过缓存API与媒体接入函数12进行通信，也可以与显示引擎11进行通信，与显示引擎11和/或媒体接入函数12通信的目标都是实现缓存的管理。当缓存管理模块13与媒体接入函数12进行通信时，需要显示引擎11将缓存管理的相关指令先通过媒体接入函数API发送给媒体接入函数12，媒体接入函数12再通过缓存API将缓存管理的相关指令通过缓存API发送给缓存管理模块13。当缓存管理模块13与显示引擎11通信时，只需要显示引擎11将场景描述文档中解析出来的缓存管理描述信息直接通过缓存API发送给缓存管理模块13即可。
如图8所述，目前MPEG#140次会议确认通过添加MPEG触觉(MPEG_haptic)801和MPEG触觉材质(MPEG_material_haptic)802的提案，且允许MPEG触觉801索引到MPEG媒体(MPEG_media)中的触觉媒体文件，多是描述与事件或环境相关的整体触觉体验，索引到的触觉媒体文件视为触觉信号，本质上能够直接驱动触觉设备来对人体产生相应的触觉反馈。
如上所述，相关技术中，每个数组元素都包含一个触觉纹理和一个表示为枚举的触觉材质属性类型，且触觉材质属性类型包括：高分辨率(High_resolution)、低分辨率(Low_resolution)、参考(Reference)以及其他(Other)。在触觉材质属性类型为高分辨率或低分辨率的情况下，纹理索引的纹理贴图或触觉纹理材质数据包中存储的不是颜色值，而是触觉值。触觉值仅代表特定属性的大小，(如摩擦系数的大小，刚度系数的大小)，并能够直接被显示引擎获取并进行触觉渲染。但当触觉材质属性类型为参考的时，纹理贴图或触觉纹理材质数据包中的每一个像素并不直接存储触觉值，而是存储着待渲染的触觉媒体文件的索引值，因此当触觉材质属性类型为参考时，根据该索引值仅能只能获取到编码并封装后的触觉媒体文件，无法获取解码后的可用于渲染的触觉值，进而导致显示引擎无法正常进行渲染。即，相关技术无法获取参考(Reference)类型的触觉材质属性的触觉值。
本申请一些实施例提供了一种支持获取参考类型的触觉材质属性的触觉值的场景描述框架，具体内容包括：场景描述文件对获取参考类型的触觉材质属性的触觉值的支持、媒体接入函数API对获取参考类型的触觉材质属性的触觉值的支持、媒体接入函数对获取参考类型的触觉材质属性的触觉值的支持、缓存API对获取参考类型的触觉材质属性的触觉值的支持、缓存管理对获取参考类型的触觉材质属性的触觉值的支持等内容。
基于场景描述框架索引三维场景中的触觉媒体文件并渲染和显示的过程包括：首先，显示引擎通过网络协议下载或者本地存储读取等方式从三维场景提供方获取场景描述文件。其中，显示引擎获取的场景描述文件包括了对整个三维场景以及场景中触觉媒体文件的描述信息，对所述触觉媒体文件的描述信息包括了所述触觉媒体文件的访问地址、所述触觉媒体文件的访问的属性信息(媒体类型，使用的编解码参数等)等。同时还包括了对所述触觉媒体 文件解码处理所得到的数据的格式要求，对所述触觉媒体解码处理后所得数据的格式要求又包括了所述触觉媒体文件处理后所得到的触觉数据的描述部分和编码部分(即图4中的描述部分41和编码部分42，后文中统称为触觉媒体文件)等信息数据的格式要求(数据类型、数据长度及数据访问方式等)以及对其他媒体数据的格式要求及地址，其中就包括后续需要使用的纹理贴图、触觉材质属性的描述信息等数据。显示引擎完成场景描述文件的解析后，调用媒体接入函数API将触觉媒体文件的访问地址、触觉媒体文件的属性信息(媒体类型，使用的编解码器等)等信息传递给媒体接入函数。同时，显示引擎将解析得到的场景描述文件中指定的解码后的触觉媒体数据的格式要求及其他媒体数据的格式要求也通过调用媒体接入函数API传递给媒体接入函数。然后，媒体接入函数根据显示引擎传递的触觉媒体文件的访问地址，向服务器请求下载指定的触觉媒体文件或者从本地获取指定的触觉媒体文件，之后媒体接入函数根据媒体类型和编解码参数等信息为该媒体文件建立相应的管线将获取到的媒体数据从封装格式转换为场景描述文件中要求的格式，即对于触觉媒体文件应在管线中转换为场景描述文件中规定的格式(数据类型、数据长度和访问方式等)。在管线中可以对该媒体文件进行解封装、解密、解码以及后处理等操作。随后管线将处理为场景描述文件规定格式的解码后的触觉媒体的信息数据以及其他媒体数据存放到指定的缓存中。该缓存可以由显示引擎通过缓存API调用缓存管理模块创建，如果显示引擎没有创建相应缓存，则可以由媒体接入函数通过缓存API调用缓存管理模块创建相应缓存。目的都是实现缓存的创建、更新、释放等。最后，显示引擎在指定的缓存中以场景描述文件规定的格式读取触觉媒体文件完成对三维场景中的触觉部分的渲染。
以下分别对支持获取参考类型的触觉材质属性的触觉值的场景描述文件、媒体接入函数API、媒体接入函数、缓存API、缓存管理进行说明。
1、支持获取参考类型的触觉材质属性的触觉值的场景描述文件
参照图9所示，图9为本申请实施例提供的支持获取参考类型的触觉材质属性的触觉值的场景描述文件的结构示意图。如图9所示，支持获取参考类型的触觉材质属性的触觉值的场景描述文件的场景描述文件中包括但不限于以下模块：MPEG媒体(MPEG_media)2601，场景描述模块(scene)2602、节点描述模块(node)2603、网格描述模块(mesh)2604、访问器描述模块(accessor)2605、缓存切片描述模块(bufferView)2606、缓存器描述模块(buffer)2607、材质描述模块2608、MPEG触觉材质(MPEG_material_haptic)2609、纹理描述模块(texture)2610、MPEG触觉(MPEG_haptic)2611、相机描述模块(camera)2612、蒙皮描述模块(skin)2613以及动画描述模块(animation)2614。以下分别对图9所示场景描述文件的各个描述模块进行说明。
为了使场景描述文件能够正确的描述触觉媒体文件本申请一些实施例对场景描述文件的MPEG媒体(MPEG_media)2601内的语法元素的取值进行了扩展，具体扩展包括以下至少一项：
扩展1、对场景描述文件的MPEG媒体的媒体列表的可选项(MPEG_media.media.alternatives)中用于说明媒体的媒体资源类型的媒体类型语法元素(MPEG_media.media.alternatives.mimeType)的值进行扩展。对媒体类型语法元素(mimeType)的扩展包括：为媒体类型语法元素(mimeType)添加与触觉媒体文件相关联的值。例如：在 场景描述文件的MPEG媒体的媒体列表包括触觉媒体时，用于说明媒体的媒体资源类型的媒体类型语法元素(MPEG_media.media.alternatives.mimeType)的值遵循ISO/IEC 23090-32标准的相关规定的情况下，当媒体资源为包括：音频、视频和触觉的媒体资源时，"mimeType"的值为"video/mp4"；当媒体资源为仅包括：音频和触觉的媒体资源时，"mimeType"的值为"video/mp4"；当媒体资源为仅包括：触觉的媒体资源时，"mimeType"的值为"haptics/mp4"。
扩展2、对场景描述文件的MPEG媒体的媒体列表的可选项的轨道数组(MPEG_media.media.alternatives.tracks)中用于声明媒体文件的轨道信息的轨道语法元素(MPEG_media.media.alternatives.tracks.track)的值进行扩展。对轨道语法元素(track)的扩展包括：当编码后的触觉媒体文件作为MPEG媒体的媒体列表的可选的轨道(MPEG_media.media.alternatives.tracks)中的一项被场景描述文件引用，应该遵循ISO/IEC23090-32标准中对参考track的选取方法。例如：当触觉媒体被ISOBMFF封装成一条MIHS轨道时，应该在MPEG媒体中引用这条MIHS轨道；当触觉媒体被ISOBMFF封装成多条MIHS轨道时，需参考ISO/IEC 23090-32的后续补充内容。
扩展3、对场景描述文件的MPEG媒体的媒体列表的可选项的轨道数组(MPEG_media.media.alternatives.tracks)中用于说明码流轨道中包含的媒体数据的编解码器类型的编解码参数语法元素(MPEG_media.media.alternatives.tracks.codecs)的值进行扩展。由于IETFRFC 6381中定义包含在码流轨道中的媒体文件的编解码参数。当码流轨道包含不同类型的编解码器(例如AdaptationSet包含具有不同编解码器的表示)时，编解码参数语法元素(codecs)可以用逗号分隔的编解码器值列表来表示，因此对编解码参数语法元素(codecs)的值的扩展包括：当媒体数据为触觉数据，且封装规则遵循23090-32标准时，编解码参数语法元素(codecs)的值应遵循ISO/IEC 23090-32触觉数据传输(Carriage of haptics data)标准中的规定进行设置。例如：在触觉数据采用ISMBOFF封装时，编解码参数语法元素(codecs)的第一组元素为表示ISO/IEC 23090-31中编解码器的代码，编解码参数语法元素(codecs)参数的形式如下："codecs":"mih1.oo"，其中"mih1"是触觉数据封装在ISOBMFF中的一类数据格式说明，该值并不唯一，允许其他符合ISOBMFF封装的样本格式名称进行替换。其中"oo"是MP4注册机构网站"对象类型"页面上定义的对象类型指示值。
即，为了使场景描述文件能够正确地描述触觉媒体文件，本申请一些实施例对场景描述文件中的MPEG媒体内的语法元素的值进行了扩展，具体扩展包括扩展如下表19所示中的一项或等多项：
表19
对场景描述文件中MPEG_media内语法元素取值执行上述扩展1～扩展3中的至少一项，即可使场景描述文件中的MPEG媒体实现对触觉媒体文件的描述。
需要说明的是，触觉媒体文件的封装基于图4所示的数据结构，触觉媒体文件不仅存在信息部分，也允许存在元数据配置部分，本申请实施例不限定触觉媒体文件的封装方式，以可以根据MPEG媒体中的触觉媒体文件的访问地址获取已编码完且封装好的触觉媒体文件为准。
在一些实施例中，支持获取参考类型的触觉材质属性的触觉值的场景描述文件中的场景描述模块(scene)2602与节点描述模块(node)2603的描述方法，包括：当三维场景中包含触觉媒体文件时，使用scene和node的描述方法对三维场景的总体结构以及触觉媒体文件在三维场景所处的结构层次和位置进行描述。当三维场景中有需要索引触觉媒体文件的节点描述模块(node)，则启用MPEG触觉(MPEG_haptic)，并在MPEG触觉中索引所需的触觉媒体文件。图9所示场景描述文件的节点描述模块(node)的MPEG触觉(MPEG_haptic)2611中包括：媒体参考(Media_reference)，媒体参考(Media_reference)中包括：触觉媒体标识语法元素(haptic_id)26111、触觉媒体标识语法元素26111的值(haptic_id)、访问器索引语法元素(accessor)26112，以及访问器索引语法元素(accessor)的值。支持获取参考类型的触觉材质属性的触觉值的场景描述文件中的MPEG触觉(MPEG_haptic)2611的描述方法，包括：通过MPEG触觉(MPEG_haptic)2611中的触觉媒体标识语法元素(haptic_id)及其值实现纹理坐标点能够索引到一个编码并封装后的触觉媒体文件，通过MPEG触觉(MPEG_haptic)2611中的访问器索引语法元素(accessor)及其值实现纹理坐标点能够索引到相应触觉媒体文件的解码数据中的一组触觉值。即，当某一三维网格承载的触觉材质属性的类别为参考(Reference)，则需要在包含该三维网格的节点对应的节点描述模块的MPEG触觉(MPEG_haptic)2611的媒体参考(Media_reference)中扩展一个触觉媒体标识语法元素(haptic_id)和一个访问器索引语法元素(accessor)，并通过触觉媒体标识语法元素(haptic_id)和访问器索引语法元素(accessor)的值对进行相关补充说明。相关补充说明具体包括：通过MPEG触觉(MPEG_haptic)2611的媒体参考(Media_reference)中的触觉媒体标识语法元素(accessor)26112及其值明确了纹理坐标点需要索引到的触觉媒体文件对应的媒体描述模块，通过MPEG触觉(MPEG_haptic)2611的媒体参考(Media_reference)中的访问器索引语法元 素(accessor)26111及其值明确了纹理坐标点能够实际访问到需要索引到的触觉媒体文件的解码数据的访问器对应的访问器描述模块。该功能与场景描述文件中对缓存器描述模块(buffer)、缓存切片描述模块(bufferview)的描述相对应，共同实现纹理坐标点对触觉媒体文件的解码数据的直接索引。
此外，每一个三维场景均使用一个独立的场景描述模块进行描述。在三维场景中，可能存在许多具体物体，比如虚拟数字人、近处的三维物体、远处的背景图片、代表用户的视点与视角的虚拟相机等，场景描述文件会将这些具体物体作为三维场景中的节点，每个节点可以代表一个物体或者是数个物体组成的一组物体集合，节点之间的关系反映了三维场景中各个组成部分的关系。每个场景描述文件可以描述一个或多个三维场景，三维场景之间只能为并列关系，不能为层级关系，即场景之间是不具有包含与被包含的关系的。节点之间可以是并列关系或是层级关系，即节点之间存在包含与被包含的关系，这使得多个具体物体可以集合在一起进行表示。如果一节点被另一个节点包含，则被包含的节点称为子节点，对于子节点使用"children"替换"node"进行表示。灵活使用节点与子节点进行组合，就可以构成层级化的节点结构。
在一些实施例中，支持获取参考类型的触觉材质属性的触觉值的场景描述文件中的网格描述模块(mesh)2604的描述方法，包括：对于表示三维物体的节点，场景描述文件将会使用三维网格的数据格式进行描述，即使用网格描述模块(mesh)2604进行描述。网格描述模块中具有多种具体信息，比如三维坐标、颜色等信息，这些信息都隶属于网格描述模块2604的基元中的属性(mesh.primitives.attribute)中的语法元素。最基本的网格描述模块包含了顶点的三维坐标、顶点间的连接关系、顶点的颜色信息以及需要纹理值的三维坐标等。在网格描述模块层面需要启用所需纹理值的三维坐标集合纹理坐标(Texcoord)，以便后续从纹理处获得的触觉值附在三维物体相应的三维坐标中。纹理坐标(Texcoord)与颜色数据等信息都隶网格描述模块的基元中的属性(mesh.primitives.attribute)。而触觉媒体暂时不需要在网格描述模块层面引用颜色数等属性(attribute)进行定义，只需要启用MPEG触觉材质(MPEG_material_haptic)2609，作为与材质描述模块(material)2604平行的内容补充，以支持对触觉材质的进一步定义。
在一些实施例中，支持获取参考类型的触觉材质属性的触觉值的场景描述文件中的材质描述模块(material)2608和纹理描述模块(texture)2610的描述方法，包括：通过材质描述模块(material)2608和纹理描述模块(texture)2610描述三维物体的表面的附加信息。在描述常规的三维物体时，仅借助网格描述模块描述三维物体的几何信息，或单调地定义三维网格中顶点的颜色，对提升虚拟物体的真实度来说远远不够，这就需要在三维物体的表面附加更多的信息。对于常规的三维模型技术，这一过程也可以简称为贴图或赋加纹理等。即，支持获取参考类型的触觉材质属性的触觉值的场景描述文件以glTF2.0为基础，也沿用了材质描述模块(material)2608和纹理描述模块(texture)2610，且材质描述模块(material)2608与纹理描述模块(texture)2610的协作关系包括：材质描述模块(material)2608与纹理描述模块(texture)2610共同定义了物体表面的颜色和物理信息。纹理描述模块(texture)2610下指定了采样器描述模块(sampler)和纹理贴图描述模块(image)，采样器描述模块(sampler)定义了如何将纹理贴图映射到物体表面，实现了纹理的具体调整和包装。纹理贴图描述模块 (image)则使用ULR对纹理贴图进行标识和索引。
在一些实施例中，支持获取参考类型的触觉材质属性的触觉值的场景描述文件中的MPEG触觉材质(MPEG_material_haptic)2609的描述方法，包括：通过MPEG触觉材质(MPEG_material_haptic)2609对触觉材质属性进行进一步描述。当三维场景中包含触觉媒体文件时，需要考虑材质特性中的触觉材质属性。MPEG触觉材质(MPEG_material_haptic)2609中包括各个触觉材质属性(刚度、摩擦力、触觉纹理、温度、振动以及自定义)对应的触觉材质属性描述模块，触觉材质属性描述模块中包括属性类型语法元素，属性类型语法元素的值可以为：高分辨率(High_resolution)、低分辨率(Low_resolution)、参考(Reference)。三维场景中一个包含触觉反馈的物体，可能包括不同类型的触觉材质属性。同一物体上的不同触觉材质属性，可以通过不同方式得到触觉值或被用户感受到该触觉值的触觉媒体，以此来实现场景描述文件中关联触觉媒体以得到触觉反馈。
由表17可知，MPEG触觉材质(MPEG_material_haptic)2509包含多个触觉材质属性(刚度、摩擦力、触觉纹理、温度、振动以及自定义)对应的触觉材质属性描述模块，每个触觉材质属性的对应的触觉材质属性描述模块包含了该触觉材质属性的类型，具体包括：低分辨率(Low_resolution)、高分辨率(High_resolution)、参考(Reference)、其它(Other)。若某一触觉材质属性的触觉材质纹理类别为低分辨率或高分辨率，则说明此时纹理坐标点上的触觉值可以从纹理贴图或其他触觉纹理材质数据包中获得，该纹理贴图或触觉纹理材质数据包上每个像素点包含的值就代表触觉值，且如表15和表16所示，不同触觉材质纹理类型的触觉值的位深不同，如高分辨率高分辨率(High_resolution)的触觉材质属性的触觉值的位深是16bit，而低分辨率(Low_resolution)的触觉材质属性的属性的位深则是8bit。
若触觉材质属性的类别为参考(Reference)，则要求该纹理坐标点或一组纹理坐标点的触觉值能够索引到一组时间上或空间上的触觉值，此时就需要启用MPEG触觉(MPEG_haptic)2611的媒体参考(Media_reference)中的触觉媒体标识语法元素(haptic_id)和访问器索引语法元素(accessor)进行相关补充说明。MPEG触觉材质(MPEG_materia_haptic)2609内部包含每种触觉材质属性的类型。当某种触觉材质属性的类型是参考(Reference)时，需要在媒体参考(Media_reference)里添加触觉媒体标识语法元素和访问器索引语法元素，目的是实现纹理坐标点能够索引到一组编码且封装后的触觉媒体文件以及该用于访问该触觉媒体文件的解码数据的访问器。即，在MPEG触觉(MPEG_haptic)2611的媒体参考(Media_reference)内部具有一个名为accessor的语法元素和一个名为haptic_id的语法元素，haptic_id明确了纹理坐标点需要索引到的触觉媒体文件的媒体描述模块，accessor明确了纹理坐标点能够实际访问到需要索引到的触觉媒体文件的解码数据的访问器对应的访问器描述模块。该功能与场景描述文件中对缓存器描述模块(buffer)、缓存切片描述模块(bufferview)的描述相对应，共同实现纹理坐标点对触觉媒体文件的直接索引。MPEG触觉(MPEG_haptic)2611的媒体参考(Media_reference)中的语法元素的定义如下表20所示：
表20
在一些实施例中，支持获取参考类型的触觉材质属性的触觉值的场景描述文件中的访问器描述模块(accessor)2605、缓存切片描述模块(bufferView)2606、缓存器描述模块(buffer)2607的描述方法，包括：通过缓存器描述模块的MPEG环形缓存器的媒体索引语法元素(buffer.MPEG_buffer_circular.media)的值指向MPEG媒体(MPEG_media)中触觉媒体文件对应的媒体描述模块的索引。即，触觉媒体文件需要在缓存器描述模块(buffer)内指定，但并非在缓存器描述模块(buffer)中直接添加触觉媒体文件的统一资源定位符(Uniform Resource Locator，URL)，而是通过缓存器描述模块(buffer)的MPEG环形缓存器的媒体索引语法元素(MPEG_buffer_circular.media)声明MPEG媒体(MPEG_media)中相应触觉媒体文件对应的媒体描述模块的索引值。
示例性的，MPEG媒体(MPEG_media)中触觉媒体文件对应的媒体描述模块为MPEG媒体(MPEG_media)的媒体列表中的第三个媒体描述模块，则可以设置缓存器描述模块的MPEG环形缓存器中的媒体索引语法元素(buffer.MPEG_buffer_circular.media)的值为"2"，从而在缓存器描述模块的MPEG环形缓存器中索引MPEG媒体中的第三个媒体描述模块中的访问地址，以通过缓存器描述模块的MPEG环形缓存器中的(MPEG_buffer_circular)媒体索引语法元素索引触觉媒体文件。
在一些实施例中，支持获取参考类型的触觉材质属性的触觉值的场景描述文件中的访问器描述模块(accessor)2605、缓存切片描述模块(bufferView)2606、缓存器描述模块(buffer)2607的描述方法，包括：通过缓存器描述模块的MPEG环形缓存器的轨道数组的轨道(buffer.MPEG_buffer_circular.tracks.track)指示访问触觉媒体文件的封装文件时的轨道信息。具体包括：当缓存器描述模块的MPEG环形缓存器的轨道数组(buffer.MPEG_buffer_circular.tracks)中包含多个轨道(track)时，管线应该对所有引用的track进行必要的处理，以生成缓存器描述模块请求的数据格式。如果缓存器描述模块的MPEG环形缓存器的轨道数组的轨道(buffer.MPEG_buffer_circular.tracks)中引用的媒体文件具有多个可选项(alternatives)，那么每个可选项(alternatives)都应该记录有轨道的选用情况。当缓存器描述模块的MPEG环形缓存器的轨道数组(buffer.MPEG_buffer_circular.tracks)未被定义时，管线应该对MPEG媒体(MPEG_media)中的所有轨道进行必要的处理，进而生成缓存器描述模块(buffer)请求的数据格式，即指导缓存器描述模块(buffer)如何正确存储MPEG媒体(MPEG_media)中的数据。
MPEG环形缓存器(MPEG_buffer_circular)中包含的语法元素如表21所示：
表21
即，基于表21中的媒体索引语法元素(media)的值的设置规则，使表21中的媒体索引语法元素(media)取值为MPEG媒体(MPEG_media)中相应触觉媒体文件对应的媒体描述模块的索引值，从而在缓存器描述模块内对触觉媒体文件进行索引。基于表21中的语法元素"tracks"的值的设置规则，使表21中的语法元素"tracks"取值为MPEG媒体中包含触觉媒体文件的轨道索引，从而在缓存器描述模块内对触觉媒体文件的轨道进行索引。
在一些实施例中，支持获取参考类型的触觉材质属性的触觉值的场景描述文件的相机描述模块(camera)2612的描述方法，包括：通过相机描述模块(camera)2612定义节点描述模块(node)的视点、视角等与观看相关的视觉信息。
在一些实施例中，支持获取参考类型的触觉材质属性的触觉值的场景描述文件的动画描述模块(animation)2613的描述方法，包括：通过动画描述模块(animation)2613定义为节点描述模块(node)添加的动画。
在一些实施例中动画描述模块(animation)可以通过位置移动、角度旋转、大小缩放中的一种或多种对为节点描述模块(node)添加的动画进行描述。
在一些实施例中，动画描述模块(animation)还可以指示为节点描述模块(node)添加的动画的开始时间、结束时间以及动画的实现方式中的至少一项。
即，在支持获取参考类型的触觉材质属性的触觉值的场景描述文件中，同样可以为代表三维物体中的物体的节点描述模块(node)增添动画。动画描述模块(animation)则通过位置移动、角度旋转、大小缩放三种方式为节点增添的动画进行描述，同时也可以规定动画的开始、结束时间以及动画的实现方式。
在一些实施例中，支持获取参考类型的触觉材质属性的触觉值的场景描述文件的蒙皮描述模块(skin)2614的描述方法，包括：通过蒙皮描述模块(skin)2614定义节点描述模块(node)中的网格(mesh)与对应的骨骼之间的运动和形变关系。
基于上述实施例对场景描述文件中的动态图像专家组媒体(MPEG_media)，场景(scene)、节点(node)、网格(mesh)、访问器(accessor)、缓存切片(bufferView)、缓存器(buffer)、蒙皮(skin)、动画(animation)、相机(camera)、材质(material)、纹理(texture)、采样器(sampler)以及纹理贴图(image)的改进和扩展，场景描述文件已能够正确地描述触觉媒体文件。
示例性的，以下结合一个包括刚度、温度以及振动触觉材质属性，单轨道封装的触觉媒体的场景描述文件为例对图9所示的场景描述文件进行说明。




上述示例中的第1行与第241行的一对大括号内包含了图9所示场景描述文件的主要内容，支持获取参考类型的触觉材质属性的触觉值的场景描述文件的场景描述文件包括：数字资产描述模块(asset)、使用扩展描述模块(extensionUsed)、MPEG媒体(MPEG_media)、场景声明(scene)、场景列表(scenes)、节点列表(nodes)、网格列表(meshes)、触觉材质 数组列表(MPEG_material_Haptic)、纹理列表(textures)、纹理贴图列表(images)、采样器列表(samplers)访问器列表(accessors)、缓存切片列表(bufferView)、缓存器列表(buffers)。以下分别对各个描述模块的内容以及在解析角度各个描述模块所包含的信息进行说明。
1、数字资产描述模块(asset)：数字资产描述模块为第2～4行。由数字资产描述模块中第3行的"version":"2.0"，因此可以确定该场景描述文件是基于glTF2.0版本编写的，该版本也是场景描述标准的参考版本。从解析角度看，显示引擎可以根据数字资产描述模块确定应该选取哪种解析器来解析该场景描述文件。
2、使用扩展描述模块(extensionUsed)：使用扩展描述模块为第6～12行。由于使用扩展描述模块包括：MPEG媒体(MPEG_media)、MPEG环形缓存器(MPEG_buffer_circular)、MPEG时变访问器(MPEG_accessor_timed)、MPEG触觉(MPEG_Haptic)以及MPEG触觉材质(MPEG_material_Haptic)五个扩展，因此可以确定该场景描述文件中使用了MPEG媒体、MPEG环形缓存器、MPEG时变访问器、MPEG触觉、MPEG触觉材质五种MPEG扩展。从解析角度看，显示引擎可以根据使用扩展描述模块(extensionUsed)的内容提前获知了后续解析所涉及的扩展项目包括：MPEG媒体、MPEG环形缓存器、MPEG时变访问器、MPEG触觉、MPEG触觉材质。
3、MPEG媒体(MPEG_media)：MPEG媒体为第14～53行。MPEG媒体实现了对三维场景中包含的触觉媒体文件的声明。MPEG媒体的媒体列表(media)中包括两个大括号(两个媒体描述模块，第17～33行的媒体描述模块和第34～50行的媒体描述模块)，表明场景描述文件的MPEG媒体内声明了两个媒体文件。第17～33行的媒体描述模块通过第23行的"mimeType":"haptic/mp4"指出了媒体文件对应的封装文件的文件类型，通过第24行的"uri":"http://www.example.com/Hapticexample.mp4"指出了对应媒体文件的访问地址，通过第27行的"track":"trackIndex＝1"指出了对应媒体文件的轨道信息，通过第28行的"codecs":""mih1.oo"指出了对应媒体文件的解码信息，通过第18行的"name":"Hapticexample1"指出了对应媒体文件的名称，通过第19行的"autoplay":true指出了对应媒体文件应该自动播放，通过第20行的"loop":true指出了对应媒体文件应该循环播放。第34～50行的媒体描述模块通过第40行的"mimeType":"video/mp4"指出了媒体文件对应的封装文件的文件类型，通过第41行的"uri":"http://www.example.com/Hapticexample.mp4"指出了对应媒体文件的访问地址，通过第44行的"track":"trackIndex＝1"指出了对应媒体文件的轨道信息，通过第45行的"codecs":""mih1.oo"指出了对应媒体文件的解码信息，通过第35行的"name":"Hapticexample2"指出了对应媒体文件的名称，通过第36行的"autoplay":true指出了对应媒体文件应该自动播放，通过第37行的"loop":true指出了对应媒体文件应该循环播放。从解析角度看，显示引擎通过解析MPEG媒体可以确定待渲染三维场景中存在两个触觉媒体文件，并且获知了访问及解析这两个触觉媒体文件的方法。
4、场景声明(scene)：场景声明为第55行。因为在理论上可一个场景描述文件以包括多个三维场景，所以上述场景描述文件首先通过第55行的"scene":0，指出了场景描述文件描述的场景为场景列表中的第一个场景描述模块对应的三维场景，即第57～63行的大括号所囊括的场景描述文件所描述的场景。
5、场景列表(scenes)：场景列表为第57～63行。场景列表中仅包含了一个大括号，说明 该场景描述文件中仅包含了一个场景描述模块(scene)，该场景描述文件中仅包含了一个三维场景，在该大括号内(场景描述模块)通过第58～61行的"nodes":[0]指出了三维场景中仅包括一个索引为0的节点。从解析角度看，该场景列表明确了整个场景描述框架应该选取场景列表中的第一个三维场景进行后续处理和渲染，且明确了三维场景的总体结构，并且指向了更详细的下一层节点描述模块(node)。
6、节点列表(nodes)：节点列表为第65～83行。节点列表中仅包含了一个大括号，说明该场景描述文件中仅包括一个节点描述模块，该三维场景中只具有一个节点，且该节点与场景描述模块中的节点中包含的索引值为0的节点是同一个节点，两者通过索引的方式实现关联。在表示该节点的大括号(节点描述模块)中，通过第67行的"name":"Hapticexample_node"指出了节点的名称为"Hapticexample_node"，通过第68行的"mesh":"0"指出了节点上所挂载的内容为网格列表(meshes)中的第一个三维网格，这与下一层的网格描述模块是对应的。同时在该节点描述模块的中第69～81行还包括了MPEG触觉(MPEG_Haptic)，MPEG触觉的媒体参考列表(Media_reference)中包括两个大括号，说明该节点描述模块包括两个媒体参考数组(第72～75行的媒体参考数组和第76～79行的媒体参考数组)。第72～75行的媒体参考数组通过第73行的"haptic_id":0指出了第72～75行的媒体参考数组对应的媒体文件为MPEG媒体的媒体列表中的第一个媒体描述模块描述的媒体文件，并通过第74行的"accessor":1指出了用于访问对应媒体文件的解析数据的访问器为访问器列表(accessors)中的第二个访问器描述模块所描述的访问器。第76～79行的媒体参考数组通过第77行的"haptic_id":1指出了第76～79行的媒体参考数组对应的媒体文件为MPEG媒体的媒体列表中的第二个媒体描述模块描述的媒体文件，并通过第78行的"accessor":2指出了用于访问对应媒体文件的解析数据的访问器为访问器列表(accessors)中的第三个访问器描述模块所描述的访问器。从解析角度看，该节点列表中的节点描述模块指出了节点上所挂载的内容是一个三维网格，以及该三维网格的网格描述模块的索引为0，还指出了能够从MPEG媒体中索引到的触觉媒体文件为MPEG媒体中的第一媒体文件和第二媒体文件，同时还指出了访问第一个媒体文件的访问器和访问第二媒体文件的访问器的访问器描述模块分别为访问器列表(accessors)中的第二个访问器描述模块和第三个访问器描述模块所描述的访问器。
7、网格列表(meshes)：网格列表为第85～102行，网格列表中仅包含了一个大括号，说明该场景描述文件中仅包括一个网格描述模块(mesh)，该三维场景中只具有一个三维网格，且该三维网格与节点描述模块中的索引值为0的三维网格是同一网格。在表示该网格的大括号(网格描述模块)中，通过第87行的"name":"Hapticexample_mesh"指出了该三维网格的名称为"Hapticexample_mesh"，该名称仅用作辨识标记，通过第88行的"primitives"指出了该三维网格具有基元，即primitives。分别通过第90行的"attributes"、第93行的"mode"以及第95行的"MPEG_material_Haptic"，指出了基元中存在存在属性(attribute)、模式(mode)以及MPEG材质触觉(MPEG_material_Haptic)三类信息。通过第91行的"TEXCOORD_0":0，指示纹理坐标对应的访问器描述模块为访问器列表中的第一个访问器描述模块。通过93行的"mode":0指示该三维网格的拓扑为散点结构。通过第95～97行的"MPEG_material_Haptic":{"index":0}指示网格描述模块对应的三维网格承载MPEG触觉材质属性，且指向了更详细的下一层触觉材质数组列表(MPEG_material_Haptic)。从解析角度看，该网格列表明确了三维网格具有的 实际数据种类和三维网格的拓扑类型。
8、触觉材质数组列表(MPEG_material_Haptic)：触觉材质数组列表为第104～130行。触觉材质数组列表(MPEG_material_Haptic)中仅包括一个大括号，说明该场景描述文件中仅包括一个触觉材质数组(第106～128行的大括号所囊括的内容)。该触觉材质数组中分别通过第107～113、第114～120、121～127行定义了三种触觉材质属性，分别为刚度(stiffness)、温度(temperature)、振动(vibration)。每种触觉材质属性包括纹理数组(texture)的纹理坐标索引两类信息。刚度(stiffness)对应的描述模块为第107～113行，通过第109行的"index"及其值定义了刚度(stiffness)指向纹理列表中的第一个纹理描述模块，通过第110行的"texCoord"定义了刚度(stiffness)的触觉材质属性将以第一个纹理坐标集合数据包作为该触觉材质属性进行纹理贴图的坐标参考,通过第112行的"type":Low_resolution定义了刚度(stiffness)的触觉材质属性的类型是低分辨率(Low_resolution)，后续得到的刚度属性的触觉值的长度均为8bit长度；温度(temperature)对应的描述模块为第114～120行，通过第116行的"index"及其值定义了温度(temperature)指向纹理描述列表中的第二个纹理描述模块，通过第119行的"type":Reference定义了温度(temperature)的触觉材质纹理的类型是参考(Reference；振动(vibration)对应的描述模块为第121～127行，通过第123行的"index"及其值定义了振动(vibration)指向纹理描述列表中的第三个纹理描述模块(texture)，通过第107行的"type"及其值定义了振动(vibration)的触觉材质纹理的类型是参考(Reference)。从解析角度看，根据触觉材质数组列表(MPEG_material_Haptic)可以确定物体所包含的触觉材质属性的纹理种类及来源并且指向了更详细的下一层描述模块纹理描述模块。
9、纹理列表(textures)：纹理列表为第132～145行。纹理列表中包含三个大括号，说明该场景描述文件中包括三个纹理描述模块，且三个纹理描述模块分别用于描述刚度(stiffness)、温度(temperature)和振动(vibration)的纹理信息。刚度(stiffness)、温度(temperature)纹理描述内容包括采样器语法元素(sampler)和纹理贴图语法元素(source)，根据采样器语法元素(sampler)和纹理贴图语法元素(source)的值可以确定刚度(stiffness)和温度(temperature)的采样器描述模块和纹理贴图的访问地址。纹理贴图的来源与下一层级的纹理贴图列表(images)呼应。
10、纹理贴图列表(images)：纹理贴图列表为第147～157行。纹理贴图列表中包含三个大括号，分别对应刚度(stiffness)、温度(temperature)和振动(vibration)的纹理描述模块中所引用的纹理贴图的访问地址。根据第148～150行的{"uri":"stiffnessTexture.png"}可以确定刚度(stiffness)的纹理贴图的访问地址为stiffnessTexture.png。根据第151～153行的{"uri":"temperatureTexture.png"}可以确定温度(temperature)的纹理贴图的访问地址为temperatureTexture.png。根据第154～156行的{"uri":"Texture.png"}可以确定振动(vibration)的纹理贴图的访问地址为vibrationTexture.png。
11、采样器列表(samplers)，采样器列表为第159～164行。采样器列表中包含了一个大括号，说明了该场景描述文件包括一个采样器描述模块，该三维场景的显示仅需要使用一个采样器，且可以通过160～163行的"magFilter":9729、"minFilter":9987、"wrapS":33648、"wrapT":33648确定对应的采样器的描述信息。
12、缓存器列表(buffers)：缓存器列表为第215～240行。缓存器列表中包含了三个大括 号，说明了该场景描述文件包括三个缓存器描述模块，该三维场景的显示仅需要使用三个缓存器。根据第216～219行的第一个大括号中(第一个缓存器描述模块)的内容可以确定第一个缓存器的大小为1000，需要存储的媒体文件的地址为"AnimatedBody.bin"，媒体文件包括需要对特定三维坐标添加纹理信息的坐标内容。根据第220～229行第二大括号中(第二个缓存器描述模块)包括MPEG环形缓存器(MPEG_buffer_circular)，可以确定第二个缓存器是一个使用MPEG扩展进行改造的环形缓存器。第225行的"media:0说明了该环形缓存器中的数据来源是前文中MPEG媒体(MPEG_media)内声明的第一个媒体文件，第226行的"tracks":"#trackIndex＝1"说明了访问触觉媒体封装文件时应该参考索引为1的轨道，在这里不对索引为1的轨道做限定，它可以是单轨道封装的触觉文件的唯一轨道，也可以是多轨道封装的触觉文件的参考轨道。此外，根据第224行的"count":3，还可以确定MPEG环形缓存器具有三个存储环节，根据第221行的"byteLength":2000，还可以确定MPEG环形缓存器的字节长度(容量)为2000字节。第三个缓存器描述模块声明的信息与第二个缓存器描述模块声明的信息类似，再次不再赘述。从解析角度看，缓存器列表实现了将MPEG媒体(MPEG_media)中声明的触觉媒体文件对应到缓存器，或者说实现了缓存器对此前仅声明但未使用的触觉媒体文件进行了引用。需要说明的是，在该处引用的触觉媒体文件是未经处理的触觉媒体文件的封装文件，触觉媒体文件的封装文件需要经过媒体访问函数的处理，才能够提取出可直接用于渲染的信息。
13、缓存切片列表(bufferViews)：缓存切片列表为第197～213行。缓存切片列表中包含了三个并列的大括号，说明该场景描述文件中包括三个缓存切片描述模块。通过第198～202的第一个大括号(第一个缓存切片描述模块)的199行的"buffer":0，可以确定第一个缓存切片描述模块对应的缓存切片为缓存列表中的第一个缓存描述模块对应的缓存器的缓存切片，通过第200行的"byteLength":1000和201行的"byteOffset":0，声明了相应缓存切片的数据切片范围为第一个缓存器的前1000个字节。通过第203～207的第二个大括号(第二个缓存切片描述模块)的204行的"buffer":1，可以确定第二个缓存切片描述模块对应的缓存切片为缓存列表中的第二个缓存描述模块对应的缓存器的缓存切片，通过第205行的"byteLength":2000和206行的"byteOffset":0，声明了相应缓存切片的数据切片范围为第二个缓存器的前2000个字节。通过第208～212的第三个大括号(第三个缓存切片描述模块)的209行的"buffer":2，可以确定第三个缓存切片描述模块对应的缓存切片为缓存列表中的第三个缓存描述模块对应的缓存器的缓存切片，通过第210行的"byteLength":4000和211行的"byteOffset":0，声明了相应缓存切片的数据切片范围为第三个缓存器的前4000个字节。从解析角度看，缓存切片列表将媒体文件的数据进行了分组，有利于后续访问器描述模块的细化定义。
14、访问器列表(accessors)：访问器列表为第166～195行。访问器列表与缓存切片列表的结构类似，都包含了三个并列的大括号，说明该三维场景描述文件中包括三个访问器描述模块(accessor)，三维场景的显示需要通过三个访问器进行媒体数据的访问。通过第168行的"bufferView":0可以第一个访问器描述模块对应的访问器用于访问缓存切片列表中的第一个缓存切片，还可以通过169行的"componentType":5121和170行的"type":"VEC2"确定对应访问器中存储的数据格式是8位无符号数构成的二维向量，通过第171行的"count":500说明了这样需要通过该访问器访问的数据有500个，每个8位无符号数占用1个字节，因此该accessor 包含了1000字节的数据，这与索引值为0的缓存切片描述模块中的规定是相同的。此外，第二个大括号(第二个访问器描述模块)和第三个大括号(第三个访问器描述模块)中都具有MPEG时变访问器部分，说明这两个访问器指向的都是MPEG定义的时变媒体。通过第179行的"bufferView":1可以确定第一个MPEG时变访问器中的内容指向了索引值为1的缓存切片，且说明了对应访问器内的语法元素取值随时间变化。通过第190行的"bufferView":2可以确定第二个MPEG时变访问器中的内容指向了索引值为2的缓存切片，且说明了对应访问器内的语法元素取值随时间变化。第二个访问器描述模块和第三个访问器描述模块同样可以通过"componentType"、"type"以及"count"声明数据类型、访问器类型以及数据数量。从解析角度看，该访问器列表完善了对渲染所需数据的完全定义，比如在缓存切片和缓存器中缺少的数据类型就在对应的访问器描述模块中进行了定义。
2、支持获取参考类型的触觉材质属性的触觉值的显示引擎
在沉浸式媒体的场景描述框架的工作流程中，显示引擎的主要功能包括：解析场景描述文件获取渲染三维场景的方法；通过媒体接入函数API向媒体接入函数发送媒体访问指令或媒体数据处理指令；通过缓存API向缓存管理模块发送缓存管理指令；从缓存中取用经过处理的数据，并根据读取的数据完成三维场景及三维场景中物体的渲染与展示，因此支持包括触觉媒体的显示引擎的功能包括：1、能够解析包含触觉媒体场景描述文件，获取相应的三维场景渲染方法；2、能够通过媒体接入函数API与媒体接入函数传递媒体访问指令或媒体数据处理指令；其中，媒体访问指令或媒体数据处理指令来自于对包含触觉媒体的场景描述文件的解析结果；3、能够通过缓存管理API向缓存管理模块发送缓存管理指令；4、能够从缓存中取用经过处理的触觉媒体文件的解码数据，并根据取用的解码数据完成三维场景及三维场景中包含的触觉媒体文件的渲染与展示。
3、支持获取参考类型的触觉材质属性的触觉值的媒体接入函数API
在沉浸式媒体的场景描述框架的工作流程中，显示引擎可以通过解析场景描述文件获取渲染三维场景的方法，且需要将渲染三维场景的方法传递给媒体接入函数或基于渲染三维场景的方法向媒体接入函数发送指令，而将渲染三维场景的方法传递给媒体接入函数或基于渲染三维场景的方法向媒体接入函数发送指令的过程就是通过媒体接入函数API实现的。
在一些实施例中，显示引擎可以通过媒体接入函数API向媒体接入函数发送媒体访问指令或媒体数据处理指令。其中，显示引擎下达的指令来自于对含有触觉媒体的属性信息的场景描述文件的解析结果，这些指令可能包含了该媒体文件的地址、媒体文件的属性信息(媒体类型，使用的编解码器等)、对经过处理的触觉媒体数据及其他媒体数据的格式要求等。
在一些实施例中，媒体接入函数也可以通过媒体接入函数API向显示引擎请求媒体访问指令或媒体数据处理指令。
4、支持获取参考类型的触觉材质属性的触觉值的媒体接入函数
在沉浸式媒体的场景描述框架的工作流程中，媒体接入函数接收到显示引擎通过媒体接入函数API下发的媒体访问指令或媒体数据处理指令后，会执行显示引擎通过媒体接入函数API下发的媒体访问指令或媒体数据处理指令。例如：获取触觉媒体文件、为触觉媒体文件建立合适的管线以将触觉媒体文件的数据转化为指令中规定的格式、同时为处理为场景描述文件中规定格式的触觉媒体数据和其他媒体数据分配合适的缓存等。
在一些实施例中，媒体接入函数获取触觉媒体文件，包括：使用网络传输服务从服务器中下载触觉媒体文件。
在一些实施例中，媒体接入函数获取触觉媒体文件，包括：从本地存储空间中读取触觉媒体文件。
媒体接入函数获取触觉媒体文件后，需要对触觉媒体文件进行处理。不同类型的媒体文件的处理过程存在较大差异，为了实现广泛的媒体类型支持，也考虑到媒体接入函数的工作效率，于是在媒体接入函数中设计了多种管线，在处理媒体文件的过程中只启用匹配于媒体类型的管线即可。
5、支持获取参考类型的触觉材质属性的触觉值的缓存API
媒体接入函数通过触觉媒体文件的数据的处理后，媒体接入函数还需要将经过处理的数据需要以规范的排列结构交付给显示引擎，这需要将经过处理的数据正确地存储在缓存中，该工作由缓存管理模块完成，但是缓存管理模块需要通过缓存API从媒体接入函数或显示引擎获取缓存管理指令。
在一些实施例中，媒体接入函数可以通过缓存API向缓存管理模块发送缓存管理指令。其中，所述缓存管理指令为显示引擎通过媒体接入函数API向媒体接入函数发送的缓存管理指令。
在一些实施例中，显示引擎可以通过缓存API向缓存管理模块发送缓存管理指令。
即，缓存管理模块可以通过缓存API与媒体接入函数进行通信，也可以通过缓存API与显示引擎进行通信，且与媒体接入函数或显示引擎进行通信的目的都是实现缓存的管理。当缓存管理模块通过缓存API与媒体接入函数进行通信时，需要显示引擎将缓存管理指令先通过媒体接入函数API发送给媒体接入函数，媒体接入函数再通过缓存API将缓存管理指令通过缓存API发送给缓存管理模块；当缓存管理模块通过缓存API与显示引擎通信时，只需要显示引擎根据场景描述文件中解析出来的缓存管理信息生成缓存管理指令，并通过缓存API发送给缓存管理模块即可。
在一些实施例中，缓存管理指令可以包括：创建缓存的指令、更新缓存的指令、释放缓存的指令中的一个或多个。
6、支持获取参考类型的触觉材质属性的触觉值的缓存管理模块
在沉浸式媒体的场景描述框架的工作流程中，媒体接入函数通过管线完成了触觉媒体文件的数据的处理后，经过处理的数据需要以规范的排列结构交付给显示引擎，这需要将经过处理的数据正确地存储在缓存器中，该工作由缓存管理模块负责。
缓存管理模块实现了缓存的创建、更新、释放等管理操作，操作的指令通过缓存API接收。缓存管理的规则在场景描述文档中进行了记录，并通过显示引擎进行解析，最终由显示引擎或媒体接入函数下达给了缓存管理模块。当媒体文件被媒体接入函数处理完毕后，需要存储在合适的缓存中，再被显示引擎进行取用，缓存管理的作用就在于良好地管理这些缓存，使其与经过处理的媒体数据的格式相匹配，而不会打乱经过处理的媒体数据。媒体管理模块的具体设计方法应该以显示引擎以及媒体接入函数的设计为参照。
在上述内容的基础上，本申请一些实施例提供了一种场景描述文件的生成方法，参照图10所示，该场景描述文件的生成方法，包括如下步骤：S271～S275：
S271、在待渲染三维场景中包括触觉媒体文件时，确定所述触觉媒体文件的各个触觉材质属性的类型。
本申请实施例中的触觉材质属性可以包括：刚度(stiffness)、摩擦力(Friction)、触觉纹理(Vibrotactile texture)、温度(Temperature)、振动(Vibration)以及自定义属性中的一种或多种。触觉材质属性的类型包括：高分辨率(High_resolution)、低分辨率(Low_resolution)以及参考(Reference)。
需要说明的是，所述触觉媒体文件中的不同触觉材质属性的类型可以不同，因此触觉媒体文件中可以包括多种不同类型的触觉材质属性，且在确定所述触觉媒体文件的各个触觉材质属性的类型时，需要逐一确定每一个触觉材质属性的类型。
在上述步骤S271中，若所述触觉媒体文件的第一触觉材质属性的类型为参考(Reference)，则执行如下步骤S272和S273：
S272、在所述待渲染三维场景的场景描述文件的目标节点描述模块中添加访问器索引语法元素(accessor)。
其中，所述目标节点描述模块为包括承载所述第一触觉材质属性的目标三维网格的节点对应的节点描述模块。
在一些实施例中，在上述步骤S272(在所述待渲染三维场景的场景描述文件的目标节点描述模块中添加访问器索引语法元素)，包括：
在所述待渲染三维场景的场景描述文件的目标节点描述模块的MPEG触觉(MPEG_haptic)中添加访问器索引语法元素。
在一些实施例中，在所述待渲染三维场景的场景描述文件的目标节点描述模块的MPEG触觉中添加访问器索引语法元素的实现方式包括如下步骤2721～步骤2723:
步骤2721、在所述目标节点描述模块的MPEG触觉中添加媒体参考列表(Media_reference)。
步骤2722、在所述媒体参考列表中添加所述触觉媒体文件对应的媒体参考数组。
步骤2723、将所述访问器索引语法元素添加到所述触觉媒体文件对应的媒体参考数组中。
示例性的，当所述承载所述第一触觉材质属性的目标三维网格对应的网格描述模块的索引值为0，所述目标节点描述模块对应的节点的名称为：Hapticexample_node，则基于上述步骤2721～步骤2723在所述目标节点描述模块中添加所述访问器索引语法元素后，目标节点描述模块可以如下所示：
其中，n+6～n+9行的内容为在目标节点描述模块的MPEG触觉中添加的媒体参考列表，n+7～n+9行的内容为在所述媒体参考列表中添加的所述触觉媒体文件对应的媒体参考数组， n+8行的内容为所述触觉媒体文件对应的媒体参考数组中添加的访问器索引语法元素。
需要说明的是，待渲染三维场景中可能会包括多个触觉媒体文件，且目标节点描述模块对应的节点可能包括承载有多个触觉媒体文件的参考类型的触觉材质属性的三维网格，当目标节点描述模块对应的节点包括承载有多个触觉媒体文件的参考类型的触觉材质属性的三维网格时，需要在所述MPEG触觉的媒体参考列表中添加每一个触觉媒体文件的对应的媒体参考数组。
S273、将所述访问器索引语法元素的值设置为触觉媒体访问器对应的访问器描述模块的索引值。
其中，所述触觉媒体访问器为用于访问所述触觉媒体文件的解码数据的访问器。
承上实施例所述，若用于访问所述触觉媒体文件的触觉媒体访问器对应的访问器描述模块的索引值为2时，则在所述媒体参考数组中添加访问器索引语法元素(accessor)，以及将所述访问器索引语法元素的值设置为所述触觉媒体访问器对应的访问器描述模块的索引值之后，目标节点描述模块可以如下所示：
其中，第n+8行的"accessor":2为在所述媒体参考数组中添加的访问器索引语法元素以及所述访问器索引语法元素的值。“2”同时为用于访问所述触觉媒体文件的触觉媒体访问器对应的访问器描述模块的索引值。
本申请实施例提供的场景描述文件的生成方法在待渲染三维场景中包括触觉媒体文件，且所述触觉媒体文件的第一触觉材质属性的类型为参考时，首先在包括承载所述第一触觉材质属性的目标三维网格的节点对应的目标节点描述模块中添加访问器索引语法元素，并将所述访问器索引语法元素的值设置为用于访问所述触觉媒体文件的解码数据的触觉媒体访问器对应的访问器描述模块的索引值。由于本申请实施例提供的场景描述文件的生成方法在待渲染三维场景中包括触觉媒体文件，且所述触觉媒体文件的第一触觉材质属性的类型为参考时，在所述目标节点描述模块中添加了访问器索引语法元素，并将所述访问器索引语法元素的值设置为了用于访问所述触觉媒体文件的解码数据的触觉媒体访问器对应的访问器描述模块的索引值，因此在根据场景描述文件进行待渲染三维场景时，可以根据所述访问器索引语法元素及其值确定用于访问所述触觉媒体文件的解码数据的触觉媒体访问器，进而根据所述触觉媒体访问器从所述触觉媒体文件的解码数据中读取所述第一触觉材质属性的触觉值，因此本申请实施例可以支持获取参考类型的触觉材质属性的触觉值，解决相关技术中无法获取参考类型的触觉材质属性的触觉值的问题。
在一些实施例中，所述场景描述文件的生成方法还包括如下步骤1和步骤2：
步骤1、在所述触觉媒体文件对应的媒体参考数组中添加触觉媒体标识语法元素(haptic_id)。
步骤2、将所述触觉媒体标识语法元素的值设置为目标媒体描述模块的索引值。
其中，所述目标媒体描述模块为所述场景描述文件的MPEG媒体的媒体列表中所述触觉媒体文件对应的媒体描述模块。
由于上述实施例还会在所述媒体参考数组中添加触觉媒体标识语法元素，以及将所述触觉媒体标识语法元素的值设置为所述触觉媒体文件对应的目标媒体描述模块的索引值，因此上述实施例可以在三维场景中包括多个触觉媒体文件时，通过触觉媒体标识语法元素及其值索引不同的触觉媒体文件，进而支持在三维场景中添加多个触觉媒体文件。
示例性的，当所述触觉媒体文件对应的目标媒体描述模块的索引值为0时，在所述媒体参考数组中添加触觉媒体标识语法元素，以及将所述触觉媒体标识语法元素的值设置为所述目标媒体描述模块的索引值之后，所述触觉媒体文件对应的媒体参考数组可以如下所示：
在一些实施例中，在上步骤S273(将所述访问器索引语法元素的值设置为用于访问所述触觉媒体文件的解码数据访问器对应的访问器描述模块的索引值)之前，本申请实施例提供的方法还包括如下步骤a～步骤c：
步骤a、在所述场景描述文件的缓存器列表(buffers)中添加触觉媒体缓存器对应的缓存器描述模块。
其中，所述触觉媒体缓存器为用于缓存所述触觉媒体文件的解码数据的缓存器。
在一些实施例中，上述步骤a(在所述场景描述文件的缓存器列表中添加触觉媒体缓存器对应的缓存器描述模块)包括如下步骤a1～a5中的至少一项：
步骤a1、在所述触觉媒体缓存器对应的缓存器描述模块中添加字节长度语法元素(byteLength)，并将所述字节长度语法元素的值设置为所述触觉媒体文件的字节长度。
步骤a2、在所述触觉媒体缓存器对应的缓存器描述模块中添加MPEG环形缓存器。
步骤a3、在所述MPEG环形缓存器中添加数量语法元素(count)，以及根据预配置的所述MPEG环形缓存器的存储环节数量设置所述数量语法元素的值。
步骤a4、在所述MPEG环形缓存器中添加媒体索引语法元素(media)，并根据所述触觉媒体文件对应的媒体描述模块的索引值设置所述媒体索引语法元素的值。
步骤a5、在所述MPEG环形缓存器中添加轨道索引语法元素(tracks)，并根据所述MPEG环形缓存器存储的数据的轨道索引值设置所述轨道索引语法元素的值。
示例性的，若在所述场景描述文件的缓存器列表中添加触觉媒体缓存器对应的缓存器描述模块包括上步骤a1～a5中的每一项，且触觉媒体文件的字节长度为8000，所述预配置的所述MPEG环形缓存器的存储环节数量为5，所述触觉媒体文件对应的媒体描述模块的索引值为0，所述MPEG环形缓存器存储的数据的轨道索引值为1，则在所述场景描述文件的缓存器列表中添加的触觉媒体缓存器对应的缓存器描述模块可以如下所示：
步骤b、在所述场景描述文件的缓存切片列表(bufferViews)中添加所述触觉媒体缓存器的缓存切片对应的缓存切片描述模块。
在一些实施例中，上述步骤b(在所述场景描述文件的缓存切片列表中添加所述触觉媒体缓存器的缓存切片对应的缓存切片描述模块)包括如下步骤b1～b3中的至少一项：
步骤b1、在所述触觉媒体缓存器的缓存切片对应的缓存切片描述模块中添加缓存器索引语法元素(buffer)，并根据所述触觉媒体缓存器对应的缓存器描述模块的索引值设置所述缓存器索引语法元素的值。
步骤b2、在所述触觉媒体缓存器的缓存切片对应的缓存切片描述模块中添加字节长度语法元素(byteLength)，并根据对应缓存切片的容量设置所述字节长度语法元素的值。
步骤b3、在所述触觉媒体缓存器的缓存切片对应的缓存切片描述模块中添加偏移量语法元素，并根据对应缓存切片的存储数据的偏移量设置所述偏移量语法元素的值。
示例性的，若在所述场景描述文件的缓存切片列表(bufferViews)中添加所述触觉媒体缓存器的缓存切片对应的缓存切片描述模块包括上步骤b1～b3中的每一项，触觉媒体缓存器对应的缓存器描述模块的索引值为1，触觉媒体缓存器的容量为8000，且触觉媒体缓存器仅包括一个缓存切片，则在所述场景描述文件的缓存切片列表中添加所述触觉媒体缓存器的缓存切片对应的缓存切片描述模块可以如下所示：
步骤c、在所述场景描述文件的访问器列表(accessors)中添加用于访问所述触觉媒体缓存器的缓存切片中的数据的所述触觉媒体访问器对应的访问器描述模块。
由于所述触觉媒体缓存器的缓存切片中的数据为所述触觉媒体文件的解码数据，因此用于访问所述触觉媒体缓存器的缓存切片中的数据的访问器即为用于访问所述触觉媒体文件的解码数据的访问器。
在一些实施例中，上述步骤c(在所述场景描述文件的访问器列表(accessors)中添加用 于访问所述触觉媒体缓存器的缓存切片中的数据的所述触觉媒体访问器对应的访问器描述模块)包括如下步骤c1～c6中的至少一项：
步骤c1、在所述访问器描述模块中添加数据类型语法元素(componentType)，并根据所述第一触觉材质属性的触觉值的数据类型设置所述数据类型语法元素的值。
步骤c2、在所述访问器描述模块中添加访问器类型语法元素(type)，并根据预配置的访问器类型设置所述访问器类型语法元素的值。
步骤c3、在所述访问器描述模块中添加数量语法元素(count)，并根据第一触觉材质属性的触觉值的数量设置所述数量语法元素的值。
步骤c4、在所述访问器描述模块中添加MPEG时变访问器(MPEG_accessor_timed)。
步骤c5、在所述MPEG时变访问器中添加缓存切片索引语法元素(bufferView)，并根据所述触觉媒体缓存器的缓存切片对应的缓存切片描述模块的索引值设置所述缓存切片索引语法元素的值。
步骤c6、在所述MPEG时变访问器中添加时变语法元素，并根据对应的访问器内的语法元素的取值是否随时间变化设置所述时变语法元素的值。
示例性的，若在所述场景描述文件的访问器列表(accessors)中添加用于访问所述触觉媒体缓存器的缓存切片中的数据的所述触觉媒体访问器对应的访问器描述模块包括上步骤c1～c6中的每一项，所述第一触觉材质属性的触觉值的数据类型为5123，所述预配置的访问器类型为SCALAR，所述第一触觉材质属性的触觉值的数量为3000，所述触觉媒体缓存器的缓存切片对应的缓存切片描述模块的索引值为1，且对应的访问器内的语法元素的取值随时间变化，则在所述场景描述文件的访问器列表(accessors)中添加的用于访问所述触觉媒体缓存器的缓存切片中的数据的所述触觉媒体访问器对应的访问器描述模块可以如下所示：
在一些实施例中，在将所述触觉媒体标识语法元素的值设置为所述目标媒体描述模块的索引值之前，所述场景描述文件的生成方法还包括：
在所述场景描述文件的MPEG媒体(MPEG_media)的媒体列表(media)中添加所述触觉媒体文件对应的目标媒体描述模块。
在一些实施例中，上述(在所述场景描述文件的MPEG媒体的媒体列表中添加所述触觉媒体文件对应的目标媒体描述模块)包括如下步骤11～19中的至少一项：
步骤11、在所述目标媒体描述模块中添加媒体名称语法元素(name)，并根据所述触觉媒体文件的名称设置所述媒体名称语法元素的值。
步骤12、在所述目标媒体描述模块中自动播放语法元素(autoplay)，并根据所述触觉媒体文件是否需要自动播放设置所述自动播放语法元素的值。
步骤13、在所述目标媒体描述模块中循环播放语法元素(loop)，并根据所述触觉媒体文件是否需要循环播放设置所述循环播放语法元素的值。
步骤14、在所述目标媒体描述模块中添加可选项(alternatives)。
步骤15、在所述可选项中添加媒体类型语法元素(mimeType)，并根据所述触觉媒体文件包含的数据设置所述媒体类型语法元素的值。
在一些实施例中，根据所述触觉媒体文件包含的数据设置所述媒体类型语法元素的值包括：
当所述触觉媒体文件包括音频和/或视频数据时，将所述目标媒体描述模块的可选项中的媒体类型语法元素的值设置为第一预设值；
当所述触觉媒体文件不包括音频和/或视频数据时，将所述目标媒体描述模块的可选项中的媒体类型语法元素的值设置为第二预设值。
示例性的，所述第一预设值可以为video/mp4，所述第一预设值可以为haptics/mp4。即，当所述触觉媒体文件仅包括触觉数据时，将媒体类型语法元素的值设置为haptics/mp4，当所述触觉媒体文件包括不但触觉数据，而且还包括音频和/或视频数据时将媒体类型语法元素的值设置为video/mp4。
当所述触觉媒体文件包括不但触觉数据，而且还包括音频和/或视频数据时，将媒体类型语法元素的值设置为video/mp4可以与现有的.mp4媒体文件向后兼容。
步骤16、在所述可选项中添加访问地址语法元素(uri)，并将所述访问地址语法元素的值设置为所述触觉媒体文件的访问地址。
步骤17、在所述可选项中添加轨道数组(tracks)。
步骤18、在所述轨道数组中添加轨道索引语法元素(track)，并将所述目标媒体描述模块的可选项的轨道数组中的轨道索引语法元素的值设置为所述触觉媒体文件的各个码流轨道的索引值。
即，当编码后的触觉媒体作为MPEG_media.alternative.tracks中的一项被场景描述文件引用，并且被引用的项符合ISO/IEC 23090-32标准里关于ISOBMFF中track的规定时：对于单轨道封装的触觉媒体文件，在MPEG_media中引用的轨道就是触觉封装的单轨道。例如，触觉媒体被ISOBMFF封装成一条MIHS轨道,则在MPEG_media中引用的轨道就是这条MIHS轨道。对于多轨道封装的触觉媒体文件，在MPEG_media中引用的轨道就是触觉媒体被封装成的多条轨道轨道。例如，提前在封装文件的配置信息alternate_group的TrackHeaderBox中定义非零值，说明此时触觉媒体被封装成多条MIHS轨道，这些轨道也是MPEG_media索引到的轨道。
步骤19、在所述轨道数组中添加编解码参数语法元素(codec)，并根据所述触觉媒体文件的各个码流轨道中的数据的编码参数和ISO/IEC 23090-32触觉数据传输标准的规定，设置所述编解码参数语法元素的值。
在IETF RFC 6381中定义的，包含在轨道数组(tracks)中的媒体的编解码器参数。当轨道数组包含不同类型的编解码器(例如AdaptationSet包含具有不同编解码器的表示)时，编解码参数语法元素可以用逗号分隔的编解码器值列表来表示。
当媒体数据为触觉数据时，应遵循ISO/IEC 23090-32触觉数据传输标准(Carriage of  haptics data)里的相关详细规定。例如在触觉数据采用ISMBOFF封装时，该编解码参数语法元素第一组元素是表示ISO/IEC 23090-31中编解码器的代码，编解码参数语法元素的值的形式如下：‘codec＝mih1.oo’其中“mih1”是触觉数据封装在ISOBMFF中的一类数据格式说明，该值并不唯一，允许其他符合ISOBMFF封装的样本格式名称进行替换。其中“oo”是MP4注册机构网站“对象类型”页面上定义的对象类型指示值。
示例性的，若在所述场景描述文件的MPEG媒体的媒体列表中添加所述触觉媒体文件对应的目标媒体描述模块包括上步骤11～19中的每一项，所述触觉媒体文件的名称为Hapticexample，所述触觉媒体文件需要自动播放和循环播放，所述触觉媒体文件不包括音频数据和/或视频数据，所述触觉媒体文件的访问地址为www.example.com/Hapticexample1.mp4，所述触觉媒体文件为单轨道封装文件，且码流轨道的索引值为1，则所述目标媒体描述模块可以如下所示：
在一些实施例中，所述场景描述文件的生成方法还包括：
在所述场景描述文件的场景列表(scenes)中添加所述待渲染三维场景对应的目标场景描述模块，以及在所述目标场景描述模块的节点索引列表(nodes)中添加所述待渲染场景中的各个节点对应的节点描述模块的索引值。
示例性的，若所述待渲染场景包括两个节点，且该两个节点对应的节点描述模块的索引值分别为0和1，则所述目标场景描述模块可以如下所示：
在一些实施例中，所述场景描述文件的生成方法还包括：
在所述场景描述文件的节点列表(nodes)中添加所述待渲染场景中的各个节点对应的节点描述模块；以及在各个节点描述模块的网格索引列表中添加各个节点中的三维网格对应的网格描述模块的索引值。
需要说明的是，在所述场景描述文件的节点列表中添加所述待渲染场景中的各个节点对应的节点描述模块时，需要根据是否包括承载有的触觉材质属性的三维网格以及承载的触觉 材质属性的类型将所述待渲染场景中的节点分为三类，对于不包括承载有的触觉材质属性的三维网格的节点，对应的节点描述模块仅需要在各个节点描述模块的网格索引列表中添加各个节点中的三维网格对应的网格描述模块的索引值即可。对于包括承载有类型为不为参考(高分辨率或低分辨率)的触觉材质属性的三维网格的节点，则需要进一步在对应的节点描述模块中添加MPEG触觉(MPEG_haptic)。对于包括承载有参考类型的触觉材质属性的三维网格的节点，对应的节点描述模块生成方式与上述实施例中的目标节点描述模块的生成方式与上述实施例类似，在此不再赘述。
示例性的：某一节点不包括承载有的触觉材质属性的三维网格，该节点的名称为Hapticexample_node1，且该节点仅包括一个对应的网格描述模块的索引值为2的三维网格，则该节点对应的节点描述模块可以如下所示：
在一些实施例中，所述场景描述文件的生成方法还包括：在所述场景描述文件的网格列表中添加所述目标三维网格对应的网格描述模块。
在所述场景描述文件的场景列表中添加所述待渲染场景的各个节点挂载的三维网格对应的网格描述模块时，需要根据是否承载有触觉材质属性将所述待渲染场景中的三维网格分为两类，分别为未承载触觉材质属性的三维网格和承载了触觉材质属性的三维网格。
对于未承载触觉材质属性的三维网格，在所述场景描述文件的网格列表中添加对应的网格描述模块，包括如下步骤d1～d5中的至少一项：
步骤d1、在该三维网格对应的网格描述模块中添加网格名称语法元素(name)，并根据该三维网格的名称设置所述网格名称语法元素的值。
步骤d2、在该三维网格对应的网格描述模块中添加属性(primitives)。
步骤d3、在所述属性中添加基元(attributes)。
步骤d4、在所述基元中添加纹理坐标语法元素(TEXCOORD_n)，并将所述纹理坐标语法元素的值设置为纹理坐标访问器对应的访问器描述模块的索引值。
其中，所述纹理坐标访问器为用于访问该三维网格的纹理坐标的访问器。
步骤d5、在该三维网格对应的网格描述模块中添加模式语法元素(mode)，并根据该三维网格的拓扑结构类型设置所述模式语法元素的值。
示例性的，若在所述场景描述文件的网格列表中添加所述目标三维网格对应的网格描述模块包括上述步骤d1～d5中的每一项，某一三维网格未承载触觉材质属性，该三维网格的名称为example_mesh，且该三维网格的拓扑结构为散点，则在所述场景描述文件的网格列表中添加的该三维网格对应的网格描述模块可以如下所示：

对于承载了触觉材质属性的三维网格(包括承载所述第一触觉材质属性的目标三维网格)，在所述场景描述文件的网格列表中添加对应的网格描述模块，包括如下步骤d1～d5中的至少一项的同时，还包括如下步骤d6和d7：
步骤d6、在该三维网格对应的网格描述模块中添加MPEG触觉材质(MPEG_material_Haptic)。
步骤d7、在所述MPEG触觉材质中添加数组索引语法元素(index)，并将所述数组索引语法元素的值设置为该三维网格的触觉材质属性数组的索引值。
示例性的，若在所述场景描述文件的网格列表中添加所述目标三维网格对应的网格描述模块包括上述步骤d1～d7中的每一项，某一三维网格承载了触觉材质属性，该三维网格的名称为Hapticexample_mesh，该三维网格的拓扑结构为散点，且该三维网格的触觉材质属性数组的索引值为2，则在所述场景描述文件的网格列表中添加的该三维网格对应的网格描述模块可以如下所示：
在一些实施例中，在将所述纹理坐标语法元素(TEXCOORD_n)的值设置为纹理坐标访问器对应的访问器描述模块的索引值之前，本申请实施例提供的方法还包括如下步骤e～步骤g：
步骤e、在所述场景描述文件的缓存器列表(buffers)中添加纹理坐标缓存器对应的缓存器描述模块。
其中，所述纹理坐标缓存器为用于缓存所述目标三维网格的纹理坐标的缓存器。
在一些实施例中，上述步骤e(在所述场景描述文件的缓存器列表中添加纹理坐标缓存器对应的缓存器描述模块)包括如下步骤e1和e2中的至少一项：
步骤e1、在所述纹理坐标缓存器对应的缓存器描述模块中添加字节长度语法元素(byteLength)，并将所述字节长度语法元素的值设置为所述目标三维网格的纹理坐标的字节长度。
步骤e2、在所述纹理坐标缓存器对应的缓存器描述模块中添加纹理坐标访问地址语法元素(uri)，并将所述纹理坐标访问地址语法元素的值设置为所述目标三维网格的纹理坐标的访 问地址。
示例性的，若在所述场景描述文件的缓存器列表(buffers)中添加纹理坐标缓存器对应的缓存器描述模块包括上步骤e1和e2，且所述目标三维网格的纹理坐标的字节长度为1000，所述目标三维网格的纹理坐标的访问地址为AnimatedBody.bin，则在所述场景描述文件的缓存器列表中添加的纹理坐标缓存器对应的缓存器描述模块可以如下所示：
如上所示，纹理坐标缓存器对应的缓存器描述模块与触觉媒体缓存器对应的缓存器描述模块的最大不同之处在于，纹理坐标缓存器对应的缓存器描述模块中年再索引MPEG媒体中声明的媒体文件，而是直接添加纹理坐标的访问地址，通过纹理坐标的访问地址进行纹理坐标的访问。
步骤f、在所述场景描述文件的缓存切片列表(bufferViews)中添加所述纹理坐标缓存器的缓存切片对应的缓存切片描述模块。
在一些实施例中，上述步骤f(在所述场景描述文件的缓存切片列表中添加所述纹理坐标缓存器的缓存切片对应的缓存切片描述模块)包括如下步骤f1～f3中的至少一项：
步骤f1、在所述纹理坐标缓存器的缓存切片对应的缓存切片描述模块中添加缓存器索引语法元素(buffer)，并根据所述纹理坐标缓存器对应的缓存器描述模块的索引值设置所述缓存器索引语法元素的值。
步骤f2、在所述纹理坐标缓存器的缓存切片对应的缓存切片描述模块中添加字节长度语法元素(byteLength)，并根据对应缓存切片的容量设置所述字节长度语法元素的值。
步骤f3、在所述纹理坐标缓存器的缓存切片对应的缓存切片描述模块中添加偏移量语法元素，并根据对应缓存切片的存储数据的偏移量设置所述偏移量语法元素的值。
示例性的，若在所述场景描述文件的缓存切片列表(bufferViews)中添加所述纹理坐标缓存器的缓存切片对应的缓存切片描述模块包括上步骤f1～f3中的每一项，纹理坐标缓存器对应的缓存器描述模块的索引值为0，纹理坐标缓存器的容量为4000，且纹理坐标缓存器仅包括一个缓存切片，则在所述场景描述文件的缓存切片列表中添加所述纹理坐标缓存器的缓存切片对应的缓存切片描述模块可以如下所示：
步骤g、在所述场景描述文件的访问器列表(accessors)中添加用于从所述纹理坐标缓存器的缓存切片中访问所述第一触觉材质属性的触觉值的访问器对应的访问器描述模块。
在一些实施例中，上述步骤g(在所述场景描述文件的访问器列表(accessors)中添加用于访问所述纹理坐标缓存器的缓存切片中的数据的访问器对应的访问器描述模块)包括如下步骤g1～g4中的至少一项：
步骤g1、在所述访问器描述模块中添加缓存切片索引语法元素(bufferView)，并根据所 述纹理坐标缓存器的缓存切片对应的缓存切片描述模块的索引值设置所述缓存切片索引语法元素的值。
步骤g2、在所述访问器描述模块中添加数据类型语法元素(componentType)，并根据所述第一触觉材质属性的触觉值的数据类型设置所述数据类型语法元素的值。
步骤g3、在所述访问器描述模块中添加访问器类型语法元素(type)，并根据预配置的访问器类型设置所述访问器类型语法元素的值。
步骤g4、在所述访问器描述模块中添加数量语法元素(count)，并根据第一触觉材质属性的触觉值的数量设置所述数量语法元素的值。
示例性的，若在所述场景描述文件的访问器列表中添加用于从所述纹理坐标缓存器的缓存切片中访问所述第一触觉材质属性的触觉值的访问器对应的访问器描述模块包括上步骤g1～g4中的每一项，所述目标三维网格的文理坐标的数据类型为5121，所述预配置的访问器类型为VEC2，所述目标三维网格的文理坐标的数量为800，所述纹理坐标缓存器的缓存切片对应的缓存切片描述模块的索引值为0，则在所述场景描述文件的访问器列表中添加的用于从所述纹理坐标缓存器的缓存切片中访问所述第一触觉材质属性的触觉值的访问器对应的访问器描述模块可以如下所示：
在一些实施例中，在将所述数组索引语法元素的值设置为所述目标三维网格的触觉材质属性数组的索引值之前，本申请实施例提供的方法还包括如下步骤h～步骤k：
步骤h、在所述场景描述文件中添加触觉材质数组列表(MPEG_material_Haptic)。
步骤j、在所述触觉材质数组列表中添加各个承载了触觉材质属性的三维网格(包括承载所述第一触觉材质属性的目标三维网格)的触觉材质属性数组。
其中，各个三维网格的触觉材质属性数组包括各个三维网格所承载的各个触觉材质属性对应的触觉材质属性描述模块，任一触觉材质属性对应的触觉材质属性描述模块包括：纹理数组(texture)和属性类型语法元素(type)；所述纹理数组包括：纹理索引语法元素(index)和纹理坐标索引语法元素(texCoord)。
由于各个三维网格的触觉材质属性数组包括各个三维网格所承载的各个触觉材质属性对应的触觉材质属性描述模块，任一触觉材质属性对应的触觉材质属性描述模块包括：纹理数组和属性类型语法元素；所述纹理数组包括：纹理索引语法元素和纹理坐标索引语法元素，因此上述步骤j(在所述触觉材质数组列表中添加各个承载了触觉材质属性的三维网格的触觉材质属性数组)，包括如下步骤j1～j3以生成所述场景描述文件的触觉材质属性数组列表(MPEG_material_Haptic)：
步骤j1、在各个三维网格的触觉材质属性数组中添加各个各个三维网格所承载的各个触觉材质属性对应的触觉材质属性描述模块。
步骤j2、在各个触觉材质属性描述模块中添加纹理数组(texture)和属性类型语法元素 (type)。
步骤j3、在各个纹理数组中添加纹理索引语法元素和纹理坐标索引语法元素(texCoord)。
步骤i、将各个触觉材质属性对应的触觉材质属性描述模块的纹理数组中的纹理索引语法元素(index)的值设置为各个触觉材质属性对应的纹理描述模块的索引值。
步骤j、将各个触觉材质属性对应的触觉材质属性描述模块的纹理数组中的纹理坐标索引语法元素(texCoord)的值设置为对应的三维网格的纹理坐标语法元素的索引值。
步骤k、将各个触觉材质属性对应的触觉材质属性描述模块中的属性类型语法元素的值设置为各个触觉材质属性的类型。
示例性的，若某一三维网格承载的触觉材质属性包括刚度和温度，所述刚度的类型的低分辨率、所述温度的类型为高分辨率，则该三维网格的触觉材质属性数组可以如下所示：
示例性的，若所述待渲染场景中的两个三维网格承载有触觉材质属性，一个三维网格承载的触觉材质属性为刚度，类型为低分辨率，另一个网格描述模块承载的触觉材质属性为温度，类型为参考，则在所述场景描述文件中添加的触觉材质数组列表可以如下所示：

在一些实施例中，所述场景描述文件的生成方法还包括：在将各个触觉材质属性对应的触觉材质属性描述模块的纹理数组中的纹理索引语法元素的值设置为各个触觉材质属性对应的纹理描述模块的索引值之前，所述方法还包括执行如下步骤l～n以生成所述场景描述文件的纹理列表(textures)：
步骤l、在所述场景描述文件的纹理列表中添加各个触觉材质属性对应的纹理描述模块。
其中，任一触觉材质属性对应的纹理描述模块包括：纹理贴图语法元素(source)和采样器语法元素(sampler)。
步骤m、将各个触觉材质属性对应的纹理描述模块中的纹理贴图语法元素的值设置为各个触觉材质属性的纹理贴图的访问地址的索引值。
步骤n、将各个触觉材质属性对应的纹理描述模块中的采样器语法元素的值设置为各个触觉材质属性的采样器对应的采样器描述模块的索引值。
示例性的，若某一触觉材质属性对应的纹理贴图的访问地址的索引值为2，且该触觉材质属性采样器对应的采样器描述模块的索引值为3，则在所述场景描述文件的纹理列表中添加的该触觉材质属性对应的纹理描述模块可以如下所示：
示例性的，若所述场景描述文件中包括三个触觉材质属性对应的纹理描述模块，则所述场景描述文件的纹理列表可以如下所示：
在一些实施例中，所述场景描述文件的生成方法还包括：在将各个触觉材质属性对应的纹理描述模块中的纹理贴图语法元素的值设置为各个触觉材质属性的纹理贴图的访问地址的 索引值之前，在所述场景描述文件的纹理贴图列表(images)中添加所述触觉媒体文件的各个触觉材质属性的纹理贴图的访问地址。
需要说明的是，若某一触觉材质属性的类型为高分辨率或低分辨率，则该触觉材质属性的纹理贴图中存储的是该触觉材质属性的触觉值，而若某一触觉材质属性的类型为参考，则该触觉材质属性的纹理贴图中存储的是该触觉材质属性对应触觉媒体文件的索引值。
示例性的，若某一触觉材质属性的纹理贴图的访问地址为temperatureTexture.png，则在所述场景描述文件的纹理列表中添加的该触觉材质属性的纹理贴图的访问地址可以如下所示：
示例性的，若所述场景描述文件中包括三个触觉材质属性的纹理贴图的访问地址，则所述场景描述文件的纹理贴图列表可以如下所示：
在一些实施例中，所述场景描述文件的生成方法还包括：在将各个触觉材质属性对应的纹理描述模块中的采样器语法元素的值设置为各个触觉材质属性的采样器对应的采样器描述模块的索引值之前，在所述场景描述文件的采样器列表(samplers)中添加所述触觉媒体文件的各个触觉材质属性的采样器对应的采样器描述模块。
示例性的，若某一触觉材质属性的采样器的Mag滤波器参数为：9729、Min滤波器参数为：9987、水平包裹参数为：33648：数值包裹参数为:33648，则该触觉材质属性的采样器的描述模块可以如下所示：
示例性的，场景描述文件中仅包括一个采样器描述模块，则场景描述文件中的采样器列表可以如下所示：

在一些实施例中，所述场景描述文件的生成方法还包括：
在所述场景描述文件中添加数字资产描述模块(asset)，在所述数字资产描述模块中添加版本语法元素(version)以及在所述场景描述文件为基于glTF2.0版本编写场景描述文时，将所述版本语法元素的值设置为2.0。
示例性的，在所述场景描述文件添加的数字资产描述模块可以如下所示：
在一些实施例中，所述场景描述文件的生成方法还包括：
在所述场景描述文件中添加扩展使用描述模块(extensionsUsed)，并在所述扩展使用描述模块中添加所述场景描述文件使用的各个MPGE对glTF2.0版本的场景描述文件的扩展。
示例性的，场景描述文件中使用的MPGE扩展包括：MPGE媒体(MPEG_media)、MPEG环形缓存器(MPEG_buffer_circular)、MPEG时变访问器(MPEG_accessor_timed)、MPEG触觉(MPEG_Haptic)以及MPEG触觉材质(MPEG_material_Haptic)，则在所述场景描述文件中添加的扩展使用描述模块可以如下所示：
在一些实施例中，所述场景描述文件的生成方法还包括：
在所述场景描述文件中添加场景声明(scene)，并将所述场景声明的值设置为所述待渲染场景对应的场景描述模块的索引值。
示例性的，待渲染场景对应的场景描述模块的索引值为1，则在所述场景描述文件中添加场景声明可以如下所示：
本申请一些实施例还提供了一种场景描述文件的解析方法，参照图11所示，该场景描述文件的生成方法，包括如下步骤：S281～S283：
S281、根据场景描述文件确定待渲染三维场景中的各个三维网格承载的各个触觉材质属性的类型。
本申请实施例中的触觉材质属性可以包括：刚度(stiffness)、摩擦力(Friction)、触觉纹理(Vibrotactile texture)、温度(Temperature)、振动(Vibration)以及自定义类型中的一种或多种。触觉材质属性的类型包括：高分辨率(High_resolution)、低分辨率(Low_resolution)以及参考(Reference)。
需要说明的是，一个三维网格可以承载多个不同类型的触觉材质属性，因此根据场景描述文件确定待渲染三维场景中的各个三维网格承载的各个触觉材质属性的类型时，需要逐一确定每一个触觉材质属性的类型。
在上步骤S281中，若所述待渲染三维场景中的目标三维网格承载的第一触觉材质属性的类型为参考(Reference)，则执行如下步骤S282和S283：
S282、获取目标节点描述模块的MPEG触觉(MPEG_haptic)的媒体参考列表(Media_reference)的目标媒体参考数组中的访问器索引语法元素(accessor)声明的索引值。
其中，所述目标节点描述模块为包括所述目标三维网格的节点对应的节点描述模块，所述目标媒体参考数组为所述第一触觉材质属性所属的触觉媒体文件对应的媒体参考数组。
由于目标节点描述模块中可能会包括多个媒体参考数组，因此在获取目标媒体参考数组中的访问器索引语法元素之前，首先需要确定第一触觉材质属性所属的触觉媒体文件，然后将第一触觉材质属性所属的触觉媒体文件对应的媒体参考数组确定为所述目标媒体参考数组。
S283、根据所述访问器索引语法元素声明的索引值获取用于访问所述目标触觉媒体文件的解码数据的触觉媒体访问器的描述信息。
本申请实施例提供的场景描述文件解析方法在获取场景描述文件后，首先根据场景描述文件确定待渲染三维场景中的各个三维网格承载的各个触觉材质属性的类型，并在待渲染三维场景中的目标三维网格承载的第一触觉材质属性的类型为参考时，获取包括所述目标三维网格的节点对应的节点描述模块，并获取目标节点描述模块的MPEG触觉中第一触觉材质属性所属的目标触觉媒体文件对应的目标媒体参考数组中的访问器索引语法元素声明的索引值，以及根据所述访问器索引语法元素声明的索引值获取用于访问所述目标触觉媒体文件的解码数据的触觉媒体访问器的描述信息。由于本申请实施例可以在目标三维网格承载的第一触觉材质属性的类型为参考时，获取目标节点描述模块的MPEG触觉的媒体参考列表的目标媒体参考数组中的访问器索引语法元素声明的索引值，以及根据所述访问器索引语法元素声明的索引值获取用于访问所述目标触觉媒体文件的解码数据的触觉媒体访问器的描述信息，因此本申请实施例可以根据所述触觉媒体访问器的描述信息构建所述触觉媒体访问器，进而通过所述触觉媒体访问器访问所述目标触觉媒体文件的解码数据，获取所述第一触觉材质属性的触觉值，因此本申请实施例可以支持获取参考类型的触觉材质属性的触觉值，解决相关技术中无法获取参考类型的触觉材质属性的触觉值的问题。
作为上述场景描述文件的解析方法的扩展和细化，本申请提供了另一种场景描述文件的解析方法，参照图12所示，该场景描述文件的解析方法包括：
S2901、从场景描述文件的场景列表(scenes)中获取所述待渲染三维场景对应的目标场景描述模块。
在一些实施例中，可以从所述场景描述文件中获取场景声明(senc)及其声明的索引值，并根据场景声明及其声明的索引值从所述场景描述文件的场景列表中获取所述待渲染三维场景对应的目标场景描述模块。
例如：场景声明及其声明的索引值为："scene":1，则可以从根据场景声明及其声明的索引值从所述场景描述文件的场景列表获取第二个场景描述模块作为所述待渲染三维场景对应的目标场景描述模块。
S2902、获取所述目标场景描述模块的节点索引列表(nodes)声明的索引值。
示例性的，当所述目标场景描述模块的节点索引列表如下所示：。2902、获取所述目标场景描述模块的节点索引列表(nodes)声明的索引值。
示例性的，当所述目标场景描述模块的节点索引列表如下所示：
则，可以获取的所述目标场景描述模块的节点索引列表声明的索引值为0。
S2903、根据所述目标场景描述模块的节点索引列表声明的索引值，从所述场景描述文件的节点列表中获取所述待渲染三维场景中的各个节点对应的节点描述模块。
例如：当所述目标场景描述模块的节点索引列表声明的索引值为0，则从所述场景描述文件的节点列表中获取第一个节点描述模块，作为所述待渲染三维场景中的各个节点对应的节点描述模块。
再例如：当所述目标场景描述模块的节点索引列表声明的索引值为0和1，则从所述场景描述文件的节点列表中获取第一个节点描述模块和第二个节点描述模块，作为所述待渲染三维场景中的各个节点对应的节点描述模块。
S2904、获取所述待渲染三维场景中各个节点对应的节点描述模块的网格索引列表(mesh)。
示例性的，当某一节点对应的节点描述模块如下所示：
则，获取的该节点对应的节点描述模块的网格索引列表为"mesh":0，1。
S2905、根据所述待渲染三维场景中各个节点对应的节点描述模块的网格索引列表，从所述场景描述文件的网格列表(meshes)中获取所述待渲染三维场景中的各个三维网格对应的网格描述模块。
示例性的，当所述待渲染三维场景中包括节点1和节点2，节点1对应的节点描述模块的网格索引列表声明索引值包括：0和1，节点2对应的节点描述模块的网格索引列表声明索引值包括：1、2、3，则可以从从所述场景描述文件的网格列表中获取第一个网格描述模块、第二个网格描述模块、第三个网格描述模块以及第四个网格描述模块。
示例性的，所述待渲染三维场景中的网格描述模块可以如下所示：

示例性的，所述待渲染三维场景中的网格描述模块也可以如下所示：
S2906、获取各个三维网格对应的网格描述模块的MPEG触觉材质(MPEG_material_Haptic)中的数组索引语法元素(index)声明的索引值。
需要说明的是，在场景描述文件的网格列表中，一些网格描述模块中包括MPEG触觉材质，一些网格描述模块中不MPEG触觉材质，因此上步骤S2906仅针对包括MPEG触觉材质的网格描述模块执行即可。
S2907、根据各个三维网格对应的网格描述模块的MPEG触觉材质中的数组索引语法元素声明的索引值，从所述场景描述文件的触觉材质数组列表中获取各个三维网格的触觉材质属性数组。
示例性的，某一三维网格对应的网格描述模块的MPEG触觉材质中的数组索引语法元素声明的索引值为1，所述场景描述文件的触觉材质数组列表如下所示：
由于该三维网格对应的网格描述模块的MPEG触觉材质中的数组索引语法元素声明的索引值为1，因此可以根据三维网格对应的网格描述模块的MPEG触觉材质中的数组索引语法元素声明的索引值，从所述场景描述文件的触觉材质数组列表中获取第二个触觉材质数组，因此从所述场景描述文件的触觉材质数组列表中获取的该三维网格的触觉材质属性数组如下所示：
S2908、根据各个三维网格对应的触觉材质属性数组确定各个三维网格承载的各个触觉材质属性的类型。
在一些实施例中，上步骤S2908(根据各个三维网格对应的触觉材质属性数组确定各个三维网格承载的各个触觉材质属性的类型)的实现方式可以包括如下步骤29081～29083：
步骤29081、从各个三维网格对应的触觉材质属性数组中获取各个三维网格承载的各个触觉材质属性对应的触觉材质属性描述模块。
承上实施例所示，三维网格的触觉材质属性数组如下所示：
因此可以确定该三维网格仅承载有温度，且温度对应的触觉材质属性描述模块如下所示：
步骤29082、获取各个触觉材质属性对应的触觉材质属性描述模块中的属性类型语法元素(type)的值。
承上实施例所述，可以根据第n+18行的"type":Reference确定温度对应的触觉材质属性描述模块中的属性类型语法元素的值为Reference。
步骤29083、根据各个触觉材质属性对应的触觉材质属性描述模块中的属性类型语法元素 的值，确定各个三维网格承载的各个触觉材质属性的类型。
承上实施例所述，温度对应的触觉材质属性描述模块中的属性类型语法元素的值为Reference，因此可以根据各个触觉材质属性对应的触觉材质属性描述模块中的属性类型语法元素的值，确定该三维网格承载的温度的类型为参考。
在上步骤S2908中，若所述待渲染三维场景中的目标三维网格承载的第一触觉材质属性的类型为参考，则执行如下步骤：
S2909、将包括所述目标三维网格的节点对应的节点描述模块确定为所述目标节点描述模块。
即，判断各个节点描述模块的所述网格索引列表中声明的索引值是否包括所述目标三维网格对应的网格描述模块的索引值；若包括，则将该节点描述模块确定为所述目标节点描述模块。
S2910、获取所述第一触觉材质属性对应的触觉材质属性描述模块的纹理数组(texture)中的纹理索引语法元素(index)声明的索引值。
示例性的，当某一触觉材质属性对应的触觉材质属性描述模块如下所示：
则，可以确定该触觉材质属性对应的触觉材质属性描述模块的纹理数组(texture)中的纹理索引语法元素(index)声明的索引值索引值为2。
S2911、根据所述第一触觉材质属性对应的触觉材质属性描述模块的纹理数组中的纹理索引语法元素声明的索引值，从所述场景描述文件的纹理列表(textures)中获取所述第一触觉材质属性对应的纹理描述模块。
承上实施例所述，某一触觉材质属性描述模块的纹理数组(texture)中的纹理索引语法元素(index)声明的索引值索引值为2，则从场景描述文件的纹理列表(textures)中获取第三个纹理描述模块作为该触觉材质属性对应的纹理描述模块。
S2912、获取所述第一触觉材质属性对应的纹理描述模块中的纹理贴图语法元素(source)声明的索引值。
示例性的，当触觉材质属性对应的纹理描述模块如下所示：
则，可以获取该触觉材质属性对应的纹理描述模块中的纹理贴图语法元素(source)声明的索引值为1。
S2913、根据所述第一触觉材质属性对应的纹理描述模块中的纹理贴图语法元素声明的索 引值，从所述场景描述文件的纹理贴图列表(images)中获取所述第一触觉材质属性的纹理贴图的访问地址。
承上实施例所述，某一触觉材质属性对应的纹理描述模块中的纹理贴图语法元素(source)声明的索引值为1，则从所述场景描述文件的纹理贴图列表(images)中获取第二个纹理贴图的访问地址作为该触觉材质属性的纹理贴图的访问地址。
S2914、根据所述第一触觉材质属性的纹理贴图的访问地址获取所述第一触觉材质属性的纹理贴图。
在一些实施例中，根据所述第一触觉材质属性的纹理贴图的访问地址获取所述第一触觉材质属性的纹理贴图，包括：根据所述第一触觉材质属性的纹理贴图的访问地址，使用网络传输服务从服务器中下载触觉媒体文件。
在一些实施例中，根据所述第一触觉材质属性的纹理贴图的访问地址获取所述第一触觉材质属性的纹理贴图，包括：据所述第一触觉材质属性的纹理贴图的访问地址，从本地存储空间中读取触觉媒体文件。
S2915、根据第一触觉材质属性的纹理贴图获取所述目标触觉媒体文件对应的目标媒体描述模块的索引值。
由于所述第一触觉材质属性的类型为参考，因此所述第一触觉材质属性的纹理贴图中存储的不是触觉值，而是保存有所述第一触觉材质属性的触觉值的触觉媒体文件对应的媒体描述模块的索引值，因此本申请实施例可以根据所述第一触觉材质属性的纹理贴图获取所述目标触觉媒体文件对应的目标媒体描述模块的索引值。
S2916、将所述目标节点描述模块中所述触觉媒体标识语法元素的值为所述目标媒体描述模块的索引值的媒体参考数组确定为所述目标媒体参考数组。
示例性的，当所述目标媒体描述模块的索引值为0，且所述目标节点描述模块的MPEG触觉中的媒体参考列表(Media_reference)如下所示：
由于媒体参考列表(Media_reference)中的第一个媒体参考数组中的触觉媒体标识语法元素(haptic_id)的值与目标媒体描述模块的索引值相同，均为0，因此将媒体参考列表(Media_reference)中的第一个媒体参考数组确定为所述目标媒体参考数组。
S2917、获取目标节点描述模块的MPEG触觉的媒体参考列表的目标媒体参考数组中的访问器索引语法元素声明的索引值。
示例性的，目标节点描述模块的MPEG触觉的媒体参考列表的目标媒体参考数组中的访问器索引语法元素及其值为"accessor":1，则可以确定目标节点描述模块的MPEG触觉的媒体参考列表的目标媒体参考数组中的访问器索引语法元素声明的索引值为1。
S2918、根据所述访问器索引语法元素声明的索引值从所述场景描述文件的访问器列表(accessors)中获取所述触觉媒体访问器对应的访问器描述模块。
示例性的，所述访问器索引语法元素声明的索引值为0，则从所述场景描述文件的访问器列表中第一个访问器描述模块作为所述触觉媒体访问器对应的访问器描述模块。
示例性的，所述访问器索引语法元素声明的索引值为2，则从所述场景描述文件的访问器列表中第三个访问器描述模块作为所述触觉媒体访问器对应的访问器描述模块。
S2919、解析所述触觉媒体访问器对应的访问器描述模块获取所述触觉媒体访问器的描述信息。
示例性的，所述触觉媒体访问器对应的访问器描述模块如下所示：
则，解析所述触觉媒体访问器对应的访问器描述模块可以获取的信息包括：数据类型为5123；访问器类型为标量；数据数量为1000；访问器基于MPEG扩展改造的时变访问器；访问器访问对象为缓存切片列表中的第二个缓存切片描述模块对应的缓存切片；访问器的参数会随时间变化。
至此，通过解析场景描述文件可以获取能够访问第一触觉材质属性的触觉值的触觉媒体访问器的描述信息，进而通过所述触觉媒体访问器的描述信息构建所述触觉媒体访问器，以读取所述第一触觉材质属性的触觉值。
在一些实施例中，本申请实施例提供的场景描述文件解析方法还包括：
根据所述第一触觉材质属性对应的纹理描述模块中的采样器语法元素(sampler)声明的索引值，从所述场景描述文件的采样器列表(samplers)中获取所述第一触觉材质属性的采样器对应的采样器描述模块，以及根据所述第一触觉材质属性的采样器对应的采样器描述模块，获取所述第一触觉材质属性的采样器的描述信息。
例如：第一触觉材质属性对应的纹理描述模块中的采样器语法元素(sampler)声明的索引值为0，则从场景描述文件的采样器列表获取第一个采样器描述模块，并根据第一个采样器描述模块获取所述第一触觉材质属性的采样器的描述信息。
在一些实施例中，本申请实施例提供的场景描述文件解析方法还包括如下步骤Ⅰ～步骤Ⅲ：
步骤Ⅰ、根据所述第一触觉材质属性对应的触觉材质属性描述模块的纹理数组(texture)中的纹理坐标索引语法元素(texCoord)声明的索引值，从所述目标三维网格对应的网格描述模块的基元(primitives)的属性(attributes)中获取所述第一触觉材质属性对应的纹理坐标语法元素(TEXCOORD_n)。
例如：所述第一触觉材质属性对应的触觉材质属性描述模块的纹理数组(texture)中的纹理坐标索引语法元素(texCoord)及其值为："texCoord":0，则从所述目标三维网格对应的网格描述模块的基元(primitives)的属性(attributes)中获取第一个纹理坐标语法元素 (TEXCOORD_0)作为所述第一触觉材质属性对应的纹理坐标语法元素。
步骤Ⅱ、根据所述第一触觉材质属性对应的纹理坐标语法元素声明的索引值，获取所述第一触觉材质属性的纹理坐标访问器对应的访问器描述模块。
例如：所述第一触觉材质属性对应的纹理坐标语法元素及其值为："TEXCOORD_0":0，则从所述场景描述文件的访问器列表(accessors)中获取第一访问器描述模块作为所述第一触觉材质属性的纹理坐标访问器对应的访问器描述模块。
步骤Ⅲ、根据所述第一触觉材质属性的纹理坐标访问器对应的访问器描述模块，获取所述第一触觉材质属性的纹理坐标访问器的描述信息。
示例性的，所述第一触觉材质属性的纹理坐标访问器对应的访问器描述模块如下所示：
则，解析所述第一触觉材质属性的纹理坐标访问器对应的访问器描述模块可以获取的信息包括：访问器访问对象为缓存切片列表中的第一个缓存切片描述模块对应的缓存切片，数据类型为5121；访问器类型为二维向量；数据数量为500。
至此，通过解析场景描述文件可以获取能够访问第一触觉材质属性对应的纹理坐标的纹理坐标访问器的描述信息，进而通过所述纹理坐标访问器的描述信息构建所述纹理坐标访问器，以读取所述第一触觉材质属性对应的纹理坐标。
在一些实施例中，所述场景描述文件解析方法还包括：
当第二触觉材质属性的类型为高分辨率或低分辨率时，获取所述第二触觉材质属性对应的触觉材质属性描述模块的纹理数组(texture)中的纹理索引语法元素(index)声明的索引值；根据所述第二触觉材质属性对应的触觉材质属性描述模块的纹理数组中的纹理索引语法元素声明的索引值，从所述场景描述文件的纹理列表(textures)中获取所述第二触觉材质属性对应的纹理描述模块；获取所述第二触觉材质属性对应的纹理描述模块中的纹理贴图语法元素(source)声明的索引值和采样器语法元素(sampler)声明的索引值；根据所述第二触觉材质属性对应的纹理描述模块中的纹理贴图语法元素(source)声明的索引值，从所述场景描述文件的纹理贴图列表(images)中获取所述第二触觉材质属性的纹理贴图的访问地址，以及根据所述第二触觉材质属性对应的纹理描述模块中的采样器语法元素(sampler)声明的索引值，从所述场景描述文件的采样器列表(samplers)中获取第二触觉触觉材质属性的采样器的描述信息。
由于当触觉材质属性的类型为高分辨率或低分辨率时，触觉材质属性的纹理贴图中存储的数据为触觉材质属性的触觉值，因此当触觉材质属性的类型为高分辨率或低分辨率时，可以通过上述实施例直接从触觉材质属性的纹理贴图中获取触觉材质属性的纹理贴图的访问地址，进而获取触觉材质属性的纹理贴图，获取触觉材质属性的触觉值。
在一些实施例中，所述场景描述文件解析方法还包括：
从所述场景描述文件的缓存器列表中获取所述待渲染三维场景的各个缓存器对应的缓存 器描述模块；根据各个缓存器对应的缓存器描述模块获取所述待渲染三维场景的各个缓存器的描述信息。
其中，缓存器的描述信息包括以下至少一项：
缓存器的容量、缓存器所缓存的数据的访问地址、是否为MPEG的环形缓存器、环形缓存器的存储环节数量、环形缓存器所缓存的媒体文件对应的媒体描述模块的索引值、环形缓存器所缓存的媒体文件的数据的轨道索引值。
示例性的，某一缓存器对应的缓存器描述模块如下所示：
则，根据该缓存器对应的缓存器描述模块可以获取该缓存器的描述信息包括：该缓存器的容量为2000字节；该缓存器为基于MPEG扩展改造的环形缓存器，环形缓存器的存储环节数量为3，环形缓存器存储的媒体文件为MPEG媒体中声明的第一个媒体文件，环形缓存器所缓存的媒体文件的数据的轨道索引值为1。
示例性的，某一缓存器对应的缓存器描述模块如下所示：
则，根据该缓存器对应的缓存器描述模块可以获取该缓存器的描述信息包括：该缓存器的容量为1000字节；该缓存器所缓存的数据的访问地址为AnimatedBody.bin。
在一些实施例中，所述场景描述文件解析方法还包括：
从所述场景描述文件的缓存切片列表中获取所述待渲染三维场景的各个缓存器缓存器的各个缓存切片对应的缓存切片描述模块；根据各个缓存切片对应的缓存切片描述模块获取各个缓存切片的描述信息；
其中，缓存切片的描述信息包括以下至少一项：
缓存切片所属的缓存器、缓存切片的容量、缓存切片的偏移量。
例如：某一缓存切片对应的缓存切片描述模块如下所示：
则，根据该缓存切片对应的缓存切片描述模块可以获取该缓存切片的描述信息包括：该缓存切片为缓存器列表中的第三个缓存器描述模块对应的缓存器的缓存切片，该缓存切片的容量为4000字节，该缓存切片的偏移量为0。
在一些实施例中，所述场景描述文件解析方法还包括：从所述场景描述文件的访问器列表中获取所述待渲染三维场景的各个访问器对应的访问器描述模块；根据各个访问器对应的访问器描述模块获取各个访问器的描述信息；
其中，访问器的描述信息包括以下至少一项：
访问器所访问的缓存切片、访问器所访问的数据的数据类型、访问器的类型、访问器的数量、是否为MPEG时变访问器、时变访问器所访问的缓存切片、访问器参数是否随时间变化。
例如：某一缓存切片对应的缓存切片描述模块如下所示：
则，解析该访问器对应的访问器描述模块获取的访问器的描述信息包括：该访问器所访问的数据的数据类型为5123，该访问器的类型为标量；该访问器的数量为1000，该访问器为基于MPEG扩展改造的时变访问器；该访问器所访问的缓存切片为索引值为1的缓存切片描述模块对应的缓存切片；该访问器的参数会随时间变化。
再例如：某一缓存切片对应的缓存切片描述模块如下所示：
则，解析该访问器对应的访问器描述模块获取的访问器的描述信息包括：该访问器所访问的缓存切片为索引值为0的缓存切片描述模块对应的缓存切片；该访问器所访问的数据的数据类型为5121；该访问器的类型为二维向量；该访问器的数量为500。
在一些实施例中，所述场景描述文件解析方法还包括：从所述场景描述文件的MPEG媒体扩展的媒体列表中获取所述待渲染三维场景中的各个媒体文件对应的媒体描述模块；根据各个媒体文件对应的媒体描述模块获取各个媒体文件的描述信息。
其中，媒体文件的描述信息包括以下至少一项：
媒体文件的名称、媒体文件是否自动播放、媒体文件是否循环播放、媒体文件的类型、媒体文件的访问地址、媒体文件的封装文件的轨道信息、媒体文件的编解码参数。
示例性的，某一媒体文件对应的媒体描述模块入下所示：

则，根据该个媒体文件对应的媒体描述模块获取的媒体文件的描述信息包括：该媒体文件的名称为Hapticexample；该媒体文件自动播放；该媒体文件循环播放、该媒体文件的类型为仅包括触觉数据的触觉媒体文件；该媒体文件的访问地址为http://www.example.com/Hapticexample1.mp4、该媒体文件的封装文件的码流轨道的索引值为1、媒体文件的编解码参数为mih1.oo。
在一些实施例中，所述场景描述文件解析方法还包括：从所述场景描述文件获取场景声明(scene)，根据所述场景声明所声明的索引值从所述场景描述文件的场景列表(scenes)中获取所述待渲染三维场景对应的目标场景描述模块。
例如：场景声明及其声明的索引值为"scene":0，则从所述场景描述文件的场景列表(scenes)中获取第一个场景描述模块作为所述待渲染三维场景对应的目标场景描述模块。
在一些实施例中，所述场景描述文件解析方法还包括：获取所述场景描述文件中的使用扩展描述模块(extensionsUsed)，根据所述使用扩展描述模块获取所述场景描述文件中使用的MPEG扩展。
例如：使用扩展描述模块如下所示：
则，根据所述使用扩展描述模块可以获取场景描述文件中使用的MPEG扩展包括：MPGE媒体(MPEG_media)、MPEG环形缓存器(MPEG_buffer_circular)、MPEG时变访问器(MPEG_accessor_timed)、MPEG触觉(MPEG_Haptic)以及MPEG触觉材质(MPEG_material_Haptic)；共5个MPEG扩展。
在一些实施例中，所述场景描述文件解析方法还包括：获取所述场景描述文件中的数字资产描述模块(version)，并根据所述数字资产描述模块确定所述场景描述文件的版本信息。
通过数字资产描述模块确定所述场景描述文件的版本信息后，基于场景描述文件的版本信息选用相应的场景描述文件解析器进行场景描述文件的解析，进而保证正确对场景描述文件进行解析。
本申请一些实施例还提供一种三维场景的渲染方法，该三维场景的渲染方法的执行主体 为沉浸式媒体描述框架中的显示引擎，参照图13所示，该三维场景的渲染方法包括如下步骤：
S3001、根据场景描述文件确定待渲染三维场景中的各个三维网格承载的各个触觉材质属性的类型。
本申请实施例中的触觉材质属性可以包括：刚度(stiffness)、摩擦力(Friction)、触觉纹理(Vibrotactile texture)、温度(Temperature)、振动(Vibration)或自定义中的一种或多种。触觉材质属性的类型包括：高分辨率(High_resolution)、低分辨率(Low_resolution)以及参考(Reference)。
需要说明的是，一个三维网格可以承载多个不同类型的触觉材质属性，因此根据场景描述文件确定待渲染三维场景中的各个三维网格承载的各个触觉材质属性的类型时，需要逐一确定每一个触觉材质属性的类型。
在上步骤S3001(根据场景描述文件确定待渲染三维场景中的各个三维网格承载的各个触觉材质属)中，若确定所述待渲染三维场景中的目标三维网格承载的第一触觉材质属性的类型为参考(Reference)，则执行如下步骤S3002～S3004：
S3002、根据目标节点描述模块的MPEG触觉的媒体参考列表(Media_reference)的目标媒体参考数组中的访问器索引语法元素(accessor)声明的索引值，确定触觉媒体访问器。
其中，所述目标节点描述模块为包括所述目标三维网格的节点对应的节点描述模块，所述目标媒体参考数组为所述第一触觉材质属性所属的触觉媒体文件对应的媒体参考数组。
上述步骤S3002中获取显示引擎确定目标节点描述模块的MPEG触觉的媒体参考列表(Media_reference)的目标媒体参考数组中的访问器索引语法元素(accessor)声明的索引值的实现方式可以参照图12所示实施例，在此不再赘述。
S3003、通过所述触觉媒体访问器访问触觉媒体缓存器的缓存切片中的数据，以获取所述第一触觉材质属性的触觉值。
其中，所述触觉媒体缓存器为用于缓存所述目标触觉媒体文件的解码数据的缓存器。
由于所述触觉媒体缓存器中保存的数据为对媒体接入函数对所述目标触觉媒体文件进行处理得到的解码数据，因此通过通过所述触觉媒体访问器访问触觉媒体缓存器的缓存切片中的数据，可以直接获取解码后的可用于渲染的所述第一触觉材质属性的触觉值。
S3004、根据所述第一触觉材质属性的触觉值对所述待渲染三维场景进行渲染。
本申请实施例提供的三维场景的渲染方法在获取场景描述文件后，首先根据场景描述文件确定待渲染三维场景中的各个三维网格承载的各个触觉材质属性的类型，并在待渲染三维场景中的目标三维网格承载的第一触觉材质属性的类型为参考时，获取包括所述目标三维网格的节点对应的节点描述模块，并获取目标节点描述模块的MPEG触觉中第一触觉材质属性所属的目标触觉媒体文件对应的目标媒体参考数组中的访问器索引语法元素声明的索引值，以及根据所述访问器索引语法元素声明的索引值确定触觉媒体访问器，再通过所述触觉媒体访问器访问用于缓存所述目标触觉媒体文件的触觉媒体缓存器的缓存切片中的数据，以获取所述第一触觉材质属性的触觉值，最后根据所述第一触觉材质属性的触觉值对所述待渲染三维场景进行渲染。由于本申请实施例可以在目标三维网格承载的第一触觉材质属性的类型为参考时，获取目标节点描述模块的MPEG触觉的媒体参考列表的目标媒体参考数组中的访问器索引语法元素声明的索引值，以及根据所述访问器索引语法元素声明的索引值确定触觉媒 体访问器，通过触觉媒体访问器获取第一触觉材质属性的触觉值，并根据所述第一触觉材质属性的触觉值对所述待渲染三维场景进行渲染，本申请实施例可以支持获取参考类型的触觉材质属性的触觉值，解决相关技术中无法获取参考类型的触觉材质属性的触觉值的问题。
本申请一些实施例还提供一种触觉媒体文件的处理方法，该触觉媒体文件的处理方法的执行主体为沉浸式媒体描述框架中的媒体接入函数，参照图14所示，该触觉媒体文件的处理方法包括如下步骤：
S3101、接收显示引擎发送的触觉媒体文件的描述信息、触觉媒体缓存器的描述信息以及所述触觉媒体缓存器的缓存切片的描述信息。
其中，所述触觉媒体文件中包括类型为参考的第一触觉材质属性的触觉值。
在一些实施例中，触觉媒体文件的描述信息可以包括以下至少一项：
媒体文件的名称、媒体文件是否自动播放、媒体文件是否循环播放、媒体文件的类型、媒体文件的访问地址、媒体文件的封装文件的轨道信息、媒体文件的编解码参数。
在一些实施例中，所述触觉媒体缓存器的描述信息可以包括以下至少一项：
缓存器的容量、是否为MPEG的环形缓存器、环形缓存器的存储环节数量、环形缓存器所缓存的媒体文件对应的媒体描述模块的索引值、环形缓存器所缓存的媒体文件的数据的轨道索引值。
在一些实施例中，所述触觉媒体缓存器的缓存切片的描述信息可以包括以下至少一项：
缓存切片所属的缓存器、缓存切片的容量、缓存切片的偏移量。
S3102、根据所述触觉媒体文件的描述信息获取所述触觉媒体文件对应的解码数据。
在一些实施例中，媒体接入函数根据所述触觉媒体文件的描述信息获取所述触觉媒体文件对应的解码数据，包括：
根据所述触觉媒体文件的描述信息创建用于处理所述触觉媒体文件的触觉媒体管线，通过所述触觉媒体管线获取所述触觉媒体文件，并对所述触觉媒体文件进行解封装和解码，以获取所述触觉媒体文件对应的解码数据。
S3103、根据所述触觉媒体缓存器的描述信息和所述触觉媒体缓存器的缓存切片的描述信息，将所述解码数据写入所述触觉媒体缓存器的缓存切片中。
将所述解码数据写入所述触觉媒体缓存器的缓存切片中后，显示引擎可以根据目标节点描述模块的MPEG触觉的媒体参考列表的目标媒体参考数组中的访问器索引语法元素声明的索引值确定触觉媒体访问器，及通过所述触觉媒体访问器访问所述触觉媒体缓存器的缓存切片中的数据获取所述第一触觉材质属性的触觉值。其中，所述目标节点描述模块为包括承载所述第一触觉材质属性的目标三维网格的节点对应的节点描述模块，所述目标媒体参考数组为所述触觉媒体文件对应的媒体参考数组。
本申请实施例提供的触觉媒体文件的处理方法在接收显示引擎发送的触觉媒体文件的描述信息、触觉媒体缓存器的描述信息以及所述触觉媒体缓存器的缓存切片的描述信息后，根据所述触觉媒体文件的描述信息获取所述触觉媒体文件对应的解码数据，并将根据所述触觉媒体缓存器的描述信息和所述触觉媒体缓存器的缓存切片的描述信息，将所述解码数据写入所述触觉媒体缓存器的缓存切片中，因此显示引擎可以根据目标节点描述模块的MPEG触觉的媒体参考列表的目标媒体参考数组中的访问器索引语法元素声明的索引值确定触觉媒体访 问器，以及通过所述触觉媒体访问器访问所述触觉媒体缓存器的缓存切片中的数据获取所述第一触觉材质属性的触觉值，因此本申请实施例可以支持获取参考类型的触觉材质属性的触觉值，解决相关技术中无法获取参考类型的触觉材质属性的触觉值的问题。
本申请一些实施例还提供一种缓存管理方法，该缓存管理方法的执行主体为沉浸式媒体描述框架中的缓存管理模块，参照图15所示，该缓存管理方法包括如下步骤：
S3201、接收触觉媒体缓存器的描述信息和所述触觉媒体缓存器的缓存切片的描述信息。
其中，所述触觉媒体缓存器为用于缓存目标触觉媒体文件的缓存器，所述目标触觉媒体文件包括类型为参考的第一触觉材质属性。
在一些实施例中，所述触觉媒体缓存器的描述信息可以包括以下至少一项：
缓存器的容量、是否为MPEG的环形缓存器、环形缓存器的存储环节数量、环形缓存器所缓存的媒体文件对应的媒体描述模块的索引值、环形缓存器所缓存的媒体文件的数据的轨道索引值。
在一些实施例中，所述触觉媒体缓存器的缓存切片的描述信息可以包括以下至少一项：
缓存切片所属的缓存器、缓存切片的容量、缓存切片的偏移量。
S3202、根据所述触觉媒体缓存器的描述信息创建所述触觉媒体缓存器。
例如：触觉媒体缓存器的描述信息包括：触觉媒体缓存器的容量为2000字节；触觉媒体缓存器为基于MPEG扩展改造的环形缓存器，环形缓存器的存储环节数量为3，环形缓存器存储的媒体文件为MPEG媒体中声明的第一个媒体文件，环形缓存器所缓存的媒体文件的数据的轨道索引值为1，则所述缓存管理模块创建一个容量为2000字节、包含3个存储环节的环形缓存器作为所述触觉媒体缓存器。
S3203、根据所述触觉媒体缓存器的缓存切片的描述信息对所述触觉媒体缓存器进行缓存切片的划分。
承上实施例所述，环形缓存器所缓存的媒体文件的数据的轨道索引值为1，则所述缓存管理模块创建一个容量为2000字节、包含3个存储环节的环形缓存器作为所述触觉媒体缓存器后，若所述触觉媒体缓存器的缓存切片的描述信息包括：所述容量为2000字节，偏移量为0，则将所述触觉媒体缓存器划分为1个缓存器切片。
在缓存管理模块根据所述触觉媒体缓存器的缓存切片的描述信息对所述触觉媒体缓存器进行缓存切片的划分后，媒体接入函数可以将所述目标触觉媒体文件的解码数据写入所述触觉媒体缓存器的缓存切片中，显示引擎可以根据目标节点描述模块的MPEG触觉的媒体参考列表的目标媒体参考数组中的访问器索引语法元素声明的索引值确定触觉媒体访问器，以及通过所述触觉媒体访问器访问所述触觉媒体缓存器的缓存切片中的数据获取所述第一触觉材质属性的触觉值。其中，所述目标节点描述模块为包括承载所述第一触觉材质属性的目标三维网格的节点对应的节点描述模块，所述目标媒体参考数组为所述目标触觉媒体文件对应的媒体参考数组。
本申请实施例提供的缓存管理方法在接收到触觉媒体缓存器的描述信息和所述触觉媒体缓存器的缓存切片的描述信息后，可以根据所述触觉媒体缓存器的描述信息创建所述触觉媒体缓存器，并根据所述触觉媒体缓存器的缓存切片的描述信息对所述触觉媒体缓存器进行缓存切片的划分，因此媒体接入函数可以将包括类型为参考的第一触觉材质属性的触觉值的触 觉媒体文件的解码数据写入所述触觉媒体缓存器的缓存切片中，显示引擎可以根据目标节点描述模块的MPEG触觉的媒体参考列表的目标媒体参考数组中的访问器索引语法元素声明的索引值确定触觉媒体访问器，及通过所述触觉媒体访问器访问所述触觉媒体缓存器的缓存切片中的数据获取所述第一触觉材质属性的触觉值。因此本申请实施例可以支持获取参考类型的触觉材质属性的触觉值，解决相关技术中无法获取参考类型的触觉材质属性的触觉值的问题。
本申请一些实施例还提供一种三维场景的渲染方法，该三维场景渲染方法包括：显示引擎所执行的场景描述文件解析方法和三维场景的渲染方法、媒体接入函数所执行的触觉媒体文件的处理方法以及缓存管理模块所执行的缓存管理方法。以下以所述待渲染场景中包括目标触觉媒体文件，所述目标触觉媒体文件的第一触觉材质属性的类型为参考为例，且将用于缓存所述目标触觉媒体文件的缓存器为触觉媒体缓存器，用于访问所述目标触觉媒体文件的缓存器为触觉媒体访问器，承载所述第一触觉材质属性的网格为目标三维网格，所述用于缓存所述目标三维网格的纹理坐标的缓存器为纹理坐标缓存器，用于访问所述目标三维网格的纹理坐标的访问器为纹理坐标访问器对本申请实施例提供的方案进行说明。参照图16所示，该方法包括如下步骤：
为例对上述本申请实施例提供的方案进行说明。参照图16所示，该方法包括如下步骤：
S3301、显示引擎获取场景描述文件。
在一些实施例中，显示引擎获取待渲染场景的场景描述文件，包括：显示引擎使用网络传输服务从服务器中下载所述场景描述文件。
在一些实施例中，显示引擎获取待渲染场景的场景描述文件，包括：从本地存储空间中读取所述场景描述文件。
S3302、显示引擎从所述场景描述文件的MPEG媒体的媒体列表(media)中获取各个媒体文件对应的媒体描述模块(包括：从所述场景描述文件的MPEG媒体的媒体列表中获取所述目标触觉媒体文件对应的媒体描述模块)。
S3303、显示引擎根据各个媒体文件对应的媒体描述模块获取各个媒体文件的描述信息(包括：根据所述目标触觉媒体文件对应的媒体描述模块获取所述目标触觉媒体文件的描述信息)。
在一些实施例中，媒体文件的描述信息包括以下至少一项：
媒体文件的名称、媒体文件是否自动播放、媒体文件是否循环播放、体文件的类型、媒体文件的访问地址、媒体文件的封装文件的轨道信息、媒体文件的编解码参数。
S3304、显示引擎向媒体接入函数发送各个媒体文件的描述信息(向媒体接入函数发送所述目标触觉媒体文件的描述信息)。
相应的，媒体接入函数接收所述显示引擎发送的各个媒体文件的描述信息。
在一些实施例中，显示引擎向媒体接入函数发送各个媒体文件的描述信息，包括：显示引擎通过媒体接入函数API向媒体接入函数发送各个媒体文件的描述信息。
在一些实施例中，媒体接入函数接收所述显示引擎发送的各个媒体文件的描述信息，包括：媒体接入函数通过媒体接入函数API接收所述显示引擎发送各个媒体文件的描述信息。
S3305、媒体接入函数根据各个媒体文件的描述信息创建用于处理各个媒体文件对应的管 线(包括根据所述目标触觉媒体文件描述信息创建用于处理所述目标触觉媒体文件的触觉媒体管线)。
在一些实施例中，所述触觉媒体管线，包括：输入模块、解封装模块、解码模块以及后处理模块；所述输入模块用于获取所述目标触觉媒体文件(封装文件)，所述解封装模块用于对所述目标触觉媒体文件进行解封装，所述解码模块用于对解封装得到的码流进行解码，所述后处理模块用于对解码得到的数据进行格式转换等后处理，以获取所述目标触觉媒体文件的解码数据。
S3306、媒体接入函数通过各个媒体文件对应的管线处理获取各个媒体文件，并对各个媒体文件进行解封装和解码，以获取各个媒体文件对应的解码数据。(包括通过所述触觉媒体管线获取所述目标触觉媒体文件，并对所述目标触觉媒体文件进行解封装和解码，以获取所述目标触觉媒体文件对应的解码数据)。
在一些实施例中，所述目标触觉媒体文件的描述信息包括所述目标触觉媒体文件的访问地址，上述步骤S3306(媒体接入函数通过各个媒体文件对应的管线处理获取各个媒体文件，并对各个媒体文件进行解封装和解码，以获取媒体文件对应的解码数据)包括：
媒体接入函数根据所述目标触觉媒体文件的访问地址获取所述目标触觉媒体文件。
在一些实施例中，所述媒体接入函数根据所述目标触觉媒体文件的访问地址获取所述目标触觉媒体文件，包括：媒体接入函数根据所述目标触觉媒体文件的访问地址向媒体资源服务器发送媒体资源请求，并接收所述媒体服务器发送的携带有所述目标触觉媒体文件的媒体资源响应。
在一些实施例中，所述媒体接入函数根据所述目标触觉媒体文件的访问地址获取所述目标触觉媒体文件，包括：媒体接入函数根据所述目标触觉媒体文件的访问地址从预设存储空间中读取所述目标触觉媒体文件。
在一些实施例中，所述目标触觉媒体文件的描述信息还包括所述目标触觉媒体文件的各个码流轨道的索引值，上述步骤S3306(媒体接入函数通过各个媒体文件对应的管线处理获取各个媒体文件，并对各个媒体文件进行解封装和解码，以获取媒体文件对应的解码数据)包括：
媒体接入函数根据所述目标触觉媒体文件的各个码流轨道的索引值对所述目标触觉媒体文件进行解封装，以获取所述目标触觉媒体文件的各个码流轨道的码流。
在一些实施例中，所述目标触觉媒体文件的描述信息还包括所述目标触觉媒体文件的各个码流轨道的编解码参数；上述步骤S3306(媒体接入函数通过各个媒体文件对应的管线处理获取各个媒体文件，并对各个媒体文件进行解封装和解码，以获取媒体文件对应的解码数据)包括：
媒体接入函数根据所述目标触觉媒体文件的各个码流轨道的编解码参数对所述目标触觉媒体文件的各个码流轨道的码流进行解码，获取所述目标触觉媒体文件的各个码流轨道的对应的解码数据，以获取目标触觉媒体文件对应的解码数据。
S3307、显示引擎从所述场景描述文件的缓存器列表(buffers)中获取各个缓存器对应的缓存器描述模块(包括：从所述场景描述文件的缓存器列表中获取用于缓存所述目标触觉媒体文件的触觉媒体缓存器对应的缓存器描述模块，以及用于缓存承载所述第一触觉材质属性 的三维网格的纹理坐标的纹理坐标缓存器对应的缓存器描述模块)。
S3308、显示引擎根据各个缓存器对应的缓存器描述模块获取各个缓存器的描述信息(包括：根据所述触觉媒体缓存器对应的缓存器描述模块获取所述触觉媒体缓存器的描述信息，以及根据所述纹理坐标缓存器对应的缓存器描述模块获取所述纹理坐标缓存器的描述信息)。
在一些实施例中，缓存器的描述信息可以包括以下至少一项：
缓存器的容量、缓存器所缓存的数据的访问地址、是否为MPEG环形缓存器、环形缓存器的存储环节数量、环形缓存器所缓存的媒体文件对应的媒体描述模块的索引值、环形缓存器所缓存的媒体文件的数据的轨道索引值。
S3309、显示引擎从所述场景描述文件的缓存切片列表(bufferViews)中获取各个缓存器的缓存切片对应的缓存切片描述模块(包括：从所述场景描述文件的缓存器列表中获取所述触觉媒体缓存器的缓存切片对应的缓存切片描述模块，以及所述纹理坐标缓存器的缓存切片对应的缓存切片描述模块)。
S3310、显示引擎根据各个缓存器的缓存切片对应的缓存切片描述模块获取各个缓存器的缓存切片的描述信息(包括：根据所述触觉媒体缓存器的缓存切片对应的缓存切片描述模块获取所述触觉媒体缓存器的缓存切片的描述信息，以及根据所述纹理坐标缓存器的缓存切片对应的缓存切片描述模块获取所述纹理坐标缓存器的缓存切片的描述信息)。
在一些实施例中，缓存器的描述信息可以包括以下至少一项：
缓存切片所属的缓存器、缓存切片的容量、缓存切片的偏移量。
在上步骤S3308和S3310之后，本申请实施例可以选择执行如下方案一或方案二，将各个缓存器的描述信息以及各个缓存器的缓存切片的描述信息发送至媒体接入函数和缓存管理模块。
方案一包括如下步骤S3311和S3312：
S3311、显示引擎向媒体接入函数发送各个缓存器的描述信息和各个缓存器的缓存切片的描述信息(包括：显示引擎向媒体接入函数发送所述触觉媒体缓存器的描述信息和所述触觉媒体缓存器的缓存切片的描述信息，以及所述纹理坐标缓存器的描述信息和所述纹理坐标缓存器的缓存切片的描述信息)。
相应的，媒体接入函数接收显示引擎发送的各个缓存器的描述信息和各个缓存器的缓存切片的描述信息(包括：媒体接入函数接收显示引擎发送的所述触觉媒体缓存器的描述信息和所述触觉媒体缓存器的缓存切片的描述信息，以及所述纹理坐标缓存器的描述信息和所述纹理坐标缓存器的缓存切片的描述信息)。
在一些实施例中，上步骤S3311(显示引擎向媒体接入函数发送各个缓存器的描述信息和各个缓存器的缓存切片的描述信息)的实现方式可以为：显示引擎通过媒体接入函数API向媒体接入函数发送各个缓存器的描述信息和各个缓存器的缓存切片的描述信息。相应的，媒体接入函数接收显示引擎发送的各个缓存器的描述信息的实现方式可以为：媒体接入函数通过媒体接入函数API接收显示引擎发送的各个缓存器的描述信息和各个缓存器的缓存切片的描述信息。
S3312、媒体接入函数向缓存管理模块发送各个缓存器的描述信息和各个缓存器的缓存切片的描述信息(包括：媒体接入函数向缓存管理模块发送所述触觉媒体缓存器的描述信息、 所述触觉媒体缓存器的缓存切片的描述信息、所述纹理坐标缓存器的描述信息、所述纹理坐标缓存器的缓存切片的描述信息)。
相应的，缓存管理模块接收媒体接入函数发送的各个缓存器的描述信息(包括：媒体接入函数接收显示引擎发送的所述触觉媒体缓存器的描述信息、所述触觉媒体缓存器的缓存切片的描述信息、所述纹理坐标缓存器的描述信息以及所述纹理坐标缓存器的缓存切片的描述信息)。
在一些实施例中，上步骤S3311(媒体接入函数向缓存管理模块发送各个缓存器的描述信息和各个缓存器的缓存切片的描述信息)的实现方式可以为：媒体接入函数通过缓存API向缓存管理模块发送各个缓存器的描述信息和各个缓存器的缓存切片的描述信息。相应的，缓存管理模块接收媒体接入函数发送的各个缓存器的描述信息和各个缓存器的缓存切片的描述信息的实现方式可以为：缓存管理模块通过缓存API接收媒体接入函数发送的各个缓存器的描述信息和各个缓存器的缓存切片的描述信息。
方案二包括如下步骤S3313和S3314：
S3313、显示引擎向媒体接入函数发送各个缓存器的描述信息和各个缓存器的缓存切片的描述信息。
相应的，媒体接入函数接收显示引擎发送的各个缓存器的描述信息和各个缓存器的缓存切片的描述信息。
S3314、显示引擎向缓存管理模块发送各个缓存器的描述信息和各个缓存器的缓存切片的描述信息。
在一些实施例中，上步骤S3314(显示引擎向缓存管理模块发送各个缓存器的描述信息和各个缓存器的缓存切片的描述信息和所述触觉媒体缓存器的缓存切片的描述信息)的实现方式可以为：显示引擎通过缓存API向缓存管理模块发送各个缓存器的描述信息和各个缓存器的缓存切片的描述信息。相应的，缓存管理模块接收显示引擎发送的各个缓存器的描述信息的实现方式可以为：缓存管理模块通过缓存API接收显示引擎发送的各个缓存器的描述信息和各个缓存器的缓存切片的描述信息。
在执行上述方案一或方案二后，继续执行如下步骤：
S3315、缓存管理模块根据各个缓存器的描述信息构建各个缓存器(包括根据所述触觉媒体缓存器的描述信息构建所述触觉媒体缓存器，以及根据所述纹理坐标缓存器的描述信息构建所述纹理坐标缓存器)。
S3316、缓存管理模块根据各个缓存器的缓存切片的描述信息对各个缓存器进行缓存切片的划分(包括根据所述触觉媒体缓存器的缓存切片的描述信息，对所述触觉媒体缓存器进行缓存切片的划分，以及根据所述纹理坐标缓存器的缓存切片的描述信息，对所述纹理坐标缓存器进行缓存切片的划分)。
S3317、媒体接入函数根据各个缓存器的描述信息和各个缓存器的缓存切片的描述信息，将各个媒体文件对应的解码数据写入各个媒体文件对应的缓存器的缓存切片中(包括：将所述目标触觉媒体文件的解码数据写入所述触觉媒体缓存器的缓存切片中)。
S3318、媒体接入函数根据所述目标三维网格的纹理坐标的访问地址获取所述目标三维网格的纹理坐标。
S3319、媒体接入函数根据所述纹理坐标缓存器的描述信息和所述纹理坐标缓存器的缓存切片的描述信息将所述目标三维网格的纹理坐标写入所述纹理坐标缓存器的缓存器切片中。
S3320、显示引擎从所述场景描述文件的访问器列表(accessors)中获取各个访问器对应的访问器描述模块(包括：显示引擎从所述场景描述文件的访问器列表中获取用于访问所述目标触觉媒体文件的触觉媒体访问器对应的访问器描述模块，以及获取用于访问所述纹理坐标的纹理坐标访问器对应的访问器描述模块)。
S3321、显示引擎根据各个访问器对应的访问器描述模块获取各个访问器的描述信息(包括：显示引擎根据所述触觉媒体访问器对应的访问器描述模块获取所述触觉媒体访问器的描述信息，以及根据所述纹理坐标访问器对应的访问器描述模块获取所述纹理坐标访问器的描述信息)。
在一些实施例中，访问器的描述信息包括以下至少一项：
访问器所访问的缓存切片、访问器所访问的数据的数据类型、访问器的类型、访问器的数量、是否为MPEG时变访问器、时变访问器所访问的缓存切片、访问器参数是否随时间变化。
S3322、显示引擎根据各个访问器的描述信息创建各个访问器(包括：显示引擎根据所述触觉媒体访问器的描述信息创建所述触觉媒体访问器，以及根据所述纹理坐标访问器的描述信息创建所述纹理坐标访问器)。
S3323、显示引擎根据场景描述文件确定待渲染三维场景中的各个三维网格承载的各个触觉材质属性的类型。
在上述步骤S3323中，若显示引擎确定待渲染三维场景中的目标三维网格承载的第一触觉材质属性的类型为参考，则执行如下步骤：
S3324、显示引擎根据目标节点描述模块的MPEG触觉的媒体参考列表的目标媒体参考数组中的访问器索引语法元素声明的索引值，确定触觉媒体访问器。
其中，所述目标节点描述模块为包括所述目标三维网格的节点对应的节点描述模块，所述目标媒体参考数组为所述第一触觉材质属性所属的目标触觉媒体文件对应的媒体参考数组。
S3325、显示引擎通过所述触觉媒体访问器访问所述触觉媒体缓存器的缓存切片中的数据，以获取所述第一触觉材质属性的触觉值。
S3326、显示引擎从所述场景描述文件的采样器列表(samplers)中获取各个触觉材质属性的采样器的描述模块(包括：显示引擎从所述场景描述文件的采样器列表中获取所述第一触觉材质属性的采样器的描述模块)。
S3327、显示引擎根据各个触觉材质属性的采样器的描述模块获取各个触觉材质属性的采样器的描述信息(包括：显示引擎根据所述第一触觉材质属性的采样器的描述模块获取所述第一触觉材质属性的采样器的描述信息)。
S3328、显示引擎根据各个触觉材质属性的采样器的描述信息构建各个触觉材质属性的采样器(包括：显示引擎根据所述第一触觉材质属性的采样器的描述信息构建所述第一触觉材质属性的采样器)。
S3329、根据各个触觉材质属性的触觉值、所述第一触觉材质属性对应的纹理坐标以及所述第一触觉材质属性的采样器对所述待渲染三维场景进行渲染。
在一些实施例中，本申请一些实施例提供了一种场景描述文件的解析装置，该场景描述文件的生成装置，包括：
存储器，被配置为存储计算机程序；
处理器，被配置为用于在调用计算机程序时，使得所述场景描述文件的解析装置实现上述任一实施例所述的场景描述文件的解析方法。
在一些实施例中，本申请一些实施例提供了一种计算机可读存储介质，所述计算机可读存储介质上存储有计算机程序，当所述计算机程序被计算设备执行时，使得所述计算设备实现上述任一实施例所述的场景描述文件的解析方法。
在一些实施例中，本申请一些实施例提供了一种计算机程序产品，当所述计算机程序产品在计算机上运行时，使得所述计算机实现上述任一实施例所述的场景描述文件的解析方法。
最后应说明的是：以上各实施例仅用以说明本申请的技术方案，而非对其限制；尽管参照前述各实施例对本申请进行了详细的说明，本领域的普通技术人员应当理解：其依然可以对前述各实施例所记载的技术方案进行修改，或者对其中部分或者全部技术特征进行等同替换；而这些修改或者替换，并不使相应技术方案的本质脱离本申请各实施例技术方案的范围。
为了方便解释，已经结合具体的实施方式进行了上述说明。但是，上述示例性的讨论不是意图穷尽或者将实施方式限定到上述公开的具体形式。根据上述的教导，可以得到多种修改和变形。上述实施方式的选择和描述是为了更好的解释原理以及实际的应用，从而使得本领域技术人员更好的使用所述实施方式以及适于具体使用考虑的各种不同的变形的实施方式。

Claims (44)

  1. 一种场景描述文件的解析方法，包括：
    根据场景描述文件确定待渲染三维场景中的各个三维网格承载的各个触觉材质属性的类型；
    当所述待渲染三维场景中的目标三维网格承载的第一触觉材质属性的类型为参考时，获取目标节点描述模块的动态图像专家组MPEG触觉的媒体参考列表的目标媒体参考数组中的访问器索引语法元素声明的索引值；所述目标节点描述模块为包括所述目标三维网格的节点对应的节点描述模块，所述目标媒体参考数组为所述第一触觉材质属性所属的目标触觉媒体文件对应的媒体参考数组；
    根据所述访问器索引语法元素声明的索引值获取用于访问所述目标触觉媒体文件的解码数据的触觉媒体访问器的描述信息。
  2. 根据权利要求1所述的方法，所述根据所述访问器索引语法元素声明的索引值获取用于访问目标触觉媒体文件的解码数据的触觉媒体访问器的描述信息，包括：
    根据所述访问器索引语法元素声明的索引值从所述场景描述文件的访问器列表中获取所述触觉媒体访问器对应的访问器描述模块；
    解析所述触觉媒体访问器对应的访问器描述模块获取所述触觉媒体访问器的描述信息。
  3. 根据权利要求1所述的方法，所述根据场景描述文件确定待渲染三维场景中的各个三维网格承载的各个触觉材质属性的类型，包括：
    从所述场景描述文件的网格列表中获取所述待渲染三维场景中的各个三维网格对应的网格描述模块；
    获取各个三维网格对应的网格描述模块的MPEG触觉材质中的数组索引语法元素声明的索引值；
    根据各个三维网格对应的网格描述模块的MPEG触觉材质中的数组索引语法元素声明的索引值，从所述场景描述文件的触觉材质数组列表中获取各个三维网格的触觉材质属性数组；
    根据各个三维网格对应的触觉材质属性数组确定各个三维网格承载的各个触觉材质属性的类型。
  4. 根据权利要求3所述的方法，所述根据各个三维网格对应的触觉材质属性数组确定各个三维网格承载的各个触觉材质属性的类型，包括：
    从各个三维网格对应的触觉材质属性数组中获取各个三维网格承载的各个触觉材质属性对应的触觉材质属性描述模块；
    获取各个触觉材质属性对应的触觉材质属性描述模块中的属性类型语法元素的值；
    根据各个触觉材质属性对应的触觉材质属性描述模块中的属性类型语法元素的值，确定各个三维网格承载的各个触觉材质属性的类型。
  5. 根据权利要求1所述的方法，在根据目标节点描述模块的动态图像专家组MPEG触觉的目标媒体参考数组中的访问器索引语法元素声明的索引值之前，所述方法还包括：
    从所述场景描述文件的节点列表中获取所述待渲染三维场景中的各个节点对应的节点描述模块；
    获取所述待渲染三维场景中的各个节点对应的节点描述模块的网格索引列表；
    当任一节点描述模块的所述网格索引列表中声明的索引值包括所述目标三维网格对应的网格描述模块的索引值时，将该节点描述模块确定为所述目标节点描述模块。
  6. 根据权利要求5所述的方法，所述从所述场景描述文件的节点列表中获取所述待渲染三维场景中的各个节点对应的节点描述模块，包括：
    从所述场景描述文件的场景列表中获取所述待渲染三维场景对应的目标场景描述模块；
    获取所述目标场景描述模块的节点索引列表声明的索引值；
    根据所述目标场景描述模块的节点索引列表声明的索引值，从所述场景描述文件的节点列表中获取所述待渲染三维场景中的各个节点对应的节点描述模块。
  7. 根据权利要求5所述的方法，在根据目标节点描述模块的MPEG触觉的目标媒体参考数组中的访问器索引语法元素声明的索引值之前，所述方法还包括：
    获取所述第一触觉材质属性对应的触觉材质属性描述模块的纹理数组中的纹理索引语法元素声明的索引值；
    根据所述第一触觉材质属性对应的触觉材质属性描述模块的纹理数组中的纹理索引语法元素声明的索引值，从所述场景描述文件的纹理列表中获取所述第一触觉材质属性对应的纹理描述模块；
    获取所述第一触觉材质属性对应的纹理描述模块中的纹理贴图语法元素声明的索引值；
    根据所述第一触觉材质属性对应的纹理描述模块中的纹理贴图语法元素声明的索引值，从所述场景描述文件的纹理贴图列表中获取所述第一触觉材质属性的纹理贴图的访问地址。
  8. 根据权利要求7所述的方法，所述方法还包括：
    根据所述第一触觉材质属性的纹理贴图的访问地址获取所述第一触觉材质属性的纹理贴图；
    根据所述第一触觉材质属性的纹理贴图获取所述目标触觉媒体文件对应的目标媒体描述模块的索引值；
    将所述目标节点描述模块中所述触觉媒体标识语法元素的值为所述目标媒体描述模块的索引值的媒体参考数组确定为所述目标媒体参考数组。
  9. 根据权利要求7所述的方法，所述方法还包括：
    根据所述第一触觉材质属性对应的纹理描述模块中的采样器语法元素声明的索引值，从所述场景描述文件的采样器列表中获取所述第一触觉材质属性的采样器对应的采样器描述模块；
    根据所述第一触觉材质属性的采样器对应的采样器描述模块，获取所述第一触觉材质属性的采样器的描述信息。
  10. 根据权利要求7所述的方法，所述方法还包括：
    根据所述第一触觉材质属性对应的触觉材质属性描述模块的纹理数组中的纹理坐标索引语法元素声明的索引值，从所述目标三维网格对应的网格描述模块的基元的属性中获取所述第一触觉材质属性对应的纹理坐标语法元素；
    根据所述第一触觉材质属性对应的纹理坐标语法元素声明的索引值，获取所述第一触觉材质属性的纹理坐标访问器对应的访问器描述模块；
    根据所述第一触觉材质属性的纹理坐标访问器对应的访问器描述模块，获取所述第一触 觉材质属性的纹理坐标访问器的描述信息。
  11. 根据权利要求7所述的方法，所述方法还包括：
    当第二触觉材质属性的类型为高分辨率或低分辨率时，获取所述第二触觉材质属性对应的触觉材质属性描述模块的纹理数组中的纹理索引语法元素声明的索引值；
    根据所述第二触觉材质属性对应的触觉材质属性描述模块的纹理数组中的纹理索引语法元素声明的索引值，从所述场景描述文件的纹理列表中获取所述第二触觉材质属性对应的纹理描述模块；
    获取所述第二触觉材质属性对应的纹理描述模块中的纹理贴图语法元素声明的索引值和采样器语法元素声明的索引值；
    根据所述第二触觉材质属性对应的纹理描述模块中的纹理贴图语法元素声明的索引值，从所述场景描述文件的纹理贴图列表中获取所述第二触觉材质属性的纹理贴图的访问地址，以及根据所述第二触觉材质属性对应的纹理描述模块中的采样器语法元素声明的索引值，从所述场景描述文件的采样器列表中获取第二触觉触觉材质属性的采样器的描述信息。
  12. 根据权利要求1-11任一项所述的方法，所述方法还包括：
    从所述场景描述文件的缓存器列表中获取所述待渲染三维场景的各个缓存器对应的缓存器描述模块；
    根据各个缓存器对应的缓存器描述模块获取所述待渲染三维场景的各个缓存器的描述信息；
    其中，缓存器的描述信息包括以下至少一项：
    缓存器的容量、缓存器所缓存的数据的访问地址、是否为MPEG环形缓存器、环形缓存器的存储环节数量、环形缓存器所缓存的媒体文件对应的媒体描述模块的索引值、环形缓存器所缓存的媒体文件的数据的轨道索引值。
  13. 根据权利要求1-11任一项所述的方法，所述方法还包括：
    从所述场景描述文件的缓存切片列表中获取所述待渲染三维场景的各个缓存器缓存器的各个缓存切片对应的缓存切片描述模块；
    根据各个缓存切片对应的缓存切片描述模块获取各个缓存切片的描述信息；
    其中，缓存切片的描述信息包括以下至少一项：
    缓存切片所属的缓存器、缓存切片的容量、缓存切片的偏移量。
  14. 根据权利要求1-11任一项所述的方法，所述方法还包括：
    从所述场景描述文件的访问器列表中获取所述待渲染三维场景的各个访问器对应的访问器描述模块；
    根据各个访问器对应的访问器描述模块获取各个访问器的描述信息；
    其中，访问器的描述信息包括以下至少一项：
    访问器所访问的缓存切片、访问器所访问的数据的数据类型、访问器的类型、访问器的数量、是否为MPEG时变访问器、时变访问器所访问的缓存切片、访问器参数是否随时间变化。
  15. 根据权利要求1-11任一项所述的方法，所述方法还包括：
    从所述场景描述文件的MPEG媒体扩展的媒体列表中获取所述待渲染三维场景中的各个 媒体文件对应的媒体描述模块；
    根据各个媒体文件对应的媒体描述模块获取各个媒体文件的描述信息；
    其中，媒体文件的描述信息包括以下至少一项：
    媒体文件的名称、媒体文件是否自动播放、媒体文件是否循环播放、媒体文件的类型、媒体文件的访问地址、媒体文件的封装文件的轨道信息、媒体文件的编解码参数。
  16. 一种场景描述文件的解析装置，包括：
    存储器，被配置为存储计算机程序；
    处理器，被配置为用于在调用计算机程序时，使得所述三维场景的渲染装置实现权利要求1-15任一项所述的场景描述文件的解析方法。
  17. 一种三维场景的渲染方法，包括：
    根据场景描述文件确定待渲染三维场景中的各个三维网格承载的各个触觉材质属性的类型；
    当待渲染三维场景中的目标三维网格承载的第一触觉材质属性的类型为参考时，根据目标节点描述模块的动态图像专家组MPEG触觉的媒体参考列表的目标媒体参考数组中的访问器索引语法元素声明的索引值，确定触觉媒体访问器；所述目标节点描述模块为包括所述目标三维网格的节点对应的节点描述模块，所述目标媒体参考数组为所述第一触觉材质属性所属的目标触觉媒体文件对应的媒体参考数组；
    通过所述触觉媒体访问器访问触觉媒体缓存器的缓存切片中的数据，以获取所述第一触觉材质属性的触觉值，所述触觉媒体缓存器为用于缓存所述目标触觉媒体文件的解码数据的缓存器；
    根据所述第一触觉材质属性的触觉值对所述待渲染三维场景进行渲染。
  18. 根据权利要求17所述的方法，在通过所述触觉媒体访问器访问触觉媒体缓存器的缓存切片中的数据，以获取所述第一触觉材质属性的触觉值之前，所述方法还包括：
    从所述场景描述文件的访问器列表中获取所述触觉媒体访问器对应的访问器描述模块；
    根据所述触觉媒体访问器对应的访问器描述模块获取所述触觉媒体访问器的描述信息；
    根据所述触觉媒体访问器的描述信息创建所述触觉媒体访问器。
  19. 根据权利要求17所述的方法，在通过所述触觉媒体访问器访问触觉媒体缓存器的缓存切片中的数据，以获取所述第一触觉材质属性的触觉值之前，所述方法还包括：
    从所述场景描述文件的MPEG媒体的媒体列表中获取所述目标触觉媒体文件对应的媒体描述模块；
    根据所述目标触觉媒体文件对应的媒体描述模块获取所述目标触觉媒体文件的描述信息；
    向媒体接入函数发送所述目标触觉媒体文件的描述信息，以使所述媒体接入函数根据所述目标触觉媒体文件的描述信息获取所述目标触觉媒体文件，并对所述目标触觉媒体文件进行解封装和解码。
  20. 根据权利要求19所述的方法，所述向媒体接入函数发送所述目标触觉媒体文件的描述信息，包括：
    通过媒体接入函数应用程序编程接口API向所述媒体接入函数发送所述目标触觉媒体文件的描述信息。
  21. 根据权利要求19所述的方法，所述方法还包括：
    从所述场景描述文件的缓存器列表中获取所述触觉媒体缓存器对应的缓存器描述模块；
    根据所述触觉媒体缓存器对应的缓存器描述模块获取所述触觉媒体缓存器的描述信息；
    从所述场景描述文件的缓存切片列表中获取所述触觉媒体缓存器的缓存切片对应的缓存切片描述模块；
    根据所述触觉媒体缓存器的缓存切片对应的缓存切片描述模块获取所述触觉媒体缓存器的缓存切片的描述信息；
    向缓存管理模块发送所述触觉媒体缓存器的描述信息和所述触觉媒体缓存器的缓存切片的描述信息，以使所述缓存管理模块根据所述触觉媒体缓存器的描述信息创建所述触觉媒体缓存器以及根据所述触觉媒体缓存器的缓存切片的描述信息对所述触觉媒体缓存器进行缓存切片的划分。
  22. 根据权利要求21所述的方法，所述向缓存管理模块发送所述触觉媒体缓存器的描述信息和所述触觉媒体缓存器的缓存切片的描述信息，包括：
    通过缓存API向所述缓存管理模块发送所述触觉媒体缓存器的描述信息和所述触觉媒体缓存器的缓存切片的描述信息。
  23. 根据权利要求22所述的方法，所述方法还包括：
    将通过媒体接入函数API向媒体接入函数发送所述触觉媒体缓存器的描述信息和所述触觉媒体缓存器的缓存切片的描述信息，以使所述媒体接入函数根据所述触觉媒体缓存器的描述信息和所述触觉媒体缓存器的缓存切片的描述信息，将对所述目标触觉媒体文件进行解封装和解码得到的解码数据写入所述触觉媒体缓存器的缓存切片中。
  24. 根据权利要求21所述的方法，所述向缓存管理模块发送所述触觉媒体缓存器的描述信息和所述触觉媒体缓存器的缓存切片的描述信息，包括：
    通过媒体接入函数API向所述媒体接入函数发送所述触觉媒体缓存器的描述信息和所述触觉媒体缓存器的缓存切片的描述信息，以使所述媒体接入函数向所述缓存管理模块转发所述触觉媒体缓存器的描述信息和所述触觉媒体缓存器的缓存切片的描述信息。
  25. 根据权利要求17-24任一项所述的方法，所述根据所述第一触觉材质属性的触觉值对所述待渲染三维场景进行渲染，包括：
    根据所述第一触觉材质属性的触觉值、所述第一触觉材质属性对应的纹理坐标以及所述第一触觉材质属性的采样器对所述待渲染三维场景进行渲染。
  26. 根据权利要求25所述的方法，在根据所述第一触觉材质属性的触觉值、所述第一触觉材质属性对应的纹理坐标以及所述第一触觉材质属性的采样器对所述待渲染三维场景进行渲染之前，所述方法还包括：
    确定所述第一触觉材质属性对应的纹理坐标访问器；
    通过所述纹理坐标访问器访问纹理坐标缓存器的缓存切片中的数据，以获取所述第一触觉材质属性对应的纹理坐标；
    其中，所述触觉媒体缓存器为用于缓存所述第一触觉材质属性的纹理坐标的缓存器。
  27. 根据权利要求26所述的方法，在通过所述纹理坐标访问器访问纹理坐标缓存器的缓存切片中的数据，以获取所述第一触觉材质属性对应的纹理坐标之前，所述方法还包括：
    从所述场景描述文件的访问器列表中获取所述纹理坐标访问器对应的访问器描述模块；
    根据所述纹理坐标访问器对应的访问器描述模块获取所述纹理坐标访问器的描述信息；
    根据所述纹理坐标访问器的描述信息创建所述纹理坐标访问器。
  28. 根据权利要求26所述的方法，在通过所述纹理坐标访问器访问纹理坐标缓存器的缓存切片中的数据，以获取所述第一触觉材质属性的纹理坐标之前，所述方法还包括：
    从所述场景描述文件的缓存器列表中获取所述纹理坐标缓存器对应的缓存器描述模块；
    根据所述纹理坐标缓存器对应的缓存器描述模块获取所述纹理坐标缓存器的描述信息；
    从所述场景描述文件的缓存切片列表中获取所述纹理坐标缓存器的缓存切片对应的缓存切片描述模块；
    根据所述纹理坐标缓存器的缓存切片对应的缓存切片描述模块获取所述纹理坐标缓存器的缓存切片的描述信息；
    向缓存管理模块发送所述纹理坐标缓存器的描述信息和所述纹理坐标缓存器的缓存切片的描述信息，以使所述缓存管理模块根据所述纹理坐标缓存器的描述信息创建所述纹理坐标缓存器以及根据所述纹理坐标缓存器的缓存切片的描述信息对所述纹理坐标缓存器进行缓存切片的划分。
  29. 根据权利要求28所述的方法，所述方法还包括：
    向媒体接入函数发送所述纹理坐标缓存器的描述信息和所述纹理坐标缓存器的缓存切片的描述信息，以使所述媒体接入函数根据所述纹理坐标缓存器的描述信息中的纹理坐标访问地址获取所述目标三维网格的纹理坐标，以及根据所述纹理坐标缓存器的描述信息和所述纹理坐标缓存器的缓存切片的描述信息将所述目标三维网格的纹理坐标写入所述纹理坐标缓存器的缓存器切片中。
  30. 根据权利要求27所述的方法，在根据所述第一触觉材质属性的触觉值、所述第一触觉材质属性对应的纹理坐标以及所述第一触觉材质属性的采样器对所述待渲染三维场景进行渲染之前，所述方法还包括：
    从所述场景描述文件的采样器列表中获取所述第一触觉材质属性的采样器的描述模块；
    根据所述第一触觉材质属性的采样器的描述模块获取所述第一触觉材质属性的采样器的描述信息；
    根据所述第一触觉材质属性的采样器的描述信息构建所述第一触觉材质属性的采样器。
  31. 一种三维场景的渲染装置，包括：
    存储器，被配置为存储计算机程序；
    处理器，被配置为用于在调用计算机程序时，使得所述三维场景的渲染装置实现权利要求17-30任一项所述的三维场景的渲染方法。
  32. 一种触觉媒体文件的处理方法，包括：
    接收显示引擎发送的目标触觉媒体文件的描述信息、触觉媒体缓存器的描述信息以及所述触觉媒体缓存器的缓存切片的描述信息；所述目标触觉媒体文件中包括类型为参考的第一触觉材质属性的触觉值；
    根据所述目标触觉媒体文件的描述信息获取所述目标触觉媒体文件的解码数据；
    根据所述触觉媒体缓存器的描述信息和所述触觉媒体缓存器的缓存切片的描述信息，将 所述解码数据写入所述触觉媒体缓存器的缓存切片中，以使所述显示引擎根据目标节点描述模块的动态图像专家组MPEG触觉的媒体参考列表的目标媒体参考数组中的访问器索引语法元素声明的索引值确定触觉媒体访问器，以及通过所述触觉媒体访问器访问所述触觉媒体缓存器的缓存切片中的数据获取所述第一触觉材质属性的触觉值，所述目标节点描述模块为包括承载所述第一触觉材质属性的目标三维网格的节点对应的节点描述模块，所述目标媒体参考数组为所述目标触觉媒体文件对应的媒体参考数组。
  33. 根据权利要求32所述的方法，所述接收显示引擎发送的目标触觉媒体文件的描述信息、触觉媒体缓存器的描述信息以及所述触觉媒体缓存器的缓存切片的描述信息，包括：
    通过媒体接入函数应用程序接口API接收所述显示引擎发送的所述目标触觉媒体文件的描述信息、所述触觉媒体缓存器的描述信息以及所述触觉媒体缓存器的缓存切片的描述信息。
  34. 根据权利要求32所述的方法，所述根据所述目标触觉媒体文件的描述信息获取所述目标触觉媒体文件的解码数据，包括：
    根据所述目标触觉媒体文件的描述信息创建用于处理所述目标触觉媒体文件的触觉媒体管线；
    通过所述触觉媒体管线获取所述目标触觉媒体文件，并对所述目标触觉媒体文件进行解封装和解码，以获取所述目标触觉媒体文件的解码数据。
  35. 根据权利要求32所述的方法，所述目标触觉媒体文件的描述信息包括所述目标触觉媒体文件的访问地址；所述根据所述目标触觉媒体文件的描述信息获取所述目标触觉媒体文件的解码数据，包括：
    根据所述目标触觉媒体文件的访问地址获取所述目标触觉媒体文件。
  36. 根据权利要求35所述的方法，所述根据所述目标触觉媒体文件的访问地址获取所述目标触觉媒体文件，包括：
    根据所述目标触觉媒体文件的访问地址向媒体资源服务器发送媒体资源请求；
    接收所述媒体服务器发送的携带有所述目标触觉媒体文件的媒体资源响应。
  37. 根据权利要求35所述的方法，所述根据所述目标触觉媒体文件的访问地址获取所述目标触觉媒体文件，包括：
    根据所述目标触觉媒体文件的访问地址从预设存储空间中读取所述目标触觉媒体文件。
  38. 根据权利要求37所述的方法，所述目标触觉媒体文件的描述信息还包括所述目标触觉媒体文件的各个码流轨道的索引值；所述根据所述目标触觉媒体文件的描述信息获取所述目标触觉媒体文件的解码数据，包括：
    根据所述目标触觉媒体文件的各个码流轨道的索引值对所述目标触觉媒体文件进行解封装，以获取所述目标触觉媒体文件的各个码流轨道的码流。
  39. 根据权利要求38所述的方法，所述目标触觉媒体文件的描述信息还包括所述目标触觉媒体文件的各个码流轨道的编解码参数；所述根据所述目标触觉媒体文件的描述信息获取所述目标触觉媒体文件的解码数据，包括：
    根据所述目标触觉媒体文件的各个码流轨道的编解码参数对所述目标触觉媒体文件的各个码流轨道的码流进行解码，获取所述目标触觉媒体文件的各个码流轨道的对应的解码数据，以获取目标触觉媒体文件的解码数据。
  40. 根据权利要求32-39任一项所述的方法，在根据所述触觉媒体缓存器的描述信息和所述触觉媒体缓存器的缓存切片的描述信息，将所述解码数据写入所述触觉媒体缓存器的缓存切片中之前，所述方法还包括：
    将所述触觉媒体缓存器的描述信息和所述触觉媒体缓存器的缓存切片的描述信息转发至缓存管理模块，以使所述管理模块根据所述触觉媒体缓存器的描述信息创建所述触觉媒体缓存器，以及根据所述触觉媒体缓存器的缓存切片的描述信息对所述触觉媒体缓存器进行缓存切片的划分。
  41. 根据权利要求40所述的方法，所述将所述触觉媒体缓存器的描述信息和所述触觉媒体缓存器的缓存切片的描述信息转发至缓存管理模块，包括：
    通过缓存API将所述触觉媒体缓存器的描述信息和所述触觉媒体缓存器的缓存切片的描述信息转发至所述缓存管理模块。
  42. 根据权利要求32-39任一项所述的方法，所述方法还包括：
    接收所述显示引擎发送的纹理坐标缓存器的描述信息和所述纹理坐标缓存器的缓存切片的描述信息；所述纹理坐标缓存器为用于缓存所述目标三维网格的纹理坐标的缓存器；所述纹理坐标缓存器的描述信息包括所述目标三维网格的纹理坐标的访问地址；
    根据所述目标三维网格的纹理坐标的访问地址获取所述目标三维网格的纹理坐标；
    根据所述纹理坐标缓存器的描述信息和所述纹理坐标缓存器的缓存切片的描述信息将所述目标三维网格的纹理坐标写入所述纹理坐标缓存器的缓存器切片中。
  43. 根据权利要求42所述的方法，在根据所述纹理坐标缓存器的描述信息和所述纹理坐标缓存器的缓存切片的描述信息将所述目标三维网格的纹理坐标写入所述纹理坐标缓存器的缓存器切片中之前，所述方法还包括：
    将所述纹理坐标缓存器的描述信息和所述纹理坐标缓存器的缓存切片的描述信息转发至缓存管理模块，以使所述管理模块根据所述纹理坐标缓存器的描述信息创建所述纹理坐标缓存器，以及根据所述纹理坐标缓存器的缓存切片的描述信息对所述纹理坐标缓存器进行缓存切片的划分。
  44. 一种触觉媒体文件的处理装置，包括：
    存储器，被配置为存储计算机程序；
    处理器，被配置为用于在调用计算机程序时，使得所述触觉媒体文件的处理装置实现权利要求32-43任一项所述的触觉媒体文件的处理方法。
PCT/CN2023/097877 2023-01-10 2023-06-01 场景描述文件的解析方法及装置 Ceased WO2024148752A1 (zh)

Applications Claiming Priority (6)

Application Number Priority Date Filing Date Title
CN202310036790 2023-01-10
CN202310036790.8 2023-01-10
CN202310275300.X 2023-03-20
CN202310275300 2023-03-20
CN202310362181.1A CN118334191A (zh) 2023-01-10 2023-04-06 一种场景描述文件的生成方法及装置
CN202310362181.1 2023-04-06

Publications (1)

Publication Number Publication Date
WO2024148752A1 true WO2024148752A1 (zh) 2024-07-18

Family

ID=91765190

Family Applications (2)

Application Number Title Priority Date Filing Date
PCT/CN2023/097877 Ceased WO2024148752A1 (zh) 2023-01-10 2023-06-01 场景描述文件的解析方法及装置
PCT/CN2024/070187 Ceased WO2024149119A1 (zh) 2023-01-10 2024-01-02 场景描述文件的生成方法及装置

Family Applications After (1)

Application Number Title Priority Date Filing Date
PCT/CN2024/070187 Ceased WO2024149119A1 (zh) 2023-01-10 2024-01-02 场景描述文件的生成方法及装置

Country Status (6)

Country Link
US (1) US20250291419A1 (zh)
EP (2) EP4745725A2 (zh)
JP (1) JP2026501767A (zh)
KR (1) KR20250121974A (zh)
CN (11) CN118334191A (zh)
WO (2) WO2024148752A1 (zh)

Citations (4)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
WO2022100985A1 (en) * 2020-11-12 2022-05-19 Interdigital Ce Patent Holdings, Sas Representation format for haptic object
WO2022145357A1 (ja) * 2020-12-28 2022-07-07 ソニーグループ株式会社 情報処理装置および方法
US20220343546A1 (en) * 2019-10-01 2022-10-27 Interdigital Ce Patent Holdings Haptic atlas coding and decoding format
CN115315943A (zh) * 2021-01-06 2022-11-08 腾讯美国有限责任公司 用于媒体场景描述的方法和设备

Family Cites Families (3)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
US9147291B2 (en) * 2012-04-20 2015-09-29 Samsung Electronics Co., Ltd. Method and apparatus of processing data to support augmented reality
US20140214211A1 (en) * 2013-01-29 2014-07-31 Electronics And Telecommunications Research Institute Outputactuator node, mpeg terminal with outputactuator node, and method for controlling actuator using the outputactuator node
EP3099030A1 (en) * 2015-05-26 2016-11-30 Thomson Licensing Method and device for encoding/decoding a packet comprising data representative of a haptic effect

Patent Citations (4)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
US20220343546A1 (en) * 2019-10-01 2022-10-27 Interdigital Ce Patent Holdings Haptic atlas coding and decoding format
WO2022100985A1 (en) * 2020-11-12 2022-05-19 Interdigital Ce Patent Holdings, Sas Representation format for haptic object
WO2022145357A1 (ja) * 2020-12-28 2022-07-07 ソニーグループ株式会社 情報処理装置および方法
CN115315943A (zh) * 2021-01-06 2022-11-08 腾讯美国有限责任公司 用于媒体场景描述的方法和设备

Also Published As

Publication number Publication date
CN118334192A (zh) 2024-07-12
KR20250121974A (ko) 2025-08-12
EP4618024A4 (en) 2026-03-04
CN118334210A (zh) 2024-07-12
CN118334191A (zh) 2024-07-12
CN118334194A (zh) 2024-07-12
US20250291419A1 (en) 2025-09-18
WO2024149119A1 (zh) 2024-07-18
EP4745725A2 (en) 2026-05-20
CN118334211A (zh) 2024-07-12
CN120226051A (zh) 2025-06-27
JP2026501767A (ja) 2026-01-16
EP4618024A1 (en) 2025-09-17
CN118338031A (zh) 2024-07-12
CN118736067A (zh) 2024-10-01
CN118334193A (zh) 2024-07-12
CN118334209A (zh) 2024-07-12
CN118333838A (zh) 2024-07-12

Similar Documents

Publication Publication Date Title
US20250166234A1 (en) Method and apparatus for generating scene description document
CN114116617B (zh) 点云媒体的数据处理方法、装置、设备及可读存储介质
WO2022037423A1 (zh) 点云媒体的数据处理方法、装置、设备及介质
WO2024149120A1 (zh) 场景描述文件的生成方法及装置
WO2024149117A1 (zh) 场景描述文件的生成方法及装置
WO2025039852A9 (zh) 点云文件封装与解封装方法、装置、设备及存储介质
CN118334194A (zh) 一种三维场景的渲染方法及装置
US20260122294A1 (en) Annotation metadata system for asset interchange
CN121304878A (zh) 场景描述文件的生成方法及装置
CN121304880A (zh) 三维场景的场景数据的处理方法及装置
CN121357366A (zh) 缓存管理方法及装置
CN121304877A (zh) 场景描述文件的解析方法及装置
CN121304879A (zh) 三维场景的渲染方法及装置
CN121353476A (zh) 数字人的动画方法及装置
HK40064955B (zh) 点云媒体的数据处理方法、装置、设备及可读存储介质

Legal Events

Date Code Title Description
121 Ep: the epo has been informed by wipo that ep was designated in this application

Ref document number: 23915516

Country of ref document: EP

Kind code of ref document: A1

NENP Non-entry into the national phase

Ref country code: DE

122 Ep: pct application non-entry in european phase

Ref document number: 23915516

Country of ref document: EP

Kind code of ref document: A1