CN117336281A

CN117336281A - Method and device for unpacking and packaging tactile media file and electronic equipment

Info

Publication number: CN117336281A
Application number: CN202311309906.7A
Authority: CN
Inventors: 胡颖; 许晓中
Original assignee: Tencent Technology Shenzhen Co Ltd
Current assignee: Tencent Technology Shenzhen Co Ltd
Priority date: 2023-10-10
Filing date: 2023-10-10
Publication date: 2024-01-02

Abstract

The application belongs to the technical field of immersion media, and particularly relates to a method for unpacking and packaging a haptic media file, a device for unpacking and packaging the haptic media file, a computer readable medium, an electronic device and a computer program product. The method for decapsulating the haptic media file comprises the following steps: obtaining a haptic media file comprising one or more haptic media tracks for encapsulating a haptic media bitstream, at least one of the haptic media tracks comprising one or more sub-samples, a data box of the sub-samples comprising a flag field for indicating a manner of division of the sub-samples; and de-encapsulating the haptic media file according to the mark field to obtain a haptic media bit stream carried in the sub-sample. The haptic media data processing flexibility can be improved.

Description

Method and device for unpacking and packaging tactile media file and electronic equipment

Technical Field

The application belongs to the technical field of immersion media, and particularly relates to a method for unpacking and packaging a haptic media file, a device for unpacking and packaging the haptic media file, a computer readable medium, an electronic device and a computer program product.

Background

The presentation of immersive media content is often accompanied by a variety of wearable or interactable devices, with the immersive media being presented in a new manner of tactile media in addition to traditional video and audio media. Haptic media represents the sensory experience that the human body obtains through contact, such as vibration, pressure, temperature, etc. The traditional media data processing means are difficult to adapt to the presentation requirement of the touch media, and have the problem of poor data processing flexibility.

Disclosure of Invention

The application provides a method for unpacking and packaging a haptic media file, a device for unpacking and packaging a haptic media file, a computer readable medium, an electronic device and a computer program product, and aims to improve the data processing flexibility of the haptic media.

Other features and advantages of the present application will be apparent from the following detailed description, or may be learned in part by the practice of the application.

According to one aspect of embodiments of the present application, there is provided a method of decapsulating a haptic media file, the method comprising:

obtaining a haptic media file comprising one or more haptic media tracks for encapsulating a haptic media bitstream, at least one of the haptic media tracks comprising one or more sub-samples, a data box of the sub-samples comprising a flag field for indicating a manner of division of the sub-samples;

And de-encapsulating the haptic media file according to the mark field to obtain a haptic media bit stream carried in the sub-sample.

According to one aspect of embodiments of the present application, there is provided a decapsulation device for haptic media files, the device comprising:

a file acquisition module configured to acquire a haptic media file, the haptic media file comprising one or more haptic media tracks for encapsulating a haptic media bitstream, at least one of the haptic media tracks comprising one or more sub-samples, a data box of the sub-samples comprising a flag field for indicating a manner of division of the sub-samples;

and the decapsulation module is configured to decapsulate the haptic media file according to the mark field to obtain a haptic media bit stream carried in the sub-sample.

According to one aspect of the embodiments of the present application, there is provided a method for encapsulating a haptic media file, the method comprising:

obtaining a haptic media bitstream to be packaged;

and packaging the haptic media bit stream to obtain a haptic media file, wherein the haptic media file comprises one or more haptic media tracks for packaging the haptic media bit stream, at least one sample in the haptic media tracks comprises one or more sub-samples, and a data box of the sub-samples comprises a mark field for indicating the division mode of the sub-samples.

According to one aspect of embodiments of the present application, there is provided an encapsulation device for a haptic media file, the device comprising:

a bitstream acquisition module configured to acquire a haptic media bitstream to be packaged;

and the packaging module is configured to perform packaging processing on the tactile media bitstream to obtain a tactile media file, wherein the tactile media file comprises one or more tactile media tracks for packaging the tactile media bitstream, at least one sample in the tactile media tracks comprises one or more sub-samples, and a data box of the sub-samples comprises a mark field for indicating the division mode of the sub-samples.

According to an aspect of the embodiments of the present application, there is provided a computer readable medium having stored thereon a computer program which, when executed by a processor, implements a method of decapsulating or a method of encapsulating a haptic media file as in the above technical solutions.

According to an aspect of the embodiments of the present application, there is provided an electronic device including: a processor; and a memory for storing executable instructions of the processor; wherein the processor is configured to execute the executable instructions to implement a method of unpacking or a method of packing a haptic media file as in the above technical solutions.

According to an aspect of embodiments of the present application, there is provided a computer program product comprising a computer program which, when executed by a processor, implements a method of decapsulating or encapsulating a haptic media file as in the above technical solution.

In the technical scheme provided by the embodiment of the application, the sub-samples can be flexibly divided when the haptic media file is unpacked or packaged by defining a new representation method of the haptic media content, and the sub-samples are supported to be packaged based on different division modes, so that the haptic media can be flexibly presented, and the flexibility of processing the haptic media data is improved.

Drawings

The accompanying drawings, which are incorporated in and constitute a part of this specification, illustrate embodiments consistent with the application and together with the description, serve to explain the principles of the application. It is apparent that the drawings in the following description are only some embodiments of the present application, and that other drawings may be obtained from these drawings without inventive effort for a person of ordinary skill in the art.

Fig. 1 shows a block diagram of a data processing architecture of an immersive media system to which the technical solution of the present application is applied.

Fig. 2 shows a schematic diagram of system interaction in an application scenario according to an embodiment of the present application.

FIG. 3 illustrates a flow chart of a method of decapsulating a haptic media file on the side of a decoding end in one embodiment of the application.

FIG. 4 illustrates a data structure of an exchange format file for representing haptic media content in one embodiment of the present application.

FIG. 5 illustrates a data structure of haptic media described in one implementation of an embodiment of the present application.

FIG. 6 illustrates a data structure of a data stream of haptic media described in one implementation of an embodiment of the present application.

FIG. 7 illustrates a schematic diagram of time cells versus haptic samples described in one implementation of an embodiment of the present application.

FIG. 8 illustrates a data structure of an exchange format file for describing haptic media content in one embodiment of the present application.

FIG. 9 illustrates a block diagram of the structure of a haptic media track in one application scenario according to an embodiment of the present application.

FIG. 10 illustrates a flow chart of a method of packaging a haptic media file on the side of an encoding end in one embodiment of the present application.

Fig. 11 schematically illustrates a block diagram of a device for decapsulating haptic media files provided by an embodiment of the application.

Fig. 12 schematically illustrates a block diagram of a device for encapsulating a haptic media file provided in an embodiment of the present application.

Fig. 13 schematically illustrates a block diagram of a computer system suitable for use in implementing embodiments of the present application.

Detailed Description

Example embodiments will now be described more fully with reference to the accompanying drawings. However, the exemplary embodiments may be embodied in many forms and should not be construed as limited to the examples set forth herein; rather, these embodiments are provided so that this disclosure will be thorough and complete, and will fully convey the concept of the example embodiments to those skilled in the art.

Furthermore, the described features, structures, or characteristics may be combined in any suitable manner in one or more embodiments. In the following description, numerous specific details are provided to give a thorough understanding of embodiments of the present application. One skilled in the relevant art will recognize, however, that the aspects of the application can be practiced without one or more of the specific details, or with other methods, components, devices, steps, etc. In other instances, well-known methods, devices, implementations, or operations are not shown or described in detail to avoid obscuring aspects of the application.

The block diagrams depicted in the figures are merely functional entities and do not necessarily correspond to physically separate entities. That is, the functional entities may be implemented in software, or in one or more hardware modules or integrated circuits, or in different networks and/or processor devices and/or microcontroller devices.

The flow diagrams depicted in the figures are exemplary only, and do not necessarily include all of the elements and operations/steps, nor must they be performed in the order described. For example, some operations/steps may be decomposed, and some operations/steps may be combined or partially combined, so that the order of actual execution may be changed according to actual situations.

In particular embodiments of the present application, related data such as haptic media files and the like are referred to, and when various embodiments of the present application are applied to particular products or technologies, user permissions or consent are required, and collection, use and processing of related data is required to comply with relevant laws and regulations and standards of the relevant countries and regions.

In the embodiment of the application, the immersive media refers to media content capable of providing immersive experience for consumers, so that the consumers immersed in the media content can obtain visual, auditory and other sensory experience in the real world. Immersive media can be classified into 3DoF media, 3dof+ media, and 6DoF media according to the degree of freedom (Degree of Freedom, doF) of consumers in consuming media content; 3DoF media may support a user to consume corresponding media content through 3DoF (i.e., three degrees of freedom), 3dof+ media may support a user to consume corresponding media content through 3dof+, and 6DoF media may support a user to consume corresponding media content through 6 DoF. Wherein 3DoF means that the user is fixed at the center point of a three-dimensional space, and the user's head can rotate around the X-axis, Y-axis and Z-axis in three degrees of freedom; 3dof+ refers to the degree of freedom in which the user's head can also undergo limited movements (e.g., translations in a limited space) along the X, Y and Z axes on a three degree of freedom basis; 6DoF refers to the degree of freedom in which the consumer can also perform free motion (e.g., free translation) along the X, Y and Z axes on a three degree of freedom basis. It should be noted that the degrees of freedom mentioned in the embodiments of the present application can be understood as degrees of freedom that support the movement of the user and generate the interaction of the content when the user views the immersive media.

In addition to traditional visual and auditory presentations, immersive media also has a new presentation mode, namely haptic presentation Haptics. Haptic presentation the haptic presentation mechanism, combined with hardware and software, allows the user to receive information through a body part, providing an embedded physical sensation, conveying critical information about the system the user is using. For example, a cell phone may vibrate to alert its owner that a message has been received. Such vibration is one type of haptic presentation. Haptic presentation may enhance both audible and visual presentation, enhancing the user experience.

Haptic media signals are used to represent a particular modality of haptic experience, which is a Signal rendered on a particular haptic device, and can generally be categorized into various categories of vibrotactile, kinematic tactile, electrotactile, and the like.

Vibrotactile is a direct tactile presentation in the form of vibration. The vibrotactile can simulate vibration of a specific frequency and intensity by motor vibration of the terminal device. For example, in shooting games, the specific effect of shooting props in use is simulated by vibration.

The kinematic touch sense is used for simulating the motion states such as the weight or the pressure of an object. For example, in a game involving the driving of a vehicle such as an automobile, the steering wheel may resist rotation when moving or operating a heavier vehicle at a higher speed. This type of feedback directly affects the muscles of the user. In the example of a driving game, the user must exert more force to obtain the desired response from the steering wheel.

Electrotactility is the feedback of information simulating a specific texture by electrical stimulation. Electrotactile presentation uses electrical impulses to provide tactile stimulation to nerve endings of a user's skin. Electrotactile presentation can create a highly realistic experience for a user wearing a suit or glove equipped with electrotactile technology. Almost any sensation can be simulated with an electrical pulse: temperature change, pressure change, moisture sensation.

From vibrotactile devices commonly used in the daily life, to diversified tactile presentations in the subdivision domain, tactile presentations themselves have been a way of presentation to which users are accustomed. With the popularity of wearable devices and interactive devices, haptic presentations perceived by users when consuming media content are no longer limited to basic vibrotactile, but rather include a more realistic haptic presentation experience that approximates the full range of bodily sensations of vibration, pressure, speed, acceleration, temperature, humidity, smell, and the like.

The embodiment of the application provides a processing method of haptic media content, which can be used for flexibly processing haptic media signals and supporting the code stream storage and transmission of haptic media based on the representation of the haptic media signals. The embodiment of the application can be particularly applied to related products related to tactile feedback, such as links of server side, player side, intermediate node and the like of an immersive media system.

As shown in fig. 1, an immersive media signal may be generated by acquisition of a signal acquisition device at the encoder side, and may include a haptic media signal, a video media signal, an audio media signal, and metadata information describing these media signals. The immersive media signals may also include point cloud media signals or other media signals having any manner of presentation.

The signal acquisition device may be a sensor device for acquiring real media signals from the real world, and may include, for example, a microphone for acquiring audio media signals, a camera for acquiring video media signals, a vibration sensor for acquiring haptic media signals, a temperature sensor, a humidity sensor, a speed sensor, and the like. In addition, the signal acquisition device may also be a computer device that generates virtual haptic signals by software simulating haptic effects.

The encoder performs format analysis (format analysis) on the video media signal or the haptic media signal based on the metadata information, and then performs Transcoding (Transcoding) on the same. The encoder performs band decomposition (frequency band decomposition) on the audio media signal based on the metadata information, and then performs key frame extraction (keyframe extraction) and wavelet processing (wavelets processing) thereon.

The various immersive media signals processed within the encoder will be co-formatted (Formatting) to form a switch format file having a specified data switch format. The exchange format file refers to a format file for data transmission and exchange between the server side and the client side, and the data exchange format may include, for example, extensible markup language XML or text data exchange format JSON. JSON (JavaScript Object Notation) is a JavaScript object numbered musical notation, and is a lightweight data exchange format. JSON stores and represents data in a text format that is completely independent of the programming language. The simple and clear hierarchical structure enables the JSON to be an ideal data exchange language, is easy to read and write by people, is easy to analyze and generate by machines, and effectively improves the network transmission efficiency.

In some application scenarios, the encoder may transmit the exchange format file directly to the decoder.

In other application scenarios, the encoder may also binary compress (Binarycompression) the exchange format file, then transmit a compressed data stream with a specified distribution format (Distribution format) to the decoder, and the decoder binary decompresses (Binarycompression) the received compressed data stream to obtain the corresponding exchange format file.

In other applications, the encoder may also encapsulate (packetize) the binary compressed data stream to transmit the data stream encapsulated in packets having a specified format (e.g., the data stream having an MIHS format specified in the standard document ISO/IEC 23090-31 MIHS) to the decoder. The decoder sequentially performs decapsulation (Depacketization) and binary decompression (Binarydecompression) on the received data stream to obtain a corresponding exchange format file.

The exchange format file after decoding processing by the decoder can be rendered and output by the renderer, so that the rendered immersive media effect is presented to the user.

Fig. 2 shows a schematic diagram of system interaction in an application scenario according to an embodiment of the present application. As shown in fig. 2, the system architecture in this application scenario may include a client 201 and a server 202.

The client 201 may include various electronic devices such as a smart phone, a tablet computer, a notebook computer, a desktop computer, a smart speaker, a smart wearable device, a smart vehicle device, a smart payment terminal, and particularly may include a wearable device or an interactable device capable of rendering haptic effects.

The server 202 may be an independent physical server, a server cluster or a distributed system formed by a plurality of physical servers, or a cloud server providing cloud services, cloud databases, cloud computing, cloud functions, cloud storage, network services, cloud communication, middleware services, domain name services, security services, CDNs, basic cloud computing services such as big data and artificial intelligence platforms, and the like.

The client 201 and the server 202 are connected by a network, which may be a communication medium of various connection types capable of providing a communication link between the client 201 and the server 202, such as a wired communication link or a wireless communication link.

The system architecture in embodiments of the present application may have any number of clients and servers, as desired for implementation. For example, the server 202 may be a server group consisting of a plurality of server devices. In addition, the technical solution provided in the embodiment of the present application may be applied to the client 201, may also be applied to the server 202, or may be implemented by the client 201 and the server 202 together, which is not limited in particular in the present application.

For example, a method for processing haptic media in embodiments of the present application may include the following process.

(1) Server 202 creates or captures haptic media signals based on the desired haptic media effects and generates an exchange format file of the haptic media signals, specifically containing information such as the underlying metadata of the haptic media, the rendering device, the rendering character, etc.

(2) The server 202 compresses the haptic media signal into a haptic media bitstream.

(3) The server 202 encapsulates the haptic media bitstream into a haptic media file F, the haptic media file comprising one or more haptic media tracks, and the samples of the haptic media tracks may be divided into one or more sub-samples.

(4) The server 202 transmits the haptic media file F to the client 201 where the user is located. Alternatively, server 202 may slice haptic media file F into a plurality of haptic media file segments Fs and transmit the plurality of haptic media file segments Fs to client 201 where the user is located.

Server 202 may transmit the complete haptic media file directly to client 201; server 202 may also transmit one or more haptic media fragments to client 201 via streaming. In streaming, the server needs to send a signaling message describing the media resource to the client 201, and the client 201 requests a specific media resource according to the signaling message. The signaling message may be, for example, media presentation description signaling MPD (media presentation description) in DASH for describing media segment information.

DASH is an adaptive bit rate streaming technology that enables high quality streaming media to be delivered over the internet via a conventional HTTP web server. DASH breaks the content into a series of small HTTP-based file segments, each containing very short lengths of playable content, while the total length of the content may be as long as several hours (e.g., a movie or sports event live). The content will be made into multiple bit rate alternative segments to provide multiple bit rate versions for selection. When media content is played by a DASH client, the client will automatically select which alternative to download and play based on current network conditions. The client will select the highest bit rate clip that can be downloaded in time for playback, thereby avoiding play-over or rebuffering events. As such, DASH clients can seamlessly adapt to changing network conditions and provide a high quality playback experience with less incidence of chunking and rebuffering.

(5) The client 201 receives the haptic media file or the haptic media fragment, selects the sub-sample of the corresponding haptic media track according to the metadata information related to the sub-sample in the haptic media file, combines with the device capability and consumption preference of the user, and the like, and performs decapsulation and decoding to obtain the exchange format of the haptic media signal and rendering and presenting.

The following describes in detail, with reference to specific embodiments, a method for decapsulating and encapsulating a haptic media file, an apparatus for decapsulating and encapsulating a haptic media file, a computer readable medium, an electronic device, and a computer program product provided in the embodiments of the present application.

Fig. 3 shows a flowchart of a method for decapsulating a haptic media file located on the side of a decoding end in one embodiment of the present application, which may be performed by the client or the server shown in fig. 2, and the embodiment of the present application is described by taking the method performed by the client as an example. As shown in fig. 3, the method of decapsulating the haptic media file may include the following steps S310 to S320.

S310: obtaining a haptic media file, the haptic media file comprising one or more haptic media tracks for encapsulating a haptic media bitstream, at least one of the haptic media tracks comprising one or more sub-samples, a data box of the sub-samples comprising a flag field for indicating a manner of division of the sub-samples;

s320: and de-encapsulating the haptic media file according to the mark field to obtain a haptic media bit stream carried in the sub-sample.

Haptic (Haptics) refers to the sensory experience that the human body obtains by contact, such as vibration, pressure, temperature, etc. Haptic media signals (Haptics signals) are used to represent a particular modality of haptic experience, rendering the presented signals on a particular device.

A bit stream (Bitstream), otherwise referred to as a code stream, refers to a binary data stream after compression encoding. A Track (Track) is a collection of media data in the media file encapsulation process, consisting of a plurality of time-sequential samples. A media file may consist of one or more tracks, such as is common: a media file may contain a video media track, an audio media track, and a subtitle media track. In particular, the metadata information may also be included in the file as a media type in the form of metadata media tracks.

Samples (samples) are packaging units in the media file packaging process, and a track is composed of a plurality of samples, and each Sample corresponds to specific timestamp information, for example: a video media track may be made up of a number of samples, typically one video frame. In embodiments of the present application, one of the haptic media tracks may be one or more haptic signals.

Subsamples (subsamples) are a portion of the data of a sample, and a sample may be partitioned into one or more subsamples in different partitions.

The method for decapsulating the haptic media file defines a new method for representing the haptic media content, can flexibly divide sub-samples when decapsulating or encapsulating the haptic media file, supports encapsulation of the sub-samples based on different division modes, can realize flexible presentation of the haptic media, and improves the flexibility of processing haptic media data, thereby supporting flexible decoding and rendering of the haptic media by a decoder.

FIG. 4 illustrates a data structure of an exchange format file for representing haptic media content in one embodiment of the present application. As shown in fig. 4, in one exchange format file, metadata, haptic objects avatar, and haptic Perception permission may be included. Wherein the haptic object avatar represents a user object experiencing a haptic effect. Haptic Perception preference represents a variety of different haptic types, such as different types of vibrotactile, kinematic haptic, or electrotactile, among others.

For haptic media signals of the same haptic sensation type, it may be represented as different haptic channels HapticChannel depending on the sensor or rendering device it is ultimately presented to. On this basis, the media signals presented on the same haptic Device can be represented as N different haptic channels. At this time, the N different haptic channels need to be mixed into one haptic media signal before being presented.

The haptic data for a single haptic channel may be contained in a set of haptic bands HapticBand defined by its frequency range, which in turn may be further broken down into haptic effects HapticEffect and Key frames.

FIG. 5 illustrates a data structure of haptic media described in one implementation of an embodiment of the present application. As shown in FIG. 5, the standard document ISO/IEC 23090-31 describes a haptic Experience consisting of perceived persistence containing Channel channels, which in turn contain Band bands.

Experience, perception, channel and frequency bands also contain metadata that is not time dependent.

The perception may additionally contain an effect library Effectlibrary comprising haptic effect definitions. The frequency band may reference haptic effects in its library of effects that contain sensations.

The frequency band also contains haptic effect definitions or references to effects in a library of effects containing sensations that will be presented at a particular time, possibly after an external event.

FIG. 6 illustrates a data structure of a data stream of haptic media described in one implementation of an embodiment of the present application. As shown in FIG. 6, the standard document ISO/IEC 23090-31 describes the MIHS format for streaming haptic media. The MIHS data stream is composed of MIHS units. Each MIHS unit consists of MIHS packets, with the boxes with broken lines representing optional packets.

The MIHS unit can include a variety of types, such as: initialization unit Initialization Unit, time Unit TemporalUnit, space Unit and silence Unit.

The MIHS data packet can also include multiple types, such as: time data Packet TimingPacket, experience metadata Packet ExpientienceMetadataPacket, perceptional metadata Packet PercentionMetadataPacket, channel metadata Packet ChannelMetadataPacket, band metadata Packet Packet, effect library data Packet Effect library Packet, and payload carrying data Packet DataPacket.

The MIHS flow starts from an initialization unit. The initialization unit contains time data packets providing a time stamp and a time scale. The initialization unit may also contain data packages with metadata for haptic experiences, sensations, channels and bands, as well as effect library data packages.

The initialization unit is followed by one or more time units or space units. Each time cell or space cell contains one or more data packets containing haptic effect data.

The time unit contains a synchronization flag indicating whether the time unit is a synchronization unit. The data packets in the synchronization unit are independent of the data in the previous data packets.

The silence units may appear in the MIHS stream at any time to establish a haptic silence period.

An initialization unit may appear in the MIHS stream from time to update timing information or to provide additional metadata or effect library information.

The metadata and effect library packets in the first initialization unit of the MIHS stream are used as decoder configuration information for the MIHS track before any time unit.

The initialization unit, the spatial unit and the silence unit are by definition synchronization units.

FIG. 7 illustrates a schematic diagram of time cells versus haptic samples described in one implementation of an embodiment of the present application. As shown in fig. 7, the haptic data of the channel band is stored in one or more time units. The data packets in each time cell form an MIHS Sample.

The MIHS samples created from the time unit as a synchronization unit are synchronization samples. The MIHS samples may also include one or more spatial units. The silence unit may correspond to an MIHS sample that is generated to contain a single packet with a payload size of zero.

In one embodiment of the present application, the Sample Entry of an MIHS Sample is defined as follows.

Sample Entry Type:'mih1'

Container: Sampledescription box('stsd')

Mandatory: Yes

Quantity: One

The sample entry is used to indicate metadata information about all samples in the track. For example, in the sample entry of a video track, it will typically contain metadata information associated with the decoder initialization.

Syntax information (syncax) of the sample entry is as follows.

aligned(8)class MIHSSampleEntry()extends HapticSampleEntry('mih1'){

MIHSConfigurationBox configuration；

HapticExperienceDescriptionBox experience；

}

It follows that in the sample entry MIHSSampleEntry of MIHS samples of one haptic media track, MIHS configuration data box MIHSConfigurationBox and haptic experience description data box HapticExperiencedescriptionBox may be included.

The definition of the MIHS configuration data box is as follows.

Box Type: 'mh1C'

Container: MIHS sample entry('mih1')

Mandatory:Yes

Quantity:One

The MIHS configuration data box MIHSConfigurationBox contains the corresponding MIHS data packets of the metadata and effects library needed to decode MIHS samples in the MIHS track. Thus, the MIHS configuration data box MIHSConfigurationBox reflects the contents of one or more initialization units in the MIHS stream and can be used to initialize the decoder. Time MIHS packets can be omitted because their contents are reflected in a standard ISOBMFF data box such as the MediaHeaderBox (for time scale). The content of the time MIHS packet, if any, should be consistent with the time information in the standard ISOBMFF data box. ISOBMFF (ISO Based Media File Format) is a media file format based on the ISO standard, and ISOBMFF is an encapsulation standard for media files, most typically ISOBMFF files, i.e. MP4 files.

The configuration data packet may include a plurality of types as shown in table 1.

TABLE 1

The syntax information of the MIHS configuration data box MIHSConfigurationBox is as follows.

Semantic information (semanticas) for each field in the MIHS configuration data box is as follows.

configuration_packet_count indicates the number of packets included in the migcon configuration record.

configuration_packet_type indicates the packet type as recorded in table 1.

configuration_packet_layer indicates the scalability layer of the packet. This value should be the same as the MIHSlayer in the MIHS packet. A zero indicates that the packet should not be skipped, a value greater than zero means that the packet may be skipped. Note that this field is not used at the container level, but is used to maintain the same syntax of the packet header defined in ISO/IEC 23090-31, and decoder initialization can be done through this field.

configuration_packet_payload_size indicates the length in bytes of the following packet payload.

The configuration_packet_payload contains a packet payload formatted for the packet type according to ISO/IEC 23090-31.

The syntax information of the MIHS samples is defined as follows.

Wherein the semantic information of each field is defined as follows.

The size_flag indicates that the packet does not contain haptic data, in which case the data_packet_payload_size should be zero.

the temporal_flag indicates that the data packet contains temporal haptic data, in which case the data_packet_payload_size should be non-zero.

The spatial_flag indicates that the data packet contains spatial haptic data, in which case the data_packet_payload_size should be non-zero.

The data_packet_layer indicates whether or not data packets can be skipped for low bit rate applications. A zero indicates that a packet cannot be skipped and a value greater than zero indicates that a packet can be skipped.

data_packet_payload_size indicates the length in bytes of the following packet payload.

The data_packet_payload contains a packet payload formatted according to ISO/IEC 23090-31.

The MIHS samples contain packets belonging to a time cell. See section 4.2.3 of the standard document ISO/IEC 23090-31 for details.

The MIHS samples are externally framed and have dimensions provided by the external framing; for example, by a sample size ("stsz") data box.

The relationship between the silnt_ flag, temporal _flag and the spatial_flag is as follows:

if the silent_flag is 1, both the temporal_flag and the spatial_flag should be zero.

If the temporal_flag or the spatial_flag is 1, the silnt_flag should be zero.

Both the temporal_flag and the spatial_flag may be 1.

In one embodiment of the present application, the haptic media bitstream may be packaged in a single track or packaged as multiple tracks. Regardless of the manner of packaging, the subsample extensions described below may be employed.

Each haptic media track sample may be divided into one or more sub-samples, which are defined according to values of flag fields (flags) of sub-sample information data using sub-sample data boxes in the haptic media track. The flag field specifies the type of subsample information in this data box

In one embodiment of the present application, the at least one sub-sample is divided in a manner including dividing the sub-samples by a packet type, packetType, or dividing the sub-samples by a packet level, packetLayer. Wherein the packet type is used to represent the packet type of the packets in the haptic media bitstream and the packet level is used to represent the importance of the packets in the haptic media bitstream.

In one embodiment of the present application, when the sub-sample is divided into sub-samples according to the packet type, the data box of the sub-sample further includes a packet type field packet_type for indicating the packet type.

In one embodiment of the present application, the data packet type includes at least one of the following data types: time information, haptic experience metadata, haptic perception metadata, haptic channel metadata, haptic band metadata, haptic code stream data, knowledge effect data.

For example, in one application scenario, the value and description of the packet type field packet_type are shown in table 2.

TABLE 2

/>

In one embodiment of the present application, when the sub-samples are divided in such a manner that the sub-samples are divided according to a data hierarchy, the data box of the sub-samples further includes a packet hierarchy field packet_layer for indicating a packet hierarchy.

packet layer indicates a level of cladding corresponding to data contained in the sub-samples, the level of cladding indicating a level of importance of the haptic media data packet. The smaller the value of this field, the higher the importance of the packet.

In one embodiment of the present application, the partitioning of at least one sub-sample includes partitioning the sub-samples by haptic sensation or haptic channels, the data in one sub-sample corresponding to one haptic sensation type, one haptic sensation, or one haptic channel.

In one embodiment of the present application, when the sub-sample is divided in a manner of dividing the sub-sample according to a haptic sensation or a haptic channel, the data box of the sub-sample further includes a sensation type field for indicating a haptic sensation type corresponding to data in the sub-sample.

In one embodiment of the present application, when the sub-sample is divided in such a manner that the sub-sample is divided according to a haptic sensation or a haptic channel, the data box of the sub-sample further includes a perception flag field for indicating whether data in the sub-sample corresponds to one haptic sensation.

When the perception flag field indicates that the data in the sub-sample corresponds to one haptic sensation, the data box of the sub-sample further includes a perception identifier field for indicating an identifier of the haptic sensation to which the data in the sub-sample corresponds.

In one embodiment of the present application, when the perception flag field indicates that the data in the sub-sample corresponds to one haptic perception, the data box of the sub-sample further includes a channel flag field for indicating whether the data in the sub-sample corresponds to one haptic channel.

When the channel flag field indicates that the data in the sub-sample corresponds to one haptic channel, the data box of the sub-sample further includes a channel identifier field for indicating an identifier of the haptic channel to which the data in the sub-sample corresponds. In one application scenario, the coding related parameter field codec_specific_parameters (i.e., the flag field) in the data box subsampleinfo box of the subsamples may have the definitions as shown in table 3.

TABLE 3 Table 3

When the flag field takes a value of 0, a sub-sample based on the haptic media stream packet type is indicated. One sub-sample contains data corresponding to one packet type defined by the MIHSPackettype.

When the flag field takes a value of 1, a subsampled based on haptic perception or channel is represented. A sub-sample contains data corresponding to a particular haptic sensation type, a particular haptic sensation, or a particular haptic channel.

When the flag field takes a value of 2, a sub-sample based on the haptic media stream packet level is represented. One sub-sample contains data corresponding to one level of haptic media stream packets.

Other flag values are reserved.

Wherein the semantic information of each field is defined as follows.

The preference_mode indicates the haptic sensation type corresponding to the data contained in the sub-sample, and the value meaning of this field is shown in table 4.

TABLE 4 Table 4

Value of	Meaning of
		0	Others
1	Pressure of
		2	Acceleration of
3	Speed of speed
		4	Position of
5	Temperature (temperature)
		6	Vibrotactile sensation
7	Humidity of the water
		8	Wind speed
9	Pressure of
		10	Vibrotactile texture
11	Electrotactile sensation
		12	Rigidity of
13	Coefficient of friction
		14	User-defined time domain data
15	User-defined airspace data
		16-255	Reservation of

When the value of the permission_id_flag is 1, the fact that a tactile perception identifier corresponding to the data contained in the sub-sample exists is indicated; and when the value is 0, the haptic identifier corresponding to the data contained in the sub-sample does not exist.

When the channel_id_flag is 1, the tactile channel identifier corresponding to the data contained in the sub-sample exists; and when the value is 0, the haptic channel identifier corresponding to the data contained in the subsamples does not exist. When the channel_id_flag has a value of 1, the period_id_flag must also have a value of 1.

The preference_id indicates the haptic sensation identifier corresponding to the data contained in the sub-sample.

The channel_id indicates a haptic channel identifier corresponding to data contained in the sub-sample.

In one embodiment of the present application, the data structure of different types of haptic media files may be implemented based on the definition of other standard files.

FIG. 8 illustrates a data structure of an exchange format file describing haptic media content in one embodiment of the present application.

As shown in FIG. 8, haptic experience metadata and multiple haptic mode patterns may be included in one haptic experience. Optionally, one or more haptic Device devices may also be included in the haptic experience.

Haptic mode metadata may be included in one haptic mode, and a plurality of haptic Channel channels. Optionally, one or more knowledge events may also be included in the haptic pattern.

Haptic channel metadata and a plurality of haptic events may be included in one haptic channel.

Haptic event metadata may be included in a haptic event. Optionally, one or more haptic event Component agents may also be included in a haptic event.

Wherein the description information of each element is shown in table 5 below.

TABLE 5

In one embodiment of the present application, one haptic event corresponds to one haptic signal, and the data structure of the haptic event may include at least one of an event type field or an event parameter field.

An event type field event_type for indicating a rendering type of the haptic event, the rendering type of the haptic event including at least one of a transient event, a duration event, or a reference event; transient events are events that complete rendering at a specified time node, persistent events are events that complete rendering within a specified time period, and reference events are knowledge events that exist in an event library for reference. The event type field event_type may be a field with an enumeration enum type, where each rendering type may be represented by a string of characters. Knowledge events provided in the event library can be referenced multiple times as reference events among a plurality of different haptic events without having to repeatedly define the knowledge event's data structure among all haptic events.

An event parameter field for indicating a rendering parameter of a haptic event, the rendering parameter of the haptic event including at least one of a rendering amplitude and a rendering frequency; the rendering amplitude is the signal maximum amplitude of the rendering haptic event, and the rendering frequency is the signal reference frequency of the rendering haptic event. A plurality of event parameter fields corresponding to different rendering parameters, such as a maximum amplitude field amplitude and a base frequency field base_frequency, may be included in a data structure of one haptic event.

Based on the above haptic experience data structure, the data packet types in embodiments of the present application include at least one of the following data types: time information, haptic experience metadata, haptic pattern metadata, haptic channel metadata, haptic event metadata, haptic code stream data, knowledge event data.

In one embodiment of the present application, the dividing of the at least one sub-sample includes dividing the sub-sample by a haptic pattern or a haptic channel, the data in one sub-sample corresponding to one haptic pattern type, one haptic pattern or one haptic channel.

In one embodiment of the present application, when the division manner of the sub-samples is to divide the sub-samples according to a haptic mode or a haptic channel, the data box of the sub-samples further includes at least one of the following fields:

In one embodiment of the present application, when the sub-sample is divided in a manner of dividing the sub-sample by a haptic mode or a haptic channel, the data box of the sub-sample further includes a mode type field for indicating a haptic mode type corresponding to data in the sub-sample.

In one embodiment of the present application, when the sub-sample is divided in a manner of dividing the sub-sample by a haptic mode or a haptic channel, the data box of the sub-sample further includes a mode flag field for indicating whether data in the sub-sample corresponds to one haptic mode.

When the mode flag field indicates that the data in the sub-sample corresponds to one haptic mode, the data box of the sub-sample further includes a mode identifier field for indicating an identifier of the haptic mode to which the data in the sub-sample corresponds.

In one embodiment of the present application, when the mode flag field indicates that the data in the sub-sample corresponds to one haptic mode, the data box of the sub-sample further includes a channel flag field for indicating whether the data in the sub-sample corresponds to one haptic channel.

When the channel flag field indicates that the data in the sub-sample corresponds to one haptic channel, the data box of the sub-sample further includes a channel identifier field for indicating an identifier of the haptic channel to which the data in the sub-sample corresponds.

In one application scenario, the flag field in the data box of the subsamples may have the definition as shown in table 6.

TABLE 6

The pattern_type indicates the type of the haptic mode corresponding to the data contained in the sub-sample, and the meaning of the value of this field can be referred to table 4 above.

When the pattern_id_flag is 1, the haptic mode identifier corresponding to the data contained in the sub-sample exists; a value of 0 indicates that there is no haptic mode identifier corresponding to the data contained in the sub-sample.

When the channel_id_flag is 1, the tactile channel identifier corresponding to the data contained in the sub-sample exists; and when the value is 0, the haptic channel identifier corresponding to the data contained in the subsamples does not exist. When the channel_id_flag has a value of 1, the pattern_id_flag must also have a value of 1.

pattern_id indicates a haptic pattern identifier corresponding to data contained in the sub-sample.

packet_type indicates the packet type corresponding to the data contained in the sub-sample, and the value of this field has the meaning shown in table 7 below.

TABLE 7

As shown in fig. 9, the samples of each haptic media track may be divided into N sub-samples, the manner of division of the sub-samples then following the definitions provided in the various embodiments above. It should be noted that multiple partitioning of the subsamples is possible in accordance with the prior art specifications. That is, there may be a subsampleinformation box in the haptic media track with a value of 0 for describing the subsampleinformation conforming to the first division; there may also be a subsampleinformation box with a flag field value of 1 for describing subsampleinformation conforming to the second division.

According to the sub-sample dividing mode provided by the embodiment of the application, the client decoder can select the sub-sample of the corresponding tactile media track according to the related metadata information of the sub-sample in the tactile media file and combining with the equipment capability, consumption preference and the like of the user to unpack and decode, so that the exchange format of the tactile media signals is obtained and rendered.

FIG. 10 illustrates a flow chart of a method of encapsulating a haptic media file on the encoding side in one embodiment of the present application, which may be performed by a client or server as illustrated in FIG. 2, which is illustrated by way of example in the embodiment of the present application. As shown in fig. 10, the encapsulation method of the haptic media file may include the following steps S1010 to S1020.

S1010: a haptic media bitstream to be packaged is obtained.

S1020: encapsulating the haptic media bitstream to obtain a haptic media file, the haptic media file including one or more haptic media tracks for encapsulating the haptic media bitstream, at least one of the haptic media tracks including one or more sub-samples, a data box of the sub-samples including a flag field for indicating a division manner of the sub-samples.

In this embodiment of the present application, the method for encapsulating the haptic media file performed on the encoding side and the method for decapsulating the haptic media file performed on the decoding side have similar embodiments, so specific implementation details of the method for encapsulating the haptic media file may be referred to the description of each embodiment above, and will not be repeated here.

It should be noted that although the steps of the methods in the present application are depicted in the accompanying drawings in a particular order, this does not require or imply that the steps must be performed in that particular order, or that all illustrated steps be performed, to achieve desirable results. Additionally or alternatively, certain steps may be omitted, multiple steps combined into one step to perform, and/or one step decomposed into multiple steps to perform, etc.

The following describes apparatus embodiments of the present application that may be used to perform the methods of the above-described embodiments of the present application.

Fig. 11 schematically illustrates a block diagram of a device for decapsulating haptic media files provided by an embodiment of the application. As shown in fig. 11, a decapsulation device 1100 for a haptic media file includes:

a file acquisition module 1110 configured to acquire a haptic media file comprising one or more haptic media tracks for encapsulating a haptic media bitstream, at least one of the haptic media tracks comprising one or more sub-samples, a data box of the sub-samples comprising a flag field for indicating a manner of division of the sub-samples;

and a decapsulation module 1120, configured to decapsulate the haptic media file according to the flag field to obtain a haptic media bitstream carried in the sub-sample.

Fig. 12 schematically illustrates a block diagram of a device for encapsulating a haptic media file provided in an embodiment of the present application. As shown in fig. 12, the encapsulation device 1200 of the haptic media file includes:

a bitstream acquisition module 1210 configured to acquire a haptic media bitstream to be packaged;

an encapsulation module 1220 configured to encapsulate the haptic media bitstream to obtain a haptic media file, the haptic media file including one or more haptic media tracks for encapsulating the haptic media bitstream, at least one of the haptic media tracks including one or more sub-samples, a data box of the sub-samples including a flag field for indicating a division manner of the sub-samples.

In some embodiments of the present application, based on the above technical solutions, the dividing of at least one of the sub-samples includes dividing the sub-samples by a packet type or by a packet hierarchy, where the packet type is used to represent a data type of a packet in a haptic media bitstream, and the packet hierarchy is used to represent a importance level of the packet in the haptic media bitstream.

In some embodiments of the present application, based on the above technical solution, when the division manner of the sub-samples is to divide the sub-samples according to the packet type, the data box of the sub-samples further includes a packet type field for indicating the packet type; when the sub-samples are divided in a manner of dividing the sub-samples according to a data hierarchy, the data box of the sub-samples further includes a data cladding level field for indicating the data packet hierarchy.

In some embodiments of the present application, based on the above technical solutions, the packet type includes at least one of the following data types: time information, haptic experience metadata, haptic perception metadata, haptic channel metadata, haptic band metadata, haptic code stream data, knowledge effect data.

In some embodiments of the present application, based on the above technical solutions, the packet type includes at least one of the following data types: time information, haptic experience metadata, haptic pattern metadata, haptic channel metadata, haptic event metadata, haptic code stream data, knowledge event data.

In some embodiments of the present application, based on the above technical solution, the dividing of at least one of the sub-samples includes dividing the sub-samples by a haptic sensation or a haptic channel, and the data in one of the sub-samples corresponds to one of the haptic sensation types, one of the haptic sensations, or one of the haptic channels.

In some embodiments of the present application, based on the above technical solution, when the sub-sample is divided into sub-samples according to a haptic sensation or a haptic channel, the data box of the sub-sample further includes at least one of the following fields:

a perception type field, configured to indicate a haptic perception type corresponding to data in the sub-sample;

a perception flag field for indicating whether the data in the sub-sample corresponds to a haptic perception;

a channel flag field for indicating whether the data in the subsamples corresponds to a haptic channel.

In some embodiments of the present application, based on the above technical solution, when the perception flag field indicates that the data in the sub-sample corresponds to one haptic sensation, the data box of the sub-sample further includes a perception identifier field for indicating an identifier of the haptic sensation to which the data in the sub-sample corresponds; when the channel flag field indicates that the data in the sub-sample corresponds to one haptic channel, the data box of the sub-sample further includes a channel identifier field for indicating an identifier of the haptic channel to which the data in the sub-sample corresponds.

In some embodiments of the present application, based on the above technical solution, the dividing of at least one of the sub-samples includes dividing the sub-samples by a haptic mode or a haptic channel, and the data in one of the sub-samples corresponds to one of the haptic mode types, one of the haptic modes, or one of the haptic channels.

In some embodiments of the present application, based on the above technical solution, when the sub-sample is divided into sub-samples according to a haptic mode or a haptic channel, the data box of the sub-sample further includes at least one of the following fields:

A pattern type field for indicating a haptic pattern type corresponding to the data in the sub-sample;

a mode flag field for indicating whether the data in the sub-sample corresponds to a haptic mode;

In some embodiments of the present application, based on the above technical solution, when the mode flag field indicates that the data in the sub-sample corresponds to one haptic mode, the data box of the sub-sample further includes a mode identifier field for indicating an identifier of the haptic mode to which the data in the sub-sample corresponds; when the channel flag field indicates that the data in the sub-sample corresponds to one haptic channel, the data box of the sub-sample further includes a channel identifier field for indicating an identifier of the haptic channel to which the data in the sub-sample corresponds.

Specific details of each device provided in each embodiment of the present application have been described in the corresponding method embodiments, and are not described herein.

Fig. 13 schematically shows a block diagram of a computer system for implementing an electronic device according to an embodiment of the present application.

It should be noted that, the computer system 1300 of the electronic device shown in fig. 13 is only an example, and should not impose any limitation on the functions and the application scope of the embodiments of the present application.

As shown in fig. 13, the computer system 1300 includes a central processing unit 1301 (Central Processing Unit, CPU) which can execute various appropriate actions and processes according to a program stored in a Read-Only Memory 1302 (ROM) or a program loaded from a storage portion 1308 into a random access Memory 1303 (Random Access Memory, RAM). In the random access memory 1303, various programs and data necessary for the system operation are also stored. The cpu 1301, the rom 1302, and the ram 1303 are connected to each other via a bus 1304. An Input/Output interface 1305 (i.e., an I/O interface) is also connected to bus 1304.

The following components are connected to the input/output interface 1305: an input section 1306 including a keyboard, a mouse, and the like; an output portion 1307 including a Cathode Ray Tube (CRT), a liquid crystal display (Liquid Crystal Display, LCD), and the like, a speaker, and the like; a storage portion 1308 including a hard disk or the like; and a communication section 1309 including a network interface card such as a local area network card, a modem, or the like. The communication section 1309 performs a communication process via a network such as the internet. The drive 1310 is also connected to the input/output interface 1305 as needed. Removable media 1311, such as magnetic disks, optical disks, magneto-optical disks, semiconductor memory, and the like, is installed as needed on drive 1310 so that a computer program read therefrom is installed as needed into storage portion 1308.

In particular, according to embodiments of the present application, the processes described in the various method flowcharts may be implemented as computer software programs. For example, embodiments of the present application include a computer program product comprising a computer program embodied on a computer readable medium, the computer program comprising program code for performing the method shown in the flowcharts. In such embodiments, the computer program may be downloaded and installed from a network via the communication portion 1309 and/or installed from the removable medium 1311. The computer programs, when executed by the central processor 1301, perform the various functions defined in the system of the present application.

It should be noted that, the computer readable medium shown in the embodiments of the present application may be a computer readable signal medium or a computer readable storage medium, or any combination of the two. The computer readable storage medium can be, for example, but not limited to, an electronic, magnetic, optical, electromagnetic, infrared, or semiconductor system, apparatus, or device, or a combination of any of the foregoing. More specific examples of the computer-readable storage medium may include, but are not limited to: an electrical connection having one or more wires, a portable computer diskette, a hard disk, a Random Access Memory (RAM), a read-Only Memory (ROM), an erasable programmable read-Only Memory (Erasable Programmable Read Only Memory, EPROM), flash Memory, an optical fiber, a portable compact disc read-Only Memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the foregoing. In the context of this document, a computer readable storage medium may be any tangible medium that can contain, or store a program for use by or in connection with an instruction execution system, apparatus, or device. In the present application, however, a computer-readable signal medium may include a data signal that propagates in baseband or as part of a carrier wave, with the computer-readable program code embodied therein. Such a propagated data signal may take any of a variety of forms, including, but not limited to, electro-magnetic, optical, or any suitable combination of the foregoing. A computer readable signal medium may also be any computer readable medium that is not a computer readable storage medium and that can communicate, propagate, or transport a program for use by or in connection with an instruction execution system, apparatus, or device. Program code embodied on a computer readable medium may be transmitted using any appropriate medium, including but not limited to: wireless, wired, etc., or any suitable combination of the foregoing.

The flowcharts and block diagrams in the figures illustrate the architecture, functionality, and operation of possible implementations of systems, methods and computer program products according to various embodiments of the present application. In this regard, each block in the flowchart or block diagrams may represent a module, segment, or portion of code, which comprises one or more executable instructions for implementing the specified logical function(s). It should also be noted that, in some alternative implementations, the functions noted in the block may occur out of the order noted in the figures. For example, two blocks shown in succession may, in fact, be executed substantially concurrently, or the blocks may sometimes be executed in the reverse order, depending upon the functionality involved. It will also be noted that each block of the block diagrams or flowchart illustration, and combinations of blocks in the block diagrams or flowchart illustration, can be implemented by special purpose hardware-based systems which perform the specified functions or acts, or combinations of special purpose hardware and computer instructions.

It should be noted that although in the above detailed description several modules or units of a device for action execution are mentioned, such a division is not mandatory. Indeed, the features and functions of two or more modules or units described above may be embodied in one module or unit, in accordance with embodiments of the present application. Conversely, the features and functions of one module or unit described above may be further divided into a plurality of modules or units to be embodied.

From the above description of embodiments, those skilled in the art will readily appreciate that the example embodiments described herein may be implemented in software, or may be implemented in software in combination with the necessary hardware. Thus, the technical solution according to the embodiments of the present application may be embodied in the form of a software product, which may be stored in a non-volatile storage medium (may be a CD-ROM, a usb disk, a mobile hard disk, etc.) or on a network, and includes several instructions to cause a computing device (may be a personal computer, a server, a touch terminal, or a network device, etc.) to perform the method according to the embodiments of the present application.

Other embodiments of the present application will be apparent to those skilled in the art from consideration of the specification and practice of the invention disclosed herein. This application is intended to cover any variations, uses, or adaptations of the application following, in general, the principles of the application and including such departures from the present disclosure as come within known or customary practice within the art to which the application pertains.

It is to be understood that the present application is not limited to the precise arrangements and instrumentalities shown in the drawings, which have been described above, and that various modifications and changes may be effected without departing from the scope thereof. The scope of the application is limited only by the appended claims.

Claims

1. A method of decapsulating a haptic media file, comprising:

2. A method of decapsulating a haptic media file according to claim 1, wherein the division of at least one of the sub-samples comprises dividing the sub-samples by a packet type representing a data type of a packet in a haptic media bitstream or by a packet level representing a importance level of the packet in the haptic media bitstream.

3. A method of decapsulating a haptic media file as defined in claim 2, wherein when the sub-sample is divided into sub-samples by packet type, the data box of the sub-sample further comprises a packet type field for indicating the packet type; when the sub-samples are divided in a manner of dividing the sub-samples according to a data hierarchy, the data box of the sub-samples further includes a data cladding level field for indicating the data packet hierarchy.

4. A method of decapsulating a haptic media file as defined in claim 3, wherein the packet type comprises at least one of the following data types: time information, haptic experience metadata, haptic perception metadata, haptic channel metadata, haptic band metadata, haptic code stream data, knowledge effect data.

5. A method of decapsulating a haptic media file as defined in claim 3, wherein the packet type comprises at least one of the following data types: time information, haptic experience metadata, haptic pattern metadata, haptic channel metadata, haptic event metadata, haptic code stream data, knowledge event data.

6. A method of unpacking a haptic media file as recited in claim 1 wherein at least one of said sub-samples is partitioned in a manner that includes partitioning the sub-samples by haptic sensation or haptic channels, data in one of said sub-samples corresponding to one of a haptic sensation type, a haptic sensation, or a haptic channel.

7. A method of decapsulating a tactile media file according to claim 6, wherein when the sub-sample is divided in a manner of dividing the sub-sample by a tactile sense or a tactile channel, the data box of the sub-sample further comprises a sense type field for indicating a tactile sense type corresponding to data in the sub-sample.

8. A method of decapsulating a tactile media file according to claim 7, wherein when the division of the sub-samples is to divide the sub-samples by tactile sense or tactile channel, the data box of the sub-samples further comprises a sense flag field for indicating whether the data in the sub-samples corresponds to one tactile sense;

9. A method of decapsulating a tactile media file according to claim 8, wherein when said perception flag field indicates that the data in said sub-sample corresponds to a tactile perception, the data box of said sub-sample further comprises a channel flag field for indicating whether the data in said sub-sample corresponds to a tactile channel;

10. A method of unpacking a haptic media file as recited in claim 1 wherein said at least one sub-sample is partitioned in a manner that includes partitioning sub-samples by haptic mode or haptic channel, data in one of said sub-samples corresponding to one of a haptic mode type, a haptic mode, or a haptic channel.

11. The method of claim 10, wherein when the sub-sample is divided in a manner of dividing the sub-sample by a haptic mode or a haptic channel, the data box of the sub-sample further comprises a mode type field for indicating a haptic mode type corresponding to data in the sub-sample.

12. A method of decapsulating a tactile media file according to claim 11, wherein when the division of the sub-samples is to divide the sub-samples by tactile pattern or tactile channel, the data box of the sub-samples further comprises a pattern flag field for indicating whether the data in the sub-samples corresponds to one tactile pattern;

13. A method of decapsulating a tactile media file according to claim 12, wherein when said mode flag field indicates that the data in said sub-sample corresponds to a tactile mode, said sub-sample's data box further comprises a channel flag field for indicating whether the data in said sub-sample corresponds to a tactile channel;

14. A method of packaging a haptic media file, comprising:

obtaining a haptic media bitstream to be packaged;

15. A device for decapsulating a haptic media file, comprising:

16. A device for packaging a haptic media file, comprising:

an encapsulation module configured to encapsulate the haptic media bitstream to obtain a haptic media file, the haptic media file including one or more haptic media tracks for encapsulating the haptic media bitstream, at least one of the haptic media tracks including one or more sub-samples, a data box of the sub-samples including a flag field for indicating a division of the sub-samples.

17. A computer readable medium, characterized in that it has stored thereon a computer program which, when executed by a processor, implements the method of any of claims 1 to 14.

18. An electronic device, comprising:

a processor; and

a memory for storing executable instructions of the processor;

wherein the processor is configured to execute the executable instructions to implement the method of any one of claims 1 to 14.

19. A computer program product comprising a computer program, characterized in that the computer program, when executed by a processor, implements the method of any one of claims 1 to 14.