WO2025195877A1

WO2025195877A1 - Encapsulating haptics into metadata integrated into coded media

Info

Publication number: WO2025195877A1
Application number: PCT/EP2025/056802
Authority: WO
Inventors: Philippe Guillotel; Quentin GALVANE; Gurvan LECUYER; João Pedro COVA REGATEIRO
Original assignee: InterDigital CE Patent Holdings SAS
Current assignee: InterDigital CE Patent Holdings SAS
Priority date: 2024-03-20
Filing date: 2025-03-12
Publication date: 2025-09-25
Anticipated expiration: 2026-09-20

Abstract

Apparatuses and methods are disclosed for integrating haptic stimuli into coded media. Techniques disclosed include obtaining haptic data defining one or more haptic stimuli to be rendered with the rendering of a media element of the coded media, then, encapsulating the haptic data into a metadata message and inserting the metadata message into the coded media at a location relative to the media element. Apparatuses and methods are also disclosed for rendering haptic stimuli integrated into coded media. Techniques disclosed include retrieving a metadata message from the coded media at a location relative to a media element of the coded media, extracting haptic data encapsulated into the metadata message, and then rendering one or more haptic stimuli, defined by the haptic data, with the rendering of the media element.

Description

ENCAPSULATING HAPTICS INTO METADATA INTEGRATED INTO CODED MEDIA

CROSS REFERENCE TO RELATED APPLICATIONS

[1] This application claims the benefit of European Application No. 24305419.4, filed on March 20, 2024, which is incorporated herein by reference in its entirety.

BACKGROUND

[2] The increasing application of haptics (tactile sensations) for the enhancement of visual and auditory immersive experiences motivates ongoing efforts made by MPEG to standardize a haptic data coding format - thereby, providing content creators and designers of rendering devices the needed cross-compatibility and interoperability. To enable synchronization of haptics with compressed media content (e.g., video and/or audio), these ongoing efforts by MPEG to develop a haptic data coding format are also aimed at facilitating the incorporation of standardized haptic data coding format into standardized system-layer-mechanisms, such as ISO base media file format (ISOBMFF), MPEG dynamic adaptive streaming over HTTP (MPEG-DASH), and immersive Scene Description (MPEG-I SD). Synchronizing haptics with compressed media content using such system-layer-mechanisms may not be efficient and may affect the bitrate distribution of the compressed media content.

SUMMARY

[3] Aspects disclosed in the present disclosure describe methods for integrating haptic stimuli into coded media. The methods comprise obtaining haptic data defining one or more haptic stimuli to be rendered with the rendering of a media element of the coded media, then, encapsulating the haptic data into a metadata message and inserting the metadata message into the coded media at a location relative to the media element. Aspects disclosed in the present disclosure also describe methods for rendering haptic stimuli that are integrated into coded media. The methods comprise retrieving a metadata message from the coded media at a location relative to a media element of the coded media, extracting haptic data encapsulated into the metadata message, and then rendering one or more haptic stimuli, defined by the haptic data, with the rendering of the media element.

[4] Aspects disclosed in the present disclosure describe apparatuses for integrating haptic stimuli into coded media. The apparatuses comprise at least one processor and memory storing instructions. The instructions, when executed by the at least one processor, cause the apparatuses to obtain haptic data defining one or more haptic stimuli to be rendered with the rendering of a media element of the coded media, to encapsulate the haptic data into a metadata message, and then to insert the metadata message into the coded media at a location relative to the media element. Further aspects disclosed in the present disclosure describe apparatuses for rendering haptic stimuli that are integrated into coded media. The apparatuses comprise at least one processor and memory storing instructions. The instructions, when executed by the at least one processor, cause the apparatuses to retrieve a metadata message from the coded media at a location relative to a media element of the coded media, to extract haptic data encapsulated into the metadata message, and then to render one or more haptic stimuli, defined by the haptic data, with the rendering of the media element.

[5] Aspects disclosed in the present disclosure describe a non-transitory computer-readable medium comprising instructions executable by at least one processor to perform methods for integrating haptic stimuli into coded media. The methods comprise obtaining haptic data defining one or more haptic stimuli to be rendered with the rendering of a media element of the coded media, then, encapsulating the haptic data into a metadata message and inserting the metadata message into the coded media at a location relative to the media element. Aspects disclosed in the present disclosure also describe a non-transitory computer-readable medium comprising instructions executable by at least one processor to perform methods for rendering haptic stimuli that are integrated into coded media. The methods comprise retrieving a metadata message from the coded media at a location relative to a media element of the coded media, extracting haptic data encapsulated into the metadata message, and rendering one or more haptic stimuli, defined by the haptic data, with the rendering of the media element.

[6] This Summary is provided to introduce a selection of concepts in a simplified form that is further described below in the Detailed Description. This Summary is not intended to identify key features or essential features of the claimed subject matter, nor is it intended to be used to limit the scope of the claimed subject matter. Furthermore, the claimed subject matter is not limited to limitations that solve any or all disadvantages noted in any part of this disclosure.

BRIEF DESCRIPTION OF THE DRAWINGS

[7] FIG. 1 is a block diagram of an example system, according to aspects of the present disclosure.

[8] FIG. 2 is a block diagram illustrating a haptic data model, as specified by the MPEG haptic data coding format, according to aspects of the present disclosure.

[9] FIG. 3 is a diagram illustrating a haptic signal, according to aspects of the present disclosure.

[10] FIG. 4 is a block diagram illustrating a haptic encoder, according to aspects of the present disclosure.

[11] FIG. 5 is a block diagram illustrating an encoder, a decoder, and a Tenderer of media augmented by haptic data, according to aspects of the present disclosure.

[12] FIG. 6 is a flow diagram illustrating an example method for integrating haptic stimuli into coded media, according to aspects of the present disclosure.

[13] FIG. 7 is a flow diagram illustrating an example method for rendering haptic stimuli integrated into coded media, according to aspects of the present disclosure.

DETAILED DESCRIPTION

[14] Apparatuses and methods are presented herein for integrating haptic stimuli into coded media and for rendering haptic stimuli integrated into coded media. A system for processing and displaying content, with which various aspects and examples described herein may be implemented, is generally described in reference to FIG. 1, followed by a description of the aspects of the present disclosure in reference to FIGS. 2-7.

[15] FIG. 1 illustrates a block diagram of an example system 100. System 100 can be embodied as a device including the various components described below and can be configured to perform one or more of the aspects described in this application. Examples of such devices, include, but are not limited to, various electronic devices such as personal computers, laptop computers, smartphones, tablet computers, digital multimedia set-top boxes, digital television receivers, personal video recording systems, connected home appliances, and servers. Elements of system 100, singly or in combination, can be embodied in a single integrated circuit, multiple integrated circuits, and/or discrete components. For example, in at least one embodiment, the processing and encoder/decoder elements of system 100 are distributed across multiple integrated circuits and/or discrete components. In various embodiments, the system 100 is communicatively coupled to other systems, or to other electronic devices, via, for example, a communications bus or through dedicated input and/or output ports. In various embodiments, the system 100 is configured to implement one or more of the aspects described in this application.

[16] The system 100 includes at least one processor 110 that can be configured to execute instructions loaded therein for implementing, for example, the various aspects described in this application. Processor 110 can include embedded memory, input and output interfaces, and various other circuitries as known in the art. The system 100 includes at least one memory 120 (e.g., a volatile memory device and/or a non-volatile memory device). System 100 includes a storage device 140, which can include non-volatile memory and/or volatile memory, including, for example, EEPROM, ROM, PROM, RAM, DRAM, SRAM, flash, magnetic disk drives, and/or optical disk drives. The storage device 140 can be an internal storage device, an attached storage device, and/or a network accessible storage device, for example.

[17] System 100 includes an encoder/decoder module 130 configured to process data to provide encoded video data or decoded video data. The encoder/decoder module 130 can include its own processor and memory. The encoder/decoder module 130 represents module(s) that can be included in a device to perform encoding and/or decoding functions. Additionally, the encoder/decoder module 130 can be implemented as a separate element of system 100 or can be incorporated within processor 110 as a combination of hardware and software as known to those skilled in the art.

[18] Program code that is to be loaded into processor 110 or into encoder/decoder 130 to perform the various aspects described in this application can be stored in a storage device 140 and subsequently loaded into memory 120 for execution by processor 110. In accordance with various embodiments, one or more of processor 110, memory 120, storage device 140, and encoder/decoder module 130 can store one or more of various items during the performance of the processes described in this application. Such stored items can include, but are not limited to, the input video, the decoded video or portions of the decoded video, the bitstream, matrices, variables, and intermediate or final results from the processing of equations, formulas, operations, and operational logic.

[19] In several embodiments, memory inside of the processor 110 and/or the encoder/decoder module 130 is used to store instructions and to provide working memory for processing functions that are needed during encoding or decoding. In other embodiments, however, memory external to the processing device (where, for example, the processing device can be either the processor 110 or the encoder/decoder module 130) can be used for one or more of these functions. The external memory can be the memory 120 and/or the storage device 140 that may comprise, for example, a dynamic volatile memory and/or a non-volatile flash memory. In several embodiments, an external non-volatile flash memory is used to store the operating system of a television. In at least one embodiment, a fast external dynamic volatile memory such as a RAM is used as working memory for video coding and decoding operations. [20] The input to the elements of system 100 can be provided through various input devices as indicated in block 105. Such input devices include, but are not limited to, (i) an RF portion that receives an RF signal transmitted, for example, over the air by a broadcaster, (ii) a Composite input terminal (COMP), (iii) a USB input terminal, and/or (iv) an HDMI input terminal.

[21] In various embodiments, the input devices of block 105 have associated respective input processing elements as known in the art. For example, the RF portion can be associated with elements suitable for (i) selecting a desired frequency (also referred to as selecting a signal, or band-limiting a signal to a band of frequencies), (ii) down-converting the selected signal, (iii) band-limiting again to a narrower band of frequencies to select, for example, a signal frequency band which can be referred to as a channel in certain embodiments, (iv) demodulating the down converted and band-limited signal, (v) performing error correction, and (vi) demultiplexing to select the desired stream of data packets. The RF portion of various embodiments includes one or more elements that perform these functions, for example, frequency selectors, signal selectors, band-limiters, channel selectors, filters, downconverters, demodulators, error correctors, and demultiplexers. The RF portion can include a tuner that performs some of these functions, including, for example, down-converting the received signal to a lower frequency (for example, an intermediate frequency or a near-baseband frequency) or to a baseband. In one set-top box embodiment, the RF portion and its associated input processing element receive an RF signal transmitted over a wired (for example, cable) medium, and perform frequency selection by filtering, down-converting, and filtering again to a desired frequency band. Various embodiments rearrange the order of the above-described (and other) elements, remove some of these elements, and/or add other elements performing similar or different functions. Added elements can include inserting elements in between existing elements, for example, inserting amplifiers and an analog-to-digital converter. In various embodiments, the RF portion includes an antenna.

[22] Additionally, the USB and/or HDMI terminals can include respective interface processors for connecting system 100 to other electronic devices across USB and/or HDMI connections. It is to be understood that various aspects of input processing, for example, Reed- Solomon error correction, can be implemented, for example, within a separate input processing integrated circuit or within processor 110 as necessary. Similarly, aspects of USB or HDMI interface processing can be implemented within separate interface integrated circuits or within processor 110 as necessary. The demodulated, error corrected, and demultiplexed stream is provided to various processing elements, including, for example, processor 110, and encoder/decoder 130 operating in combination with the memory and storage elements to process the datastream as necessary for presentation on an output device.

[23] Various elements of system 100 can be provided within an integrated housing. Within the integrated housing, the various elements can be interconnected and transmit data therebetween using a suitable connection arrangement 115, for example, an internal bus as known in the art, including the I2C bus, wiring, and printed circuit boards.

[24] The system 100 includes a communication interface 150 that enables communication with other devices via communication channel 190. The communication interface 150 can include, but is not limited to, a transceiver configured to transmit and to receive data over communication channel 190. The communication interface 150 can include, but is not limited to, a modem or network card. The communication channel 190 can be implemented, for example, within a wired and/or a wireless medium.

[25] Data are streamed to the system 100, in various embodiments, using a Wi-Fi network such as IEEE 802.11. The Wi-Fi signal of these embodiments is received over the communication channel 190 and the communication interface 150 which are adapted for WiFi communications. The communication channel 190 of these embodiments is typically connected to an access point or router that provides access to outside networks including the Internet for allowing streaming applications and other over-the-top communications. Other embodiments provide streamed data to the system 100 using a set-top box that delivers the data over the HDMI connection of the input block 105. Still other embodiments provide streamed data to the system 100 using the RF connection of the input block 105.

[26] The system 100 can provide an output signal to various output devices, including a display device 165, an audio device (e.g., speaker(s)) 175, and other peripheral devices 185. The other peripheral devices 185 include, in various examples of embodiments, haptic devices, one or more of a stand-alone DVR, a disk player, a stereo system, a lighting system, and other devices that provide a function based on the output of the system 100. In various embodiments, control signals are communicated between the system 100 and the display device 165, the audio device 175, or other peripheral devices 185 using signaling such as AV. link, CEC, or other communication protocols that enable device-to-device control with or without user intervention. The output devices can be communicatively coupled to system 100 via dedicated connections through respective interfaces 160, 170, and 180. Alternatively, the output devices can be connected to system 100 using the communication channel 190 via the communication interface 150. The display device 165 and the audio device 175 can be integrated in a single unit with the other components of system 100 in an electronic device, for example, a television. In various embodiments, the display interface 160 includes a display driver, for example, a timing controller (T Con) chip.

[27] The display device 165 and the audio device 175 can alternatively be separate from one or more of the other components, for example, if the RF portion of input 105 is part of a separate set-top box. In various embodiments in which the display device 165 and the audio device 175 are external components, the output signal can be provided via dedicated output connections, including, for example, HDMI ports, USB ports, or COMP outputs.

[28] Rendering of haptic data, describing various tactile sensations, can enhance the immersive experience provided to users by visual and/or auditory media content. Tactile sensations can be produced by one or more haptic devices, such as actuators that are attached to (or are in touch with) a user’s body to form vibrotactile or kinaesthetic sensations. Such haptic devices operate by input haptic signals synthesized based on decoded haptic data that specify the desired vibrotactile or kinaesthetic sensations. To deliver the haptic data to the decoder and synthesizer that feed the haptic devices, the haptic data are encoded according to a known format. To be realistic, the operation of the haptic devices needs to be synchronized with the viewed and/or heard media content. To that end, means are required to signal (or decode) the haptic data in conjunction with the signaling (decoding) of the media content.

[29] Given the increasing use of haptics by content creators and designers of rendering devices, ongoing efforts are made by MPEG to standardize a haptic data coding format (as specified in ISO/IEC 23090-31, referred to herein as the MPEG haptic data coding format) - thereby, providing industry players with the needed cross-compatibility and interoperability. These ongoing efforts by MPEG to develop a haptic data coding format are also aimed at facilitating the incorporation of haptics into system-layer-mechanisms (such as ISOBMFF, MPEG-DASH, and MPEG-I SD) to allow standardized incorporation of haptic data into media content. Thus, the standardized representation of haptic data (according to the MPEG haptic data coding format) enables distribution and rendering of haptic data in conjunction with the distribution and rendering of other types of media (e.g., video and/or audio). That standardized representation of haptic data is modeled by a hierarchical data structure, namely, a haptic data model, through which information required to generate the haptic signals is defined. The haptic data model is further described in reference to FIG. 2. [30] FIG. 2 is a block diagram illustrating a haptic data model 200, as specified by the MPEG haptic data coding format. The top layer of that hierarchical data structure contains high level metadata 210, avatar(s) 212, and a list of N_P perceptions 214-218. Typically, each perception (e.g., 214) is represented by one or more haptic signals. Haptic signals, of haptic tracks 224- 228, are each designed to create a desired tactile sensation that corresponds to a haptic modality (e.g., a vibration, force, position, velocity, or temperature). Each haptic perception is also associated with a haptic avatar, that is, a body representation in reference to which the haptic signals of the perception can be spatially directed to respective locations on the user’s body, as further explained below.

[31] As shown in FIG. 2, a data structure of a perception (e.g., 214) contains metadata 220 describing information specific to the perception, devices 222, and a list of Nt tracks. Each track (e.g., 224) represents a haptic signal that can be decomposed into Nb frequency bands, each band represents a part of the signal’s energy that is within a respective frequency range (i.e., low band or high band). Thus, a track is described by a data structure, containing metadata 230 describing information specific to the track, and Nb haptic bands 232-238. A haptic band, in turn, is described by a data structure, containing band data 240 and a list of N_e haptic effects 242-248. Each effect (e.g., 242) in the list is described by a data structure, containing effect data 250 and a list of keyframes 252-258. Generally, a keyframe encodes a point (a sample) at a given position in time and/or space. Thus, the bands, their respective Effects, and the Key frames of each effect, together define a signal of a haptic track. To reconstruct the signal of the haptic track (e.g., 224), the different haptic bands 232-238 can be combined as further described in reference to FIG. 3.

[32] As mentioned above, an avatar 212 is a body representation used to indicate the location on the user’s body where haptic stimuli should be applied via a haptic signal. Each haptic perception is associated with a haptic avatar, so that haptic signals (e.g., 224-228) of a perception (e.g., 214) can be spatially directed to respective locations on the body that are determined by referencing the body representation of the avatar. Each perception may be associated with a different avatar based on the level of detail used in the avatar’s body representation and on the spatial acuity required by the haptic modality of the perception. Haptic modalities may be temporal in nature (pressure, acceleration, velocity, position, temperature, vibrotactile, water, wind, force, and electrotactile) or may be spatial in nature (vibrotactile texture, stiffness, and friction). Thus, each haptic perception (e.g., 214) can be defined by metadata 220 (e.g., identifying the haptic modality and the avatar 212 to be used), information characterizing the targeted haptic devices 222, and haptic tracks 224-228 used to generate the intended haptic stimuli of the perception.

[33] FIG. 3 is a diagram illustrating a haptic signal 300. A haptic signal (e.g., of haptic track 224) can be decomposed into multiple bands 232-238, each band represents an energy part of the haptic signal that is within a frequency range of the band. In the example of FIG. 3, the haptic signal (denoted “Track 1”) is decomposed into two frequency bands - a low frequency band (denoted “Band 1”) and a high frequency band (denoted “Band 2”). By adding the high and low frequency bands, the original signal (Track 1) can be reconstructed.

[34] Four types of haptic bands are described in the haptic data coding format: a transient band, a curve band, a vectorial wave band, and a wavelet band. As illustrated in FIG. 3, each band is composed of a series of haptic effects (e.g., defined by data elements 242-248 in FIG. 2) and each haptic effect is represented by a list of keyframes (e.g., defined by data elements 252-258 in FIG. 2). Data defining each effect and its associated keyframes are interpreted differently for each of the types of haptic bands, as follows. For a transient band, each effect stores a set of keyframes defining a position, an amplitude, and a frequency, where a keyframe represents a transient event. For a curve band (e.g., Band 1 in FIG. 3), each effect stores a set of keyframes defining a position and an amplitude, where the keyframes represent the control points of the curve. The type of interpolation (e.g., cubic or linear) used to generate the curve band is specified in the metadata of the band. For a vectorial wave band, the effect stores a set of keyframes defining aposition, an amplitude, and a frequency. For a wavelet band (e.g., Band 2 in FIG. 3), each effect stores the content of one wavelet block, including a keyframe for every quantized wavelet coefficient using only the amplitude values of the coefficients. The coefficients can be scaled to a range of [-1,1], and the original maximum amplitude as well as the maximum number of used bits can be stored in additional respective keyframes. The manner in which the haptic signals are encoded are next described in reference to FIG. 4.

[35] FIG. 4 is a block diagram illustrating a haptic encoder 400. The overall architecture of this haptic encoder 400 is specified in the MPEG haptic data coding format and generally described herein. In the example of FIG. 4, the haptic encoder 400 receives haptic data (to be encoded) that are provided by a descriptive content 402 - that is, a composition of primitives that are encoded in a human-readable (textual) format, such as JSON, the HJIF proposed by MPEG, the AHAP by Apple, or the IVS by Immersion. The haptic encoder 400 also receives haptic data (to be encoded) that are provided by waveform content 404 - a quantized haptic signal encoded in format such as WAVE (or WAV).

[36] In addition, the haptic encoder 400 receives haptic information 406 from which a metadata extractor 410 extracts metadata 412, 414. The haptic information 406 can be represented by a textual format, such as the object haptic metadata (OHM) format. The extracted metadata 412 may provide the name of the haptic files from which the descriptive content 402 and the waveform content 404 can be retrieved. The extracted metadata 414 also provide information that can be mapped to the data model 200 described in reference to FIG. 2. The encoder 400 handles the descriptive content 402 and the waveform content 404 (retrieved from haptic files indicated by the metadata 412) along two corresponding processing paths, as further described below.

[37] The descriptive content of the haptic data 402 (retrieved from file formats such as IVS, AHAP, or HJIF) is encoded using a process path including an analyzer 415, a transcoder 425, and a formatter 445. First, the encoder 400 semantically analyses, by the analyzer 415, the input file information to detect the format in which the descriptive content 402 is represented with (or extract this information from the OHM file). Then, depending on the detected format, the encoder 400 may transcode, by the transcoder 425, the descriptive content into a common format (e.g., HJIF). For example, if the identified format is HJIF, transcoding is not necessary; otherwise, if the identified format is AHAP or IVS, the descriptive content is transcoded 425 into an HJIF.

[38] The waveform content of the haptic data 404 is encoded using a process path including a decomposer 420, a keyframe coder 430, a wavelet coder 440, and the formatter 445. First, the encoder 400 decomposes the waveform content into multiple frequency bands by the decomposer 420. In the example of FIG. 4, the waveform is decomposed into a low band and a high band. Then, as illustrated, the encoder encodes the low band by a keyframe coder 430, using a keyframe representation - that is, keyframes are extracted from the low band and are used for its representation. Additionally, the keyframe coder 430 reconstructs the low band based on its keyframe representation. The reconstructed low band is then subtracted 435 from the original low band, forming a residual low band. The encoder 400 then encodes, by the wavelet coder 440, the summation of the residual low band and the high band.

[39] Next, the encoder 400 generates, by the formatter 445, the data model 200 based on the descriptive content (output of the transcoder 425 or the analyzer 415), the waveform content (outputs of the keyframe coder 430 and the wavelet coder 440), and the metadata 414. The io formatter 445 then outputs the generated data model 200 in an interchange format representation, that is, the coded haptic data 460. The coded haptic data 460 may be compressed, by a binary coder 450, into a binary representation 462 (in a binary format such as HMPG), that can be further packetized, by a packet generator 455, into a packetized bitstream 464 (defined by a streaming format such as MIHS).

[40] Hence, reversing the operations of the packet generator 455 and the binary coder 450, a haptic decoder, given a packetized stream 464, can generate the coded haptic data 460. And, from the interchange format representation of the coded haptic data 460, the data model 200 can be reconstructed. Tactile sensations can then be performed as prescribed by that reconstructed data model.

[41] To synchronize compressed media (e.g., video and/or audio) with haptic data, it is necessary to use a system layer that provides a mechanism to spatiotemporally signal the different components of the haptic data (e.g., ISOBMFF, DASH or other MPEG system layer mechanisms). In a simpler approach, as described herein, haptic data can be directly associated with the compressed media content using metadata. Various media coding formats support the integration of metadata that provide additional information. For example, metadata integrated into a video coding format can provide information regarding the coded video (e.g., the color model used) or information regarding the post-processing of the decoded video (e.g., how to crop the reconstructed images, a specific noise to be added, parameters of the camera or the encoder, time codes, closed captions, lyrics, or copyright info). In MPEG, such metadata can be provided through video usability information (VUI) messages or through generic supplemental enhancement information (SEI) messages. A complete specification has been developed in MPEG for SEI messages, called VSEI (ISO/IEC 23002-7). In other standards (such as AVI or JPEG) similar means are defined to integrate metadata into the coding format.

[42] Hence, SEI (or equivalent metadata) messages can be extended to support integration of haptic stimuli into coded media. To that end, the SEI (or the equivalent metadata) messages can contain or identify haptic data, modeled according to the MPEG haptic data coding format. Such messages can be inserted into coding formats of various media. For example, when the media is video, SEI messages can be inserted into coding formats such as AVC, HEVC, VVC, EVC, AVI or any other coding format that supports the inclusion of metadata. Generally, for a video, SEI messages can be inserted per a period, where a period can correspond to an intra period, a group of pictures (GOP), any number of pictures, or any time duration. [43] According to aspects, coded haptic data (e.g., 460, 462, or 464) can be encapsulated into an SEI (or equivalent metadata) message. Synchronization of haptic sensation to coded media can then be achieved by associating the SEI message (containing corresponding haptic data) to a coding element of the coded media. For example, synchronization of haptic sensation to a coded video can be achieved by associating the SEI message to a coding element such as a frame (e.g., identified by a frame number or a timestamp), a GOP, or any time duration (e.g., determined by a number of frames) of the coded video. In another example, synchronization of haptic sensation to a coded audio can be achieved by associating the SEI message to a time duration of the coded audio.

[44] Table 1 presents an SEI message, identifying coded haptic data 460, 462, 464. In this case, the coded haptic data are provided by a URL.

[45] Table 1: Haptic SEI message for haptic data identified by a URL

[46] The data structure of the SEI message, namely, SEI haptic url, shown in Table 1, includes a “type” data element indicating the type of a haptic file format (e.g., a JSON format (HJIF), a binary format (HMPG), or a streaming format such as MIHS) and a “url” data element providing the URL address of the haptic file to be read. The haptic file can be stored in a distant server (e.g., accessible by the prefix ftp:// or http://) or in a server local to the decoder (e.g., accessible by the prefix file://).

[47] Alternatively, or in addition, SEI messages can be used to provide haptic data that are directly integrated into the coded media content, as illustrated by Table 2. Table 2 shows an SEI message that can be inserted into coded media. The integration of the SEI message can be done by the designer or by any equipment for inserting data into an existing media stream (e.g., a slicer or a multiplexer). For example, the SEI can be inserted (e.g., on demand) relative to a coding element, at the sequence level, at the GOP level, or at a frame level.

[48] Table 2: Haptic SEI message for integrated haptic data.

[49] The data structure of the SEI message, namely, SEI haptic bin, shown in Table 2, includes a “type” data element indicating the type of a haptic data format (e.g., a JSON format (HJIF), a binary format (HMPG), or a streaming format such as MIHS) and a “data[]” buffer storing the content of the haptic data.

[50] In another aspect, instead of including in an SEI message the content of a full haptic file (potentially a large file describing, for example, one or more haptic experiences), haptics effect(s) (e.g., 242-248) can be included in one or more respective SEI messages, as illustrated in Table 3. This may be useful, for example in a video case, where a representation (e.g., in a JSON format) including one or more effects has to be associated with a certain coding element, such as a certain GOP or a certain frame of the coded video. In this manner, an SEI message can be embedded at a desired location (or time) in the media content. This approach is advantageous since, first, these haptic SEI messages are still compatible with the MPEG haptic data coding format, and, second, these haptic SEI messages have smaller data size, and, therefore, the respective data rate is distributed across time (avoiding spikes in data rate along the bitstream of the coded media). Furthermore, this approach is independent of the type of the media - it can be used to add haptic effects also to audio formats, such as MP3 or AAC audio formats.

[52] The data structure of the SEI message, namely, SEI mpeg haptic, shown in Table 3, includes a “type” data element indicates the type of a haptic data format (e.g., a JSON format (HJIF), a binary format (HMPG), or a streaming format such as MIHS) and a “data[]” buffer storing a set of haptic effects. This set of haptic effects can be employed up to the next haptic SEI message or for a limited duration of time.

[53] Since SEI messages, by design, are independent of each other, each haptic SEI message (or the haptic data therein) should be self-contained. Thus, the haptic data encapsulated into each SEI message should include all necessary information to define the haptic sensation represented by the haptic data. For example, in a case in which an SEI message is formed to represent a perception 214, the haptic data that are embedded into the SEI message should include all necessary information to define that perception (e.g., according to the hierarchical data structure of the haptic data model 200). Note that when there are no haptic effects during a period of time in a coded media, no haptic SEI messages need to be inserted into the coded media during that time. Alternatively silent packets as described in the MIHS format might be used.

[54] FIG. 5 is a block diagram illustrating an encoder, a decoder, and a Tenderer of media augmented by haptic data 500 (each of which is employable by the system 100 of FIG. 1). In the example of FIG. 5, the encoder 510 comprises a haptic encoder 515, an encapsulator 520, and a media encoder 525; the decoder 530 comprises a haptic decoder 535, an extractor 540, and a media decoder 545; and the Tenderer 550 comprises a synthesizer (a haptic Tenderer) 555, (optionally) a synchronization manager 560, and a media Tenderer 565. The haptic encoder 515 can be configured to encode haptic data (e.g., as described in reference to FIG. 4). The encapsulator 520 can be configured to encapsulate coded haptic data (e.g., the output of the haptic encoder 460, 462, or 464) into metadata message(s) (e.g., any of the SEI message(s) described in reference to Tables 1-3). And, the media encoder 525 can be configured to encode media data. The media data may be video data, audio data, or any other (multi) media content. The media encoder 525 can be further configured to insert into the coded media data the metadata message(s) provided by the encapsulator 520, generating coded media data augmented by haptic data. The media decoder 545 can be configured to decode the coded media received from the media encoder 525. The decoded media data can next be rendered by the media Tenderer 565, outputting media signal(s) readily available for playback by a player 580. The encoding 525 and the decoding 545 of the media data typically conform to the appropriate media coding format. For example, in the case of a video, the media encoder 525 and the media decoder 545 can conform to a video coding format such as AVC, HEVC, VVC, or AVI.

[55] According to various aspects, the media encoder 525 and the haptic encoder 515 can operate in real-time or offline. For example, the haptic encoder can encode the haptic data offline and the encapsulator 520 can encapsulate the encoded haptic data into metadata message(s) that are provided in real-time to the media encoder to be inserted into the coded media. [56] Concurrently with the decoding 545, the rendering 565, and the playing 580 of the media data, the media decoder 545 can be configured to retrieve the metadata message(s) (inserted into the coded media data obtained from the media encoder 525) and to send the retrieved metadata message(s) to the extractor 540. Based on the type of the metadata message(s) (e.g., see Tables 1-3), the extractor 540 makes available to the haptic decoder 535 the coded haptic data (generated by the haptic encoder 515). For example, if an SEI message is an SEI haptic url message, the extractor can retrieve the coded haptic data from a location defined by the URL provided in the message. If the SEI message is an SEI haptic bin message, the coded haptic data can be extracted from the message body itself and be provided to the haptic decoder 535. Otherwise, if the SEI message is an SEI_mpeg_haptic message the extractor can extract the set of effects stored in the message and provide it to the haptic decoder 535. Receiving the coded haptic data, the haptic decoder 535 can be configured to decode the haptic data into decoded haptic data, represented by the haptic data model 200. The decoded haptic data are then synthesized (rendered) by the synthesizer 555, generating therefrom the haptic signal(s). These haptic signal(s) drive corresponding haptic devices 570 to generate the intended tactile sensations.

[57] The operation of the synthesizer 555 should be coordinated with that of the media renderer 565 to allow haptic perceptions to be generated at or during the intended time. In an aspect, such coordination is achieved by virtue of integrating a metadata message (containing or identifying haptic data) into a location in the media (e.g., relative to a video or an audio frame) with respect to which a haptic stimulus (defined by the haptic data) should be generated. In such a case, the haptic stimulus is generated by the targeted haptic device 570 when it is received (e.g., via the media decoder 545, the extractor 540, the haptic decoder 535, and then the synthesizer 555). In another aspect, a synchronization manager 560 can be employed to more accurately synchronize the haptic stimulus with the media data. The synchronization manager 560, in a typical implementation, can use buffers - for example, for buffering a media signal outputted by the media renderer 565 and a haptic signal outputted by the synthesizer 555 - and timestamps embedded into these signals to manage their synchronization.

[58] FIG. 6 is a flow diagram illustrating an example method for integrating haptic stimuli into coded media 600, according to aspects of the present disclosure. The method 600 may be implemented by the encoder 510 of FIG. 5. The method 600 can begin, in step 610, by obtaining haptic data. The obtained haptic data define one or more haptic stimuli to be rendered with the rendering of a media element of the coded media. In step 620, the haptic data can be encapsulated into a metadata message. Then, in step 630, the metadata message can be inserted into the coded media at a location in the coded media that is relative to the location of the media element.

[59] FIG. 7 is a flow diagram illustrating an example method for rendering haptic stimuli integrated into coded media 700, according to aspects of the present disclosure. The method 700 may be implemented by the decoder 530 and the Tenderer 565 of FIG. 5. The method 700 begins, in step 710, by retrieving a metadata message from the coded media. The retrieved metadata message is located relative to a media element of the coded media. In step 720, the haptic data encapsulated into the metadata message can be extracted. Then, in step 730, one or more haptic stimuli defined by the haptic data can be rendered with the rendering of the media element. As described above, further synchronization between these haptic stimuli and the media element can be achieved by the synchronization manager 560 of FIG. 5.

[60] With respect to both methods 600, 700, the metadata message can be an SEI message. As described in reference to Tables 1-3, the metadata message can include a syntax element indicating a type of a haptic format describing the haptic data, where the haptic format can be a readable textual format (e.g., JSON), a binary format, or a streaming format. The metadata message can further include a syntax element indicating: a url address of a file containing the haptic data (see Table 1), a buffer containing the haptic data (see Table 2), or a buffer containing a set of haptic effects included in the haptic data (see Table 3). Furthermore, the coded media can be a coded video, and so, for example, the media element can be a frame, a GOP, or any time duration of the coded video. The coded media can also be a coded audio, and so, for example, the media element can be a time duration in the coded audio. As described above, the haptic data can be represented by a haptic data model 200 (see, FIG. 2), defined by the MPEG haptic data coding format.

[61] The illustrations of the aspects described herein are intended to provide a general understanding of the structure, function, and operation of the various aspects. The illustrations are not intended to serve as a complete description of all of the elements and features of apparatuses and systems that utilize the structures or methods described herein. Many other aspects may be apparent to those of skill in the art upon reviewing the disclosure. Other aspects may be utilized and derived from the disclosure, such that structural and logical substitutions and changes may be made without departing from the scope of the disclosure. Accordingly, the disclosure and the figures are to be regarded as illustrative rather than restrictive. [62] The description of the aspects is provided to enable the making or use of the aspects. Various modifications to these aspects will be readily apparent, and the generic principles defined herein may be applied to other aspects without departing from the scope of the disclosure. Thus, the present disclosure is not intended to be limited to the aspects shown herein but is to be accorded the widest scope possible consistent with the principles and novel features as defined by the following claims.

Claims

1. A method for integrating haptic stimuli into coded media, comprising: obtaining haptic data defining one or more haptic stimuli to be rendered with the rendering of a media element of the coded media; encapsulating the haptic data into a metadata message; and inserting the metadata message into the coded media at a location relative to the media element.

2. The method according to claim 1, wherein the metadata message is a supplemental enhancement information (SEI) message.

3. The method according to claim 1 or 2, wherein the metadata message comprises: a syntax element indicating a type of a haptic format describing the haptic data, wherein the type is one of a readable textual format, a binary format, or a streaming format.

4. The method according to any one of claims 1-3, wherein the metadata message further comprises: a syntax element indicating a url address of a file containing the haptic data.

5. The method according to any one of claims 1-3, wherein the metadata message further comprises: a syntax element indicating a buffer containing the haptic data.

6. The method according to any one of claims 1-3, wherein the metadata message further comprises: a syntax element indicating a buffer containing a set of haptic effects included in the haptic data.

7. The method according to any one of claims 1-6, wherein the coded media is a coded video, and the media element is a frame, a group of pictures (GOP), or a time duration of the coded video.

8. The method according to any one of claims 1-6, wherein the coded media is a coded audio, and the media element is a time duration in the coded audio.

9. The method according to any one of claims 1-8, wherein the haptic data are represented by a haptic data model defined by the MPEG haptic data coding format.

10. A method for rendering haptic stimuli integrated into coded media, comprising: retrieving a metadata message from the coded media at a location relative to a media element of the coded media; extracting haptic data encapsulated into the metadata message; and rendering one or more haptic stimuli, defined by the haptic data, with the rendering of the media element.

11. The method according to claim 10, wherein the metadata message is a supplemental enhancement information (SEI).

12. The method according to claim 10 or 11, wherein the metadata message comprises: a syntax element indicating a type of a haptic format describing the haptic data, wherein the type is one of a readable textual format, a binary format, or a streaming format.

13. The method according to any one of claims 10-12, wherein the metadata message further comprises: a syntax element indicating a url address of a file containing the haptic data.

14. The method according to any one of claims 10-12, wherein the metadata message further comprises: a syntax element indicating a buffer containing the haptic data.

15. The method according to any one of claims 10-12, wherein the metadata message further comprises: a syntax element indicating a buffer containing a set of haptic effects included in the haptic data.

16. The method according to any one of claims 10-15, wherein the coded media is a coded video, and the media element is a frame, a GOP, or a time duration of the coded video.

17. The method according to any one of claims 10-15, wherein the coded media is a coded audio, and the media element is a time duration in the coded audio.

18. The method according to any one of claims 10-17, wherein the haptic data are represented by a haptic data model defined by the MPEG haptic data coding format.

19. An apparatus for integrating haptic stimuli into coded media, comprising: at least one processor; and memory storing instructions that, when executed by the at least one processor, cause the apparatus to: obtain haptic data defining one or more haptic stimuli to be rendered with the rendering of a media element of the coded media, encapsulate the haptic data into a metadata message, and insert the metadata message into the coded media at a location relative to the media element.

20. The apparatus according to claim 19, wherein the metadata message is a supplemental enhancement information (SEI), comprising: a syntax element indicating a type of a haptic format describing the haptic data, wherein the haptic format is one of a readable textual format, a binary format, or a streaming format.

21. The apparatus according to claim 20, wherein the SEI message further comprises: a syntax element indicating one of a url address of a file containing the haptic data, a buffer containing the haptic data, or a buffer containing a set of haptic effects of the haptic data.

22. An apparatus for rendering haptic stimuli integrated in coded media, comprising: at least one processor; and memory storing instructions that, when executed by the at least one processor, cause the apparatus to: retrieve a metadata message from a location in the coded media relative to a location of a media element of the coded media, extract haptic data encapsulated into the metadata message, and render one or more haptic stimuli, defined by the haptic data, with the rendering of the media element.

23. The apparatus according to claim 22, wherein the coded media is a coded video and the media element is a frame, a GOP, or a time duration of the coded video.

24. The apparatus according to claim 22, wherein the coded media is a coded audio and the media element is a time duration in the coded audio.

25. The apparatus according to any one of claims 22-23, wherein the haptic data are represented by a haptic data model defined by the MPEG haptic dada coding format.

26. A non-transitory computer-readable medium comprising instructions executable by at least one processor to perform a method for integrating haptic stimuli into coded media, the method comprising: obtaining haptic data defining one or more haptic stimuli to be rendered with the rendering of a media element of the coded media; encapsulating the haptic data into a metadata message; and inserting the metadata message into the coded media at a location relative to the media element.

27. A non-transitory computer-readable medium comprising instructions executable by at least one processor to perform a method for rendering haptic stimuli integrated into coded media, the method comprising: retrieving a metadata message from the coded media at a location relative to a media element of the coded media; extracting haptic data encapsulated into the metadata message; and rendering one or more haptic stimuli, defined by the haptic data, with the rendering of the media element.