WO2019072389A1

WO2019072389A1 - Extending a haptic signal

Info

Publication number: WO2019072389A1
Application number: PCT/EP2017/076051
Authority: WO
Inventors: José ARAÚJO; Soma TAYAMON; Zeid AL-HUSSEINY; Aitor Hernandez Herranz
Original assignee: Telefonaktiebolaget Lm Ericsson (Publ)
Priority date: 2017-10-12
Filing date: 2017-10-12
Publication date: 2019-04-18

Abstract

It is provided a method for extending a first haptic signal. The method is performed in a haptic signal generator and comprises: receiving a first haptic signal from a user device, the first haptic signal representing haptic content captured by the user device; determining a second haptic signal based at least on the first haptic signal, the second haptic signal representing the haptic content; and extending the first haptic signal by combining the first haptic signal with the second haptic signal, resulting in a combined haptic signal.

Description

EXTENDING A HAPTIC SIGNAL

TECHNICAL FIELD

The invention relates to a method for extending a haptic signal, haptic signal generators for extending a haptic signal, a corresponding computer program, and a corresponding computer program product.

BACKGROUND

Haptic internet is seen as the next step on mobile networking. Users are currently able to efficiently communicate over voice, video and text, but in the future networked society, it is envisioned that people will be able to communicate the sense of touch via haptic devices. There has been a large amount of research on devices that allow such communication to take place, particularly devices that are able to be worn by the user such as gloves or vests, but also integrated in smartphones and smartwatches. Several haptic interfaces are now appearing in the market, which deliver haptic feedback using different actuation technologies such as ultrasound, vibrotactile, electrostatic and piezoelectric.

Haptic perception consists of both kinaesthetic and tactile sense and relates to the sensation of the size, shape, mass, texture and stiffness of physical objects, surfaces, etc. Kinaesthetic information refers to the information perceived when moving joints, muscles and tendons, while tactile

information refers to information retrieved via the skin.

Several haptic feedback solutions for handheld devices exist and typically proide the user with haptic user interfaces (feedback when clicking in buttons or sliders) or haptic feedback as an augmentation to audio or video content being displayed to the user. In other use cases, users using a haptic device may capture haptic signals and transmit the haptic signals between each other, the haptic signals representing haptic content e.g. in the form of the sense of touch. However, capturing haptic signals is time consuming and requires computing resources the user.

SUMMARY

It is an object of this invention to provide an improved solution for capturing haptic signals.

According to a first aspect of the invention, it is provided a method for extending a first haptic signal. The method is performed in a haptic signal generator and comprises: receiving a first haptic signal from a user device, the first haptic signal representing haptic content captured by the user device; determining a second haptic signal based at least on the first haptic signal, the second haptic signal representing the haptic content; and extending the first haptic signal by combining the first haptic signal with the second haptic signal, resulting in a combined haptic signal. In this way, a user only needs to capture the first haptic signal instead of a complete haptic signal to generate the complete haptic signal. This allows the user to capture less haptic data, which saves both time and resources, hence decreasing battery consumption and improving the user experience.

The determining a second haptic signal may comprise obtaining the second haptic signal from a database based on at least one property of the first haptic signal. The database is an efficient way to store possible second haptic signals to be retrieved when desired.

The at least one property may comprise an identifier of a source of the haptic content. In other words, the first haptic signal is characterised by an identity, which is used to retrieve the second haptic signal from the database. This allows the second haptic signals to be managed based on their identities, which can be of use, e.g. for different physical products or for different identifiable experiences.

The determining a second haptic signal may comprise obtaining the second haptic signal based on one or more characteristics of the first haptic signal. The characteristics can be mathematically derived based on the first haptic signal.

The one or more characteristics may comprise at least one of the following characteristics of the first haptic signal: average amplitude, average frequency, median amplitude, median frequency, minimum amplitude, minimum frequency, maximum amplitude, maximum amplitude, average variation of amplitude, average variation of frequency, minimum variation of amplitude, minimum variation of frequency, maximum variation of amplitude, and maximum variation of frequency. The determining a second haptic signal may comprise generating a haptic model based on the first haptic signal, in which case the second haptic signal is based on the haptic model. Using the model, gaps in haptic data coverage can be covered and/or extrapolations of haptic data coverage can be achieved without the need to have corresponding actual captured haptic data. The second haptic signal may comprise haptic model parameters to enable a device to render the combined haptic signal in accordance with the haptic model. The haptic model parameters can take up much less data space compared to an actual haptic signal.

The second haptic signal may extend a duration of the first haptic signal. The second haptic signal may extend a region represented by the first haptic signal.

The second haptic signal may extend the first haptic signal in level of detail.

The method may further comprise providing the combined haptic signal to a recipient device. The determining a second haptic signal may comprise reducing resource requirements of the second haptic signal based on capabilities of the recipient device. This reduces resource usage when the recipient device might not be able to render the second haptic signal fully. The determining a second haptic signal may comprise: rendering alternative second haptic signals to a user, and receiving user input indicating a user selection of which second haptic signal to use in the extending the first haptic signal. The determining a second haptic signal may be based on another user device having captured the haptic content. In this way, multiple user devices can be used to reciprocally extend each other's haptic signals.

The method may further comprise: receiving an additional media signal representing at least one of video content, audio content, and olfactory content of a source of the haptic content, the at least one of video content, audio content, and olfactory content having been captured by the user device. In this case, the determining a second haptic signal is based also on the additional media signal. The additional media signal can thus be used to improve accuracy of determining the second haptic signal. According to a second aspect of the invention, it is provided a haptic signal generator for extending a first haptic signal. The haptic signal generator comprises: a processor; and a memory storing instructions that, when executed by the processor, cause the haptic signal generator to: receive a first haptic signal from a user device, the first haptic signal representing haptic content captured by the user device; determine a second haptic signal based at least on the first haptic signal, the second haptic signal representing the haptic content; and extend the first haptic signal by combining the first haptic signal with the second haptic signal, resulting in a combined haptic signal. The instructions to determine a second haptic signal may comprise

instructions that, when executed by the processor, cause the haptic signal generator to obtain the second haptic signal from a database based on at least one property of the first haptic signal.

The at least one property may comprise an identifier of a source of the haptic content. The instructions to a determine second haptic signal may comprise instructions that, when executed by the processor, cause the haptic signal generator to obtain the second haptic signal based on one or more

characteristics of the first haptic signal. The one or more characteristics may comprise at least one of the following characteristics of the first haptic signal: average amplitude, average frequency, median amplitude, median frequency, minimum amplitude, minimum frequency, maximum amplitude, maximum amplitude, average variation of amplitude, average variation of frequency, minimum variation of amplitude, minimum variation of frequency, maximum variation of amplitude, and maximum variation of frequency.

The instructions to determine a second haptic signal may comprise instructions that, when executed by the processor, cause the haptic signal generator to generate a haptic model based on the first haptic signal. In this case, the second haptic signal is based on the haptic model.

The second haptic signal may comprise haptic model parameters to enable a device to render the combined haptic signal in accordance with the haptic model.

The second haptic signal may extend the first haptic signal in level of detail.

The haptic signal may further comprise instructions that, when executed by the processor, cause the haptic signal generator to provide the combined haptic signal to a recipient device.

The instructions to determine a second haptic signal may comprise instructions that, when executed by the processor, cause the haptic sig generator to reduce resource requirements of the second haptic signal based on capabilities of the recipient device.

The instructions to determine a second haptic signal may comprise instructions that, when executed by the processor, cause the haptic signal generator to: render alternative second haptic signals to a user, and receive user input indicating a user selection of which second haptic signal to use in the instructions to extend the first haptic signal.

The instructions to determine a second haptic signal may be based on another user device having captured the haptic content. The haptic signal generator may further comprise instructions that, when executed by the processor, cause the haptic signal generator to: receive an additional media signal representing at least one of video content, audio content, and olfactory content of a source of the haptic content, the at least one of video content, audio content, and olfactory content having been captured by the user device, and wherein the instructions to determine a second haptic signal comprise instructions that, when executed by the processor, cause the haptic signal generator to determine the second haptic signal based also on the additional media signal.

According to a third aspect of the invention, it is provided a haptic signal generator comprising: means for receiving a first haptic signal from a user device, the first haptic signal representing haptic content captured by the user device; means for determining a second haptic signal based at least on the first haptic signal, the second haptic signal representing the haptic content; and means for extending the first haptic signal by combining the first haptic signal with the second haptic signal, resulting in a combined haptic signal.

According to a fourth aspect of the invention, it is provided a computer program for extending a first haptic signal. The computer program comprises computer program code which, when run on a haptic signal generator causes the haptic signal generator to: receive a first haptic signal from a user device, the first haptic signal representing haptic content captured by the user device; determine a second haptic signal based at least on the first haptic signal, the second haptic signal representing the haptic content; and extend the first haptic signal by combining the first haptic signal with the second haptic signal, resulting in a combined haptic signal.

According to a fifth aspect of the invention, it is provided a computer program product comprising a computer program according to the fourth aspect of the invention and a computer readable means on which the computer program is stored. Generally, all terms used in the claims are to be interpreted according to their ordinary meaning in the technical field, unless explicitly defined otherwise herein. All references to "a/an/the element, apparatus, component, means, step, etc." are to be interpreted openly as referring to at least one instance of the element, apparatus, component, means, step, etc., unless explicitly stated otherwise. The steps of any method disclosed herein do not have to be performed in the exact order disclosed, unless explicitly stated.

BRIEF DESCRIPTION OF THE DRAWINGS

The invention is now described, by way of example, with reference to the accompanying drawings, in which: Fig 1 is a schematic diagram illustrating an environment in which

embodiments presented herein can be applied;

Figs 2A-C are schematic diagrams illustrating extending duration of a haptic signal according to embodiments of the invention;

Figs 3A-B are schematic diagrams illustrating extending a region of a haptic signal according to embodiments of the invention;

Figs 4A-D are schematic diagrams illustrating various embodiments of the haptic signal generator; Figs 5A-C are flow charts illustrating embodiments of the method for extending a first haptic signal;

Fig 6 is a schematic diagram illustrating components of the haptic signal generator of Figs 4A-D according to embodiments of the invention; Fig 7 is a schematic diagram showing functional modules of the haptic signal generator of Figs 4A-D according to embodiments of the invention; and

Fig 8 shows an embodiment of a computer program product comprising computer readable means.

DETAILED DESCRIPTION

The invention will now be described more fully hereinafter with reference to the accompanying drawings, in which certain embodiments of the invention are shown. This invention may, however, be embodied in many different forms and should not be construed as limited to the embodiments set forth herein; rather, these embodiments are provided by way of example so that this disclosure will be thorough and complete, and will fully convey the scope of the invention to those skilled in the art. Like numbers refer to like elements throughout the description.

Fig 1 is a schematic diagram illustrating an environment in which

embodiments of the invention presented herein can be applied. A first user 5a has a user device 2 which is used to capture haptic content using sensors that record haptic inputs. The sensors can comprise e.g. any one or more haptic sensors such as motion sensors (e.g., any one or more of acceleration, velocity, position, and rotation sensors), force sensors, deformation sensors, temperature sensors, etc. The user device 2 may also comprise one or more components for providing haptic feedback to the user via haptic actuators, which can be implemented using e.g. vibration motors, electric motors, voice coil actuators, electroactive polymers, pneumatic actuators, piezoelectric actuators, etc. The user device can further comprise a display, such as a touchscreen or head-mounted display (HMD), or may comprise one or more interfaces for being operatively connected with an external display or HMD. The user device 2 may optionally comprise a haptic touchscreen.

The user device 2 comprises an interface for communicating over a

communications network 6. The communications network 6 can be any one or more interconnected networks, comprising any one or more of a Local Area Network (LAN), such as Ethernet, a Wireless LAN (WLAN), a Wide Area Network (WAN), a cellular network such as 5G NR (New Radio), LTE (Long Term Evolution), W-CDMA (Wideband Code Division Multiplex), EDGE (Enhanced Data Rates for GSM (Global System for Mobile

communication) Evolution), GPRS (General Packet Radio Service),

CDMA2000 (Code Division Multiple Access 2000), or any other current or future wireless network. The communication over the communications network 6 can be effected using any one or more of the Internet Protocol (IP), the Transport Control Protocol (TCP), the Hypertext Transfer Protocol (HTTP), the Constrained Application Protocol (CoAP), etc.

Also connected to the network 6 is a server 3 and a receiver device 7. The receiver device 7 comprises haptic actuators 8 to thereby render haptic content represented by a haptic signal so that a second user 5b can

experience the haptic content represented by the haptic signal. For instance, the haptic actuators 8 can comprise any one or more of vibration motors, electric motors, voice coil actuators, electroactive polymers, pneumatic actuators, piezoelectric actuators, etc. The receiver device 7 can further comprise a display, such as a touchscreen or an HMD. The receiver device 7 may optionally comprise a haptic touchscreen. While the actuators 8 are in Fig 1 illustrated as forming part of a glove, worn on a hand of the second user 5b, the actuators 8 alternatively can be provided in any suitable way that allows haptic perception for the user 5b. For instance, the actuators 8 could form part of an exoskeleton or any type of haptic garment, e.g., a haptic vest, etc. The user device 2 generates a haptic signal 4 based on the captured haptic content. The haptic signal 4 is an analogue or a digital signal which can be transmitted to the receiver device 7 via the network 6. This allows the receiver device 7 to render the haptic content represented by the haptic signal, to allow the second user 5b to experience the haptic content.

Using the environment of Fig 1, the second user 5b can haptically experience a physical environment which can be located remotely, as captured by the user device 2 of the first user 5a. It is to be noted that the user device 2 may also be configured to be a receiver device 7 and vice versa, i.e., devices 2 and 7 may be of the same, or different types.

Alternatively, or additionally, the first user 5a can send synthesised haptic signals to the second user 2b. A synthesised haptic signal is not captured from the real world. Rather, the synthesised haptic signal is generated within the user device 2 or any other electronic device. Furthermore, the haptic signal 4 representing the haptic content could be used for a web or file search using the user device 2. Alternatively or additionally, the haptic signal could be stored in the user device 2 for rendering or transmission to the second user at a later point in time.

According to embodiments of the invention presented herein, the haptic signal representing the haptic content is extended by another haptic signal representing the same haptic content. This extension is performed in a haptic signal generator which can be embodied in various devices, as illustrated in Figs 4A-D and described below.

Figs 2A-C are schematic diagrams illustrating extending duration of a haptic signal according to an embodiment. Time is represented along the horizontal axis.

In Fig 2A, the first haptic signal 20 is depicted as a time-varying signal values. The first haptic signal 20 has a certain duration, represented by its extension along the horizontal axis. The first haptic signal 20 can represent any haptic content, e.g. motion (including any one or more of acceleration, velocity, position and rotation) or force, over time. In Fig 2B, a second haptic signal 21 is shown which extends the first haptic signal 20, yielding a combined haptic signal 22. The second haptic signal 21 is determined based on the first haptic signal 20 and represents the same haptic content as the first haptic signal 20, but for a different time period. In this way, the combined haptic signal 22 represents a greater extent (in this case in terms of duration) of the haptic content. In Fig 2B, the second haptic signal 21 is subsequent to the first haptic signal 20.

Fig 2C is similar to Fig 2B, but the second haptic signal 21 precedes the first haptic signal 20.

Figs 3A-B are schematic diagrams illustrating extending a region of a haptic signal according to an embodiment. The term region is here to be interpreted as geometric extent over which the haptic signal represents the haptic content. The region can be in one dimension, two dimensions or three dimensions. In this example, the region is in two dimensions.

In Fig 3A, there is a physical object being a source 10 of haptic content in the shape of a T-shirt. A user can capture a first haptic signal representing a first region 30 of the source 10. For instance, the user may capture the haptic content by sliding the user device 2 on the source 10. In Fig 3B, the dotted area indicates a second region 31 is shown which is represented by a second signal, which accordingly also represents haptic content of the source 10. In this case, the second signal extends the first haptic signal in terms of region, i.e., the geometric extent of the haptic content. It is to be noted that this extension of the first haptic signal does not need to be a time variant signal. In other words, the extension of the example illustrated in Figs 3A-B is only related to extending the region of the haptic signal.

Figs 4A-D are schematic diagrams illustrating various embodiments of the haptic signal generator. In Fig 4A, an embodiment of the haptic signal generator 1 being implemented in user device 2 is shown. The user device 2 is thus the host device for the haptic signal generator 1. For instance, the user device 2 can be implemented as a smartphone, a table or a smartwatch with haptic sensors and haptic actuators, such as a haptic touchscreen, but also a glove with haptic sensors and haptic actuators.

In Fig 4B, an embodiment of the haptic signal generator 1 being implemented in the server 3 is shown. The server 3 is thus the host device for the haptic signal generator 1. The server 3 can optionally include a database including haptic data for a set of sources of haptic content (e.g. physical objects). In one example, the server also provides haptic data for objects which are offered for sale, e.g. for a furniture, clothing, etc.

In Fig 4C, an embodiment of the haptic signal generator 1 being implemented in a receiver device 7 is shown. The receiver device 7 is thus the host device for the haptic signal generator 1.

In Fig 4D, an embodiment of the haptic signal generator 1 being implemented as a stand-alone device 1 is shown.

Figs 5A-C are flow charts illustrating embodiments of methods for extending a first haptic signal, the method being performed in a haptic signal generator. First, embodiments of methods illustrated by Fig 5A will be described.

The method may start when the user device recognises that the user is performing a haptic signal composition, e.g. creating a haptic message to be transmitted to another user, recording a haptic signal to be consumed by the user or an application being executed in the user device, or storing a haptic signal in the user device or cloud.

In another embodiment, the haptic signal composition may be initiated by an application that the user is currently using on the user device. As an example, the user can begin to compose a message in a user device comprising haptic sensors, and the user selects the option to start recording haptic information. The message may, e.g., be an email message, a chat message, or a Multimedia Messaging Service (MMS) message. As another example, the user may indicate, either by pressing a certain button, or using speech input to the user device, that the user wants to initiate the composition of a haptic signal. In a receive first haptic signal step 40, the haptic signal generator receives a first haptic signal from a user device. The first haptic signal represents haptic content captured by the user device. The haptic signal comprises parameters, in a suitable data format, relating to haptic parameters of haptic content. Such haptic parameters can e.g. be associated with motion, force, texture, friction, stiffness, temperature, etc. The haptic parameters can be time- varying and/or spatially-varying. The haptic content may be applied to the user device (e.g. the user applies movement or force on the device) or the user may use the user device to capture the data (e.g. the user slides the user device over the surface of a sofa). In one embodiment, the context of the haptic content is also recorded. As an example, the context may be that the user is composing a haptic message to another user. In another example, the context may be that the user is performing an information query (e.g. a web query) relating to haptic content. In one embodiment, haptic input is acquired from one or more haptic sensors and recorded.

In a determine second haptic signal step 44, the haptic signal generator determines a second haptic signal based at least on the first haptic signal. The second haptic signal represents the same haptic content as the first haptic signal.

The second haptic signal can e.g. be obtained from a database based on at least one property of the first haptic signal.

For instance, the property can be an identifier of the source of the haptic content represented by the first haptic signal. As an example, the first haptic signal may refer to a source being a sofa. The first haptic signal then contains captured haptic parameters of a specific object, as e.g., the stiffness of a fabric covering a sofa. The identifier can then be determined based on the haptic properties. In another example, the first haptic signal may be based on the source being an event e.g. a rollercoaster ride or a trip on a bicycle that the user has performed, where the identifier then points to the specific

rollercoaster ride or the specific bicycle trip. For the second haptic signal to be determined, a matching is performed between the recorded first haptic signal and labelled second haptic signals stored in the database (in the haptic signal generator). Optionally, the context of the first haptic signal is also used in the determination of the second haptic signal. The context can comprise any one or more values e.g. related to time, date, location, or information about the user (e.g., age, sex, social network data, etc.).

The second haptic signal can be determined based on one or more

characteristics of the first haptic signal. For instance, the one or more characteristics comprise at least one of the following characteristics of the first haptic signal: average amplitude, average frequency, median amplitude, median frequency, minimum amplitude, minimum frequency, maximum amplitude, maximum amplitude, average variation of amplitude, average variation of frequency, minimum variation of amplitude, minimum variation of frequency, maximum variation of amplitude, and maximum variation of frequency.

In one embodiment, the second haptic signal is determined by generating a haptic model based on the first haptic signal, and wherein the second haptic signal is based on the haptic model. In such a case, the second haptic signal can comprise haptic model parameters to enable a device to render the combined haptic signal in accordance with the haptic model. The model can be time dependent or static over time. Additionally or alternatively, the model can be spatially variant, in which case the model can be in one, two, or three dimensions. The model defines how haptic parameters vary over time and/or space. For instance, a variation frequency of one or more haptic parameters in the first haptic signal can be extrapolated to generate the model.

In one embodiment, resource requirements of the second haptic signal are reduced based on capabilities of the recipient device. The capabilities can e.g. relate to haptic rendering capabilities, current battery level, limitations on a communication channel etc., thus reducing requirements for rendering, sending and/or receiving the second haptic signal. This can reduce the amount of energy needed by the haptic actuators when rendering the output content on the user device and/or reduce the number of haptic actuators used for the haptic output signal. Alternatively or additionally, the energy consumed by the haptic actuators when rendering the output content on the user device can be reduced and/or the amount of haptic data to be

transmitted from the user device to a second user device can be reduced, e.g. by reducing a duration of the second haptic signal, reducing/eliminating certain frequency components, reducing resolution and/or amplitude, reducing range of amplitude and/or frequency.

In one embodiment, the second haptic signal is determined based on another user device having captured the haptic content. For instance, when at least two users capture haptic content of the same source, the determination of the second haptic signal is performed at each individual device. In such embodiment, the determination of the second haptic signal at each device could then depend on haptic content captured by the other device

For instance, consider the case where the haptic content relates to a two- dimensional region. A first user device captures one property of the haptic content and a second user device captures another property. It may be the case that some regions have data captured from both user devices while some regions may only have data captured by one of the user devices, since one user has missed those areas with one of the devices. Furthermore, the first user device could be used to capture texture and the second user device could be used to capture stiffness. The determination of the second haptic signal then occurs in the areas where data captured from one of the devices is missing, to complete the missing data. This extends the haptic signal for either of the two user devices, resulting in a combined haptic signal which contains more haptic

information.

In one embodiment, the second haptic signal is determined to cover haptic content which provides a good representation of the haptic content. In this case, the second haptic signal is the haptic signal stored in the database, where the time frame for the second haptic signal extends from a first time Ti to a second time T2.

In one embodiment, the first time Ti corresponds to a point in time at the end of an overlap time period where the first haptic signal matches the second haptic signal.

In one embodiment, the first time Ti corresponds a point in time within the second haptic signal that better describes the information that the user aims to transmit. As an example, the user may want to transmit the received first haptic signal but a new second haptic signal is generated from the stored haptic signal which is known to contain the most relevant haptic signal for the haptic content. In a related embodiment, the second time T2 is the is equal to the end of the stored haptic signal.

In another embodiment, the second time T2 is prior to the end of the stored haptic signal forming the base for the second haptic signal. In other words, the second haptic signal is then a subset of the stored haptic signal. In such an embodiment, the second time T2 may optionally be selected by the user (e.g. the user manually selects the second time T2).

In one embodiment, in addition to the second haptic signal, additional media output signals are also determined as output content. The additional media output signals represent content of the same source which is the source of the haptic content represented by the first haptic data. The additional media output signals can relate to audio, visual and/or olfactory content. The additional media output signals may be rendered in parallel with the second haptic signal. Such additional media output signals may correspond to the received additional media signal, or the output content may be stored content obtained from the database.

In one embodiment, the second haptic signal is a haptic signal which is generated from a non-haptic signal, based on content such as audio, video or olfactory content, which is combined with the first haptic signal. As an example, the haptic output signal may be extracted from a video and audio signal. As another example, concurrently with rendering the first haptic signal by the user device, visual and/or audio content relating to the first and/or haptic signals may also be rendered. Using the video and/or audio content, it is possible for the user to select which time frame or object locations delimiting the second haptic signal and/or the first haptic signal.

In an extend first haptic signal step 46, the haptic signal generator extends the first haptic signal by combining the first haptic signal with the second haptic signal, resulting in a combined haptic signal.

The second haptic signal can extend the first haptic signal in different ways. For instance, the second haptic signal can be used to gain a combined haptic signal which has a longer duration than what is covered by the first haptic signal, as illustrated in Figs 2A-C and described above. Alternatively or additionally, the second haptic signal can extend a region represented by the first haptic signal, as illustrated in Figs 3A-B and described above. The region can be a one-dimensional, a two-dimensional region of coverage or a three- dimensional region of coverage. Alternatively or additionally, the second haptic signal can extend the first haptic signal in level of detail, e.g. a resolution of the haptic signal or a model accuracy, when a model is used. Alternatively or additionally, the second haptic signal can extend the first haptic signal in haptic extent by adding another haptic property that is absent from the first haptic signal. l8

In the following, and with reference to Fig 5B, only new or modified steps compared to the steps of Fig 5A will be described.

In an optional receive additional media signal step 42, the haptic signal generator receives an additional media signal representing at least one of video content, audio content, and olfactory content of a source of the haptic content. The at least one of video content, audio content, and olfactory content has previously, or concurrently with the capturing of the first haptic signal, been captured by the user device, and the determining a second haptic signal is based also on the additional media signal. As an example, an additional media signal, representing video content captured by a camera in the user device, is used to detect that the user is touching a certain sofa or a part of a fabric. This additional media signal is then used to determine the second haptic signal, which is related to the identified haptic content.

In an optional provide combined haptic signal step 48, the haptic signal generator provides the combined haptic signal to a recipient device. In one embodiment, the combined haptic signal can be sent in its entirety to the recipient. Alternatively, a first signal is sent to the recipient device, and the combined haptic signal is stored in a memory which is remotely accessible.

The memory can e.g. be a file server, web server, etc. In this way, the combined haptic signal does not need to form part of the first signal. The first signal contains a reference to enable the recipient to obtain the combined haptic signal from the remotely accessible memory.

Subsequently, the recipient device can render the combined haptic signal, optionally along with any additional media signals. Additionally or alternatively, to providing the combined haptic signal to the recipient device, the combined haptic signal is stored in memory for later use. This storage of the combined haptic signal can, but does not need to, be combined with the storage mentioned above with reference to the first signal containing a reference to the stored combined haptic signal. For instance, the combined haptic signal can be stored in the memory of the user device, analogously to how pictures and other media items are stored. The user can then use the user device to render the combined haptic signal when desired.

Now, and with reference to Fig 5C, optional substeps of the determine second haptic signal step 44 of Figs 5A-B will be described. In an optional render alternatives substep 44a, the haptic signal generator renders alternative second haptic signals to a user of the user device. For instance, the haptic signal generator can determine that there are several possible second haptic signals based on the first haptic signal (greater than a configured probability). This can then result in all of these alternatives, or a configured number of the most probable second haptic signals, are rendered to the user to allow the user to select one of these possible alternatives.

In an optional receive user input substep 44b, the haptic signal generator receives user input. The user input indicates a user selection of which second haptic signal (from the alternative second haptic signals rendered in the render alternatives substep 44a) to use in the extending the first haptic signal. In this way, the user can experience the different haptic signals, e.g. with a finger or hand, and select which second haptic signal to be used in the combined haptic signal.

Using embodiments presented herein, a user only needs to input the first haptic signal instead of a complete haptic signal to generate the complete haptic signal. This allows the user to acquire less haptic data, which saves both time and resources, hence decreasing battery consumption and improving the user experience.

In the following, a few illustratory examples are presented in which embodiments presented herein are applied.

In a first example, consider a user who wants to transmit the haptic information of how it feels to touch a T-shirt the user wants to buy at a store to a recipient. The user utilises the user device to capture a haptic signal related to the haptic properties of the T-shirt at the store. A T-shirt is only used here as a means of example, where other objects such as a furniture object, a pet, a ride in a rollercoaster or other could be used.

Haptic signals related to the T-shirt are provided by the store on their store database and made available to the user device. After the user acquires the first haptic signal, a matching occurs between the first haptic signal and database. Then, the combined signal is transmitted to a recipient device.

Optionally, the user selects between alternative haptic signals to be transmitted to the recipient. Optionally, the image of the T-shirt is also displayed in the user device to aid the user identifying the correct T-shirt. Then, the haptic signal is generated after the user confirms that the T-shirt is correctly identified.

In another example, a sound sample of touching a drum is also rendered so that the user can hear the sound corresponding to playing the drum and confirm that the object has been correctly classified. In addition to these examples, the user may also select the visual and audio components to be transmitted together with the message.

Fig 6 is a schematic diagram illustrating components of the haptic signal generator 1 of Figs 4A-D according to an embodiment. It is to be noted that any one or more of the components described here can optionally be shared with a host device, such as a user device (2 of Fig 4A), a server (3 of Fig 4B) or a receiver device (7 of Fig 4C). A processor 60 is provided using any combination of one or more of a suitable central processing unit (CPU), multiprocessor, microcontroller, digital signal processor (DSP), application specific integrated circuit etc., capable of executing software instructions 67 stored in a memory 64, which can thus be a computer program product. The processor 60 can be configured to execute the methods described with reference to Figs 5A-C above.

The memory 64 can be any combination of random access memory (RAM) and read only memory (ROM) and can be located within the haptic signal generator 1 or be available remotely or locally over a communication link or network. The memory 64 also comprises persistent storage, which, for example, can be any single one or combination of magnetic memory, optical memory, solid-state memory or even remotely mounted memory. The memory 64 can comprise a database which can be used to look up the second haptic signal based on the first haptic signal.

The haptic signal generator 1 further comprises an I/O interface 62 for communicating with other external entities. Optionally, the haptic signal generator 1 also comprises a user interface, comprising a display, such as a touchscreen or head-mounted display (HMD), which is particularly applicable when the haptic signal generator 1 forms part of the user device 2.

Fig 7 is a schematic diagram showing functional modules of the haptic signal generator of Figs 4A-D according to an embodiment of the invention. The functional modules are implemented using software instructions such as a computer program executing in the haptic signal generator. Alternatively or additionally, the functional modules are implemented using hardware, such as any one or more of an ASIC (Application Specific Integrated Circuit), an FPGA (Field Programmable Gate Array), or discrete logical circuits. The functionality implemented by the functional modules corresponds to the steps in the methods illustrated in Figs 5A-C. The functionality implemented by a haptic signal receiver 80 corresponds to step 40 of Figs 5A-B. The functionality implemented by an additional media receiver 82 corresponds to step 42 of Fig 5B. The functionality implemented by a signal determiner 84 corresponds to step 44 of Figs 5A-C. The

functionality implemented by an alternative renderer 84a corresponds to step 44a of Fig 5C. The functionality implemented by a user input receiver 84b corresponds to step 44b of Fig 5C. The functionality implemented by an extender 86 corresponds to step 46 of Figs 5A-B. The functionality

implemented by a signal provider 88 corresponds to step 48 of Fig 5B.

Fig 8 shows one example of a computer program product comprising computer readable means 90. On this computer readable means 90, a computer program 91 can be stored, which computer program can cause a processor to execute a method according to embodiments described herein. In this example, the computer program product 90 is an optical disc, such as a CD (compact disc) or a DVD (digital versatile disc) or a Blu-Ray disc. As explained above, the computer program product could also be embodied in a memory of a device, such as the computer program product 67 of Fig 6. While the computer program 91 is here schematically shown as a track on the depicted optical disk, the computer program can be stored in any way which is suitable for the computer program product, such as a removable solid state memory, e.g. a Universal Serial Bus (USB) drive.

The invention has mainly been described above with reference to a few embodiments. However, as is readily appreciated by a person skilled in the art, other embodiments than the ones disclosed above are equally possible within the scope of the invention, as defined by the appended patent claims.

Claims

1. A method for extending a first haptic signal, the method being performed in a haptic signal generator (1) and comprising:

receiving (40) a first haptic signal from a user device (2), the first haptic signal (20) representing haptic content captured by the user device (2);

determining (44) a second haptic signal (21) based at least on the first haptic signal, the second haptic signal (21) representing the haptic content; and

extending (46) the first haptic signal (20) by combining the first haptic (20) signal with the second haptic signal (21), resulting in a combined haptic signal (22).

2. The method according to claim 1, wherein the determining (44) a second haptic signal (21) comprises obtaining the second haptic signal (21) from a database (3) based on at least one property of the first haptic signal.

3. The method according to claim 2, wherein the at least one property comprises an identifier of a source of the haptic content (10).

4. The method according to claim 2, wherein the determining (44) a second haptic signal comprises obtaining the second haptic signal based on one or more characteristics of the first haptic signal (20).

5. The method according to claim 4, wherein the one or more

characteristics comprise at least one of the following characteristics of the first haptic signal (20): average amplitude, average frequency, median amplitude, median frequency, minimum amplitude, minimum frequency, maximum amplitude, maximum amplitude, average variation of amplitude, average variation of frequency, minimum variation of amplitude, minimum variation of frequency, maximum variation of amplitude, and maximum variation of frequency.

6. The method according to claim 1, wherein the determining (44) a second haptic signal (21) comprises generating a haptic model based on the first haptic signal, and wherein the second haptic signal (21) is based on the haptic model.

7. The method according to claim 6, wherein the second haptic signal (21) comprises haptic model parameters to enable a device to render the combined haptic signal in accordance with the haptic model.

8. The method according to any one of the preceding claims, wherein the second haptic signal (21) extends a duration of the first haptic signal.

9. The method according to any one of the preceding claims, wherein the second haptic signal (21) extends a region represented by the first haptic signal.

10. The method according to any one of the preceding claims, wherein the second haptic signal (21) extends the first haptic signal in level of detail.

11. The method according to any one of the preceding claims, further comprising:

providing (48) the combined haptic signal (22) to a recipient device.

12. The method according to any one of the preceding claims, wherein the determining (44) a second haptic signal comprises reducing resource requirements of the second haptic signal based on capabilities of the recipient device.

13. The method according to any one of the preceding claims, wherein the determining (44) a second haptic signal comprises:

rendering (44a) alternative second haptic signals to a user, and receiving (44b) user input indicating a user selection of which second haptic signal to use in the extending (46) the first haptic signal.

14. The method according to any one of the preceding claims, wherein the determining (44) a second haptic signal (21) is based on another user device having captured the haptic content.

15. The method according to any one of the preceding claims, further comprising:

receiving (42) an additional media signal representing at least one of video content, audio content, and olfactory content of a source of the haptic content, the at least one of video content, audio content, and olfactory content having been captured by the user device, and wherein the

determining (44) a second haptic signal is based also on the additional media signal.

16. A haptic signal generator (1) for extending a first haptic signal, the haptic signal generator (1) comprising:

a processor (60); and

a memory (64) storing instructions (67) that, when executed by the processor, cause the haptic signal generator (1) to:

receive a first haptic signal from a user device (2), the first haptic signal (20) representing haptic content captured by the user device (2);

determine a second haptic signal (21) based at least on the first haptic signal, the second haptic signal (21) representing the haptic content; and extend the first haptic signal (20) by combining the first haptic (20) signal with the second haptic signal (21), resulting in a combined haptic signal (22).

17. The haptic signal generator (1) according to claim 16, wherein the instructions to determine a second haptic signal (21) comprise

instructions (67) that, when executed by the processor, cause the haptic signal generator (1) to obtain the second haptic signal (21) from a

database (3) based on at least one property of the first haptic signal.

18. The haptic signal generator (1) according to claim 17, wherein the at least one property comprises an identifier of a source of the haptic

content (10).

19. The haptic signal generator (1) according to claim 17, wherein the instructions to determine a second haptic signal comprise instructions (67) that, when executed by the processor, cause the haptic signal generator (1) to obtain the second haptic signal based on one or more characteristics of the first haptic signal (20).

20. The haptic signal generator (1) according to claim 19, wherein the one or more characteristics comprise at least one of the following characteristics of the first haptic signal (20): average amplitude, average frequency, median amplitude, median frequency, minimum amplitude, minimum frequency, maximum amplitude, maximum amplitude, average variation of amplitude, average variation of frequency, minimum variation of amplitude, minimum variation of frequency, maximum variation of amplitude, and maximum variation of frequency.

21. The haptic signal generator (1) according to claim 16, wherein the instructions to determine a second haptic signal (21) comprise

instructions (67) that, when executed by the processor, cause the haptic signal generator (1) to generate a haptic model based on the first haptic signal, and wherein the second haptic signal (21) is based on the haptic model.

22. The haptic signal generator (1) according to claim 21, wherein the second haptic signal (21) comprise haptic model parameters to enable a device to render the combined haptic signal in accordance with the haptic model.

23. The haptic signal generator (1) according to any one of claims 16 to 22, wherein the second haptic signal (21) extends a duration of the first haptic signal.

24. The haptic signal generator (1) according to any one of claims 16 to 23, wherein the second haptic signal (21) extends a region represented by the first haptic signal.

25. The haptic signal generator (1) according to any one of claims 16 to 24, wherein the second haptic signal (21) extends the first haptic signal in level of detail.

26. The haptic signal generator (1) according to any one of claims 16 to 25, further comprising instructions (67) that, when executed by the processor, cause the haptic signal generator (1) to provide the combined haptic signal (22) to a recipient device.

27. The haptic signal generator (1) according to any one of claims 16 to 26, wherein the instructions to determine a second haptic signal comprise instructions (67) that, when executed by the processor, cause the haptic signal generator (1) to reduce resource requirements of the second haptic signal based on capabilities of the recipient device.

28. The haptic signal generator (1) according to any one of claims 16 to 27, wherein the instructions to determine a second haptic signal comprise instructions (67) that, when executed by the processor, cause the haptic signal generator (1) to:

render alternative second haptic signals to a user, and

receive user input indicating a user selection of which second haptic signal to use in the instructions to extend the first haptic signal.

29. The haptic signal generator (1) according to any one of claims 16 to 28, wherein the instructions to determine a second haptic signal (21) is based on another user device having captured the haptic content.

30. The haptic signal generator (1) according to any one of claims 16 to 29, further comprising instructions (67) that, when executed by the processor, cause the haptic signal generator (1) to:

receive an additional media signal representing at least one of video content, audio content, and olfactory content of a source of the haptic content, the at least one of video content, audio content, and olfactory content having been captured by the user device, and wherein the

instructions to determine a second haptic signal comprise instructions (67) that, when executed by the processor, cause the haptic signal generator (1) to determine the second haptic signal based also on the additional media signal.

31. A haptic signal generator comprising:

means for receiving a first haptic signal from a user device (2), the first haptic signal (20) representing haptic content captured by the user device (2);

means for determining a second haptic signal (21) based at least on the first haptic signal, the second haptic signal (21) representing the haptic content; and

means for extending the first haptic signal (20) by combining the first haptic (20) signal with the second haptic signal (21), resulting in a combined haptic signal (22).

32. A computer program (67, 91) for extending a first haptic signal, the computer program comprising computer program code which, when run on a haptic signal generator (1) causes the haptic signal generator (1) to:

33. A computer program product (64, 90) comprising a computer program according to claim 32 and a computer readable means on which the computer program is stored.