WO2018017934A1

WO2018017934A1 - Haptic system for delivering audio content to a user

Info

Publication number: WO2018017934A1
Application number: PCT/US2017/043242
Authority: WO
Inventors: Joseph VERBEKE; Adam BOULANGER; Davide Di Censo; Stefan Marti
Original assignee: Harman International Industries, Incorporated
Priority date: 2016-07-22
Filing date: 2017-07-21
Publication date: 2018-01-25
Also published as: JP2019523497A; CN109475079A; JP6944513B2; WO2018017210A1; CN109478102A; US10915175B2; JP2019530049A; EP3488427A1; CN109478354A; US20190276051A1; JP7144400B2; CN109478354B; EP3487739B1; EP3487280A1; EP3488326A1; CN109477721A; JP7248568B2; JP6992045B2; US20190268693A1; US11126263B2

Abstract

One embodiment of the present disclosure sets forth a technique for delivering audio content to a user. The technique includes analyzing audio content to determine a frequency at which to vibrate an effector. The technique further includes configuring one or more haptic output devices to generate haptic output that generates a haptic trap proximate to the effector. The technique further includes configuring the one or more haptic output devices to modify the haptic output to cause the haptic trap to vibrate the effector at the determined frequency. The effector vibrates to transmit the audio content. In one embodiment, the effector vibrates a skull of a user.

Description

HAPTIC SYSTEM FOR DELIVERING AUDIO CONTENT TO A USER

CROSS-REFERENCE TO RELATED APPLICATIONS

[0001] This application claims benefit of United States provisional patent application serial number 62/365,960, filed July 22, 2016, which is hereby incorporated herein by reference.

BACKGROUND

Field of the Embodiments

[0002] The various embodiments relate generally to human-machine interfaces and, more specifically, to a haptic system for delivering audio content to a user.

Description of the Related Art

[0003] Audio is a major part of consumer electronics. Audio devices translate signals from electronic devices into audible mechanical waves that a user may hear. For example, a user may attach headphones or speakers to a laptop computer in order to hear sounds associated with applications on the laptop. In addition, users may hear audio from in-vehicle infotainment systems via speakers located within the car.

[0004] Conventionally, headphones are placed over ears of a user, which may deliver sounds directly to the user. Such devices may deliver audio content in a private manner that may allow a user to listen to audio content without concern that other individuals in the environment may hear the audio content. In addition, speakers also deliver audio content to users via audible mechanical waves. Speakers broadcast audio content into a large area, which one or more users may hear.

[0005] One drawback of conventional approaches is that headphones may reduce a user's ability to hear sounds from the environment. In particular, headphones may absorb and/or reflect sound waves in the user's environment, thereby obstructing the sound waves from reaching eardrums of the user. Another drawback of conventional approaches is that generally speakers deliver audio content to a large area such that a user near the speakers may hear the audio content. Therefore, a user may be unable to listen to audio content privately when other individuals are nearby the speakers.

[0006] Conventional bone conduction headphones may deliver audio content to a user privately, without affecting the user's ability to hear sound waves from the environment. Generally, conventional bone conduction headphones vibrate bones in a skull of a user in order to cause a cochlea of the user to resonate, which may deliver audio content to the user in a private manner. Eardrums of a user are not affected by the conventional bone conduction headphones, and so the user may hear sound waves from the environment while listening to audio content from conventional bone conduction headphones.

[0007] However, conventional bone conduction headphones are often large in size. Large headphones may be cumbersome and uncomfortable for a user to wear for long periods of time. In addition, conventional bone conduction headphones require a power source, such as batteries. Batteries add to the bill of materials of the system, require replacement, need to be recharged after a given period of time, and/or add to the weight of the headphones.

[0008] As the foregoing illustrates, techniques for more effectively delivering private audio communications to a user would be useful.

SUMMARY

[0009] Embodiments of the present disclosure set forth a method for delivering audio content to a user. The method includes analyzing audio content to determine a frequency at which to vibrate an effector. The method further includes configuring one or more haptic output devices to generate haptic output that generates a haptic trap proximate to the effector. The method further includes configuring the one or more haptic output devices to modify the haptic output to cause the haptic trap to vibrate the effector at the determined frequency. The effector vibrates to transmit the audio content.

[0010] Further embodiments provide, among other things, a system and a non-transitory computer-readable storage medium configured to implement the techniques set forth above.

[0011] At least one advantage of the techniques described herein is that audio content can be transmitted to a user by a mid-air haptic system. Using haptic output devices to transmit audio content to a user via bone conduction enables private delivery of audio content to a user, without reducing the user' s ability to hear sounds in the external environment. Accordingly, the techniques described herein can be implemented to design mid-air haptic systems for audio content delivery in environments that place a high auditory demand on users, such as public transportation hubs. BRIEF DESCRIPTION OF THE SEVERAL VIEWS OF THE DRAWINGS

[0012] So that the manner in which the recited features of the one or more embodiments set forth above can be understood in detail, a more particular description of the one or more embodiments, briefly summarized above, may be had by reference to certain specific embodiments, some of which are illustrated in the appended drawings. It is to be noted, however, that the appended drawings illustrate only typical embodiments and are therefore not to be considered limiting of its scope in any manner, for the scope of the various embodiments subsumes other embodiments as well.

[0013] Figure 1 illustrates a system configuration for implementing various embodiments of the present disclosure;

[0014] Figure 2 illustrates a block diagram of a computing device configured to implement one or more aspects of the present disclosure, according to various embodiments;

[0015] Figure 3 illustrates a system configuration for temporarily placing an effector near a user's body part in order to transmit audio content generated by audio content generators to the user, according to various embodiments;

[0016] Figure 4 illustrates a system for transmitting audio content to a user by vibrating a diaphragm, according to various embodiments;

[0017] Figure 5 illustrates a system for generating electricity on a piezoelectric film via haptic output, according to various embodiments; and

[0018] Figure 6 illustrates a flow diagram of method steps for delivering audio content to a user via haptic output, according to various embodiments.

DETAILED DESCRIPTION

[0019] Figure 1 illustrates a system configuration for implementing various embodiments of the present disclosure. As shown, system environment 105 includes computing device 100, audio content generator 160, and sensors 140. In addition, system environment 105 includes haptic output devices 150 that generate haptic output 155. Haptic output 155 generates haptic trap 170 proximate to effector 180.

[0020] Effector 180 is any type of object that is capable of vibrating a surface coupled to the effector 180 when a vibrational force is applied to effector 180. Effector 180 may be constructed out of a variety of materials including plastics, metals, alloys, and so forth. In various embodiments, effector 180 is a rigid spherical object. In other embodiments, effector 180 may be a rigid object that has one of a variety of shapes including polyhedral shapes, cones, freeform shapes, and so forth. In addition, effector 180 may be constructed to a variety of sizes and weights. For example, effector 180 could be a spherical object with a 1- centimeter radius.

[0021] In one embodiment, effector 180 is located proximate to a user. In particular, the body of the user may include a skull. In such embodiments, effector 180 may be located proximate to the skull of the user. For example, effector 180 could be grafted into or mounted onto a bone in the skull of the user, could be attached to a plate that is grafted into or mounted onto the skull of the user, and/or can be included in a piece of body jewelry that is in contact with a bone in the skull of the user. In particular, effector 180 may be attached to a temporal bone in the skull of the user (e.g., a squamosal suture). Additionally or alternatively, effector 180 may be located on a piece of body jewelry that is attached to an ear of the user, an eyebrow of the user, a nose of the user, and so forth. In such embodiments, the piece of jewelry is in contact with a bone in the skull of the user. Furthermore, in some embodiments, a body of a user may include a head. Effector 180 may be temporarily placed proximate to the head of the user. In such embodiments, effector 180 may be located within an effector enclosure and the effector enclosure may be placed proximate on the head of the user. In further embodiments, effector 180 is located proximate to a diaphragm in a speaker.

Moreover, in other embodiments, effector 180 is located proximate to a piezoelectric material in a piezoelectric film.

[0022] In operation, effector 180 mechanically transfers vibrational forces to a surface. In particular, a user may have a skull. Effector 180 may be located proximate to the skull of the user. In one embodiment, effector 180 is in contact with the skull of the user. Applying a force to effector 180 displaces effector 180. The displacement of effector 180 generates a force in the direction of the displacement on the skull of the user. Thus, applying a vibrational force to effector 180 causes effector 180 to apply a vibrational force on the skull of the user. Forces applied on the skull of the user may be mechanically transferred to a cochlea of the user, and thus may cause the cochlea to vibrate. When the cochlea vibrates, one or more action potentials may travel along an auditory nerve of the user and may be registered by the user as audio sounds. Various parameters of the vibration of effector 180 (e.g., the frequency, amplitude, and phase) may determine the characteristics of the action potentials transmitted along the auditory nerve (e.g., the number, shape, and amplitude of the action potentials). The characteristics of the action potentials transmitted along the auditory nerve may determine the specifics of the audio sounds that the user hears.

[0023] Effector 180 may include various electronic components for providing location information associated with the effector 180 (also referred to herein as "localization data"). For example, effector 180 could include radio frequency identification (RFID) chips,

Bluetooth® low energy transmitters, infrared (IR) emitters, capacitive sensing components, and other types of small localization components.

[0024] Sensors 140 include any type of sensor that can generate localization data associated with the location of effector 180. Sensors 140 may include various types of imaging sensors, including visual sensors (e.g., RGB image sensors), patterned and pattern- less IR imagers, time-of-flight sensors (e.g., depth cameras), thermal sensors, radar based sensors (e.g., short range and long range sensors), laser based sensors (e.g., LIDAR), ultrasound based sensors, microwave based sensors, driver facing cameras, capacitive sensors and transmitters, Bluetooth® sensors (e.g., Bluetooth® low energy sensors), ultrasound based sensors for triangulation and location, and so forth.

[0025] In various embodiments, sensors 140 are collocated with effector 180. In other embodiments, sensors 140 are located in an external platform positioned around the user, such as on a vehicle console. In a third embodiment, sensors 140 are included in a mobile or wearable device, such as a mobile phone, smart watch, or other device on the user. Sensors 140 transmit the localization data to computing device 100. Sensors 140 may transmit the localization data via a controller area network, a local interconnect network, a FlexRay®, and so forth.

[0026] In one embodiment, haptic output devices 150 include ultrasonic transducers. Ultrasonic transducers generate haptic output 155 in the form of ultrasonic waves. In various embodiments, haptic output devices 150 receive one or more control signals from computing device 100 that indicate how one or more parameters of haptic output devices 150 should be configured. These parameters include the intensity, frequency, and phase of haptic output 155 generated by each of haptic output devices 150. Based on the parameter values specified by the control signal(s), haptic output devices 150 generate haptic output 155. [0027] In particular, haptic output 155 generated by two or more haptic output devices 150 may occupy a particular location in space at a particular time, enabling haptic output 155 of one of haptic output devices 150 to interfere constructively and/or destructively with haptic output 155 of one or more other haptic output devices 150. Constructive interference between haptic output 155 generated by different haptic output devices 150 may produce haptic output maxima at the locations at which haptic output 155 constructively interferes. A haptic output maximum is a location at which the intensity of haptic output 155 is large relative to at least one other location that also receives haptic output 155. Destructive interference between haptic output 155 generated by different haptic output devices 150 may produce haptic output minima at the locations at which haptic output 155 destructively interferes. A haptic output minimum is a location at which the intensity of haptic output 155 is small relative to at least one other location that also receives haptic output 155.

[0028] The haptic output maxima and haptic output minima are herein referred to collectively as haptic output extrema. The parameters of the haptic output extrema include an intensity profile and a position. The intensity profile indicates the intensity of the haptic output 155 at various positions in and around the haptic output extrema. Haptic output 155 produces haptic trap 170. The parameters of the haptic output extrema determine the characteristics of haptic trap 170, which include a size, shape, intensity spectrum, and location. For example, as shown in system environment 105, haptic trap 170 includes one or more haptic output minima located between two rows of haptic output maxima. The shape of haptic trap 170 is determined by the positions of haptic output extrema. In system environment 105, the positions of the haptic output maxima define the center of each section of haptic trap 170. The positions of the haptic output minima define the center of haptic trap 170. Furthermore, the intensity profiles and positions of the haptic output extrema determine the size and location of haptic trap 170. Additionally or alternatively, in some embodiments, haptic trap 170 may include haptic output minima that are fully or partially encircled by haptic output maxima. Each configuration of haptic output extrema corresponds to haptic trap 170 with a different size, shape, intensity spectrum, and location.

[0029] In various embodiments, one or more parameters of the haptic output extrema may be adjusted by configuring haptic output devices 150 to modify one or more parameters of haptic output 155 that produces haptic trap 170. In particular, the positions of each of the haptic output extrema may be shifted, and the intensity profiles of each of the haptic output extrema may be modified. As disclosed above, varying the parameters of the haptic output extrema modifies one or more parameters of haptic trap 170. In this manner, the control signal(s) configure haptic output devices 150 to modify various parameters of haptic trap 170.

[0030] For example, the intensity, phase and/or frequency of haptic output 155 generated by each of haptic output devices 150 could be modified in order to shift the positions of haptic output extrema upwards. In addition, the number of haptic output maxima and haptic output minima can be increased and/or decreased and/or the intensity profile of the haptic output extrema can be modified. As a result, various characteristics of haptic trap 170 could be adjusted, including the size, shape, location, and intensity spectrum of haptic trap 170.

[0031] In various embodiments, effector 180 is located at a haptic output minimum in haptic trap 170. Again, haptic output minimum is a location at which the intensity of haptic output 155 is small relative to at least one other location that also receives haptic output 155. Shifting the position of that haptic output minimum applies a restoring force on effector 180 in a direction that points towards the new location of the haptic output minimum. The restoring force is caused by the increased intensity of haptic output present outside of a haptic output minimum. When the position of the haptic output minimum shifts, a section of effector 180 no longer lies at the center of the haptic output minimum. The section of effector 180 that lies outside of the new position of the haptic output minimum experiences an increased intensity of haptic output. The increased intensity of haptic output 155 exerts a restoring force on effector 180 in the direction that the haptic output minimum shifted. Thus, the restoring force pushes effector 180 back towards the center of the haptic output minimum. In this manner, shifting the position of the haptic output extrema shifts the position of effector 180.

[0032] Furthermore, shifting the position of the haptic output minimum between a first position and a second position can generate a vibrational force on effector 180. The vibrational force can cause effector 180 to vibrate with a particular amplitude and frequency. In particular, the distance between the first position of the haptic output minimum and the second position of the haptic output minimum determines the amplitude of the vibration of effector 180. The number of cycles that effector 180 completes between the first position and the second position per unit time determines the frequency of the vibration of effector 180.

[0033] In some embodiments, such as the one shown in system environment 105, effector 180 is located on the user's head. When haptic trap 170 exerts the vibrational force on effector 180, the effector vibrates, thereby vibrating the user's skull. The skull vibrates at a frequency and amplitude may match the frequency and amplitude of the audio content generated by the audio content generators 160. The vibration of the skull may be mechanically transferred to the user's cochlea, which may vibrate the user's cochlea at the specified frequency and amplitude. When the cochlea vibrates, the cochlea may produce action potentials that may travel along the user's auditory nerves. The amplitude, shape, and time separation between the cochlea generating each action potential may be determined by the frequency and amplitude of vibration of the cochlea. Thus, the action potentials may be reflective of the frequency and amplitude of the audio content generated by the audio content generators 160. Furthermore, the action potentials may be registered by the user as audio sound. In particular, the amplitude and shape of the action potentials and the time separation between individual action potentials may deliver the audio content generated by the audio content generators 160 to the user.

[0034] As shown in system environment 105, haptic output devices 150 are arranged in an array, however, other arrangements of haptic output devices 150 are within the inventive concept. For example, haptic output devices 150 can be arranged in a linear manner, in a circular or elliptical manner, or may be randomly placed at various distances from and locations about effector 180. As shown, haptic output devices 150 are arranged below effector 180, however, haptic output devices 150 can be located at any position around effector 180. For example, haptic output devices 150 could be located above effector 180 and at various angles and distances to effector 180. Regardless of the arrangement and locations of haptic output devices 150, haptic output 155 generated from haptic output devices 150 may produce haptic trap 170 with various shapes and sizes. For example, haptic output devices 150 arranged in a 5 x5 array may generate haptic output 155 that produces haptic trap 170 with a circular or elliptical and with various diameters, positions, and intensity spectra. Furthermore, the 5 x5 array of haptic transducer devices 150 may further modify haptic output 155 to produces haptic trap 170 that generates a vibrational force on effector 180 with a specific amplitude and frequency.

[0035] Audio content generators 160 include any type of device capable of generating audio content. Audio content may include a single frequency audio tone or more complex audio content. Audio content generators 160 may be included in an in-vehicle infotainment system, a driver assistance system, and so forth. Audio content generators 160 may also include audio content from a cloud streaming service, audio content produced by a mobile and/or wearable device, audio content from an in-vehicle infotainment system, audio content stored in computing device 100, audio content received via a microphone system, and so forth. [0036] Computing device 100 may be a microprocessor, an application-specific integrated circuit (ASIC), a system-on-a-chip (SoC), and so forth. As shown, computing device 100 may be independent of audio content generators 160. In alternative embodiments, computing device 100 may be embedded in audio content generators 160 and/or may share one or more processing units, input/output (I/O) interfaces, and/or memory devices (e.g., a database) with audio content generators 160. As noted above, in various embodiments, computing device 100 receives localization data from sensors 140 and audio content from audio content generators 160. Furthermore, computing device 100 transmits control signals to haptic output device 150 that configure haptic output devices 150 to generate a particular type of haptic output 155 that produces haptic trap 170 proximate to effector 180.

[0037] Figure 2 illustrates a block diagram of computing device 100 configured to implement one or more aspects of the present disclosure, according to various embodiments. As shown, computing device 100 includes processing unit 210, I/O interfaces 220, and memory 230. Memory 230 further includes haptic actuation application 232 and database 234. [0038] Input/output (I/O) interfaces 220 may include one or more interfaces that coordinate the transfer of data between computing device 100 and sensors 140, haptic output devices 150, audio content generators 160, and other systems that transmit or receive data from computing device 100. In various embodiments, I/O interfaces 220 coordinate the transfer of audio content between audio content generators 160 and computing device 100. In addition, I/O interfaces 220 coordinate the transfer of localization data between sensors 140 and computing device 100. Furthermore, I/O interfaces 220 coordinate the transfer of control signals between computing device 100 and haptic output device 150. I/O interfaces 220 may transmit and/or receive various types of data via a controller area network, a local interconnect network, a FlexRay®, and so forth. In addition, I/O interfaces 220 may receive electrical signals from one or more microphones external to computing device 100 that detect audio waves in the environment.

[0039] Processing unit 210 may include a central processing unit (CPU), digital signal processing unit (DSP), a sensor processing unit, a controller unit, and so forth. Processing unit 210 may be physically embedded into computing device 100, may be part of a cloud-based computing environment, and/or may be physically introduced into computing device 100 by a user, such as in a mobile or wearable device. In some embodiments, processing unit 210 may be an element of control units 160, such as a microprocessor within an in-vehicle infotainment system or a driver assistance system. In various embodiments, processing unit 210 is configured to execute applications included in memory 230.

[0040] Memory 230 includes haptic actuation application 232 configured to transmit data to and receive data from database 234. Processing unit 210 executes haptic actuation application 232 to implement the overall functionality of computing device 100. Memory 230 may be embedded in control units 160 or introduced into computing device 100 by a user. Moreover, memory 230 may be included in a cloud-based computing environment.

[0041] Database 234 may store various types of data including, user preferences data (e.g., a preferred range of amplitudes and frequencies at which to vibrate effector 180), various pre- set configurations for one or more of sensors 140 and haptic output devices 150,

configurations set by the user, and so forth. For example, database 234 could store the type, position, orientation, frequency range, and intensity range of haptic output devices 150.

Database 234 may also store look-up tables, algorithms, etc. for analyzing localization data in order to identify the position of effector 180. Furthermore, database 234 may store

instructions for determining a type of haptic trap 170 to generate proximate to effector 180 and may store one or more characteristics of effector 180, including the size, weight, and shape of effector 180.

[0042] Haptic actuation application 232 receives audio content from audio content generators 160 and analyses the audio content to determine frequency and amplitude patterns in the audio content. Furthermore, for audio content with multiple simultaneous frequency patterns, haptic actuation application 232 determines phase relationships between pairs of frequency patterns. When the frequency patterns are delivered to the user, haptic actuation application 232 preserves the phase relationships between pairs of frequency patterns in order to produce a high-integrity reproduction of the audio content generated by audio content generators 160. In some embodiments, haptic actuation application 232 may receive electrical signals via I/O devices 120 from one or more microphones that detect audio waves in the environment. In such embodiments, haptic actuation application 132 analyses the electrical signals to determine frequency and amplitude patterns and phase relationships between frequency patterns, as described above. [0043] In operation, haptic actuation application 232 configures haptic output devices 150 to generate haptic output 155 that creates haptic trap 170. Haptic actuation application 232 further configures haptic output devices 150 to modify haptic output 155 to cause haptic trap 170 to vibrate effector 180 at a particular frequency and amplitude. Haptic actuation application 232 selects the frequency and amplitude of the vibration force to match the frequency and amplitude of the audio content generated by audio content generators 160. Vibrating effector 180 at the selected frequency and amplitude causes effector 180 to vibrate any surface proximate to effector 180 at that particular frequency and amplitude. In various embodiments, haptic actuation application 232 selects the frequency and amplitude of vibration of effector 180 to be slightly higher than the frequency and amplitude of the audio content generated by audio content generators 160. Thus, dissipative forces acting on effector 180 and the surface cause the surface to vibrate at a substantially similar frequency and amplitude to the frequency and amplitude of the audio content generated by the audio content generators 160.

[0044] In various embodiments, haptic actuation application 232 analyzes localization data from sensors 140 to determine the position of effector 180. Haptic actuation application 232 may access data from database 234 to determine one or more characteristics of effector 180, including the size, weight, and shape of effector 180. Haptic actuation application 232 may determine various parameters of haptic trap 170 based on the characteristics of effector 180.

[0045] In addition, haptic actuation application 232 also accesses data from database 234 that indicates the position and orientation of haptic output devices 150. Based on the position and orientation of haptic output devices 150 relative to the position of effector 180, haptic actuation application 232 determines a directionality of haptic output 155 and a location at which to generate haptic trap 170 relative to effector 180. In one embodiment, effector 180 is located at a haptic output minimum of haptic trap 170. Based on the directionality, haptic actuation application 232 determines an amplitude, frequency, and phase with which each haptic output device in the haptic output devices 150 should generate haptic output 155 in order to generate haptic trap 170 at the determined location.

[0046] Haptic actuation application 232 determines characteristics of haptic trap 170, including the size, shape, and intensity spectrum of haptic trap 170. The intensity spectrum of haptic trap 170 includes the amplitude and gradient of the haptic output intensity at each position in haptic trap 170. Haptic actuation application 232 may analyze audio content from audio content generators 160 as well as user preferences data in database 234 to determine an amplitude range and a frequency range within which to vibrate effector 180. Based on the characteristics of effector 180 and the determined amplitude and frequency ranges, haptic actuation application 232 may determine a size, shape, and intensity spectrum of haptic trap 170. For example, haptic trap 170 could be larger for vibrations with a large amplitude and smaller for vibrations with a smaller vibration. In addition, the intensity spectrum of haptic trap 170 could be larger for vibrations of a high frequency and smaller for vibrations of a low frequency. Furthermore, the shape of haptic trap 170 is determined by the location and intensity profiles of the haptic output extrema. Thus, haptic trap 170 defined by a haptic output minimum encircled by haptic output maxima may have a circular or elliptical shape. Haptic trap 170 defined by haptic output maxima located are two opposite sides of a haptic output minimum may have the shape of a lemniscate. Haptic trap 170 defined by haptic output maxima located at more than two sides of a haptic output minimum may have the rose- petal shape.

[0047] Furthermore, haptic actuation application 232 may determine an intensity profile for various haptic output extrema. In particular, haptic actuation application 232 determines the intensity profile of the haptic output minimum in which effector 180 is located. The intensity profile of the haptic output minimum determines the magnitude of the restoring force applied to effector 180. As stated above, when the position of the haptic output minimum shifts, a restoring force is applied to effector 180. The magnitude of the restoring force applied to effector 180 is related to the intensity profile of the haptic output minimum. Haptic actuation application 232 determines the intensity profile of the haptic output minimum based on the size, weight and shape of effector 180 and the frequency and amplitude of the audio content being transmitted to the user.

[0048] Haptic actuation application 232 can increase the gradient of the intensity profile in order to exert a restoring force that sharply increases in magnitude as the position of the effector 180 moves away from the haptic output minimum. Such an intensity profile may decrease the phase lag between the movement of the haptic output minimum and the movement of effector 180. Such an intensity profile may also be implemented with larger effectors 180 and/or with power constrained haptic output devices 150. Haptic actuation application 232 may increase the gradient of the intensity profile by reducing the distance between the haptic output minimum and haptic output maxima located proximate to the haptic output minimum. Additionally or alternatively, haptic actuation application 232 may configure haptic output devices 150 to generate haptic output 155 that increases the intensity difference between the haptic output maxima and the haptic output minimum.

[0049] Haptic actuation application 232 can decrease the gradient of the intensity profile in order to exert a restoring force that slowly increases in magnitude as the position of the haptic output minimum shifts. Such an intensity profile may also be implemented with smaller effectors 180 and/or with power constrained haptic output devices 150. Haptic actuation application 232 may decrease the gradient of the intensity profile by increasing the distance between the haptic output minimum and haptic output maxima located proximate to the haptic output minimum. Additionally or alternatively, haptic actuation application 232 may configure haptic output devices 150 to generate haptic output 155 that decreases the intensity difference between the haptic output maxima and the haptic output minimum.

[0050] In some embodiments, haptic actuation application 232 may access database 234 to determine the allowable range of frequencies and amplitudes of audio content being

transmitted to a user. In particular, if the maximum frequency and/or amplitude is above the allowable frequency and/or amplitude range, haptic actuation application 232 may decrease the average frequency and/or amplitude of the audio content to bring the highest frequency and/or amplitude present in the audio content into the range of allowable frequencies and amplitudes. In addition, if the minimum frequency and/or amplitude is below the allowable frequency and/or amplitude range, haptic actuation application 232 may increase the average frequency and/or amplitude to bring the lowest frequency and/or amplitude of the audio content into the range of allowable frequencies and amplitudes.

[0051] Haptic actuation application 232 generates control signals that configure haptic output devices 150 to generate haptic output 155 that produces haptic trap 170 at the determined position proximate to effector 180. In addition, haptic actuation application 232 generates control signals that configure haptic output devices 150 to modify haptic output 155 in a manner that adjusts one or more parameters of haptic output extrema in haptic trap 170. In such a manner, haptic actuation application 232 can change various parameters of haptic trap 170. For example, haptic actuation application 232 could cause the location of the haptic output extrema located in haptic trap 170 to shift upwards, thereby exerting a restoring force on effector 180 that pushes effector 180 upwards. Specifically, haptic actuation application 232 configures the haptic output devices 150 to generate haptic output 155 that shifts the position of the haptic output minimum proximate to effector 180 upwards, thereby exerting a restoring force on effector 180. The restoring force shifts the position of effector 180 upwards.

[0052] Haptic actuation application 232 further configures haptic output devices 150 to generate haptic output 155 that repeatedly shifts the position of the haptic output minimum proximate to effector 180. In one embodiment, haptic actuation application 232 configures haptic output devices 150 to generate haptic output 155 that shifts the position of the haptic output minimum between a first position and a second position. Such a sequence of movements exerts a sequence of restoring forces on effector 180 that repeatedly switch between pushing effector 180 from the first position towards the second position and then pushing effector 180 from the second position towards the first position. Such a sequence of restoring forces is herein defined as a vibrational force. Haptic actuation application 232 configures haptic output devices 150 to generate haptic output 155 that varies the position of various haptic output extrema included in haptic trap 170, including the haptic output minimum proximate to effector 180, in order to exert a vibrational force on effector 180. Haptic actuation application 232 further configures haptic output devices 150 to modify haptic output 155 in a manner that causes haptic trap 170 to exert a vibrational force on effector 180 with a specific frequency and amplitude.

[0053] Figure 3 illustrates a system configuration for temporarily placing effector 180 near a body part of a user in order to transmit audio content generated by audio content generators 160 to the user, according to various embodiments. As shown, system environment 300 includes effector 380, effector enclosure 390, and haptic trap 370.

[0054] Effector 380 is located within effector enclosure 390. Effector enclosure 390 is any type of hollow object that allows haptic output 155 to travel into and out of the object, is large enough to allow effector 380 to move within effector enclosure 390, and is capable of vibrating a surface when vibrated. As describe above, haptic actuation application 232 configures haptic output devices 150 to generate haptic output 155 that generates haptic trap 370 proximate to effector 380.

[0055] As further described above, haptic actuation application 232 configures haptic output devices 150 to modify haptic output 155 in order to exert a correction force on effector 380. Exerting a correction force on effector 380 may cause effector 380 to strike the inner surface of effector enclosure 390, thereby exerting a force on effector enclosure 390. Exerting a force on effector enclosure 390 causes effector enclosure 390 to displace in the direction in which effector 380 applied the force.

[0056] When effector enclosure 390 is located proximate to a user's head, the displacement of effector enclosure 390 applies a force to a skull of the user in the direction in which effector enclosure 390 displaced. The direction and amplitude of the correction force exerted on effector 380 causes effector 380 to strike various sections of the inside surface of effector enclosure 390 in a particular order and with a particular amount of force. For example, when haptic trap 370 exerts a correction force on effector 380 in an upwards direction, effector 380 could strike an upper section of the inside surface of haptic trap 370. Effector 380 could exert a force on effector enclosure 390 in an upwards direction that causes effector enclosure 390 to move upwards. When effector enclosure 390 is in contact with the user's head, a displacement of effector enclosure 390 in an upward direction exerts a force on the user's head in an upwards direction.

[0057] In some embodiments, haptic actuation application 232 configures haptic output devices 150 to modify haptic trap 370 in a manner that exerts a vibrational force on effector 380 with a specific frequency and amplitude. Such a vibrational force may cause effector 380 to repeatedly strike opposite sides of the inner surface of effector enclosure 390 with a substantially similar force, thereby exerting a vibrational force on effector enclosure 390. Exerting a vibrational force on effector enclosure 390 causes effector enclosure 390 to vibrate, which exerts a vibrational force on the user's skull. The frequency and amplitude with which the skull vibrates is determined by the frequency and amplitude of the vibration of effector 380. Haptic actuation application 232 controls the frequency and amplitude at which effector 380 vibrates by configuring haptic output devices 150 to generate haptic output 155 that modifies the positions of haptic output extrema in haptic trap 370 with a particular frequency and amplitude. [0058] As stated above, haptic actuation application 232 analyzes audio content generated by audio content generators 160 to determine the frequency and amplitude at which effector 380 should vibrate. Further, the user's skull vibrates with a frequency and amplitude that is substantially similar to the frequency and amplitude determined by haptic actuation application 232, which may transmit audio content to the user by causing a cochlea of the user to vibrate at the specified frequency and amplitude.

[0059] Figure 4 illustrates a system 400 for transmitting audio content to a user by vibrating diaphragm 490, according to various embodiments. As shown, system 400 includes haptic output devices 150 that generate haptic output 155. Haptic output 155 generates haptic trap 470 proximate to effector 480. Effector 480 is in physical contact with diaphragm 490. Haptic actuation application 232 configures haptic output devices 150 to modify haptic output 150 in order to shift the positions of haptic output extrema in haptic trap 470. [0060] Haptic actuation application 232 further determines a frequency and an amplitude at which the one or more parameters of the haptic output extrema should be modified (e.g., the position), thereby exerting a vibrational force on effector 480. Exerting a vibrational force on effector 480 causes effector 480 to vibrate at the specified frequency and amplitude, thereby vibrating diaphragm 490 at the specified frequency and amplitude. Vibrating diaphragm 490 generates audio waves in speaker enclosure 430 at a substantially similar frequency and amplitude to the frequency and amplitude at which effector 480 vibrates. Thus, speaker system 400 receives audio content from audio content generators 160 and generates audio waves 440 that match the frequency and amplitude of the received audio content. [0061] In various embodiments, effector 480 may be embedded in diaphragm 490 or may be attached to effector 480. In addition, effector 480 may be located in an effector enclosure. The effector enclosure may be placed in contact with diaphragm 490 in order to initiate production of audio waves 440. When the effector enclosure is removed from diaphragm 490, audio waves 440 may be no longer produced by speaker system 400. [0062] Figure 5 illustrates a system 500 for generating electricity on piezoelectric film 530 via haptic output 155, according to various embodiments. As shown, piezoelectric system 500 includes haptic output devices 150 that generate haptic output 155. Haptic output 155 generates haptic trap 570 proximate to effector 580. Effector 580 is in physical contact with piezoelectric film 530. Haptic actuation application 232 configures haptic output devices 150 to generate haptic output 155 that generates haptic trap 570 proximate to effector 580. Haptic actuation application 232 analyzes audio content from audio content generators 160 in order to determine a frequency and amplitude at which to vibrate piezoelectric film 530. Haptic actuation application 232 configures haptic output devices 150 to modify haptic output 155 in order to generate a vibrational force on effector 580 via haptic trap 570 at the determined frequency and amplitude. Effector 580 exerts a vibrational force on piezoelectric film 530 that causes piezoelectric film 530 to vibrate at the determined frequency and amplitude.

[0063] In various embodiments, when a strain is applied to piezoelectric film 530, a voltage difference is generated across electrical leads 540. When effector 580 moves, effector 580 exerts a force on piezoelectric film 530 that causes piezoelectric film 530 to deform, thereby applying a strain on piezoelectric film 530. The strain causes a voltage difference to be generated across electrical leads 540 proportional to the amplitude of the deformation. In addition, when effector 580 applies a vibrational force to piezoelectric film 530, piezoelectric film 530 vibrates with a frequency that is substantially similar to the frequency at which effector 580 vibrates. The vibration generates a voltage difference across electrical leads 540 that is proportional to the frequency and the amplitude of the vibration of piezoelectric film 530. Thus, in various embodiments, the voltage difference generated across electrical leads 540 is proportional to the frequency and amplitude of the audio content generated by audio content generators 160.

[0064] In one embodiment, electric leads 540 of piezoelectric film 530 are connected to an external speaker. Haptic actuation application 232 receives audio content from audio content generators 160 and analyzes the audio content to determine a frequency and amplitude. Haptic actuation application 232 configures haptic output devices 150 to generate haptic output 155 that exerts a vibrational force on effector 580, causing effector 580 to vibrate at the specified frequency and amplitude. When effector 580 vibrates, effector 580 exerts a vibrational force on piezoelectric film 530. When effector 580 exerts a vibrational force on piezoelectric film 530, piezoelectric film 530 vibrates at the specified frequency and amplitude, generating a voltage difference across electrical leads 540 proportional to the frequency and amplitude of the vibration of piezoelectric film 530. A processor within the external speaker analyzes the voltage difference to determine a frequency and amplitude at which to generate audio waves. The frequency and amplitude determined by the processer is substantially similar to the frequency and amplitude of the audio content generated by audio content generators 160. The processor configures the external speaker to generate audio waves at the determined frequency and amplitude, thereby reproducing the audio content of audio content generators 160 through the external speaker.

[0065] Figure 6 illustrates a flow diagram of method steps for delivering audio content to a user via haptic output, according to various embodiments. Although the method steps are described in conjunction with the system of Figures 1-5, persons skilled in the art will understand that any system configured to perform the method steps, in any order, falls within the scope of the present disclosure.

[0066] As shown in Figure 6, a method 600 begins at step 610, where haptic actuation application 232 receives audio content from audio content generators 160. Audio content generators 160 may include microphone systems, in-vehicle infotainment systems, audio stored in database 234, and so forth. At step 620, haptic actuation application 232 analyzes the audio content in order to determine a frequency at which to oscillate effector 180. Haptic actuation application 232 may access user preferences data in database 234 to determine whether the frequency and amplitude ranges of the audio content lie within the allowable frequency and amplitude ranges stored in database 234. If not, then haptic actuation application 232 may increase or decrease the average frequency and/or amplitude of the audio content to bring the audio content within the allowable frequency and amplitude range.

[0067] At step 630, haptic actuation application 232 analyzes sensor data to determine a position of effector 180. In various embodiments, effector 180 is located proximate to a user's head. Sensors 140 may be collocated with effector 180, may be included in an external sensing environment located proximate to a user, and/or may be included in a mobile or wearable device. In addition, effector 180 may include various localization elements, such as Bluetooth® low energy transmitters to facilitate haptic actuation application 232 in locating effector 180.

[0068] At step 640, haptic actuation application 232 configures haptic output devices 150 to generate haptic output 155 that generates haptic trap 170 proximate to effector 180. Haptic trap 170 includes haptic output extrema that are spatially arranged so that effector 180 is located in a haptic output minimum and one or more haptic output maxima are located at various sides of and distances away from the haptic output minimum.

[0069] At step 650, haptic actuation application 232 configures haptic output devices 150 to modify haptic output 155 to cause haptic trap 170 to vibrate effector 180 at the determined frequency. For example, haptic actuation application 232 may configure haptic output devices 150 to generate haptic output 155 that displaces the position of one or more haptic output extrema upwards, thereby deforming the shape of haptic trap 170. In particular, haptic actuation application 232 may configure haptic output devices 150 to modify haptic output 155 in order to repeatedly shift the position of one or more haptic output extrema between a first position and a second location in haptic trap 170. When effector 180 is located in such a haptic output extremum, haptic trap 170 exerts a vibrational force on effector 180 that causes effector 180 to vibrate at the frequency and amplitude at which the haptic extremum shifts between the first position and the second position. When effector 180 is located proximate to a user's head, effector 180 exerts vibrational forces on a user's skull that may cause a cochlea to vibrate at the specified frequency and amplitude, which may transmit the audio content generated by audio content generators 160 to the user. [0070] In sum, the haptic actuation application receives audio content from an audio content generator. The haptic actuation application analyzes the audio content to determine a frequency and amplitude at which to vibrate the effector. In addition, the haptic actuation application receives localization data from one or more sensors that indicates the position of the effector. The haptic actuation application configures haptic output devices to generate haptic output that produces a haptic trap proximate to the effector. The haptic trap includes haptic output extrema disposed within the haptic trap, such that the effector is located at a haptic output minimum. The haptic actuation application configures the haptic output devices to modify the haptic output generated in order to repeatedly shift the position of the haptic output minimum between a first and second position at the determined frequency. Repeatedly shifting the position of the haptic output minimum exerts a vibrational force on the effector that causes the effector to vibrate at the determined frequency and amplitude. When the effector is located proximate to a user's head, the effector vibrates a user's skull at the determined frequency and amplitude. The user's skull may vibrate a user's cochlea, which may transmit the audio content to the user.

[0071] At least one advantage of the techniques described herein is that audio content can be transmitted to a user by a mid-air haptic system. Using haptic output devices to transmit audio content to a user via bone conduction enables private delivery of audio content to a user, without reducing the user' s ability to hear sounds in the external environment. Accordingly, the techniques described herein can be implemented to design mid-air haptic systems for audio content delivery in environments that place a high auditory demand on users, such as public transportation hubs. [0072] The descriptions of the various embodiments have been presented for purposes of illustration, but are not intended to be exhaustive or limited to the embodiments disclosed. Many modifications and variations will be apparent to those of ordinary skill in the art without departing from the scope and spirit of the described embodiments.

[0073] Aspects of the present embodiments may be embodied as a system, method or computer program product. Accordingly, aspects of the present disclosure may take the form of an entirely hardware embodiment, an entirely software embodiment (including firmware, resident software, micro-code, etc.) or an embodiment combining software and hardware aspects that may all generally be referred to herein as a "circuit," "module" or "system." Furthermore, aspects of the present disclosure may take the form of a computer program product embodied in one or more computer readable medium(s) having computer readable program code embodied thereon. [0074] Any combination of one or more computer readable medium(s) may be utilized. The computer readable medium may be a computer readable signal medium or a computer readable storage medium. A computer readable storage medium may be, for example, but not limited to, an electronic, magnetic, optical, electromagnetic, infrared, or semiconductor system, apparatus, or device, or any suitable combination of the foregoing. More specific examples (a non-exhaustive list) of the computer readable storage medium would include the following: an electrical connection having one or more wires, a portable computer diskette, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or Flash memory), an optical fiber, a portable compact disc read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the foregoing. In the context of this document, a computer readable storage medium may be any tangible medium that can contain, or store a program for use by or in connection with an instruction execution system, apparatus, or device. [0075] Aspects of the present disclosure are described above with reference to flowchart illustrations and/or block diagrams of methods, apparatus (systems) and computer program products according to embodiments of the disclosure. It will be understood that each block of the flowchart illustrations and/or block diagrams, and combinations of blocks in the flowchart illustrations and/or block diagrams, can be implemented by computer program instructions. These computer program instructions may be provided to a processor of a general purpose computer, special purpose computer, or other programmable data processing apparatus to produce a machine, such that the instructions, which execute via the processor of the computer or other programmable data processing apparatus, enable the implementation of the functions/acts specified in the flowchart and/or block diagram block or blocks. Such processors may be, without limitation, general purpose processors, special-purpose processors, application-specific processors, or field-programmable processors or gate arrays.

[0076] The flowchart and block diagrams in the figures illustrate the architecture, functionality, and operation of possible implementations of systems, methods and computer program products according to various embodiments of the present disclosure. In this regard, each block in the flowchart or block diagrams may represent a module, segment, or portion of code, which comprises one or more executable instructions for implementing the specified logical function(s). It should also be noted that, in some alternative implementations, the functions noted in the block may occur out of the order noted in the figures. For example, two blocks shown in succession may, in fact, be executed substantially concurrently, or the blocks may sometimes be executed in the reverse order, depending upon the functionality involved. It will also be noted that each block of the block diagrams and/or flowchart illustration, and combinations of blocks in the block diagrams and/or flowchart illustration, can be implemented by special purpose hardware-based systems that perform the specified functions or acts, or combinations of special purpose hardware and computer instructions.

[0077] While the preceding is directed to embodiments of the present disclosure, other and further embodiments of the disclosure may be devised without departing from the basic scope thereof, and the scope thereof is determined by the claims that follow.

Claims

What is claimed is:

1. A system for delivering audio content, the system comprising:

a memory storing instructions; and

a processor configured to execute the instructions to:

analyze audio content to determine a frequency at which to vibrate an effector; configure one or more haptic output devices to generate haptic output that generates a haptic trap proximate to the effector; and

configure the one or more haptic output devices to modify the haptic output to cause the haptic trap to vibrate the effector at the determined frequency, wherein the effector is configured to vibrate to transmit the audio content.

2. The system of claim 1, wherein the processor is further configured to execute the instructions to receive localization data from one or more sensors, wherein the localization data indicates a position of the effector.

3. The system of claim 1, wherein the processor is further configured to execute the instructions to analyze the audio content to determine the frequency at which to vibrate the effector by:

analyzing the audio content to determine a frequency pattern;

analyzing the audio content to determine an amplitude pattern; and

determining the frequency at which to vibrate the effector based on the frequency pattern and the amplitude pattern.

4. The system of claim 1, wherein the haptic trap includes a haptic output maximum and a haptic output minimum.

5. The system of claim 4, wherein the processor is further configured to execute the instructions to configure the one or more haptic output devices to generate haptic output by configuring the one or more haptic output devices to generate haptic output that generates the haptic trap, wherein the haptic output minimum is located at a first position that is collocated with a position of the effector.

6. The system of claim 5, wherein the processor is further configured to execute the instructions to configure the haptic output devices to modify the haptic trap by configuring the haptic output devices to generate second haptic output, wherein the haptic output minimum shifts from the first position to a second position based on the second haptic output.

7. The system of claim 6, wherein the second position is determined by an amplitude associated with the audio content.

8. The system of claim 6, wherein the processor is further configured to configure the haptic output devices to generate repeated haptic output that shifts the haptic output minimum between the first position and the second position with a frequency associated with the audio content.

9. The system of claim 8, wherein the repeated haptic output:

varies a location of the second position based on an amplitude associated with the audio content;

varies the frequency at which the haptic output minimum repeatedly shifts between the first position and the second position based on a frequency associated with the audio content.

10. The system of claim 8, wherein the repeated haptic output causes the effector to transmit audio content via one of passive mechanical waves, audio wave transmission, and voltage differences.

11. The system of claim 1, wherein the effector vibrates one of a diaphragm, and a piezoelectric film.

12. The system of claim 1, wherein the one or more haptic output devices comprise ultrasonic transducer devices.

13. A system for delivering audio content, the system comprising:

one or more haptic output devices configured to:

generate haptic output that generates a haptic trap proximate to an effector; modify the haptic output to cause the haptic trap to vibrate the effector, wherein the effector is configured to vibrate to transmit audio content.

14. The system of claim 13, wherein the haptic trap vibrates the effector at a frequency associated with the audio content.

15. The system of claim 13, wherein the haptic trap includes a haptic output maximum and a haptic output minimum.

16. The system of claim 15, wherein the one or more haptic output devices are configured to:

generate haptic output that generates the haptic trap, wherein the haptic output minimum is located at a first position that is collocated with a position of the effector;

generate second haptic output, wherein the haptic output minimum shifts from the first position to a second position based on the second haptic output; and generate repeated haptic output that shifts the haptic output minimum between the first position and the second position with a frequency associated with the audio content.

17. The system of 13, wherein the haptic trap vibrates the effector at a frequency that is higher than a frequency associated with the audio content.

18. The system of 13, wherein the haptic trap causes an effector enclosure that includes the effector to vibrate, and the effector enclosure causes the effector to vibrate.

19. The system of claim 13, wherein the one or more haptic output devices comprise ultrasonic transducer devices.

20. A method for delivering audio content to a user, the method comprising:

analyze audio content to determine a frequency at which to vibrate an effector;

configure one or more haptic output devices to generate haptic output that generates a haptic trap proximate to the effector; and configure the one or more haptic output devices to modify the haptic output to cause the haptic trap to vibrate the effector at the determined frequency, wherein the effector vibrates to transmit the audio content to the user.