WO2005004534A1 - The production of augmented-reality audio - Google Patents

The production of augmented-reality audio

Info

Publication number
WO2005004534A1
WO2005004534A1 PCT/AU2004/000902 AU2004000902W WO2005004534A1 WO 2005004534 A1 WO2005004534 A1 WO 2005004534A1 AU 2004000902 W AU2004000902 W AU 2004000902W WO 2005004534 A1 WO2005004534 A1 WO 2005004534A1
Authority
WO
Grant status
Application
Patent type
Prior art keywords
audio
signals
input
listener
external
Prior art date
Application number
PCT/AU2004/000902
Other languages
French (fr)
Inventor
Simon Carlile
Craig Jin
Johahn Leung
Schaik Andre Van
Original Assignee
Vast Audio Pty Ltd
Priority date (The priority date is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the date listed.)
Filing date
Publication date

Links

Classifications

    • HELECTRICITY
    • H04ELECTRIC COMMUNICATION TECHNIQUE
    • H04RLOUDSPEAKERS, MICROPHONES, GRAMOPHONE PICK-UPS OR LIKE ACOUSTIC ELECTROMECHANICAL TRANSDUCERS; DEAF-AID SETS; PUBLIC ADDRESS SYSTEMS
    • H04R5/00Stereophonic arrangements
    • H04R5/033Headphones for stereophonic communication
    • HELECTRICITY
    • H04ELECTRIC COMMUNICATION TECHNIQUE
    • H04SSTEREOPHONIC SYSTEMS 
    • H04S1/00Two-channel systems
    • H04S1/002Non-adaptive circuits, e.g. manually adjustable or static, for enhancing the sound image or the spatial distribution
    • H04S1/005For headphones
    • HELECTRICITY
    • H04ELECTRIC COMMUNICATION TECHNIQUE
    • H04RLOUDSPEAKERS, MICROPHONES, GRAMOPHONE PICK-UPS OR LIKE ACOUSTIC ELECTROMECHANICAL TRANSDUCERS; DEAF-AID SETS; PUBLIC ADDRESS SYSTEMS
    • H04R1/00Details of transducers, loudspeakers or microphones
    • H04R1/10Earpieces; Attachments therefor ; Earphones; Monophonic headphones
    • H04R1/1016Earpieces of the intra-aural type
    • HELECTRICITY
    • H04ELECTRIC COMMUNICATION TECHNIQUE
    • H04RLOUDSPEAKERS, MICROPHONES, GRAMOPHONE PICK-UPS OR LIKE ACOUSTIC ELECTROMECHANICAL TRANSDUCERS; DEAF-AID SETS; PUBLIC ADDRESS SYSTEMS
    • H04R1/00Details of transducers, loudspeakers or microphones
    • H04R1/10Earpieces; Attachments therefor ; Earphones; Monophonic headphones
    • H04R1/1083Reduction of ambient noise
    • HELECTRICITY
    • H04ELECTRIC COMMUNICATION TECHNIQUE
    • H04RLOUDSPEAKERS, MICROPHONES, GRAMOPHONE PICK-UPS OR LIKE ACOUSTIC ELECTROMECHANICAL TRANSDUCERS; DEAF-AID SETS; PUBLIC ADDRESS SYSTEMS
    • H04R2420/00Details of connection covered by H04R, not provided for in its groups
    • H04R2420/07Applications of wireless loudspeakers or wireless microphones

Abstract

A method for the delivery of augmented-reality audio includes creating an occlusion (12) in an auditory canal (14) of a listener so as to limit the transmission of external sounds in the range of human hearing from the environment to at least one eardrum of the listener. An acoustic sensor (20) is arranged within the auditory canal distally of the occlusion (12) and the sensor (20) is used to sense the external sounds as they arrive at the sensor (20). An acoustic actuator (22) is arranged within the auditory canal proximally of the occlusion (12). Auxiliary input audio signals are received from an external source to be rendered as virtual audio to the listener. The actuator (22) is used to deliver an output audio signal to the eardrum (18), the output audio signal constituting at least one of input audio signals corresponding to the external sounds, the auxiliary input audio signals and combinations of the foregoing.

Description

"The production of augmented-reality audio" Field of the Invention The present invention relates to the production of augmented-reality audio. More particularly, the invention relates to a method of, and equipment for, the production of augmented-reality audio and , more specifically, for rendering virtual spatial audio concurrently with the real sound environment in such a manner that a listener clearly perceives both the real and virtual sound sources as localized within the listener's personal auditory space.

Background to the Invention The Applicant is aware of various techniques for producing virtual spatial audio as well as for delivering electronically-produced audio to the ears such that the real sound environment remains audible and accessible, but not necessarily properly spatialised, when the electronically-produced audio is played back. Still further, the Applicant is aware of the use of earphones, commonly referred to as "open tubephones", that sit near the opening of the ear canal, but do not block the canal's entrance, and allow the presentation of both virtual and real sound sources such that both are properly spatialised within the listener's personal auditory space. Some of the techniques for producing virtual spatial audio apply electronic filtering to the sound signals for the left and right ear using filters that are designed to simulate the acoustic filtering of the listeners' outer ears. These techniques include accurately estimating the filter functions of the listener's outer ears for any location in space around the listener in a manner which generates for the listener a virtual sound field that is nearly indistinguishable from the real sound field. Some of the techniques for producing virtual spatial audio use "closed" earphones to acoustically deliver the filtered signals for the left and right ear to the ears of the listener. Closed earphones, such as circumaural headphones that fit around the outer ear, or ear-bud headphones that fit in the ear, are referred to as closed earphones because they significantly occlude the sound path to the ear, thereby substantially distorting the external sound input to the ear. These types of earphones significantly reduce the quality of the external sound field that is heard by the listener. Thus, a disadvantage of the techniques that use a closed earphone is that the closed earphones that deliver the virtual spatial audio also attenuate and/or distort sounds in the real, external environment. Yet other techniques for producing virtual spatial audio use "open" earphones.

Some open earphones provide a low acoustic impedance seal (a partial seal) around the outer ear and thereby produce less occlusion for external sounds than closed earphones. A disadvantage of these techniques that use open earphones with a low acoustic impedance seal is that the low acoustic impedance seal can still distort the acoustic filtering of the outer ear. This distortion disrupts the normal acoustic spatialisation cues associated with outer ear filtering and results in poor acoustic spatialisation of external sound sources. Still other techniques for producing virtual spatial audio use "open" tubephones that sit near, but do not block, the ear canal. These earphones allow the presentation of both virtual and real sound sources such that both are properly spatialised within the listener's personal auditory space. A disadvantage of these techniques is that one does not have full control of the sound signal presented to the listener. More specifically, one cannot modify the intensity level or characteristics of the real sound sources. Another disadvantage is the relatively high power required to drive the open tubephones because of the distance they sit away from the eardrum. A further disadvantage is that the relatively cumbersome and delicate nature of the placement of the tubephones detracts from the general utility of such devices.

Summary of the Invention According to a first aspect of the invention, there is provided a method for the delivery of augmented-reality audio, the method including the steps of: creating an occlusion in an auditory canal of a listener so as to limit the transmission of external sounds in the range of human hearing from the environment to at least one eardrum of the listener; arranging an acoustic sensor within the auditory canal distally of the occlusion and using the sensor to sense the external sounds as they arrive at the sensor; arranging an acoustic actuator within the auditory canal proximally of the occlusion; receiving auxiliary input audio signals to be rendered as virtual audio to the listener; and using the actuator to deliver an output audio signal to the eardrum, the output audio signal constituting at least one of input audio signals corresponding to the external sounds, the auxiliary input audio signals and combinations of the foregoing. In this specification the term "distally of the occlusion" means on the side of the occlusion facing outwardly from the ear and, conversely, the term "proximally of the occlusion" means on the side of the occlusion facing towards the eardrum. The method may include selecting the acoustic sensor so as to minimise disruption to acoustic filtering of an outer ear of the listener. Further, the method may include implementing the sensor as a microphone with an input signal frequency range comparable to that of normal human hearing, i.e. with a frequency range of about 50 Hz to 16 kHz. In this way, the acoustic sensor may be operable to capture the real sound field. The method may include implementing the acoustic actuator as a miniature loudspeaker, or miniature loudspeaker array, with a frequency range comparable to that of normal human hearing. The method may include processing the auxiliary audio signals using methods of virtual auditory space to create the illusion for the listener that sounds representative of the auxiliary audio signals originate at specific locations in the listener's personal auditory space around the listener's head. Further, the method may include using signal processing to modify and mix any combination of input audio signals to produce the output audio signal fed to the acoustic actuator. Thus, the method may include processing the input audio signals corresponding to the external sounds to one of amplifying and attenuating the input audio signals corresponding to the external sounds. In addition, the method may include processing at least one of the input audio signals corresponding to the external sounds and the auxiliary audio signals to perfomi enhancement of at least one class of sounds of interest within the external sounds. For example, the method may include processing the input audio signals to perform speech enhancement. Still further, the method may include processing the input audio signals corresponding to the external sounds to effect active noise cancellation. Also, the method may include processing the input audio signals corresponding to the external sounds to assist in protecting the listener's hearing by rapidly attenuating loud external sounds. In addition, the method may include playing back the output audio signal through the acoustic actuator with a time delay after sensing of the input audio signals corresponding to the external sounds by the sensor. The method may include transmitting the input audio signals corresponding to the external sounds to an external devices before or after processing for further processing, delayed playback to the listener, or playback to other listeners. Further, the method may include using a device controller to control operation of at least certain components, the components being the sensor, the acoustic actuator and the signal processor. Additionally, the method may include transmitting audio signals to other devices and/or receiving signals from other devices. The method may also include receiving information and/or instructions from the other devices, such as a wearable computer or other devices, in regard to how the auxiliary audio signals are to be rendered so as to generate acoustic characteristics such as the spatial location or the motion trajectory of the object in extra-personal space, the loudness of the auditory object, the timbre of the auditory object, the ambient acoustic characteristics associated with a virtual environment of an auditory object. Thus, the method may include manipulating audio filters used to generate the auxiliary input audio signals to modify the auditory object in virtual auditory space. Preferably, the method is implemented at both ears of the listener, but may only be performed at one of the listener's ears. According to a second aspect of the invention, there is provided equipment for the delivery of augmented-reality audio, the equipment including: an occluder for occluding an auditory canal of a listener so as to limit the transmission of external sounds in the range of human hearing from the environment to at least one eardrum of the listener; a sensor for sensing the external sounds, the sensor being arranged on one side of the occlusion to lie within the auditory canal distally of the occlusion; an acoustic actuator arranged on an opposed side of the occlusion so as to lie in the auditory canal proximally of the occlusion; and a receiver for receiving auxiliary input audio signals from a source of the auxiliary input audio signals to be rendered as virtual audio to the listener, the receiver and the sensor being in communication with the actuator so that the actuator is operable to deliver an output audio signal to the eardrum, the output audio signal constituting at least one of input audio signals corresponding to the external sounds, the auxiliary input audio signals and combinations of the foregoing. The sensor may be configured to minimise disruption to acoustic filtering of an outer ear of the listener. Thus, the sensor may be in the form of a microphone having a frequency range comparable to that of normal human hearing. The actuator may be in the form of a loudspeaker with a frequency range comparable to that of normal human hearing. The equipment may include a signal processor for processing the input audio signals corresponding to the external sounds and the auxiliary input audio signals prior to the signals being received by the actuator. The signal processor may modify and mix any combination of input audio signals to produce the output audio signal fed to the acoustic actuator. Further, the signal processor may effect one of amplification and attenuation of the input audio signals corresponding to the external sounds. Also, the signal processor processes at least one of the input audio signals corresponding to the external sounds and the auxiliary audio signals to perform enhancement of at least one class of sounds of interest within the external sounds. In addition, the signal processor may process the input audio signals corresponding to the external sounds to effect active noise cancellation. The signal processor may further process the input audio signals corresponding to the external sounds to assist in protecting the listener's hearing. The signal processor may include a time delay for playing back the output audio signal through the acoustic actuator with a time delay after sensing of the input audio signals corresponding to the external sounds by the sensor. The equipment may include an amplifier for amplifying signals including the signals from the sensor, a signal output from the signal processor and the auxiliary input audio signals. The equipment may include an external transmitter device for transmission of the input audio signals corresponding to the external sounds sensed by the sensor. The equipment may include a device controller for enabling the listener to control operation of at least certain components, the components being the sensor, the acoustic actuator and the signal processor. The receiver may be configured to be received in the auditory canal. More particularly, the receiver may be contained in the occluder. The receiver may be in wireless communications with the source by an appropriate wireless communications protocol such as radio frequency, infra red, ultrasonic, or the like. It will also be appreciated that, instead of a wireless link, the receiver could be wired to the source. The receiver may form part of a transceiver to enable signals to be transmitted to other devices. The equipment may include an adjustable audio filter arrangement used to enable the auxiliary input audio signals to be manipulated to modify an auditory object in virtual auditory space. Further, the equipment may include an energy storage unit to supply power. Preferably, the equipment is configured for each ear of the listener. Brief Description of the Drawings The invention is now described by way of example with reference to the accompanying drawings in which: - Figure 1 shows a schematic representation of equipment, in accordance with an embodiment of the invention, for the delivery of augmented-reality audio in use; Figure 2 shows a schematic, block diagram of the equipment; and Figure 3 shows a front view of another version of the equipment.

Detailed Description of Exemplary Embodiment In the drawings, reference numeral 10 generally designated equipment, in accordance with an embodiment of the invention, for the delivery of augmented-reality audio. The equipment 10 includes an occluder 12 receivable in an auditory canal 14 of an ear 16 of a listener. The occluder 12 is shaped and configured effectively to block the auditory canal 14 and to inhibit sounds reaching an eardrum 18 of the listener. More particularly, the occluder 12 reduces, by about 10 dB, the transmission of sounds in the range of human hearing from the environment to the eardrum 18. In the embodiment shown in Figure 1 of the drawings, the occluder 12 is individually moulded to fit in a bony part of the auditory canal 14 of the listener. The occluder 12 can be made of any suitable hard, malleable compound or soft compound or a mixture of the two. A sensor 20 is mounted on a distal side of the occluder 12 to face an entrance to the auditory canal 14. The sensor 20 senses sounds from the external environment within the range of human hearing. Thus, for example, the sensor 20 is implemented as a microphone having an input signal frequency range of about 50 Hz to 16 kHz. The sensor 20 senses the sounds from the external environment and generates input audio signals corresponding to the external sounds. For ease of explanation, these input audio signals will be referred to below as the external audio input signals. An acoustic actuator 22 is arranged on a distal side of the occluder 12. It is therefore positioned in the auditory canal 14 and delivers sounds to the eardrum 18 of the listener, as will be described in greater detail below. The acoustic actuator 22 is implemented in the form of a miniature loud speaker or miniature loud speaker array which has an audible frequency range comparable to that of normal human hearing. A signal processor 24 is contained within a body of the occluder 12, as shown in greater detail in Figure 2 of the drawings. The signal processor 24 processes incoming signals prior to transmission of the signals to the actuator 22. An amplifier 26 is interposed between the signal processor 24 and the actuator 22. The equipment 10 further includes a receiver constituting part of a transceiver 28 contained within the body of the occluder 12. The transceiver 28 communicates wirelessly with a source 30 of auxiliary, input audio signals. For example, the source 30 is a cellular telephone or some other device. The transceiver 28 communicates with the source 30 via any suitable wireless communications protocol, more particularly, a suitable low power wireless connection. The wireless connection can take the form of a radio-frequency signal conforming to an accepted standard such as 801.1 lb. Instead, the wireless connection could be an infra red connection using infrared as a carrier signal or the wireless connection could be an ultrasonic connection using an ultrasonic carrier signal. Instead of a wireless connection, the transceiver 28 could be wired to the source 30 with the source 30, for example, being worn behind the ear 16 of the listener.. The equipment 10 also includes an auxiliary communications device 32 carried or worn by the listener. The device 32 allows devices with multiple transmission protocols to communicate with the transceiver 28 either wirelessly or in a wired manner. Still further, the equipment includes an external device 34. The external device 34 receives signals sent from the parts of the equipment 10 contained in the auditory canal 14 of the listener for onward transmission to other devices (not shown). The device 34 communicates with the sensor 20 via the transceiver 28 or auxiliary communications devices 32. The equipment 10 also includes a controller 36 for controlling operation of the equipment. The controller 36 communicates with the source 30 as well as with a data reading unit 38 for input of information to the controller 36. The controller 36 is accessible by the listener for controlling operation of the equipment 10. For ease of explanation, the parts of the equipment 10 contained within the auditory canal 14 shall be referred to below as an earphone 40. It will be appreciated, that, in use, the equipment 10 comprises two such earphones 40, one for each auditory canal 14 of the listener for enabling sounds to be delivered to both eardrums 18 of the listener 10. In the embodiment shown in Figure 1 of the drawings, the occluder 12 of the earphone 40 is a mouldable, individualized element moulded specifically for the auditory canal 14 of a particular listener. The occluder 12 fits snugly within the bony portion of the auditory canal 14 and it is possible to wear it semi-permanently. In another, more generalised version of the equipment 10, the earphones 40 are carried on stalks 42 projecting radially inwardly from a holder 44 worn on a user's head. Other components of the equipment 10 are carried in a holder 46 on the holder 44. In this embodiment, the equipment 10 approximates a conventional set of headphones but with the modification that the earphones 40 are still received completely in the auditory canal 14 effectively to block the auditory canal 14. Referring again to Figure 2 of the drawings it is to be noted that the equipment 10 includes an energy storage unit or battery 48 contained in the body of the occluder 12. The battery 48 could, for example, be a re-chargeable battery. The equipment 10 is intended for delivery of augmented-reality audio to a listener. Accordingly, in use, an earphone 40 is placed completely in each auditory canal 14 of the listener. As indicated above, the earphones 40 effectively occlude the auditory canal 14 and substantially inhibit the passage of external sounds from the environment to the eardrums 18 of the listener. It is to be noted that, by having the earphones 40 completely within the auditory canals 14 of the listener, the acoustic filtering of the outer ear 16 of the listener is preserved to enable external sounds to be detected by each sensor 20. In this way, monaural spectral cues that are derived from the filtering by the outer ear 16 are available for sound localisation and generation to enable the listener to have a precise perception of the external auditory environment. By having the earphones 40 within the auditory canals 14 and the proximity of the acoustic actuator 22 to its associated eardrum 18, the power required to generate sounds at an appropriate sound level is reduced, reducing the power consumption of the equipment 10 and minimising drain on the battery 48. In addition, acoustic leakage around the earphones 40 is minimised. Using the embodiment shown in Figure 3 of the drawings, a less deep placement of the earphone 40 within each auditory canal 14 may result and less individualised methods of fitting the earphones 40 may need to be employed. Although these factors may reduce the utility of the equipment 10 slightly it is compensated for by the ease and convenience of the processes required for fitting, inserting and removing the earphones 40. The earphones 40 are designed to preserve the full range of hearing frequencies for human listeners in the frequency range from about 50 Hz to 16 kHz. The earphones 40 are also able to transmit external sounds collected by the sensors 20 to the actuators 22 as the external audio input signals and then to the eardrum without substantial distortion of modification. It will, however, be appreciated that, by appropriate manipulation of the signal processor 24 sounds emanating from the environment can be transformed in a variety of ways. These transformations include, but are not limited to: compensating, using a predicted or measured transfer function for the auditory canal 14, for the loss of auditory canal resonance resulting from the occlusion of the auditory canal 14 by the earphone 40; filtering sounds from the external environment so as to compensate for the occlusion effect produced by the occluder 12; producing rapid changes in the gain of the external sound that can be triggered by the incoming sound level or by a signal from some external device communicated to the earphone 40 via the transceiver 28 or the communications device 32; producing signals that result in active noise cancellation at the eardrum 18; or producing changes in the gain of specific classes of sounds based on their spectral-temporal characteristics (such as speech) while varying the gain independently for other sounds discriminated in terms of their different spectral- temporal characteristics. However, the main purpose of the signal processor 24 is to combine the external audio input signals from the sensor 20 and the auxiliary, input audio signals from the transceiver 28 to feed these combined signals to the acoustic actuator 22. The acoustic actuator 22 emits a composite signal constituting the external audio input signals from the sensor 20 and the auxiliary, input audio signals from the transceiver 28 to the eardrum 18. The controller 36 adjusts and/or changes audio filters used to create auditory objects in virtual auditory space. The unit 38 comprises a memory card reader, smartcard reader or other data reading device for input to the controller 36 of information such as the audio filters required to render auditory objects in virtual auditory space. The source 30 communicates with the earphones 40 either via the transceiver 28 or the auxiliary communications device 32. As indicated above, this communications between the source 30 and the transceiver 28 or the communications device 32 is, preferably, effected wirelessly. The signal processor 24 receives the audio input from the transceiver 28, possibly in addition with information from the controller 36. The signal processor 24 mixes the auxiliary, input audio signals from the transceiver 28 with the external audio input signals from the sensor 20 to effect augmented-reality audio made available to the listener. Using the Applicant's methods for virtual auditory space as disclosed in the

Applicant's International Patent Application, Publication No. WO 01/54453 entitled "The generation of customised three-dimensional sound effects for individuals", the contents of which are incorporated herein by reference, the additional input is processed by the signal processor 24 so that, when delivered to the listener via the actuator 22, an auditory object or objects is/are rendered for the perception of the listener at some phantom or virtual location or locations in space around the listener. The source 30 could be a wearable computer or some other intelligent device that takes input from the listener or from other sources in regard to how the signal might be rendered so as to generate various characteristics of the auditory object. These characteristics include, but are not limited to, the spatial location or the motion trajectory of the object in extra-personal space, the loudness of the auditory objection, its timbre or the ambient acoustic characteristics associated with the virtual environment of the object such as reverb erance. The source 30 can perform the filtering required to render sounds for the right and left earphones 40. Still further, the source 30, optionally, sends only the signal to be rendered or the signal with tokens, coefficients and other information for the signal processor 24 to enable the signal processor 24 to perform the necessary filtering. Still further, the listener could select an information channel from the source 30 and, using the controller 36, select the characteristics to be rendered for the information channel received by the earphones 40. A common example of this is the audio channel from a mobile communications device such as a cellular telephone which is then rendered by the signal processor 24 in the earphones 40 to appear at a particular location in auditory space as selected by the listener. In this case, the coefficients necessary to carry out such filtering are loaded into the signal processor 24 via the controller 36 and the unit 38. The sensors 20 of the earphones 40 could also be used as input devices for secondary communications and other intelligent devices. Hence, the utterances of the listener are encoded by the sensors 20 and then transmitted to the device 34 using either the transceiver 28 or the auxiliary communications device 32, as the case may be. In the latter case, only low powered transmission is required between the earphones 40 and the auxiliary communications device 32. This, in turn, reduces current drain on the battery 48 of the earphones 40. Thus, by means of the equipment 10, both the real external sound field and the virtual auditory field can be appreciated simultaneously and with high fidelity. On the one hand, sounds from the real external sound field can be passed without distortion from the input side of the earphone 40 to the output side and thence to the eardrum 18. In this way, the earphone 40 generates the same pattern of sound waves at the eardrum 18 that would have occurred in the absence of the earphone 40. On the other hand, using the methods of virtual auditory space, the rendered input can be combined with the external audio input signals by the signal processor 24 in the earphone 40. When delivered to the eardrums 18 of the right and left ears this results in a high fidelity rendering of virtual auditory space that can be appreciated in conjunction and simultaneously with the appreciation of the real external sound field. Another advantage of the invention is that the equipment 10 provides for the introduction of rendered signals from a variety of sources. This can include, but is not limited to, a cellular telephone or other communications device, voice or other sound interfaces from intelligent devices and music or other sounds from entertainment devices. The link to the signal processor 24 in the earphones 40 also allows listener control over the characteristics of these inputs such as, but not limited to, their virtual location, loudness and tonal quality. As the output of each earphone 40 is almost entirely sealed from the real external sound field, these sounds can also be appreciated by the listener and not heard by other listeners near the listener using the equipment 10. A further advantage of the invention arises from the ability of human listeners to parse or separate multiple concurrent sources of information based on differences in their locations in space around the listener. This is referred to as the "cocktail party effect" and a common example is the ability of listeners to attend to one conversation in a situation where there are multiple talkers and to switch attention from one conversation to another at will. For example, if speech is rendered into a specific virtual location near the listener then the listener is able to selectively attend to the rendered talker or to ignore the rendered talker as desired The equipment 10 enhances this ability of the listener using the equipment 10. Yet another advantage of the invention is that the virtual auditory object can be used to render information where the location of a particular sound may convey information at the same time as the listener is engaged in other activities. An example of such an application is in providing verbal or other spatial directions using a navigational guidance system for buildings or other terrain. Still a further advantage of the invention is that hearing protection functions can also be managed using the equipment 10. In environments with high background noise that is either relatively continuous or impulsive, the gain of the earphones 40 can be dynamically and rapidly adjusted to significantly alter the overall gain of the sound reaching the eardrums 18 of the listener. This derives in part from the fact that the earphone can be tightly sealed in its associated auditory canal 14 at a location close to the eardrum 18 and, when the sound transmitting function is attenuated or turned off, will provide a very effective occlusive barrier to sound from the real external sound field to the eardrum. In addition, the signal processor 24 in each earphone 40 can also modify the sound delivered to the eardrum 18 so that, for instance, by inverting its phase and adding an appropriate short delay, the sound will actively cancel sound that may leak from the real external sound field to the eardrum from around or through the earphone. An additional advantage of the invention is that hearing enhancement functions for the hearing-impaired listener can also be implemented in the earphones 40 using the signal processor 24 of each earphone 40. This can include variable gain function, background noise cancellation, amplitude compression or amplification and speech enhancement. The equipment 10 exploits the completely-in-the-canal wearing/fitting configuration and produces no perceptually relevant acoustical distortion of the normal filtering properties of the outer ear. Of note, these properties are directionally sensitive and distortions cannot be compensated for without having prior knowledge as to the direction of the incoming sound. The distortion of the transmission properties of the auditory canal 14 produced by occluding the canal can be compensated for using a single gain function in the signal processor 24 that approximates the resonance gain of the canal. Changes to the overall input impedance of the outer ear resulting from the occlusion of the auditory canal 14 by the earphone 40 may result in changes in the filtering characteristics of the outer ear (for instance increased gain of the resonant modes of the concha - the cavity at the entrance of the ear canal). However, those skilled in the art will recognize that virtual auditory objects rendered using filter functions recorded by microphones occluding the canal are of high fidelity and do not degrade the appreciation of, or the performance of tasks reliant on, the spatial information so rendered. It will be appreciated by persons skilled in the art that numerous variations and/or modifications may be made to the invention as shown in the specific embodiments without departing from the spirit or scope of the invention as broadly described. The present embodiments are, therefore, to be considered in all respects as illustrative and not restrictive.

Claims

CLAIMS:
1. A method for the delivery of augmented-reality audio, the method including the steps of: creating an occlusion in an auditory canal of a listener so as to limit the transmission of external sounds in the range of human hearing from the environment to at least one eardrum of the listener; arranging an acoustic sensor within the auditory canal distally of the occlusion and using the sensor to sense the external sounds as they arrive at the sensor; arranging an acoustic actuator within the auditory canal proximally of the occlusion; receiving auxiliary input audio signals to be rendered as virtual audio to the listener; and using the actuator to deliver an output audio signal to the eardrum, the output audio signal constituting at least one of input audio signals corresponding to the external sounds, the auxiliary input audio signals and combinations of the foregoing.
2. The method of claim 1 which includes selecting the acoustic sensor so as to minimise disruption to acoustic filtering of an outer ear of the listener.
3. The method of claim 1 or claim 2 which includes implementing the sensor as a microphone with an input signal frequency range comparable to that of normal human hearing.
4. The method of any one of the preceding claims which includes implementing the acoustic actuator as a miniature loudspeaker with a frequency range comparable to that of normal human hearing.
5. The method of any one of the preceding claims which includes processing the auxiliary audio signals using methods of virtual auditory space to create the illusion for the listener that sounds representative of the auxiliary audio signals originate at specific locations in the listener's personal auditory space around the listener's head.
6. The method of any one of the preceding claims which includes using signal processing to modify and mix any combination of input audio signals to produce the output audio signal fed to the acoustic actuator.
7. The method of claim 6 which includes processing the input audio signals corresponding to the external sounds to one of amplifying and attenuating the input audio signals corresponding to the external sounds.
8. The method of claim 6 or claim 7 which includes processing at least one of the input audio signals corresponding to the external sounds and the auxiliary audio signals to perform enhancement of at least one class of sounds of interest within the external sounds.
9. The method of any one of claims 6 to 8 which includes processing the input audio signals corresponding to the external sounds to effect active noise cancellation.
10. The method of any one of claims 6 to 9 which includes processing the input audio signals corresponding to the external sounds to assist in protecting the listener's hearing.
11. The method of any one of the preceding claims which includes playing back the output audio signal through the acoustic actuator with a time delay after sensing of the input audio signals corresponding to the external sounds by the sensor.
12. The method of any one of the preceding claims which includes transmitting the input audio signals corresponding to the external sounds to an external device.
13. The method of any one of the preceding claims which includes using a device controller to control operation of at least certain components.
14. The method of any one of the preceding claims which includes transmitting audio signals to other devices.
15. The method of any one of the preceding claims which includes receiving signals from other devices.
16. The method of claim 15 which includes receiving information and/or instructions from the other devices in regard to how the auxiliary audio signals are to be rendered so as to generate acoustic characteristics associated with a virtual environment of an auditory object.
17. The method of claim 16 which includes manipulating audio filters used to generate the auxiliary input audio signals to modify the auditory object in virtual auditory space.
18. The method of any one of the preceding claims which is implemented at both ears of the listener.
19. Equipment for the delivery of augmented-reality audio, the equipment including: an occluder for occluding an auditory canal of a listener so as to limit the transmission of external sounds in the range of human hearing from the environment to at least one eardrum of the listener; a sensor for sensing the external sounds, the sensor being arranged on one side of the occlusion to lie within the auditory canal distally of the occlusion; an acoustic actuator arranged on an opposed side of the occlusion so as to lie in the auditory canal proximally of the occlusion; and a receiver for receiving auxiliary input audio signals from a source of the auxiliary input audio signals to be rendered as virtual audio to the listener, the receiver and the sensor being in communication with the actuator so that the actuator is operable to deliver an output audio signal to the eardrum, the output audio signal constituting at least one of input audio signals corresponding to the external sounds, the auxiliary input audio signals and combinations of the foregoing.
20. The equipment of claim 19 in which the sensor is configured to minimise disruption to acoustic filtering of an outer ear of the listener.
21. The equipment of claim 19 or claim 20 in which the sensor is in the form of a microphone having a frequency range comparable to that of normal human hearing.
22. The equipment of any one of claims 19 to 21 in which the actuator is in the form of a loudspeaker with a frequency range comparable to that of normal human hearing.
23. The equipment of any one of claims 19 to 22 which includes a signal processor for processing the input audio signals corresponding to the external sounds and the auxiliary input audio signals prior to the signals being received by the actuator.
24. The equipment of claim 23 in which the signal processor modifies and mixes any combination of input audio signals to produce the output audio signal fed to the acoustic actuator. 5
25. The equipment of claim 23 or claim 24 in which the signal processor effects one of amplification and attenuation of the input audio signals corresponding to the external sounds.
10 26. The equipment of any one of claims 23 to 25 in which the signal processor processes at least one of the input audio signals corresponding to the external sounds and the auxiliary audio signals to perform enhancement of at least one class of sounds of interest within the external sounds.
15 27. The equipment of any one of claims 23 to 26 in which the signal processor processes the input audio signals corresponding to the external sounds to effect active noise cancellation.
28. The equipment of any one of claims 23 to 27 in which the signal processor 0 processes the input audio signals corresponding to the external sounds to assist in protecting the listener's hearing.
29. The equipment of any one of claims 23 to 28 in which the signal processor includes a time delay for playing back the output audio signal through the acoustic 5 actuator with a time delay after sensing of the input audio signals corresponding to the external sounds by the sensor.
30. The equipment of any one of claims 23 to 29 which includes an amplifier for amplifying signals including the signals from the sensor, a signal output from the signal 0 processor and the auxiliary input audio signals.
31. The equipment of any one of claims 19 to 30 which includes an external transmitter device for transmission of the input audio signals corresponding to the external sounds sensed by the sensor. 5
32. The equipment of any one of claims 19 to 31 which includes a device controller for enabling the listener to control operation of at least certain components.
33. The equipment of any one of claims 19 to 32 in which the receiver is configured to be received in the auditory canal.
34. The equipment of claim 33 in which the receiver is contained in the occluder.
35. The equipment of any one of claims 19 to 34 in which the receiver is in wireless communications with the source.
36. The equipment of any one of claims 19 to 35 in which the receiver forms part of a transceiver to enable signals to be transmitted to other devices.
37. The equipment of any one of claims 19 to 36 which includes an adjustable audio filter arrangement used to enable the auxiliary input audio signals to be manipulated to modify an auditory object in virtual auditory space.
38. The equipment of any one of claim 19 to 37 which includes an energy storage unit to supply power.
39. The equipment of any one of claims 19 to 38 which is configured for each ear of the listener.
PCT/AU2004/000902 2003-07-04 2004-07-02 The production of augmented-reality audio WO2005004534A1 (en)

Priority Applications (2)

Application Number Priority Date Filing Date Title
AU2003903414 2003-07-04
AU2003903414 2003-07-04

Publications (1)

Publication Number Publication Date
WO2005004534A1 true true WO2005004534A1 (en) 2005-01-13

Family

ID=31983048

Family Applications (1)

Application Number Title Priority Date Filing Date
PCT/AU2004/000902 WO2005004534A1 (en) 2003-07-04 2004-07-02 The production of augmented-reality audio

Country Status (1)

Country Link
WO (1) WO2005004534A1 (en)

Cited By (10)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
EP1519625A2 (en) * 2003-09-11 2005-03-30 Starkey Laboratories, Inc. External ear canal voice detection
EP1796427A1 (en) * 2005-12-07 2007-06-13 Phonak AG Hearing device with virtual sound source
WO2008119122A1 (en) * 2007-03-30 2008-10-09 Personal Audio Pty Ltd An acoustically transparent earphone
US7580536B2 (en) 2003-08-11 2009-08-25 Vast Audio Pty Ltd. Sound enhancement for hearing-impaired listeners
JPWO2008099590A1 (en) * 2007-02-14 2010-05-27 パナソニック株式会社 Optical disk device
US8009848B2 (en) 2005-06-27 2011-08-30 Phonak Ag Hearing device system, hearing device maintenance system, and method for maintaining a hearing device system
WO2011161487A1 (en) * 2010-06-21 2011-12-29 Nokia Corporation Apparatus, method and computer program for adjustable noise cancellation
EP2413615A2 (en) * 2010-07-28 2012-02-01 Pantech Co., Ltd. Apparatus and method for merging acoustic object information
US8477973B2 (en) 2009-04-01 2013-07-02 Starkey Laboratories, Inc. Hearing assistance system with own voice detection
US9219964B2 (en) 2009-04-01 2015-12-22 Starkey Laboratories, Inc. Hearing assistance system with own voice detection

Citations (2)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
US20010046301A1 (en) * 2000-05-24 2001-11-29 Matsushita Electric Industrial Co., Ltd. Active noise control system
US20030035551A1 (en) * 2001-08-20 2003-02-20 Light John J. Ambient-aware headset

Patent Citations (2)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
US20010046301A1 (en) * 2000-05-24 2001-11-29 Matsushita Electric Industrial Co., Ltd. Active noise control system
US20030035551A1 (en) * 2001-08-20 2003-02-20 Light John J. Ambient-aware headset

Cited By (22)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
US7580536B2 (en) 2003-08-11 2009-08-25 Vast Audio Pty Ltd. Sound enhancement for hearing-impaired listeners
US7929713B2 (en) 2003-09-11 2011-04-19 Starkey Laboratories, Inc. External ear canal voice detection
EP1519625A3 (en) * 2003-09-11 2006-12-06 Starkey Laboratories, Inc. External ear canal voice detection
US9036833B2 (en) 2003-09-11 2015-05-19 Starkey Laboratories, Inc. External ear canal voice detection
EP1519625A2 (en) * 2003-09-11 2005-03-30 Starkey Laboratories, Inc. External ear canal voice detection
US9369814B2 (en) 2003-09-11 2016-06-14 Starkey Laboratories, Inc. External ear canal voice detection
US8009848B2 (en) 2005-06-27 2011-08-30 Phonak Ag Hearing device system, hearing device maintenance system, and method for maintaining a hearing device system
EP2271136A1 (en) * 2005-12-07 2011-01-05 Phonak Ag Hearing device with virtual sound source
EP2273800A1 (en) * 2005-12-07 2011-01-12 Phonak Ag Hearing device with virtual sound source
EP1796427A1 (en) * 2005-12-07 2007-06-13 Phonak AG Hearing device with virtual sound source
JPWO2008099590A1 (en) * 2007-02-14 2010-05-27 パナソニック株式会社 Optical disk device
WO2008119122A1 (en) * 2007-03-30 2008-10-09 Personal Audio Pty Ltd An acoustically transparent earphone
US9699573B2 (en) 2009-04-01 2017-07-04 Starkey Laboratories, Inc. Hearing assistance system with own voice detection
US8477973B2 (en) 2009-04-01 2013-07-02 Starkey Laboratories, Inc. Hearing assistance system with own voice detection
US9712926B2 (en) 2009-04-01 2017-07-18 Starkey Laboratories, Inc. Hearing assistance system with own voice detection
US9094766B2 (en) 2009-04-01 2015-07-28 Starkey Laboratories, Inc. Hearing assistance system with own voice detection
US9219964B2 (en) 2009-04-01 2015-12-22 Starkey Laboratories, Inc. Hearing assistance system with own voice detection
US9858912B2 (en) 2010-06-21 2018-01-02 Nokia Technologies Oy Apparatus, method, and computer program for adjustable noise cancellation
US9275621B2 (en) 2010-06-21 2016-03-01 Nokia Technologies Oy Apparatus, method and computer program for adjustable noise cancellation
WO2011161487A1 (en) * 2010-06-21 2011-12-29 Nokia Corporation Apparatus, method and computer program for adjustable noise cancellation
EP2413615A3 (en) * 2010-07-28 2013-08-21 Pantech Co., Ltd. Apparatus and method for merging acoustic object information
EP2413615A2 (en) * 2010-07-28 2012-02-01 Pantech Co., Ltd. Apparatus and method for merging acoustic object information

Similar Documents

Publication Publication Date Title
US5492129A (en) Noise-reducing stethoscope
US6275596B1 (en) Open ear canal hearing aid system
US20040208333A1 (en) Directional hearing enhancement systems
EP2237573A1 (en) Adaptive feedback cancellation method and apparatus therefor
US20090154739A1 (en) Systems and methods employing multiple individual wireless earbuds for a common audio source
US20060182287A1 (en) Audio monitoring system
US20090041260A1 (en) Active noise cancellation in hearing devices
US7310427B2 (en) Recreational bone conduction audio device, system
US20070121956A1 (en) Device and method for integrating sound effect processing and active noise control
EP0421681A2 (en) Electro-acoustic transducer and sound reproducing system
US20080008341A1 (en) Method and apparatus for a binaural hearing assistance system using monaural audio signals
US5117461A (en) Electroacoustic device for hearing needs including noise cancellation
US20080112581A1 (en) Vibrating earphone with enhanced base sound effect
EP1640972A1 (en) System and method for separation of a users voice from ambient sound
US5001763A (en) Electroacoustic device for hearing needs including noise cancellation
US20080118078A1 (en) Acoustic system, acoustic apparatus, and optimum sound field generation method
US20090116657A1 (en) Simulated surround sound hearing aid fitting system
US20130051585A1 (en) Apparatus and Method for Audio Delivery With Different Sound Conduction Transducers
US20110293105A1 (en) Earpiece and a method for playing a stereo and a mono signal
US20130208909A1 (en) Dynamic hearing protection method and device
US20110301729A1 (en) Sound system and a method for providing sound
US20070147635A1 (en) System and method for separation of a user's voice from ambient sound
US6771778B2 (en) Method and signal processing device for converting stereo signals for headphone listening
US20070086600A1 (en) Dual ear voice communication device
US20150163602A1 (en) Hearing aid device for hands free communication

Legal Events

Date Code Title Description
AK Designated states

Kind code of ref document: A1

Designated state(s): AE AG AL AM AT AU AZ BA BB BG BR BW BY BZ CA CH CN CO CR CU CZ DE DK DM DZ EC EE EG ES FI GB GD GE GH GM HR HU ID IL IN IS JP KE KG KP KR KZ LC LK LR LS LT LU LV MA MD MG MK MN MW MX MZ NA NI NO NZ OM PG PH PL PT RO RU SC SD SE SG SK SL SY TJ TM TN TR TT TZ UA UG US UZ VC VN YU ZA ZM ZW

AL Designated countries for regional patents

Kind code of ref document: A1

Designated state(s): BW GH GM KE LS MW MZ NA SD SL SZ TZ UG ZM ZW AM AZ BY KG KZ MD RU TJ TM AT BE BG CH CY CZ DE DK EE ES FI FR GB GR HU IE IT LU MC NL PL PT RO SE SI SK TR BF BJ CF CG CI CM GA GN GQ GW ML MR NE SN TD TG

121 Ep: the epo has been informed by wipo that ep was designated in this application
122 Ep: pct app. not ent. europ. phase