TECHNICAL FIELD
-
The present invention relates to wearable microphone array apparatus comprising at least two microphones within a garment, further comprising a control unit within the garment connected with the microphones and adapted to receive signals from the microphones, wherein the control unit is adapted to provide an output signal related to the localization of a sound source upon reception of a sound signal by at least one of the microphones.
PRIOR ART
-
A wearable microphone array apparatus with the features of the preamble of claim 1 is known from the master thesis from
Ravi R Shenoy "Design of e-textiles for acoustic applications" written at the Virginia polytechnic institute in 2003.
-
The publication from Ravi Shenoy discloses a way to implement a wearable microphone array for sound localization. Five microphones are distributed over the surface of a garment to provide electrical signals upon reception of a sound by the different microphones. These electrical signals are sent to a low integrated electronic for amplification of these signals. Analog electrical signals are used for signal transmission between microphones and board. The amplified signals are then transmitted to an external analog to digital conversion board connected to a desktop computer for future processing.
-
Another wearable phased array for sound localization and enhancement is known from
Sumit Basu, Steve. Schwartz and Alex Pentland from the article "Wearable phased array for sound localization and enhancement" in Proceedings of the IEEE Intl. Symposium on Wearable Computing 2000, Atlanta, USA. Microphones are disposed on electronic boards that are attached to a textile. Three USB microphones are distributed in this way and connected to a desktop computer for future processing. It is an object of this prior art to provide high quality audio without the use of a headset microphone. Furthermore the problem of speaker-change detection is addressed within this document.
-
Although it is not mentioned within these documents, one important application of such garments is providing a hearing aid for impaired people. These people more and more request unnoticeable hearing aids which are especially not viewable by third persons.
-
Such hearing aids that are able to filter the sound according to their incoming direction are as such inherently limited by the spatial extent of the device, which means the distance between microphones should compare with main wavelengths of a filtered sound, which results in limited performance improvement.
-
In order to circumvent this physical limitation
EP 1 540 991 discloses hearing spectacles, but the use of microphones in spectacles limits the possibilities of signal treatment.
-
EP 2 350 683 provides a wearable sound source localization device which is limited to gunshot detection and is not included in a fabric but enclosed in an element placed on the users shoulder.
SUMMARY OF THE INVENTION
-
Based on this prior art, the invention is intended to provide an improved approach for a sound improvement device, which can especially also be used as hearing aid. However, the application of the invention is not limited to such hearing aids.
-
The apparatus according to the invention comprises a reproduction device connected with the control unit and adapted to reproduce a real-time signal to a user of the apparatus. Said user can be the wearer or a distant user with remote access to the information.
-
The invention comprises a piece of clothing, made of, for example, flexible material, which can also be the fabric of an underwear or the felt of a headdress, that is used as a holding structure for other elements. These other elements of the system are arranged on the surface or inside the fabric and include two or more microphones collecting sound (a mechanical wave-field of frequency from 1Hz to 10MHz) measures directly as a digital signals or comprising an analog to digital converter for received analog sound signals, a power source, a processing unit and a communication unit for a wired or wireless transmittal of the data to other systems. Said microphones can be replaced in the invention by any device able to measure the sound, notably but not limited to, ultrasound-transducers or accelerometers. The connection between these elements on the fabric are provided via conductive wires, weaved or hidden in the fabric itself. All the elements interact in a way that the signal coming from the microphones are processed in real time and can be sent to the user via the communication unit with a minimal delay that is not noticeable for a human recipient. This is especially important within the hearing aid application. It is also important for other applications where the reaction of the human user who wears the apparatus or who remotely monitors the output signal of such a device upon detection of a sound is intended.
-
The described invention can be used for a sound reproduction device which outputs the received sound signal to an ear of the user. Such sound reproduction unit can be placed near a user ear, inside the ear canal or implanted. Said sound reproduction devices receive the sound signal from the transmission unit preferably wirelessly but not limited to.
-
When the described invention is used as an input for a hearing aid then the processing unit outputs one or two sound signals that are transmitted e.g. wireless by the communication unit to the hearing aid usually placed inside or near the user's ear. The processing unit is then able to detect the sound sources and sort them out by their direction of arrival. The processing unit could also select a subset of sound sources that are considered relevant for the user in the present situation and could selectively extract the sound produced by the sources from the input sound, increasing by that means the relevant signal-to-noise ratio (SNR) of the reproduced sound.
-
It is an advantage over known wearable microphone arrays, as mentioned above, for implementing acoustic applications, that the present device is embedded in the fabric in a way that the fabric can be worn under clothes without being noticeable.
-
The processing, powering and communication elements are embedded in the garment whereas within the prior art documents the recording, processing and powering is mainly done externally. Also these documents do not provide communication units integrated into the system. The invention is autonomous and can thus be worn all day long by a user without inconvenience.
-
The overall processing scheme satisfies the real time constraint; whereas the prior art documents make no reference to real-time computing: the sound as received is recorded and processed later.
-
In contrary to state of the art multi-microphone hearing aids where microphones are distant of few centimetres, the large spatial extension of the disclosed apparatus enables covering a wide spectrum of frequencies which improves the directionality of the array and the intelligibility of recorded voices. A greater number of microphones, e.g. 20 microphones, can be used, especially more than 5 to 10.
-
The distance between the microphones and the hearing aid in such an application avoids the problem of feedback even when providing a highly amplified output signal for severely hearing impaired people.
-
Within further preferred embodiments of the invention, such a device is used as a sound sources localization device that enables the user wearing the system or a remote user to monitor in real time the direction of arrival of a given sound. The communication device sends the signals to a distance receiver and/or to a local reproduction device. This reproduction device can be a touch screen worn by the user that displays the localization of a sound source and characteristics or give a direct feedback, e.g. through providing a stimulation signal in the direction of the sound. Such a stimulation signal can preferably be combined with the electronic placement of the microphones and may comprise a vibrating unit or a shock inducing unit, which alerts the user from the direction of a sound source.
-
A device using the advantages of the invention comprises a fabric, a plurality of transducers placed on, under or inside the fabric, one or more power supplies placed on, under or inside the fabric that each powers one or more of the transducers and at least one control unit as computing and signal processing unit which processes in real-time at least one signal from the said transducers and outputs at least one processed signals. The transducers are microphone units. The fabric can be a garment and especially an underwear. A communication unit for transferring the processed signal can be a RF unit, an infrared unit, Bluetooth unit or a Wifi unit. Such communication can comprise a transmittal to a sound reproduction device, e.g, a hearing aid and the communication can comprise the transmittal of information relating to the location of the sound source to the user. In case of a hearing aid, the computing unit can use the input microphone signals to generate an output sound signal that has preferred directivity characteristics.
-
The reproduction device can comprise one or more actuators adapted to transmit a tactile feedback signal on the inside of the garment towards the user. Such a signal can be a vibration when the actuator is a vibrator. It can be an electric discharge, if the reproduction device is an electrode. The actuator can also comprise an embossment creating or pin extending unit, so that the user perceives this tactile feedback.
-
On the other hand, reproduction device can comprise a hearing aid, especially a wireless hearing aid, which optionally receives control signals through a wireless interface from a transmission unit connected with the control unit.
-
The control unit can comprise a speech-to-text conversion unit adapted to provide an output signal based on speech or specific predetermined sound signals to a reproduction device. Specific predetermined sound signals can be the horn signal of an ambulance, the acoustic signal of a police car, the sound of an incoming shell, the ringing sound of a telephone, of a door bell or the sound of any security relevant source.
-
The reproduction device can comprise a visual display screen, e.g. on a tablet computer, which can be connected by wire or wireless to the control unit. Said control unit then comprises a coding unit providing an output signal for a visual display of the speech or specific predetermined sound signals.
-
The reproduction device can comprise an array of actuators forming a Braille cell, e.g. an array of 2 times 3 or 2 times 4 actuators, which can be controlled at the same time to transmit Braille coded information. Then of course the control unit comprises a coding unit providing an output signal for a Braille representation of the speech or specific predetermined sound signals within the Braille cell.
-
The reproduction device can comprise at least or only one actuator and then the control unit comprises a coding unit providing an output signal for a Morse or another sequentially coded representation of the speech or specific predetermined sound signals.
-
The reproduction device can also comprise a plurality of at least two, preferably four or more actuators provided in a preferably similar or equal angularly spaced relationship on the circumference of the garment, wherein the control unit comprises a source localization unit and is adapted to actuate in real-time the actuator being closest in view of the direction of a specific sound event, so that the user is incited to turn himself towards the source of speech, especially if unwanted noise sources are filtered out.
-
The apparatus according to the invention, especially when the reproduction unit is either worn by the wearer or located in a distant place (e.g. at the command unit in a military application case) comprises within the reproduction unit a visual display screen that displays information related to the location of sound events optionally together with characteristics of said sound events. It is also interesting to provide for one reproduction unit that it receives the signal from several devices and merges them to display an improved map of the sound events.
BRIEF DESCRIPTION OF THE DRAWINGS
-
Preferred embodiments of the invention are described in the following with reference to the drawings, which are for the purpose of illustrating the present preferred embodiments of the invention and not for the purpose of limiting the same. In the drawings,
- Fig. 1
- shows a front side view of a shirt as schematical representation of a device according to an embodiment of the invention;
- Fig. 2
- shows the shirt of Fig. 1 from the back side of the shirt;
- Fig. 3
- shows a front side view of a collar as schematical representation of a device according to a further embodiment of the invention;
- Fig. 4
- shows the collar of Fig. 3 from the back side of the collar;
- Fig. 5
- shows a front side view of a shirt as schematical representation of a device according to a further embodiment of the invention, based on the array distribution of Fig. 1;
- Fig. 6
- shows the shirt of Fig. 5 from the back side of the shirt;
- Fig. 7
- shows a block diagram of the connections of the device according to Fig. 1 as the basic device; and
- Fig. 8
- shows a block diagram of the connections of the device according to Fig. 1 with control elements realized within the signal processing unit for additional processing steps.
DESCRIPTION OF PREFERRED EMBODIMENTS
-
Fig. 1 shows a front side view of a shirt 10 as garment as a schematical representation of a device according to an embodiment of the invention. The device comprises a plurality of microphone units 20 distributed on the front side of shirt 10. They are provided in three lines of three microphones 20 on the height of the torso area of the shirt as well as comprising two microphone units 20 on the shoulder areas of the shirt.
-
A microphone unit 20 preferably comprises a microphone as such, as well as signal processing elements to provide a pre-transformed electrical signal relating to the detected sound as usual. The transmission is preferably done digitally in order to improve noise immunity of the sound signal. This is especially important if the user has a mobile communication device e.g. cell phone, wifi transmitter. Further noise immunity can be obtained using differential signal e.g LVDS. The microphones or the signal processing element part of the microphone units are connected via electrical wires 21, shown as solid lines, with a specific signal combining signal processing unit 30, which provides e.g. a localisation information of a specific sound from e.g. a speaker as known from the prior art. The signal transmission is then prepared to be transmitted to the transmission unit 40 through wire 31. Transmission unit 40 comprises a wireless transponder, shown as the symbol 41 for wireless transfer to the remote hearing aid, which is usually an ear loudspeaker, not shown in Fig. 1.
-
Power lines 22 connect a battery 23 with each of the microphones 20 as well as the signal processing unit 30.
-
Fig. 2 shows the shirt 10 of Fig. 1 from the back side of shirt 10. As usual all identical or similar features of the various embodiments receive the same of similar reference numerals. The back side of shirt 10 also comprises eleven microphone units 20, three columns with three lines forming almost a rectangle on the back of shirt 10 as well as two additional shoulder microphone elements. These two elements are provided almost on the shoulder but separate from the elements connected with the front side microphones 20. It could be the case that these microphones on the shoulder parts are provided quite near one another. The signal handling can be separated between the signals issued by the back side and the front side of the shirt reducing the localization effort of the signal processing unit 30 through previous filtering. It is preferred that there is nevertheless one single battery for both arrays, the front and the back array, and one single signal processing unit 30.
-
Fig. 3 shows a front side view of a collar 110 as schematical representation of a device according to a further embodiment of the invention; and Fig. 4 shows the collar 110 of Fig. 3 from the back side of the collar 110;
-
Collar 110 has a superposing area 111, which can be used, e.g. with a velcro type connection to close the collar 110 around the neck of a user, as shown in Fig. 3 and 4.
-
Here one array of microphones 20 is provided on the front side of the collar 110 in two lines of 4 and 6 microphone units 20. The microphone units are interconnected and connected with the signal processing unit 30 via lines 21 as well as via power lines 22 with the battery 23.
-
Fig. 5 shows a front side view of a shirt 10 as schematical representation of a device according to a further embodiment of the invention, based on the array distribution of Fig. 1. Fig. 6 shows the shirt of Fig. 5 from the back side of the shirt. Beside the microphone units 20 which are disposed identically to the embodiment of Fig. 1 there are provided actuators 50.
-
The embodiment as shown in Fig. 5 and 6 shows a combination of two embodiments which can be used separately or together. One embodiment uses two times two actuators 50 distributed on the front as well as on the back of the garment 10, as shown in Fig. 6. Beside these upper actuators 50 or replacing these actuators, within another embodiment, there is provided a line of here 9 plus 9 actuator elements 50 around the waist line of the garment 10. The actuator units 50 in the lower area provide a complete 360 degree coverage with actuators 50 in view of an user of this garment 10, almost in regularly spaced intervals in a horizontal direction. Eighteen actuators 50 allow an angle resolution of approximately 360/18 = 20 degrees with a slightly higher value on the side of the body. It is also possible to use a different number of actuators, like 4 plus 4 which gives roughly an angle resolution of 45 degrees (=360/8).
-
The advantage of the use of these actuators 50 becomes clear in view of possible applications. Profoundly hearing impaired people are not able to understand speech if they cannot use side channel information as lipreading. The microphone array with microphone units 20 analyses the incoming sound, detects the interesting source like speech with prior art techniques of sound analysis and can identify the direction where the speaker is standing. Then the actuator 50 in the direction of the source is activated and performs his function.
-
Actuators 50 can comprise a vibrator and as such give a tactile feedback on the skin or the under garment of the user. It can also provide a different answer, like a heat pulse, a pin moving in direction of the skin of the user extending over the casing of the actuator 50. A contact actuator 50 is a transceiver that can transmit information to the wearer being in contact with its skin, e.g. a vibration device or an electrode are two preferred embodiments. When the two times two actuators 50 are used, it is also possible to provide height information of the sound source, when the upper or lower actuators 50 are activated.
-
Within one embodiment the sound source is analysed for the presence of speech and the speech is converted into the corresponding text message by the signal processing unit 30, 38 based on the different incoming signals from the microphone units 20 over the wires 21. This text message is then transmitted to the actuators 50 with a predetermined coding. This can be a Morse code on one or more specific actuators 50 or an array of two times three actuators 50 (two columns, three lines; not shown in the drawings; or eight actuators with a fourth additional line) is used as the usual known Braille coding to communicate the spoken words. It is also possible that the signal processing unit 30 provides a cable or wireless 41 link to a screen, e.g. on a tablet computer for a visual representation of the spoken words.
-
The aim of the actuators 50 is therefore at least twofold. The above mentioned possibility provides identification of a speaker, indication of the direction to the user and providing a coded (Braille, Morse as tactile or by visual means) signal on the spoken information. The processing unit 30 can also ensure that the speaker in front of the user wearing the garment 10 is the sound source which is used for speech decoding, since the microphone units 20 allow to distinguish this source in view of the orientation of the garment 10.
-
A further advantage of the system incorporated in the processing unit 30 is the localization of possibly dangers. The microphone units on the front, the back and different heights on the garment 10 allow a better analysis of incoming sound signals for the processing unit 30 which can also be programmed to discern different noise patterns, e.g. the signal of a police or ambulance, the specific signal of a tramway to attract the user's attention to such surrounding dangers. This can also include shouting people to warn. This is specifically useful, if such a sound source is localized at the back of the user so that activation of the backwards oriented actuators 50 provides a direct indication to a possibly problematic sound source in this direction.
-
There are also other applications as sound localization on the battlefield. Sound information is vital for soldiers on the field as it may enable to localize a hidden enemy who is shooting. This localization can be especially problematic with the ears because the direction of the shock wave created by a supersonic bullet does not directly point towards the shooter, inducing a moment of confusion that can be fatal. Additional problems are created by reverberant environment that prevent the localization by human ears. Electronics means can enable an accurate and instantaneous localization through the use of the array of microphone units 20. However, the spatial extension of the microphone array (radius bigger than 50cm) required for an accurate localization is unacceptable to mount on a firearm, or to carry on a battlefield. The present innovation can be used to overcome this problem by embedding the array of microphones 20 into the garments 10 of the soldier as the user so as to benefit from a large spatial extent as well as a reduced space requirement.
-
The device includes a reproduction part that enables a real-time and easy understanding of the acoustic scene. The preferred embodiment here includes a set of incorporated contact actuators 50 arranged on a circular way so as to cover all directions in view of a standing person. When a sound is localized by the signal processing unit 30, the actuator 50 that corresponds to the direction is activated.
-
Another application related to sound localization on the battlefield is to distantly monitor the position of the enemies. The localization of the gunshots given by the device worn by a soldier is transmitted to a control operation centre together with soldier location information e.g. GPS position of the soldier, and displayed on a real-time updated map. In this case, the reproduction device of the disclosed apparatus includes e.g. a screen displaying e.g. a map of the battlefield. The information from several apparatus can be further combined to be displayed on a single map.
-
It is noted that the signal wires 51 for the actuators 50 can use or be identical to the signal wires 21 of the microphone units 20, since they are both connected to the signal processing unit 30. It is also possible to combine the power wires 22 and 52.
-
In an embodiment not shown in the drawings, microphone units 20 and actuators 50 are combined on the same support, i.e. a printed circuit board. The actuators 50 can also be provided within the microphone units 20 within the same chip element.
-
Fig. 7 shows a block diagram of the connections of the device according to Fig. 1 as a basic device and Fig. 8 shows a block diagram of the connections of the device according to Fig. 1 with control elements realized within the signal processing unit 30 for additional processing steps. Same reference numerals are used in the block diagram of Fig. 7 and 8 for elements provided in the actual garment 10.
-
The power unit 23 comprises usually a battery and sometimes a converter. The power unit 23 is connected via power wires 22 to the array of microphone units 20, the processing unit 30 and, if provided the transmission unit 40. The signal transmission is realized via wires 21 from the microphones 20 to the signal processing unit 30 and from the signal processing unit 30 towards the optional transmission unit 40 via wire 31. The signal wires 21, 31 (and 51) can be replaced by wireless transmission means.
-
Within Fig. 8, the signal wires 21 from the microphone units 20 are provided to the signal processing unit 30 of Fig. 7 which comprises a source localization unit 35, a relevance estimation unit 36, a beamformer 37, a speech-to-text conversion unit 38, a coding unit 39 and a reproduction device 130.
-
As explained above the signal processing unit 30 comprises a source localization unit 35 which provides as an output on signal wire 135 an information relative to the angular position of the sound source e.g. azimuth and altitude angles relatively to the user wearing the garment 10. A relevance estimation unit 36 also receives the source sound signal and the source localization signal. The relevance estimation unit 36 determines the relative relevance of the signals, depending on the conditions. The voice of an interlocutor should be magnified whereas other conversations from other groups of person should be suppressed (known in the prior art as the cocktail party problem). Outside of buildings an emergency signal from an ambulance is an important relevant sound source whereas indoor, it is not the case. This relevancy signal is transmitted via wire 136 to the beamformer unit 37. The beamformer unit or spatial filtering unit 37 is a signal processing technique used in sensor arrays for directional signal transmission or reception. This is achieved by combining elements in the array in such a way that signals at particular angles experience constructive interference while others experience destructive interference. Beamforming is used on the receiving end in order to achieve spatial selectivity and provides an important receive/transmit gain. The combination of units 35, 36 and 37 relates to an adaptive beamforming used to detect and estimate the signal-of interest at the output of the microphone units 20 by means of a spatial filtering and interference rejection. For the spatial filtering a least-squares method can be used.
-
The thus optimised incoming sound signal over wires 21 is forwarded via wire 137 to the speech-to-text conversion unit providing the spoken words in a computer understandable form. These elements are then forwarded to a coding unit 39, which codes the information in a way which is understood by the reproduction device 130. This can comprise a 2x3 or 2x4 Braille coding for transmittal to the actuators 50 or a simpler Morse-code transfer to a single or a specific number of actuators 50 for sequential transmittal of the spoken words or as a coding for a visual transfer into ASCII-codes for display on a screen as reproduction device 130. Furthermore hearing aids for the ear, either implanted or to be introduced into the ear canal are such reproduction devices 130. In such a case, preferably a wireless transmitter unit 40 is used but a wire transfer as line 139 is also possible.
LIST OF REFERENCE SIGNS
-
| 10 |
garment |
41 |
wireless transfer |
| 20 |
microphone unit |
50 |
actuator |
| 21 |
signal wire |
51 |
signal wire |
| 22 |
power wire |
52 |
power wire |
| 23 |
power source |
110 |
Collar |
| 30 |
signal processing unit |
111 |
superposing area |
| 31 |
transmission transfer wire |
130 |
reproduction device |
| 35 |
source localization unit |
135 |
signal wire |
| 36 |
relevance estimation unit |
136 |
signal wire |
| 37 |
beamformer |
137 |
signal wire |
| 38 |
speech-to-conversion unit |
138 |
signal wire |
| 39 |
coding unit |
139 |
signal wire |
| 40 |
transmission unit |
|
|
