KR102578147B1 - Method for detecting user voice activity in a communication assembly, its communication assembly - Google Patents

Abstract

Many headsets include automatic noise cancellation (ANC), which dramatically reduces perceived background noise and improves the user listening experience. Unfortunately, voice microphones in these devices often capture the ambient noise that the headsets output to other users during phone calls or other communication sessions. In response to this, many headsets and communication devices provide passive mute circuitry, but users often forget to turn the mute on and/or off, creating additional problems as they communicate. To solve this, the present inventors have, among other things, used a matrix of transfer functions to detect the absence or presence of user speech based on two signals derived from the microphones, automatically muting the voice microphone without user intervention, and An exemplary headset was designed to unmute. Some embodiments strengthen the relationships between feedback and feedforward signals in the ANC circuit to detect user speech, avoiding adding extra hardware to the headset. Other embodiments also enhance the speech detection function to activate and deactivate keyword detectors, and/or sidetone circuits, thus extending battery life.

Description

Method for detecting user voice activity in a communication assembly, its communication assembly

Copyright Notice and Permission

Portions of this patent document contain material that is subject to copyright protection. The copyright holder has no objection to facsimile reproduction by anyone of the patent document or patent disclosure as it appears in the U.S. Patent and Trademark Office patent files or records, but otherwise reserves all copyright rights whatsoever. The following notice applies to this document: Copyrightⓒ 2017, AVNERA CORPORATION.

Related Applications

This application relates to U.S. Patent Application No. 15/711,793, filed September 21, 2017; and U.S. Provisional Patent Application No. 62/456,100, filed February 7, 2017; U.S. Provisional Patent Application No. 62/459,055, filed February 14, 2017; and U.S. Provisional Patent Application No. 62/532,964, filed July 14, 2017. All four applications are hereby incorporated by reference in their entirety.

Technology field

Various embodiments of the present invention generally relate to automatic detection of user voice activity in various types of headsets, such as those with automatic noise reduction.

Headsets typically include a microphone along with one or two ear devices or earplugs that are worn over, on, or within the user's ears to facilitate electronic communications. Many modern headsets also include automatic noise reduction (ANR) or automatic noise cancellation (ANC) circuitry to automatically detect and cancel significant portions of ambient noise, thereby allowing the user to Improve your listening experience. The ANC circuit is simple in principle but complex in implementation, and many headsets contain the same six microphones: two for feedback (fb) ANC, two for feedforward (ff) ANC, and a user microphone. 1 or 2 to pick up voice signals.

One problem the inventors recognized is that while ANC headsets improve the listening experiences of users who wear them, they do little to improve the quality of signals transmitted from the headset to other devices. For example, in high noisy environments, users wearing headsets with mobile phones typically manually switch their phones to mute mode to prevent background noise from being transmitted by the phone to other phones and devices. are asked to do so. Although this is effective in preventing unwanted noise transmission, it often results in gaps in communications as users who mute their phones forget to unmute them when they start speaking again. Moreover, in conference calls where many users manually mute and unmute, the potential for communication gaps is even greater.

In attempting to solve the problem, some designers have provided circuitry to monitor voice mic output and automatically mute or unmute based on a comparison of the mic output level to one or more thresholds. However, the monitoring circuit suffers from at least two problems that have prohibited its use in mobile phones. Firstly, it consumes significant power and thus shortens battery life. Second, the circuit creates its own communication gaps not only because of its slow response to stopping and starting speech, but also because it confuses external noises, for example the voices of nearby people, with the user's voice. .

Accordingly, the present inventors have recognized the need for better ways to reduce transmission of ambient noise through ANC and other types of headsets.

To address one or more of these and/or other needs or problems, the present inventors have proposed, among other things, one method for automatically detecting the presence or absence of user speech and activating associated muting or other voice- or speech-dependent functionality. The above exemplary systems, kits, methods, devices, assemblies, and/or components have been designed. Some embodiments include a voice mic, at least two control mics, and processing circuitry, where the processing circuitry is configured to mute the voice mic based on a relationship between the control mic output signals.

In a first embodiment, two control mics, for example a left earpiece mounted mic and a right earpiece mounted mic, are configured for substantially symmetrical acoustic coupling to the user's voice area. The processing circuitry determines whether the user is speaking based on the measured temporal symmetry -- e.g., phase relationship -- of the output signals from the two mics, with high symmetry indicating that the user's speech (arriving at both mics at approximately the same time) and low symmetry indicates ambient noise or non-user speech. The two control mics are ANC headset inner ear left and right feedback mics in some variations of the first embodiment. Some other variations use the complex coherence of sampled mic outputs to measure or estimate the phase relationship, activating a mute circuit whenever the real part of the complex coherence falls below a threshold.

In a second embodiment, the two control mics are configured for asymmetric acoustic coupling to the user's voice region, ie one mic has better acoustic coupling to the user's voice region than the other. The processing circuitry is configured to turn muting on or off based on the relative magnitude or energy level of the two mic output signals. For example, in some ANC embodiments, the mic with good acoustic coupling is a feedback error mic in the ANC earpiece and another is a feedforward mic. The processing circuitry compares the ratio of the feedback error mic output to the feedforward mic output to a threshold when determining whether to turn the mute function on or off. Some variations may use two mics placed on the boom or cord rather than ANC controlled mics.

The processing circuitry compares the ratio of the feedback error mic output to the feedforward mic output to a threshold when determining whether to turn the mute function on or off. Some variations may use two mics placed on a boom or cord, rather than ANC controlled mics. More general embodiments use two or more microphones to generate two signals, the first signal being more closely coupled to the user's speech than the second signal, and the second signal being more sensitive to ambient noise or sound than the first signal. more closely coupled.

A third embodiment estimates a transfer function that accounts for both phase and magnitude differences between signals from two mics, such as a feedback error mic and a feedforward ambient mic. Some variations of this embodiment may have no ANC mic and use other pairs of mics, such as a voice mic and another mic. The transfer function can be estimated using a variety of methods, such as Kalman filtering or other types of optimal estimators.

One or more embodiments also include other features. For example, some embodiments include an attenuator to mute music or other audio signals output through a loudspeaker within an earpiece associated with one or more of the microphones. Some embodiments include mute delay and hold features to reduce the risk of muting between spoken words or during brief pauses. Still other embodiments include buffer memory and look ahead functionality to ensure timely unmute of the voice mic and thus avoid partial loss of user speech.

Various embodiments are described herein with reference to the accompanying drawings (Figs) below. These drawings are annotated with reference numbers for various features and components, and such numbers are used as teaching aids in the following description, with like numbers referring to the same or similar features and components.
1 is a block diagram of an example headset system or assembly 100 corresponding to one or more embodiments of the present invention.
2 is a block diagram of an example method of operating the user voice activity detector portion of assembly 100, corresponding to one or more embodiments of the present invention.
3 is a half-conceptual system diagram of a headset system 300 corresponding to one or more embodiments of the present invention.
Figure 4 is a block diagram of a general electroacoustic system with multiple audio transducers and audio input signals, corresponding to one or more embodiments of the present invention.
Figure 5 is a block diagram of a user voice activity detector corresponding to one or more embodiments of the present invention.

This document, including the drawings and claims, describes one or more specific embodiments of one or more inventions. These embodiments, which are provided only to illustrate and teach the invention and not to limit it, are shown and described in sufficient detail to enable any person skilled in the art to implement or practice the invention(s). Moreover, where appropriate to avoid obscuring the invention(s), the description may omit certain information known to those skilled in the art.

1 shows an example ANR headset assembly or system 100 mounted on ears 102A and 102B of a user's head 101 (top view). The user head 101 includes left and right ears 102A and 102B, respectively, a mouth 103, and a user voice area 104. User voice area 104 includes a user mouth and voice box (not shown). Assembly 100 includes left and right earpieces 110A and 110B, optional headband 110C, in-line or boom voice microphone 120, processing circuitry 130, and host or external device 140. .

Take in-the-ear, over-the-ear, or on-the-ear configurations and define the vertical axis 110D. Capable earpieces 110A and 110B include respective ANR drivers (DA and DB), feedforward mics (FFA and FFB), and respective feedback mics (FBA and FBB). ANR drivers (DA and DB) divide the internal volumes of each earpiece 110A and 110B into front cavity and back cavity portions (not individually labeled). The feedforward peripheral mics (FFA and FFB) are located within or adjacent to the rear cavities and are positioned further away from the respective ears 102A and 102B than the feedback mics (FBA and FBB), so that they Ensures greater acoustic coupling to ambient noise than the feedback error mics (FBA and FBB) and less acoustic coupling to the user's head 101, user ear canals, and speech area 104. Feedback error mics (FBA and FBB) are similarly positioned within or on their respective earpieces to transmit the user's voice on axis 110D via head or bone (body) conduction paths 104A and 104B. Ensures generally equal or symmetrical acoustic coupling across regions 104. Additionally, feedback mics have generally symmetrical acoustic coupling to off-axis external noise sources, such as an ambient noise source (N), via air conduction paths (P1 and P2). (Feedback and feedforward mics within the same earpiece have highly asymmetric responses to user speech and ambient.) Microphones and drivers are coupled to processing circuitry 130 via respective wired or wireless communication links 131 and 132. do.

Processing circuitry 130, which in some embodiments takes the form of a digital signal processor with associated memory and other integrated components completely separate, partially or fully integrated within device 140, includes a set of input/output nodes ( 133), ANR processing module 134, user voice detection module 135, mute module 136, speech dependent modules 137, and on-the-ear detection module 137.

The input/output nodes 133 include a voice mic output signal node (Vm), a feedback mic left signal node (FBA(fbl)), a feedback mic right signal node (FBB(fbR)), and a feedforward mic left signal node ( FFA(ffl)), feedforward mic right signal node (FFB(ffl)), and destination device audio/music signal nodes (RxA RxB). (Rx, as used herein, may refer individually or collectively to RxA or RxB and may include a telephone audio signal.)

ANR processing module 134 provides circuitry and machine-executable instructions for canceling ambient noise within earpiece front volumes, contained within users' ear cavities. In particular, module 134 receives output signals from feedback mics (FFA and FFB), representing the sum of the total acoustic energy within each ear kernel or cavity. Module 134 also receives earphone audio signals from device 140, such as a smartphone, music player, two-way radio, or other electronic audio source. In response to the ambient noise signal from the feedforward microphones (FFA and FFB), the ANR processing circuit generates noise cancellation signals and mixes or sums the cancellation signals and the earphone audio signal (Rx) to the ANR drivers (DA and DB). ), which generates acoustic energy that cancels a substantial portion of the perceived ambient noise and provides an acoustic representation of the earphone audio signal. In some embodiments, ANR processing module 134 includes adaptive ANR processing, such as the type described in U.S. patent application Ser. No. 15/069,271, filed March 4, 2016, which is incorporated herein by reference. .

User voice activity detection (UVAD) module 135 uses feedforward mics (FFA and FFB) and It includes logic circuitry and/or stored machine-executable instructions for processing input signals derived from two or more of the feedback mics (FBA and FBB). Specifically, the example embodiment provides two methods for detecting user voice activity. (Some embodiments provide UVAD using any peripheral coupled mic in combination with a feedback mic. Thus, for example, a voice mic and a feedback mic could be used.)

The first method enhances insight because the user's head is sufficiently symmetrical that the acoustic paths 104A and 104B through the user's head are generally of the same length and substantially similar in material composition. This means that the speech components of the feedback mic outputs are substantially the same in magnitude and phase when the user is speaking and are substantially different, at least in phase, when the user is not speaking. In other words, the output signals from the feedback mics have nearly identical speech data components and thus exhibit high coherence (in phase relationship) when the user is speaking. Moreover, the distance of the two mics from the ambient noise is usually unequal or asymmetric, as indicated by paths P1 and P2 in Figure 1, meaning that the coherence is rotated in phase. (It may still be coherent, meaning that the magnitude may still be very close to 1, but the angle will not be zero, indicating an 'in-phase' situation.

Accordingly, the example embodiment uses the complex coherence of the left and right feedback error mic signals (fbL and fbR) within a threshold test to determine whether user voice activity is likely present or absent. If the threshold criterion is met, then a mute command signal is provided to mute module 136, otherwise unmute command signs are provided. More specifically, the criticality test takes the form of

Here, mean() represents the average function (over frequencies); real() represents the real part of the complex argument; Cxy(freq_range) represents the complex coherence of signals (x and y) over the frequency range indicated by freq_range; The subscript x represents the fbL error mic signal (left fb mic signal, FBA) and the subscript y represents the fbR error mic signal (right fb mic signal, FBB); CT represents the coherence threshold. The real part of complex coherence (Cxy) is used because it is a measure of how "in phase" two signals are. Note: abs(Cxy) is 1 if only one average is taken, but this can still be useful as important information is still in phase. The freq_range for which the average is calculated can be changed. However, because body-conducted sounds in the human voice box are primarily low frequencies, some embodiments use a range of 70 to 700 Hz or 100 to 500 Hz. In response to an indication from the Cxy detector circuitry or logic that the user is not speaking (i.e., no user speech), example embodiments use mute module 136 to silence one or more microphones, such as a voice mic, and/or Mutes or attenuates the output of feedforward microphones.

Additionally, in some embodiments, the coherence threshold (CT) ranges from 0.7 to 0.9 with slight variations in performance. Setting it very close to 1 makes it easier for the detector to produce false negatives (speech not detected when it is present) if there is significant background noise, and setting it very low makes it more prone to false positives (i.e. , which results in unmuting when there is no speech. In some embodiments, the coherence threshold may be dynamically adjusted based on system or environmental parameters, such as signal (speech) to noise ratio. That is, if the speech is much louder than the noise, then some embodiments shift the threshold closer to 1, for example 0.9, and if it is very loud, these embodiments shift the threshold level to avoid muting the user's speech. reduce. An exemplary threshold between 0.7 and 0.8, such as 0.75, is a good compromise.

This left-right symmetry based approach is effective, but vulnerable to on-axis ambient noise that may arise, for example, if the second speaker is on-axis to the user (or generally equidistant from the left and right earpieces). can do. It also suffers from poor signal-to-noise ratio (SNR). On-axis noise exhibits high coherence in feedback error microphones and will therefore be misperceived as speech. Interestingly, one way to identify these vulnerabilities is in Denmark. of This UVAD approach (e.g. left and right feedforward or positioning the headset (with left and right feedback mics, or more commonly, two mics configured to be approximately equidistant from the audio area). The headsets can then be noticed to automatically mute in response to noise sources that are approximately equidistant from the mics, eg, directly in front of, behind, and above the HATS.

To reject on-axis ambient noise sources, such as non-user speech, and enable better performance in noisy environments, some embodiments use a second method of user voice activity detection, which also uses a single ear This has the added advantage of being useful in one-piece systems, or more generally in any situation where two mics have asymmetric coupling to user and ambient sounds. This method ensures that the amount of energy generated by feedback mics (more generally, mics with better acoustic coupling to the user's speech region) is greater than the amount of energy generated by the user speaking due to the asymmetry of the acoustic paths of the speech signals to the feedback and feedforward mics. Enables substantially greater insight than occurs with feedforward mics when speaking rather than when not. Feedback mics are located in the front cavity of the earpieces within or near the user's ear canal. For this positioning, the feedback mics receive the user's voice energy via bone conduction paths 104A and 104B with very low attenuation and via air conduction with strong attenuation due to the seal typically present in ANR systems. Receives ambient noise. Therefore, the feedback mic to feedforward mic output ratios (fb/ff energy ratio) are much higher when a user's voice is present than that of ambient noise without speech, regardless of the noise direction. Because of this, some embodiments use the following logic to determine user voice activity:

Here, fb_left/ff_left represents the ratio of the energy of the output signal of the left feedback mic (FBA) to the energy of the output signal of the left feedforward mic (FFA), and DT represents the selected detection threshold for user speech. In this embodiment, DT is platform dependent; Typically, when the user is speaking, the ratio will increase perceptibly through non-speech. (The ratio is the transfer ratio, which in this case is a real number.)

Notably, some embodiments may use a complex transfer ratio, which includes both magnitude and phase, thus providing improved performance. When dealing with thresholds in these complex cases, the threshold is no longer a scaler, but a line (perhaps curved or trimmed, which splits the complex plane. For example, the line for the imaginary part >0 is a line). (or the threshold may be such that the positive real and imaginary parts represent speech, defining a quadrant of the complex plane.) Some embodiments may use right-hand, rather than left-hand, feedback and feedforward microphones. Pay attention to the point. Still other embodiments can automatically determine whether to use a right or left earpiece. For example, if the left earpiece is removed, the on-ear detection circuitry determines that only the right earpiece is in place and operates the voice detection module based on the right feedback and feedforward microphones.

In determining the energy levels at the output of feedback and feedforward microphones, the complex transfer function (TF) Txy can be calculated as

Here, Sxy is the cross spectral density between x and y, and Sxx is the power spectral density of x. Therefore, Sxy can be estimated through FFTs in the following way:

Or, if only one fft is used (and no averaging) Txy is defined as

For the ratio of fb (feedback mic) energy to ff (feed forward mic) energy, x is the left feedforward mic signal ffL(FFA) and y is the left feedback mic signal fbL(FBA). The ratio of fb energy to ff energy is actually |Tff2fb|^2.

Accordingly, in embodiments that use fft as a core basis, the ratio of the squared absolute values of the fft results can be bounded. Alternatively, if you were implementing without fft, you could simply calculate the moving average of the sampled temporal signals, after passing it through a bandpass filter and achieving similar results. Using complex transfer function estimates (Txy..), important phase information is available.

The detection threshold (DT) is generally set based on the physical implementation of the earpieces and the quality of seal they provide for the front cavity volume. For good ANC headphones, you can expect 20dB of attenuation when the user is not speaking. This rises to approximately 0 dB when the user is talking due to the bone conduction mechanism.

However, the degree of variation is different for each type of headset, being more significant for the in-the-ear type of headphones, less significant for the on-the-ear type, and least significant for the around-the-ear type. This refers to the use of thresholds. For example, for ITE headphones, at certain frequencies, such as 100 Hz, it is expected that there will be almost 20 to 30 dB more sound pressure in the blocked ear (fb mic) than outside it (ff mic) due to user speech. do. This effect can also be enhanced in circumaurals, but the difference in in-the-ear sound pressure for speech at 100 Hz relative to external sound pressure in a feedforward mic is probably only a few decibels.

Some embodiments may also dynamically change the threshold based on measured parameters, such as, for example, ambient noise or average signal-to-noise ratios, or alternatively, the user may use an app hosted on device 140. A coordination mechanism can be provided. Some embodiments use a dynamic detection threshold because the fb/ff ratio is a function of the overall attenuation within the ANC headset and can therefore vary over a fairly wide range. In one embodiment, the detection threshold is estimated during the two-ear smart mute period, specifically as a function of the moving average of the energy to mute and the moving average of the energy to unmute. Some embodiments measure the attenuation of the system both actively and passively, with the threshold ultimately being a function of the active attenuation.

For TF estimates, some embodiments conditionally update the mean. Some embodiments also recognize that the feedback mic signal includes multiple components: audio input signal (Rx) (from device 140), ambient noise, user speech, and measurement noise. This ensures good signal levels that are not correlated with ambient noise. Alternatively, to estimate the noise transfer function, some embodiments update the average when the energy ratio of fb/ff is low or highly correlated, ensuring that TF estimates convergence faster than it would otherwise. .

This second approach to user voice activity detection (UVAD), which is based on the complex transfer characteristics of two control mics within a specific frequency range, is believed to be particularly robust when used on any two mics with transfer characteristics that satisfy the following constraints: do. The first constraint is that the transmission characteristics change in a significant way for speech compared to interference. The second constraint is that the transfer characteristic remains relatively clear (i.e., relatively clear) by changes in the relative direction of interference. These conditions are met by feedforward and feedback mics in in-the-ear headsets (and in other situations where one mic is more strongly acoustically coupled to the user's voice field than another).

For ambient noise, the output of ambient pressure at the ambient mic responds first and has a leading phase relative to the ear-coupled feedback mic for two reasons. The first reason is that it is directly coupled to the surroundings and is usually closer to the acoustic path length to the noise source. The second reason is that typical headset earpieces have some amount of passive attenuation, which is almost a kind of low-pass filter, i.e. it doesn't make any difference at very low frequencies, but as the frequencies rise, the ear-coupled mic (fb mic) It attenuates it more significantly. All causal low-pass filters induce phase delay and all physical filters are causal.

For user speech, in addition to the acoustic path from the user's mouth to the ear and then to the ear-coupled microphone, there is another path from the vocal cords through the body. The speed of sound through the body is significantly faster, almost 4 to 5 times faster, or 3 to 4.5 milliseconds versus below 1 millisecond. (The speed of sound in air is approximately 340 meters/sec; in the skin it is approximately 1500 meters/sec; in the skull and bones it is 4000 meters/sec.) As a result, the Sound travels much faster than airborne acoustic signals through the mouth.

Looking very closely at the transfer characteristics between peripheral coupled and ear coupled mics, the peripheral microphone will guide the ear coupled mics independently of the directional arrival. For user speech, the ear-coupled mic will guide the peripheral microphone. Thus, it can be clearly appreciated that the lack of large ambiguity caused by the direction of interference and asymmetry in the complex transfer function (or any other basis) is far superior to ambient and ear-coupled microphones from the same ear. .

To fully utilize the bone conduction path of the ear-coupled mic, referred to herein as the feedback mic (fb), some embodiments describe incoming audio “Rx” from an external device, such as device 140. (In some embodiments, Rx may be mixed with one or more internally generated audio notification signals, such as beeps or tones indicating system events, prompts, etc.) This incoming audio is typically transmitted through bone conduction. It has properties that are strongly similar to speech, meaning that it has an fb_mic amplitude that is much stronger than that of the surrounding mic (feed forward mic(ff)), and can therefore lead to erroneous user-speech detections.

One approach to alleviate this issue is to use an Rx canceller, which mathematically cancels or removes the effect of the Rx signal from UVAD calculations. An exemplary embodiment uses an Rx canceller based on decomposing the fb signal as follows:

here,

fb _Rx Rx is the fb mic signal due to the Rx signal played outside the ear-coupled speaker;

fb _ambient is the fb mic signal due to ambient noise;

fb _{speech_BC} is the fb mic signal due to bone conduction.

Additionally, fb_Rx and fb_ambient can be additionally defined as follows:

Here, T _rx2tb is the transfer function from Rx to fb mic with all other inputs being zero and T _ff2fb is the transfer function from feedforward mic to feedback mic with only noise stimulus and no speech or Rx. T _rx2fb and T _ff2fb can be estimated using several methods. For example, some embodiments use a Kalman filter, or a traditional estimate based on auto and cross spectra, for example being careful not to update the means for Tff2fb when Rx is present. Care also needs to be taken not to update estimates when user speech is present, but this is a problem that is greatly alleviated as UVAD for this step does not need to catch all speech, but rather has high confidence that speech is not present. .

If estimates are available for these mainly static transfer functions, then we can use them to estimate the fb _{speech_BC} signal in near real time. Estimates of Trx2fb and Tff2fb will be averaged over time. The example embodiment uses fast fourier transforms (FFTs) to calculate the estimates; Some embodiments use an arbitrary basis that sufficiently spans the subspace containing bone conduction information.

Recognizing Trx2fb and Tff2fb, fb _{speech_BC} can be expressed as follows:

or

Here, fb^ _{speech_BC} is an estimate of fb _{speech_BC} .

Therefore, user speech detection is based on an estimated signal that is primarily free of interference from ambient noise and incoming audio.

Notably, this version of the asymmetric approach (using same-side feedback and feedforward mics) relies on a bone conduction path between the user's voice area and the feedback mic. Therefore, placement of a headset using this asymmetric approach on a conventional HATS (e.g. the B&K 4128-C simulator mentioned above) will generally prevent proper operation of the UVAD since conventional HATS have no bone conduction pathways. In other words, a headset mounted on a HATS will not be able to properly mute and unmute (or otherwise accurately detect user voice activity) in response to user voice signals in the appropriate frequency range input to the HATS (from the voice domain). There is a path to the feedback mic due to vibration, but this will be a very weak coupling compared to actual bone conduction.)

Some embodiments combine symmetric and asymmetric based threshold tests together, as follows:

Notably, implementing this detection logic involves the use of one of three control mics, left and right feedback error mics and feedforward mics. Additionally, this logic only allows the asymmetric threshold test (ratio of feedback mic energy to feedforward mic energy) to control unmute. Other embodiments may allow both to trigger unmute.

Additionally, some embodiments provide buffers within processing circuitry 130 to perform voice detection on a delayed version of the relevant signal data. More specifically, some embodiments utilize an For example, one embodiment may store a set of 20 milliseconds of sampled data from system mics such that the detector detects user speech at sample n and then all previous samples taken in the previous 20 millisecond period. allows to unmute the user's phrases, thus avoiding skipping or muting the first part of the user's phrase. In some embodiments, the length of the look ahead period can be adjusted or calibrated by the user, in others it is based on the user's detected speech intonation, for example a rolling average of the distances between peaks in the speech signal. So it can be determined dynamically.

The mute module 136 provides a mute function in response to command signals from the user voice detection module 135. In some embodiments, this involves turning off signal paths associated with the voice mic and/or one or more other mics within the assembly. However, to improve the user experience, some embodiments delay the activation or onset of mute for 3, 5, 10, 15, or 20 milliseconds to avoid shortening the ends of phrases or silencing between words. Add a margin of safety for In some embodiments, the duration of this delay may be set by the user or determined dynamically based on the user's measured speech intonation. Additionally, in some embodiments, a visual, audible, or haptic indication is provided in response to activating and deactivating the mute function to alert the user of the change in mute state. In some embodiments, one or more of these indicators are provided on the headset itself and/or on device 140. In some instances, the visual indicator takes the form of an illuminated and/or flashing light emitting diode (LED) on the headset and/or an illuminated or flashing or altered coloration or shape of the microphone icon on the device display. Take . In some embodiments, the user can override the mute function through manual control elements on the headset and/or device, where overriding is for a predetermined period of time, e.g., 1, 2, or 3 minutes, or between a phone call and The same has an effect until termination of the current communication session. At the end of the ignore period, automatic muting and unmuting will resume.

More generally, some embodiments elevate state changes between mute on and mute off to avoid very fast microphone gain changes that can otherwise produce audible pops or clicks that are annoying and indicative of poor quality audio components. to slow down or intentionally slow down (the same goes for the reverse). This is usually handled by having the gain change gradually rather than immediately. For example, in one embodiment, the “attack” off-to-on occurs over approximately 100 msec, which is slow enough to avoid pops and long enough to minimize look ahead memory requirements. One embodiment uses a decay function of the form:

Some embodiments facilitate more effective automatic muting by detecting when a two earpiece system has only one earpiece properly mounted. Some of these embodiments rely on an On Ear Detector (OED) to optimize performance. Details of the OED are further described in commonly-owned U.S. patent application Ser. No. 14/850,859, filed September 10, 2015, the disclosure of which is incorporated herein by reference in its entirety.

Speech dependence module 136 represents one or more other functions (of processor circuitry 130 and/or device 140) that receive a binary speech-present or speech-absent signal from speech activity detection module 135. . Some of these modules use signals to activate or deactivate module functions, thereby conserving processing power, memory, and/or battery life. For example, in some embodiments, speech dependency module 137 includes a speech or keyword recognition module configured to listen for specific keyword commands or perform more generalized speech recognition functions.

In some other embodiments, module 137 further includes a noise reduction module that provides additional processing to reduce noise in the voice mic signal. This noise reduction module may, in some embodiments, be tailored to the user's specific environment. And, in yet other embodiments, speech dependent module 136 includes a sidetone module or circuit that receives voice mic output and generates a 3-10% user sidetone signal for one or both earpieces. Generating sidetones consumes power, so switching this feature off conserves battery life when the user is not speaking. See U.S. Provisional Patent Application No. 62/530,049, filed July 7, 2017, which is incorporated herein by reference.

2 shows a flow diagram 200 of an improved auto-mute system for an ANR headset with two earpieces, where the flow diagram 200 includes processing blocks 210-280.

Block 210 involves performing OED (On-Ear Detection) to determine the status of the earpieces. (See co-pending U.S. patent application Ser. No. 14/850,859, filed September 10, 2015, which is incorporated herein by reference.) The implementation then proceeds to determine whether the ambient noise level is low. Continues at block 220 with following steps: If ambient noise is low, the smart mute capability of module 134 is disabled at block 230 and execution returns to block 220; Otherwise, execution continues at block 240.

Block 240 involves determining whether both earpieces are mounted on the user. In the example embodiment, this involves another call to the OED module. Once both earbuds are installed, execution branches to block 250, which determines whether to mute using the symmetry-based or combined symmetric-asymmetric mute functions described above, both of which are muted from both earpieces. Signals are needed. Execution from here loops back to block 240 to determine if both earpieces are still mounted. If it is determined that both earbuds are not equipped, then execution proceeds to block 260, which determines whether one earpiece is equipped. (It also determines which one is equipped.) If one is equipped, execution branches to block 270 to perform the smart mute function based on asymmetric thresholding as described above. Execution then cycles back to block 260 to determine if one earpiece is still attached. (In some embodiments, an earpiece can still be mounted, but has insufficient battery power.) If one earpiece is not mounted, no smart mute is performed and execution branches back to block 220. .

3 shows a monaural system model 300 of system 100, including the following blocks: T _p is the passively attenuated transfer function; T _dm is the driver-to-feedback-mic transfer function; H _ff is a feedforward filter; H _fb is a feedback filter; V is the user-speech-to-feedforward-mic acoustic path (transfer function); W is the user-speech-to-feedback-mic bone-conduction path (transfer function). The model also includes the following signals: s is the user speech signal; a is the ambient noise signal; n is the feedforward mic measurement (or more generally, the mic furthest from or less acoustically coupled to the speech region); m is the feedback mic measurement (or more generally, the mic furthest from or more acoustically coupled to the speech area); u is the RX signal and/or any system audio notification signals; d is the DAC (driver) output.

More particularly, the Figure 3 system has both feedforward and feedback filters (Hff and Hfb) present. However, some embodiments omit these filters, meaning (H _ff = H _fb = O) and the headset is passive. The exemplary embodiment uses the following linear statistical model:

Substituting D for M gives:

Sorting out the terms yields:

Substituting N gives:

Sorting out the terms yields:

here,

The goal in linear modeling is to decompose the feedback microphone measurement (M) into a linear combination of independent components (ambient noise, Rx, and user speech). This model is applicable to narrowband signals, i.e. specific frequency bins. To model a broadband system, we will add frequency exponents to each term.

The variances can be expressed as follows:

Also, this is for the narrowband case. Calculating the variances at all frequencies will yield power spectra of A, S, and N. These variances are instantaneous values (since ambient noise and speech are non-stationary). , )am. Time exponents are dropped for notation convenience. The covariance (E[MN ^* ]) is defined as

It can be rewritten as below

Note that calculating the covariance at all frequencies yields a cross power spectrum.

The regression coefficient (G) is defined as

here, is the user-speech-to-ambient-noise SNR. Substituting F ₁ and F _s into G yields

When user speech is present, non approaches 1 (as the user-speech-to-ambient-noise SNR increases). When user speech is absent, is 0. This means that the instantaneous regression coefficient G(t) remains in the line segment with the end points F ₁ and F ₁ + F _s V ^-1 . Note that calculating the regression coefficients at all frequencies yields a transfer function.

Estimation: Tracking the regression coefficient (G(t)) can be a challenging problem as it changes over time. It's bad yet that when Rx is present, the coherence between M and N is reduced, it increases the variance of the estimates. Using a reference to U simplifies the estimation problem, although it is not required. Some embodiments formulate the estimation problem in a state space framework using a measurement model (M(t)), defined below:

Here, r(t) is a Gaussian random variable with zero mean and unit variance, and σ _r is an adjustable parameter that accounts for unmodeled behavior (i.e., slight nonlinearities in the measurements). H(t) is the regression coefficient that accounts for the contribution of the rx/music signal to the feedback mic measurement.

Some embodiments use the following process model:

Here, q ₁ (t) and q ₂ (t) are independent Gaussian random variables with zero means and unit variances. α ₁ and α ₂ are adjustable parameters that govern how quickly G(t) and H(t) can change over time.

The state space framework is useful because there are efficient algorithms for state estimation Recursive Least Square (RLS), Least Mean Square (LMS), and Kalman filter, for example. Some embodiments estimate the states G(t) and H(t) in several frequency bins by using a Kalman filter in each bin.

Some embodiments include a more generalized approach to user voice activity detection that avoids the need to explicitly characterize the electroacoustic parameters of the system. A generalized approach is to generate two signals ((θ), each of which is a function of at least two of user speech (s), ambient noise (a), and/or incoming audio (u), according to a matrix of transfer functions (T). )(theta) and ϕ(phi)) are used. (Incoming audio u may be a mixture of externally generated audio signals, such as Rx/Music from a communication device, and internally generated audio signals, such as system event prompts, notifications, or alarms.) The matrix of transfer functions (T) is determined by how the speech, ambient and incoming audio (u) appears on two or more transducers and how the transducers are combined with a reference to Rx/Music to generate phi and theta. do. Mathematically, this can be expressed as

Here, T represents the matrix of transfer functions and is defined as

Here, T _sθ represents the transfer function from user speech (s) to signal (θ); T _aθ represents the transfer function from ambient noise (a) to signal (θ); T _sΦ represents the transfer function from user speech (s) to signal (Φ); T _aΦ represents the transfer function from ambient noise (a) to signal (Φ). For these models, trust detection of user voice activity requires adherence to the following asymmetry constraints:

Here, z represents the asymmetric threshold (z). It indicates that the speech-to-ambient sensitivity ratios should differ in magnitude and/or phase. In some embodiments, z is equal to zero and in other embodiments z is equal to 0.01, 0.1, 0.5, 0.75, 1.0, 2.0. In still other embodiments, z is greater than 2.

Mapping the terms of this more generalized model to the specific electroacoustic implementation of Figure 3 gives the corresponding T matrix as

here,

Some embodiments may use alternative asymmetry constraints in the form of

It requires that the ratio of speech to ambient signal power in signal θ is greater than the ratio of speech to ambient signal power in signal Φ.

Figure 4 shows a block diagram of a general electroacoustic system 400 illustrating the generation of signal θ from a linear combination of mic inputs and Rx/music (incoming audio) inputs. System 400 measures a set of microphones or transducers provided through respective gain modules or blocks (K ₁ to K _N ) to filter H with respective transfer functions (H ₁ to H _N ). s or inputs (M ₁ ...M _N) and Rx/music signal references (U1 and U2). The filter outputs a feed to a summer, which generates theta. Therefore, signal θ is a filtered combination of the converter and reference inputs.

FIG. 5 shows a block diagram of a generalized UVAD module 500 that can be used as part of UVAD module 135 and within the FIG. 2 process. In addition to the input signals user speech (s), ambient noise (a), and incoming device audio (u), module 500 includes an electroacoustic system model block 510, an estimator block 520, a summer 530, Includes a variance ratio estimator block 540, and a decision block 550. Electroacoustic system model block 510 generally represents T, a matrix of transfer functions (T), and any electroacoustic system, such as system 100 or 500.

Estimator block 520 iteratively predicts or estimates (theta) from pi and u, and the prediction error signal (e) from summer block 530 is fed back to update each new prediction. In an exemplary embodiment, estimator block 520 takes the form of a Kalman filter (as described above). However, other embodiments use forms of linear estimators, such as RLS and LMS estimators. θΦ.

The variance ratio estimator block 540 estimates the variance of the signal (Φ, S _Φ ) and the variance of the prediction error (S _e ) and calculates the ratio S _e /S _Φ . The ratio is provided to decision block 550, which compares the ratio to a detection threshold (DT). If the ratio exceeds the threshold, the user voice activity detection signal is set to 1, indicating the presence of user speech. Otherwise, the detection signal is set to 0.

conclusion

In the foregoing specification, certain example embodiments have been described. However, one skilled in the art understands that various modifications and changes may be made without departing from the scope of the invention as set forth in the claims below. Accordingly, the specification and drawings are to be regarded in an illustrative rather than a restrictive sense, and all such modifications are intended to be included within the scope of the present teachings.

The benefits, advantages, solutions to problems, and any element(s) that may give rise to or make more significant any benefit, advantage, or solution are material to any or all of the claims; They should not be construed as required or essential features or elements. The invention is defined solely by the appended claims, including any amendments made during the pendency of this application and all equivalents of those claims as issued.

Moreover, in this literature, correlational terms, such as second, top, bottom, etc., refer to one entity or action from another without necessarily requiring or implying any actual such relationship or ordering between the entities or actions. Can only be used to differentiate. Terms “comprises”, “comprising”, “has”, “having”, “includes”, “including”, “contains” “Contains,” “containing,” or any other variation thereof is intended to cover the non-exclusive inclusion of a process, method, or article that has, has, includes, or contains a list of elements. , or a device may not only include those elements, but may also include other elements not explicitly listed or inherent in such process, method, article, or device. "comprises a", "has ... a)", "includes ... a)", "contains ... a) "An element referred to by" does not, without further limitations, exclude the presence of additional identical elements in a process, method, article, or apparatus that has, has, includes, or contains the element. Terms (“a” and “an”) are defined herein as more than one unless clearly stated otherwise. The terms “substantially,” “essentially,” “approximately,” “about” or any other version thereof are defined as proximate as understood by a person of ordinary skill in the art, and in non-limiting embodiments , the term is defined as within 10%, in other embodiments within 5%, in other embodiments within 1%, and in other embodiments within 0.5%. As used herein, the term “coupled” is defined as connected, although not necessarily directly and not necessarily mechanically. A device or structure that is “configured” in a particular way is configured in at least that way, but may also be configured in ways not listed. Additionally, the term “exemplary” is used herein as an adjective to modify one or more nouns, such as an embodiment, system, method, or device, and is meant to specifically indicate that the nouns are provided as non-limiting examples.

Claims (38)

An audio processing system, comprising:
A plurality of inputs receiving a plurality of audio signals - the plurality of inputs comprising: a first input receiving a first audio signal from a first transducer disposed proximate to the user's first ear; a second input for receiving a second audio signal received from a second transducer, a third input for receiving a third audio signal from a third transducer disposed proximate to the user's speech area, and to the first ear of the user. a fourth input receiving a fourth audio signal from a fourth transducer disposed proximate but farther from the first ear of the user than the first transducer; and
Coupled to the plurality of inputs and receiving the first, second, third, and fourth audio signals, the amplitude of speech components of the first audio signal, the second audio signal, and the fourth audio signal and comparing phase, wherein the magnitude and phase of the speech components of the first audio signal and the second audio signal are substantially equal and the magnitude of the speech components of the first audio signal and the fourth audio signal. processing circuitry configured to determine that the speech components are provided by the user in response to determining that the speech components are substantially different, and to provide a speech activity detection signal indicating whether the speech components are provided by the user.
An audio processing system comprising: delete delete According to paragraph 1,
wherein the processing circuitry is further configured to at least one of mute or attenuate the third audio signal in response to determining that the speech components are not provided by the user. According to paragraph 1,
The first, second, third, and fourth transducers are microphones. According to paragraph 1,
The audio processing system may be used in an in-the-ear headset, over-the-ear-headset, or on-the-ear headset. -headset), an audio processing system. According to paragraph 1,
wherein the processing circuit determines that the speech components are not provided by the user in response to the comparison indicating that the phases of the speech components of at least the first audio signal and the second audio signal are not substantially the same. , audio processing system. delete delete According to paragraph 1,
The processing circuit is configured to determine if the comparison determines that the magnitude and phase of the speech components of the first audio signal and the second audio signal are substantially equal and that the magnitude of the first audio signal is at least substantially greater than the magnitude of the fourth audio signal. and determining that the speech components are provided by the user in response to indicating that the audio processing system is loud. An audio processing system, comprising:
a plurality of inputs receiving a plurality of audio signals, the plurality of inputs comprising: a first input receiving a first audio signal from a first transducer disposed proximate to the user's first ear; a second input receiving a second audio signal from a second transducer disposed farther from the first ear than the first transducer, and a third input receiving a third audio signal from a third transducer disposed proximate to the user's speech area. an input, and a fourth input receiving a fourth audio signal from a fourth transducer disposed proximate the user's second ear; and
Coupled to the plurality of inputs and receiving the first, second, third, and fourth audio signals, the amplitude of speech components of the first audio signal, the second audio signal, and the fourth audio signal and comparing phase, wherein the magnitude and phase of the speech components of the first audio signal and the fourth audio signal are substantially equal and the magnitude of the speech components of the first audio signal and the second audio signal. processing circuitry configured to detect that the speech components are provided by the user in response to determining that the speech components are substantially different, and provide a speech activity detection signal indicating whether the speech components are provided by the user.
An audio processing system comprising: delete delete According to clause 11,
wherein the processing circuitry is further configured to at least one of mute or attenuate the third audio signal in response to determining that the speech components are not provided by the user. According to clause 11,
The first, second, third, and fourth transducers are microphones, and the audio processing system is one of an in-the-ear headset, an over-the-ear headset, or an on-the-ear headset. system. According to clause 11,
The processing circuit is configured to cause the speech components to be generated by the user in response to the comparison indicating that the magnitude of the speech components of the first audio signal is at least not substantially greater than the magnitude of the speech components of the second audio signal. The audio processing system determines that it is not provided. According to clause 11,
The processing circuit is configured to cause the speech components to be transmitted by the user in response to the comparison indicating that the speech components of the first audio signal arrive substantially faster in time than the speech components of the second audio signal. An audio processing system, which determines that it is provided. A method of processing audio signals, comprising:
Receiving a first audio signal from a first transducer disposed proximate to the user's first ear;
Receiving a second audio signal from a second transducer disposed proximate to the user's second ear;
Receiving a third audio signal from a third transducer disposed close to the user's voice area;
receiving a fourth audio signal from a fourth transducer disposed close to the first ear of the user but further away from the first ear of the user than the first transducer;
comparing the magnitude and phase of speech components of the first audio signal, the second audio signal, and the fourth audio signal;
Based on the comparison, it is determined that the magnitude and phase of the speech components of the first audio signal and the second audio signal are substantially the same and the magnitude of the speech components of the first audio signal and the fourth audio signal are determining that the speech components are provided by the user in response to determining that they are substantially different; and
providing a speech activity detection signal indicating whether the speech components are provided by the user.
How to include . delete delete As a communication assembly,
an incoming audio signal node configured for connection to a communication device providing an incoming audio signal; and
A signal processing circuit configured to output a voice activity detection signal indicating the absence or presence of user speech, the signal processing circuit based on the first signal (Φ), the second signal (θ), and the third signal. configured to determine the absence or presence of user speech, wherein the Φ and θ signals are derived from two or more transducers in response to at least ambient noise (a), user speech (s), and the third signal, wherein the Φ and θ signals are constrained

satisfy,
T _sθ represents the transfer function from user speech (s) to signal (θ),
T _aθ expresses the transfer function from ambient noise (a) to signal (θ),
T _sΦ represents the transfer function from user speech (s) to signal (Φ),
T _a Expresses the transfer function from the ambient noise (a) to the signal (Φ),
The third signal is based at least in part on the incoming audio signal or at least one audio notification signal, and the communication assembly recognizes one or more keywords in user speech signals derived from the voice microphone output signal and detects the absence of user speech. The communication assembly further comprising a keyword recognition module configured to enter a shutdown or sleep state to conserve power or memory in response to the voice activity detection signal indicating. According to clause 21,
wherein the two or more transducers include first and second microphones, the first microphone being positioned with substantially greater acoustic coupling to the user's voice area than the second microphone. According to clause 22,
and wherein the first microphone is positioned to receive more acoustic energy from the user's speech area via a bone conduction path than the second microphone. According to clause 21,
an auto-mute module configured to attenuate a voice microphone output signal in response to a voice activity detection signal indicating the absence of user speech and to unattenuate the voice microphone output signal in response to a voice activity detection signal indicative of the presence of user speech. A communication assembly further comprising: According to clause 24,
and wherein the automatic mute module is configured to mute or attenuate the voice microphone output signal for a predetermined amount of time following receipt of the voice activity detection signal. delete According to clause 21,
Additionally, a sidetone module that provides a sidetone signal to the loudspeaker within the headset earpiece in response to user speech and enters a shutdown or sleep state to conserve power or memory in response to a voice activity detection signal indicating the absence of user speech. Including, communication assemblies. According to clause 21,
further comprising a noise reduction module configured to reduce noise in a first manner responsive to a speech activity detection signal indicating the presence of user speech and to reduce noise in a second manner responsive to a speech activity detection signal indicating the absence of user speech. Including, communication assemblies. According to clause 21,
further comprising at least one speech dependent module configured for operation in a manner dependent on the speech content of the speech microphone output signal and entering a power and/or memory conservation mode in response to a speech activity detection signal indicating the absence of user speech. Including, communication assemblies. According to clause 21,
a first node configured for connection to a loudspeaker associated with the headset earpiece, a second node configured for connection to a feedforward peripheral microphone associated with the headset earpiece, and a feedback error microphone associated with the headset earpiece. Further comprising a third node configured for connection to,
wherein the signal processing circuit is coupled to the first, second and third nodes and configured to provide an anti-noise signal to the loudspeaker in response to signals originating from the feedforward peripheral microphone and the feedback error microphone; ,
and the Φ and θ signals are derived from signals originating from the feedforward peripheral microphone and the feedback error microphone. According to clause 30,
wherein the signal processing circuit is further configured to indicate the absence of the user's speech when the first and second transducers are placed on a standard head and torso simulator (HATS) that is outputting a speech signal in range. According to clause 30,
A headset earpiece in an in-the-ear earpiece having an ear canal portion containing the feedback error microphone. 1. A method of operating a communications assembly having a voice microphone output signal and a set of two or more transducers, comprising:
determining whether a user of the communication assembly is speaking based on the output of the two or more transducers by deriving a first signal (Φ) and a second signal (θ), wherein the Φ and θ signals are an ambient noise signal ( a), a user speech signal (s), and a third signal, wherein the Φ and θ signals are constrained

satisfy,
T _sθ represents the transfer function from user speech (s) to signal (θ),
T _aθ expresses the transfer function from ambient noise (a) to signal (θ),
T _sΦ represents the transfer function from user speech (s) to signal (Φ),
T _aϕ expresses the transfer function from ambient noise (a) to signal (Φ),
the third signal is a function of at least an incoming audio input signal from an external device or at least one audio notification signal; and
In response to determining that the user is not speaking, changing the operating state of one or more speech dependent modules associated with the communication assembly to a resource conservation state, wherein the one or more speech dependent modules are configured to produce one or more spoken words or phrases. A method comprising a keyword recognition module configured to recognize. According to clause 33,
The one or more speech dependent modules include:
a mute module configured to mute the voice microphone output signal in response to determining that the user is not speaking; and
A sidetone module configured to generate a sidetone signal based on the voice microphone output signal.
A method further comprising at least one of: As a communication assembly,
first and second transducers configured to be worn on the user's head, the first transducer configured for substantially greater acoustic coupling to the user's speech area than the second transducer; and
a signal processing circuit configured to output a speech activity detection signal indicative of the absence or presence of user speech in a predetermined frequency range when the first and second transducers are mounted on a user's head, the assembly having a loudspeaker; an incoming audio signal node configured for connection to an earpiece, a feedforward peripheral microphone, a feedback error microphone, a voice microphone, and a communication device providing an incoming audio signal, the signal processing circuit comprising: a first signal (Φ); configured to determine the absence or presence of user speech based on a second signal (θ) and a third signal, wherein the Φ and θ signals are dependent on ambient noise (a), user speech (s), and the third signal. In response, at least output signals are derived from at least two of the feedforward peripheral microphone, the feedback error microphone, and the voice microphone, wherein the Φ and θ signals are constrained

satisfy,
T _sθ represents the transfer function from user speech (s) to signal (θ),
T _aθ expresses the transfer function from ambient noise (a) to signal (θ),
T _sΦ represents the transfer function from user speech (s) to signal (Φ),
T _aϕ expresses the transfer function from ambient noise (a) to signal (Φ),
The third signal is based at least in part on an incoming audio signal or at least one audio notification signal, and the assembly further comprises a set of one or more speech dependent modules, wherein the set of one or more speech dependent modules comprises one or more spoken words. or a keyword recognition module configured to recognize phrases. According to clause 35,
wherein the signal processing circuit is further configured to indicate the absence of user speech when the first and second transducers are positioned on a standard head and torso simulator (HATS) outputting a speech signal in the predetermined frequency range, Communication assembly. According to clause 35,
wherein the processing circuitry is further configured to provide an anti-noise signal to the loudspeaker in response to signals originating from the feedforward peripheral microphone and the feedback error microphone, each speech dependent module being configured to indicate the absence of user speech. A communications assembly, wherein the communication assembly enters a power or memory conservation state in response to a voice activity detection signal and exits the power or memory conservation state in response to a voice activity detection signal indicating the presence of user speech. According to clause 37,
A set of one or more of the speech dependent modules includes a voice microphone output module configured to output a voice microphone signal to the communication device, and a sidetone module configured to generate a sidetone signal based on the voice microphone output signal. A communication assembly further comprising:

Images