JP2021511755A - Speech recognition audio system and method - Google Patents

Abstract

Speech recognition A method of recognizing the external acoustic environment while the user wearing the audio system and headset is listening to music or any other audio source. Adjustable acoustic recognition zones give the user the flexibility to avoid hearing far away speech. External acoustics can be analyzed in the frequency domain, oscillation frequency candidates can be selected, and in the time domain it can be determined whether the oscillation frequency candidate is a signal of interest. If the signal intended for the external sound is determined to be the signal of interest, the external sound is mixed with the audio from the audio source.

Description

The present invention relates to a system and method for a user wearing a headset to recognize an external acoustic environment while listening to music or any other audio source.

Voice activity detection (VAD), known as speech activity detection or speech detection, is a technique used in speech processing to detect the presence or absence of human speech. Various VAD algorithms are known. Conventional algorithmic solutions used in VADs are known to have the drawback of inferior detection scores when the input signal is noisy.

VADs play a role in many speech processing applications, including speech recognition systems, speech compression systems, and noise reduction systems. In FIG. 1, features are extracted from a framed input signal, then a multidimensional threshold is adopted based on the information captured from the last few frames, and then features are used to determine whether the frame is utterance or noise. The basic principle of conventional VAD, which consists of comparing with this threshold, is drawn. In general, there is typically a final stage, a decision hangover, whose purpose is to ensure a continuous speech stream that includes a normal short silence period that appears in the sentence. Since the duration of 10 ms to 40 ms corresponds to a time window in which the utterance can be considered to be statistically stationary, the frame length is generally chosen to be a duration of 10 ms to 40 ms.

The criterion for detecting utterances is to look for a periodic voiced part with a well-defined mathematical structure that can be used in the algorithm. Another approach is to use a statistical model for speech, estimate speech parameters from captured data samples, and use the classical results of judgment theory to reach frame speech / noise classification.

FIG. 2 illustrates techniques that have been used in the time domain method to detect utterances. Techniques include short-term energy, zero crossover, cross-correlation, periodic measures, linear prediction analysis, and pitch estimation. FIG. 3 illustrates techniques that have been used in the frequency domain method to detect utterances. Techniques include subband energy, Wiener entropy, cepstrum, energy entropy, harmonics, and spectral peak analysis. Traditional VAD algorithms use time domain or frequency domain features, or use statistical algorithms or other specific algorithmic mechanisms. Some conventional VADs use a collection of features including long-term spectral divergence, cepstrum peaks, MEL filtered spectra, and spectral-time modulation in the time domain or frequency domain.

It is known that VAD performance decreases as the amount of noise increases. Conventional solutions should use a noise reduction (NR) module before the VAD system. One known limitation when processing speech signals with noise reduction (NR) is the potential appearance of music noise, which is added to the input signal and misdirects the VAD module. May lead to false positives.

Another drawback with using traditional NR modules is that it is difficult to set internal parameters to allow the system to operate properly for different noise levels and categories, and even so. Is impossible. As an example, when choosing a set of internal parameters to address a very noisy environment, relatively significant distortions appear in a silent or quiet environment.

To overcome the above drawbacks that not only affect audio quality but can even compromise the performance of VAD modules, it provides an improved mechanism for detecting noise level environments and dynamics of NR internal parameters. It is desirable to be able to set.

It is desirable to provide an improved, noise-robust VAD method, and a system that allows the user to recognize the external acoustic environment while listening to music or any other audio source.

The present invention relates to a speech recognition audio system and a method for a user wearing a headset to recognize an external acoustic environment while listening to music or any other audio source. The present invention relates to the concept of adjustable acoustic recognition zones that give the user flexibility and avoid hearing speech far away. The system of the present invention uses headphone features as described herein in US Patent Application Publication No. 2016/0241947, which is incorporated herein by reference. In one embodiment, the headphones include a microphone array with four input microphones. This microphone array provides spatial acoustic acquisition selectivity and allows the microphone array to be oriented in the direction of interest. A new audio architecture that uses beam forming methods and combines with different technologies such as noise reduction systems, fractional delay processing, and the voice activity detection (VAD) algorithm of the present invention, with improved performance in noisy environments. I will provide a.

The present invention includes different signal processing modules, including noise reduction and array processing. In particular, a procedure for estimating noise levels is provided, called Noise Sensing (NS). This procedure adapts the noise reduction parameters so that the output acoustic quality is optimized. When audio is detected, the user can be alerted via the headphone signal without disturbing the music or other audio source they are listening to. This is done by mixing the external audio with the headphone read signal. Considering psychoacoustic characteristics, use a mixing mechanism that allows final mixing without reducing the volume of the music signal while maximizing intelligibility at the same time.

Typical applications of the voice recognition audio system of the present invention are within the following scenarios: voices such as human screams, conversations or calls, baby crying, public transport announcements, doors that someone is ringing. Bells and warnings such as courier-activated doorbells, homes, cars, and other warnings, car horns, police and ambulance air raid sirens, and other scenarios such as whistles can appear. is there. The present invention will be described more fully with reference to the drawings below.

It is a schematic diagram of the principle of the prior art of speech activity detection (VAD). It is the schematic of the time domain utterance detection technique of an exemplary prior art. It is the schematic of the frequency domain utterance detection technique of an exemplary prior art. It is the schematic of the voice recognition audio system which mixes the external voice of interest with the music of the user according to the teaching of this invention. It is the schematic of the adjustable acoustic recognition zone used in the speech recognition audio system of this invention. It is the schematic of the microphone array used in the headphone of this invention. It is a flow chart of the method for detecting voice activity according to the teaching of this invention. It is a schematic diagram of an utterance signal. It is a schematic diagram of logarithmic Wiener entropy. It is a schematic diagram of the simplified logarithmic Wiener entropy. FIG. 6 is a schematic representation of a voice activity detection architecture system that includes a data buffer structure around a noise reduction (NR) module and a voice activity detection (VAD) module. It is a schematic diagram of the state mechanical diagram of the hangover procedure. It is the schematic of the utterance signal with a buffer length of 128. It is a schematic diagram of the logarithmic Wiener entropy of the signal shown in FIG. 11A. FIG. 11A is a schematic diagram of a simplified logarithmic Wiener entropy of the signal shown in FIG. 11A. It is the schematic of the utterance signal with a buffer length of 256. It is a schematic diagram of the logarithmic Wiener entropy of the signal shown in FIG. 12A. FIG. 12A is a schematic diagram of a simplified logarithmic Wiener entropy of the signal shown in FIG. 12A. It is the schematic of the utterance signal with a buffer length of 512. It is a schematic diagram of the logarithmic Wiener entropy of the signal shown in FIG. 13A. It is a schematic diagram of the simplified logarithmic Wiener entropy of the signal shown in FIG. 13A. It is the schematic of the adaptive noise reduction method by teaching of this invention. It is the schematic of the input signal including noise. It is the schematic of the phase difference of the left front microphone and the right front microphone. It is the schematic of the right front microphone and the right rear microphone. FIG. 5 is a flow diagram of a method of improving the output quality of speech activity detection (VAD), including localization and beam formation using a microprocessor array. It is a schematic diagram which improves the robustness of speech activity detection (VAD) with respect to diffuse noise. FIG. 5 is a flow chart of a method of increasing the robustness of speech activity detection (VAD) for unwanted speech within a recognition zone. It is a flow chart of the method for implementing a speech recognition audio system including adaptive spectrum equalization. FIG. 20A is a graph of music with poor speech intelligibility. FIG. 20B is a graph of music with good speech intelligibility using the concept of adaptive EQ. FIG. 21A is a schematic diagram of poor utterance intelligibility. FIG. 21B is a schematic diagram of good speech intelligibility achieved using the concept of intelligibility improvement based on HRTFs. It is a flow chart of a special processing method using the processing based on compression. It is a schematic diagram of the process which brings about poor intelligibility. FIG. 3 is a schematic representation of an implementation of a special process that uses compression-based processing to provide good intelligibility.

Next, the preferred embodiments of the present invention will be referred to in more detail, and some examples of the preferred embodiments will be illustrated in the accompanying drawings. Whenever possible, use the same reference numbers throughout drawings and descriptions to refer to the same or similar parts.

The voice recognition audio system of the present invention allows any user wearing headphones to recognize the external acoustic environment while listening to music or any other audio source. In one embodiment, the speech recognition audio system can be implemented as headphones with four input microphones, as described, for example, in US Patent Application Publication No. 2016-0241946. The user is stimulated by listening to a voice or a set of defined sounds of interest when the signal coming from the microphone of the headphones is recognized as the desired signal. When the signal coming from the microphone is not analyzed as voice or any signal of interest, the listener is not confused by the microphone signal, only hears the read signal.

FIG. 4 illustrates a possible scenario for a voice recognition audio system 10 when a person B comes towards a person A who is wearing headphones 12 and is listening to music or watching a television screen or the like with audio output. To do. As soon as person B speaks to person A, voice is detected through one or more microphones 15 arranged in ear pads 14 so that person A recognizes the spoken message spoken by person B. It is mixed with the lead signal. External sound needs to be mixed with music only when it is desirable, such as human voice, so as not to disturb it. The speech recognition system 10 can also detect other typical sounds such as alarms, ringing sounds, horns, alarms, sirens, bells, and whistles.

As depicted in FIG. 5, a subsystem called an adjustable Sound Awareness Zone (ASAZ) can be used with the speech recognition audio system 10. The user can use the application program interface (API) associated with the headphone 12 so that the speech recognition system 10 responds only to normal, non-whispering speech within a defined sphere radius. It has the ability to define a variable sphere radius around the user's head. Any other normal voice that is not a scream, located outside the defined sphere, is also undetected. The three levels that regulate the speech recognition system 12 can be defined as wide, medium, and narrow. Wide adjustment corresponds to a radius RL with a large length, medium adjustment corresponds to a radius RM with a medium length smaller than the RL, and narrow adjustment corresponds to a radius RM smaller than the radius RM. Corresponds to the radius RS having a dimension. For example, a radius RL can have a length in the range of about 75 feet to about 30 feet, a radius RM can have a length in the range of about 50 feet to about 20 feet, and a radius RS. Can have lengths ranging from about 25 feet to about 1 foot.

Referring to FIG. 4, the speech recognition audio system 10 uses a noise reduction (NR) method or a noise reduction (NR) method for estimating noise levels so that it can quickly harmonize with any of the internal parameters of the noise reduction (NR) algorithm. Includes noise reduction (NR) algorithm. This provides the best audio quality for a wide range of noise levels. Furthermore, this procedure, called Noise Sensing (NS), is used to dynamically adjust sensitive thresholds or other internal parameters to achieve good performance.

In one embodiment, the headphones 12 have one or more omnidirectional microphones 15 located within the ear pads 14. The headphones 12 can include four omnidirectional microphones 15, as shown in FIG. The headphones 12 include a rectangular or trapezoidal array of four omnidirectional microphones 15. This configuration allows the use of different virtual directional / heart-shaped microphones with pairs that combine elements in a straight line or even diagonally. The omnidirectional microphone 15 is located in the lower portion 16 of the ear pad 14 mounted in a unique position in order to realize a 360 ° audio image of the user's surrounding environment. An array processing algorithm is used to determine the localization of interest, such as the location of the speaker. Once localization is performed, the user can easily direct the equivalent antenna emission pattern in that direction. Then, the noise energy in one or more omnidirectional microphones 15 can be reduced, and the external sound is enhanced. As described below, the effect of beam formation has a decisive effect on noise reduction performance. One or more speakers 17 can be associated with the microphone 15. In an alternative embodiment, the headphones 12 may include any type of speaker array associated with a type of structure.

FIG. 7 is a schematic diagram of a method for voice activity detection 20 that can be implemented in the voice recognition audio system 10. An implementation of the present invention is to use both the frequency domain and the time domain. At block 22, the frequency domain can be used to detect periodic patterns. Block 22 can be called the first guessing step. The block 22 is a rough determination process for the purpose of selecting a potential oscillation frequency candidate. After block 22, block 24 can be performed. The block 24 can be a time domain procedure to check whether the selected oscillation frequency candidate has been confirmed or not. A large buffer can be used for the frequency estimation step at block 22, and for noise immunity, and a relatively low threshold can be used to minimize the rate of false negative verdicts. If the detected oscillation frequency candidates are incorrect, recursively use the results of the time domain algorithm analysis operating on the subframes inside the frame used for the analysis of the first step of the frequency domain. , The second and final determination process in the block 24 is performed in the time domain.

In some implementations of block 22, Wiener entropy or spectral flatness is used to reduce the computational burden associated with two consecutive steps. Further, as described below, the FFT of the input buffer can be used for noise reduction.

Some implementations of block 24 use a pitch estimation algorithm. In one embodiment, the pitch estimation algorithm is based on a robust YIN algorithm. The estimation process can be simplified to a detection-only process, or the pitch value estimated between successive frames can be used to make the algorithm even more robust against errors. Continuity can be ensured.

The accuracy of an algorithm called the Wiener Entropy YIN (Wiener Entropy YIN) algorithm is improved by continuously determining the overlap between large frames in addition to the subframes in the frame.

In one embodiment of VAD, block 22 can perform this method with different combinations of features within the frequency domain to detect potential pitch voiced frame candidates that are reanalyzed in the time domain of block 24. ..

The Wiener entropy is obtained by the following equation,

The above equation can be calculated using the following equation.

This equation leads to the following equation.

Wiener entropy can be calculated in different bands B _i , i = 1, ..., L. As a result, the candidate selection process is performed by calculating the amount of L scalars.

These are sent to the selection process after the threshold determination step.

When the frame is designed as an uttered candidate, time domain inspection is initiated at block 24. The YIN algorithm can be used over K subframes of length M, as in the following equation.
N = KM,
During the ceremony
N = 2 ^L
Is the frame length used in the spectral region, chosen to be a power of 2 so that the FFT can be used.

Change the YIN algorithm from a pitch estimation algorithm to a pitch detection algorithm. Therefore, the frequency band

Is specified to correspond to the expected minimum and maximum pitch frequency values, which leads to the time interval [τ _min , τ _{max] of the following equation.}

In the equation, F _S is a sampling frequency that can be a fraction of the original sampling frequency used for processing in the frequency domain.

Are the floor rounding operator and the ceiling rounding operator, respectively. As an example

If, then [τ _min , τ _max ] = [20,115].

We specify the following matrix for delays due to time delays.

In the equation, <> is a rounding operator to the nearest integer, and (0; m) = (0 1 2 ... m-1 m). Reconsider the above example as follows.

Using this selection, the calculation of the YIN difference function is performed according to the lag values in the first and second rows of the matrix Δ. The first column of this matrix gives a relative index to start the difference function calculation.

A set of difference function values inherited from continuous intervals of length H is defined over this frame. These values are organized in the form of a matrix with the number of rows and columns defined as follows.

The YIN difference matrix dd is defined by its general elements as follows.

Next, consider the following equation.

In addition, consider the following quantities.

The algorithm restarts by calculating the following equation.

Then look for the smallest.
rr (i) = min (Dn (τ _min : τ _max ))
Compare the above equation with the threshold.

When this minimum is smaller than the threshold value, the utterance determination βi = 1 regarding the subframe i is obtained.

When the determination is made for K consecutive sub-frames in the present frame, the determination regarding the presence of utterance is made over all the frames by continuously making a majority vote.

In the formula, Q may be chosen to be K / 2 (but not limited).

In one embodiment, at block 22, Wiener entropy simplification can be used. Expensive square root vector operation

To avoid, choose and use the following equation:

FIG. 8A shows an utterance signal. FIG. 8B shows the logarithm of Wiener entropy. FIG. 8C shows the logarithm of the simplified Wiener entropy. The results show that the simplified Wiener entropy is the correct instruction for voiced speech.

In one embodiment, block 24 can use the YIN simplification. The following YIN version can be used for the time domain portion.

In this last equation, the square difference function is replaced by an absolute value to reduce the number of operations.

There is an overlap of J samples between two consecutive frames (the judgment of utterance is correct only for the first J samples).

If r _k (i + 1) is the kth row of the _{matrix dd i + 1 at time i + 1, then}

Wherein, r _m (i + 1) is the m-th column of the matrix _{dd i + 1, dd i (} 2: n columns, :) from column 2 to n columns, associated with the present frame i dd It is a matrix extracted from.

From the above equation, the following equation is easily deduced.

Or
Dd _{i + 1} = Dd _i − r _i (i) + r _{n column} (i + 1)
Therefore, it is not necessary to calculate all the elements of the matrix dd before calculating the sum of the rows of the matrix dd. Instead, the vector Dd (i) is updated by computing the r-th column (i) and the n-th column (i).

FIG. 9 is a schematic diagram of an implementation of the method 20 in the voice activity detection architecture system 30 combined with the noise detection architecture system 50. As shown in FIG. 1, a voice activity detection (VAD) architecture system 30 and a noise detection architecture system (NS) 50 are implemented in a voice recognition audio system 10 to provide noise-robust voice activity detection (VAD). can do. Referring to FIG. 9, the input buffer 31 receives the input signal 29. The Fast Fourier Transform FFT and the concatenation of the input signals 29 in the input buffer 31 determine the frame 32. Candidates can be detected using frame 32 in the Wiener entropy module 33. The Wiener entropy module 33 performs block 22 as shown in FIG.

With reference to FIG. 9, the frame 32 can also be divided into K consecutive subframes 34. A downsampling process 35 can be used for the subframe 34 before the YIN pitch detection module 36. The YIN pitch detection module 36 executes the block 24 as shown in FIG. With reference to FIG. 9, the Wiener entropy module 33 and the YIN detection module 36 determine the subframe determination 37. In front of the module 40 for determining the presence of an utterance, the subframe determination 37 and the determinations from other subframes 38 can be introduced into the hangover module 39. Low energy regions can be found within the sentence, and the method 20 of the present invention may consider these regions as non-speech frames. Listening at the output can be frustrating if there are too many interruptions. Confusion can be eliminated by using the hangover module 39. Also, the frame 32 can be transferred to the noise detection (NS) architecture 50.

FIG. 10 is a schematic view of a state machine 60 that can be used in the hangover module 39. The permanent state 1 indicating that there is an utterance is drawn by the circle 61 at the hangover module output, and the permanent state 0 indicating that there is no utterance is drawn by the circle 63 at the hangover module output. Each determination arrow (0 or 1) that exits the circle 61 and the box 64, and the circle 63 and the box 65 is obtained after frame processing. If the determination is the same as the previous determination, XY or XN are accumulated as spoken or unuttered, respectively. If they are not the same, XY and XN are reset to their initial value of 0. When one of these variables becomes equal to NY or NN, switching from one state to another is triggered.

In this method or algorithm, decVad means a determination input coming from the utterance detection module 40 shown in FIG. When the state machine of FIG. 10 defines the position index idx and the determination output decHov value related to the state of the index, as a result, the state [0] = 0 and the state [1] = 1.

11 to 13 show the influence of the input buffer data on the Wiener entropy value. 11A, 12A, and 13A show utterance signals at buffer lengths of 128, 256, and 512, respectively. 11B, 12B, and 13B show log Wiener entropy at buffer lengths of 128, 256, and 512, respectively. 11C, 12C, and 13C show simplified log Wiener entropy at buffer lengths of 128, 256, and 512, respectively. It has been shown that increasing the input data buffer length has the effect of smoothing the Wiener entropy curve.

In one embodiment, the noise detection (NS) architecture 50 is optimized to provide noise reduction (NR) audio quality output for all possible noise levels, while avoiding the appearance of music noise as much as possible. As shown in FIG. 14, the noise detection (NS) output 51 can be used in the adaptive noise reduction (NR) module 70. The noise energy detection architecture system 72 is used to estimate noise using the module 73 and the noise reduction module 74, which combines the combiner 75 and the output. The amount of noise is estimated by the noise reduction module 74, which derives the selection of noise reduction (NR) algorithm parameters. The distance calculation module 76 can determine the distance between the detected noise and the headphones 12.

The output obtained from the distance calculation module 76 is used in the hangover determination module 77. To control the frequency of switching between noise level states, three noise level states are defined as noise, intermediate stage, and no noise, in which the speech recognition audio system 10 causes sudden noise or impulse noise. It is determined by the hangover determination module 77 so that it cannot be switched to. The adaptive noise reduction module 78 processes the signal obtained from the hangover determination module 77 to reduce noise. Both the raw signal G180 and the processed signal 82 G2 are mixed with a mixer 84 to provide a clean signal 85 and a voice activity determination (VAD) architecture system using an adaptive convex linear combination. Transmitted to 30
y = G1x1 + (1-G1) x2
In the equation, x1 is the raw microphone input, x2 is the NR module output, and y is the VAD module input.

G1 depends on the root mean square (RMS) value ξ that can be calculated in the time domain or frequency domain.

The NR algorithm and the internal configuration parameters corresponding to those algorithms can be adjusted to minimize music noise and audio artifacts while reducing environmental noise to the maximum.

In one embodiment, the voice recognition audio system 10 can include a microphone array and headphones 12 having, for example, a 4-channel procedure. The advantage of multi-channel procedures is that they provide innovative features that increase efficiency. Since the speaker is localized in space, the propagation of speaker acoustics to the microphone array follows a coherent path as opposed to noise diffusion. Typically, the audio picked up by one microphone is a delayed copy of the audio recorded by the second microphone. 15A-15C exemplify the phase difference pattern. The signals are timed to one speaker in the front and one speaker in the rear (about 2 seconds to about 6 seconds), and two speakers, one in the front and one in the rear (about 6 seconds to about 6 seconds). This is the first track of a drawn 4-channel recording microphone array representing (about 10 seconds). Noise is artificially added to the input signal, as shown in FIG. 15A. The phase difference between MLF and MLB (broadside) is shown in FIG. 15B, and the phase difference between MRF and MRB (endfire) I is shown in FIG. 15C. For both arrays, it has been shown that the phase difference patterns do not look similar with or without utterances.

The microphone array acts as a spatial filter to attenuate sound coming from undesired directions, while enhancing sound coming from one or more selected directions. The use of microphone arrays can help improve acoustic quality and / or improve VAD noise robustness and detection accuracy.

FIG. 16 illustrates an implementation of a speech recognition audio system 10 that includes a noise detection architecture system 50 that receives a noisy signal and determines a clean signal. The clean signal is used in the voice activity detection architecture system 30. The microphone array 100 can be used with the localization module 102 and the beam forming module 104.

When one of the microphones 15 in the microphone array 100 detects voice in one direction, the localization module 102 localizes the speaker arrival direction. The beam forming module 104 points the microphone detecting the sound in the determined direction, and as a result, attenuates the noise coming from the other direction. The beam forming module 104 attenuates external noise statistically and spatially to deliver the enhanced audio signal to the speaker 17 of the headphones 12, as shown in FIG.

In the alternative embodiment, the noise comes from all directions. For example, noise can occur in all directions on trains, planes, ships, etc. In these places, the noise is mainly due to the motor engine and the direction of arrival is totally due to the echoing of the cabin acoustics. it's not correct. Conversely, the speaker of interest is always located at a single point in space. Reverberation is rarely a problem because it is near the speaker, for example up to a few meters.

FIG. 17 is a speech recognition audio system 10 that includes a noise detection architecture system 50 that receives a noisy signal and determines a clean signal, and the use of a microphone array that takes advantage of the difference between the noise and the signal. An example of the implementation form is shown. In parallel with the noise reduction (NR) module 70 and the voice activity detection architecture system 30, incident signals coming from different directions, such as forward and backward, are received by the beam forming module 104 and compared by similar modules 106. If there is an utterance, the difference between the two spectra should be observed, taking into account that the speakers cannot be placed in multiple positions at the same time. In the absence of speech, small differences between the spectra can be observed, taking into account that the noise is more or less the same no matter which direction the headphones are facing. The signal determined by the similar module 106 can be mixed with the voiced signal and, in many cases, the artifacts from the voice activity detection architecture system 30 with the mixer 107. The use of such similarity-based features may help eliminate false alarms in voice activity detection architecture systems to increase the robustness of the signal to noise.

FIG. 18 illustrates an implementation of a speech recognition audio system 10 that includes undesired speech cancellation when multiple speakers are placed around the user. The user wants to talk to one speaker in a particular direction, eg, from the front. The microphone array 100 is used in recognition zone 108 to remove all signals coming from undesired directions with the beam forming module 104 and preprocess the signals to reduce noise reduction (NR) module 70 and voice activity detection architecture. It can be a noisy signal coming only from the recognition zone before entering the system 30.

It is preferable that the voice recognition audio system 10 secures a high degree of intelligibility. When the user is interrupted by external voice, it is desirable to add external voice while keeping the music level constant and ensuring that the user hears the voice message clearly. This advantage can be achieved by controlling both voice false alarm detection and listening conditions. The voice false alarm can be determined by the voice activity detection architecture system 30. In one embodiment, the invention provides a step of mixing external utterances detected by the voice activity detection architecture system 30 with music coming from the headphones 12, as shown in FIG.

It is desirable to ensure that the user understands the speaker sound delivered from the headphones 12. In one embodiment, the acoustic level of the music is muted, or at least reduced, while the utterance is detected and transmitted. Mixed strategies for improving speech intelligibility can include adaptive space equalization, spatial separation, and special studio-inspired processing that can be processed separately or together.

Listening to an utterance signal mixed with music dramatically reduces the intelligibility of the utterance signal, especially when the music already contains an audio signal. According to many sources, there is evidence that increasing the signal-to-noise ratio (SNR) with respect to the fundamental frequency of speech enhances speech comprehension. As a result, the higher the SNR for all harmonics, the better the utterance comprehension.

In the present invention, both the voice coming from the voice activity detection (VAD) architecture system 30 and the music played by the user on the headphones 12 can be used. In one embodiment, the energies of both signals can be compared, especially in the fundamental frequency band and associated harmonic bands, when the signal obtained from the voice activity detection (VAD) architecture system 30 is compared to music. If it is relatively low, it can be increased.

FIG. 19 illustrates an implementation of a speech recognition audio system 10 that includes an adaptive spectrum equalization method 200. Each time the voice is detected, the adaptive space equalization method 200 can be performed. At block 201, an estimate of the spectral density power of the music is determined. At block 202, an estimate of the spectral density power of the utterance is determined. At block 203, the estimated value and formant of the fundamental frequency of the utterance obtained from block 202 are determined. At block 204, the energy ratio between the utterance formant obtained from block 203 and the music obtained from block 201 is calculated to determine the voice-to-music ratio (VMR) for each spectral band. At block 205, an FFT-based equalizer (EQ) is applied to the band with the low VMR determined by block 204.

FIG. 20A illustrates a power and frequency graph 300 with respect to the utterance spectrum 301 compared to a music spectrum 302 with poor intelligibility. For bands 304, where the audio formant energy is relatively low relative to the music determined by block 204, an FFT-based equalizer is applied at block 205 to enhance those bands. FIG. 20B illustrates a power and frequency graph 300 with respect to the utterance spectrum 301 compared to a music spectrum 302 with good intelligibility after enhancement.

21A and 21B illustrate certain implementations of a speech recognition audio system 10 that includes a spatial decomposition 400. This strategy assumes that once a signal of interest is detected, an embedded microphone array can be used to localize this signal of interest. For example, through a cross-correlation-based method. FIG. 21A illustrates poor intelligibility with monaural utterance at position 402 and stereo music at position 403. An HRTF-based filter is applied to the signal delivered by the voice activity detection (VAD) 30 according to the speaker arrival direction, and the signal is embodied according to the actual speaker position (3D effect).

This allows the user 401 to separate the acoustic signals in space. At position 406, music is perceived at the center of the head, while at position 404, speech is perceived outside the head, as shown in FIG. 20B, which illustrates good intelligibility. At the same time, the music can be temporarily switched from stereo to monaural. Restoring spatial hearing is known to significantly enhance speech intelligibility.

FIG. 22 illustrates an implementation of a speech recognition audio system 10 that includes a compression-based process 500 that enhances the presence of speech when mixed with music and a special processing algorithm can be used. At block 501, the audio signal is copied, compressed, and then the compressed signal is copied back to the original audio signal. At block 502, light saturation is applied to the resulting signal. At block 503, a special equalizer is applied.

At block 501, compression reduces the intensity difference between phonemes, resulting in a reduced time-series masking effect and increased speech loudness. The sum of both the compressed voice and the original voice ensures that the voice still sounds natural. Block 502 provides more harmonics. For example, it is known that not only the fundamental frequency (F0) but also the harmonic information of F1 and F2 is crucial for vowel identification and consonant perception. Block 503 removes low frequency noise and increases the frequency band of interest, for example, -18 dB / octave up to 70 Hz, -3 dB around 250 Hz, -2 dB around 500 Hz, and around 3.3 kHz. The purpose is to clean the audio signal with a low frequency cut of + 2.5 dB and + 7 dB around 10 kHz.

FIG. 23A illustrates a poor intelligibility in which the gain 602 of the audio signal 601 is combined with the music signal 604 and the mixer 605 to provide the input 606 to the driver. FIG. 23B illustrates a system 600 that implements compression-based processing 500. The audio signal 601 is applied to the compression module 607 to provide a compressed signal. The compressed signal is combined with the gain 602 of the audio signal 601 by the mixer 608. The output of the mixer 608 is applied to the saturation module 609 to perform light saturation of block 502, and to the equalization module 610 to apply a special equalizer. The output of the equalization module 610 is combined with the music signal 604 on the mixer 612 to provide input 614 to the driver.

The noise-robust VAD method or algorithm of the present invention uses a strategic approach of selection and then confirmation. The first step is performed in the frequency domain with a relatively large input buffer that allows the effects of noise to be reduced. The presence of a voiced speech signal is detected via the multi-band Wiener entropy feature and shows how the complexity can be reduced without compromising the characteristics of classical Wiener entropy.

The second part of the algorithm is done in the time domain using a simplified version of the YIN algorithm, where pitch estimation is replaced by a simple detection of pitch. To further reduce the complexity, use the absolute value difference instead of the classical square difference. This algorithm operates over a series of subframes along the entire input frame.

The present invention provides the derivation of an adjustable acoustic recognition zone system. Using the amplitude of the input signal and some features that help distinguish the user from distant external voice, the system heads a spherical region where the VAD algorithm allows the user to consider normal voice. Be able to specify around the part. If the user is speaking at a normal volume outside the sphere, the system rejects that volume.

The present invention provides the derivation of a noise detection system.

Not only noise reduction methods or algorithms, but also other major modules such as VAD and array processing algorithms, these internal settings facilitate all possible noise levels, from quiet to very noisy situations. It may have the disadvantage of not being able to handle it. In order to improve the performance of the present system, how to significantly improve the noise reduction and the performance of the VAD algorithm by deriving the noise detection mechanism of the present invention and integrating this mechanism into the system of the present invention. It is shown whether it will improve. In fact, noise detection provides self-adjustable internal parameters including interacting related modules such as VAD, noise reduction, voice localization and beam formation using a microphone array system, and computational complexity reduction consisting of different algorithms. It allows for the architecture of reconfigurable algorithms.

The present invention shows how the burden of computational complexity can be significantly reduced. This reduces power consumption or provides more room for further processing. The present invention provides the derivation of an audio mixing method performed under the constraint of keeping the volume of music constant while increasing the intelligibility of speech.

Alternative embodiments of the invention may be implemented as pre-programmed hardware components, other related components, or as a combination of hardware and software components, including a hardware processor. An embodiment of the invention is implemented in connection with a dedicated or general purpose processor device that includes both hardware and / or software components, or a dedicated or general purpose computer adapted to have processing power. You can.

Embodiments may also include a physical computer-readable medium and / or an intangible computer-readable medium for carrying or having computer executable instructions, data structures, and / or data signals stored therein. Such physical computer-readable media and / or intangible computer-readable media can be any available medium accessible by a general purpose computer or a dedicated computer. As an example, but not limited to, such physical computer readable media can be RAM, ROM, EEPROM, CD-ROM or other optical disk storage area, magnetic disk storage area or magnetic storage device, other semiconductor storage medium, or computer execution. It may include any other physical medium that can be used to store the desired data in the form of possible instructions, data structures, and / or data signals, and that can be accessed by a general purpose computer or a dedicated computer. it can. Inside a general-purpose computer or a dedicated computer, an intangible computer-readable medium may include electromagnetic means for transmitting a data signal from one part of the computer to another, such as through a circuit resident in the computer. it can.

Computer-executable instructions, data structures, and / or data signals (eg, wiring, cables) when transmitting or providing information to a computer over a network or another communication connection (wired, wireless, or a combination of wired or wireless). Hardware equipment for transmitting and receiving (optical fibers, electronic circuits, chemicals, etc.) should, of course, be viewed as a physical computer-readable medium, while computer-executable instructions, data structures, and The radio carrier or radio medium for transmitting and / or receiving data signals (eg, radio communication, satellite communication, infrared communication, etc.) should, of course, be viewed as an intangible computer-readable medium. The above combinations should also be included in the scope of computer readable media.

Computer-executable instructions include, for example, instructions, data, and / or data signals that cause a general purpose computer, a dedicated computer, or a dedicated processing device to perform certain functions or groups of functions. Although not essential, the aspects of the invention have been described herein in the general context of computer-executable instructions such as program modules being executed by a computer in a networked and / or non-networked environment. In general, a program module includes routines, programs, objects, components, and content structures that perform a particular task or implement a particular abstract content type. Computer-executable instructions, associated content structures, and program modules represent examples of program code for performing aspects of the methods disclosed herein.

The embodiments also include a physical computer-readable medium having computer-readable program code stored therein, comprising computer-executable instructions that cause the system to perform the methods of the invention when executed by a processor. It may include computer program products for use in the system.

The embodiments described above provide only a few examples of many possible unique embodiments that can represent applications of the principles of the invention. Without departing from the spirit and scope of the present invention, a number of other arrangements can be easily devised by those skilled in the art according to these principles.

Claims (24)

A voice recognition audio system
Headphones configured to receive audio from an audio source, and
At least one microphone associated with the headphones, configured to detect external acoustics in an external acoustic environment and generate a signal targeting the external acoustics.
When the signal targeting the external sound is the signal of interest and includes an analyzer module for determining whether or not the signal targeting the external sound is the signal of interest. The external sound is a speech recognition audio system that is mixed with the audio from the audio source. The analyzer module analyzes the signal targeting the external sound in the frequency domain, selects an oscillation frequency candidate, and determines whether the oscillation frequency candidate is the signal of interest in the time domain. The voice recognition audio system according to claim 1, wherein the voice recognition audio system is configured to determine. The analyzer module receives the signal of interest to the external sound in the input buffer, and the analysis in the frequency domain uses the FFT of the signal in the input buffer to generate an input frame. The voice recognition audio system according to claim 2, wherein the analysis in the time domain recursively uses a subframe together with the input frame. The speech recognition audio system of claim 3, wherein the analysis in the frequency domain is performed using Wiener entropy or simplified Wiener entropy. The speech recognition audio system of claim 3, wherein the analysis in the time domain is performed using pitch estimation or a YIN algorithm. The voice recognition audio system according to claim 1, wherein the analyzer module further includes a hangover module for determining whether or not there is an utterance in the signal of interest determined in the time domain. The voice according to claim 2, wherein the noise reduction algorithm uses the analysis in the frequency domain to estimate the noise level in the external acoustic environment and adjust the voice recognition audio system based on the noise level. Recognition audio system. An adjustable sound recognition zone having one or more control zones is defined around the headphones, and the external sound is said to be of interest when it is inside a predetermined one of the one or more control zones. The voice recognition audio system according to claim 1, which is determined to be a certain signal. The voice recognition audio system according to claim 1, wherein the audio is music. The headphones include an array of microphones arranged to attenuate or amplify audio coming from a selected direction, the microphones of the array of microphones providing a 360 ° audio image of the user's ambient environment. The voice recognition audio system according to claim 1, which is directed in various directions in order to do so. An adjustable sound recognition zone having one or more control zones is defined around the headphones, and the external sound is said to be of interest when it is inside a predetermined one of the one or more control zones. The voice recognition audio system according to claim 10, wherein the microphone array is determined to be a certain signal, removes a signal arriving from an undesired direction, and directs the microphone array in a direction of interest. A method for a user wearing headphones configured to receive audio from an audio source to recognize the external acoustic environment.
a. A step of detecting external acoustics in the external acoustic environment using at least one microphone associated with the headphones.
b. The step of generating a signal targeting the external sound, and
c. A step of determining whether the signal targeting the external sound is a signal of interest, and
d. A method comprising the step of mixing the external sound with the audio from the audio source when the signal for the external sound is determined to be the signal of interest. 12. In step b, the external sound is analyzed in the frequency domain, an oscillation frequency candidate is selected, and in the time domain, it is determined whether or not the oscillation frequency candidate is the signal of interest. The method described in. 13. The method of claim 13, wherein the analysis in the frequency domain is performed using Wiener entropy or simplified Wiener entropy. 13. The method of claim 13, wherein the analysis in the time domain is performed using pitch estimation or a YIN algorithm. 13. The method of claim 13, further comprising a step of determining whether or not there is an utterance in the signal of interest determined in the time domain. Further provided with a step of estimating the noise level in the outer acoustic environment,
The step c includes a step of adjusting based on the noise level to determine whether the signal targeting the external sound is the signal of interest.
The method according to claim 12. A step is further provided around the headphones to define an adjustable acoustic recognition zone having one or more adjustment zones, in step c where the external sound is a predetermined of the one or more adjustment zones. When inside one, it is determined to be the signal of interest,
The method according to claim 12. The at least one microphone is an array of microphones, and after detecting sound in step a, the direction of the sound is localized and the array of microphones is directed toward the determined localization direction. The method according to claim 12, further comprising. e. The step of determining whether or not the signal in step b is a noisy signal, and
f. When it is determined that the signal is noisy, the step of generating a clean signal and
g. In step c, the step of determining the signal from the first direction and the second direction, and
h. It is a step of estimating the similarity of the signal obtained from the first direction and the second direction, and is obtained from the signal obtained from the first direction and the second direction in the step h. 19. The method of claim 19, further comprising mixing the signals in step d when it is determined that the signals are similar. 18. The method of claim 18, further comprising removing all signals coming from undesired directions in the adjustable acoustic recognition zone. The sound is music
The step of estimating the spectral density power of the acoustic and
The step of estimating the spectral density power of the utterance in the external sound, and
The step of estimating the fundamental frequency of the utterance and determining the utterance format,
A step of calculating the energy ratio between the speech format and the spectral power of the music format block to determine the voice-to-music ratio (VMR) for each spectral band.
12. The method of claim 12, further comprising applying an FFT-based equalizer (EQ) to the spectral band with a predetermined VMR. Headphones configured to receive audio from an audio source, a computer program product implemented in a non-temporary computer-readable storage medium for judging sound in an external acoustic environment, said program. , A program code for detecting an external sound in the external acoustic environment using at least one microphone related to the head, a program code for generating a signal for the external sound, and the external. To determine whether the signal targeting sound is a signal of interest and a program code for determining whether the signal targeting external sound is a signal of interest. A computer program product comprising the program code of the above and a program code for mixing the external sound with the audio from the audio source when it is determined that the external sound is of interest. 23. The computer of claim 23, wherein in the frequency domain, the external acoustics are analyzed, oscillation frequency candidates are selected, and in the time domain, it is determined whether the oscillation frequency candidate is the signal of interest. Program product.

Images