JP4866958B2 - Noise reduction in electronic devices with farfield microphones on the console - Google Patents

Description

[Priority claim]
This application claims the benefit of Patent Document 1, which is commonly assigned to this application and is the same as the present application, the entire disclosure of which is incorporated herein by reference. This application claims the benefit of Patent Document 2 that is commonly assigned to this application and is the same as the present application, the entire disclosure of which is incorporated herein. This application also claims the benefit of US Pat. No. 6,057,056, whose assignee is common to this application and is co-pending with this application, the entire disclosure of which is incorporated herein by reference. This application also claims the benefit of US Pat. No. 6,057,056, whose assignee is common to this application and is co-pending with this application, the entire disclosure of which is incorporated herein by reference. This application also claims the benefit of Patent Document 5, which is common to the present application and is assigned at the same time as the present application, the entire disclosure of which is incorporated herein. This application also claims the benefit of US Pat. No. 6,057,097, whose assignee is common to this application and co-pending with this application, the entire disclosure of which is incorporated herein by reference. This application also claims the benefit of Patent Document 7, which is commonly assigned to this application and is pending at the same time as this application, the entire disclosure of which is incorporated herein by reference. This application also claims the benefit of U.S. Pat. No. 6,053,075, whose assignee is common to this application and is co-pending with this application, the entire disclosure of which is incorporated herein by reference. This application also claims the benefit of U.S. Pat. No. 6,053,075, whose assignee is common to this application and is co-pending with this application, the entire disclosure of which is incorporated herein by reference. This application also claims the benefit of US Pat. No. 6,057,056, whose assignee is common to this application and is pending at the same time as this application, the entire disclosure of which is incorporated herein by reference.
US Patent Application No. 11 / 381,727, Shadon Mao, "NOISE REMOVAL FOR ELECTRONIC DEVICE WITH FAR FIELD MICROPHONE ON CONSOLE", filed May 4, 2006, (Attorney Docket Number SCEA05073US00) US Patent Application No. 11 / 381,729, Shadon Mao, "ULTRA SMALL MICROPHONE ARRAY", filed May 4, 2006, (Attorney Docket Number SCEA05062US00) US Patent Application No. 11 / 381,728, Shadon Mao, "ECHO AND NOISE CANCELATION", filed May 4, 2006, (Attorney Docket Number SCEA05064US00) US Patent Application No. 11 / 381,725, Shadon Mao, "METHODS AND APPARATUS FOR TARGETED SOUND DETECTION", filed May 4, 2006, (Attorney Docket Number SCEA05072US00), US Patent Application No. 11 / 381,724, Shadon Mao, "METHODS AND APPARATUS FOR TARGETED SOUND DETECTION AND CHARACTERIZATION", filed May 4, 2006, (Attorney Docket Number SCEA05079US00) US Patent Application No. 11 / 381,721, Shadon Mao, "SELECTIVE SOUND SOURCE LISTENING IN CONJUNCTION WITH COMPUTER INTERACTIVE PROCESSING", filed May 4, 2006, (Attorney Docket Number SCEA04005 JUMBOUS) PCT Application PCT / US06 / 17483, Shadon Mao, "SELECTIVE SOUND SOURCE LISTENING IN CONJUNCTION WITH COMPUTER INTERACTIVE PROCESSING", filed May 4, 2006, (Attorney Docket Number SCEA04005 JUMBOPCT) US Patent Application No. 11 / 418,988, Shadon Mao, "METHODS AND APPARATUSES FOR ADJUSTING A LISTENING AREA FOR CAPTURING SOUNDS", filed May 4, 2006, (Attorney Docket Number SCEA-00300) US Patent Application No. 11 / 418,989, Shadon Mao, "METHODS AND APPARATUSES FOR CAPTURING AN AUDIO SIGNAL BASED ON VISUAL IMAGE", filed May 4, 2006, (Attorney Docket SCEA-00400) US Patent Application No. 11 / 429,047, Shadon Mao, "METHODS AND APPARATUSES FOR CAPTURING AN AUDIO SIGNAL BASED ON A LOCATION OF THE SIGNAL", filed May 4, 2006, (Attorney Docket Number SCEA-00500)

  In many consumer electronic devices, consoles including various user control devices and input devices are used. In many applications, such as video game consoles, cable TV set / top boxes, and digital video recorders, it is desirable for the console to incorporate a microphone. In order to reduce costs, the microphone is typically a conventional omnidirectional microphone that does not have a preferred receiving direction. Unfortunately, such electronic device consoles also include noise sources such as cooling fans, hard disk drives, CD-ROM drives, and digital video disk (DVD) drives. A microphone located on the console can often interfere with a desired voice input, such as a user voice command. In order to cope with this problem, a technique for removing noise from these noise sources by filtering processing is implemented in these devices.

  The conventional method is effective in removing noise distributed over a wide band by a filtering process. For example, since noise from a fan follows a Gaussian distribution, it is distributed over a wide frequency band. Such noise can be simulated by a Gaussian distribution and can be canceled from the input signal of the console microphone. For example, noise from a disk drive such as a hard disk or a DVD drive is characterized by a narrow band frequency distribution such as a gamma distribution or a narrow band Laplace distribution. Unfortunately, there is no definitive method for gaussian noise and suitable for removing noise that follows a gamma distribution.

[Summary of Invention]
Embodiments of the present invention aim to reduce noise in devices that have a console with one or more microphones, and a narrowband distributed noise source is also located on the console. A microphone signal including a desired sound distributed in a wide band and noise distributed in a narrow band is divided into a plurality of frequency bins. For each frequency bin, it is determined whether a portion of the signal in that frequency bin belongs to the narrow band distribution characteristic of the narrow band noise source located at the console. In order to reduce the narrow band noise, the frequency bin including a part of the signal belonging to the narrow band distribution is filtered.

The teachings of the present invention can be readily understood by considering the following detailed drawings in conjunction with the accompanying drawings, in which:
1 is a schematic view of an electronic device according to an embodiment of the present invention. 2 is a flowchart of a noise reduction method in the apparatus of the type shown in FIG. 3A-3B are graphs illustrating narrowband noise reduction according to embodiments of the present invention and are graphs showing microphone signals as a function of frequency. 4A-4B are graphs illustrating narrowband noise reduction according to another embodiment of the present invention, showing the microphone signals of different microphones as a function of frequency.

[Description of Specific Embodiment]
The following detailed description includes specific details for the purpose of description, but it is normal in the art that many variations and modifications of the following details are possible within the scope of the present invention. It will be understood by those who have knowledge of. Accordingly, the description of the embodiments of the present invention described below does not lose the generality of the invention described in the scope of claims and is not limited thereto.

  As shown in FIG. 1, an electronic device 100 according to an embodiment of the present invention includes a console 102 having one or more microphones 104A, 104B. As used herein, the term console generally refers to a stand-alone unit that includes electronic components that perform computational and / or signal processing functions. The console may receive input from one or more external input devices, such as joystick 106. The console may then provide output to one or more external output devices, such as the monitor 108. The console 102 may include a CPU 110 and a memory 112. The console may optionally include a fan 114 for cooling the components of the console. The console 102 may be a console of a video game system such as Sony PlayStation (registered trademark) or a cable TV set / top box, and is a TiVo digital video recorder provided by TiVoInc in Alviso, California. It may be.

  The processor unit 110 and the memory 112 may be connected to each other via a system bus 116. Microphones 104A and 104B may be connected to a processor and / or memory through input / output (I / O) elements 118. As used herein, the term input / output (I / O) generally refers to any program, operation, or device that transfers data to / from console 100 and to / from peripheral devices. Point to. All data transfers could be considered output from one device and input to another device.

  The device 100 may include one or more additional peripheral units internal to or external to the console 102. Peripheral devices include input only devices such as a keyboard and mouse, output only devices such as a printer, and devices that operate as input and output devices such as an overwritable CD-ROM. Peripheral devices are external devices such as mouse, keyboard, printer, monitor, microphone, game controller, camera, external Zip drive, scanner, CD-ROM drive, CD-R drive, hard disk drive, DVD drive, internal modem And other peripheral devices such as a flash memory reader / writer, hard drive, and the like.

The console includes at least one narrow band distributed noise source, such as a disk drive 120. Narrowband noise from the disk drive 120 is filtered from the digital signal generated from the microphone inputs x _A (t), x _B (t). Thereby, for example, a desired sound such as a sound from the remote source 101 can be prevented from being erased by the sound of the disk drive 120. Narrowband noise may be characterized by a gamma distribution. The desired sound from the source 101 is preferably characterized by a broadband probability density function, such as a Gaussian probability density function.

  Memory 112 may include coded instructions that are executable by processor 110 and / or data 115 that facilitates removal of narrowband disk drive noise. In particular, the data 115 may include a distribution function created from long-time training data of sound recordings from the disk drive. The distribution function may be stored in the form of a lookup table.

The encoded instructions 113 may perform a method 200 for reducing narrowband distributed noise in a device of the type shown in FIG. According to method 200, one or more console microphone input signals 104A, 104B are divided into frequency bins, as shown in step 202. The step of dividing the signal into a plurality of frequency bins includes obtaining a portion of the signal clipped by the time window (eg, microphone signal x _A (t)), and subtracting the portion of the signal clipped by the time window. It may include the steps of converting to a frequency domain signal x (f) (eg, using fast Fourier transform) and dividing the frequency domain signal into frequency bins. In step 204, for example, approximately 32 milliseconds of microphone data may be stored in a buffer for classification into frequency bins. For each frequency bin, it is determined whether the signal within that frequency bin that is part of the signal belongs to the narrow band distribution characteristic of the narrow band disk drive noise. As shown in step 206, frequency bins containing a portion of the signal belonging to the narrowband distribution are removed from the input signal by a filtering process.

  The input signal filtering process will be understood with reference to FIGS. 3A-B. Specifically, as shown in FIG. 3A, the frequency domain signal x (f) could be considered as a combination of a wideband signal 302 and a narrowband signal 304. As shown in FIG. 3B, when these signals are divided into frequency bins 306, each bin includes a value corresponding to a portion of the wideband signal 302 and a portion of the narrowband signal 304. The portion of signal x (f) due to narrowband signal 304 in a given frequency bin (shown in FIG. 3B by a dashed bar graph) could be predicted from the training data. This portion may be subtracted from the value in the frequency bin 306 to remove narrowband noise in that bin by filtering.

  Narrowband signal 304 may be predicted as follows. Initially, narrowband signal samples are collected in a large volume for training the distribution model. Distribution models are well known to those skilled in the art of pattern recognition techniques such as speech modeling. The distribution model for narrowband signal 304 approximates the model used in speech modeling with some exceptions. Specifically, unlike speech that is considered a wideband distribution with a Gaussian distribution, the narrowband noise in the narrowband signal 304 has a “gamma” distribution density function. This distribution model is known as a “gamma mix model”. On the other hand, in a speech application such as speaker / language recognition, a “Gaussian distribution mixed model” is usually used. These two models are very close and differ only in the underlying distribution function. The model training approach follows an “Estimate-Maximize” (EM) algorithm that is widely available in speech modeling. The EM algorithm is an interactive likelihood maximization method that predicts a set of model parameters from a training data set. A feature vector is generated directly from the logarithm of the power spectrum. In contrast, speech models typically apply further compression, such as DCT or septum-coefficient. This is because the signal of interest is distributed in a narrow band and band averaging that leads to attenuation in the broadband background is not desired. In real time, the model is used to predict narrowband noise power spectral density (PSD).

  The algorithm for such a model proceeds as follows.

First, the signal x (t) is transformed from the time domain to the frequency domain.
X (k) = fft (x (t))
Here, k is a frequency index.

Next, a power spectrum is acquired from the frequency domain signal X (k).
S _yy (k) = X (k). ^* Conj (X (k))
Here, “conj” refers to a complex conjugate.

Next, a feature vector V (k) is acquired from the logarithm of the power spectrum.
V (k) = log (S _yy (k))

The term “feature vector” is a word that is widely used in pattern recognition. Basically, arbitrary pattern matching includes 1) a pretrained model that defines a distribution in a priori feature space, and 2) a feature vector whose runtime is observed. The task is to match feature vectors against the model. _Given a pretrained gamma <model>, the probability < _N (k)> that narrowband noise is present can be obtained from this observed feature V (k).
P _n (k) = Gamma (Model, V (k))

The narrowband noise PSD is adaptively updated.
S _nn (k) = α ^* S _nn (k) + (1-α) ^* S _yy (k) ^* P _n (k) + S _nn (k) ^* (1-P _n (k))
If Pn (k) is 0, that is, there is no narrowband noise and S _nn (k) does not change. If P _n (k) = 1, that is, the frequency <k> is completely narrowband noise. Then
S _nn (k) = α ^* S _nn (k) + (1−α) ^* S _yy (k)
Holds. This is basically a statistical periodogram average. Here, α is a smoothing factor.

Predicting a clean speech signal given the prediction noise PSD is not complicated. Examples of algorithms for performing such prediction are well known and are based on MMSE described in Non-Patent Literature 1 and Non-Patent Literature. The disclosures of both documents are incorporated herein by reference.
Y. Ephraim and D. Malah, "Speech enhancement using a minimum mean-square error short-time spectral amplitude estimator," IEEE Trans. Acoust, Speech, Signal Processing, Vol. ASSP-32, 1109-112, 1984 12 Moon D. Malah, "Speech enhancement using a minimum mean-square error log- spectral amplitude estimator," IEEE Trans. Acoust., Speech, Signal Processing, Vol. ASSP-33, pp. 443-445, April 1985

In another embodiment, the presence of two or more microphones on the console 102 may be advantageously utilized in the filtering process. When there are two microphones 104A, 104B on the console 102, one of them (104B) may be closer to the disk drive than the other (104). As a result, the arrival time of noise from the disk drive 120 differs between the microphone input signals x _A (t) and x _B (t). As shown in FIGS. 4A-B, the difference in arrival time is obtained when the input signals x _A (t) and x _B (t) are frequency-converted to x _A (f) and x _B (f), respectively. This results in a difference in frequency distribution. On the other hand, with respect to the fact that the frequency from the remote source is distributed over a wide band, there will not be much difference between x _A (t) and x _B (t). However, the frequency distribution of the narrowband signal 304A from the microphone 104A will be shifted in frequency with respect to the frequency distribution 304B from the microphone 104B. The narrow band noise contribution to the frequency bin 306 can be determined by generating a feature vector V (k) from the frequency domain signals x _A (f), x _B (f) from the two microphones 104A, 104B. .

For example, the first feature vector V (k, A) is generated from the power spectrum S _yy (k, A) for the microphone 104A.
V (k, A) = log (S _yy (k, A))

The second feature vector V (k, B) is generated from the power spectrum S _yy (k, B) for the microphone 104B.
V (k, B) = log (S _yy (k, B))

The feature vector V (k) is obtained by simply connecting V (k, A) and V (k, B).
V (k) = [V (k, 1), V (k, 2)]

  The rest of the model training and real-time detection is the same except that the model size and feature vector dimensions are doubled. The above approach does not use array beam forming or an approach that depends on arrival time differences, but in practice, spatial information is implicitly included in the trained model and runtime feature vectors. And greatly improve the accuracy of detection.

  Embodiments of the present invention may be used as presented herein and may be used with other user input mechanisms. A mechanism for tracking and measuring the azimuth direction and volume of sound, and / or a mechanism for actively or passively tracking the position of an object, a mechanism using machine vision, a combination thereof, and the like. The tracked object may include auxiliary controls and buttons that manipulate feedback to the system. Such feedback may include, but is not limited to, emission of light from the light source, sound quality distortion means, other suitable transmitters, modulators, control devices, buttons, pressure pads, etc. It is not a thing. It may affect the transfer and modulation of the same coding state and / or transfer instructions to and from the device being tracked by the system. Such devices are part of, or interact with, or affect the system used in connection with embodiments of the present invention.

  The above is a complete description of the preferred embodiment of the present invention, but various other variations, modifications, and equivalents are possible. The scope of the invention should, therefore, be determined not by the above description, but should be determined by the following claims, including their full equivalents. The features described herein may be combined with any of the features described herein, whether or not they are preferred. In the following claims, unless expressly stated otherwise, the quantity of each element is one or more. The claims appended hereto are understood to include means plus function limitations in the given claims, except where explicitly indicated using the phrase “means for”. Must not be done.

Claims (21)

A processor readable medium storing a processor readable instruction set for implementing a noise reduction method in an electronic device having a console, wherein the console has one or more microphones, and the console has a narrowband distributed noise. Noise source is located, the processor is connected to the microphone, the memory is connected to the processor,
The processor readable instructions are:
When executed, causes the apparatus to acquire from the one or more microphones a signal including a desired sound distributed in a wide band and narrowband distributed noise from a noise source located on the console. Instructions and
When executed, the instructions allowed to perform the step of dividing the signal into a plurality of frequency bins,
When executed, for each frequency bin, generate a feature vector from the logarithm of the power spectrum of the signal and match the pre-trained model for that feature vector, so that the signal in the frequency bin part, the instructions allowed to perform the step of determining whether belonging to the narrow band distribution characteristic from the noise source located on the console,
When executed, from the signal data generated from a signal from the one or more microphones, and instructions allowed to perform the step of filtering the frequency bins comprising a portion of the signals belonging to the narrow band distribution characteristic,
A processor readable medium comprising:   Determining whether a portion of the signal in the frequency bin belongs to a narrowband distribution characteristic includes: a value corresponding to the portion of the signal in the frequency bin and a noise source located on the console. The processor-readable medium of claim 1, comprising comparing a value derived from a known signal and stored as a value for that frequency bin. The one or more microphones include a first microphone and a second microphone;
Obtaining a signal from the one or more microphones includes obtaining a first signal from the first microphone and obtaining a second signal from the second microphone;
The step of determining whether a part of the signal in the frequency bin belongs to a narrowband distribution characteristic is a step of determining a first vector characteristic from the first signal and obtaining a second vector characteristic from the second signal The processor readable medium of claim 1, comprising: forming a combined feature vector from the first signal and the second signal and matching the combined feature vector against a model. The step of dividing the signal into a plurality of frequency bins comprises:
Capturing a portion of the signal clipped by a time window;
Converting a portion of the signal clipped in the time window into a frequency domain signal;
Dividing the frequency domain signal into a plurality of frequency bins;
The processor readable medium of claim 1 comprising:   The processor-readable medium according to claim 1, wherein the desired sound distributed in a wide band is a voice.   The processor-readable medium of claim 1, wherein the noise source of the narrowband distributed noise is a disk drive.   The processor-readable medium of claim 1, wherein the desired sound distributed over a wide band is characterized by a Gaussian probability density function.   The processor readable medium of claim 1, wherein the narrowband noise is characterized by a gamma distribution probability density function. An electronic device,
Console,
One or more microphones located on the console;
A narrowband noise source located on the console;
A processor connected to the microphone;
A memory connected to the processor and storing a processor readable instruction set for implementing a noise reduction method;
The processor readable instructions are:
When executed, causes the apparatus to acquire from the one or more microphones a signal including a desired sound distributed in a wide band and narrowband distributed noise from a noise source located on the console. Instructions and
When executed, the instructions allowed to perform the step of dividing the signal into a plurality of frequency bins,
When executed, for each frequency bin, generate a feature vector from the logarithm of the power spectrum of the signal and match the pre-trained model for that feature vector, so that the signal in the frequency bin part, the instructions allowed to perform the step of determining whether belonging to the narrow band distribution characteristic from the noise source located on the console,
When executed, from the signal data generated from a signal from the one or more microphones, and instructions allowed to perform the step of filtering the frequency bins comprising a portion of the signals belonging to the narrow band distribution characteristic,
Including electronic devices. Instructions allowed to perform the step of determining whether belonging to the narrow band distribution characteristic of the noise source a portion of the signal in that frequency bin for each frequency bin in the are executed is located on the console,
When executed, a value corresponding to a portion of the signal in the frequency bin and a value stored as the value of the frequency bin derived from a known signal from a noise source located on the console The apparatus of claim 9, comprising one or more instructions for comparing to. Further comprising a lookup table stored in the memory;
The apparatus of claim 10, wherein the look-up table includes the stored value.   The apparatus of claim 9, wherein the one or more microphones include a first microphone and a second microphone. Instructions allowed to perform the step of obtaining a signal when the run from the one or more microphones,
One or more instructions that, when executed, cause the apparatus to perform the steps of obtaining a first signal from the first microphone and obtaining a second signal from the second microphone;
The step of determining whether a part of the signal in the frequency bin belongs to a narrowband distribution characteristic is a step of determining a first vector characteristic from the first signal and obtaining a second vector characteristic from the second signal And forming a combined feature vector from the first signal and the second signal, and matching the combined feature vector to a model. Wherein the step of dividing the signal into a plurality of frequency bins when executed allowed to execute instructions,
Capturing a portion of the signal clipped by a time window;
Converting a portion of the signal clipped in the time window into a frequency domain signal;
Dividing the frequency domain signal into a plurality of frequency bins;
10. The device of claim 9, comprising instructions that cause the device to execute.   The apparatus according to claim 9, wherein the desired sound distributed in a wide band is a voice.   The apparatus according to claim 9, wherein the noise source of the narrow band distributed noise is a disk drive.   The apparatus of claim 9, wherein the desired sound distributed over a wide band is characterized by a Gaussian probability density function.   The apparatus of claim 9, wherein the narrowband noise is characterized by a gamma distribution probability density function.   The apparatus of claim 9, wherein the console is a video game console.   The apparatus according to claim 9, wherein the console is a digital video recorder or a cable TV set top box. A method of reducing noise in a device having a console, the console comprising one or more microphones and a narrowband distributed noise source located on the console;
Obtaining from the one or more microphones a signal comprising a desired sound distributed over a wide band and narrowband distributed noise from a noise source located on the console;
Dividing the signal into a plurality of frequency bins;
By generating a feature vector from the logarithm of the power spectrum of the signal and matching against a pretrained model for that feature vector, for each frequency bin, a portion of the signal in that frequency bin is Determining whether it belongs to a narrowband distribution characteristic from a noise source located above;
Filtering frequency bins including a portion of the signal belonging to a narrowband distribution characteristic from signal data generated from signals from the one or more microphones ;
A method comprising:

Images