JP5507596B2 - Speech enhancement - Google Patents

Abstract

A method for enhancing speech includes extracting a center channel of an audio signal, flattening the spectrum of the center channel, and mixing the flattened speech channel with the audio signal, thereby enhancing any speech in the audio signal. Also disclosed are a method for extracting a center channel of sound from an audio signal with multiple channels, a method for flattening the spectrum of an audio signal, and a method for detecting speech in an audio signal. Also disclosed is a speech enhancer that includes a center-channel extract, a spectral flattener, a speech-confidence generator, and a mixer for mixing the flattened speech channel with original audio signal proportionate to the confidence of having detected speech, thereby enhancing any speech in the audio signal.

Description

  The present specification provides a method and apparatus for extracting a central channel of sound from an audio signal with a plurality of channels, flattening the spectrum of the audio signal, detecting speech in the audio signal, and enhancing the speech. explain. The method of extracting a central channel of sound from an audio signal by a plurality of channels includes (1) a first channel of an audio signal smaller than a ratio α of candidate central channels and (2) an audio signal smaller than a ratio α of candidate central channels. Multiplying the conjugate of the second channel to approximately minimize α and multiplying the candidate center channel by the generally minimized α to form an extracted center channel.

  A method for flattening the spectrum of an audio signal separates the estimated speech channel into perceptual bands to determine which of the perceptual bands has the most energy and has less energy It may include flattening the spectrum of any speech in the audio signal by increasing the gain of the perceptual band. This increase may include increasing the gain of the perceptual band having less energy to a maximum.

  A method for detecting speech in an audio signal measures the spectral variation in a candidate central channel of the audio signal, measures the spectral variation of an audio signal less than the candidate central channel, and compares these spectral variations to determine the audio signal. Detecting speech at.

  The method of enhancing speech enhances any speech in the audio signal by extracting the center channel of the audio signal, flattening the spectrum of the center channel, and mixing the audio signal into the flattened speech channel May include. The method may further include generating a confidence level for speech detection in the center channel, and the mixing causes the audio signal in the flattened speech channel to be proportional to the confidence level with the detected speech. Mixing may be included. Its confidence can vary from the lowest likelihood probability to the highest probability, so its generation is for values that are higher than the lowest probability and lower than the highest probability. It may further include limiting the generated confidence. The extraction may include extracting the center channel of the audio signal using the method described above. The flattening described above may include flattening the center channel spectrum using the method described above. The generation described above may include generating confidence in speech detection in the center channel using the method described above.

  The extraction described above may include extracting the central channel of the audio signal using the method described above, and the flattening described above may include flattening the spectrum of the central channel using the method described above. , Generating as described above may include generating confidence in speech detection in the center channel using the method described above.

  This specification includes not only a computer-readable recording medium storing a computer program for executing any of the above-described methods, but also a CPU, the recording medium, and a bus that couples the CPU and the recording medium. Teach computer system including.

FIG. 3 is a functional block diagram of a speech enhancer according to one embodiment of the present invention. FIG. 5 represents a suitable set of filters with a spacing of 1 ERB that provides a total of 40 bands. FIG. 6 illustrates a mixing process according to one embodiment of the present invention. FIG. 3 illustrates a computer system according to one embodiment of the invention.

  FIG. 1 is a functional block diagram of a speech enhancer according to one embodiment of the present invention. The speech enhancer 1 includes an input signal 17, discrete Fourier transformers 10a and 10b, a center channel extractor 11, a spectrum flattener 12, a speech activity detector 13, variable gain amplifiers 15 and 15c, an inverse discrete Fourier transformer 18a, 18b and an output signal 18. The input signal 17 comprises left and right channels 17a and 17b, respectively, and similarly the output signal 18 comprises left and right channels 18a and 18b, respectively.

  Each discrete Fourier transformer 18 receives as input the left and right channels 17a, 17b of the input signal 17 and forms transforms 19a, 19b as outputs. The center channel extractor 11 receives the transformation 19 and forms a temporary center channel C20 as output. Spectral flattener 12 receives as input a temporary central channel C20 and forms a shaped central channel 24 as an output, while vocal activity detector 13 receives the same input C20, while variable gain amplifier 14a. And the control signal 22 for the variable gain amplifier 14b on the other hand are formed as outputs.

  The amplifier 14a receives the left channel conversion 19a and the output control signal 22 of the vocal activity detector 13 as input and control signals, respectively. Similarly, amplifier 14c receives right channel conversion 19b and voicing activity detector output control signal 22 as input and control signals, respectively. The amplifier 14b receives the spectrally shaped central channel 24 and the output speech activity detector control signal 21 of the spectral flattener 12 as input and control signals.

  The mixer 15a receives the gain adjusted left transform 23a, which is the output from the amplifier 14, and the gain adjusted spectrally shaped central channel 25 and forms the signal 26a as output. Similarly, mixer 15b receives gain adjusted right conversion 23b from amplifier 14c and gain adjusted spectrally shaped center channel 25 and forms signal 26b as an output.

Inverse converters 18a, 18b receive the respective signals 26a, 26b and form derived left and right channel signals L′ 18a and R′18b, respectively.
The operation of the speech enhancer 1 will be described in further detail below. The process of center channel extraction, spectral flattening, vocal activity detection and mixing will be described in order, first generally and then in more detail, according to one embodiment.
Center channel extraction Assuming that:
(1) The signal of the object 17 includes speech.
(2) For multi-channel signals (ie left and right, or stereo), the speech is panned to the center.
(3) The actual panned center consists of the sound source left / right signal ratio alpha (α).
(4) The result of the ratio subtraction is a pair of orthogonal signals.

  Operating under these assumptions, the center channel extractor 11 extracts from the stereo signal 17 the center panned content C20. For content that is panned to the center, the same area of both the left and right channels contains the content panned to the center. The content panned to the center is extracted by removing the same part from both the left and right channels.

  For the remaining left and right signals (using a method that is continuously updated on the frame of the block or each time a new block enters), calculate LR * = 0 (where * indicates the conjugate) and calculate the ratio α Adjustment may be made until the value becomes sufficiently close to zero.

Spectral flattening The auditory filter separates speech in the estimated speech channel into perceptual bands. The band with the most energy is determined for each block of data. The spectral shape of the speech channel for that block is modified to compensate for the low energy in the remaining bands. This spectrum is flattened. The r band with low energy has a gain increased to some maximum. In one embodiment, all bands may share maximum gain. In an alternative embodiment, each band may have its own maximum gain. (If it is not desirable that all bands have the same energy, the spectrum is already flat. It would also be considered that no spectral shaping occurs or that the spectral shaping is achieved by the same function. )
Spectral flattening occurs independently of the channel content. Non-speech may be processed, but this is not used later in the system. Since non-speech has a very different spectrum than speech, the flattening for non-speech is usually not the same as for speech.
Speech Activity Detector Once the estimated speech is separated into a single channel, it is analyzed for speech content (does it include speech?). Content is analyzed independently of spectral flattening. Speech content is determined by measuring spectral variations in adjacent frames of data. (Each frame can consist of many blocks of data, but a frame is typically 2, 4, or 8 blocks at a 48 kHz sample rate.)
Where the speech channel is extracted from stereo, the remaining stereo signal will be useful for speech analysis. This concept is generally applied by adjacent channels in any multi-channel source.
When mixing speech is considered to be present, the flattened speech channel is mixed with the original signal at a rate that is related to the confidence that the speech channel actually contains speech. In general, when the reliability is high, more flattened speech channels are used. When the confidence level is low, fewer flattened speech channels are used.

The process of center channel extraction, spectral flattening, vocal activity detection and mixing will be described in turn in more detail by one embodiment.
Temporary center and surround channel extraction from a two-channel source With speech enhancement, it is desirable to extract, process, and reinsert only center-panned audio. In stereo mixing, speech is most often panned to the center.

  Here, the extraction from the two-channel mixture of the audio panned in the center (provisional center channel) will be described. Mathematical proof forms the first part. The second part applies this proof to the real environment stereo signal to derive a temporary center.

  When the temporary center is removed from the original stereo, a stereo signal having orthogonal channels remains. A similar method derives a temporary surround channel from the audio panned to the periphery.

Center channel extraction-mathematical proof Given a two channel signal, the channel will be divided into left (L) and right (R). Each of the left and right channels includes not only common information but also unique information. Common information can be represented as C (panned in the center) and unique information can be represented as L and R for left only and right only, respectively.

“Inherent” means that L and R are orthogonal to each other.

Dividing L and R into real and imaginary parts,

Here, Lr is the real part of L, Li is the imaginary part of L, and the same applies to R. Now, it is considered that an orthogonal pair (L and R) is formed from a non-orthogonal pair (L and R) by subtracting C panned in the center from L and R.

Where C = αC (where C is the estimated center channel and α is the magnification)

Substituting Equation (6) and Equation (7) into Equation (3),

Equation (8) takes the form of a quadratic equation,

Here, the power root is obtained as follows.

Here, C in Equation (6) and Equation (7) is

As follows:

Then, in the quadratic equation (9),

Substituting Equation (14), Equation (15) and Equation (16) into Equation (10) and solving for α,

A negative root is chosen for the solution for α and α is limited to the range {0, 0.5} to avoid confusion due to information panned around it (however, its value is not important to the present invention). The provisional central channel type is as follows.

here,

It is. (The min {} and max {} functions limit α to the range {0, 0.5}, but its value is not important to the present invention.)
The temporary surround channel can be derived as follows.

Where S is the audio panned around in the original stereo pair (L, R) and S is assumed to be (LR). Again, a negative root is chosen for the solution for β and β is limited to the range {0, 0.5} to avoid confusion due to information panned to the periphery (however, its value is not important to the present invention). Absent).

  Now that C and S have been derived, they can be removed from the original stereo pair (L and R) to form four channels of audio from the two original channels. That is,

Here, L ′ is a derived left channel, C is a derived center channel, R ′ is a derived right channel, and S is a derived surround channel.

Central Channel Extraction-Application As mentioned above, for speech enhancement methods, the main concern is the extraction of the central channel. In this part, the techniques described above are applied to complex frequency domain representations of audio signals.

The first stage of tentative center channel extraction is to perform a DFT on the block of audio samples and obtain the resulting transform coefficients. The DFT block size depends on the sampling rate. For example, at a sampling rate f _S of 48 kHz, a block size of N = 512 samples is possible. A block of samples is weighted prior to applying the transform by a windowing function w [n], such as a Hamming window.

Where n is an integer and N is the number of samples in the block.

  The DFT coefficient is calculated by the following equation (25) as follows.

Where x [n, c] is the sample number n in channel c of block m, j is the imaginary unit (j ² = −1), and X _m [k, c] is the channel for the sample in block m. The conversion coefficient k in c. Note that the number of channels is three: left, right, and tentative center (in the case of x [n, c], only left and right). In the following equations, the left channel is represented as c = 1, the temporary center channel is represented as c = 2 (not yet derived), and the right channel is represented as c = 3. Fast Fourier transform (FFT) efficiently performs DFT.

  In principle, the sum and difference between left and right were determined for each frequency bin. The real and imaginary parts were grouped and squared. Each bin was smoothed between blocks prior to calculating α. This smoothing reduces audible artifacts (which occur when the power in the bin changes abruptly between blocks of data). Smoothing may be performed, for example, with a leaky integrator, a non-linear smoother, a linear and multipolar low-pass smoother, or a more sophisticated smoother.

Here, Re {} is a real part, Im {} is an imaginary part, and λ ₁ is a leaky integration circuit coefficient. The leakage integrator circuit has a low-pass filtering effect, with a typical value for λ ₁ being 0.9. Next, the extraction coefficient α for the block m is derived as follows using Equation (19).

And the temporary center channel about the block m is derived | led-out as follows using Formula (18).

Spectral flattening Examples of spectral flattening according to the present invention will be described below. Assuming a single channel that is mostly speech, the speech signal is transformed to the frequency domain by a discrete Fourier transform (DFT) or related transform. The amplitude spectrum is converted to a power spectrum by squaring the conversion frequency bin.

  The frequency bins are then classified into bands that are possible with a critical or auditory filter scale. Dividing a speech signal into critical bands is very similar to the human auditory system (especially the cochlea). These filters exhibit a generally rounded exponential shape and are evenly spaced on an equivalent length rectangular bandwidth (ERB) scale. This ERB scale is just a measure used in psychoacoustics and approximates the bandwidth and spacing of the auditory filter. FIG. 2 represents a suitable set of filters with 1 ERB spacing, resulting in a total of 40 bands. Audio data banding also helps to eliminate audible artifacts (which in principle occur when processing bin by bin). The critical band power is then smoothed over time. That is, smoothing is performed over adjacent blocks.

  Find the maximum output of the smoothed critical bands and calculate the corresponding gain for the remaining (non-maximum) bands to approximate those outputs to the maximum output. Gain compensation is similar to the compression (nonlinear) characteristics of the basement membrane. These gains are limited to maximum values to avoid saturation. In order to apply these gains to the original signal, they must be converted back to the DFT format. Thus, the per band output gain is first converted back to a frequency bin output gain, and then the per bin output gain is converted to an amplitude gain by taking the square root of each bin. Thus, the original signal conversion bin can be multiplied by the calculated per-bin amplitude gain. The spectral flattening signal is then transformed back from the frequency domain to the time domain. In the tentative center, this is first mixed with the original signal prior to returning to the time domain. FIG. 3 illustrates the process.

The spectral flattening system described above does not take into account the characteristics of the input signal. If the non-speech signal is flattened, the perceptible change in sound quality will be severe. In order to avoid processing of non-speech signals, the method described above can be tied to the speech activity detector 13. When speech activity detector 13 indicates the presence of speech, flattened speech is used.

  Assume that the signal to be flattened has already been transformed to the frequency domain as described above. For simplicity, the channel notation used above is omitted. The DFT coefficients are converted to output and then converted from the DFT domain to the critical band.

Here, H [k, p] is a P critical band filter.

  The output in each band is then smoothed between blocks, similar to the time integration that occurs at the cortical level of the brain. Smoothing may be performed, for example, by a leaky integration circuit, a non-linear smoother, a linear and multipole low-pass smoother, or a more sophisticated smoother. This smoothing also helps to eliminate the transition behavior, which causes the gain to suddenly fluctuate between blocks, resulting in audible pumping. Next, the peak output is obtained as follows.

Here, E _m [p] is the smoothed critical band output, λ ₂ is the leakage integrator circuit coefficient, and E _max is the peak output. The leakage integrator circuit has a low-pass filtering effect, and its gain is typically 0.9 for λ ₂ .

  Then find the band-by-band output gain and limit the maximum gain to avoid excessive compensation,

Get. Here, G _m [p] is an output gain to be applied to each band, G _max is an allowable maximum output gain, and γ determines the degree of spectrum flattening. In practice, γ approximates 1. G _max depends on the dynamic range (or no distortion limit) if the system performs processing as well as other general limits on the amount of gain specified. A typical value for G _max is 20 dB.

  Next, the output gain per band is converted into the output per bin, and the square root is taken to obtain the amplitude gain per bin.

Here, Y _m [k] is an amplitude gain per bin.

The amplitude gain is then modified based on the voice activity detector outputs 21,22. A method for voicing activity detection is described below according to one embodiment of the present invention.
Voice activity detection The spectral bundle measures the rate at which the output spectrum of the signal changes and compares the output between adjacent frames of audio. (A frame is a plurality of blocks of audio data.) Spectral bundles indicate voicing activity detection or “other decisions in speech vs. audio classification”. In many cases, additional indicators are used and the results are accumulated to make a determination as to whether the audio is really speech.

In general, the spectral bundle of speech is slightly higher than that of music. That is, the music spectrum tends to be more stable than the speech spectrum between frames.
In the case of stereo, where the center channel of the spectrum is extracted, the DFT coefficients are first divided into center and side audio (original stereo minus the temporary center). This is different from traditional mid / horizontal stereo processing, where traditional mid / horizontal stereo processing is typically (L + R) / 2, (LR) / 2, whereas center / lateral processing. Is C, L + R-2C.

  According to the signal converted to the frequency domain as described above, the DFT coefficients are converted to the output, and then converted from the DFT domain to the critical band domain. The critical band output is then used to calculate both the central and lateral spectral bundles.

Where X _m [p] is a temporary central critical band representation, S _m [p] is the critical band representation of the remaining signal (the sum of left and right subtracted from the center), and H [k, p] is a P critical band filter as described above.

  Two frame buffers are formed (for center and lateral amplitude) from the 2J block of data to puncture.

The next stage calculates the weight W for the center channel from the average output of the current frame and the previous frame. This is done over a limited range of bands.

The range of the band is limited to the main bandwidth of speech, about 100-8000 Hz. The unweighted spectral bundle for both the center and the side is calculated as follows:

Here, F _x (m) is a center unweighted spectrum bundle, and F _s (m) is a side unweighted spectrum bundle.

  Therefore, an estimated value with a biased spectrum bundle is calculated as follows.

If it is,

Otherwise,

It is. Here, F _Tot (m) is the total bundle estimated value, and W _min is the allowable minimum weight. W _min depends on the dynamic range, but a typical value is W _min = −60 dB.

The final smoothed value for the spectral bundle is calculated by low pass filtering the value of F _Tot (m) with a simple first order IIR low pass filter. This filter is dependent on the size of the sample rate and block signals, in one embodiment, can be defined by the primary low-pass filter with a normalized cutoff f s ₌ 48kHz for 0.025 * f _s. Where f _s is the sample rate of the digital system.

F _Tot (m) is thus shortened to the following range: That is,

Because

(The min {} and max {} functions limit F _Tot (m) to {0, 1} according to this embodiment.)
Mixing The flattened center channel is mixed with the original audion signal based on the output of the vocal activity detector.

The amplitude gain Y _m [k] per bin for spectral flattening (as described above) is applied to the temporary central channel X _m [k, 2] (as derived above).

When the speech activity detector 13 detects speech, F _Tot (m) = 1, and when it detects non-speech, F _Tot (m) = 0. Values between 0 and 1 are possible, which is obtained when the speech activity detector 13 makes a soft decision on the presence of speech.

  For the left channel

Similarly, for the right channel,

In practice, F _Tot is limited to a narrow range of values. For example

Preserves a small amount of both the flattening signal and the original signal in the final mixture.

  A bin-by-bin amplitude gain is then applied to the original input signal, which is transformed via the inverse DFT back to the time domain.

here

Is an enhanced form of x, the original stereo input signal.

  FIG. 4 shows a computer 4 according to an embodiment of the present invention. The computer 4 includes a memory 41, a CPU 42, and a bus 43. The bus 43 is connected so as to communicate with the memory 41 and the CPU 42. The memory 41 stores a computer program for executing any method described above.

  Several embodiments of the invention have been described. Nevertheless, those skilled in the art will understand how to make various modifications to the described embodiments without departing from the spirit and scope of the present invention. For example, although the description includes a discrete Fourier transformer, those skilled in the art will understand various alternative methods of transforming from the time domain to the frequency domain and vice versa.

Conventional technology

Schaub, A.M. and P.M. , “Spectral sharpening for speech enhancement noise reduction”, Proc. ICASSP. 1991, Toronto, Canada, May 1991, pp. 993-996.

Sondhi, "New methods of pitch extraction", Audio and Electroacoustics, IEEE Transactions, June 1968, Volume 16, Issue 2, pp 266-266.

Villchul, E .; , "Signal Processing to Improve Speech Integility for the Healing Implied", 99th Audio Engineering Society Convention, 1995.

Thomas, I.D. and Niederjohn, R.A. , “Preprocessing of Speech for Added Intelligent in High Ambient Noise”, 34th Audio Engineering Society Convention, March 1968.

Moore, B.B. et. al. "A Model for the Prediction of Thresholds, Loudness, and Partial Loudness", J. et al. Audio Eng. Soc, Vol. 45, no. 4, April 1997.

Moore, B.B. and Oxenham, A.A. , “Psychoacoustic consensus of compression in the peripheral auditory system, The Journal of the Austicum of Ace. 2962-2966

Prior Art Spectral Flattening U.S. Pat. No. 6,673,073 B1 Title of invention “Spectral enhancement of acoustic signals to provided imprinted recognition of speech”

U.S. Patent No. 0993480 B1 Title of Invention "Voice intelligence enhancement system"

US Patent No. 2006/026320 A1 “Apparatus and method for noise reduction and speech enhancement with microphones and loudspeakers”

U.S. Pat. No. 07191122 Name of invention “Speech compression system and method”

US Patent No. 2007/0094017 Title of invention “Frequency domain format enhancement”

International patent

WO 2004/013840 Al No. of Invention “Digital Signal Processing Technologies For Improving Audio Clarity And Intelligence”

WO 2003/015082 Title of Invention “Sound Intelligent Enhancement Using A Psychoacoustic Model And An Oversampled Filterbank”


paper

Sallberg, B.M. et. “Analog Circuit Implementation for Speech Enhancement Proposals Signals”; Systems and Computers, 2004. al; Conference Record of the Thirty-Eighth Asilomar Conference.

Magotra, N .; and Sirivara, S .; "Real-time digital speech processing strategies for the sharing implied"; Acoustics, Speed, and Signal Processing. 1997. ICASSP-97. 1997 page (s): 1211-1214 vol. 2

Walker, G.M. Byrne, D .; , And Dillon, H .; ; "The effects of multichannel compression / expansion amplification on the intelligibility of nonsense syllables in noise"; The Journal of the Acoustical Society of America - September 1984 - Volume 76, Issue 3, pp. 746-757

Conventional technology Central extraction

Adobe Audit has a vocal / instrument extraction function
http: // www. Adobeforms. eom / cgi-bin / webx /. 3bc3a3e5

"Center cut" for winamp
http: // www. hvdrogenaudio. org / forums / loloversion / index. php / tl7450. html

Prior art Spectrum bundle

Vinton, M, and Robinson C; "Automated Speech / Other Discrimination for Loudness Monitoring," AES 118th Conv. 2005

Scheiler E. , And Slaney M .; , “Construction and evaluation of a robotic multi-spec feature / music discriminator”, IEEE Transactions on Acoustics, Speech, and Signal Prop. 1331 --- 1334.

Claims (7)

A method for enhancing speech,
Extract the center channel of the audio signal
Generating a confidence level for speech detection in the central channel;
Flatten the spectrum of the central channel,
A method comprising enhancing any speech in an audio signal by mixing the audio signal into the flattened speech channel in proportion to the reliability of the detected speech. The method of claim 1, wherein
The reliability varies from the lowest possibility probability to the highest possibility probability,
The generation is
The method further comprising further limiting the confidence level generated to a value that is higher than the lowest probability of probability and lower than the highest probability of probability. The method of claim 1, wherein extracting the central channel of the audio signal comprises:
Obtaining a temporary center channel from the sum of the first channel of the audio signal and the second channel of the audio signal;
Multiplying the first channel of the audio signal smaller than the temporary central channel ratio α by the second channel conjugate with the first channel of the audio signal smaller than the temporary central channel ratio α. Calculate the product,
Obtain the extraction coefficient from the value of α that minimizes this product,
Obtaining the extracted center channel by multiplying the temporary center channel by the extraction factor. The method of claim 3, wherein flattening the spectrum of the center channel comprises:
Separating the central channel into perceptual bands;
Determine which of the perceptual bands has the most energy,
Increasing the gain of a perceptual band having less energy. A computer-readable recording medium storing a computer program for executing the method according to claim 1. A computer system,
CPU,
A recording medium according to claim 5;
A computer system including a bus for coupling the CPU and the recording medium. A speech enhancer,
A center channel extractor for extracting the center channel of the audio signal;
A spectrum flattener for flattening the spectrum of the central channel;
A speech confidence generator for generating confidence in speech detection in the central channel;
A speech enhancer comprising: a mixer that enhances any speech in the audio signal by mixing the flattened speech channel with the original audio signal in proportion to the reliability of the detected speech.