JP5666444B2 - Apparatus and method for processing an audio signal for speech enhancement using feature extraction - Google Patents

Description

  The present invention relates to the field of audio signal processing, and in particular, to the field of speech enhancement of audio signals where the speech content of the processed signal has objective and subjective speech intelligibility.

  Speech enhancement is applied in various applications. The main application is the use of digital signal processing in hearing aids. Digital signal processing in hearing aids provides a new and effective means for the recovery of hearing impairment. Besides higher acoustic signal quality, digital hearing aids enable the implementation of specific speech processing strategies. In many of these strategies, it is desirable to estimate the speech to noise ratio (SNR) of the acoustic environment. Specifically, a complex algorithm for speech processing may be optimized for a specific acoustic environment, but in situations where it does not meet specific assumptions, such an algorithm does not work well there is a possibility. This is especially true for noise reduction schemes that can introduce processing artifacts in quiet environments or situations where the SNR is below a predetermined threshold. Since the optimal choice of compression algorithm and amplification parameters may depend on the speech-to-noise ratio, adjusting the parameter set in response to the SNR estimate helps to be effective. Furthermore, SNR estimation can also be used directly as a control parameter for noise reduction techniques such as Wiener filtering or spectral subtraction techniques.

  Another application is in the area of speech enhancement for movie audio. Many people have found it difficult to understand the speech content of movies, for example due to hearing impairment. In order to follow the story of a movie, it is important to understand the relevant speech of the audio track, such as monologue, conversation, announcements and narration. People who find it difficult to hear often feel that the level of background sounds, such as ambient noise and music, is too high compared to speech. In this case, it is desirable to increase the level of the speech signal and weaken the background sound. In other words, it is desirable to increase the level of the speech signal relative to the overall level.

  The main method of speech enhancement is spectral weighting, also called short-time spectral attenuation, as shown in FIG. The output signal y [k] is calculated by attenuating the subband signal X (ω) of the input signal x [k] according to the noise energy in the subband signal.

In the following, it is assumed that the input signal x [k] is an additive mixture of the desired speech signal s [k] and the background noise b [k].

  Speech enhancement is an improvement in the objective clarity and / or subjective quality of speech.

The frequency domain representation of the input signal is computed by a short time Fourier transform (STFT), other time-frequency transform, or a filter bank, as shown in block 30. The input signal is then filtered in the frequency domain according to equation (2), while the frequency response G (ω) of the filter is calculated to reduce noise energy. The output signal is calculated by the inverse processing of the time-frequency conversion or the filter bank, respectively.

を使用し、あるいは線形サブバンドＳＮＲ

を使用して、各スペクトル値についてブロック３１において計算される。重み付けされたスペクトル値は、ブロック３２において再び時間ドメインへと変換される。雑音抑制規則の有名な例は、スペクトルサブトラクション法（非特許文献１）及びウィーナーフィルタリングである。入力信号がスピーチ信号と雑音信号との加算混合物であり、スピーチ及び雑音が相関していないと仮定すると、スペクトルサブトラクション法のためのゲイン値は式（３）において与えられる。

Appropriate spectral weights G (ω) are used to determine the input signal spectrum X (ω) and noise spectrum estimation.

Or linear subband SNR

Is calculated at block 31 for each spectral value. The weighted spectral values are converted back to the time domain at block 32. Famous examples of noise suppression rules are the spectral subtraction method (Non-Patent Document 1) and Wiener filtering. Assuming that the input signal is a summed mixture of speech and noise signals and the speech and noise are not correlated, the gain value for the spectral subtraction method is given in equation (3).

からも導出される。
チャネル

Similar weights are used for linear subband SNR estimation according to equation (4).

Is also derived from
channel

  Various extensions of spectral subtraction methods: use of oversubtraction coefficients and spectral floor parameters (non-patent document 2), generalized form (non-patent document 3), use of perceptual criteria (eg non-patent document 4), And multiband spectral subtraction (eg, Non-Patent Document 5) have been proposed in the past. However, a critical part of the spectral weighting method is the estimation of the instantaneous noise spectrum or subband SNR, which tends to suffer from errors, especially when the noise is not static. Errors due to noise estimation lead to artifacts described as residual noise, distortion of speech components or musical noise ("warbling with tonal quality") (non-patent document 6).

A simple approach for noise estimation is to measure and average the noise spectrum during speech pauses. This approach does not give satisfactory results when the noise spectrum changes over time during speech activity and when speech pause detection fails. Methods for estimating the noise spectrum during speech activities have been proposed in the past, and according to Non-Patent Document 6,
・ Minimum tracking algorithm ・ Time recursive average algorithm ・ Histogram-based algorithm

  Non-patent document 7 proposes estimation of a noise spectrum using minimum statistics. This method is based on tracking the local minimum of signal energy in each subband. Non-patent document 8 proposes a non-linear update rule for noise estimation and faster update.

  The time-recursive averaging algorithm estimates and updates the noise spectrum whenever the estimated SNR for a particular frequency band is very low. This is done by recursively calculating a past noise estimate and a weighted average of the current spectrum. For example, in Non-Patent Document 9 and Non-Patent Document 10, the weight is determined as a function of the probability that speech is present, or as a function of the estimated SNR of a specific frequency band.

  Histogram-based methods are based on the assumption that histograms of subband energy often have two modes. Large low energy modes accumulate the energy values of segments that do not include speech or include the low energy portion of speech. The high energy mode accumulates the energy values of the segments including voiced speech and noise. The noise energy of a specific subband is determined from the low energy mode (Non-Patent Document 11). Reference is made to Non-Patent Document 6 for a comprehensive recent review.

  Methods for estimating subband SNR based on supervised learning using the characteristics of amplitude modulation are reported in Non-Patent Document 12 and Non-Patent Document 13.

  Other techniques for speech enhancement include pitch-synchronous filtering (eg, Non-Patent Document 14), spectral time modulation (STM) filtering (eg, Non-Patent Document 15), and a sine wave model of an input signal. This is filter processing based on expression (for example, Non-Patent Document 16).

  The method for estimating subband SNR based on supervised learning using the features of amplitude modulation, as reported in Non-Patent Document 12 and Non-Patent Document 13, requires two spectrogram processing stages. This is disadvantageous. The first spectrogram processing stage is to generate a time / frequency spectrogram of the time domain audio signal. Then, another “time / frequency” transform that transforms the spectral information from the spectral domain to the modulation domain is required to generate a modulation spectrogram. This additional conversion operation poses a problem because of the inevitable delays from the system, as well as the time / frequency resolution problems inherent in any conversion algorithm.

  A further result of this procedure is that the noise estimation becomes quite inaccurate in situations where the noise is not static and various noise signals can occur.

S. Boll, “Suppression of acoustic noise in speech using spectral subtraction”, IEEE Trans. On Acoustics, Speech, and Signal Processing, vol. 27, no. 2, pp. 113-120, 1979 M. Berouti, R. Schwartz, J. Makhoul, “Enhancement of speech corrupted by acoustic noise”, Proc.of the IEEE Int. Conf. On Acoustics, Speech, and Signal Processing, ICASSP, 1979 J. Lim, A. Oppenheim, “Enhancement and bandwidth compression of noisy speech”, Proc. Of the IEEE, vol 67, no. 12, pp. 1586-1604, 1979 N. Virag, “Single channel speech enhancement based on masking properties of the human auditory system”, IEEE Trans. Speech and Audio Proc., Vol. 7, no. 2, pp. 126-137, 1999 S. Kamath, P. Loizou, “A multi-band spectral subtraction method for enhancing speech corrupted by colored noise”, Proc. Of the IEEE Int. Conf. Acoust. Speech Signal Processing, 2002 P.Loizou, Speech Enhancement: Theory and Practice, CRC Press, 2007 R. Martin “Spectral subtraction based onminimum statistics”, Proc.of EUSIPCO, Edingburgh, UK, 1994 G. Doblinger, “Computationally Efficient Speech Enhancement By Spectral Minima Tracking In Subbands”, Proc. Of Eurospeech, Madrid, Spain, 1995 I. Cohen, “Noise estimation by minima controlled recursive averaging for robust speech enhancement”, IEEE Signal Proc. Letters, vol. 9, no. 1, pp. 12-15, 2002 L. Lin, W. Holmes, E. Ambikairajah, “Adaptive noise estimation algorithm for speech enhancement”, Electronic Letters, vol. 39, no. 9, pp. 754-755, 2003 H. Hirsch, C. Ehrlicher, “Noise estimation techniques for robust speech recognition”, Proc. Of the IEEE Int. Conf. On Acoustics, Speech, and SignalProcessing, ICASSP, Detroit, USA, 1995 J. Tchorz, B. Kollmeier, “SNR Estimation based on amplitude modulation analysis with applications to noise suppression”, IEEE Trans. On Speech and Audio Processing, vol. 11, no. 3, pp. 184-192, 2003 M. Kleinschmidt, V. Hohmann, “Sub-band SNR estimation using auditory feature processing”, Speech Communication: Special Issue on Speech Processing for HearingAids, vol. 39, pp. 47-64, 2003 R. Frazier, S. Samsam, L. Braida, A. Oppenheim, “Enhancement of speech by adaptive filtering”, Proc. Of the IEEE Int. Conf.on Acoustics, Speech, and Signal Processing, ICASSP, Philadelphia, USA, 1976 N. Mesgarani, S. Shamma, “Speech enhancement based on filtering the spectro-temporal modulations”, Proc. Of the IEEE Int. Conf.on Acoustics, Speech, and Signal Processing, ICASSP, Philadelphia, USA, 2005 J. Jensen, J. Hansen, “Speech enhancement using aconstrained iterative sinusoidal model”, IEEE Trans. On Speech and Audio Processing, vol. 9, no. 7, pp.731-740, 2001 H. Hermansky, N. Morgan, “RASTA Processing of Speech”, IEEE Trans. On Speech and Audio Processing, vol. 2, no. 4, pp. 578-589, 1994 H. Hermansky, “Perceptual Linear Predictive Analysis for Speech”, J. Ac. Soc. Am., Vol. 87, no. 4, pp. 1738-1752, 1990

  It is an object of the present invention to provide an improved concept for speech enhancement.

  According to a first aspect, the object is an apparatus for processing audio signals to obtain control information for a speech enhancement filter, obtaining a time sequence of a short-time spectral representation of the audio signals, A feature extractor that extracts at least one feature in each frequency band of a plurality of frequency bands for short-time spectrum expression, wherein the at least one feature is one short in one frequency band of the plurality of frequency bands. Control for the speech enhancement filter for one time portion of the audio signal by combining a feature extractor representing the spectral shape of the time spectral representation with the at least one feature of each frequency band using a combining parameter And a feature combiner for obtaining information.

  According to a second aspect, the object is a method for obtaining control information for a speech enhancement filter by processing an audio signal, obtaining a time sequence of a short-time spectral representation of the audio signal; Extracting at least one feature in each frequency band of a plurality of frequency bands for a short-time spectral representation of the plurality of frequency bands, wherein the at least one feature is one short time in one frequency band of the plurality of frequency bands Representing a spectral shape of a spectral representation and combining the at least one feature of each frequency band using a coupling parameter to obtain control information for the speech enhancement filter for one time portion of the audio signal And the method comprising: steps.

  According to a third aspect, the object is an apparatus for speech enhancement in an audio signal, wherein the audio signal is obtained in order to obtain filter control information for a plurality of bands representing a time portion of the audio signal. Achieved by a device comprising: a processing device; and a controllable filter that can be controlled based on the control information to variably attenuate one band of the audio signal compared to another band Is done.

  According to a fourth aspect, the object is a method for speech enhancement of an audio signal, wherein the audio signal is processed and filter control information is obtained for a plurality of bands representing one time portion of the audio signal. And controlling the filter based on the control information so that a certain band of the audio signal is variably attenuated as compared with the other bands.

  According to a fifth aspect, the object is a device that trains the feature combiner to determine a combination parameter of the feature combiner, wherein control information for each frequency band to the speech enhancement filter is A feature of acquiring a time sequence of a short-time spectrum expression of the learning audio signal that is known and extracting at least one feature in each frequency band of a plurality of frequency bands for a plurality of short-time spectrum expressions A feature extractor, wherein the at least one feature represents a spectral shape of one short-time spectral representation in one frequency band of the plurality of frequency bands; and the at least one feature in each frequency band. Supplying to the feature combiner, calculating the control information using intermediate coupling parameters, changing the intermediate coupling parameters; An optimized controller that compares the changed control information with the known control information and updates the intermediate coupling parameter when the changed intermediate coupling parameter results in control information that better matches the known control information And achieved by a device comprising:

  According to the sixth aspect, the object is a method for causing the feature combiner to learn in order to determine a combination parameter of the feature combiner, and the control information for each frequency band to the speech enhancement filter is known. For a learning audio signal, a step of obtaining a time sequence of a short-time spectrum expression of the learning audio signal, and a step of extracting at least one feature in each frequency band of a plurality of frequency bands for a plurality of short-time spectrum expressions. The at least one feature represents a spectral shape of one short-time spectral representation in one frequency band of the plurality of frequency bands, and provides the at least one feature of each frequency band to the feature combiner Computing the control information using intermediate coupling parameters; and the intermediate coupling A step of changing a parameter, a step of comparing the changed control information with the known control information, and a case where the changed intermediate coupling parameter results in control information that better matches the known control information, Updating the intermediate coupling parameters.

  According to a seventh aspect, the object is achieved by a computer program for performing any one of the methods of the present invention when executed on a computer.

  The present invention is based on the finding that per-band information about the spectral shape of an audio signal within a specific band is a very useful parameter for determining control information for a speech enhancement filter. Specifically, spectral shape information features determined for each band for multiple bands and multiple consecutive short-time spectral representations can be useful for characterizing audio signals for audio signal speech enhancement processing. provide. Specifically, there is a set of spectral shape features, where each spectral shape feature is in multiple spectral bands (such as the Bark band, or generally a band with a variable bandwidth in the frequency range). The feature set can be used to determine the signal / noise ratio for each band. For this purpose, the spectral shape features of multiple bands are processed by a feature combiner that combines these features using a combination parameter, and for some time portion of the audio signal control information is sent to the speech enhancement filter. can get. Preferably, the feature combiner comprises a neural network that is controlled by a number of coupling parameters, which are determined in a learning phase that is performed before actually performing the speech enhancement filtering. Specifically, the neural network executes a neural network regression method. A particular advantage is that the coupling parameters can be determined in a learning phase with audio material that may be different from the actual speech-enhanced audio material, so that the learning phase only needs to be performed once and this learning phase After that, the coupling parameter is fixedly set and can be applied to each unknown audio signal having a speech characteristic similar to that of the learning signal. Such speech characteristics may be characteristics of a single language, for example, or may be characteristics of a group of languages, such as European versus Asian languages.

  Preferably, the inventive concept is to estimate noise by learning feature of speech using feature extraction and neural network, and the feature extracted according to the present invention is an efficient and easy method It is a simple low-level spectral feature that can be extracted with, and what is important is that it can be extracted without incurring the large delays that inevitably arise from the system. Therefore, the inventive concept is particularly useful for providing accurate noise or SNR estimation even in situations where the noise is non-static and various noise signals are generated.

Preferred embodiments of the present invention will now be described in more detail with reference to the accompanying drawings.
FIG. 2 is a block diagram of a preferred apparatus or method of the present invention for processing an audio signal. FIG. 2 is a block diagram of an apparatus or method for learning a feature combiner according to a preferred embodiment of the present invention. 1 is a block diagram illustrating a speech enhancement apparatus and method according to a preferred embodiment of the present invention. An outline of the procedure for learning a feature combiner and applying a neural network regression method using optimized coupling parameters is shown. FIG. 6 is a plot showing gain factors as a function of SNR, where applied gain (solid line) is compared with spectral subtraction gain (dotted line) and Wiener filter (dashed line). Fig. 2 is a summary of the characteristics for each frequency band and preferred further characteristics for the total bandwidth. 6 is a flowchart illustrating a preferred embodiment of a feature extractor. FIG. 6 shows a flow chart for explaining a preferred embodiment of calculation of a gain coefficient for each frequency value and subsequent calculation of a speech enhanced audio signal part. An example of spectrum weighting is shown, showing an input time signal, an estimated subband SNR, an estimated SNR within the frequency bin after interpolation, a spectrum weight, and a time signal after processing. FIG. 2 is a schematic block diagram of a preferred embodiment of a feature combiner using a multilayer neural network.

  FIG. 1 shows a preferred apparatus for processing an audio signal 10 to obtain control information 11 for a speech enhancement filter 12. The speech enhancement filter filters the audio signal 10 using control information for each frequency band for each of a plurality of frequency bands, and includes many controllable filters such as a speech enhancement audio output signal 13. Can be realized by the method. As described below, the controllable filter can also be realized as a time / frequency converter, in which case individually calculated gain factors are applied to the spectral values or spectral bands, after which the frequency / time conversion is performed. Executed.

  The apparatus of FIG. 1 includes a feature extractor 14 that acquires a time sequence of a short time spectral representation of an audio signal and extracts at least one feature of each frequency band of a plurality of frequency bands for a plurality of short time spectral representations. The at least one feature represents a spectral shape of a short-time spectrum expression in each frequency band of a plurality of frequency bands. Further, the feature extractor 14 may be configured to extract other features apart from the spectral shape features. Several features of the audio short-time spectrum are output at the output of the feature extractor 14, and these several features are in each frequency band of multiple (at least 10, preferably more, 20-30, etc.) frequency bands. Including at least spectral shape features. These features can be used as is to obtain raw or averaged features for each band, or geometric or arithmetic mean or median processing or other statistical moment processing (dispersion, skewness, etc.) ), Etc., or any other process can be used. As a result, all of these raw and / or averaged features are input to the feature combiner 15. The feature combiner 15 combines a plurality of spectral shape features and preferably additional features using a coupling parameter that can be introduced via the coupling parameter input 16 but does not require the coupling parameter input 16. Thus, the coupling parameters may be incorporated into the feature combiner 15 or programmed. At the output of the feature combiner, control information for the speech enhancement filter is obtained for each time portion of the audio signal for each frequency band or each subband of a plurality of frequency bands or “subbands”.

  Preferably, the feature combiner 15 is realized as a neural network regression circuit. However, the feature combiner 15 applies an arbitrary combination operation to the feature output by the feature extractor 14 to obtain necessary control information (for each band). It can also be implemented as any other numerically or statistically controlled feature combiner that ultimately yields SNR values or gain factors per band. In a preferred embodiment of the neural network application, a learning phase (“learning phase training phase” means a phase in which learning using examples is performed) is required. In this learning phase, a device for learning the feature combiner 15 as shown in FIG. 2 is used. Specifically, FIG. 2 shows an apparatus that causes the feature combiner 15 to learn to determine the combination parameters of the feature combiner. For this purpose, the apparatus of FIG. 2 comprises a feature extractor 14 which is preferably configured similar to the feature extractor 14 of FIG. Further, the feature combiner 15 is configured in the same manner as the feature combiner 15 of FIG.

  In addition to FIG. 1, the apparatus of FIG. 2 includes an optimization controller 20 that receives control information 21 of a learning audio signal as an input. The learning phase is performed based on a known learning audio signal having a known speech / noise ratio in each band. The speech part and the noise part are for example brought separately from each other and the actual SNR for each band is measured on the fly, i.e. during the learning operation. Specifically, the optimization controller 20 controls the feature combiner so that the feature combiner 15 is supplied with the feature from the feature extractor 14. Based on these features and the intermediate coupling parameters resulting from the previous iteration, the feature combiner 15 calculates the control information 11. This control information 11 is sent to the optimization controller, and is compared with the control information 21 of the learning audio signal in the optimization controller 20. The intermediate coupling parameters are changed in response to an instruction from the optimization controller 20, and an additional set of control information is calculated by the feature combiner 15 using the changed coupling parameters. If this additional control information better matches the control information 21 of the learned audio signal, the optimization controller 20 updates the coupling parameters and uses these updated coupling parameters 16 as intermediate coupling parameters in the next run. To the feature combiner 15. Alternatively or in addition, the updated binding parameters can be stored in memory for further use.

FIG. 4 shows an overview of a spectrum weighting process using feature extraction in the neural network regression method. In the learning phase, the neural network parameter w is calculated using features from the reference subband SNR value R _t and the learning item x _t [k] (this is shown on the left side of FIG. 4). The noise estimation and speech enhancement filtering is shown on the right side of FIG.

  The concept proposed by the present invention follows a spectral weighting approach and uses a novel method to calculate spectral weights. Noise estimation is based on supervised learning and uses the feature set of the present invention. This feature aims to distinguish between tone signal components and noise signal components. Furthermore, the features proposed by the present invention take into account the evolution of signal characteristics on longer time scales.

  The noise estimation method proposed by the present invention makes it possible to cope with various fluctuating background sounds. A robust SNR estimate for fluctuating background noise is obtained by feature extraction and neural network regression as shown in FIG. Real-valued weights are calculated from SNR estimates for frequency bands that are approximately close to the Bark scale. The spectral resolution of SNR estimation is rather coarse to allow measurement of the spectral shape of the band.

The left side of FIG. 4 corresponds to a learning stage that basically needs to be executed only once. The procedure on the left side of FIG. 4 shown as the learning side 41 includes a reference SNR calculation block 21 that generates control information 21 of the learning audio signal input to the optimization controller 20 of FIG. The feature extraction device 14 in FIG. 4 located on the learning side corresponds to the feature extractor 14 in FIG. In particular, although FIG. 2 has been described as an apparatus that receives a learning audio signal, the learning audio signal is composed of a speech portion and a background portion. As you can perform useful references, background portion b _t and speech parts s _t is available separately from one another, are summed by adder 43 before input to the feature extractor 14. Therefore, the output of the adder 43 corresponds to the learning audio signal input to the feature extractor 14 in FIG.

  The neural network learning apparatus indicated by reference numerals 15 and 20 in FIG. 4 corresponds to the blocks 15 and 20 in FIG. 2, and is a set of coupling parameters that can be stored in the memory 40 by the same connection as in FIG. 2 or other similar connections. Bring w. Next, when the concept of the present invention is applied as shown by the application side 42 in FIG. 4, these combination parameters are used in the neural network regression device 15 corresponding to the feature combiner 15 in FIG. 4 corresponds to the controllable filter 12 of FIG. 1, and the feature extractor 14 on the right side of FIG. 4 corresponds to the feature extractor 14 of FIG.

  A concise implementation of the concept proposed by the present invention is described in detail below. The feature extractor 14 of FIG. 4 operates as follows.

  A set 21 of different features was investigated to identify the best feature set for subband SNR estimation, these features combined in various settings and objective measurements And evaluated by informal listening. The feature selection process results in a set of features including spectral energy, spectral flux, spectral flatness, spectral skewness, LPC, and RASTA-PLP coefficients. Spectral energy, flux, flatness, and skewness characteristics are calculated from spectral coefficients depending on the critical band scale.

  The above features will be described in detail with reference to FIG. Further features are a delta feature of spectral energy and a delta-delta feature of low pass filtered spectral energy and spectral flux.

  The structure of the neural network used in block 15, 20 or 15 of FIG. 4 or preferably used in the feature combiner 15 of FIG. 1 or 2 will be described with reference to FIG. In particular, the preferred neural network includes a layer 100 of input neurons. In general, n input neurons can be used, i.e. one neuron for each input feature. Preferably, the neural network has 220 input neurons corresponding to the number of features. The neural network further comprises a hidden layer 102 having p hidden layer neurons. In general, p is less than n, and in a preferred embodiment, the hidden layer has 50 neurons. The neural network includes an output layer 104 having q output neurons on the output side. In particular, the number of output neurons is equal to the number of frequency bands so that each output neuron provides control information for each frequency band, such as SNR (speech-to-noise ratio) information for each frequency band. For example, if there are 25 different frequency bands, preferably with increasing bandwidth from low to high frequencies, the number of output neurons q is equal to 25. In this way, the neural network is applied to subband SNR estimation from the calculated low level features. As described above, the neural network has 220 input neurons and one hidden layer 102 having 50 neurons. The number of output neurons is equal to the number of frequency bands. Preferably, the hidden neuron includes an activation function that is a hyperbolic tangent and the activation function of the output neuron is identity.

  In general, each neuron from layer 102 or 104 receives all corresponding inputs (for layer 102, the output of all input neurons). Next, each neuron of the layer 102 or 104 performs weighted addition with a weighting parameter corresponding to the connection parameter. The hidden layer can include a bias value in addition to the parameters. Therefore, the bias value also belongs to the coupling parameter. Specifically, each input is weighted by its corresponding coupling parameter, and the output of the weighting operation shown in FIG. 10 by the exemplary box 106 is input to an adder 108 in each neuron. The output of the adder or the input to the output neuron can comprise a non-linear function 110, and such a non-linear function can optionally be placed at the output and / or input of the neuron in the hidden layer, for example. .

  The neural network weights are learned with a mixture of clean speech signals and background noise, but their reference SNR is calculated using the separated signals. The learning process is shown on the left side of FIG. Speech and noise are mixed with an SNR of 3 dB per item and fed to the feature extractor. This SNR is a broadband SNR value that is constant over time. The data set includes 2304 combinations of 48 speech signals and 48 noise signals, each 2.5 seconds long. The speech signal is from different speakers in seven languages. The noise signal is a recording of traffic noise, crowd noise, and various natural environmental sounds.

  For a particular spectral weighting rule, two definitions of the output of the neural network are appropriate. That is, the neural network can be trained using a reference value for a time-varying subband SNR value R (ω), or it can be trained with a spectral weight G (ω) (derived from the SNR value). be able to. In informal listening, better objective results and better performance were obtained in the simulation with the subband SNR as the reference value compared to the neural network trained by spectral weighting. The neural network is trained using 100 iteration cycles. In this work, a learning algorithm based on scaled conjugate gradients is used.

  A preferred embodiment of the spectrum weighting operation 12 will now be described.

へと変換される。線形サブバンドＳＮＲが、推定誤差からもたらされる可能性があるアーチファクトを少なくするために、ＩＩＲ低域通過フィルタ処理を使用して時間及び周波数に沿って平滑化される。スペクトル重み付けのインパルス応答がＤＦＴフレームの長さを超える場合に生じる周回畳み込み（circular convolution）の影響を軽減するために、周波数に沿った低域通過フィルタ処理がさらに必要とされる。上記フィルタ処理が２回実行される一方で、第２のフィルタ処理は、結果としてのフィルタがゼロの位相を有するように、（最後のサンプルから出発して）逆順で行われる。 The estimated subband SNR estimate is linearly interpolated to the frequency resolution of the input spectrum and the linear ratio

Converted to. The linear subband SNR is smoothed along time and frequency using IIR low pass filtering to reduce artifacts that may result from estimation errors. In order to mitigate the effects of circular convolution that occurs when the spectral weighted impulse response exceeds the length of the DFT frame, further low pass filtering along the frequency is required. While the above filtering process is performed twice, the second filtering process is performed in reverse order (starting from the last sample) so that the resulting filter has a phase of zero.

  FIG. 5 shows the gain factor as a function of SNR. The applied gain (solid line) is compared to the spectral subtraction gain (dashed line) and the Wiener filter (dashed line).

Spectral weighting is calculated according to the modified spectral subtraction rule of equation (5) and limited to -18 dB.

  The parameters α = 3.5 and β = 1 are determined experimentally. This special attenuation above the SNR of 0 dB has been chosen to avoid distortion of the speech signal at the expense of residual noise. The attenuation curve as a function of SNR is shown in FIG.

  FIG. 9 shows examples of input and output signals, estimated subband SNR, and spectral weights.

  Specifically, FIG. 9 shows an example of spectral weighting, showing the input time signal, the estimated subband SNR, the estimated SNR of the frequency bin after interpolation, the spectral weight, and the processed time signal. Yes.

  FIG. 6 shows an overview of preferred features to be extracted by the feature extractor 14. The feature extractor represents a feature representing the spectral shape of the short-time spectral representation in each frequency band with each low resolution, ie, each of the 25 frequency bands that require SNR or gain values. It is desirable to extract. The spectral shape in a band represents the distribution of energy within that band and can be configured by several different calculation rules.

  A preferred spectral shape feature is spectral flatness (SFM), which is the geometric mean of spectral values divided by the arithmetic mean of spectral values. In the definition of geometric mean / arithmetic mean, power can be applied to each spectral value in the band prior to performing the n th order root or mean operation.

  In general, spectral flatness can also be calculated when the power to process each spectral value in the denominator of the formula for SFM is larger than the power used for the numerator. In that case, both the denominator and the numerator can contain arithmetic expressions. Typically, the wrinkle in the numerator is 2 and the wrinkle in the denominator is 1. In general, in order to obtain generalized spectral flatness, the power used for the numerator need only be larger than the power used for the denominator.

  The SFM for bands where energy is evenly distributed throughout the frequency band is less than 1 and approaches a small value close to 0 in a number of frequency lines, while the energy falls to just one spectral value in the band. In the case of concentration, it is clear from this calculation that, for example, the SFM value is equal to 1. That is, a high SFM value represents a band where energy is concentrated at a certain position in the band, while a low SFM value indicates that the energy is equally distributed in the band.

  Another spectral shape characteristic is spectral skewness, which is an index of asymmetry of the distribution centered on the center of gravity. There are other features regarding the spectral shape of the short-time frequency representation within a frequency band.

  While the spectral shape is calculated for one frequency band, there are other features calculated for one frequency band and shown in detail in FIG. There are also additional features that do not necessarily have to be calculated for each frequency band, but are calculated for the entire bandwidth.

Spectral energy Spectral energy is calculated for each time frame and each frequency band and normalized by the total energy of that frame. Further, the spectral energy is low pass filtered over time using a second order IIR filter.

Spectral Flux Spectral flux SF is defined as the dissimilarity between 20 consecutive frames of spectrum and is often performed by a distance function. In this work, the spectral flux uses the Euclidean distance according to Equation (6), the spectral coefficient X (m, k), the time frame index m, the subband index r, and the lower and upper frequency bands l _r and u _r. Calculated by

Spectral Flatness There are various definitions for the calculation of vector flatness or spectral tonality (with inverse correlation to spectral flatness). The spectral flatness SFM used here is calculated as the ratio of the geometric mean and arithmetic mean of the L spectral coefficients of the subband signal, as shown in equation (7).

Spectral skewness Distribution skewness is a measure of asymmetry centered on the center of gravity and is defined as the cubic central moment of a random variable divided by its standard deviation cube.

が最小になるように予測する全極型フィルタの係数である。

Linear prediction coefficient LPC calculates the actual value x (k) of the time series from the previous value by the square error

Is the coefficient of the all-pole filter that is predicted to be minimum.

  LPC is calculated by the autocorrelation method.

Mel frequency cepstrum coefficient The power spectrum is warped according to the Mel scale using a triangulation weighting function with unit weights for each frequency band. The MFCC is calculated by taking the logarithm and calculating the discrete cosine transform.

Relative spectrum perceptual linear prediction coefficient RASTA-PLP coefficient (Non-patent Document 17) is calculated from the power spectrum in the following steps.
1. 1. Compression of spectral coefficient magnitude 2. Bandpass filtering of subband energy over time 3. Expansion of the size related to the reverse processing of step 2. 4. Multiplication of weights corresponding to equal loudness curves 5. Simulation of perception of loudness by raising the coefficient to 0.33. Calculation of all-pole model of result spectrum by autocorrelation method

Perceptual Linear Prediction (PLP) Coefficient PLP values are calculated without applying steps 1 to 3 as in RASTA-PLP (Non-patent Document 18).

Delta features Delta features have been successfully applied to automatic speech recognition and audio content classification in the past. There are various methods for calculating delta features. Here, it is calculated by convolving the feature time series with a linear gradient having a length of 9 samples (the feature time series sampling rate is equal to the STFT frame rate). The delta-delta feature is obtained by performing a delta operation on the delta feature.

As described above, it is preferable to have band separation in the low resolution frequency band, similar to the human auditory system. Therefore, logarithmic band separation or bark-shaped band separation is preferred. This means that the band with a low center frequency is narrower than the band with a high center frequency. In the calculation of the spectral flatness, for example, usually from one of the lowest frequency is the value value l _r in the band, the calculation of the sum of up to the maximum count value u _r is the spectral values within that predetermined band Is executed. In order to obtain better spectral flatness, it is preferable to use some or all spectral values from at least the lower and / or upper adjacent frequency bands in the lower band. This means, for example, that the spectral flatness for the second band is calculated using the spectral values of the second band and the spectral values of the first band and / or the third band. In the preferred embodiment, not only the spectral values of either the first or the third band are used, but the spectral values of the first band and the third band are also used. This is because when calculating the SFM of the second band, formula q (7) is the highest spectral value of the third band from the equal l _r the first (lowest) spectral values of the first band Means up to u _r equal to. Until the value l _r and u _r reaches a predetermined band of sufficient number of spectral values within that low resolution frequency band itself, in the manner described above, based on the larger number of spectral values, the spectral shape The characteristics of can be calculated.

For the linear prediction coefficients extracted by the feature extractor, either the LPC coefficient a _j in equation (8) or the residual / error value remaining after optimization is used, or the LPC features extracted by the feature extractor It is preferable to use any combination of coefficients and error values, such as by multiplication or addition with a normalization coefficient, so that both the coefficient and the square error value affect.

  The advantage of the spectral shape feature is that it is a low-dimensional feature. For example, when considering a frequency bandwidth with 10 complex or real spectral values, it would not be beneficial to use all of these 10 complex or real spectral values, and waste computational resources. Conceivable. Therefore, spectral shape features having a lower dimension than the raw data dimension are extracted. For example, when energy is considered, the raw data has a dimension of 10 because there are 10 square spectral values. In order to extract spectral shape features that can be used efficiently, spectral shape features having a lower dimension (preferably 1 or 2) than that of the raw data are extracted. Similar dimension reduction for raw data can be achieved, for example, when a low-level polynomial fit is made to the spectral envelope of the frequency band. For example, if only two or three parameters are fitted, the spectral shape features will include these two or three parameters of a polynomial or any other parameterized system. In general, all parameters that represent the distribution of energy within a frequency band and that have a low dimension of less than 5% of the raw data dimension, or at least less than 50%, or just less than 30% are useful.

Although the use of spectral shape features alone has been found to provide advantageous behavior for an apparatus for processing audio signals, it is preferable to use at least additional per-band features. The characteristics of each band of the useful additional in providing improved results, calculated for each time frame and frequency band and normalized by the total energy of the frame, it is the spectral energy of each band is also Is shown . This feature may or may not be low pass filtered. In addition, the addition of spectral flux features advantageously improves the performance of the device of the present invention, where the spectral shape features of each band add to the spectral energy features of each band and the spectral flux features of each band. It has been found that an efficient procedure that yields good performance when obtained is obtained. In addition to the additional features described above, this also improves the performance of the device of the present invention.

  As described with respect to the spectral energy feature, application of low pass filtering of this feature over time or moving average normalization over time can be applied, but is not necessarily required. In the former case, for example, an average of five preceding spectral shape features for the corresponding band is calculated and the result of this calculation is used as the spectral shape feature for the current band of the current frame. However, this averaging can also be applied bi-directionally so that not only features from the past but also features from the “future” are used to calculate the current feature in the averaging operation.

  Next, FIGS. 7 and 8 will be described to present a preferred embodiment of the feature extractor 14 as shown in FIG. 1, FIG. 2 or FIG. In the first stage, as shown in step 70, a window is applied to the audio signal to provide a block of audio sampling values. Preferably, overlap is applied. This means that due to the overlap range, one and the same audio sample occurs in two consecutive frames, and a 50% overlap for the audio sample value is preferred. In step 71, a time / frequency conversion is performed on the block of audio sampling values to which the window has been applied to obtain a frequency representation at a first resolution, which is a high resolution. For this purpose, a short-time Fourier transform (STFT) realized with an efficient FFT is used. When step 71 is applied several times to temporally consecutive blocks of audio sample values, a spectrogram is obtained as is known in the art. In step 72, high resolution spectral information, i.e., high resolution spectral values, are grouped into low resolution frequency bands. For example, if an FFT with 1024 or 2048 input values is applied, there are 1024 or 2048 spectral values, but such high resolution is not required or intended. Instead, the grouping step 72 results in splitting the high resolution spectrum into a small number of bands, such as bands with varying bandwidth as known from the Berk band or logarithmic band splitting. Next, following the grouping step 72, a spectral shape feature and preferably another feature calculation step 73 is performed for each of the low resolution bands. Although not shown in FIG. 7, additional features regarding the entire frequency band can be calculated using the data obtained in step 70. This is because for these full bandwidth features, any spectral separation obtained by step 71 or step 72 is not required.

  Step 73 results in spectral shape features having a dimension of m that is less than n, preferably 1 or 2 per frequency band. This means that information for each frequency band existing after step 72 is compressed into low-dimensional information existing after step 73 by the operation of the feature extractor.

  As shown in FIG. 7, in the vicinity of step 71 and step 72, the time / frequency conversion and grouping steps can be replaced with another operation. The output of step 70 can be filtered, for example, with a low resolution filter bank so that 25 subband signals are obtained at the output. A high resolution analysis of each subband can then be performed to obtain raw data for spectral shape feature calculation. This processing can be done, for example, by FFT analysis of the subband signal, or by any other analysis of the subband signal, such as by a further cascade filter bank.

  FIG. 8 illustrates a preferred procedure for implementing the controllable filter 12 of FIG. 1 or the spectral weighting feature described in FIG. 3 or indicated by reference numeral 12 in FIG. As shown in step 80, following the step of determining control information for each low resolution band such as the subband SNR value output by the neural network regression block 15 of FIG. Linear interpolation is performed.

  For each spectral value obtained in step 30 of FIG. 3 or obtained by a short-time Fourier transform performed in step 71, or obtained by an alternative procedure shown to the right of steps 71 and 72, a weighting factor is obtained. The goal is to finally get. After step 81, the SNR value for each spectral value is obtained. However, since this SNR value is still in the log domain, step 82 results in a transformation from the log domain to the linear domain for each high resolution spectral value.

  In step 83, the linear SNR value (ie, high resolution) of each spectral value is applied, eg, any moving average operations using an IIR low pass filter or FIR low pass filter, etc. By doing so, it is smoothed in time and frequency. In step 84, a spectral weight for each high resolution frequency value is calculated based on the smoothed linear SNR value. This calculation is based on the function shown in FIG. 5, but the function shown in this figure is given in logarithmic terms, while in step 84 the spectral weight for each high resolution frequency value is Calculated in the linear domain.

  Next, in step 85, each spectral value is multiplied by the determined spectral weight to obtain a set of high resolution spectral values multiplied by the set of spectral weights. This processed spectrum is frequency-to-time converted in step 86. Depending on the application scenario and depending on the overlap used in step 80, between two blocks of time-domain audio sampling values obtained by two successive frequency-time conversion stages to deal with blocking artifacts. Crossfading operations can be performed.

  Additional windows can be applied to reduce the circular convolution artifact.

  The result of step 86 is a block of audio sample values that have improved speech performance, i.e., can better perceive speech compared to a corresponding audio input signal that has not been speech enhanced.

  Depending on the specific implementation requirements of the inventive methods, the inventive methods can be implemented in hardware or in software. Implementation is achieved using a digital storage medium, in particular a disk, DVD or CD, on which electronically readable control signals are stored that cooperate with a computer system programmable to carry out the method of the invention. It can be carried out. Accordingly, in general, the present invention is a computer program product having program code stored on a machine-readable carrier, and when the computer program product is executed on a computer, the program code is stored in the computer program product. Operate to carry out the inventive method. Thus, in other words, the method of the present invention is a computer program having program code for executing at least one of the methods of the present invention when executed on a computer.

  The above-described embodiments are merely illustrative of the principles of the present invention. It should be understood that variations and modifications to the arrangements and details described herein will be apparent to those skilled in the art. Accordingly, the present invention is limited only by the scope of the appended claims and is not limited by the specific details presented by the descriptions and descriptions of the embodiments herein.

Claims (15)

An apparatus for processing audio signals to obtain control information for a speech enhancement filter,
A feature extractor for acquiring a time sequence of a short-time spectrum representation of the audio signal and extracting at least one feature in each frequency band of a plurality of frequency bands for a plurality of short-time spectrum representations, A feature extractor wherein one feature represents a spectral shape of one short-term spectral representation in one of the plurality of frequency bands;
A feature combiner that combines the at least one feature of each frequency band using a coupling parameter across the plurality of frequency bands to obtain control information for the speech enhancement filter for one time portion of the audio signal;
An optimization controller that causes the feature combiner to learn to determine a combining parameter of the feature combiner;
The feature extractor acquires a time series of short-time spectral representations of the learning audio signal for learning audio signals whose control information for each frequency band to the speech enhancement filter is known, and for a plurality of short-time spectral representations Extracting at least one feature in each frequency band of the plurality of frequency bands, wherein the at least one feature represents a spectral shape of one short-time spectrum representation in one frequency band of the plurality of frequency bands,
The feature extractor supplies the at least one feature of each frequency band extracted from the learning audio signal to the feature combiner;
The feature combiner calculates the control information using intermediate combining parameters;
The optimization controller changes the intermediate coupling parameter, compares the control information after the change with the known control information, and obtains control information that the intermediate coupling parameter after the change better matches the known control information. If so, the apparatus updates the intermediate coupling parameters to determine the coupling parameters . The feature extractor extracts at least one additional feature that represents a characteristic of the short-term spectral representation, and the characteristic of the short-term spectral representation represented by the additional feature is a spectral shape of the short-term spectral representation and Are different characteristics,
The apparatus of claim 1, wherein the feature combiner combines the at least one additional feature and the at least one feature for each frequency band using the combination parameter.   2. The frequency extraction operation according to claim 1, wherein the feature extractor performs a frequency conversion operation that obtains a time sequence of a spectral representation having a frequency band with a non-uniform bandwidth that increases in bandwidth as the center frequency of the frequency band increases. Equipment. The feature extractor calculates, as a first feature, a spectral flatness for each band representing an energy distribution within the band, or as a second feature based on the total energy of the signal frame from which the spectral representation is derived. To calculate the normalized energy for each band,
The apparatus of claim 1, wherein the feature combiner uses spectral flatness per band or normalized energy per band.   The feature extractor further extracts, for each band, spectral flux representing similarity or dissimilarity between temporally continuous spectral representations, or spectral skewness representing asymmetry about the centroid; The apparatus according to claim 1.   The feature extractor further extracts an LPC feature including an LPC error signal, a linear prediction coefficient up to a predetermined order, or a combination of an LPC error signal and a linear prediction coefficient, or the feature extractor includes a PLP coefficient, RASTA- The apparatus of claim 1, further extracting PLP coefficients, mel frequency cepstrum coefficients, or delta features.   The feature extractor calculates a feature of the linear prediction coefficient for a block of time-domain audio samples, and the block is used to extract the at least one feature representing a spectral shape of each frequency band. The apparatus of claim 6 comprising audio samples.   When the feature extractor uses spectral information of one or two frequency bands that are directly adjacent to each other in order to calculate the shape of the spectrum of one frequency band, and when only the spectral information of the frequency band is used. The apparatus of claim 1 comprising:   The feature extractor extracts individual feature information for each feature for each block of audio samples and combines the sequence of individual feature information of one frequency band to obtain the at least one feature for the frequency band; The apparatus of claim 1.   The feature extractor calculates several spectral values for each frequency band and combines the several spectral values to obtain the at least one feature representing a spectral shape, wherein the at least one feature is the frequency The apparatus of claim 1, wherein the apparatus has a dimension that is less than the number of spectral values in a band. A method of processing an audio signal to obtain control information for a speech enhancement filter,
For a learning audio signal for which control information for each frequency band to the speech enhancement filter is known, obtaining a time sequence of a short-time spectral representation of the learning audio signal;
Extracting at least one feature in each frequency band of a plurality of frequency bands for a plurality of short-time spectral representations for the learning audio signal, wherein the at least one feature is one of the plurality of frequency bands; Representing a spectral shape of one short-term spectral representation in a frequency band; and
Calculating the control information from the at least one feature in each frequency band of the learning audio signal using intermediate coupling parameters;
Changing the intermediate coupling parameters;
Comparing the control information after the change with the known control information;
Updating the intermediate coupling parameter to determine a coupling parameter when the changed intermediate coupling parameter results in control information that better matches the known control information;
Obtaining a time sequence of a short-time spectral representation of the audio signal;
Extracting at least one feature in each frequency band of a plurality of frequency bands for a plurality of short-term spectral representations, wherein the at least one feature is one short in one frequency band of the plurality of frequency bands; Representing the spectral shape of the temporal spectral representation; and
The at least one feature in each frequency band is combined using the combination parameters over the plurality of frequency bands, the steps for one time portion of the audio signal to obtain control information for the speech enhancement filter,
Including methods. A device for speech enhancement in an audio signal,
The apparatus of claim 1, wherein the apparatus processes the audio signal to obtain filter control information for a plurality of bands representing a time portion of the audio signal;
A controllable filter that can be controlled based on the control information to variably attenuate one band of the audio signal compared to other bands;
With a device. The processing device comprises a time-frequency converter that provides spectral information having a higher resolution than the spectral resolution of the control information;
The control information is further interpolated to a high resolution, the control information after interpolation is smoothed, and further provided with a post-processing processor for control information to obtain post-processed control information,
The apparatus of claim 12, wherein controllable filter parameters of the controllable filter are set based on the post-processed control information. A method for speech enhancement of an audio signal,
The method of claim 11, wherein the audio signal is processed to obtain filter control information for a plurality of bands representing one time portion of the audio signal;
Controlling a filter based on the control information so that a certain band of the audio signal is variably attenuated compared to other bands;
Including methods. 15. A computer program for executing the method of claim 11 or 14 when executed on a computer.

Images