JP6567478B2 - Sound source enhancement learning device, sound source enhancement device, sound source enhancement learning method, program, signal processing learning device - Google Patents

Description

  The present invention relates to a sound source enhancement technique that emphasizes a sound source signal of a specific sound source from a sound source signal in which sound source signals of various sound sources are mixed, and particularly to a sound source enhancement technique using deep learning.

  Up to now, the network parameters (for example, weight matrix, bias vector) that make up deep learning are set to random initial values and optimized according to the error back-propagation method. We have built a neural network for sound source enhancement that outputs a group (for example, target sound / noise PSD (Power Spectral Density), Wiener filter, prior SNR (Signal-Noise Ratio)).

  There is Non-Patent Document 1 as a conventional sound source separation / emphasis method. A sound source signal to be emphasized (hereinafter referred to as a target sound) is represented as s (t), and noise other than the target sound is represented as n (t) (where t is a time index). s (t) and n (t) are time domain signals.

s (t), n (t) is a frequency domain target sound that is developed in the frequency domain, and frequency domain noise is represented as S (τ, ω), N (τ, ω) (where τ, ω are , Time frame index and frequency index respectively). At this time, the frequency domain mixed sound X (τ, ω) is expressed as follows. Also, let x (t) be the time domain representation of X (τ, ω).

ここで、Kは特性の異なる雑音の数、N_i(τ, ω)はi番目の雑音特性に属する雑音源の周波数領域雑音である。 Noise may be divided into a plurality according to characteristics. Here, the noise characteristics are, for example, the property of strong non-stationarity like music, the property of strong stationaryity like air-conditioning noise and the property of strong low-frequency components, using a microphone with 2ch or more In the case of observation, it is a property that the direction of arrival is different. If the noise characteristics are different, the noise can be modeled as follows.

Here, K is the number of noises having different characteristics, and N _i (τ, ω) is the frequency domain noise of the noise source belonging to the i-th noise characteristic.

  In the following, first, a conventional sound source enhancement learning apparatus 800 will be described with reference to FIGS. FIG. 12 is a block diagram illustrating a configuration of the sound source enhancement learning apparatus 800. FIG. 13 is a flowchart showing the operation of the sound source enhancement learning apparatus 800. As illustrated in FIG. 12, the sound source enhancement learning device 800 includes a feature quantity / label generation unit 810 and a pre-learning unit 820.

  The sound source enhancement learning device 800 is connected to the learning data recording unit 890. In the learning data recording unit 890, the monaural mixed sound X (τ, ω) used for pre-learning, the target sound S (τ, ω), and the noise N (τ, ω) constituting the same are recorded as learning data. . Hereinafter, X (τ, ω), S (τ, ω), and N (τ, ω) are referred to as frequency domain mixed sound learning data, frequency domain target sound learning data, and frequency domain noise learning data, respectively.

The feature quantity / label generation unit 810 generates a feature quantity from the frequency domain mixed sound learning data X (τ, ω), the frequency domain target sound learning data S (τ, ω), and the frequency domain noise learning data N (τ, ω). And a label are generated (S810). There are various methods for designing feature quantities, but the simplest example is to use a smoothed value of the mixed sound spectrum | X (τ, ω) | ² or spectrum | X (τ, ω) | ² it can. Although there are various label design methods, as the simplest example, a binary mask I (τ, ω) can be used. In the following description, the mixed sound spectrum | X (τ, ω) | ² is used as the feature quantity and the binary mask I (τ, ω) is used as the label. | X (τ, ω) | ² and I (τ, ω) are prepared for each frame time and for each frequency.

ここで、NL(τ, ω)は観測時点の雑音混在レベルであり、θはバイナリマスクの値（0または1）を決定する閾値である。 The binary mask I (τ, ω) is calculated by the following equation.

Here, NL (τ, ω) is a noise mixing level at the time of observation, and θ is a threshold value for determining a binary mask value (0 or 1).

  Therefore, the label (0 or 1) is given for each frequency or frequency band.

  The threshold value θ is often set to about θ = 0 dB. This corresponds to determining whether or not the target sound source is the most important sound source in the corresponding frequency-time frame. In addition, the floor value (that is, the value of I (τ, ω) when NL (τ, ω) is smaller than θ) is 0 in Equation (3), but in reality, 0 <α <1 In many cases, the value α to be satisfied is used. For example, α is set to a value of about 0.05 to 0.3.

NL (τ, ω) is calculated by the following equation.

  The prior learning unit 820 learns the network parameter p from the combination of the feature quantity and the label (S820). The network parameter p is a parameter used in the binary mask estimation unit 920 constituting the sound source enhancement apparatus 900 described below. For the learning framework, any supervised learning method such as DNN (Deep Neural Network), CNN (Convolutional Neural Network), or RNN (Recurrent Neural Network) may be used. That is, the network parameter p is a set of parameters optimized for each sound source by prior supervised learning. When DNN is used, the network parameter p is a weight matrix and a bias vector.

  Next, a conventional sound source enhancement apparatus 900 will be described with reference to FIGS. FIG. 14 is a block diagram illustrating a configuration of the sound source enhancement device 900. FIG. 15 is a flowchart showing the operation of the sound source enhancement apparatus 900. As illustrated in FIG. 14, the sound source enhancement device 900 includes a frequency domain conversion unit 910, a binary mask estimation unit 920, a signal enhancement unit 930, and a time domain conversion unit 940.

  The sound source emphasizing apparatus 900 is connected to the learning result recording unit 990. In the learning result recording unit 990, the network parameter p learned in advance by the sound source enhancement learning apparatus 800 is recorded.

  The frequency domain transform unit 910 performs frequency domain transform on the time domain observation signal x (t) to generate the frequency domain observation signal X (τ, ω) (S910). The observation signal x (t) may be a 1ch signal or a 2ch or more signal.

  The binary mask estimation unit 920 generates a binary mask I (τ, ω) from the frequency domain observation signal X (τ, ω) using the network parameter p read from the learning result recording unit 990 (S920). The binary mask estimation method is based on the premise that a method corresponding to the learning method used in the sound source enhancement learning apparatus 800 is used.

  In addition, when inputting signals of 2ch or more, a signal group (that is, a linearly converted signal group of X (τ, ω)) each emphasizing the target sound and noise by beam forming or the like is input, and a binary mask I ( τ, ω) may be output. In this case, it is necessary to execute the same processing for the generation of the feature amount in the feature amount / label generation unit 810 and the pre-learning of the network parameter in the pre-learning unit 820.

The signal enhancement unit 930 performs signal enhancement (sound source enhancement) from the frequency domain observation signal X (τ, ω) and the binary mask I (τ, ω) according to the equation (5), and the frequency domain enhanced sound ^ S (τ, ω) is generated (S930).

  The time domain transform unit 940 generates an enhanced sound ^ s (t) that is an estimated sound source signal in the time domain from the frequency domain enhanced sound ^ S (τ, ω) (S940).

Y. Wang, A. Narayanan and D.L.Wang, “On training targets for supervised speech separation”, IEEE / ACM Transactions on Audio, Speech, and Language Processing, vol.22, No.12, pp.1849-1858, 2014.

  In the method of the prior art, in the estimation of the binary mask used for sound source enhancement, a function (hereinafter referred to as a binary mask function) that outputs a binary mask using DNN, CNN, RNN, etc. (frequency domain) as a mixed sound input is designed. doing. The binary mask function is expressed by using a neural network, but the network parameters are learned without any physical restrictions on the structure of the neural network. For example, the number of network layers and the number of nodes are determined heuristically, and initial values of network parameters (such as a weight matrix and a bias vector) are also randomly determined. In other words, it can be said that the binary mask function for estimating the binary mask from the mixed sound is black boxed. Therefore, it is difficult to determine whether or not a neural network suitable for estimating a latent variable group (for example, target sound / noise PSD) for highly accurate and stable sound source enhancement can be constructed. there were. Actually, the PSD of the target sound / noise could not be estimated with high accuracy at the calculation stage in the middle of the neural network.

  In view of the above, an object of the present invention is to provide a sound source enhancement learning technique for learning network parameters for highly accurate and stable sound source enhancement by incorporating a model for sound source enhancement into the structure of a neural network. . It is another object of the present invention to provide a signal processing learning technique for learning network parameters for signal processing with high accuracy and stability by incorporating the structure of a signal processing algorithm into the structure of a neural network.

  According to one aspect of the present invention, a frequency domain mixed sound learning data is labeled from a set of frequency domain mixed sound learning data, frequency domain target sound learning data, and frequency domain noise learning data, and a set of target sound PSD and noise PSD is labeled. Using the feature quantity and the label, and learning the network parameters that characterize the neural network for sound enhancement using the observation signal as input and the estimated target sound PSD and the estimated noise PSD as outputs A sound source enhancement learning device having a pre-learning unit, wherein the pre-learning unit is a target sound BF output power that is a power of a beamforming output signal that emphasizes a target sound generated from the frequency domain mixed sound learning data Target sound non-negative auto encoder calculation unit that calculates target sound spectrum characteristic vector modeling target sound spectrum characteristics Noise non-negative auto encoder calculation that calculates noise spectrum characteristic vector modeling noise spectrum characteristic from noise BF output power which is power of beamforming output signal emphasizing noise generated from frequency domain mixed sound learning data A complementary subtraction neural network calculation unit for calculating the estimated target sound PSD and the estimated noise PSD from the target sound spectrum characteristic vector and the noise spectrum characteristic vector, the estimated target sound PSD and the estimated noise PSD, A network parameter optimizing unit that optimizes the network parameter using the target sound PSD and the noise PSD;

  One aspect of the present invention is characterized by a signal processing neural network that receives a signal as an input and outputs an estimation processing result of the signal using learning data including a set of a feature amount related to the signal and a label that is a processing result of the signal. A signal processing learning device for learning a network parameter, wherein an estimation processing result calculated in an intermediate stage from a feature amount calculated from the signal to an output of the estimation processing result is an intermediate stage estimation processing result i (1 ≦ i ≦ n, where n is an integer equal to or greater than 1, and using the feature amount, the neural network 1 calculation unit that calculates the intermediate stage estimation process result 1 and the feature quantity or the intermediate stage estimation process result 1 to 1 A neural network j calculation unit for calculating the intermediate stage estimation processing result j using any of the intermediate stage estimation processing result j-1 (2 ≦ j ≦ n), the feature Using the amount or the intermediate stage estimation process result 1 to the intermediate stage estimation process result n, the neural network n + 1 calculation unit for calculating the estimation process result, the estimation process result and the label A network parameter optimization unit for optimizing the network parameter.

  According to the present invention, a network for enhancing sound sources with high accuracy and stability by incorporating a model for sound source enhancement as a neural network structure and estimating a latent variable group such as PSD of target sound / noise. It becomes possible to learn parameters. In addition, learning the network parameters for highly accurate and stable signal processing by incorporating the structure of the signal processing algorithm as the structure of the neural network and estimating the intermediate stage estimation processing results corresponding to the latent variables. Is possible.

The figure which shows the structure of the neural network 500 for sound source emphasis. FIG. 2 is a block diagram showing a configuration of a sound source enhancement learning device 100. 5 is a flowchart showing the operation of the sound source enhancement learning apparatus 100. The block diagram which shows the structure of the prior learning part 120. FIG. The flowchart which shows operation | movement of the prior learning part 120. FIG. FIG. 3 is a block diagram showing a configuration of a network parameter calculation unit 122. 6 is a flowchart showing the operation of network parameter calculation 122; FIG. 2 is a block diagram showing a configuration of a sound source enhancement apparatus 200. 5 is a flowchart showing the operation of the sound source emphasizing apparatus 200. The block diagram which shows the structure of the Wiener filter estimation part 220. FIG. The flowchart which shows operation | movement of the Wiener filter estimation part 220. FIG. The block diagram which shows the structure of the sound source emphasis learning apparatus. 5 is a flowchart showing the operation of the sound source enhancement learning apparatus 800. FIG. 3 is a block diagram showing a configuration of a sound source enhancement apparatus 900. 5 is a flowchart showing the operation of the sound source enhancement apparatus 900.

  Hereinafter, embodiments of the present invention will be described in detail. In addition, the same number is attached | subjected to the component which has the same function including the sound source emphasis learning apparatus 800 and the sound source emphasis apparatus 900, and duplication description is abbreviate | omitted.

_ (Underscore) represents a subscript. For example, ^xy_z represents that _yz is a superscript to x, and _{xy_z} represents that _yz is a subscript to x.

First, the principle of the invention will be described.
<Principle of the invention>
In the conventional method, the number of network layers, the number of nodes, and the connection between nodes are determined heuristically. In other words, the neural network is designed (learned) with the combination of the feature value and the label as input without providing any particular restrictions on the physical structure of the neural network.

  In this application, a neural network that reflects a physical model for sound source enhancement is designed. The structure of the neural network to be designed is shown in FIG. As shown in FIG. 1, the sound source enhancement neural network 500 includes a target sound non-negative auto encoder 510, a noise non-negative auto encoder 515, and a complementary subtraction neural network 520. The target sound non-negative auto encoder 510 and the noise non-negative auto encoder 515 are neural networks that independently model the target sound spectral characteristics and noise spectral characteristics. The complementary subtraction neural network 520 is a neural network that estimates the target sound PSD and noise PSD.

  When there are a plurality of target sounds having different characteristics and noises having different characteristics, the neural network for sound source enhancement 500 includes a plurality of target sound non-negative auto encoders 510 and noise non-negative auto encoders 515 corresponding to the respective target sounds / noises. It may be included. However, for example, instead of considering a plurality of noise models such as a model of air conditioning noise and a model that uses TV sound as noise, one noise non-negative auto encoder 515 is designed using the added value of these noises. But you can. In the following, for the sake of simplicity, it is assumed that there is one target sound non-negative auto encoder 510 and one noise non-negative auto encoder 515.

Sound enhancement training the neural network 500, the power phi _{Y_S} beamformed output signal Y _S emphasizing the target sound obtained by applying beamforming frequency domain mixed sound X (τ, ω) (τ , ω) ( Hereinafter, φ _{Y_S} (τ, ω) is referred to as target sound BF output power) and beam forming output signal Y _N emphasizing noise obtained by applying beam forming to frequency domain mixed sound X (τ, ω). Power φ _{Y_N} (τ, ω) (hereinafter referred to as φ _{Y_N} (τ, ω) is referred to as noise BF output power) and the estimated target sound PSD q _S ⁽⁴⁾ and estimated noise PSD q _N ⁽⁴⁾ are combined. Configure to output estimated PSD q ⁽⁴⁾ . Here, Ω is the number of frequency bands, and φ _{Y_S} (τ, ω), φ _{Y_N} (τ, ω), q _S ⁽⁴⁾ , and q _N ⁽⁴⁾ are Ω-dimensional real value vectors. That is, the target sound BF output power φ _{Y_S} (τ, ω) and the noise BF output power φ _{Y_N} (τ, ω) are calculated for each frame and for each frequency band.

As learning data used for learning of the neural network 500 for sound source enhancement, the frequency domain mixed sound X (τ, ω), the frequency domain target sound S (τ, ω) constituting it, and the frequency domain noise N (τ, ω) Is given. _Target sound BF output power φ _{Y_S} (τ, ω) from frequency domain mixed sound learning data X (τ, ω), frequency domain target sound learning data S (τ, ω), frequency domain noise learning data N (τ, ω) The noise BF output power φ _{Y_N} (τ, ω) is generated before learning. Also, the target sound PSDφ _S and noise PSDφ _N used to control learning, that is, to optimize the network parameter p are also frequency domain target sound learning data S (τ, ω) and frequency domain noise learning data N (τ, generated from ω).

  Hereinafter, the target sound non-negative auto encoder 510, the noise non-negative auto encoder 515, and the complementary subtraction neural network 520 will be described.

[Modeling of target sound spectrum characteristics and noise spectrum characteristics by non-negative auto encoder]
First, the target sound non-negative auto encoder 510 and the noise non-negative auto encoder 515, which are neural networks for modeling the target sound spectral characteristics and noise spectral characteristics, will be described.

ここで、Ωは周波数バンド数、Υは解析する時間フレーム数、βはスペクトル基底数を表す。なお、Rは実数の集合を表す。 There is a non-negative matrix factorization (Non-Negative Matrix Factorization) as a method of using the spectral characteristics of a sound source for sound source separation. In non-negative matrix decomposition, a non-negative spectrogram S is expressed as a product of a non-negative spectral basis matrix B and a non-negative activation matrix A as in the following equation.

Here, Ω represents the number of frequency bands, Υ represents the number of time frames to be analyzed, and β represents the number of spectrum bases. R represents a set of real numbers.

As a method of non-negative matrix decomposition of S, for example, there is a method using a least square error, and when the least square error is used, the activation matrix A and the spectrum basis matrix B can be optimized (reference non-patent document 1). ).
[Reference Non-Patent Document 1] DD Lee et al, “Algorithms for non-negative matrix factorization”, Proc. NIPS 2000, pp.556-562, 2000.

  The idea of decomposing a non-negative power spectrum into a product of a spectrum base group and an activation group, similar to non-negative matrix decomposition, can be expressed using a neural network. A method for modeling the spectral characteristics of the target sound using the fully coupled DNN, that is, the structure of the target sound non-negative auto encoder 510 will be described.

First, the target sound BF output power φ _{Y_S} (τ, ω) is input to the input layer (first layer) of the target sound non-negative auto encoder 510. In other words, if the input to the first layer is q _S ⁽¹⁾ ,

ここで、W_S ⁽²⁾ _i,jは、W_S ⁽²⁾の(i、j)番目の要素を表す。 Next, the weight matrix W _S ⁽²⁾ , which is a β × Ω real-valued matrix, is regarded as an aggregate of β spectral bases, and a non-negative matrix constraint is imposed on W _S ⁽²⁾ . That means

Here, W _S ⁽²⁾ _{i, j} represents the (i, j) -th element of W _S ⁽²⁾ .

ただし、b_S ⁽²⁾はβ次元実数値ベクトルで表されるバイアスベクトル、f⁽²⁾ (・)は次式で表されるランプ関数(ReLU)である。

ここでは、活性化関数（伝達関数）の例としてランプ関数を用いるとしたが、これに限られるものではない。従来から用いられているシグモイド関数など、様々な非線形関数を用いることができる。 A β × 1 real-valued matrix (β-dimensional real-valued vector) q _S ⁽²⁾ that is the output of the first layer, that is, the input of the second layer, is obtained by the following calculation.

However, b _S ⁽²⁾ is a bias vector represented by a β-dimensional real value vector, and f ⁽²⁾ (·) is a ramp function (ReLU) represented by the following equation.

Here, the ramp function is used as an example of the activation function (transfer function), but the present invention is not limited to this. Various nonlinear functions such as a sigmoid function used conventionally can be used.

When calculated in this way, q _S ⁽²⁾ can be regarded as an activation for each of β spectral bases for one frame.

ここで、Tは転置を表す。

ただし、b_S ⁽³⁾はΩ次元実数値ベクトルで表されるバイアスベクトル、f⁽³⁾ (・)は式(15)で表されるランプ関数(ReLU)である。 Next, the weight matrix W _S ⁽³⁾ is used as a transposed matrix of W _S ⁽²⁾ (formula (12)), and the spectrogram is restored by formulas (13) to (15).

Here, T represents transposition.

However, b _S ⁽³⁾ is a bias vector represented by an Ω-dimensional real value vector, and f ⁽³⁾ (·) is a ramp function (ReLU) represented by Expression (15).

The target sound non-negative auto encoder 510 outputs an Ω × 1 real value matrix (Ω-dimensional real value vector) q _S ⁽³⁾ (hereinafter, q _S ^{(3) is referred} to as a target sound spectrum characteristic vector). Configuring the target sound non-negative auto encoder 510 using the non-negative weight matrices W _S ⁽²⁾ and W _S ⁽³⁾ corresponds to modeling the spectral characteristics of the target sound.

Incidentally, a neural network having the structure of Expression (12) is generally called an auto encoder (Reference Non-Patent Document 2).
[Non-Patent Document 2] Takayuki Okaya, “Deep Learning (Machine Learning Professional Series)”, Kodansha, 2015.

Similarly, with _respect to the noise BF output power φ _{Y_N} (τ, ω), it is possible to model the spectral characteristics of noise by configuring a non-negative auto encoder that is a neural network that performs the same calculation. The output of the noise non-negative auto encoder 515 is assumed to be an Ω × 1 real value matrix (Ω-dimensional vector) q _N ⁽³⁾ (hereinafter, q _N ^{(3) is referred} to as a noise spectrum characteristic vector).

[Estimation of target sound / noise PSD by complementary subtraction]
Next, from the output q _S ^{(3) of} the target sound non-negative auto encoder 510 that models the spectral characteristic of the target sound and the output q _N ^{(3) of} the noise non-negative auto encoder 515 that models the noise spectral characteristic, the Wiener filter The complementary subtraction neural network 520, which is a neural network for estimating the target sound PSD and noise PSD used for generation, will be described.

The output of the second layer, that is, the target sound spectrum characteristic vector q _S ⁽³⁾ and the noise spectrum characteristic vector q _N ⁽³⁾ , which are the inputs of the third layer, are changed to q ⁽³⁾ = [q _S ^{(3) T} , q _N ^{(3) T} ] Combined as ^T , and at the output of the third layer, the estimated target sound PSD q _S ⁽⁴⁾ is generated by subtracting the noise component remaining in the target sound spectrum characteristic vector q _S ⁽³⁾ Then, the estimated noise PSD q _N ⁽⁴⁾ is generated by subtracting the target sound component remaining in the noise spectrum characteristic vector q _N ⁽³⁾ . In other words, by constructing the weight matrix W ^(4), which is a 2Ω × 2Ω real-valued matrix, to perform complementary subtraction, the estimated target sound PSD q _S ⁽⁴⁾ and the estimated noise PSD q _N ⁽⁴⁾ can be accurately obtained. Think about estimating.

When receiving sound using a symmetric microphone array or symmetric beam forming, the average sensitivity D _ω of beam forming is determined independently of the direction of arrival of the target sound. Therefore, since the structure of W ⁽⁴⁾ is determined without depending on the direction of arrival of the target sound, it is considered that the optimization calculation is easy to converge.

ただし、λ_{S,ω_i}、λ_{N,ω_i}は正の値、γ_{S-N,ω_i}、γ_{N-S,ω_i}は負の値（1≦i≦Ω）とする。 As an example, to design W ⁽⁴⁾ Initial value W _init ⁽⁴⁾ of the following equation.

However, λ _{S, ω_i} , λ _{N, ω_i} are positive values, and γ _{SN, ω_i} , γ _{NS, ω_i} are negative values (1 ≦ i ≦ Ω).

At this time, no restriction is imposed on the range of values of each element of W ⁽⁴⁾ .

The output q ⁽⁴⁾ of the third layer is calculated by the following equation.

The output of layer 3 q ⁽⁴⁾ is q ⁽⁴⁾ = [q _S ^{(4) T} , q _N ^{(4) T} ] ^T (where q _S ⁽⁴⁾ and q _N ⁽⁴⁾ (Numerical vector), and q _S ⁽⁴⁾ and q _N ⁽⁴⁾ are the estimated target sound PSD and the estimated noise PSD, respectively.
[Optimization of network parameters of neural network for sound source enhancement]
Finally, the entire three layers are optimized by the error back propagation method using the target sound PSDφ _S and the noise PSDφ _N , and the network parameter p is optimized. Optimization is performed by the number of learning data.

Here, the network parameter p is a weight matrix W _S ⁽²⁾ , W _N ⁽²⁾ , W _S ⁽³⁾ , W _N ⁽³⁾ , W ^{(4) for three} layers and a bias vector b for two layers. _S ⁽²⁾ , b _N ⁽²⁾ , b _S ⁽³⁾ , b _N ⁽³⁾ . W _S ⁽²⁾ and W _N ⁽²⁾ are β × Ω real-valued matrices, W _S ⁽³⁾ and W _N ⁽³⁾ are Ω × β real-valued matrices, and W ⁽⁴⁾ is 2Ω × 2Ω real Numerical matrices, b _S ⁽²⁾ and b _N ⁽²⁾ are β-dimensional real-valued vectors, and b _S ⁽³⁾ and b _N ⁽³⁾ are Ω-dimensional real-valued vectors. W _S ⁽²⁾ and W _S ⁽³⁾ satisfy Equation (12), W _N ⁽²⁾ and W _N ⁽³⁾ satisfy the following relationship, and W ⁽⁴⁾ has an initial value of Equation (16 ).

In addition, after optimizing the target sound non-negative auto encoder 510 and the noise non-negative auto encoder 515 in advance, the third layer parameter W ⁽⁴⁾ (that is, the complementary subtraction neural network 520) is optimized by the error back propagation method. You may keep it.

<Embodiment 1>
Hereinafter, the sound source enhancement learning apparatus 100 according to the first embodiment will be described with reference to FIGS. FIG. 2 is a block diagram illustrating a configuration of the sound source enhancement learning apparatus 100. FIG. 3 is a flowchart showing the operation of the sound source enhancement learning apparatus 100. As illustrated in FIG. 2, the sound source enhancement learning device 100 includes a feature quantity / label generation unit 110 and a pre-learning unit 120.

  The sound source enhancement learning device 100 is connected to the learning data recording unit 890 as with the sound source enhancement learning device 800. The learning data recording unit 890 records frequency domain mixed sound learning data X (τ, ω), frequency domain target sound learning data S (τ, ω), and frequency domain noise learning data N (τ, ω). .

The feature quantity / label generation unit 110 obtains a feature quantity from the frequency domain mixed sound learning data X (τ, ω), the frequency domain target sound learning data S (τ, ω), and the frequency domain noise learning data N (τ, ω). And a label are generated (S110). Here, frequency domain mixed sound learning data X (τ, ω) is used as a feature quantity. Further, the target sound PSDφ _S and noise PSDφ _N calculated from the frequency domain target sound learning data S (τ, ω) and the frequency domain noise learning data N (τ, ω) are used as labels.

The pre-learning unit 120 learns the network parameter p from the set of the feature amount X (τ, ω) and the labels φ _S and φ _N (S120). Hereinafter, the pre-learning unit 120 will be described with reference to FIGS. FIG. 4 is a block diagram illustrating a configuration of the pre-learning unit 120. FIG. 5 is a flowchart showing the operation of the pre-learning unit 120. As shown in FIG. 4, the pre-learning unit 120 includes a beamforming output power calculation unit 121 and a network parameter calculation unit 122.

First, the beamforming output power calculation unit 121 calculates the target sound BF output power φ _{Y_S} (τ, ω) and the noise BF output power φ _{Y_N} (τ, ω) from the feature amount X (τ, ω) (S121). . As described above, the target sound BF output power φ _{Y_S} (τ, ω) is applied to the frequency domain mixed sound learning data X (τ, ω), which is the feature quantity, and the beam forming output signal that emphasizes the target sound. After generating Y _S , its power φ _{Y_S} (τ, ω) may be calculated. The noise BF output power φ _{Y_N} (τ, ω) can be obtained in the same manner.

Next, the network parameter calculation unit 122 labels the target sound PSDφ _S and the noise PSDφ _N that are labeled as the target sound BF output power φ _{Y_S} (τ, ω) and the noise BF output power φ _{Y_N} (τ, ω) calculated in S121. The network parameter p is calculated from (S122). Hereinafter, the network parameter calculation unit 122 will be described with reference to FIGS. FIG. 6 is a block diagram illustrating a configuration of the network parameter calculation unit 122. FIG. 7 is a flowchart showing the operation of the network parameter calculation unit 122. As shown in FIG. 6, the network parameter calculation unit 122 includes a target sound non-negative auto encoder calculation unit 126, a noise non-negative auto encoder calculation unit 127, a complementary subtraction neural network calculation unit 128, and a network parameter optimization unit 129.

First, the target sound non-negative auto encoder calculation unit 126 corresponds to the target sound non-negative auto encoder 510 of the neural network for sound source enhancement, and the target sound spectrum characteristic vector q from the target sound BF output power φ _{Y_S} (τ, ω). _S ⁽³⁾ is calculated (S126). Similarly, the noise non-negative auto encoder calculation unit 127 corresponds to the noise non-negative auto encoder 515 of the neural network for sound source enhancement, and the noise spectrum characteristic vector q _N ⁽³ from the noise BF output power φ _{Y_N} (τ, ω). ⁾ Is calculated (S127).

Next, the complementary subtraction neural network calculation unit 128 corresponds to the complementary subtraction neural network 520 of the sound source enhancement neural network, and the target sound spectral characteristic vector q _S ⁽³⁾ calculated in S126 and the noise calculated in S127. The estimated target sound PSD q _S ⁽⁴⁾ and the estimated noise PSD q _N ⁽⁴⁾ are calculated from the spectrum characteristic vector q _N ⁽³⁾ (S128).

Finally, the network parameter optimization unit 129 uses the estimated target sound PSD q _S ⁽⁴⁾ and estimated noise PSD q _N ⁽⁴⁾ calculated in S128, and the target sound PSDφ _S and noise PSDφ _N calculated in S110, The network parameter p is optimized (S129). As described in <Principle of the Invention>, the network parameter p is a weight matrix W _S ⁽²⁾ , W _N ⁽²⁾ , W _S ⁽³⁾ , W _N ⁽³⁾ , W ^{(4) for} three layers. And two layers of biases b _S ⁽²⁾ , b _N ⁽²⁾ , b _S ⁽³⁾ , b _N ⁽³⁾ .

  The processing of S121 to S129 is repeated for the number of learning data.

  Note that there may be a plurality of target sound non-negative auto encoder calculation units 126 and noise non-negative auto encoder calculation units 127 depending on, for example, the number of target sound sources and the number of noise sound sources having different characteristics.

  According to the invention of the present embodiment, a neural network in which physical constraints are introduced, specifically, a target sound spectrum characteristic, a non-negative auto encoder unit that independently models a noise spectrum characteristic, an estimated target sound PSD, and Sound source enhancement neural network composed of complementary subtraction neural network to generate estimated noise PSD, (large amount) speech data, that is, frequency domain mixed sound learning data, frequency domain target sound learning data, frequency domain noise learning data It is possible to estimate the target sound PSD and the noise PSD with high accuracy and stability by learning using the set. In addition, by using the estimated target sound PSD and noise PSD, it is possible to generate a Wiener filter with high accuracy and stability.

<Embodiment 2>
In the first embodiment, a neural network for sound source enhancement is obtained from a set of frequency domain mixed sound learning data X (τ, ω), frequency domain target sound learning data S (τ, ω), and frequency domain noise learning data N (τ, ω). The method of learning the network parameter p of was described. Here, a method for generating an emphasized sound from an observation signal collected by a microphone using the network parameter p learned in the first embodiment will be described. As a result, it is possible to output an enhanced sound in which the target sound in the observation signal is sound source enhanced.

  Note that the value of the network parameter p used here is the value of p at the end of learning by the sound source enhancement learning apparatus 100 of the first embodiment.

  Hereinafter, the sound source emphasizing apparatus 200 will be described with reference to FIGS. FIG. 8 is a block diagram illustrating a configuration of the sound source enhancement device 200. FIG. 9 is a flowchart showing the operation of the sound source emphasizing apparatus 200. As illustrated in FIG. 8, the sound source enhancement device 200 includes a frequency domain conversion unit 910, a Wiener filter estimation unit 220, a signal enhancement unit 230, and a time domain conversion unit 940.

  The sound source emphasizing apparatus 200 is connected to the learning result recording unit 990 as with the sound source emphasizing apparatus 900. The learning result recording unit 990 records the value of the network parameter p at the end of the above learning.

  The frequency domain transform unit 910 performs frequency domain transform on the observation signal x (t), which is a time domain mixed sound, and generates a frequency domain observation signal X (τ, ω) (S910).

  The Wiener filter estimation unit 220 generates the Wiener filter G (τ, ω) from the frequency domain observation signal X (τ, ω) using the network parameter p read from the learning result recording unit 990 (S220). Hereinafter, the Wiener filter estimation unit 220 will be described with reference to FIGS. FIG. 10 is a block diagram illustrating a configuration of the Wiener filter estimation unit 220. FIG. 11 is a flowchart showing the operation of the Wiener filter estimation unit 220. As shown in FIG. 10, the Wiener filter estimation unit 220 includes a beamforming output power calculation unit 121, a target sound non-negative auto encoder unit 221, a noise non-negative auto encoder unit 222, a complementary subtraction unit 223, and a Wiener filter calculation unit 224. including.

First, the beamforming output power calculation unit 121 calculates the target sound BF output power φ _{Y_S} (τ, ω) and the noise BF output power φ _{Y_N} (τ, ω) from the frequency domain observation signal X (τ, ω) ( S121).

Next, the target sound non-negative auto encoder unit 221 _converts the target sound BF output power φ _{Y_S} (τ, ω) calculated in S121 from the input q _S ⁽¹⁾ of the first layer to the target sound spectrum characteristic vector q _S ^(3). Is calculated (S221). The target sound non-negative auto encoder unit 221 is set with network parameters p (weight matrices W _S ⁽²⁾ , W _S ⁽³⁾ and bias vectors b _S ⁽²⁾ , b _S ⁽³⁾⁾ ) at the end of learning. This is different from the target sound non-negative auto encoder calculation unit 126 only in that point.

Similarly, the noise non-negative auto encoder unit 222 calculates the noise spectrum characteristic vector q _N ⁽³⁾ from the input q _N ⁽¹⁾ of the first layer with the noise BF output power φ _{Y_N} (τ, ω) calculated in S121. (S222). The noise non-negative auto encoder unit 222 is set with network parameters p (weight matrices W _N ⁽²⁾ , W _N ⁽³⁾ and bias vectors b _N ⁽²⁾ , b _N ⁽³⁾ ) at the end of learning. Is different from the noise non-negative auto encoder calculation unit 127 only in FIG.

Next, the complementary subtraction unit 223 calculates the estimated target sound PSD q _S ⁽⁴⁾ and the estimated noise PSD q from the target sound spectrum characteristic vector q _S ⁽³⁾ and the noise spectrum characteristic vector q _N ⁽³⁾ calculated in S221 and S222. _N ⁽⁴⁾ is calculated (S223). The complementary subtraction unit 223 differs from the complementary subtraction neural network calculation unit 128 only in that the network parameter p (weight matrix W ⁽⁴⁾ ) at the end of learning is set.

Finally, the Wiener filter calculation unit 224 calculates the Wiener filter G (τ, ω) from the estimated target sound PSD q _S ⁽⁴⁾ and the estimated noise PSD q _N ⁽⁴⁾ calculated in S223 (S224).

The signal enhancement unit 230 executes signal enhancement (sound source enhancement) from the frequency domain observation signal X (τ, ω) and the Wiener filter G (τ, ω) estimated in S220 according to the following equation, and the frequency domain enhancement sound ^ S ( (τ, ω) is generated (S230).

  The time domain transform unit 940 generates an enhanced sound ^ s (t) that is an estimated sound source signal in the time domain from the frequency domain enhanced sound ^ S (τ, ω) (S940).

  Note that there may be a plurality of target sound non-negative auto encoder units 221 and noise non-negative auto encoder units 222 similar to the target sound non-negative auto encoder calculation unit 126 and noise non-negative auto encoder calculation unit 127, and in this case, the same number is included. .

  According to the invention of the present embodiment, it is possible to output an enhanced sound obtained by emphasizing a target sound in an observation signal with high accuracy and stably.

<Modification>
In the first embodiment, in order to perform sound source enhancement processing on a mixed sound, a neural network having a physical structure that reflects a sound source enhancement processing algorithm is learned. Here, instead of the Wiener filter that is finally required for sound source enhancement, a neural network that outputs the target sound spectrum characteristic vector, which is a latent variable, a neural network that outputs the noise spectrum characteristic vector, the estimated target sound PSD / estimated noise By learning each neural network that outputs PSD, the features of the algorithm for sound source enhancement processing were incorporated as the physical structure of the neural network, and the neural network that can perform sound source enhancement processing with high accuracy and stability was learned.

  Similarly, in addition to processing other than sound source enhancement for acoustic signals and processing for image signals, each processing algorithm is also in the middle of processing for all signals in which radio waves and light waves are converted into electrical signals It is possible to construct a neural network that outputs the vectors (intermediate stage estimation processing results) generated in step (b) and obtains final processing results (estimation processing results) using these neural networks as parts. In this way, a neural network corresponding to each processing algorithm can be configured, and the neural network can execute each processing with high accuracy and stability.

<Supplementary note>
The apparatus of the present invention includes, for example, a single hardware entity as an input unit to which a keyboard or the like can be connected, an output unit to which a liquid crystal display or the like can be connected, and a communication device (for example, a communication cable) capable of communicating outside the hardware entity Can be connected to a communication unit, a CPU (Central Processing Unit, may include a cache memory or a register), a RAM or ROM that is a memory, an external storage device that is a hard disk, and an input unit, an output unit, or a communication unit thereof , A CPU, a RAM, a ROM, and a bus connected so that data can be exchanged between the external storage devices. If necessary, the hardware entity may be provided with a device (drive) that can read and write a recording medium such as a CD-ROM. A physical entity having such hardware resources includes a general-purpose computer.

  The external storage device of the hardware entity stores a program necessary for realizing the above functions and data necessary for processing the program (not limited to the external storage device, for example, reading a program) It may be stored in a ROM that is a dedicated storage device). Data obtained by the processing of these programs is appropriately stored in a RAM or an external storage device.

  In the hardware entity, each program stored in an external storage device (or ROM or the like) and data necessary for processing each program are read into a memory as necessary, and are interpreted and executed by a CPU as appropriate. . As a result, the CPU realizes predetermined functions (respective constituent requirements expressed as the above-described units,.

  The present invention is not limited to the above-described embodiment, and can be appropriately changed without departing from the spirit of the present invention. In addition, the processing described in the above embodiment may be executed not only in time series according to the order of description but also in parallel or individually as required by the processing capability of the apparatus that executes the processing. .

  As described above, when the processing functions in the hardware entity (the apparatus of the present invention) described in the above embodiments are realized by a computer, the processing contents of the functions that the hardware entity should have are described by a program. Then, by executing this program on a computer, the processing functions in the hardware entity are realized on the computer.

  The program describing the processing contents can be recorded on a computer-readable recording medium. As the computer-readable recording medium, for example, any recording medium such as a magnetic recording device, an optical disk, a magneto-optical recording medium, and a semiconductor memory may be used. Specifically, for example, as a magnetic recording device, a hard disk device, a flexible disk, a magnetic tape or the like, and as an optical disk, a DVD (Digital Versatile Disc), a DVD-RAM (Random Access Memory), a CD-ROM (Compact Disc Read Only). Memory), CD-R (Recordable) / RW (ReWritable), etc., magneto-optical recording medium, MO (Magneto-Optical disc), etc., semiconductor memory, EEP-ROM (Electronically Erasable and Programmable-Read Only Memory), etc. Can be used.

  The program is distributed by selling, transferring, or lending a portable recording medium such as a DVD or CD-ROM in which the program is recorded. Furthermore, the program may be distributed by storing the program in a storage device of the server computer and transferring the program from the server computer to another computer via a network.

  A computer that executes such a program first stores, for example, a program recorded on a portable recording medium or a program transferred from a server computer in its own storage device. When executing the process, the computer reads a program stored in its own recording medium and executes a process according to the read program. As another execution form of the program, the computer may directly read the program from a portable recording medium and execute processing according to the program, and the program is transferred from the server computer to the computer. Each time, the processing according to the received program may be executed sequentially. Also, the program is not transferred from the server computer to the computer, and the above-described processing is executed by a so-called ASP (Application Service Provider) type service that realizes the processing function only by the execution instruction and result acquisition. It is good. Note that the program in this embodiment includes information that is used for processing by an electronic computer and that conforms to the program (data that is not a direct command to the computer but has a property that defines the processing of the computer).

  In this embodiment, a hardware entity is configured by executing a predetermined program on a computer. However, at least a part of these processing contents may be realized by hardware.

100 Sound Source Enhancement Learning Device 110 Feature Quantity / Label Generation Unit 120 Pre-Learning Unit 121 Beamforming Output Power Calculation Unit 122 Network Parameter Calculation Unit 126 Target Sound Non-Negative Auto Encoder Calculation Unit 127 Noise Non-Negative Auto Encoder Calculation Unit 128 Complementary Subtraction Neural Network Calculation Unit 129 Network parameter optimization unit 200 Sound source enhancement device 220 Wiener filter estimation unit 221 Target sound non-negative auto encoder unit 222 Noise non-negative auto encoder unit 223 Complementary subtraction unit 224 Wiener filter calculation unit 230 Signal enhancement unit 800 Sound source enhancement learning device 810 Label generation unit 820 Pre-learning unit 890 Learning data recording unit 900 Sound source enhancement device 910 Frequency domain conversion unit 920 Binary mask estimation unit 930 Signal enhancement unit 940 Time domain change Replacement unit 990 Learning result recording unit

Claims (6)

Features / labels that generate frequency domain mixed sound learning data as features and sets target sound PSD and noise PSD as labels from a set of frequency domain mixed sound learning data, frequency domain target sound learning data, and frequency domain noise learning data A generator,
A sound source enhancement learning apparatus comprising: a pre-learning unit that learns a network parameter that characterizes a neural network for sound source enhancement having an observation signal as an input and an estimated target sound PSD and an estimated noise PSD as an output, using the feature amount and the label. There,
The pre-learning unit
The target sound that calculates the target sound spectrum characteristic vector that models the spectral characteristic of the target sound from the target sound BF output power that is the power of the beamforming output signal that emphasizes the target sound generated from the frequency domain mixed sound learning data Non-negative auto encoder calculation unit,
A noise non-negative auto encoder calculation unit that calculates a noise spectrum characteristic vector that models a noise spectrum characteristic from a noise BF output power that is a power of a beamforming output signal that emphasizes noise generated from the frequency domain mixed sound learning data When,
A complementary subtraction neural network calculation unit for calculating the estimated target sound PSD and the estimated noise PSD from the target sound spectrum characteristic vector and the noise spectrum characteristic vector;
A sound source enhancement learning apparatus comprising: the estimated target sound PSD, the estimated noise PSD, and a network parameter optimization unit that optimizes the network parameter using the target sound PSD and the noise PSD. The sound source enhancement learning device according to claim 1,
Among the network parameters,
Network parameters used in the target sound non-negative auto encoder calculation unit are non-negative weight matrices W _S ⁽²⁾ and W _S ⁽³⁾ and bias vectors b _S ⁽²⁾ and b _S ⁽³⁾ , and W _S ^{(2 )} And W _S ⁽³⁾ satisfy the following relationship:

Network parameters used in the noise non-negative auto encoder calculation unit are non-negative weight matrices W _N ⁽²⁾ , W _N ⁽³⁾ and bias vectors b _N ⁽²⁾ , b _N ⁽³⁾ , and W _N ⁽²⁾ And W _N ⁽³⁾ satisfy the following relationship:

The network parameter used in the complementary subtraction neural network calculation unit is a weight matrix W ⁽⁴⁾ , the weight matrix W ⁽⁴⁾ is an initial value W _init ⁽⁴⁾ ,

( _Wherein λ _{S, ω_i} , λ _{N, ω_i} are positive values, γ _{SN, ω_i} , γ _{NS, ω_i} are negative values, 1 ≦ i ≦ Ω) . A sound source emphasizing device for generating an emphasis sound obtained by emphasizing the observation signal from the observation signal using a neural network for sound source emphasis in which the network parameter generated using the sound source emphasis learning device according to claim 1 is set. There,
A frequency domain transform unit for generating a frequency domain observation signal from the observation signal;
Using the neural network for sound source enhancement, an estimated target sound PSD and an estimated noise PSD are generated from the frequency domain observation signal, and a Wiener filter estimating unit that estimates a Wiener filter from the estimated target sound PSD and the estimated noise PSD;
Using the Wiener filter, a signal enhancement unit that generates a frequency domain enhanced sound from the frequency domain observation signal;
A sound source emphasizing device comprising: a time domain transforming unit that performs time domain transformation on the frequency domain enhanced sound and generates the enhanced sound. Features / labels that generate frequency domain mixed sound learning data as features and sets target sound PSD and noise PSD as labels from a set of frequency domain mixed sound learning data, frequency domain target sound learning data, and frequency domain noise learning data Generation step;
A sound source enhancement learning method including a pre-learning step of learning a network parameter that characterizes a neural network for sound source enhancement having an observation signal as an input and an estimated target sound PSD and an estimated noise PSD as an output using the feature amount and the label. There,
The pre-learning step includes
The target sound that calculates the target sound spectrum characteristic vector that models the spectral characteristic of the target sound from the target sound BF output power that is the power of the beamforming output signal that emphasizes the target sound generated from the frequency domain mixed sound learning data Non-negative auto encoder calculation step,
Noise non-negative auto encoder calculation step for calculating a noise spectrum characteristic vector modeling the noise spectrum characteristic from the noise BF output power, which is the power of a beamforming output signal emphasizing noise generated from the frequency domain mixed sound learning data When,
Complementary subtraction neural network calculation step of calculating the estimated target sound PSD and the estimated noise PSD from the target sound spectrum characteristic vector and the noise spectrum characteristic vector;
A sound source enhancement learning method comprising: the estimated target sound PSD, the estimated noise PSD, and a network parameter optimization step that optimizes the network parameter using the target sound PSD and the noise PSD.   A program for causing a computer to function as the sound source enhancement learning device according to claim 1 or 2, or the sound source enhancement device according to claim 3. Signal processing that learns network parameters that characterize a neural network for signal processing that takes a signal as an input and outputs an estimation processing result of the signal, using learning data consisting of a set of a feature amount related to the signal and a label that is a processing result of the signal A learning device,
The estimation process result calculated in the middle stage from the feature quantity calculated from the signal to the output of the estimation process result is the middle stage estimation process result i (1 ≦ i ≦ n, n is an integer of 1 or more). ,
Using the feature amount, a neural network 1 calculation unit that calculates the intermediate stage estimation processing result 1;
A neural network j calculation unit for calculating the intermediate stage estimation process result j using any one of the feature quantity or the intermediate stage estimation process result 1 to the intermediate stage estimation process result j-1 (2 ≦ j ≦ n ),
Using either the feature amount or the intermediate stage estimation process result 1 to the intermediate stage estimation process result n, a neural network n + 1 calculation unit that calculates the estimation process result;
A signal processing learning device comprising: a network parameter optimization unit that optimizes the network parameter using the estimation processing result and the label.

Images