JP2023025458A - Sound source separation device, sound source separation method, and sound source separation program - Google Patents

Abstract

To provide a sound source separation device which has improved estimation accuracy for a sound signal only from observation signals by matching combination of frequency directions of separation signals even if positioning of a microphone is unknown.SOLUTION: A sound source separation device 100 comprises: a masking unit 103 that masks a frequency band for a plurality of predetermined different frequency bands with respect to a separation signal generated by separating an observation signal including a plurality of mixed construction sounds according to a separation matrix; a spectrogram generation unit 104 that generates a compensation spectrogram using the masked separation signal and a predetermined sound source model, and generates a separation signal spectrogram in each frequency band of the separation signal; and a separation matrix correction unit 106 that corrects the separation matrix so as to achieve rearrangement of frequency bands in a reallocation target assigned so that a distance from the compensation becomes closer to each of the separation signal spectrum.SELECTED DRAWING: Figure 3

Description

The disclosed technology relates to a sound source separation device, a sound source separation method, and a sound source separation program.

Blind source separation (BSS) is a technique for estimating each sound source signal only from an observed signal in which a plurality of sound source signals are mixed under the condition that the sound source or the transfer characteristics from the sound source to the microphone are unknown. BSS approaches formulated in the frequency domain, such as frequency-domain independent component analysis (FDICA), can express the sound source mixing process as an instantaneous mixing system that does not include convolution operations. It has the advantage of being able to implement a highly efficient algorithm.

However, in FDICA, since the order of the separated signals obtained for each frequency is arbitrary, permutation matching processing for grouping independent components for each frequency derived from the same sound source is separately required in the subsequent stage. As permutation matching processing, a solution using the direction of arrival of the sound source obtained from the correlation of the power of adjacent frequencies or the position information of the microphone as a clue, and a method combining the two have been proposed (Non-Patent Document 1). .

On the other hand, not as a post-processing, but by modeling the inter-frequency component dependencies of the sound sources and incorporating them into the BSS optimization problem in the form of constraints or costs, permutation matching and frequency-wise sound source separation can be performed simultaneously. In recent years, many methods for solving this problem have been proposed, and their effects have been demonstrated.

For example, in Independent Low-Rank Matrix Analysis (ILRMA), the power spectrogram of each sound source is represented by the product of two non-negative matrices. This corresponds to the assumption that the power spectrum can be approximated at each time frame by a linear sum of scaled basis spectra with time-varying amplitudes. This constraint allows ILRMA to simultaneously solve the frequency-wise source separation and permutation matching problems while cuing the spectral structure of the sources.

In addition, in the multichannel variational autoencoder (MVAE) method, the generation model of the sound source spectrogram is represented by conditional VAE (Conditional VAE: CVAE), and pre-learning using the training sample It makes it possible to capture the dependence of components between frequencies and between times of sound sources. High-precision sound source separation can be performed by estimating the separation matrix so that each separated signal matches this sound source generation model as much as possible.

Hiroshi Sawada, Ryo Mukai, Shoko Araki, and Shoji Makino, “A robust and precise method for solving the permutation problem of frequency-domain blind source separation,” IEEE transactions on speech and audio processing, vol. 12, no. 5, pp. 530-538, 2004.

Even with a technique that simultaneously solves permutation matching and sound source separation for each frequency, permutation mismatch can occur for each band block. This is called the block permutation problem, and is caused by the fact that the sound source model does not adequately capture the dependency between distant frequency bands, or the sound source model has too high expressive power. If this block permutation problem can be solved, further improvement in separation accuracy can be expected.

The block permutation problem can be regarded as an "assignment problem" to find which sound source the separated signal for each frequency corresponds to, using an appropriate cost as a clue. However, since the separation matrix estimated for each frequency when estimating each sound source signal from only the observed signal is obtained by BSS, the frequency direction of the separation result is Finding a suitable combination is difficult. Moreover, even if restrictions are placed on the structure of the sound source in the frequency direction, the structure between distant frequency bands cannot be fully restricted, and a separation result in which the values of the frequency bands are interchanged may be obtained. Conventionally, the combination of frequency directions is matched using the direction of arrival of the sound source obtained from the correlation of the power of adjacent frequencies or the positional information of the microphone. However, the technique of using microphone position information as in Non-Patent Document 1 impairs the advantage of BSS that can be operated even if the microphone arrangement is unknown.

The disclosed technology has been made in view of the above points, and improves the accuracy of estimating the sound source signal from only the observed signal by matching the combination of the separated signals in the frequency direction even if the placement of the microphones is unknown. It is an object of the present invention to provide a sound source separation device, a sound source separation method, and a sound source separation program.

A first aspect of the present disclosure is a sound source separation device, in which a separated signal obtained by separating an observed signal in which a plurality of constituent sounds are mixed is separated by a separation matrix, and a predetermined plurality of different frequency bands are separated from each other. A spectrogram generator for generating a complementary spectrogram using a masking unit for masking, the separated signal masked by the masking unit, and a predetermined sound source model, and for generating a separated signal spectrogram in each of the frequency bands of the separated signal. and a separation matrix correction unit that corrects the separation matrix so as to rearrange the frequency bands to be rearranged so that the distance between each of the separated signal spectrograms and the complementary spectrogram is reduced. and including.

A second aspect of the present disclosure is a sound source separation method, wherein for a separated signal obtained by separating an observed signal in which a plurality of constituent sounds are mixed by a separation matrix, each of a plurality of predetermined different frequency bands is divided into the respective frequency bands. Masking, generating an interpolated spectrogram using the masked separated signal and a predetermined sound source model, generating a separated signal spectrogram in each of the frequency bands of the separated signal, and generating a separated signal spectrogram for each of the separated signal spectrograms The computer executes a process of correcting the separation matrix so as to realize rearrangement of the frequency bands to be rearranged so that the distance to the complementary spectrogram becomes closer to the complementary spectrogram.

A third aspect of the present disclosure is a program for causing a computer to function as the sound source separation device of the first aspect.

According to the disclosed technique, by matching the combination of the separated signals in the frequency direction even if the arrangement of the microphones is unknown, the separated signal can be obtained with higher accuracy than when the combination of the separated signals in the frequency direction is not matched. be able to.

FIG. 3 is a diagram showing an algorithm of the HBP method proposed in an embodiment of the disclosed technique; 2 is a block diagram showing the hardware configuration of the sound source separation device; FIG. 2 is a block diagram showing an example of a functional configuration of a sound source separation device; FIG. 4 is a flowchart showing the flow of sound source separation processing by the sound source separation device; FIG. 4 is a diagram showing the arrangement of microphones and sound sources in a separation experiment; FIG. 4 is a diagram showing an example of separated signals of 9 sound sources;

An example of embodiments of the technology disclosed herein will be described below with reference to the drawings. In each drawing, the same or equivalent components and portions are given the same reference numerals. Also, the dimensional ratios in the drawings are exaggerated for convenience of explanation, and may differ from the actual ratios.

<Overview of this embodiment>
First, an outline of this embodiment will be described.

In this embodiment, first, after estimating a sound source model representing the structure of the sound source in the frequency direction and a separation matrix to separate the signals, for each of a plurality of predetermined different frequency bands, the frequency band of the separated signal is deleted ( Masking) is performed, and then an interpolated spectrogram restored using the sound source model is prepared. Then, the separated signals of each missing frequency band are rearranged using the Hungarian method, which is an efficient algorithm, so that the distance to the complementary spectrogram becomes closer. As a result, even if the microphone arrangement is unknown, it is possible to match the combination of separated signals in the frequency direction.

<Principle of this embodiment>
Next, the technical principle of this embodiment will be described.

（１）

（２）
とする。ただし、ここではＩ＝Ｊの優決定条件を考える。ここで（ ）^Ｔは転置を表し、ｆとｎはそれぞれ周波数と時間のインデックスである。 <Formulation of frequency domain multi-channel sound source separation problem>
Consider the case of observing signals coming from J sound sources with I microphones. Let x _i (f, n) and s _j (f, n) be the complex spectrograms of the observed signal of microphone i and the signal of sound source j, respectively. Also, a vector with these elements as

(1)

(2)
and However, here, the over-determination condition of I=J is considered. where ( ) ^T represents the transpose and f and n are the frequency and time indices, respectively.

（３）

（４）
を仮定することができる。ここで、Ｗ^Ｈ（ｆ）は分離行列を表し、（ ）^Ｈはエルミート転置である。 Under the condition of I=J, when the reverberation time is shorter than the analysis window length, the instantaneous separation system

(3)

(4)
can be assumed. where W ^H (f) represents the separation matrix and ( ) ^H is the Hermitian transpose.

（５） Under the assumption of the instantaneous mixing system described above, the likelihood function of the mixed signal can be expressed as Equation (5).

(5)

の複素正規分布

（６）
に従う確率変数と仮定する。ここで、パワースペクトログラムＶ_ｊ＝｛ｖ_ｊ（ｆ，ｎ）｝_ｆ，ｎを非負値行列積で表現したバージョンがＩＬＲＭＡのモデル、ＣＶＡＥのデコーダで表現したバージョンがＭＶＡＥ法のモデルにそれぞれ対応する。ｓ_ｊ（ｆ，ｎ）とｓ_ｊ’（ｆ，ｎ）、ｊ≠ｊ’が統計的に独立のとき、数式（６）により、ｓ（ｆ，ｎ）は

（７）
に従う。ここで、Ｖ（ｆ，ｎ）はｖ_１（ｆ，ｎ），・・・，ｖ_Ｉ （ｆ，ｎ）を対角要素に持つ対角行列である。 The ILRMA method and the MVAE method assume a Local Gaussian Model (LGM). That is, the complex spectrogram s _j (f, n) of the sound source signal j has a mean of 0 and a variance of

complex normal distribution of

(6)
is assumed to be a random variable that follows Here, the version of the power spectrogram V _j ={v _j (f, n)} _{f, n} represented by the non-negative matrix product corresponds to the ILRMA model, and the version represented by the CVAE decoder corresponds to the MVAE method model. . When s _j (f, n) and s _j′ (f, n), j≠j′ are statistically independent, s(f, n) is given by Equation (6) as

(7)
obey. Here, V(f, n) is a diagonal matrix having v ₁ (f, n), . . . , v _I (f, n) as diagonal elements.

が与えられた下での分離行列

と、各音源のパワースペクトログラム

の対数尤度関数は、

（８）
となる。 From equations (5) and (7), the observed signal

given the separation matrix

and the power spectrogram of each sound source

The log-likelihood function of is

(8)
becomes.

（９）
のように、Ｗ（ｆ）に置換行列Ｐ_ｋを乗じたものが局所解として推定されていることになる。数式（９）で、Ｆ_ｋはｋ番目の帯域ブロック内の周波数ビンの集合であり、Ｐ_ｋは当該帯域における正解音源成分と分離成分の順番とを対応付ける置換行列である。ブロックパーミュテーション問題は、帯域ｋごとにＰ^－１ _ｋ＝Ｐ^Ｔ _ｋを推定し、

（１０）
により、正解の分離行列を見つける問題となる。なお、周波数ビンごとに帯域を分割した場合、ブロックパーミュテーション問題は通常のパーミュテーション問題に帰着する。 <Formulation of block permutation problem>
If the equation (8) can be maximized on the assumption of an ideal sound source model V, it will be possible to simultaneously solve sound source separation and permutation matching for each frequency. However, many existing sound source models can flexibly adapt to spectrograms in which some frequency band components are completely replaced with other sound source components. As a result, separated signals having different sound source components for each band are obtained. In a situation where each frequency component in a certain band _Fk is similarly exchanged between sound sources, if the correct separation matrix is W(f),

(9)
, W(f) multiplied by the permutation matrix _Pk is estimated as the local solution. In Equation (9), F _k is a set of frequency bins in the k-th band block, and P _k is a permutation matrix that associates correct sound source components with the order of separated components in the band. The block permutation problem estimates P ⁻¹ _k =P ^T _k for each band k,

(10)
, it becomes a problem of finding the correct separation matrix. Note that when the band is divided for each frequency bin, the block permutation problem is reduced to a normal permutation problem.

<Explanation of quota problem and Hungarian law>
The permutation problem can be regarded as an "assignment problem" of finding which sound source the separated signal for each frequency corresponds to, using an appropriate cost as a clue. In this embodiment, the Hungarian method, which is one of the methods for solving the assignment problem, is used to solve the permutation problem. Therefore, first, the allocation problem and the Hungarian method will be outlined below.

とする。ただし、ｐ＝１，・・・，Ｍとｑ＝１，・・・，Ｍは、それぞれ作業員と仕事のインデックスである。この場合、割当問題は

（１１）
を満足する分配行列

を求める最適化問題として定式化することができる。ただし、＜ ， ＞は行列の内積を表す。 The assignment problem is the problem of finding the most efficient job assignment when assigning M jobs to M workers. Let c _pq be the cost required for worker p to do job q, and the cost matrix with c _pq as the element of p row and q column is

and where p=1, . . . , M and q=1, . In this case the allocation problem is

(11)
A distribution matrix that satisfies

can be formulated as an optimization problem for However, < , > represent the inner product of matrices.

（１２）

（１３）
を満足する実数集合

及び

が存在することを仮定する。これらのΠ及び∇は双対問題により求められる。最適化問題のコストｚは、数式（１４）のように表せる。

（１４） When solving this optimization problem by full enumeration, M! An increase in M results in a combinatorial explosion, since there are valid solution candidates. The Hungarian method is an algorithm that efficiently solves this optimization problem. under Hungarian law

(12)

(13)
real number set satisfying

as well as

Suppose that there exists These Π and ∇ are obtained by the dual problem. The cost z of the optimization problem can be expressed as Equation (14).

(14)

Equation (14) shows that subtracting the constants u _p and r _q from any row and column of the cost matrix, respectively, does not affect the optimal allocation. The Hungarian method can use this property to find the optimal allocation while modifying the cost matrix. The specific procedure is as follows.

(Step 1) Find the minimum value in each row and subtract that minimum value from each element in that row. Then similarly find the minimum value in each column and subtract that minimum value from each element in that column.

(Step 2) Determine whether one 0 can be selected from each row and column of the matrix after subtracting the minimum value. If a choice can be made, that set of coordinates is the optimal allocation proposal. If you can't choose, go to the next step.

(Step 3) Draw as few lines as possible on rows or columns to obscure all 0 entries in the matrix after subtracting the minimum.

(Step 4) Subtract the minimum value among those elements from the elements of the matrix not covered by the line drawn in step 3, and apply the minimum Add the value and go back to step 2.

Note that the computation time order is O(M!) for the full enumeration method, while it is O(M ³ ) for the Hungarian method. That is, as the number of M increases, the Hungarian method can find the most efficient work assignment more efficiently than the full enumeration method.

<Hungarian block permutation method>
Next, we describe a block permutation matching method using the Hungarian method. A block permutation matching method using the Hungarian method is called an HBP (Hungarian Block Permutation) method. The conventional permutation matching method, which uses the correlation between components at adjacent frequencies of separated signals or the direction of arrival as a clue, has drawbacks such as the need for iterative calculations and the fact that the microphone placement must be known. On the other hand, the HBP method proposed in this embodiment is a method that can use the sound source model used in the sound source separation algorithm as it is, and is a method that can be applied even when the microphone arrangement is unknown. Specifically, in the HBP method, in the middle of the sound source separation algorithm, (1) the high-band components of each separated signal are artificially masked (zeroed), and (2) the missing components of the band using the sound source model and (3) performing block permutation matching by the Hungarian method based on the restored value.

とすると、ｌ番目の帯域を欠損（マスキング）した後に当該帯域の補完を行う過程は

（１５）
と表せる。ただし、

は行列の要素積を表す。 Let R(·) be a function (missing band complementer) that receives as input a spectrogram in which some band components are missing and outputs a spectrogram in which the missing region is interpolated. A matrix V j whose elements are v _j (f, n) corresponding to the separated signal _j is

Then, the process of complementing the band after missing (masking) the l-th band is

(15)
can be expressed as however,

represents the element product of matrices.

の全要素が１であるような行列Ｖ_ｊと同じサイズのバイナリ行列を表す。 Assuming that the frequency band to be masked is G _l , M _l ε{0, 1} ^F×N has all elements of row fεG _l being 0, row

represents a binary matrix of the same size as the matrix V _j such that all elements of are ones.

はＶｊに比べて、当該音源が本来もつべきスペクトログラムに近いものになっていることが期待できる。このことを利用し、

を用いて適切なコスト行列を設計できれば、ハンガリアン法を応用してブロックパーミュテーション整合を行うことができる。つまり本実施形態で提案する手法は、各ｌの欠損帯域を補完したスペクトログラム

を用いて各ｋの帯域内のブロックパーミュテーション整合を行うアルゴリズムとなる。ある（ｌ，ｋ）において、割当問題のコスト行列Ｃ^{（ｌ，ｋ）}の各要素ｃ^{（ｌ，ｋ）} _ｊｊ’は、

が帯域Ｆ_ｋにおいて分離信号ｊ’とどれくらい適合しているかを測る尺度となっていれば良い。本実施形態では、要素ｃ^{（ｌ，ｋ）} _ｊｊ’は、数式（８）の対数尤度関数に関連させて、

と、

との板倉齋藤距離である

（１６）
とした。勿論、コスト行列Ｃ^{（ｌ，ｋ）}の各要素ｃ^{（ｌ，ｋ）} _ｊｊ’の決め方は係る例に限定されるものではない。 Although there are various options for the concrete form of the function R(·) and the pre-learning method, the sound source model used in the sound source separation algorithm may be used as it is, as will be described later. If the missing band compensating ability of the function R(·) is sufficiently high, and if permutation mismatch occurs in the high range of V _j , the processing of formula (15) yields

can be expected to be closer to the spectrogram that the sound source should originally have than Vj. Taking advantage of this

can be used to design an appropriate cost matrix, the Hungarian method can be applied to perform block permutation matching. In other words, the method proposed in this embodiment is a spectrogram

is used to perform block permutation matching within each k band. For some (l, k), each element c ^{(l, k)} _jj' of the allocation problem cost matrix C ( ^{l, k} ) is

is a measure of how well it matches the separated signal j' in the band _Fk . In this embodiment, the element c ^(l,k) _jj' is related to the log-likelihood function of equation (8) as follows:

and,

is the Itakura-Saito distance between

(16)
and Of course, the method of determining each element c ^(l,k) _jj' of the cost matrix C ^(l,k) is not limited to this example.

The method of determining the band _Fk and the masking frequency band _Gl is not limited to a specific method. For example, _Fk and _Gl may be randomly determined. Also for example, F _k may be such that each band block is a different single frequency bin, ie, F _k ={k}(k=F ₀ , . . . , F). Further, for example, G _l may be only one type of set consisting of frequency bins with missing bands equal to or higher than the Nyquist frequency, that is, G _l ={F ₀ , . . . , F}.

を求めている。そして図１に示したＨＢＰ法のアルゴリズムは、数式（１６）によってコスト行列Ｃ^{（ｌ，ｋ）}を計算する。続いて図１に示したＨＢＰ法のアルゴリズムは、ハンガリアン法によりコスト行列Ｃ^{（ｌ，ｋ）}から置換行列Ｐ_ｋを求め、数式（１０）により正解の分離行列を求める。 FIG. 1 is a diagram showing an algorithm of the HBP method proposed in this embodiment. An outline of the algorithm of the HBP method shown in FIG. 1 will be described. The algorithm of the HBP method first masks the frequency band G _l of V _j using Equation (15), and the missing band interpolator

I am looking for Then, the algorithm of the HBP method shown in FIG. 1 calculates the cost matrix C ^{(l, k)} by Equation (16). Subsequently, the algorithm of the HBP method shown in FIG. 1 obtains the permutation matrix P _k from the cost matrix C ^{(l, k)} by the Hungarian method, and obtains the correct separation matrix by Equation (10).

<Hardware configuration>
FIG. 2 is a block diagram showing the hardware configuration of the sound source separation device 100. As shown in FIG.

As shown in FIG. 2, the sound source separation apparatus 100 includes a CPU (Central Processing Unit) 11, a ROM (Read Only Memory) 12, a RAM (Random Access Memory) 13, a storage 14, an input section 15, a display section 16, and a communication interface. (I/F) 17. Each component is communicatively connected to each other via a bus 19 .

The CPU 11 is a central processing unit that executes various programs and controls each section. That is, the CPU 11 reads a program from the ROM 12 or the storage 14 and executes the program using the RAM 13 as a work area. The CPU 11 performs control of each configuration and various arithmetic processing according to programs stored in the ROM 12 or the storage 14 . In this embodiment, the ROM 12 or storage 14 stores a sound source separation program for separating each sound source signal from the observed signal.

The ROM 12 stores various programs and various data. The RAM 13 temporarily stores programs or data as a work area. The storage 14 is configured by a storage device such as a HDD (Hard Disk Drive) or an SSD (Solid State Drive), and stores various programs including an operating system and various data.

The input unit 15 includes a pointing device such as a mouse and a keyboard, and is used for various inputs.

In this embodiment, the input unit 15 receives an observed signal in which a plurality of sound source signals are mixed. The observation signal received by the input unit 15 is separated into each sound source signal by the CPU 11 .

The display unit 16 is, for example, a liquid crystal display, and displays various information. The display unit 16 may employ a touch panel system and function as the input unit 15 .

The communication interface 17 is an interface for communicating with other devices. The communication uses, for example, a wired communication standard such as Ethernet (registered trademark) or FDDI, or a wireless communication standard such as 4G, 5G, or Wi-Fi (registered trademark).

<Functional configuration>
Next, the functional configuration of the sound source separation device 100 will be described.

FIG. 3 is a block diagram showing an example of the functional configuration of the sound source separation device 100. As shown in FIG.

As shown in FIG. 3 , the sound source separation apparatus 100 includes a learning unit 101, a model storage unit 102, a masking unit 103, a spectrogram generation unit 104, a band allocation unit 105, a separation matrix correction unit 106, and a sound source separation unit. 107. Each functional configuration is realized by the CPU 11 reading out a sound source separation program stored in the ROM 12 or the storage 14, developing it in the RAM 13, and executing it.

The learning unit 101 performs machine learning of the sound source model used for correcting the separation matrix. Although the machine learning processing of the sound source model in this embodiment will be described, the machine learning processing of the sound source model is not limited to the one described below.

（１７）

（１８）
ｐ^＋ _θ（Ｓ｜ｚ，ｃ）、ｑ^＋ _φ（ｚ｜Ｓ）、ｒ^＋ _ψ（ｃ｜Ｓ）は、それぞれエンコーダ分布、デコーダ分布、クラス識別器分布を表し、θ、φ、ψは対応するネットワークのパラメータである。学習部１０１が学習した音源モデルは、欠損帯域補完器Ｒ（・）として用いることができる。 For machine learning by the learning unit 101, a large number of pairs of a spectrogram S of clean speech, a spectrogram S' obtained by masking an appropriate band of the spectrogram S, and a speaker label c are prepared in advance. The learning unit 101 learns a sound source model by inputting S′ or S for each pair and using S as a target value. When S′ is input, the following two are defined as reconstruction errors.

(17)

(18)
p ⁺ _θ (S|z,c), q ⁺ _φ (z|S), r ⁺ _ψ (c|S) represent the encoder distribution, the decoder distribution, and the class discriminator distribution, respectively, and θ, φ, and ψ are Corresponding network parameters. The sound source model learned by the learning unit 101 can be used as the missing band interpolator R(·).

The model storage unit 102 stores a sound source model used for correcting the separation matrix. The sound source model may be prepared in advance or machine-learned by the learning unit 101 .

The masking section 103 masks a plurality of predetermined different frequency bands with respect to the separated signal obtained by separating the observed signal mixed with the constituent sounds by the separating matrix. For example, the masking section 103 masks high-band components in the separated signals.

The spectrogram generation unit 104 uses the separated signals masked by the masking unit 103 and a predetermined sound source model stored in the model storage unit 102 to generate a complementary spectrogram that complements the missing part due to masking. Also, the spectrogram generator 104 generates a separated signal spectrogram in each frequency band of the separated signal.

Band allocation section 105 allocates a frequency band to each of the separated signal spectrograms generated by spectrogram generation section 104 so that the distance from the complementary spectrogram is short. Specifically, band allocation section 105 allocates the frequency band by calculating cost matrix C ^{(l, k)} using Equation (16) above.

Separation matrix correction section 106 corrects the separation matrix so as to realize rearrangement of frequency bands corresponding to allocation by band allocation section 105 . Specifically, separating matrix correction section 106 obtains permutation matrix P _k from cost matrix C ^{(l, k)} using the Hungarian method, and obtains the correct separating matrix by Equation (10), thereby obtaining the separating matrix fix it.

Sound source separation section 107 separates an observed signal in which a plurality of sound source signals are mixed into sound source signals using a separation matrix by BSS. In the present embodiment, the sound source separation unit 107 uses the separation matrix corrected by the separation matrix correction unit 106 at a predetermined timing as the separation matrix, and separates an observed signal in which a plurality of sound source signals are mixed into each sound source signal. do.

By having such a configuration, the sound source separation apparatus 100 can modify the separation matrix used when separating the observed signal in which the component sounds are mixed so as to match the combination of the separated signals in the frequency direction. Sound source separation apparatus 100 corrects the separation matrix so as to match the combination of separated signals in the frequency direction, thereby obtaining separated signals with higher accuracy than when the separation matrix is not corrected.

<Action>
Next, the operation of the sound source separation device 10 will be described.

FIG. 4 is a flowchart showing the flow of sound source separation processing by the sound source separation device 10. As shown in FIG. The CPU 11 reads the sound source separation program from the ROM 12 or the storage 14, develops it in the RAM 13, and executes it, thereby performing sound source separation processing.

In step S101, the CPU 11 masks a plurality of predetermined different frequency bands with respect to the separated signal obtained by separating the observed signal mixed with the constituent sounds by the separation matrix. Specifically, the CPU 11 masks high-band components in the separated signals.

Following step S101, in step S102, the CPU 11 uses a predetermined sound source model to generate a complementary spectrogram that complements the missing portion due to masking. The sound source model may be prepared in advance or machine-learned by the CPU 11 .

After step S102, in step S103, the CPU 11 generates a separated signal spectrogram in each frequency band of the separated signal. Note that the order of steps S102 and S103 may be reversed.

Following step S103, in step S104, the CPU 11 allocates frequency bands to each of the separated signal spectrograms generated in step S103 so that the distance to the complementary spectrogram is short. Specifically, in step S104, the CPU 11 allocates the frequency band by calculating the cost matrix C ^{(l, k)} according to Equation (16) above.

After step S104, in step S105, the CPU 11 corrects the separation matrix so as to realize the rearrangement of the frequency bands corresponding to the allocation performed in step S104. Specifically, in step S105, the CPU 11 obtains the permutation matrix P _k from the cost matrix C ^{(l, k)} using the Hungarian method, and obtains the correct separation matrix by Equation (10), thereby obtaining the separation matrix fix it.

When separating the observed signal mixed with the constituent sounds, the CPU 11 uses the separation matrix corrected by the series of processes of steps S101 to S105, so that the series of processes of steps S101 to S105 may or may not be performed. Separation signals can be obtained with high accuracy by comparison.

<effect>
In order to verify the speech separation performance of the sound source separation device 10 according to this embodiment, an arbitrary speaker separation experiment was conducted using the WSJ0 speech database. About 25 hours of data of 101 speakers included in the si_tr_s folder of the WSJ0 database was used as learning data, and data of 18 speakers in the si_dt_05 folder and the si_et_05 folder were used to create evaluation data. For verification, mixed signals with sound sources of {2, 3, 6, 9, 12, 15, 18} were created. Impulse responses were generated by the mirror image method with a wall reflection coefficient of 0.2. FIG. 5 is a diagram showing the arrangement of microphones and sound sources in a separation experiment. Ten mixed signals were generated for each condition. In addition, 10 sentences of a mixed signal were created for each condition using speech in which all utterances were repeated. A short-time Fourier transform was performed with a sampling frequency of 16 kHz for all audio signals, a frame length of 256 ms, and a shift of 128 ms, and a spectrogram was calculated.

There are FastMVAE method and FastMVAE2 method as high-speed versions of the MVAE method. In the FastMVAE method and the FastMVAE2 method, in order to realize a high-speed separation algorithm, the former is a VAE with a class discriminator (Auxiliary Classifier VAE: ACVAE), and the latter is a ChimeraACVAE that integrates an ACVAE encoder and a class discriminator. The approach is to prelearn the generative model of the sound source spectrogram and the inference process of its latent variables. The ChimeraACVAE network structure was used in this segregation experiment. The number of iterations of the algorithm was set to 60, and the HBP method of this embodiment was performed every 10 iterations. Also, Source-to-Distortion Ratio (SDR) was used as an evaluation criterion.

Table 1 shows the average SDR values.

According to Table 1, it was confirmed that the sound source separation performance was improved by the HBP method for all the number of sound sources. In addition, it was found that the improvement value is large for repeated data. This is thought to be an improvement due to the reduction in silent intervals and the increase in spaces with articulatory structures that serve as clues for solving permutations. FIG. 6 is a diagram showing an example of separated signals of 9 sound sources. FIG. 6 shows, from the top, the correct signal, the separated signal not using the HBP method by FastMVAE2, and the separated signal using the HBP method by FastMVAE2 generated by the sound source separation apparatus 10. Input SDR and SDR are shown above each spectrogram. Improvement values are indicated.

As described above, according to the embodiments of the present disclosure, even if the microphone arrangement is unknown, by matching the combination of the separated signals in the frequency direction, the separated signals can be obtained with higher accuracy than when they are not matched. can be done.

Note that the sound source separation processing executed by the CPU by reading the software (program) in each of the above embodiments may be executed by various processors other than the CPU. The processor in this case is a PLD (Programmable Logic Device) whose circuit configuration can be changed after manufacturing such as an FPGA (Field-Programmable Gate Array), and an ASIC (Application Specific Integrated Circuit) for executing specific processing. A dedicated electric circuit or the like, which is a processor having a specially designed circuit configuration, is exemplified. In addition, the sound source separation processing may be performed by one of these various processors, or a combination of two or more processors of the same or different type (for example, multiple FPGAs and a combination of a CPU and an FPGA). etc.). More specifically, the hardware structure of these various processors is an electric circuit in which circuit elements such as semiconductor elements are combined.

Also, in each of the above-described embodiments, the sound source separation program has been pre-stored (installed) in the storage 14, but the present invention is not limited to this. The program is stored in non-transitory storage media such as CD-ROM (Compact Disk Read Only Memory), DVD-ROM (Digital Versatile Disk Read Only Memory), and USB (Universal Serial Bus) memory. may be provided in the form Also, the program may be downloaded from an external device via a network.

The following additional remarks are disclosed regarding the above embodiments.
(Appendix 1)
memory;
at least one processor connected to the memory;
including
The processor
Masking each of a plurality of predetermined different frequency bands for a separated signal obtained by separating an observed signal in which constituent sounds are mixed by a separation matrix,
generating a complementary spectrogram using the masked separated signal and a predetermined sound source model, and generating a separated signal spectrogram in each of the frequency bands of the separated signal;
A sound source configured to modify the separation matrix so as to realize rearrangement of frequency bands to be rearranged so that each of the separated signal spectrograms is closer to the complementary spectrogram. separation device.

(Appendix 2)
A non-temporary storage medium storing a program executable by a computer to perform sound source separation processing,
The sound source separation processing includes:
Masking each of a plurality of predetermined different frequency bands for a separated signal obtained by separating an observed signal in which constituent sounds are mixed by a separation matrix,
generating a complementary spectrogram using the masked separated signal and a predetermined sound source model, and generating a separated signal spectrogram in each of the frequency bands of the separated signal;
A non-temporary storage medium that modifies the separation matrix so as to rearrange frequency bands to be rearranged so that each of the separated signal spectrograms is closer to the complementary spectrogram.

100 Sound source separation device 101 Learning unit 102 Model storage unit 103 Masking unit 104 Spectrogram generation unit 105 Band allocation unit 106 Separation matrix correction unit 107 Sound source separation unit

Claims (7)

a masking unit for masking a plurality of predetermined different frequency bands with respect to a separated signal obtained by separating an observed signal in which a plurality of constituent sounds are mixed by a separation matrix;
a spectrogram generation unit that generates a complementary spectrogram using the separated signal masked by the masking unit and a predetermined sound source model, and generates a separated signal spectrogram in each of the frequency bands of the separated signal;
A separation matrix correction unit that corrects the separation matrix so as to realize rearrangement of frequency bands to be rearranged so that the distance between each of the separated signal spectrograms and the complementary spectrogram is reduced;
A sound source separation device. 2. The sound source separation according to claim 1, wherein the separation matrix correction unit obtains a permutation matrix using a Hungarian method for the cost matrix generated using the complementary spectrogram, and corrects the separation matrix using the permutation matrix. Device. 3. The sound source separation apparatus according to claim 1, further comprising a sound source separation unit that separates the observed signals using the separation matrix corrected by the separation matrix correction unit. The sound source separation device according to any one of claims 1 to 3, further comprising a learning unit that obtains the sound source model by predetermined machine learning. 5. The sound source separation according to claim 4, wherein the learning unit receives as input a spectrogram of clean speech or a spectrogram obtained by masking a part of the band of the spectrogram of clean speech, and learns the sound source model using the spectrogram of clean speech as a target value. Device. Masking each of a plurality of predetermined different frequency bands for a separated signal obtained by separating an observed signal in which a plurality of constituent sounds are mixed by a separation matrix,
generating a complementary spectrogram using the masked separated signal and a predetermined sound source model, and generating a separated signal spectrogram in each of the frequency bands of the separated signal;
A computer executes a process of correcting the separation matrix so as to rearrange the frequency bands to be relocated so that the distance from the complementary spectrogram is reduced for each of the separated signal spectrograms, sound source separation method. A sound source separation program for causing a computer to function as the sound source separation device according to any one of claims 1 to 5.

Images