JP2023025457A - Signal analysis device, signal analysis method, and signal analysis program - Google Patents

Abstract

To precisely separate each of constituent tones from a mixture signal in which the constituent tones are mixed while reducing the calculation cost.SOLUTION: A learning unit 32 learns an encoder that estimates a latent vector series in response to an input of a spectrogram of a tone on the basis of a spectrogram of each constituent tone and an attribute class thereof, an identifier that identifies an attribution class which indicates the attribute of the tone in response to an input of the spectrogram of the tone, and a decoder that generates distribution of the spectrogram of the tone in response to an input of the latent vector series and the attribute class. A parameter estimation unit 36 estimates a separate matrix and a scale parameter to optimize an objective function in response to an input of an observation signal in which the constituent tones are mixed.SELECTED DRAWING: Figure 5

Description

The technology disclosed herein relates to a signal analysis device, a signal analysis method, and a signal analysis program.

Blind source separation (BSS) is a technology that separates and extracts individual sound source signals from only the observed mixed signal without using prior information such as information about the sound source and the transfer function between the sound source and the microphone. be. Under overdetermined conditions where the number of microphones is greater than or equal to the number of sound sources, Independent Component Analysis (ICA) is effective for estimating separation filters so as to maximize the independence between sound source signals. It is known that there is one, and many techniques that extend the principle have been proposed. Among them, the technique formulated in the time-frequency domain has the advantage of being able to effectively utilize various assumptions about the sound source that hold in the time-frequency domain and the assumptions about the frequency response of the microphone array. For example, Independent Low-Rank Matrix Analysis (ILRMA) described in Non-Patent Document 1 models the power spectrogram of each sound source signal with the product of two non-negative matrices (low-rank non-negative matrix). It is based on the assumption that it is possible. However, the separation performance of this method is inevitably limited for sound sources that do not follow this assumption.

In recent years, attempts have been made to improve the separation accuracy by introducing deep neural network (DNN) into techniques based on signal processing such as ICA. The Multichannel Variational Autoencoder (MVAE) method described in Non-Patent Document 2 pre-learns and separates the generation model of the sound source spectrogram represented by the conditional VAE (CVAE). In some cases, the method estimates the CVAE decoder input together with the separation matrix, and achieves particularly high separation accuracy among the methods using the DNN. In this method, the parameters are updated so that the likelihood function rises at each iteration, thus ensuring convergence of the likelihood function to a stationary point, while updating the decoder input values using error backpropagation ( Backpropagation) is used, so there is a problem in that a high calculation cost is required.

The FastMVAE method described in Non-Patent Document 3 is a method proposed for the purpose of reducing the calculation cost of the MVAE method. In addition, by learning the discriminator distribution and encoder distribution that approximate the distribution of the sound source class and the posterior distribution of the latent variable, the decoder input value that maximizes the posterior distribution using the discriminator and encoder obtained by learning is a method of predicting Although this method can speed up the sound source separation algorithm compared to the MVAE method, the separation performance tends to decrease when the conditions during testing do not match those during training, such as in the case of unknown speakers or long reverberations. .

Daichi Kitamura, Nobutaka Ono, Hiroshi Sawada, Hirokazu Kameoka, and Hiroshi Saruwatari, “Determined blind source separation unifying independent vector analysis and nonnegative matrix factorization,” IEEE/ACM Transactions on Audio, Speech, and Language Processing, vol. 24, no. 9, pp. 1626-1641, Sep. 2016. Hirokazu Kameoka, Li Li, Shota Inoue, and Shoji Makino, “Supervised Determined Source Separation with Multichannel Variational Autoencoder,” Neural Computation, vol. 31, no. 9, pp.1891-1914, Sep. 2019. Li Li, Hirokazu Kameoka, Shota Inoue, and Shoji Makino, “FastMVAE: A Fast Optimization Algorithm for the Multichannel Variational Autoencoder Method,” IEEE Access, vol. 8, pp. 228740-228753, Dec. 2020.

In the MVAE method, parameters are updated so that the logarithmic likelihood increases in each iteration, so there is the advantage of guaranteeing the convergence of the logarithmic likelihood to a stationary point. The problem was that updating the parameters of the model required a great deal of computational cost.

On the other hand, the FastMVAE method of Non-Patent Document 3 realizes a significant speed-up of the sound source separation algorithm by predicting the update value of the parameter using the encoder and classifier that have been trained in advance together with the decoder. bottom. However, since the output values of the encoder and discriminator in the FastMVAE method are only approximations of the updated values in the steepest rising direction of the logarithmic likelihood related to the parameter, the FastMVAE method is inferior to the MVAE method in terms of sound source separation accuracy. has been confirmed experimentally.

The disclosed technology has been made in view of the above points, and is a signal analysis device, method, and method capable of accurately separating each component sound from a mixed signal in which each component sound is mixed, while suppressing calculation costs. and to provide programs.

A first aspect of the present disclosure is a signal analysis device, an encoder that estimates a latent vector sequence based on a spectrogram of each component sound and an attribute class indicating the attribute of the component sound, using a sound spectrogram as an input; A learning unit that learns a discriminator that uses the sound spectrogram as an input to identify an attribute class indicating the sound attribute, and a decoder that receives the latent vector sequence and the attribute class as input and generates the variance of the sound spectrogram. and the latent vector sequence estimated for each component sound separated by the separation matrix by the learned encoder, the separation matrix by the learned discriminator, with the observation signal mixed with each component sound as input. the attribute class identified for each component sound separated by, for each component sound, calculated from the variance of the component spectrogram generated by the learned decoder, and a scale parameter, the configuration A spectrogram of a sound, a scale parameter of the spectrogram of each component sound, a separation matrix for separating a mixed sound in which each component sound is mixed in the time-frequency domain into each component sound, and a signal obtained by separating the observed signal into each component sound. a parameter estimator for estimating the separation matrix and the scale parameter so as to optimize an objective function expressed using .

A second aspect of the present disclosure is a signal analysis method, in which a learning unit estimates a latent vector sequence based on a spectrogram for each component sound and an attribute class indicating the attribute of the component sound, with the spectrogram of the sound as input. an encoder that receives the spectrogram of sound as an input and identifies an attribute class indicating the attribute of the sound; and a decoder that receives the latent vector sequence and the attribute class as input and generates a variance of the spectrogram of the sound. The latent vector sequence estimated for each constituent sound separated by the separation matrix by the learned encoder, the learned from the attribute class identified by the classifier for each component sound separated by the separation matrix, the variance of the component sound spectrogram generated by the learned decoder for each component sound, and a scale parameter; The calculated spectrogram of the component sound, the scale parameter of the spectrogram of each component sound, the separation matrix for separating the mixed sound in which the component sounds are mixed in the time-frequency domain into each component sound, and the observed signal, estimating the separation matrix and the scale parameter so as to optimize an objective function represented using the signal separated into constituent sounds;

A third aspect of the present disclosure is a program for causing a computer to function as the signal analysis device of the first aspect.

According to the disclosed technology, it is possible to obtain the effect of being able to accurately separate each component sound from a mixed signal in which each component sound is mixed, while suppressing the calculation cost.

FIG. 2 is a conceptual diagram for explaining configurations of a teacher encoder and a teacher decoder according to this embodiment; FIG. 2 is a conceptual diagram for explaining configurations of an encoder, a discriminator, and a decoder according to this embodiment; FIG. 3 is a diagram showing a configuration example of an encoder and discriminator, and a configuration example of a decoder according to this embodiment; 1 is a schematic block diagram of an example of a computer that functions as a signal analysis device of this embodiment; FIG. 1 is a block diagram showing the configuration of a signal analysis device of this embodiment; FIG. 4 is a flowchart showing a learning processing routine in the signal analysis device of this embodiment; 4 is a flowchart showing a signal analysis processing routine in the signal analysis device of this embodiment; It is a figure which shows the arrangement|positioning of the microphone and the sound source in an experimental example. FIG. 4 is a diagram showing computation time in each iteration by the method of the present embodiment and the conventional method;

An example of embodiments of the technology disclosed herein will be described below with reference to the drawings. In each drawing, the same or equivalent components and portions are given the same reference numerals. Also, the dimensional ratios in the drawings are exaggerated for convenience of explanation, and may differ from the actual ratios.

<Overview of this embodiment>
First, an outline of this embodiment will be described.

In this embodiment, the encoder and classifier in the FastMVAE method are integrated as a single multitask NN (neural network) to achieve further speedup. In addition, knowledge distillation (KD) is performed on the multitask NN and decoder so that each output distribution is as close as possible to each output distribution of the encoder and decoder obtained by pre-learning in the MVAE method. , make each NN behave like the parameter update of the speech production model in the MVAE method, approaching the high separation accuracy of the MVAE method. These ideas realize high-speed and high-accuracy sound source separation even for unknown speakers compared to the conventional FastMVAE method.

（１）

（２）
とする。ただし、ここではＩ＝Ｊの優決定条件を考える。ここで（ ）^Ｔは転置を表し、ｆとｎはそれぞれ周波数と時間のインデックスである。 <Principle of this embodiment>
<Formulation of multi-channel sound source separation problem under over-determined condition>
Consider the case of observing signals coming from J sound sources with I microphones. Let x _i (f, n) and s _j (f, n) be the complex spectrograms of the observed signal of microphone i and the signal of sound source j, respectively. Also, a vector with these elements as

(1)

(2)
and However, here, the over-determination condition of I=J is considered. where ( ) ^T represents the transpose and f and n are the frequency and time indices, respectively.

（３）

（４）
を仮定することができる。ここで、Ｗ^Ｈ（ｆ）は分離行列を表し、（ ）^Ｈはエルミート転置である。 Under the condition of I=J, the instantaneous separation system

(3)

(4)
can be assumed. where W ^H (f) represents the separation matrix and ( ) ^H is the Hermitian transpose.

の複素正規分布

（５）
に従う確率変数とすると、各音源信号ｓ_ｊ（ｆ，ｎ）とｓ_ｊ’（ｆ，ｎ）、ｊ≠ｊ’が統計的に独立のときには、音源信号ｓ（ｆ，ｎ）は

（６）
に従う。 Under the assumption of the instantaneous mixing system described above, the complex spectrogram s _j (f, n) of sound source j has mean 0 and variance

complex normal distribution of

(5)
_, the sound source signal _s (f,n) is given by

(6)
obey.

（７）
に従う。 Here, V(f, n) is a diagonal matrix having v ₁ (f, n), . . . , v _I (f, n) as elements. From equations (3) and (6), the observed signal x is

(7)
obey.

（８）
となる。ここで、＝^ｃはパラメータに依存する項のみに関する等号を表す。音源パワースペクトログラムｖ_ｊ（ｆ，ｎ）に制約がない場合、式（８）は周波数ｆごとの項に分解されるため、式（８）に基づいて求めるＷで得られた分離信号のインデックスにはパーミュテーションの任意性が生じる。ｖ_ｊ（ｆ，ｎ）が周波数方向に構造的制約を持つ場合、その制約を活かすことでパーミュテーション整合と音源分離を同時解決するアプローチを導くことができる。ＩＬＲＭＡやＭＶＡＥ法がその例である。 Therefore, given the observed signal X={x(f, n)} _{f, n,} the separation matrix W={W(f)} _f and the power spectrogram V={v _j (f, n )} The log-likelihood function of _{j, f, n} is

(8)
becomes. where = ^c stands for equality only for the parameter dependent terms. If there are no constraints on the sound source power spectrogram v _j (f, n), Equation (8) is decomposed into terms for each frequency f. gives rise to permutation arbitrariness. If v _j (f, n) has structural constraints in the frequency direction, exploiting those constraints can lead to an approach that simultaneously solves permutation matching and sound source separation. Examples are the ILRMA and MVAE methods.

（９）

（１０）
と置く。ただし、分散σ^＊ _θ ^２（ｆ，ｎ；ｚ，ｃ）はデコーダネットワークの出力であり、ｇはパワースペクトログラムのスケールを表す変数である。 <Conventional technology 1: MVAE method>
In the MVAE method, the CVAE decoder distribution with the sound source class label as an auxiliary input is used as a complex spectrogram generation model for each sound source. Let S={s(f,n)} _f,n be the complex spectrogram of a sound source signal, and let c be the corresponding sound source class label. FIG. 1 shows a conceptual diagram of CVAE. CVAE is such that the encoder distribution q ^* _φ (z|S,c) and the decoder distribution p ^* _θ (S|z,c) are consistent, that is, q ^* _φ (z|S,c) and p ^* _θ ^The _NN ^parameter _φ , θ. Here, the decoder distribution of CVAE is a probability model isomorphic to the local Gaussian source model of equation (5)

(9)

(10)
and put. where the variance σ ^* _θ ² (f,n;z,c) is the output of the decoder network and g is the variable representing the scale of the power spectrogram.

（１１）
と仮定する。 On the other hand, the encoder distribution q ^* _φ (z|S, c) is the standard normal distribution

(11)
Assume that

（１２）
が最大となるように学習される。 where μ ^* _φ (S,c) and σ ^* _φ2 (S,c) are the encoder outputs ^. CVAE parameters θ and φ are obtained using training samples {S _m , _cm } ^M _{m = 1} of complex spectrograms of sound source signals of various classes.

(12)
is learned to maximize

は学習サンプルによる標本平均を表し、ＫＬ［・｜｜・］はＫｕｌｌｂａｃｋ－Ｌｅｉｂｌｅｒ（ＫＬ）ダイバージェンスである。以上により学習したデコーダ分布ｐ^＊ _θ（Ｓ｜ｚ，ｃ，ｇ）をＣＶＡＥ音源モデルと呼ぶ。ＣＶＡＥ音源モデルは、学習サンプルに含まれる様々なクラスの音源の複素スペクトログラムを表現可能な生成モデルとなっており、ｃを音源クラスのカテゴリカルな特徴を調整する役割と見なすことができ、ｚを、クラス内の変動を調整する役割を担った変数と見なすことができる。

represents the sample mean from the training samples, and KL[·||·] is the Kullback-Leibler (KL) divergence. The decoder distribution p ^* _θ (S|z, c, g) learned as described above is called a CVAE excitation model. The CVAE sound source model is a generative model that can express the complex spectrograms of the sound sources of various classes included in the training samples. , can be viewed as the variable responsible for adjusting for within-class variation.

（１３）

（１４）
を用いることができる。 Complex spectrogram S _j ={s _j (f, n)} of sound source j By expressing the generative model of _{f, n} by the decoder distribution with input of z _j , c _j , g _j , the parameters of the sound source model The likelihood function can be reduced to a likelihood function isomorphic to equation (8). Therefore, by repeatedly updating the separation matrix W, the CVAE sound source model parameter Ψ={z _j , c _j } _j , and the scale parameter G={g _j } _j so as to increase the likelihood function of equation (8), We can search for a stationary point in equation (8). Iterative projection method (IP) is used in the same way as ILRMA to update W that raises equation (8).

(13)

(14)
can be used.

であり、ｅ_ｊはＩ×Ｉの単位行列の第ｊ列ベクトルである。また式（８）を上昇させるΨの更新は誤差逆伝播法、Ｇの更新は

（１５）
により行うことができる。ただし、式（１５）はＷとΨが固定された下で式（８）を最大にする更新式である。以上よりＭＶＡＥの推論プロセスは以下のようにまとめられる。 however,

and e _j is the j-th column vector of the I×I identity matrix. Also, the update of Ψ that raises equation (8) is the error backpropagation method, and the update of G is

(15)
It can be done by However, equation (15) is an update equation that maximizes equation (8) under the condition that W and Ψ are fixed. From the above, the inference process of MVAE is summarized as follows.

1. θ and φ are learned based on equation (12).
2. Initialize W to the identity matrix and initialize Ψ.
3. The following steps a to c are repeated for each j.
(Step a) {w _j (f)} _{j and f} are updated by equations (13) and (14).
(Step b) Update Ψ _j ={z _j , c _j } by error backpropagation.
(Step c) Update g _j by equation (15).

<Conventional technology 2: Fast MVAE method>
In the MVAE method _, the parameters are _updated so that the log-likelihood increases in each iteration. The problem is that updating the parameters z _j and c _j that maximize _j |S _j ) by the error backpropagation method requires a great deal of computational cost. In the FastMVAE method of Non-Patent Document 3, the posterior distribution p _θ (z, c | S) is decomposed into a product of two conditional distributions such as p _θ (z | S, c) p _θ (c | S). , the distributions q ^* _φ (z|S, c) and r ^* _ψ (c|S) are represented by NNs so as to approximate each distribution, and pre-learned. As a result, the parameter search by the error backpropagation method in the MVAE method can be replaced by the forward calculation of each NN, enabling high-speed inference. However, the output values of the encoder q ^* _φ (z|S,c) and the discriminator r ^* _ψ (c|S) in the FastMVAE method are approximations of updated values in the direction of the steepest increase in the logarithmic likelihood of the relevant parameter. Therefore, it has been experimentally confirmed that the FastMVAE method is inferior to the MVAE method in terms of sound source separation accuracy.

<Method of this embodiment>
In the FastMVAE2 method used in this embodiment, first, it is assumed that the latent variable z and the sound source attribute class c are conditionally independent. Given a given spectrogram S, this corresponds to assuming that the speaker information c and the utterance content information z are independent. That is, it differs from the conventional method in that it is assumed that the posterior probability p _θ (z, c|S) can be expressed as p _θ (z|S)p _θ (c|S). If approximate distributions of these two conditional distributions are obtained, parameter search can be performed at high speed by forward calculation of NN, like the FastMVAE method.

<Chimera ACVAE sound source model>
ACVAE is an extended version of CVAE that was originally proposed for the purpose of applying it to speech conversion. In order to emphasize the influence of the input class label c on the decoder output, the mutual information I This method learns the encoder and decoder using (c, S|z) as a regularization term. Although it is not easy to directly optimize the criterion including I(c, S|z), a variational lower bound is introduced as in CVAE learning, and the variational lower bound and J(φ, θ) are combined. By raising the standard, we can indirectly increase the original standard. I(c,S|z) is given by the sum of the expected value of log p(c|S) and a constant, where p(c|S) is replaced by an appropriate auxiliary distribution r(c|S) It is a lower bound on I(c, S|z). By modeling this auxiliary distribution r(c|S) with the NN of the parameter _ψ , ψ can be learned together with φ and θ with the above lower bound as a reference. An auxiliary distribution represented by NN of the parameter ψ is denoted as r _ψ (c|S) and called a discriminator.

On the other hand, "ChimeraACVAE" of the present embodiment is a model in which the encoder and discriminator of ACVAE are expressed as an integrated multitask NN. In other words, it becomes a model in which the distributions q ⁺ _φ (z|S) and r ⁺ _ψ (c|S) of z and c are simultaneously inferred from the spectrogram S. FIG. 2 shows a conceptual diagram of ChimeraACVAE.

Since ChimeraACVAE has a structure that extracts the latent variable z only from the input spectrogram, it is possible to avoid the inference error of z due to the estimation error of the class label c. In addition, since it can be described with a more compact network structure than the conventional ACVAE model, it is expected that faster inference will become possible.

（１６）
および、相互情報量

（１７）
の和を含む。また、ラベル付き学習サンプル｛Ｓ_ｍ，ｃ_ｍ｝^Ｍ _ｍも学習に用いることができるため、学習データＳ_ｍと対応するクラスラベルｃ_ｍの負の交差エントロピー

（１８）
も、学習するための規準に含めることができる。ここまではモデル構造を除けば従来のＡＣＶＡＥと同様である。 The criterion for learning the ChimeraACVAE, ie the objective function to be maximized with respect to the NN parameters θ, φ, ψ, is the learning criterion for the CVAE

(16)
and mutual information

(17)
including the sum of In addition, since the labeled training samples {S _m , _cm } ^M _m can also be used for training, the negative cross entropy of the training data S _m and the corresponding class label c _m

(18)
can also be included in the criteria for learning. Up to this point, it is the same as the conventional ACVAE, except for the model structure.

However, ACVAE learned by the above criteria enables highly accurate inference when the test conditions and learning conditions match, but when they do not match, the estimated latent variables tend to deviate from the assumed distribution. , the generalization ability of the model was not sufficient.

（１９）

（２０） Therefore, in addition to the above criteria, the following criteria and knowledge distillation are used in learning ChimeraACVAE in order to improve the generalization ability of the model. In ChimeraACVAE, the spectrogram S can be reconstructed using the estimated class information. Since this process is also used at the time of inference, it is thought that learning a model using a criterion for evaluating the accuracy of the spectrogram S reconstructed in the same process will lead to an improvement in the accuracy at the time of inference. Therefore, the reconstruction criterion of equation (19) to be maximized and the class discrimination criterion of equation (20) are also included in the criteria for learning.

(19)

(20)

（２１）

（２２）
を用いても良い。ただし

である。 Alternatively, for simplicity of implementation, an approximation of

(21)

(22)
may be used. however

is.

Knowledge Distillation (KD) is a methodology for using a large NN trained with a large amount of data in advance as a teacher model, and inheriting that knowledge to a student model with a lightweight or another NN structure. It is known that a student model with a high Here, a CVAE model capable of achieving high separation accuracy even for unknown speakers is used as a teacher model, and the latent variable distribution q ^* _φ (z|S, c) learned by CVAE and the spectrogram generative model p ^* _θ Consider inheriting the knowledge of (S|z, c) to ChimeraACVAE, which is a student model. Specifically ^{, the} normal distribution N ⁽ 0 _, diag ₍ σ ^* _θ2 ⁽ Let z, c))) be the output distribution of the student model q ⁺ _φ (z | S) and the prior distribution of the normal distribution using the decoder output σ ⁺ _φ ² , and learn so that the output of the student model approaches the prior distribution Let However, the degree of divergence between the distributions of the teacher model and the student model is measured using KL divergence, and is used as a knowledge distillation criterion as shown in equations (23) to (25).

（２３）

（２４）

（２５）

(23)

(24)

(25)

（２６）
となる。ここで、λは非負値であり、各規準の重み係数である。図２に知識蒸留を用いたＣｈｉｍｅｒａＡＣＶＡＥの学習の概念図を示す。 From the above, the criteria that should be maximized when learning ChimeraACVAE are

(26)
becomes. where λ is a non-negative value and is the weighting factor for each criterion. FIG. 2 shows a conceptual diagram of ChimeraACVAE learning using knowledge distillation.

FIG. 3 shows an example network structure of ChimeraACVAE. Each layer of the encoder and discriminator is composed of a convolution layer, Layer Normalization (LN) and Sigmoid Linear Unit (SiLU), and each layer of the decoder is composed of a deconvolution layer, LN and SiLU. Here, by using LN, it is possible to avoid inconsistency in normalization calculation methods during learning and inference. SiLU is a data-driven activation function that gate-controls information passed between layers, similar to the Gated Linear Unit (GLU) used in the CVA sound source model, and can halve the number of GLU parameters.

<FastMVAE2 method: fast inference algorithm>
By using the encoder and discriminator learned by ChimeraACVAE, the maximization step of p _θ (z _j , c _j |S _j ) in the conventional MVAE method is replaced by q ⁺ _φ (z _j |S _j ) and r ⁺ _ψ ( c _j |S _j ). Therefore, the following algorithm is obtained. This is called FastMVAE2 method.

1. θ, φ, and ψ are learned using equation (26) as a criterion for learning.
2. Initialize W to be the identity matrix.
3. Repeat steps a to c below for each j.
(Step a) {w _j (f)} _{j and f} are updated by equations (13) and (14).
(Step b) The spectrogram separated using W is input, and _zj and _cj are updated to the mean of the Gaussian distribution output from the encoder and the output value (continuous value vector) of the discriminator, respectively.
(Step c) Update g _j by equation (15).

<Configuration of signal analysis apparatus according to the present embodiment>
FIG. 4 is a block diagram showing the hardware configuration of the signal analysis device 100 of this embodiment.

As shown in FIG. 4, the signal analysis apparatus 100 includes a CPU (Central Processing Unit) 11, a ROM (Read Only Memory) 12, a RAM (Random Access Memory) 13, a storage 14, an input unit 15, a display unit 16, and a communication interface. (I/F) 17. Each component is communicatively connected to each other via a bus 19 .

The CPU 11 is a central processing unit that executes various programs and controls each section. That is, the CPU 11 reads a program from the ROM 12 or the storage 14 and executes the program using the RAM 13 as a work area. The CPU 11 performs control of each configuration and various arithmetic processing according to programs stored in the ROM 12 or the storage 14 . In this embodiment, the ROM 12 or the storage 14 stores a learning program for executing learning processing and a signal analysis program for executing signal analysis processing. The learning program and the signal analysis program may be one program, or may be a program group composed of a plurality of programs or modules.

The ROM 12 stores various programs and various data. The RAM 13 temporarily stores programs or data as a work area. The storage 14 is configured by a HDD (Hard Disk Drive) or SSD (Solid State Drive), and stores various programs including an operating system and various data.

The input unit 15 includes a pointing device such as a mouse and a keyboard, and is used for various inputs.

As learning data, the input unit 15 receives, for each of a plurality of constituent sounds, the time-series data of the signal of the constituent sound and the attribute class indicating the attribute of the signal of the constituent sound. The input unit 15 also receives time-series data of a mixed signal (hereinafter referred to as an observed signal) in which a plurality of constituent sounds are mixed as data to be analyzed. The attribute class indicating the attribute of the component sound signal may be given manually. Further, the attribute of the constituent sound signal is, for example, gender, adult/child, speaker ID, and the like.

The display unit 16 is, for example, a liquid crystal display, and displays various information. The display unit 16 may employ a touch panel system and function as the input unit 15 .

The communication interface 17 is an interface for communicating with other devices, and uses standards such as Ethernet (registered trademark), FDDI, and Wi-Fi (registered trademark), for example.

Next, the functional configuration of the signal analysis device 100 will be described. FIG. 5 is a block diagram showing an example of the functional configuration of the signal analysis device 100. As shown in FIG.

As shown in FIG. 5, the signal analysis apparatus 100 functionally includes a time-frequency expansion unit 24, a teacher learning unit 30, a learning unit 32, a sound source signal model storage unit 34, a parameter estimation unit 36, and an output unit 38 .

The time-frequency expansion unit 24 calculates, for each component sound, a power spectrogram representing the spectrum at each time based on the time-series data of the signal of the component sound. Also, the time-frequency expansion unit 24 calculates a power spectrogram representing the spectrum at each time based on the time-series data of the observed signal. In this embodiment, time-frequency expansion such as short-time Fourier transform or wavelet transform is performed.

Based on the spectrogram and attribute class of each constituent sound input as learning data, the teacher learning unit 30 includes a teacher encoder for estimating a latent vector sequence using the sound spectrogram and attribute class as inputs, and a latent vector sequence and attribute A teacher decoder that generates the variance of the sound spectrogram using the class as an input is learned and stored in the sound source signal model storage unit 34 .

Specifically, for each constituent sound, the teacher learning unit 30 calculates the error between the power spectrogram generated by the teacher decoder and the power spectrogram in the signal of the original constituent sound, and the latent value estimated by the teacher encoder. Train the teacher encoder and teacher decoder so as to maximize the value of the objective function of the above equation (12), which is expressed using the distance between the vector sequence and the latent vector sequence in the signal of the original constituent sound. and stored in the sound source signal model storage unit 34 . Here, each of the teacher encoder and teacher decoder is configured using a convolutional network or a recursive network.

Based on the spectrogram and attribute class for each component sound input as learning data, the learning unit 32 identifies an encoder that estimates a latent vector sequence with the spectrogram of the sound as input, and an attribute class with the spectrogram of the sound as input. A classifier is trained and a decoder that takes the latent vector sequences and attribute classes as input and generates the variance of the sound spectrogram.

Specifically, the learning unit 32 receives the encoder output and the learning criterion for evaluating the decoder output, the decoder output and the mutual information of the attribute class, the encoder output and the discriminator output, and the and a reconstruction criterion for evaluating a spectrogram generated using the output of the decoder, and the output of the classifier whose input is the spectrogram generated using the output of the decoder whose input is the output of the encoder and the output of the classifier and a knowledge distillation criterion for matching the output of the encoder and the output of the learned teacher encoder, and the output of the decoder and the output of the learned teacher decoder. Encoders, discriminators, and decoders are trained so as to maximize the criterion of the above equation (26), and stored in the sound source signal model storage unit 34 . Here, the encoder and discriminator are an integral neural network, and the encoder and discriminator share some layers. Also, each of the encoder, discriminator, and decoder is constructed using a convolutional network or a recursive network.

The parameter estimating unit 36 receives as input an observed signal in which each component sound is mixed based on the power spectrogram of the observed signal, and a latent vector sequence estimated for each component sound separated by a separation matrix by a learned encoder, from the attribute class identified for each component sound separated by the separation matrix by the learned classifier, the variance of the component sound spectrogram generated by the learned decoder for each component sound, and the scale parameter. The calculated spectrogram of the constituent sounds, the scale parameter of the spectrogram of each constituent sound, the separation matrix for separating the mixed sound in which the constituent sounds are mixed in the time-frequency domain into each constituent sound, and the observed signal for each constituent A separation matrix and a scale parameter are estimated so as to optimize the objective function of the above equation (8) expressed using the sound-separated signal.

Specifically, the parameter estimation unit 36 includes an initial value setting unit 40 , a separation matrix update unit 42 , a latent variable class update unit 44 , a scale parameter update unit 46 and a convergence determination unit 48 .

The initial value setting unit 40 sets initial values for the separation matrix, the latent vector sequence of each component sound, the attribute class of each component sound, and the scale parameter of each component sound.

The separation matrix updating unit 42 updates the power spectrogram of the observed signal, the latent vector sequence of each component sound that was last updated or has an initial value set, the attribute class of each component sound, the scale parameter of each component sound, and Based on the separation matrix, the separation matrix is updated according to the above formulas (13) and (14) so as to increase the objective function shown in the above formula (8).

The latent variable class updating unit 44 inputs the power spectrogram of each constituent sound obtained using the observed signal and the separation matrix, and updates the latent vector sequence of each constituent sound obtained using the average of the Gaussian distribution of the output of the encoder. In addition to updating, the attribute class of each constituent sound is updated using the output of the discriminator whose input is the power spectrogram of each constituent sound obtained using the observed signal and separation matrix.

Based on the power spectrogram of the observed signal and the updated latent vector sequence of each component sound, attribute class of each component sound, scale parameter of each component sound, and separation matrix, the scale parameter update unit 46 calculates the above formula The scale parameter is updated according to the above equation (15) so as to increase the objective function shown in (8).

The convergence determination unit 48 determines whether or not the convergence condition is satisfied, and until the convergence condition is satisfied, the update processing in the separation matrix update unit 42, the update processing in the latent variable class update unit 44, and the scale parameter update unit 46 Repeat the update process.

As a convergence condition, for example, it is possible to use that the number of iterations reaches the upper limit number of times. Alternatively, as the convergence condition, it is possible to use that the difference between the value of the objective function of the above equation (8) and the value of the previous objective function is less than or equal to a predetermined threshold.

Based on the latent vector sequence of each component sound, the attribute class of each component sound, and the scale parameter of each component sound obtained by the parameter estimation unit 36, the output unit 38 outputs each component sound generated using the decoder. A power spectrogram is obtained, and from the power spectrogram of each component sound, a signal of each component sound is generated and output.

<Action of the Signal Analysis Apparatus According to the Present Embodiment>
Next, the operation of the signal analysis device 100 according to this embodiment will be described.

FIG. 6 is a flowchart showing the flow of learning processing by the signal analysis device 100. As shown in FIG. The learning process is performed by the CPU 11 reading the learning program from the ROM 12 or the storage 14, developing it in the RAM 13, and executing it. In addition, for each of a plurality of component sounds, time-series data of the signal of the component sound and an attribute class indicating the attribute of the signal of the component sound are input to the signal analysis apparatus 100 as learning data.

First, in step S100, the CPU 11, as the time-frequency expansion unit 24, calculates a power spectrogram representing the spectrum at each time for each component sound based on the time-series data of the signal of the component sound.

In the next step S102, the CPU 11 serves as the teacher learning unit 30, based on the spectrogram and attribute class of each component sound input as learning data, and the teacher who uses the spectrogram and attribute class of the sound as input and estimates a latent vector sequence. We train an encoder for , and a teacher decoder that takes the latent vector sequences and attribute classes as input and generates the variance of the sound spectrogram.

In step S104, the CPU 11, as the learning unit 32, uses an encoder for estimating a latent vector sequence based on the spectrogram and attribute class of each constituent sound input as learning data, and an encoder for estimating a latent vector sequence based on the spectrogram of the sound and the spectrogram of the sound. A discriminator that identifies an attribute class as input, and a decoder that generates the variance of a sound spectrogram with the latent vector sequence and attribute class as input, and the parameters of the learned encoder, discriminator, and decoder are used as the sound source signal. Stored in the model storage unit 34 .

FIG. 7 is a flowchart showing the flow of signal analysis processing by the signal analysis apparatus 100. As shown in FIG. The CPU 11 reads out the signal analysis program from the ROM 12 or the storage 14, develops it in the RAM 13, and executes it, thereby performing signal analysis processing. In addition, time-series data of an observation signal in which each component sound is mixed is input to the signal analysis apparatus 100 .

First, in step S120, the CPU 11, as the time-frequency expansion unit 24, calculates a power spectrogram representing the spectrum at each time based on the time-series data of the observed signal.

In step S122, the CPU 11, as the initial value setting unit 40, sets initial values for the separation matrix, the latent vector sequence of each component sound, the attribute class of each component sound, and the scale parameter of each component sound.

In step S124, the CPU 11, as the separating matrix updating unit 42, performs the power spectrogram of the observed signal calculated in step S120, the latent vector sequence of each component sound that was last updated or the initial value was set, each Based on the component sound attribute class, the scale parameter of each component sound, and the separation matrix, the separation matrix is generated according to the above formulas (13) and (14) so as to increase the objective function shown in the above formula (8). Update.

In step S126, the CPU 11, as the latent variable class updating unit 44, converts the latent vector sequence of each constituent sound into a Gaussian distribution of the output of the encoder with the input of the power spectrogram of each constituent sound obtained using the observation signal and separation matrix. updated to the latent vector sequence of each constituent sound obtained using the average of , and using the output of the discriminator with the input of the power spectrogram of each constituent sound obtained using the observed signal and the separation matrix, each configuration Update the sound attribute class.

In step S128, the CPU 11, as the scale parameter updating unit 46, updates the power spectrogram of the observed signal calculated in step S120, the updated latent vector sequence of each constituent sound, the attribute class of each constituent sound, and each constituent sound. and the separation matrix, the scale parameters are updated according to the above equation (15) so as to increase the objective function shown in the above equation (8).

Next, in step S130, it is determined whether or not the convergence condition is satisfied. When the convergence condition is satisfied, the process proceeds to step S132, and when the convergence condition is not satisfied, the process proceeds to step S124, and the processes of steps S124 to S128 are repeated.

In step S132, based on the latent vector sequence of each component sound, the attribute class of each component sound, and the scale parameter of each component sound finally updated in steps S124 to S128, each component sound is calculated using a decoder. is generated, a signal of each component sound is generated from the power spectrogram of each component sound, and output from the output unit 38, and the signal analysis processing is completed.

<Experimental results>
In order to verify the sound source separation performance of the technique of this embodiment, we conducted speaker-dependent separation experiments using the Voice Conversion Challenge (VCC) 2018 speech database and arbitrary speaker separation experiments using the WSJ0 speech database. The comparison target is ILRMA described in Non-Patent Document 1, the MVAE method described in Non-Patent Document 2, and the FastMVAE method described in Non-Patent Document 3, and the evaluation criteria are source-to-distortion ratio (SDR) and source-to-interference ratio. (SIR) and sources-to-artifacts ratio (SAR) were used. In all methods, the separation matrix W(f) was initialized to a unit matrix and updated 60 times.

The base number of ILRMA was set to 2. Table 1 shows the number of parameters for each model.

ChimeraACVAE was able to reduce the number of parameters by 40% compared to ACVAE. Table 2 shows the experimental results.

Under any conditions, the method of the present embodiment (FastMVAE2 method) exhibited higher separation performance than the ILRMA and FastMVAE methods, and greatly narrowed the difference from the MVAE method.

In order to evaluate the separation performance and computational time of each method for the number of sound sources greater than two, we extracted mixed signals with the number of sound sources {2, 3, 6, 9} from the WSJ0 speech database using utterances of 18 speakers. Created. Impulse responses were generated by the mirror image method with a wall reflection coefficient of 0.2. Figure 8 shows the arrangement of microphones and sound sources. Ten mixed signals were generated for each condition. All processing was computed using an Intel(R) Xeon(R) Gold 6130 CPU @ 2.10 GHz and a Tesla V100 GPU. Table 3 shows the average value of SDR under each condition.

Also, FIG. 9 shows the calculation time for each iteration of each method. Performance improvement was confirmed in the method of this embodiment (FastMVAE2, FastMVAE2_CPU, FastMVAE2_GPU). It was also confirmed that the method of this embodiment can achieve separation in a calculation time equivalent to that of ILRMA when there are three or fewer sound sources, and can achieve separation in a shorter calculation time than ILRMA when there are three or more sound sources.

As described above, the signal analysis apparatus according to the present embodiment includes an encoder for estimating a latent vector sequence using a sound spectrogram as an input, and A discriminator for identifying an attribute class indicating the attribute of the sound and a decoder for generating the variance of the spectrogram of the sound with the latent vector sequence and the attribute class as inputs are trained. Then, the signal analysis device receives as input an observation signal in which each component sound is mixed, a latent vector sequence estimated for each component sound separated by a separation matrix by a learned encoder, a separation matrix by a learned discriminator, an attribute class identified for each component note separated by, for each component note, the component note spectrogram calculated from the variance of the component note spectrogram generated by the learned decoder and the scale parameter, It is expressed using the scale parameter of the spectrogram of each component sound, the separation matrix for separating the mixed sound into each component sound in the time-frequency domain, and the signal obtained by separating the observed signal into each component sound. Estimate the separation matrix and scale parameters to optimize the objective function for As a result, it is possible to accurately separate each component sound from a mixed signal in which each component sound is mixed, while suppressing the calculation cost.

<Modification>
The present invention is not limited to the above-described embodiments, and various modifications and applications are possible without departing from the gist of the present invention.

For example, the case of calculating the power spectrogram of the observed signal and the power spectrogram of the constituent sounds has been described as an example, but the present invention is not limited to this, and the amplitude spectrogram of the observed signal and the amplitude spectrogram of the constituent sounds are calculated. good too. In this case, based on the amplitude spectrogram and attribute class of each constituent sound, the learning unit 32 includes an encoder for estimating the latent vector sequence with the amplitude spectrogram of the sound as input, and an attribute class with the amplitude spectrogram of the sound as input. A classifier that discriminates and a decoder that takes the latent vector sequence and the attribute class as input and generates the variance of the sound amplitude spectrogram is trained. In addition, the parameter estimating unit 36 receives the observed signal as an input, and the latent vector sequence estimated by the learned encoder, the attribute class identified by the learned classifier, and the learned decoder for each constituent sound. Amplitude spectrogram of constituent sounds, scale parameter of amplitude spectrogram of each constituent sound, scale parameter of amplitude spectrogram of each constituent sound, mixed sound in which each constituent sound is mixed in time-frequency domain A separation matrix and a scale parameter are estimated so as to optimize an objective function expressed using a separation matrix for separating each component sound and a signal obtained by separating an observed signal into each component sound.

Also, since the order of parameters to be updated is arbitrary, it is not limited to the order in the above embodiment.

Further, the various processes executed by the CPU by reading the software (program) in each of the above-described embodiments may be executed by various processors other than the CPU. The processor in this case is a PLD (Programmable Logic Device) whose circuit configuration can be changed after manufacturing such as an FPGA (Field-Programmable Gate Array), and an ASIC (Application Specific Integrated Circuit) for executing specific processing. A dedicated electric circuit or the like, which is a processor having a specially designed circuit configuration, is exemplified. Also, the learning process and signal analysis process may be performed by one of these various processors, or by a combination of two or more processors of the same or different type (e.g., multiple FPGAs, and CPU and FPGA). , etc.). More specifically, the hardware structure of these various processors is an electric circuit in which circuit elements such as semiconductor elements are combined.

Further, in each of the above-described embodiments, the learning program and the signal analysis program have been pre-stored (installed) in the storage 14, but the present invention is not limited to this. The program is stored in non-transitory storage media such as CD-ROM (Compact Disk Read Only Memory), DVD-ROM (Digital Versatile Disk Read Only Memory), and USB (Universal Serial Bus) memory. may be provided in the form Also, the program may be downloaded from an external device via a network.

The following additional remarks are disclosed regarding the above embodiments.

(Appendix 1)
memory;
at least one processor connected to the memory;
including
The processor
An encoder for estimating a latent vector sequence with the spectrogram of a sound as an input, based on a spectrogram for each constituent sound and an attribute class indicating the attribute of the constituent sound; and an attribute class indicating the attribute of the sound with the spectrogram of the sound as an input. and a decoder that generates the variance of the sound spectrogram with the latent vector sequence and the attribute class as inputs,
The latent vector sequence estimated for each component sound separated by the separation matrix by the learned encoder using an observed signal mixed with each component sound as an input, and separated by the separation matrix by the learned discriminator. the attribute class identified for each component sound, the variance of the component sound spectrogram generated by the learned decoder for each component sound, and a scale parameter of the component sound, calculated from the Using a spectrogram, a scale parameter of the spectrogram of each component sound, a separation matrix for separating a mixed sound in which each component sound is mixed in the time-frequency domain into each component sound, and a signal obtained by separating the observed signal into each component sound A signal analysis device configured to estimate the separation matrix and the scale parameter so as to optimize an objective function represented by .

(Appendix 2)
A non-transitory storage medium storing a program executable by a computer to perform signal analysis processing,
The signal analysis processing includes
An encoder for estimating a latent vector sequence with the spectrogram of a sound as an input, based on a spectrogram for each constituent sound and an attribute class indicating the attribute of the constituent sound; and an attribute class indicating the attribute of the sound with the spectrogram of the sound as an input. and a decoder that generates the variance of the sound spectrogram with the latent vector sequence and the attribute class as inputs,
The latent vector sequence estimated for each component sound separated by the separation matrix by the learned encoder using an observed signal mixed with each component sound as an input, and separated by the separation matrix by the learned discriminator. the attribute class identified for each component sound, the variance of the component sound spectrogram generated by the learned decoder for each component sound, and a scale parameter of the component sound, calculated from the Using a spectrogram, a scale parameter of the spectrogram of each component sound, a separation matrix for separating a mixed sound in which each component sound is mixed in the time-frequency domain into each component sound, and a signal obtained by separating the observed signal into each component sound A non-transitory storage medium for estimating the separation matrix and the scale parameter so as to optimize an objective function represented by .

11 CPUs
14 storage 15 input unit 16 display unit 24 time-frequency expansion unit 30 teacher learning unit 32 learning unit 34 sound source signal model storage unit 36 parameter estimation unit 38 output unit 40 initial value setting unit 42 separation matrix update unit 44 latent variable class update unit 46 Scale parameter update unit 48 Convergence determination unit 100 Signal analysis device

Claims (6)

An encoder for estimating a latent vector sequence with the spectrogram of a sound as an input, based on a spectrogram for each constituent sound and an attribute class indicating the attribute of the constituent sound; and an attribute class indicating the attribute of the sound with the spectrogram of the sound as an input. a learning unit that learns a discriminator that identifies and a decoder that generates the variance of the sound spectrogram with the latent vector sequence and the attribute class as inputs;
The latent vector sequence estimated for each component sound separated by the separation matrix by the learned encoder using an observed signal mixed with each component sound as an input, and separated by the separation matrix by the learned discriminator. the attribute class identified for each component sound, the variance of the component sound spectrogram generated by the learned decoder for each component sound, and a scale parameter of the component sound, calculated from the Using a spectrogram, a scale parameter of the spectrogram of each component sound, a separation matrix for separating a mixed sound in which each component sound is mixed in the time-frequency domain into each component sound, and a signal obtained by separating the observed signal into each component sound a parameter estimator that estimates the separation matrix and the scale parameter so as to optimize the objective function represented by
signal analysis equipment including A teacher encoder for estimating a latent vector sequence with the sound spectrogram and the attribute class as input, based on the spectrogram and the attribute class for each constituent sound, and the sound spectrogram with the latent vector sequence and the attribute class as input. further comprising a supervised learning unit for training a supervised decoder that produces the variance of
The learning unit is arranged so that the output of the encoder and the learned output of the teacher encoder correspond, and the output of the decoder and the learned output of the teacher decoder correspond. , the encoder and the decoder are trained. 3. The signal analysis apparatus according to claim 1, wherein said encoder and said discriminator are an integral neural network, and said encoder and said discriminator share some layers. The learning unit
a learning criterion for evaluating the output of the encoder and the output of the decoder;
mutual information of the output of the decoder and the attribute class;
a reconstruction criterion for evaluating the spectrogram generated using the output of the decoder with the output of the encoder and the output of the discriminator as inputs;
A class discrimination criterion for evaluating the output of the discriminator whose input is the spectrogram generated using the output of the decoder whose input is the output of the encoder and the output of the discriminator;
The signal analysis apparatus according to any one of claims 1 to 3, wherein the encoder, the discriminator, and the decoder are trained so as to optimize a criterion including An encoder for estimating a latent vector sequence with the sound spectrogram as input, and the sound attribute with the sound spectrogram as input, based on the spectrogram for each component sound and the attribute class indicating the attribute of the component sound. and a decoder that generates the variance of the sound spectrogram with the latent vector sequence and the attribute class as inputs,
A parameter estimating unit receives as input an observed signal in which each component sound is mixed, the latent vector sequence estimated for each component sound separated by the separation matrix by the learned encoder, and the learned classifier calculated from the attribute class identified for each component sound separated by the separation matrix, the variance of the component sound spectrogram generated by the learned decoder for each component sound, and a scale parameter , a spectrogram of the constituent sounds, a scale parameter of the spectrogram of each constituent sound, a separation matrix for separating the mixed sound into each constituent sound in which each constituent sound is mixed in the time-frequency domain, and the observation signal to each constituent sound A signal analysis method for estimating the separation matrix and the scale parameter so as to optimize an objective function represented using separated signals. A signal analysis program for causing a computer to function as each part of the signal analysis apparatus according to any one of claims 1 to 4.

Images