JP6992873B2 - Sound source separation device, sound source separation method and program - Google Patents

Description

The present disclosure relates to a non-temporary computer-readable medium in which a sound source separation device, a sound source separation method, and a program are stored.

A technique for separating a mixed signal including a plurality of sound source signals such as voice emitted by a plurality of speakers into individual sound source signals is being studied. In relation to such a technique, Patent Document 1 describes an acoustic process for storing a first base matrix that is a sound source learned in advance and that shows the acoustic characteristics of the first sound source that does not include the sound of the second sound source. The device is disclosed. The sound processing device uses a non-negative matrix factorization using the first basis matrix to obtain the first from the observation matrix showing the time series of the spectrum of the acoustic signal showing the mixed sound of the sound of the first sound source and the sound of the second sound source. A two-base matrix and a second coefficient matrix are generated. Then, the acoustic processing apparatus generates the acoustic signals of the first sound source and the second sound source by using the acoustic signal corresponding to the first basis matrix and the first coefficient matrix and the second basis matrix and the second coefficient matrix. ..

Further, Non-Patent Document 1 is disclosed in relation to the above technique. Non-Patent Document 1 discloses a sound source separation method in which a voice emitted by a certain speaker is used as a sound source and the voice emitted by a plurality of speakers at the same time is separated into the voices of individual speakers. The sound source separation method receives a single channel mixed signal, converts the received mixed signal into a time-frequency representation (spectrogram), and extracts a feature vector from each time-frequency bin using a deep neural network. Then, by clustering the extracted feature vectors, the time-frequency bins were classified into the same number of clusters as the target number of sound sources (number of speakers), and each cluster was reconstructed from the time-frequency bins contained therein. Create a sound source signal for each speaker from the spectrogram.

The deep neural network disclosed in Non-Patent Document 1 is prepared by prior training (learning). The data used for learning is a collection of many sound source signals spoken by various speakers. These are all independent sound source signals, not mixed signals spoken by multiple speakers at the same time. In Non-Patent Document 1, first, a short-time Fourier transform is performed on the training data, and each sound source signal is converted into a spectrogram. Next, the spectrograms of the two sound source signals are superimposed to generate a spectrogram of the mixed signal, and for each time-frequency bin, it is determined which speaker belongs to and a speaker label is given. Here, the speaker label is determined from the amplitude of each original sound source signal. That is, it is assumed that the time-frequency bin belongs to the speaker having the larger amplitude. Subsequently, the feature vector is extracted from each time-frequency bin using the deep neural network obtained at that time. Next, a loss function for calculating a scale for measuring consistency with the speaker label is calculated, and the parameters of the deep neural network for feature extraction are updated so that the loss function is reduced.

Japanese Unexamined Patent Publication No. 2013-33196

JR Hershey, Z. Chen, J. Le Roux, and S. Watanabe, “Deep clustering: Discriminative embeddings for segmentation and separation,” in Proc. Of the 41st IEEE International Conference on Acoustics, Speech, and Signal Processing (ICASSP 2016) , Mar. 2016.

The technique disclosed in Patent Document 1 uses a first sound source as a reference learned in advance to separate individual sound sources from a mixed signal in which a plurality of sound sources are mixed. Further, the technique disclosed in Non-Patent Document 1 separates individual sound sources by using a mixed signal artificially generated by superimposing two or more different sound source signals. That is, the training data used in Patent Document 1 and Non-Patent Document 1 are different from the actually observed mixed signals.

Here, in an actual environment, noise and reverberation are usually present, so that the actually observed mixed signal is different from a simple superposition of spectrograms of individual sound source signals. The reason is that the amplitude ratio when superimposing the sound source signal depends on the positional relationship between the microphone and the speaker, so that the actually observed mixed signal is not always constant in all observations. .. In addition, since there is interaction between speakers in conversation, the actually observed mixed signal is not always constant in time. Therefore, if the data used for learning is not the actually observed mixed signal, a mismatch occurs between the data used for learning and the actually observed mixed signal. If a mismatch occurs between the data used for learning and the actually observed mixed signal, proper learning cannot be performed. Therefore, if proper learning is not performed, it is not possible to accurately separate individual sound source signals from the mixed signal. That is, since the techniques disclosed in Patent Document 1 and Non-Patent Document 1 described above do not use the actually observed mixed signal as learning data, the sound source is separated accurately from the actually observed mixed signal. I can't.

An object of the present disclosure is to solve such a problem, and a sound source separation device, a sound source separation method, and a program capable of accurately separating individual sound source signals from a mixed signal are stored. It is to provide a non-temporary computer-readable medium.

The sound source separation device according to the present disclosure is a feature vector using a feature extractor to which the parameters used for feature extraction are applied for each time frequency bin in a spectrogram obtained by converting a mixed signal in which a plurality of sound source signals are mixed. For each classified cluster using a feature extraction means for extracting, a clustering means for classifying the extracted feature vector into a plurality of clusters, and a time frequency bin included in each of the classified plurality of clusters. A separation means for generating a sound source signal and a parameter updating means for updating the parameters based on the learning mixed signal including the observed mixed signal are provided.

Further, the sound source separation method according to the present disclosure uses a feature extractor to which the parameters used for feature extraction are applied for each time frequency bin in a spectrogram obtained by converting a mixed signal in which a plurality of sound source signals are mixed. Using the extraction of the feature vector, the classification of the extracted feature vector into a plurality of clusters, and the time frequency bin included in each of the classified clusters, the sound source is used for each classified cluster. It is a sound source separation method including generating a signal and updating the parameter based on the learning mixed signal including the observed mixed signal.

Further, the non-temporary computer-readable medium according to the present disclosure is a feature extraction in which parameters used for feature extraction are applied for each time frequency bin in a spectrogram obtained by converting a mixed signal in which a plurality of sound source signals are mixed. The feature vectors were extracted using a device, the extracted feature vectors were classified into a plurality of clusters, and the time frequency bins included in each of the classified clusters were used for classification. A non-temporary computer-readable device that contains a program that causes a computer to generate a sound source signal for each cluster and update the parameters based on the learning mixed signal including the observed mixed signal. It is a medium.

According to the present disclosure, it is possible to provide a non-temporary computer-readable medium containing a sound source separation device, a sound source separation method, and a program capable of accurately separating individual sound source signals from a mixed signal.

It is a figure which shows the outline of the sound source separation apparatus 1 which concerns on embodiment of this disclosure. It is a block diagram which shows the structural example of the sound source separation apparatus which concerns on Embodiment 1. FIG. It is a figure explaining the sound source label in a related technique. It is a figure explaining the sound source label in Embodiment 1. FIG. It is a flowchart which shows the operation example of the sound source separation apparatus which concerns on Embodiment 1. FIG. It is a flowchart which shows the operation example of the sound source separation apparatus which concerns on Embodiment 1. FIG. It is a flowchart which shows the operation example of the sound source separation apparatus which concerns on Embodiment 1. FIG. It is a block diagram which shows the structural example of the sound source separation apparatus which concerns on Embodiment 2. It is a figure which shows the structural example of the sound source separation apparatus which concerns on other embodiment.

(Outline of embodiment)
Prior to the description of the embodiments of the present disclosure, an outline of the embodiments will be described. FIG. 1 is a diagram showing an outline of the sound source separation device 1 according to the embodiment of the present disclosure.

The sound source separation device 1 includes a feature extraction unit 2 that functions as a feature extraction means, a clustering unit 3 that functions as a clustering means, a separation unit 4 that functions as a separation means, and a parameter update unit 5 that functions as a parameter update means. To be equipped.

The feature extraction unit 2 extracts a feature vector for each time-frequency bin using a feature extractor to which the parameters used for feature extraction are applied in a spectrogram obtained by converting a mixed signal in which a plurality of sound source signals are mixed. ..

The clustering unit 3 classifies the extracted feature vector into a plurality of clusters.
The separation unit 4 generates a sound source signal for each of the classified clusters by using the time frequency bin included in each of the plurality of classified clusters.
The parameter updating unit 5 updates the parameters based on the learning mixed signal including the observed mixed signal.

The sound source separation device 1 according to the embodiment uses a actually observed mixed signal as the learning mixed signal, not a mixed signal in which a plurality of sound source signals are artificially superimposed. Therefore, by using the sound source separation device 1, it is possible to acquire the optimum feature vector for the mixed signal separated into the individual sound source signals, so that the mixed signal can be accurately separated into the individual sound source signals. It becomes. Therefore, by using the sound source separation device 1 according to the embodiment, it is possible to accurately separate individual sound source signals from the mixed signal.

Even if the sound source separation method in the sound source separation device 1 is used, it is possible to accurately separate individual sound source signals from the mixed signal. Further, even if a program capable of executing the sound source separation method is used, it is possible to accurately separate individual sound source signals from the mixed signal.

(Embodiment 1)
Hereinafter, embodiments of the present disclosure will be described with reference to the drawings.
<Sound source separation device configuration example>
First, a configuration example of the sound source separation device 10 according to the first embodiment will be described with reference to FIG. FIG. 2 is a configuration diagram showing a configuration example of the sound source separation device according to the first embodiment.

The sound source separation device 10 may be, for example, a computer such as a server device or a personal computer device. The sound source separation device 10 includes a learning mixed signal storage unit 11, a learning label data storage unit 12, a feature extractor learning unit 13, and a sound source separation unit 14. The learning mixed signal storage unit 11, the learning label data storage unit 12, the feature extractor learning unit 13, and the sound source separation unit 14 are a learning mixed signal storage unit, a learning label data storage unit, and a feature extractor learning unit, respectively. And functions as a sound source separation means.

The learning mixed signal storage unit 11 stores the mixed signal actually observed and acquired in advance as learning data. The learning mixed signal is a signal emitted from a plurality of sound sources, and is, for example, audio data in which voices spoken by a plurality of speakers are recorded in monaural (single channel). In the present embodiment, the learning mixed signal is not a mixed signal in which a plurality of sound sources are artificially superimposed like the learning mixed signal in the related technique shown as Non-Patent Document 1, but is actually observed. It is a mixed signal. The mixed signal may be, for example, a sampling frequency of 8 kHz, a sample size of 16 bits, and an uncompressed linear PCM (Pulse Code Modulation). The format of the mixed signal is not limited to the above contents, and may be another format.

The learning label data storage unit 12 stores label data representing a time interval of each sound source signal determined by analyzing the mixed signal stored in the learning mixed signal storage unit 11 in advance. Specifically, the label data is data indicating which time interval each sound source is included in each mixed signal, and is, for example, data set in association with the sound source type and the start and end of the time interval. Is. For example, when it is analyzed that the sound source of the speaker A is included from O minutes o seconds to P minutes p seconds in a certain mixed signal, the label data is the sound source type: speaker A, the beginning: O minutes o seconds. , End: P minutes, p seconds, etc., the sound source type, the start and end of the time interval are set in association with each other. As a matter of course, the above-mentioned label data is an example, and other information may be set.

The feature extractor learning unit 13 learns feature extraction parameters indicating parameters used in feature extraction applied to the feature extractor. The feature extractor may be a neural network or another algorithm may be used. In the following description, the feature extractor may be described as a neural network.
The sound source separation unit 14 separates a mixed signal in which a plurality of sound sources are mixed into individual sound source signals using a neural network which is a feature extractor.

Subsequently, the details of the feature extractor learning unit 13 and the sound source separation unit 14 will be described.
The feature extractor learning unit 13 includes a feature extraction unit 101, a feature extraction parameter storage unit 102, a parameter update unit 103, a supervised loss function calculation unit 104, and an unsupervised loss function calculation unit 105. The feature extraction unit 101, the feature extraction parameter storage unit 102, and the parameter update unit 103 function as feature extraction means, feature extraction parameter storage means, and parameter update means, respectively. Further, the supervised loss function calculation unit 104 and the unsupervised loss function calculation unit 105 function as a supervised loss calculation means and an unsupervised loss function calculation means, respectively. The feature extraction unit 101 and the feature extraction parameter storage unit 102 are functional units shared with the sound source separation unit 14.

The feature extraction unit 101 corresponds to the feature extraction unit 2 in the outline of the embodiment. The feature extraction unit 101 acquires all the mixed signals stored in the learning mixed signal storage unit 11. The feature extraction unit 101 applies a short-time Fourier transform (STFT) to each acquired mixed signal to convert it into a time-frequency representation (spectrogram). Further, the feature extraction unit 101 also applies a short-time Fourier transform to the mixed signal to be determined, which is separated into individual sound source signals, to convert it into a spectrogram.

Further, the feature extraction unit 101 acquires the feature extraction parameter applied to the feature extractor (neural network) stored in the feature extraction parameter storage unit 102. The feature extraction unit 101 divides the spectrogram converted from the mixed signal into a predetermined number of time-frequency bins, and inputs the partial spectrogram corresponding to each time-frequency bin into the neural network. Then, the feature extraction unit 101 uses the result output from the neural network as a feature vector.

In the present disclosure, the time-frequency bin divided by the feature extraction unit 101 is represented as (t, f), and the partial spectrogram corresponding to the time-frequency bin (t, f) is represented as x (t, f). It is assumed that the feature vector is expressed as vt and _f . Further, in the present disclosure, the time-frequency bin may be described as a time frequency bin.

For example, assume that the format of the mixed signal is a sampling frequency of 8 kHz, a sample size of 16 bits, and an uncompressed linear PCM. Then, the spectrogram obtained by the short-time Fourier transform is obtained by transforming the mixed signal, for example, by shifting the Fourier transform at a window width of 32 msec (256 points) per frame every 8 msec (64 points). .. In this case, the resolution in the frequency direction is 31.25 Hz (256 points at 8 kHz), and the number of time-frequency bins is 125 per second in the time direction and 256 in the frequency direction. For the time-frequency bin (t, f), it is effective to consider the context before and after including (t, f) when obtaining the feature vectors vt _{, f} . For example, a 100-dimensional vector that collects bins for 100 frames including t is given to the neural network as inputs x _{t and f} . The outputs _{dt and f} of the neural network are usually set to be lower than the input, and for example, when the input is 100 dimensions, the output may be set to about 50 dimensions.

The feature extraction parameter storage unit 102 stores the feature extraction parameters used when the feature extraction unit 101 extracts the feature vector. Specifically, the feature extraction parameter storage unit 102 stores the feature extraction parameters determined by the parameter update unit 103, which will be described later. In the initial stage where the feature extraction parameter is undecided, the feature extraction parameter storage unit 102 stores the initialized feature extraction parameter by performing a process such as generating (generating) a random number by the parameter update unit 103.

The parameter update unit 103 corresponds to the parameter update unit 5 in the outline of the embodiment. The parameter update unit 103 updates the feature extraction parameters applied to the neural network based on the learning mixed signal which is the learning data.

The parameter update unit 103 acquires the feature vector extracted by the feature extraction unit 101 and the feature extraction parameters stored in the feature extraction parameter storage unit 102. The parameter update unit 103 outputs the acquired information to the supervised loss function calculation unit 104 and the unsupervised loss function calculation unit 105, which will be described later. The parameter update unit 103 evaluates the feature vector extracted by the feature extraction unit 101 using the evaluation criteria determined by the supervised loss function calculation unit 104 and the unsupervised loss function calculation unit 105. The parameter update unit 103 determines the feature extraction parameter so that a better feature vector is generated based on the evaluation result of the feature vector, and updates the feature extraction parameter to the determined feature extraction parameter.

When the feature extraction parameter is determined (updated), the parameter update unit 103 determines the feature extraction parameter by applying an iterative solution method used in the learning of the neural network, for example, an error backpropagation method. The parameter update unit 103 stores the determined feature extraction parameter in the feature extraction parameter storage unit 102, and updates the determined feature extraction parameter so that it is applied to the neural network. That is, the parameter update unit 103 defines the evaluation standard used when determining the feature extraction parameter as a loss function indicating a mathematically defined evaluation function. Then, the parameter update unit 103 iteratively determines the feature extraction parameter by using a numerical method such as a stochastic gradient descent (SGD) so that the loss function is minimized. And update the feature extraction parameters.

ここで、θは特徴抽出パラメタであり、Ｌ_θ ^（Ｓ）は教師付き損失関数であり、Ｌ_θ ^（Ｕ）は教師なし損失関数であり、λは重み係数である。また、Ｘは学習用データから得られる全てのスペクトログラムの集合であり、ＶはＸから得られる全ての特徴ベクトルの集合である。さらに、Ｙ＝（ｙ_ｔ，ｆ）は特徴ベクトルｖ_ｔ，ｆに対応する時間－周波数ビン（ｔ，ｆ）がどの音源に対応するかを表現した音源ラベルである。例えば、音源が話者である場合、ｙ_ｔ，ｆは話者を一意に特定する話者ラベルとなる。The parameter update unit 103 defines a loss function indicating an evaluation function related to the feature extraction parameter of the neural network as the following equation (11). Specifically, the parameter update unit 103 uses the evaluation function related to the feature extraction parameter as a supervised loss function indicating the first evaluation function and an unsupervised loss function indicating the second evaluation function. , To be defined.

Here, θ is a feature extraction parameter, L _θ ^(S) is a supervised loss function, L _θ ^(U) is an unsupervised loss function, and λ is a weighting factor. Further, X is a set of all spectrograms obtained from the training data, and V is a set of all feature vectors obtained from X. Further, Y = (yt _{, f} ) is a sound source label expressing which sound source the time-frequency bin (t, f) corresponding to the feature vectors vt _{, f} corresponds to. For example, when the sound source is a speaker, _{yt and f} are speaker labels that uniquely identify the speaker.

For example, suppose a mixed signal contains two speakers, and the time-frequency bin (t, f) contains the voice of the first speaker more strongly than the second speaker. , Yt _{, f} are two-dimensional vectors (1,0). On the other hand, if the time-frequency bin (t, f) contains the voice of the second speaker more strongly than the first speaker, _{yt, f} becomes a two-dimensional vector (0, 1). .. Thus, when N speakers (that is, N sound sources) are included, these vectors are N-dimensional, only one element of the N-dimensional vector is 1, and the other (N-). The element of 1) becomes 0.

The supervised loss function calculation unit 104 calculates the supervised loss function in the equation (11). As described above, since the supervised loss function can be said to be the first evaluation function, the supervised loss function calculation unit 104 can also be said to be the first calculation means. Further, it can be said that the supervised loss function calculation unit 104 calculates the first evaluation value by using the supervised loss function indicating the first evaluation function.

The supervised loss function calculation unit 104 acquires the label data stored in the learning label data storage unit 12 and generates the sound source labels yt _{and f} . The supervised loss function calculation unit 104 sets a sound source label for a time-frequency bin included in a time interval in which only a single sound source exists. On the other hand, the supervised loss function calculation unit 104 does not set the sound source label for the time-frequency bin included in the time interval in which a plurality of sound sources coexist.

Further, the supervised loss function calculation unit 104 acquires the feature vectors vt _{and f} from the parameter update unit 103, and calculates the supervised loss function L _θ ^(S) which is the first term on the right side in the equation (11). The calculation result is output to the parameter update unit 103. The details of the supervised loss function L _θ ^(S) and the sound source label will be described later.

The unsupervised loss function calculation unit 105 calculates the unsupervised loss function in the equation (11). As described above, since the unsupervised loss function can be said to be the second evaluation function, the unsupervised loss function calculation unit 105 can also be said to be the second calculation means. Further, it can be said that the unsupervised loss function calculation unit 105 calculates the second evaluation value by using the unsupervised loss function indicating the second evaluation function.

The unsupervised loss function calculation unit 105 acquires the feature vectors vt _{and f} from the parameter update unit 103 and the sound source labels yt _{and f} from the supervised loss function calculation unit 104. The unsupervised loss function calculation unit 105 calculates the unsupervised loss function L _θ ^(U) , which is the second term on the right side of the equation (11), and outputs the calculation result to the parameter update unit 103.

The feature extraction unit 101, the parameter update unit 103, the supervised loss function calculation unit 104, and the unsupervised loss function calculation unit 105 operate repeatedly while interacting with each other, and feature extraction stored in the feature extraction parameter storage unit 102. Update the parameters sequentially. The feature extraction unit 101, the parameter update unit 103, the supervised loss function calculation unit 104, and the unsupervised loss function calculation unit 105 update the feature extraction parameters a sufficient number of times so that the feature extraction parameters converge. Then, when the feature extraction parameters converge, the final feature extraction parameters are stored in the feature extraction parameter storage unit 102.

ここで、ＶＶ^Ｔは特徴ベクトルｖ_ｔ，ｆの全てのペアに関する余弦類似度（正規化された内積）を要素に持つ行列であり、ＹＹ^Ｔは音源ラベルｙ_ｔ，ｆの全てのペアに関する内積をもつ行列となる。ＶＶ^ＴおよびＹＹ^Ｔは、時間－周波数ビン（ｔ，ｆ）と（ｔ’，ｆ’）が同じ音源クラスに属している場合、１となり、同じ音源クラスに属していない場合、０となる。また、||・||_Ｆは、フロベニウスノルムであり、行列の全ての要素の自乗和の平方根を表す。すなわち、同じ音源クラスに属する特徴ベクトルのペアの余弦類似度が１に近く、同じ音源クラスに属さない特徴ベクトルのペアの余弦類似度が０に近くなるほど、式（１２）の損失関数は小さくなる。この場合、教師付き損失関数Ｌ_θ ^（Ｓ）は、特徴抽出パラメタθは、音源クラスをよく表す特徴ベクトルＶを抽出出来ていると言える。Here, the details of the supervised loss function and the unsupervised loss function included in the above equation (11) will be described.
The supervised loss function L _θ ^(S) is defined by the following equation (12). The supervised loss function L _θ ^(S) is a function representing the loss related to the feature vector V = (v _{t, f} ) extracted from the time-frequency bin in which the sound source label is set.

Here, VVT is a matrix having cosine similarity (normalized inner product) for all pairs of feature vectors vt _{and f} as an element, and YY ^T is an inner product for all pairs of sound source labels ^yt _{and f} . It becomes a matrix with. VV ^T and YY ^T are 1 when the time-frequency bins (t, f) and (t', f') belong to the same sound source class, and 0 when they do not belong to the same sound source class. Also, || · || _F is the Frobenius norm and represents the square root of the sum of squares of all the elements of the matrix. That is, the closer the cosine similarity of the pair of feature vectors belonging to the same sound source class is to 1, and the closer the cosine similarity of the pair of feature vectors not belonging to the same sound source class is to 0, the smaller the loss function of equation (12) becomes. .. In this case, it can be said that the supervised loss function L _θ ^(S) can extract the feature vector V that well represents the sound source class by the feature extraction parameter θ.

The sound source class is information indicating individual sound sources included in the mixed signal. For example, when the mixed signal includes the sound sources of the speaker A and the speaker B, the sound source of the speaker A is the first sound source class, and the sound source of the speaker B is the second sound source class.

Here, with reference to FIGS. 3 and 4, the difference between the sound source label set by the supervised loss function calculation unit 104 and the sound source label in the related technology as in Non-Patent Document 1 will be described. FIG. 3 is a diagram illustrating a sound source label in a related technique. FIG. 4 is a diagram illustrating a sound source label according to the first embodiment.

In related arts such as Non-Patent Document 1, it is assumed that all time-frequency bins yt _{, f} are known. As described above, in Non-Patent Document 1, since a plurality of sound source signals are superimposed to artificially generate a mixed signal, the loss function is defined only by the above equation (12). In a related technique such as Non-Patent Document 1, the time-frequency bin (t, f) amplitude of each sound source signal is known. Therefore, by obtaining the sound source signal having the maximum amplitude, all the time- A sound source label can be set for the frequency bin (t, f).

FIG. 3 shows an example of a sound source label set in the time-frequency bin in Non-Patent Document 1. FIG. 3 shows that a mixed signal obtained by mixing sound source signals with two speakers as sound sources is converted into a spectrogram, and a sound source label is set for each time-frequency bin. As shown in FIG. 3, in Non-Patent Document 1, since a plurality of sound source signals are superimposed to artificially generate a mixed signal, the speaker is used for each time-frequency bin (t, f). The sound source label of A or speaker B is set.

On the other hand, in the present embodiment, it is assumed that the sound source labels yt _{and f} of some of the time-frequency bins among all the time-frequency bins are unknown, and the time-frequency bins satisfying a predetermined condition are satisfied. The sound source label is set for. In the present embodiment, unlike the related art described above, an audio label is given to the spectrogram obtained by converting the actually observed mixed signal. FIG. 4 is an example in which a sound source label is set for a mixed signal using two speakers as sound sources as in FIG. As shown in FIG. 4, a sound source label is set for a time-frequency bin included in a time interval in which only the sound source of speaker A or speaker B is included in the mixed signal. In other words, the sound source label is set for the time-frequency bin in the time interval in which only a single sound source is included in the mixed signal.

As described above, the sound source label is set by the supervised loss function calculation unit 104 based on the label data stored in the learning label data storage unit 12. For example, assuming that the mixed signal includes the sound sources of the speaker A and the speaker B, the label data is set to the start and end of the time interval including the sound source of the speaker A. Similarly, the start and end of the time interval including the sound source of the speaker B are set. Since the supervised loss function calculation unit 104 can determine which speaker's sound source is included in which time interval by referring to the label data, the sound source label is set based on the label data. Can be done.

FIG. 4 shows that the sound source label of speaker A is set for the time-frequency bins from the front to the eighth in the time domain. Similarly, in the time domain, it is shown that the sound source label of the speaker B is set for the 11th to 16th time-frequency bins from the front. On the other hand, for a time-frequency bin in which multiple sound sources coexist, multiple sound sources are included and it is not known which sound source it is, so the sound source label is not set because the sound source label is unknown. .. As shown in FIG. 4, in the time domain, since the sound sources of speaker A and speaker B are mixed for the 9th and 10th time-frequency bins from the front, the sound source label is unknown and the sound source is sound source. Do not set the label. The reason is that for the actually observed mixed signal, the start and end of the time interval of each sound source signal can be set relatively easily, whereas for all time-frequency bins of each sound source signal. On the other hand, it is almost impossible to give a sound source label. Therefore, in the present embodiment, the supervised loss function calculation unit 104 sets the sound source label in the time-frequency bin included in the time interval in which only a single sound source exists among all the time-frequency bins. Further, the supervised loss function calculation unit 104 does not set a sound source label for the time-frequency bin included in the time interval in which a plurality of sound sources coexist.

For the time-frequency bin whose amplitude is sufficiently small in the mixed signal, "no sound source" may be added to indicate a special label different from the sound source label. This special label can be given automatically by simple signal processing. In this disclosure, the special label is not included in the sound source label.

As described above, in the present embodiment, the time-frequency bin in which the sound source label cannot be set is included. Therefore, it is necessary to define a loss function for the feature vector extracted from the time-frequency bin where the sound source label is not set.

ここで、ｙ_ｔ，ｆ＝ＮＵＬＬは音源ラベルが設定されていない時間－周波数ビンを表し、γ_{ｔ，ｆ，ｉ}は音源クラスｉに対する特徴ベクトルｖ_ｔ，ｆの帰属率であり、ｃは音源クラス数である。また、μ_ｉは音源クラスｉに属する時間－周波数ビン（ｔ，ｆ）にわたる特徴ベクトルｖ_ｔ，ｆの平均である。音源クラスｉに対する特徴ベクトルｖ_ｔ，ｆの帰属率は、どの音源に帰属するかを示す指標値である。また、音源クラス数は、ラベルデータより決定することが出来る。Therefore, in the present embodiment, the unsupervised loss function calculation unit 105 is provided, and the unsupervised loss function is defined as the following equation (13). That is, in the present embodiment, a loss function for a time-frequency bin whose sound source label is unknown in FIG. 4 for which the sound source label is not set is defined. By using the following equation (13), it is determined which sound source includes the time-frequency bin whose sound source label is unknown and whose sound source label is not set.

Here, y _{t, f} = NULL represents a time-frequency bin in which the sound source label is not set, γ _{t, f, i} are the attribution ratios of the feature vectors vt _{, f} to the sound source class i, and c is the sound source. The number of classes. Further, μ _i is the average of the feature vectors v _{t, f} over the time-frequency bin (t, f) belonging to the sound source class i. The attribution rate of the feature vectors vt _{and f} with respect to the sound source class i is an index value indicating which sound source belongs to. Further, the number of sound source classes can be determined from the label data.

ここで、帰属率γ_{ｔ，ｆ，ｉ}は、例えば、ｉ＝ａｒｇｍｉｎ_ｊ｜ｖ_ｔ，ｆ―μ_ｊ｜が成り立つ場合、γ_{ｔ，ｆ，ｉ}＝１となり、そうではない場合、γ_{ｔ，ｆ，ｉ}＝０となるような、最近傍法に基づく離散的な帰属率を定義することができる。 _μi is calculated according to the following equation (14).

Here, the attribution rate γ _{t, f, i} is, for example, γ _{t, f, i} = 1 when i = argmin _j | v _{t, f} ―μ _j |, and γ t, when i = argmin j | v t, f ―μ j | _. It is possible to define a discrete attribution rate based on the nearest neighbor method such that _{f, i} = 0.

Equations (13) and (14), and the unsupervised loss function with discrete attribution, are a form of hard clustering that classifies feature vectors into unique clusters based on Euclidean distance, as is clear from the above. Is. The unsupervised loss function is, in particular, quasi-supervised hard clustering containing feature vectors of known and unknown sound source classes. In other words, it can be said that the unsupervised loss function calculation unit 105 calculates the loss function based on hard clustering.

The unsupervised loss function described above is an example and is not limited to this. For example, instead of measuring the closeness of the feature vector by the Euclidean distance (L2 norm), it is also possible to measure by the Manhattan distance (L1 norm) or a similarity scale such as Lp norm or cosine similarity. In particular, the cosine similarity is suitable because it is highly consistent with the supervised loss function of Eq. (12).

Further, the attribution rates γ _{t, f, i} may be continuous, and for example, γ _{t, f, i} and μ _i may be defined based on soft clustering assuming a Gaussian mixture distribution. In general, in clustering, any similarity measure and loss function can be defined, so the unsupervised loss function of the present embodiment may be defined as well. Further, when the sound source label to which the sound source label cannot be attached is sufficiently smaller than the time-frequency bin in which the sound source label to which the sound source label can be attached is known for the unknown time-frequency bin, the right side of the above equation (14). The second term of the molecule and denominator of is negligible. That is, in the above equation (14), the term relating to the feature vector to which the sound source label is not attached can be ignored.

Returning to FIG. 2, the sound source separation unit 14 will be described. The sound source separation unit 14 separates the mixed signal into individual sound source signals using a neural network that is a feature extractor. The sound source separation unit 14 includes a feature extraction unit 101, a feature extraction parameter storage unit 102, a clustering unit 106, and a separation unit 107. The clustering unit 106 and the separation unit 107 function as a clustering means and a separation unit, respectively. Further, the feature extraction unit 101 and the feature extraction parameter storage unit 102 are functional units shared with the feature extractor learning unit 13.

The feature extraction unit 101 acquires the mixed signal, converts the mixed signal into the spectrogram X, and generates the feature vectors vt _{, f from the partial spectrograms x t, f ,} as in the configuration in the feature extractor learning unit 13. ..

The clustering unit 106 corresponds to the clustering unit 3 in the outline of the embodiment. The clustering unit 106 applies any algorithm of, for example, K-means, mean-shift, shortest / longest distance method, Ward's method, etc., to the feature vector _vt . _{, F} are classified into multiple clusters.

The separation unit 107 corresponds to the separation unit 4 in the outline of the embodiment. The separation unit 107 generates a sound source signal for each of the classified clusters by using the time frequency bin included in each of the plurality of clusters classified by the clustering unit 106. Specifically, the separation unit 107 performs an inverse Fourier transform on the spectrogram reconstructed only from the time-frequency bins (t, f) included in each cluster for each cluster classified by the clustering unit 106, and individually. Generates the sound source signal of.

<Operation example of sound source separator>
Subsequently, an operation example of the sound source separation device 10 will be described with reference to FIGS. 5 to 7. 5 to 7 are flowcharts showing an operation example of the sound source separation device according to the first embodiment.

First, the overall operation of the sound source separation device 10 will be described with reference to FIG. As shown in FIG. 5, the sound source separation device 10 executes the feature extractor learning process (step A1) and the sound source separation process (step A2).

Specifically, the sound source separation device 10 learns the feature extraction parameters of the neural network which is the feature extractor by using the actually observed mixed signal in the feature extractor learning process (step A1).

Next, in the sound source separation process, the sound source separation device 10 separates the mixed signal into individual sound source signals by using the feature extractor to which the feature extraction parameters determined in step A1 are applied (step A2).

Subsequently, the feature extractor learning process will be described with reference to FIG. The flowchart shown in FIG. 6 is a flowchart executed in step A1 of FIG. 5, and is executed by the feature extractor learning unit 13. The operation shown below is clearly different from the operation disclosed in Non-Patent Document 1.

First, the feature extraction unit 101 sequentially acquires the mixed signals stored in the learning mixed signal storage unit 11, performs a short-time Fourier transform, and converts them into a spectrogram (step B1).

Next, the feature extraction unit 101 acquires the feature extraction parameters stored in the feature extraction parameter storage unit 102. The feature extraction unit 101 extracts feature vectors vt _, f from each time-frequency bin (t, f) in the converted spectrogram by using a neural network which is a feature extractor to which the acquired parameters are applied (. Step B2).

In the initial stage where the feature extraction parameters are undecided, the parameter update unit 103 performs an operation such as generating a random number in an initialization step (not shown) to initialize the feature extraction parameters, and the feature extraction parameter storage unit 102 in advance. Output to.

Next, the parameter update unit 103 acquires the feature vector extracted by the feature extraction unit 101 from the feature extraction unit 101, and calculates a loss function, which is a scale for measuring the quality of the feature vector, based on the equation (11). Specifically, the parameter update unit 103 calculates the loss function shown in the equation (11) by using the calculation results calculated in steps B3 and B4, which will be described later, for the loss function.

In step B3, the supervised loss function calculation unit 104 calculates the supervised loss function shown in the equation (12) (step B3). Specifically, the supervised loss function calculation unit 104 acquires the feature vector extracted by the feature extraction unit 101 via the parameter update unit 103. Further, the supervised loss function calculation unit 104 acquires label data representing a time interval of each sound source stored in the learning label data storage unit 12. The supervised loss function calculation unit 104 sets a sound source label in the time-frequency bin in the time interval in which only a single sound source exists among the time-frequency bins of each sound source based on the acquired label data. Then, the supervised loss function calculation unit 104 calculates the supervised loss function based on the equation (12) with respect to the time-frequency bin in which the sound source label is set.

In step B4, the unsupervised loss function calculation unit 105 calculates the unsupervised loss function shown in the equation (13) (step B4). Specifically, the unsupervised loss function calculation unit 105 acquires the feature vector extracted by the feature extraction unit 101 via the parameter update unit 103. Further, the unsupervised loss function calculation unit 105 acquires the sound source label set by the supervised loss function calculation unit 104. The unsupervised loss function calculation unit 105 calculates the unsupervised loss function based on the equations (13) and (14) for the time-frequency bin in which the sound source label is not set.

The parameter update unit 103 updates the feature extraction parameter based on the calculation result of the loss function shown in the equation (11) (step B5). Specifically, the parameter update unit 103 uses the calculation result of the supervised loss function calculated in step B3 and the calculation result of the unsupervised loss function calculated in step B4, and the loss represented by the equation (11). Compute the function. The parameter update unit 103 determines the feature extraction parameter so that the calculation result of the loss function shown in the equation (11) is reduced. Then, the parameter update unit 103 stores the determined feature extraction parameter in the feature extraction parameter storage unit 102, and updates the feature extraction parameter.

Next, the parameter update unit 103 determines whether or not a predetermined convergence condition is satisfied, for example, the decreasing tendency of the calculation result of the loss function shown in the equation (11) disappears (step B6). The parameter update unit 103 may determine in step B6 whether the processes of steps B2 to B5 have been performed a predetermined number of times.

In step B6, when the parameter update unit 103 determines that the predetermined convergence condition is satisfied (YES in step B6), the process ends.
On the other hand, when the parameter update unit 103 determines that the predetermined convergence condition is not satisfied (NO in step B6), the parameter update unit 103 returns to step B2 and performs the processing after step B2 again.

Subsequently, the sound source separation process will be described with reference to FIG. 7. The flowchart shown in FIG. 7 is a flowchart executed in step A2 of FIG. 5, and is executed by the sound source separation unit 14.

First, the feature extraction unit 101 performs a short-time Fourier transform on the mixed signal to be determined to be separated into individual sound source signals, and converts it into a spectrogram (step C1). The mixed signal to be determined may be a mixed signal observed by the sound source separation device 10 with a microphone (not shown), or may be a mixed signal pre-recorded and stored.

Next, the feature extraction unit 101 acquires the feature extraction parameters stored in the feature extraction parameter storage unit 102. The feature extraction unit 101 extracts feature vectors vt _, f from each time-frequency bin (t, f) in the converted spectrogram by using a neural network which is a feature extractor to which the acquired feature extraction parameters are applied. (Step C2).

Next, the clustering unit 106 clusters the feature vectors _{dt and f} extracted by the feature extraction unit 101 (step C3). Specifically, the clustering unit 106 clusters the feature vectors _{dt and f} extracted by the feature extraction unit 101 to cluster the time-frequency bins in the same number as the number of sound sources assumed to be included in the mixed signal. Classify into.

The clustering unit 106 performs clustering by applying any of the algorithms such as the K-means method, the mean shift method, the shortest / longest distance method, and the Ward method. You may. Further, the clustering unit 106 may determine the number of clusters for classifying the feature vectors vt _{and f} according to the prior information such as "a conversation between two speakers". Alternatively, the clustering unit 106 may use the method for determining the number of clusters provided by any of the above algorithms in the absence of the above prior information.

Next, the separator 107 performs an inverse Fourier transform on the spectrogram reconstructed from the time-frequency bin contained in each of the plurality of classified clusters, and is separated into a single sound source for each classified cluster. The sound source signal is generated and output (step C4).

As described above, the sound source separation device 10 according to the present embodiment uses the actually observed mixed signal and the label data of the time interval of each sound source assigned to the mixed signal, and features the feature extractor. Determine the extraction parameters. Further, the sound source separation device 10 according to the present embodiment uses a loss function including two loss functions, a supervised loss function and an unsupervised loss function, when determining the feature extraction parameter, and the calculation result of each loss function. Update to the feature extraction parameter that minimizes the sum of. Therefore, by using the sound source separation device 10 according to the present embodiment, the optimum feature extractor for the actually observed mixed signal is obtained instead of the artificially created mixed signal, and the mixed signal is obtained. Can be accurately separated into individual sound source signals. That is, by using the sound source separation device 10 according to the present embodiment, it is possible to accurately separate individual sound source signals from the mixed signal.

(Embodiment 2)
Subsequently, the second embodiment will be described.
<Sound source separation device configuration example>
The sound source separation device 80 according to the second embodiment will be described with reference to FIG. FIG. 8 is a configuration diagram showing a configuration example of the sound source separation device according to the second embodiment. As shown in FIG. 8, the sound source separation device 80 according to the present embodiment includes a sound source separation program 81, a data processing device 82, and a storage device 83. Further, the storage device 83 includes a feature extraction parameter storage area 831, a learning mixed signal storage area 832, and a learning label data storage area 833. It should be noted that this embodiment is a configuration example in which the feature extractor learning unit 13 and the sound source separation unit 14 in the first embodiment are realized by a computer operated by a program.

The sound source separation program 81 is read into the data processing device 82 and controls the operation of the data processing device 82. The sound source separation program 81 describes the operations of the feature extractor learning unit 13 and the sound source separation unit 14 in the first embodiment using a programming language.

Specifically, the data processing device 82 executes the same processing as the processing of the feature extractor learning unit 13 and the sound source separation unit 14 in the first embodiment under the control of the sound source separation program 81. That is, the data processing device 82 has the feature extraction parameter stored in the feature extraction parameter storage area 831, the learning mixed signal storage area 832, and the learning label data storage area 833 in the storage device 83, respectively, the feature extraction parameter, the learning mixed signal, and the learning. Get the label data for. Then, the data processing device 82 processes the feature extractor learning unit 13 and the sound source separation unit 14 in the first embodiment of the first embodiment.

More specifically, in the first embodiment, the data processing device 82 includes a feature extraction unit 101, a parameter update unit 103, a supervised loss function calculation unit 104, an unsupervised loss function calculation unit 105, a clustering unit 106, and a separation unit. Each process carried out by 107 is performed.

As described above, the sound source separation device 80 according to the second embodiment also performs each process executed by each functional unit in the first embodiment, so that the same effect as that of the first embodiment can be obtained. Become. That is, by using the sound source separation device 80 according to the present embodiment, the optimum feature extractor for the actually observed mixed signal, not the artificially created mixed signal, is obtained, and the mixed signal is obtained. Can be accurately separated into individual sound source signals. Therefore, by using the sound source separation device 80 according to the present embodiment, it is possible to accurately separate individual sound source signals from the mixed signal.

Further, by using the sound source separation program 81 according to the second embodiment, it is possible to obtain the same effect as that of the first embodiment. That is, according to the sound source separation program 81 according to the present embodiment, it is possible to accurately separate individual sound source signals from the mixed signal.

(Other embodiments)
The sound source separation device according to the above-described embodiment may have the following hardware configuration. FIG. 9 is a block diagram showing a configuration example of the sound source separation devices 1, 10 and 80 (hereinafter, referred to as sound source separation device 1 and the like) described in the above-described embodiment. Referring to FIG. 9, the sound source separation device 1 and the like include the processor 1201 and the memory 1202.

The processor 1201 reads software (computer program) from the memory 1202 and executes it to perform processing of the sound source separation device 1 and the like described by using the flowchart in the above-described embodiment. The processor 1201 may be, for example, a microprocessor, an MPU (Micro Processing Unit) or a CPU (Central Processing Unit). Processor 1201 may include a plurality of processors.

The memory 1202 is composed of a combination of a volatile memory and a non-volatile memory. Memory 1202 may include storage located away from processor 1201. In this case, processor 1201 may access memory 1202 via an I / O interface (not shown).

In the example of FIG. 9, memory 1202 is used to store software modules. By reading these software modules from the memory 1202 and executing the processor 1201, the processor 1201 can perform the processing of the sound source separation device 1 and the like described in the above-described embodiment.

As described with reference to FIG. 9, each of the processors included in the sound source separator 1 and the like executes one or more programs including a set of instructions for causing the computer to perform the algorithm described with reference to the drawings.

In the above example, the program can be stored and supplied to the computer using various types of non-transitory computer readable medium. Non-temporary computer-readable media include various types of tangible storage media. Examples of non-temporary computer-readable media include magnetic recording media (eg, flexible disks, magnetic tapes, hard disk drives), magneto-optical recording media (eg, magneto-optical disks). Further, examples of non-temporary computer-readable media include CD-ROM (Read Only Memory), CD-R, and CD-R / W. Further, examples of non-temporary computer readable media include semiconductor memory. The semiconductor memory includes, for example, a mask ROM, a PROM (Programmable ROM), an EPROM (Erasable PROM), a flash ROM, and a RAM (Random Access Memory). The program may also be supplied to the computer by various types of transient computer readable medium. Examples of temporary computer readable media include electrical, optical, and electromagnetic waves. The temporary computer-readable medium can supply the program to the computer via a wired communication path such as an electric wire and an optical fiber, or a wireless communication path.

The present disclosure is not limited to the above embodiment, and can be appropriately modified without departing from the spirit. Further, the present disclosure may be carried out by appropriately combining the respective embodiments.

1, 10, 80 Sound source separator 2, 101 Feature extraction unit 3, 106 Clustering unit 4, 107 Separation unit 5, 103 Parameter update unit 11 Learning mixed signal storage unit 12 Learning label data storage unit 13 Feature extractor learning unit 14 Sound source separation unit 102 Feature extraction parameter storage unit 104 Supervised loss function calculation unit 105 Unsupervised loss function calculation unit 81 Sound source separation program 82 Data processing device 83 Storage device

Claims (9)

A feature extraction means for extracting a feature vector using a feature extractor to which the parameters used for feature extraction are applied for each time-frequency bin in a spectrogram obtained by converting a mixed signal in which a plurality of sound source signals are mixed.
A clustering means for classifying the extracted feature vector into a plurality of clusters,
A separation means for generating a sound source signal for each classified cluster using a time frequency bin included in each of the plurality of classified clusters.
A parameter updating means for updating the parameter based on the learning mixed signal including the observed mixed signal is provided .
In the spectrogram obtained by converting the mixed signal for learning, the sound source label is set in the time frequency bin satisfying a predetermined condition, and the first evaluation value for the feature vector extracted from the time frequency bin in which the sound source label is set is set. The first calculation means calculated using the first evaluation function and
Further, a second calculation means for calculating a second evaluation value for a feature vector extracted from a time frequency bin in which the sound source label is not set by using a second evaluation function is provided.
The parameter updating means is a sound source separation device that updates the parameters based on the first evaluation value and the second evaluation value . The sound source separation device according to claim 1 , wherein the parameter updating means updates the parameters so as to reduce the total value of the first evaluation value and the second evaluation value. The first calculation means sets the sound source label in the time frequency bin in the time interval in which a single sound source exists, based on the label data indicating the time interval in which each sound source signal is included in the learning mixed signal. The sound source separation device according to claim 1 or 2 , wherein the sound source label is not set in the time frequency bin in the time interval in which a plurality of sound sources exist. The sound source separation device according to any one of claims 1 to 3 , wherein the second evaluation function is a loss function based on at least one of hard clustering and soft clustering. The sound source separation device according to any one of claims 1 to 4 , wherein the first evaluation function is a supervised loss function, and the second evaluation function is an unsupervised loss function. The sound source separation device according to claim 5 , wherein the supervised loss function is the following equation (1), and the unsupervised loss function is the following equation (2).

Here, θ is the parameter, X is a set of all spectrograms obtained from the mixed signal for learning, and (t, f) and ( _t ', f') are time frequency bins, and vt. _{, F} is the feature vector of the time frequency bin (t, f), V is the set of all feature vectors v _{t, f} obtained from X, and Y = (y _{t, f} ) is the feature vector v _{t ,.} It is a sound source label of the time frequency bin (t, f) corresponding to _f .

Here, y _{t, f} = NUML is a time frequency bin in which the sound source label is not set, γ _{t, f, i} are the attribution ratios of the feature vectors v _{t, f} to the sound source class i, and c is the sound source class. It is a number, and μ _i is the average of the feature vectors vt, _f over the time frequency bins (t, f) belonging to the sound source class i, and is determined by the equation (3).

The sound source separation device according to any one of claims 1 to 6 , wherein the feature extractor is a neural network. In a spectrogram obtained by converting a mixed signal in which a plurality of sound source signals are mixed, a feature vector is extracted for each time-frequency bin using a feature extractor to which the parameters used for feature extraction are applied.
By classifying the extracted feature vectors into multiple clusters,
Using the time frequency bin included in each of the plurality of classified clusters, the sound source signal is generated for each classified cluster, and
Updating the above parameters based on the learning mixed signal including the observed mixed signal
In the spectrogram obtained by converting the mixed signal for learning, the sound source label is set in the time frequency bin satisfying a predetermined condition, and the first evaluation value for the feature vector extracted from the time frequency bin in which the sound source label is set is set. To calculate using the first evaluation function and
Using the second evaluation function, the second evaluation value for the feature vector extracted from the time frequency bin in which the sound source label is not set is calculated.
A sound source separation method comprising updating the parameters based on the first evaluation value and the second evaluation value . In a spectrogram obtained by converting a mixed signal in which a plurality of sound source signals are mixed, a feature vector is extracted for each time-frequency bin using a feature extractor to which the parameters used for feature extraction are applied.
By classifying the extracted feature vectors into multiple clusters,
Using the time frequency bin included in each of the plurality of classified clusters, the sound source signal is generated for each classified cluster, and
Updating the above parameters based on the learning mixed signal including the observed mixed signal
In the spectrogram obtained by converting the mixed signal for learning, the sound source label is set in the time frequency bin satisfying a predetermined condition, and the first evaluation value for the feature vector extracted from the time frequency bin in which the sound source label is set is set. To calculate using the first evaluation function and
Using the second evaluation function, the second evaluation value for the feature vector extracted from the time frequency bin in which the sound source label is not set is calculated.
A program that causes a computer to update the parameters based on the first evaluation value and the second evaluation value .

Images