JP2008233866A - Signal separating device, signal separating method, and computer program - Google Patents

Abstract

<P>PROBLEM TO BE SOLVED: To perform highly accurate separation processing in consideration of a delay amount for mixed sound signals having various delay amounts. <P>SOLUTION: Observation spectrograms generated by converting an input sound signal into a time-frequency domain is interpreted as observation signals subjected to convolution mixtures in the time-frequency domain and an independent component analysis solving the convolutive mixtures is performed to generate signal separated results. Alternatively, modulation spectrograms generated by short-time Fourier transform (STFT) is interpreted as instantaneous mixting and an independent component analysis solving the instantaneous mixting is performed to generate signal separated results. Therefore, highly accurate separation processing is performed by taking into account a delay amount for mixed sound signals having various delay amounts such as direct waves and reflected waves. <P>COPYRIGHT: (C)2009,JPO&INPIT

Description

  The present invention relates to a signal separation device, a signal separation method, and a computer program. More specifically, the present invention relates to a signal separation device, a signal separation method, and a computer program for separating a signal in which a plurality of signals are mixed into each signal using independent component analysis (ICA). About.

  Independent component analysis (ICA), which uses only statistical independence to separate and reconstruct the original signal when multiple original signals are linearly mixed by unknown coefficients, is the signal processing method. It is attracting attention in the field. By applying this independent component analysis, voice signals can be separated and restored even in situations where the speaker and the microphone are separated and the microphone picks up sounds other than the speaker's voice. It becomes possible.

  ICA is a type of multivariate analysis, and is a technique for separating multidimensional signals using the statistical properties of signals. For details of ICA itself, refer to Non-Patent Document 1 ("Introduction / Independent Component Analysis" (Noboru Murata, Tokyo Denki University Press)).

First, a method for separating a signal in which a plurality of signals (especially sound signals) are mixed in the time-frequency domain by using independent component analysis in the time-frequency domain will be described, and then problems of the method will be described. As shown in FIG. 1, a situation is considered in which different sounds are produced from N sound sources (signal sources) and these are observed by n microphones (sensors). When sounds (original signals) from multiple sound sources reach the microphone, the sound acquired by the microphone includes direct waves and reflected waves, and there is a time delay based on the distance to each sound source. The signal (observation signal) observed at j (where 1 ≦ j ≦ n) is expressed as a summation of the convolution operation of the original signal and the transfer function for all sound sources, as shown in Equation [1.1] below. (Hereinafter referred to as “convolution mixing”). Furthermore, if the observation signals for all the microphones 1 to n are expressed by one equation, it can be expressed as equation [1.2]. However, x (t), s (t) respectively _x k _(t), a column vector with _s k (t) of the ^{elements, A [l]} is a matrix of n × N whose elements _{a kj} is there. (Hereafter, n = N is assumed.)

The following two methods are known as a method for solving such folding mixing.
(1) Solve convolutional mixing directly in the time domain. (Time domain deconvolution)
(2) Convert the observed signal to the time-frequency domain and solve it as an instantaneous mixing problem.
Below, each method is demonstrated.

(1) Method for directly solving the convolution mixture in the time domain (time domain deconvolution) In order to solve the convolution of the above equation [1.2], the observation signal of the following equation [2.1] Prepare a convolution mixing formula.

An expression for convolutional mixing of observation signals such as the above expression [2.1] is prepared, and components y ₁ (t) to y _n (t) of the separation result y (t) are all t The separation matrices W ^{[0] to} W ^{[L ′]} are determined so as to be most independent of each other (hereinafter, W ^{[0] to} W ^{[L ′]} are collectively referred to as a separation filter). For this purpose, Expressions [2.1] to [2.4] are repeated until the separation matrix and the separation result converge (hereinafter, such repetition is referred to as “learning.” In addition, the separation matrix is updated. Formulas and formulas for calculating ΔW are called “learning rules”). In the equation [2.3], E _t [] represents an average with respect to t. Φ in the equation is a function called a score function or an activation function. For details of the equation for solving convolutional mixing in the time domain, see, for example, Non-Patent Document 2 (“Detailed Independent Component Analysis” (Aapo Hyvarenen et al., Tokyo Denki University Press) 19.2 Separation in the dark of convolutional mixing, Refer to 19.2.3 Natural gradient method.

(2) Method for transforming observed signals into the time-frequency domain and solving them as an instantaneous mixing problem Time-domain convolutional mixing is known to be represented by instantaneous mixing in the time-frequency domain. It is ICA (Independent Component Analysis) in the time-frequency domain. Regarding the time-frequency domain ICA itself, Non-Patent Document 2 (“Detailed Independent Component Analysis” (Aapo Hyvarinen et al., Tokyo Denki University Press “19.2.4 Fourier Transform Method”)) and Patent Document 1 (Special See, for example, 2006-238409 “Audio Signal Separation Device / Noise Removal Device and Method”).

In the independent component analysis in the time frequency domain, instead of directly estimating A and s (t) from x (t) in the above equation [1.2], x (t) is converted into a signal in the time frequency domain, The signals corresponding to A and s (t) are estimated in the time frequency domain. Below, the point which is mainly related to this invention is demonstrated. When both sides of the above formula [1.2] are subjected to short-time Fourier transform, the following formula [3.1] is approximately obtained. That is, X (ω, t) and S (ω, t) are obtained by short-time Fourier transforming the signal vectors x (t) and s (t) through a window of length L, respectively, and the matrix A (t) is the same. If A (ω) is the result of short-time Fourier transform, the above formula [1.2] in the time domain can be expressed by the following formula [3.1] in the time frequency domain. However, ω indicates a frequency bin number (1 ≦ ω ≦ M), and t indicates a frame number (1 ≦ t ≦ T). In the independent component analysis in the time-frequency domain, S (ω, t) and A (ω) in Equation [3.1] are estimated in the time-frequency domain.

  In the above equation [3.1], ω is a frequency bin number, and t is a frame number. If ω is fixed, this equation can be regarded as instantaneous mixing. Therefore, in order to separate the observation signals, an equation such as Equation [3.5] is prepared, and the separation matrix W (ω) is determined so that each component of Y (ω, t) is most independent.

The number of frequency bins is essentially the same as the window length L, and each frequency bin has frequency components obtained by equally dividing -R / 2 to R / 2 (R is a sampling frequency) into L equal parts. To express. However, the negative frequency component is a conjugate complex number of the positive frequency component, and can be obtained as X (−ω) = conj (X (ω)) (conj (·) is a conjugate complex number).
In order to estimate S (ω, t) and A (ω) in the time-frequency domain, first consider the following equation (4). In equation [3.5], Y (ω, t) represents a column vector whose element is Yk (ω, t) obtained by short-time Fourier transform of yk (t) through a window of length L, and W (ω) Represents an n-by-n matrix (separation matrix) having wij (ω) as an element.

  In the conventional time-frequency domain ICA, there is a problem called “permutation problem” in which “which component is separated into which channel” is different for each frequency bin. The above-mentioned patent document 1 (Japanese Patent Application Laid-Open No. 2006-238409 “Audio Signal Separation Device / Noise Removal Device and Method”) is substantially solved.

  Since the present invention is a theoretical development version of this published patent application, JP-A-2006-238409, the features of JP-A-2006-238409 will also be described below.

  Conventionally, that is, before the technique disclosed in Patent Document 1 (Japanese Patent Application Laid-Open No. 2006-238409) is disclosed, [3.5], which is an expression for each frequency bin, is used as an expression for separating the time-frequency domain. Further, a separation matrix W (ω) that maximizes independence for each frequency bin is obtained.

  In other words, when ω is fixed and t is changed, W1 (Y (ω, t) to Yn (ω, t) is statistically independent (in practice, the independence is maximized). ω). As will be described later, there is a solution other than W (ω) = A (ω) −1 because of the indefiniteness of permutation and scaling in independent component analysis in the time-frequency domain. If Y1 (ω, t) to Yn (ω, t) that are statistically independent are obtained for all ω, the time domain separation signal y (t) can be obtained by performing inverse Fourier transform on them. it can.

  An outline of conventional independent component analysis in the time-frequency domain will be described. s1 to sn are independent original signals emitted from n sound sources, and s is a vector having them as elements. The observation signal x observed by the microphone is obtained by performing the convolution / mixing operation of the above equation [1.2] on the original signal s. Next, short-time Fourier transform is performed on the observation signal x to obtain a signal X in the time-frequency domain. If the element of X is Xk (ω, t), Xk (ω, t) takes a complex value. A diagram in which | Xk (ω, t) |, which is the absolute value of Xk (ω, t), is expressed by color shading is called a spectrogram. In the spectrogram, for example, | Xk (ω, t) |, which is the absolute value of Xk (ω, t), is expressed by color shading, where the horizontal axis is t (frame number) and the vertical axis is ω (frequency bin number). FIG. Subsequently, the separated signal Y is obtained by multiplying each frequency bin of the signal X by W (ω). Then, the separation signal Y is inverse Fourier transformed to obtain a separation signal y in the time domain.

  However, in the time-domain independent component analysis described above, signal separation processing is performed for each frequency bin, and the relationship between the frequency bins is not considered. Therefore, even if the separation itself is successful, there is a possibility that scaling and separation destination inconsistency may occur among the frequency bins. Among these, inconsistency of scaling can be solved by a method of estimating an observation signal for each sound source. On the other hand, the separation destination is inconsistent, for example, a phenomenon in which a signal derived from S1 appears in Y1 at ω = 1, whereas a signal derived from S2 appears in Y1 at ω = 2. It is called the problem of mutation.

On the other hand, in Patent Document 1 (Japanese Patent Laid-Open No. 2006-238409), a separation matrix that maximizes the independence of the entire spectrogram using the following equation [4.4] that is an expression representing the separation of the entire spectrogram. A method of obtaining W is employed.

  Specifically, as the independence of the entire spectrogram, the Kullback-Leiblar information amount I (Y) represented by the formula [4.5] is introduced, and the separation matrix W that minimizes I (Y) is obtained. . In independent component analysis, there are various variations on what scale to express independence and what algorithm maximizes independence. As one of the methods, there is Kullback-Leibler information amount (KL information amount) for independence. The Kullback-Leiblar information amount I (Y) is obtained by subtracting the simultaneous entropy of the entire spectrogram from the total entropy of each spectrogram, and is minimum (ideally 0 when all spectrograms are independent of each other). )

The KL information amount I (Y) is defined as in the above formula [4.5] as described above. In this equation [4.5], H (Y _k ) represents the entropy for one spectrogram for each channel, and H (Y) represents the simultaneous entropy for one spectrogram for all channels. FIG. 2 shows the relationship between H (Y _k ) and H (Y) when n = 2. In FIG. 2, P (Y _k (t)) is a probability density function of Y _k (t), and H (Y _k ) is an entropy for one spectrogram for each channel. The Kullback-Leiblar information amount I (Y) is obtained by subtracting the simultaneous entropy 13 of the entire spectrogram from the sum of entropies 11 and 12 for each spectrogram, and is minimum (ideal) when all spectrograms are independent of each other. 0).

In order to minimize the KL information amount I (Y) in the entire spectrogram, the equations [5.1] to [5.3] are repeated until W and Y converge.

  Note that ΔW (ω), W (ω), and Y (ω, t) appearing in Equation [5.3] are elements corresponding to the ω-th frequency bin from ΔW, W, and Y (t), respectively. This is the extracted submatrix. By doing so, it became possible to obtain separation results without permutation problems.

However, the method of solving the above two convolution mixture, ie,
(1) Solve convolutional mixing directly in the time domain. (Time domain deconvolution)
(2) Convert the observed signal to the time-frequency domain and solve it as an instantaneous mixing problem.
There are problems with these two approaches. That is,
(1) Solve convolutional mixing directly in the time domain. (Time domain deconvolution)
This method has a problem of slow convergence. The reason for the slow convergence is that the entire waveform changes when the coefficient of the separation filter changes, and that the calculation amount of the update formula of the separation filter is proportional to the square of the tap number L ′. Therefore, when the number of taps L ′ of the filter is large, it is difficult to separate in a practical time unless a value as close as possible to the convergence value is obtained in advance as the initial value of the separation filter. In order to cope with the reverberation of the real environment, the number of taps in the order of several thousand is required. Therefore, the method (1) requires a calculation amount of several thousand squares.

On the other hand, (2) transform the observed signal into the time-frequency domain and solve it as an instantaneous mixing problem. This method has a problem that a trade-off exists between the window length of short-time Fourier transform (STFT) and the separation accuracy. When the observed signal is long reverberation, that is, convolutional confusion with a large number of taps, in order to express it by instantaneous mixing in the time-frequency domain, it is necessary to increase the STFT window length (= tap number). (If the window length is less than the reverberation length, the reverberation spans a plurality of frames and cannot be expressed by instantaneous mixing.) However, it is known that if the window length is too long, the separation accuracy decreases. For this trade-off, refer to the following documents, for example.
Patent Document 2 (Japanese Patent Laid-Open No. 2003-271168 “Signal Extraction Method and Signal Extraction Apparatus, Signal Extraction Program and Recording Medium Recording the Program”)
Non-Patent Document 3 ("Study on blind sound source separation by subband processing" Akiko Araki, Robert Aichner, Shoji Makino, Takeki Nishikawa, Hiroshi Saruwatari Proceedings of the Acoustical Society of Japan, March 2002, pp. 619--620)
Non-Patent Document 4 ("Optimization of the number of band divisions in the Blind Source Separation using band division type ICA" Takeki Nishikawa, Akiko Araki, Shoji Makino, Hiroshi Saruwatari Proceedings of the Acoustical Society of Japan, March 2001 pp.569- -570)

  The reason why the separation accuracy decreases when the window length is increased is that the longer the window length (= the larger the number of taps), the more gradually the change in the time direction of the generated spectrogram, that is, the change in the time direction envelope. . The time frequency domain ICA separates the observation signals by focusing on the independence between the envelopes, but the independence between the gentle envelopes tends to be calculated lower than the independence between the rapidly changing envelopes. . That is, even envelopes derived from different sound sources may be determined to have “correlation”, resulting in poor separation accuracy.

  As described above, (2) the problem in the method of converting the observation signal into the time-frequency domain and solving it as an instantaneous mixing problem is that there is a trade-off between the window length of short-time Fourier transform (STFT) and the separation accuracy. It is to be. Hereinafter, the results of experiments conducted by the inventor regarding the trade-off between window length and separation accuracy will be shown. FIG. 3 is a graph plotting the relationship between the window length of the STFT and the separation accuracy of the time frequency domain ICA.

  In FIG. 3, the horizontal axis is the STFT window length (64, 128, 256, 512, 1024, 2048, 4096), the vertical axis is the signal-interference-ratio (SIR), which is a measure of separation accuracy, and the solid line is (2) SIR obtained as a method of converting the observation signal into the time-frequency domain and solving as an instantaneous mixing problem is the result of separation by the method disclosed in Japanese Patent Laid-Open No. 2006-238409 (details of the experiment will be described later). Also, the upper graph in FIG. 3 is a waveform-based SIR, and the lower graph is a frequency bin-based SIR. FIG. 4 is a graph showing the window length on the horizontal axis in actual seconds. Both graphs show that there is a peak of separation accuracy in the middle. (Window length = 1024 is peak for waveform base. Window length = 512 is peak for frequency bin base.)

  That is, in the ICA in the time-frequency domain, even if the STFT window is lengthened so as to cope with long reverberation, there is a problem that the separation performance is deteriorated if it exceeds a certain level.

In summary, the following two methods, which are methods of independent component analysis (ICA), are as follows:
(1) Solve convolutional mixing directly in the time domain. (Time domain deconvolution)
(2) Convert the observed signal to the time-frequency domain and solve it as an instantaneous mixing problem.
Both of these two methods have a problem that the separation accuracy is insufficient for convolutional mixing with a large number of taps.

  Note that Non-Patent Document 5 (Serviere, C. Separation of speech) discloses a process corresponding to the assumption that “if STFT is performed using a window shorter than the reverberation length, convolution still remains on the spectrogram”. signals under reverse conditions. In Proc. EUSIPCO04, pp. 1693-1696 (2004)).

  Non-Patent Document 5 considers the observation signal to be convolutional mixture in the time-frequency domain, and proposes a deconvolution algorithm in the time-frequency domain as a method for solving it. In other words, it is a process close to the method of “solving convolutional mixture directly in the time frequency domain”. However, the algorithm disclosed in Non-Patent Document 5 is limited to the case of two inputs and two outputs, that is, two output sound sources for audio signals and two microphones as input units. In addition, in this document, separation and deconvolution are individually performed for each frequency bin, and a “permutation problem”, which component is separated into which channel, is different for each frequency bin. The problem occurs.

As described above, there are some prior arts disclosed regarding the separation processing of a sound signal in which a plurality of signals are mixed, but high-precision separation processing for each signal using independent component analysis (ICA). In the signal separation process to realize
(1) Coping with reverberation exceeding the window length (= analysis frame length),
(2) coping with permutation problems,
(3) Dealing with input / output configurations that exceed two inputs / outputs,
At present, sufficient solutions to these various problems have not been presented.
JP 2006-238409 A JP2003-271168A "Introduction and Independent Component Analysis" (Noboru Murata, Tokyo Denki University Press) JP 2002-342198 “Detailed analysis of independent components” (Aapo Hyvarinen et al., Tokyo Denki University Press) “Study on blind sound source separation by subband processing” Akiko Araki, Robert Aichner, Shoji Makino, Takeki Nishikawa, Hiroshi Saruwatari Proceedings of the Acoustical Society of Japan March 2002 pp. 619--620 “Optimization of the number of band division in Blind Source Separation using band division type ICA” Takeki Nishikawa, Akiko Araki, Shoji Makino, Hiroshi Saruwatari Proceedings of the Acoustical Society of Japan, March 2001 pp. 569--570 Servier, C.I. Separation of speech signals under reverberant conditions. In Proc. EUSIPCO04, pp. 1693-1696 (2004)

  The present invention has been made in view of such a situation, and realizes high-accuracy separation processing for each signal by using independent component analysis (ICA) for a sound signal in which a plurality of signals are mixed. Signal separation device, signal separation method, and computer program, and in particular, a signal separation device and a signal separation method with improved separation accuracy for convolutional mixing with a large number of taps, An object is to provide a computer program.

The first aspect of the present invention is:
A signal separation device that inputs a signal in which a plurality of signals are mixed and separates them into individual signals,
A signal conversion means for converting the input signal into the time-frequency domain and generating an observed signal spectrogram;
Signal separation means for generating a signal separation result from the observed signal spectrogram generated by the signal conversion means,
The signal separating means includes
The signal separation device is configured to interpret the observation signal spectrogram as an observation signal mixed in the time-frequency domain and generate a signal separation result by executing a process for solving the time-frequency domain convolutional mixing. .

  Furthermore, in one embodiment of the signal separation device of the present invention, the signal conversion means performs a short-time Fourier transform (STFT) on the input signal to convert it into the time-frequency domain and generate an observation signal spectrogram. It is the structure which performs.

  Furthermore, in one embodiment of the signal separation device of the present invention, the signal separation means convolves the separation signal Y (t) of the frame number (t) with the observation signals X (t−L ′) to X (t). The configuration is characterized in that the signal separation result is generated by a process which is set as mixing and increases the independence of each of the individual signal components Y1 (t) to Yn (t) included in the separation signal Y (t). To do.

  Furthermore, in one embodiment of the signal separation device of the present invention, the signal separation means increases the independence of each of the individual signal components Y1 (t) to Yn (t) included in the separated signal Y (t). As a process to increase, a configuration in which a Kullback-Leiblar information amount I (Y) that is an independence calculation scale is applied, and a signal separation result is generated by updating a separation matrix that minimizes the Kullback-Leiblar information amount I (Y). It is characterized by being.

  Furthermore, in one embodiment of the signal separation device of the present invention, the signal separation means generates a first signal separation result by processing applying instantaneous mixed ICA (Independent Component Analysis) to the observed signal spectrogram. , Unnecessary channel removal processing determined not to correspond to any sound source from the first signal separation result, and processing for solving the convolutional mixture in the time-frequency domain for the observed signal spectrogram remaining after the removal processing And generating a signal separation result.

  Furthermore, in one embodiment of the signal separation device of the present invention, the processing using the instantaneous mixing ICA generates a separation signal in the time frequency domain from the observation signal in the time frequency domain and the separation matrix, Modify the separation matrix until the separation signal and the separation matrix calculated by the multidimensional score function derived from the multidimensional probability density function converge, and apply the modified separation matrix to the time-frequency domain separation signal. It is a process to generate.

Furthermore, the second aspect of the present invention provides
A signal separation device that inputs a signal in which a plurality of signals are mixed and separates them into individual signals,
First signal conversion means for converting an input signal into a time-frequency domain and generating an observation signal spectrogram;
Second signal conversion means for performing data conversion on the observed signal spectrogram generated by the first signal conversion means to generate a modulation spectrogram;
Signal separation means for generating a signal separation result from the modulation spectrogram generated by the second signal conversion means;
The signal separating means includes
In the signal separation device, the modulation spectrogram is interpreted as instantaneous mixing and a signal separation result is generated.

  Furthermore, in one embodiment of the signal separation device of the present invention, the first signal conversion means performs a short-time Fourier transform (STFT) on the input signal to convert it into a time-frequency domain and generate an observation signal spectrogram. It is the structure which performs the process to perform.

  Furthermore, in one embodiment of the signal separation device of the present invention, the second signal conversion means generates a modulation spectrogram as a result of performing a short-time Fourier transform (STFT) in the time direction on the observed signal spectrogram. The signal separation means is configured to generate a signal separation result by a process for increasing the independence of each of the signal components Y1 ′ to Yn ′ corresponding to the separation signal included in the modulation spectrogram. To do.

  Furthermore, in one embodiment of the signal separation device of the present invention, the signal separation means is a process for increasing the independence of each of the signal components Y1 ′ to Yn ′ corresponding to the separated signal, and is a Kullback-Leiblar which is an independence calculation measure. A signal separation result is generated by updating a separation matrix that applies the information amount and minimizes the Kullback-Leiblar information amount.

  Furthermore, in an embodiment of the signal separation device of the present invention, the signal separation device further performs inverse Fourier transform on each of the signal components Y1 ′ to Yn ′ corresponding to the separation signal obtained by the signal separation means. And an inverse Fourier transform means for generating spectrograms Y1 to Yn corresponding to the separated signals.

  Furthermore, in one embodiment of the signal separation device of the present invention, the signal separation device further applies instantaneous mixed ICA (Independent Component Analysis) to the observation signal spectrogram generated by the first signal conversion means. The first signal separation result is generated by the processing, and unnecessary channel removal means for executing an unnecessary channel removal process determined from the first signal separation result as not corresponding to any sound source is provided, and the second signal The conversion means and the signal separation means generate a signal separation result by executing only the processing on the signal after unnecessary channel removal.

  Furthermore, in one embodiment of the signal separation device of the present invention, the processing using the instantaneous mixing ICA generates a separation signal in the time frequency domain from the observation signal in the time frequency domain and the separation matrix, Modify the separation matrix until the separation signal and the separation matrix calculated by the multidimensional score function derived from the multidimensional probability density function converge, and apply the modified separation matrix to the time-frequency domain separation signal. It is a process to generate.

Furthermore, the third aspect of the present invention provides
A signal separation device that inputs a signal in which a plurality of signals are mixed and separates them into individual signals,
A signal conversion means for converting the input signal into the time-frequency domain and generating an observed signal spectrogram;
Signal separation means for generating a signal separation result from the observed signal spectrogram generated by the signal conversion means,
The signal separating means includes
The observed signal spectrogram is shifted in the frame direction to generate an observed signal spectrogram shift set in which data having different shift amounts are stacked, and an instantaneous mixed ICA (Independent Component Analysis) is generated for the generated observed signal spectrogram shift set. ) To generate a signal separation result by the processing applied.

  Furthermore, in one embodiment of the signal separation device of the present invention, the processing using the instantaneous mixing ICA generates a separation signal in the time frequency domain from the observation signal in the time frequency domain and the separation matrix, Modify the separation matrix until the separation signal and the separation matrix calculated by the multidimensional score function derived from the multidimensional probability density function converge, and apply the modified separation matrix to the time-frequency domain separation signal. It is a process to generate.

  Furthermore, in one embodiment of the signal separation device of the present invention, the signal separation means is adapted for a plurality of channels in which a plurality of observation signal spectrogram shift sets generated corresponding to the observation signals of a plurality of signal input sources are stacked. A signal separation result is generated by applying instantaneous mixing ICA to the observed signal spectrogram shift set.

  Furthermore, in one embodiment of the signal separation device of the present invention, the signal separation means sets the gap generated during the shift by copying zero or a value close to zero or values at both ends of the observed signal spectrogram. Then, the observation signal spectrogram shift set is generated.

  Furthermore, in one embodiment of the signal separation device of the present invention, the signal separation means executes a cyclic shift process of copying the data at one end that protrudes from the shift to the other end.

  Furthermore, in one embodiment of the signal separation device of the present invention, the signal separation means sets the minimum shift amount as 0 and the maximum shift amount as the number of frame taps [L ′] when generating the separation result from the observation signal. A plurality of shift data is generated, and an observation signal spectrogram shift set in which the generated data having different shift amounts are stacked is generated.

  Furthermore, in one embodiment of the signal separation device of the present invention, the signal separation means generates the observation signal spectrogram shift set by changing the number of frame taps [L ′] according to the frequency. .

  Furthermore, in one embodiment of the signal separation device of the present invention, the signal separation means generates a first signal separation result by processing applying instantaneous mixed ICA (Independent Component Analysis) to the observed signal spectrogram. Then, an unnecessary channel removal process determined not to correspond to any sound source is executed from the first signal separation result, and an observation signal spectrogram shift set remaining after the removal process is shifted in the frame direction to obtain an observation signal spectrogram shift set. It is the structure which produces | generates a signal separation result by applying instantaneous mixing ICA with respect to the produced | generated observation signal spectrogram shift set.

Furthermore, the fourth aspect of the present invention provides
A signal separation device that inputs a signal in which a plurality of signals are mixed and separates them into individual signals,
A signal conversion means for converting the input signal into the time-frequency domain and generating an observed signal spectrogram;
Signal separation means for generating a signal separation result from the observed signal spectrogram generated by the signal conversion means,
The signal separating means includes
For the observed signal spectrogram, signal separation results Y1 to Yn are generated by processing using instantaneous mixed ICA (Independent Component Analysis),
A signal spectrogram corresponding to each of the signal separation results Y1 to Yn is shifted in the frame direction to generate an observation signal spectrogram shift set in which data having different shift amounts are stacked, and for the generated observation signal spectrogram shift set The signal is characterized in that it performs a dereverberation process by a process to which instantaneous mixing ICA (Independent Component Analysis) is applied, and generates a signal separation result from which the dereverberation is removed by an integration process of the spectrogram after dereverberation. In the separation device.

  Furthermore, in one embodiment of the signal separation device of the present invention, the processing using the instantaneous mixing ICA generates a separation signal in the time frequency domain from the observation signal in the time frequency domain and the separation matrix, Modify the separation matrix until the separation signal and the separation matrix calculated by the multidimensional score function derived from the multidimensional probability density function converge, and apply the modified separation matrix to the time-frequency domain separation signal. It is a process to generate.

Furthermore, the fifth aspect of the present invention provides
In the signal separation device, a signal separation method for executing a process of inputting a signal obtained by mixing a plurality of signals and separating the signal into individual signals,
A signal conversion step in which the signal conversion means converts the input signal into the time-frequency domain and generates an observation signal spectrogram;
The signal separation means includes a signal separation step of generating a signal separation result from the observed signal spectrogram generated in the signal conversion step;
The signal separation step includes
The signal separation method is a step of interpreting the observed signal spectrogram as an observation signal mixed in the time-frequency domain and generating a signal separation result by executing a process for solving the time-frequency domain convolutional mixture. .

  Furthermore, in an embodiment of the signal separation method of the present invention, the signal converting step performs a short-time Fourier transform (STFT) on the input signal to convert it to a time frequency domain to generate an observation signal spectrogram. It is the step which performs.

  Furthermore, in one embodiment of the signal separation method of the present invention, the signal separation step convolves the separation signal Y (t) of the frame number (t) with the observation signals X (t−L ′) to X (t). It is a step of generating a signal separation result by a process of setting as mixing and increasing the independence of each of the individual signal components Y1 (t) to Yn (t) included in the separation signal Y (t). To do.

  Furthermore, in one embodiment of the signal separation method of the present invention, the signal separation step is performed to determine the independence of each of the individual signal components Y1 (t) to Yn (t) included in the separated signal Y (t). As a process of increasing, applying the Kullback-Leiblar information amount I (Y), which is an independence calculation scale, and generating a signal separation result by updating a separation matrix that minimizes the Kullback-Leiblar information amount I (Y). Features.

  Furthermore, in one embodiment of the signal separation method of the present invention, the signal separation step generates a first signal separation result by processing applying instantaneous mixed ICA (Independent Component Analysis) to the observed signal spectrogram. , Unnecessary channel removal processing determined not to correspond to any sound source from the first signal separation result, and processing for solving the convolutional mixture in the time-frequency domain for the observed signal spectrogram remaining after the removal processing It is a step of executing and generating a signal separation result.

  Furthermore, in one embodiment of the signal separation method of the present invention, the process using the instantaneous mixing ICA generates a time frequency domain separation signal from the time frequency domain observation signal and the separation matrix, and generates the generated time frequency domain separation signal. Modify the separation matrix until the separation signal and the separation matrix calculated by the multidimensional score function derived from the multidimensional probability density function converge, and apply the modified separation matrix to the time-frequency domain separation signal. It is a process to generate.

Furthermore, the sixth aspect of the present invention provides
In the signal separation device, a signal separation method for executing a process of inputting a signal obtained by mixing a plurality of signals and separating the signal into individual signals,
A first signal converting step in which a first signal converting means converts an input signal into a time-frequency domain and generates an observed signal spectrogram;
A second signal converting step in which a second signal converting means performs data conversion on the observed signal spectrogram generated in the first signal converting step to generate a modulation spectrogram;
A signal separation step of generating a signal separation result from the modulation spectrogram generated in the second signal conversion step;
The signal separation step includes
In the signal separation method, the modulation spectrogram is interpreted as instantaneous mixing to generate a signal separation result.

  Furthermore, in one embodiment of the signal separation method of the present invention, the first signal conversion step performs short-time Fourier transform (STFT) on the input signal to convert it to the time-frequency domain to generate an observation signal spectrogram. It is the step which performs the process to perform.

  Furthermore, in one embodiment of the signal separation method of the present invention, the second signal conversion step generates a modulation spectrogram as a result of performing a short-time Fourier transform (STFT) in the time direction on the observed signal spectrogram. The signal separation step is characterized in that a signal separation result is generated by a process of increasing the independence of each of the signal components Y1 ′ to Yn ′ corresponding to the separation signal included in the modulation spectrogram.

  Furthermore, in one embodiment of the signal separation method of the present invention, the signal separation step is a process of increasing the independence of each of the signal components Y1 ′ to Yn ′ corresponding to the separated signal, and is a Kullback-Leiblar which is an independence calculation measure. The information amount is applied, and a signal separation result is generated by updating a separation matrix that minimizes the Kullback-Leiblar information amount.

  Furthermore, in one embodiment of the signal separation method of the present invention, the signal separation method further includes: an inverse Fourier transform unit for each of the signal components Y1 ′ to Yn ′ corresponding to the separation signal obtained in the signal separation step. And performing an inverse Fourier transform to generate spectrograms Y1 to Yn corresponding to the separated signals.

  Furthermore, in one embodiment of the signal separation method of the present invention, the signal separation method is further characterized in that the unnecessary channel removing means performs instantaneous mixed ICA (Independent) on the observed signal spectrogram generated by the first signal converting means. A first signal separation result is generated by a process applying Component Analysis), and an unnecessary channel removal step is executed to execute an unnecessary channel removal process that is determined not to correspond to any sound source from the first signal separation result. The second signal conversion unit and the signal separation unit generate a signal separation result by executing only the processing on the signal after unnecessary channel removal.

  Furthermore, in one embodiment of the signal separation method of the present invention, the process using the instantaneous mixing ICA generates a time frequency domain separation signal from the time frequency domain observation signal and the separation matrix, and generates the generated time frequency domain separation signal. Modify the separation matrix until the separation signal and the separation matrix calculated by the multidimensional score function derived from the multidimensional probability density function converge, and apply the modified separation matrix to the time-frequency domain separation signal. It is a process to generate.

Furthermore, the seventh aspect of the present invention provides
In the signal separation device, a signal separation method for inputting a signal in which a plurality of signals are mixed and separating them into individual signals,
A signal conversion step in which the signal conversion means converts the input signal into the time-frequency domain and generates an observation signal spectrogram;
The signal separation means includes a signal separation step of generating a signal separation result from the observed signal spectrogram generated in the signal conversion step;
The signal separation step includes
The observed signal spectrogram is shifted in the frame direction to generate an observed signal spectrogram shift set in which data having different shift amounts are stacked, and an instantaneous mixed ICA (Independent Component Analysis) is generated with respect to the generated observed signal spectrogram shift set. The signal separation method is a step of generating a signal separation result by the processing to which (1) is applied.

  Furthermore, in one embodiment of the signal separation method of the present invention, the process using the instantaneous mixing ICA generates a time frequency domain separation signal from the time frequency domain observation signal and the separation matrix, and generates the generated time frequency domain separation signal. Modify the separation matrix until the separation signal and the separation matrix calculated by the multidimensional score function derived from the multidimensional probability density function converge, and apply the modified separation matrix to the time-frequency domain separation signal. It is a process to generate.

  Furthermore, in one embodiment of the signal separation method according to the present invention, the signal separation step corresponds to a plurality of channels in which a plurality of observation signal spectrogram shift sets generated corresponding to the observation signals of the plurality of signal input sources are stacked. A signal separation result is generated by applying instantaneous mixing ICA to the observed signal spectrogram shift set.

  Further, in one embodiment of the signal separation method of the present invention, in the signal separation step, the gap generated during the shift is set by copying zero or a value close to zero or values at both ends of the observed signal spectrogram. Then, the observation signal spectrogram shift set is generated.

  Furthermore, in one embodiment of the signal separation method of the present invention, the signal separation step is characterized in that a cyclic shift process is performed in which the data at one end protruding from the shift is copied to the other end.

  Furthermore, in one embodiment of the signal separation method of the present invention, in the signal separation step, the minimum shift amount is set to 0, and the maximum shift amount is set as the number of frame taps [L ′] when the separation result is generated from the observation signal. A plurality of shift data is generated, and an observation signal spectrogram shift set in which the generated data having different shift amounts are stacked is generated.

  Furthermore, in one embodiment of the signal separation method of the present invention, the signal separation step generates the observed signal spectrogram shift set by changing the number of frame taps [L ′] according to the frequency. .

  Furthermore, in one embodiment of the signal separation method of the present invention, the signal separation step generates a first signal separation result by processing applying instantaneous mixed ICA (Independent Component Analysis) to the observed signal spectrogram. Then, an unnecessary channel removal process determined not to correspond to any sound source is executed from the first signal separation result, and an observation signal spectrogram shift set remaining after the removal process is shifted in the frame direction to obtain an observation signal spectrogram shift set. The step of generating a signal separation result by applying the instantaneous mixing ICA to the generated observed signal spectrogram shift set.

Furthermore, the eighth aspect of the present invention provides
In the signal separation device, a signal separation method for inputting a signal in which a plurality of signals are mixed and separating them into individual signals,
A signal conversion step in which the signal conversion means converts the input signal into the time-frequency domain and generates an observation signal spectrogram;
The signal separation means includes a signal separation step of generating a signal separation result from the observed signal spectrogram generated in the signal conversion step;
The signal separation step includes
For the observed signal spectrogram, signal separation results Y1 to Yn are generated by processing using instantaneous mixed ICA (Independent Component Analysis),
A signal spectrogram corresponding to each of the signal separation results Y1 to Yn is shifted in the frame direction to generate an observation signal spectrogram shift set in which data having different shift amounts are stacked, and for the generated observation signal spectrogram shift set The signal is characterized in that it is a step of performing a dereverberation process by a process to which instantaneous mixing ICA (Independent Component Analysis) is applied, and generating a signal separation result from which the dereverberation is removed by an integration process of the dereverberation spectrogram. In the separation method.

  Furthermore, in one embodiment of the signal separation method of the present invention, the process using the instantaneous mixing ICA generates a time frequency domain separation signal from the time frequency domain observation signal and the separation matrix, and generates the generated time frequency domain separation signal. Modify the separation matrix until the separation signal and the separation matrix calculated by the multidimensional score function derived from the multidimensional probability density function converge, and apply the modified separation matrix to the time-frequency domain separation signal. It is a process to generate.

Furthermore, the ninth aspect of the present invention provides
In the signal separation device, a computer program for executing a signal separation process for inputting a signal obtained by mixing a plurality of signals and separating the signal into individual signals,
A signal conversion step for causing the signal conversion means to convert the input signal into the time-frequency domain and generate an observed signal spectrogram;
A signal separation step for causing the signal separation means to generate a signal separation result from the observed signal spectrogram generated in the signal conversion step;
The signal separation step includes
The computer program is a step of interpreting the observed signal spectrogram as an observation signal mixed in the time-frequency domain and generating a signal separation result by executing a process for solving the convolutional mixing in the time-frequency domain. .

Furthermore, the tenth aspect of the present invention provides
In the signal separation device, a computer program for executing a signal separation process for inputting a signal obtained by mixing a plurality of signals and separating the signal into individual signals,
A signal conversion step of causing the first signal conversion means to convert the input signal into the time-frequency domain and generate an observation signal spectrogram;
A second signal converting step for causing the second signal converting means to perform data conversion on the observed signal spectrogram generated in the first signal converting step to generate a modulation spectrogram;
A signal separation step of causing a signal separation means to generate a signal separation result from the modulation spectrogram generated in the second signal conversion step;
The signal separation step includes
The computer program is a step of interpreting the modulation spectrogram as an instantaneous mixture and generating a signal separation result.

Furthermore, an eleventh aspect of the present invention is
In the signal separation device, a computer program for executing a signal separation process for inputting a signal obtained by mixing a plurality of signals and separating the signal into individual signals,
A signal conversion step for causing the signal conversion means to convert the input signal into the time-frequency domain and generate an observed signal spectrogram;
A signal separation step for causing the signal separation means to generate a signal separation result from the observed signal spectrogram generated in the signal conversion step;
The signal separation step includes
The observed signal spectrogram is shifted in the frame direction to generate an observed signal spectrogram shift set in which data having different shift amounts are stacked, and an instantaneous mixed ICA (Independent Component Analysis) is generated for the generated observed signal spectrogram shift set. ) Is a step of generating a signal separation result by the processing applied.

  The computer program of the present invention is, for example, a storage medium or communication medium provided in a computer-readable format to a computer system capable of executing various program codes, such as a CD, FD, or MO. It is a computer program that can be provided by a recording medium or a communication medium such as a network. By providing such a program in a computer-readable format, processing corresponding to the program is realized on the computer system.

  Other objects, features, and advantages of the present invention will become apparent from a more detailed description based on embodiments of the present invention described later and the accompanying drawings. In this specification, the system is a logical set configuration of a plurality of devices, and is not limited to one in which the devices of each configuration are in the same casing.

  According to the configuration of an embodiment of the present invention, in a signal separation process in which an input signal mixed with a plurality of signals is converted into a time-frequency domain to generate an observation signal spectrogram, and a signal separation result is generated from the observation signal spectrogram. Interpret the signal spectrogram as an observation signal convoluted and mixed in the time-frequency domain, and generate a signal separation result by solving the convolution mixture in the time-frequency domain, or a short-time Fourier transform (STFT) in the time direction for the observed signal spectrogram Since the modulation spectrogram is generated by the above, and the modulation spectrogram is interpreted as instantaneous mixing and the signal separation result is generated, the delay amount for mixed sound signals with various delay amounts such as direct wave and reflected wave Highly accurate separation process considering There is realized.

  Hereinafter, a signal separation device, a signal separation method, and a computer program according to the present invention will be described in detail with reference to the drawings. The present invention performs signal separation processing for performing processing for separating and restoring the original signal by signal analysis of the mixed signal obtained by mixing a plurality of original signals as described above, and includes independent component analysis (ICA). : Independent Component Analysis).

  Specifically, as shown in FIG. 5, different sound is produced from N sound sources 111-1 to 111 -N, and these are observed by n microphones 121-1 to 121-n. , Signal separation processing by independent component analysis (ICA) is performed based on the mixed signal acquired by the microphones 121-1 to 121-n.

As described above, a signal (observation signal) observed by a certain microphone j (where 1 ≦ j ≦ n) is a convolution of the original signal and the transfer function as shown in the above-mentioned equation [1.1]. The calculation can be expressed as a summation expression for all sound sources ("convolution mixing"). Furthermore, when the observation signals for all the microphones 1 to n are expressed by one equation, it can be expressed as the above-described equation [1.2], and there are two methods for solving this convolution mixture, that is,
(1) Solve convolutional mixing directly in the time domain. (Time domain deconvolution)
(2) Convert the observed signal to the time-frequency domain and solve it as an instantaneous mixing problem.
There were these methods,
(2) Assuming that the process of converting the observed signal into the time-frequency domain and solving it as an instantaneous mixing problem is performed, in the conventional time-frequency domain ICA framework, the time-domain convolutional mixture is represented by instantaneous mixing in the time-frequency domain. I was thinking. In contrast, in the present invention, convolutional mixing is still considered in the time-frequency domain. This concept will be described with reference to FIG.

FIG. 6A shown in FIG. 6 is a vertically stacked spectrogram of the original signal, that is, the original signal output from each of the sound sources 111-1 to 111-N shown in FIG. The spectrogram of each sound source is S ₁ , S ₂ , and S is the result of vertically stacking both. In addition, as described above, the spectrogram is represented by | Xk (ω, t) |, which is the absolute value of Xk (ω, t), where t (frame number) is the horizontal axis and ω (frequency bin number) is the vertical axis. FIG.

  In the spectrogram of the original signal shown in FIG. 6A, a signal representing the signal of the t-th frame as a vector is S (t). Note that one frame of the spectrogram is called a spectrum.

  Conventionally, S (t) was considered to reach the microphone without frame delay, but in the present invention, it is considered that there is a frame delay. That is, referring to FIG. 5, vectors named spectrum are independently generated in each of the sound sources 111-1 to 111 -N, and they are microphones 121-1 to 121-1 as sensors with a delay of 0 or more. 121-n. These include direct waves and reflected waves.

  Various signals such as direct waves from different sound sources, direct waves and reflected waves, simple reflections and complex reflections will be acquired by the microphone, and it is estimated that there are various amounts of delay in the signals Is done. Here, assuming that the maximum value of the delay is L + 1, the influence of the spectrum S (t) which is the vector representation of the t-th frame signal in the spectrogram of the original signal shown in FIG. It will span the frame.

  FIG. 6B is a spectrogram of the observation signal. The observation signal spectrogram X generated by executing short-time Fourier transform (STFT) on the observation signals acquired by the microphones 121-1 to 121-n. is there.

The short-time Fourier transform (STFT) will be described with reference to FIG. For example, FIG. 7A shows an observation signal x _k recorded by the k-th microphone in the environment shown in FIG. The observed signal frame 171-173 is cut data cut out a predetermined length from x _k exerts a window function such as a Hanning window or sine window. The cut unit is called a frame. The length to be cut out (number of sample points) may be the same value as the length (in the vicinity of 512 points or 1024 points according to FIG. 3) at which the most accurate separation result is obtained in the time frequency domain ICA of the conventional method.
A spectrum Xk (t), which is data in the frequency domain, is obtained by performing discrete Fourier transform (Fourier transform in a finite interval; abbreviated DFT) or fast Fourier transform (FFT) on one frame of data ( t is a frame number).

  There may be overlap between frames to be cut out as in the frames 171 to 173 shown in the figure, so that the spectra Xk (t−1) to Xk (t + 1) of successive frames can be changed smoothly. Can do. A spectrum arranged in accordance with the frame number is called a spectrogram. FIG. 7B is an example of a spectrogram.

  If there is an overlap between frames to be cut out in the short-time Fourier transform (STFT), the inverse transform result (waveform) waveforms for each frame are also overlapped with each other in the inverse Fourier transform (FT). This is called overlap addition. The inverse transformation result may be obtained by applying a window function such as a sine window again before overlap addition, which is called weighted overlap add (WOLA). With WOLA, noise derived from discontinuity between frames can be reduced.

FIG. 6B is a spectrogram of observation signals obtained by processing with reference to FIG. 7, and the spectrograms are vertically stacked. The spectrogram of each sensor (microphone) is X ₁ , X ₂ , and X is the result of vertically stacking both. If the spectrogram of the observed signal is X, L + 1 frames from X (t) to X (t + L) are affected by the original signal spectrum S (t). Conversely, the observation signal X (t) of the t-th frame in the observation signal of FIG. 6B is affected by the original signal for the L + 1 frame before that.

Thus, considering that the observation signal X (t) of the t-th frame in the observation signal is influenced by the original signal for the L + 1 frame before that, the observation signal X (t) is It can be expressed by convolutional mixing as shown in the following formula [6.1].

Note that equation [6.1] is similar to equation [1.2] described above, but equation [6.1] is a time-frequency domain equation. When L = 0, this is equivalent to conventional instantaneous mixing. That is, when the observation signal X (t) is affected only by the original signal spectrum S (t), L = 0, which is equivalent to the conventional instantaneous mixing.
To distinguish between the convolutions of both,
L in Equation [1.2] is “the number of time taps”,
L in Equation [6.1] is “the number of frame taps”,
It is defined as

The above equation [6.1] is strictly established when the frame shift width is 1 in the STFT. Even when the frame shift width is set to 2 or more, it is approximately established. For details on this point, refer to the following literature which is the inventor's own paper.
[Hiroe, A. et al. Blind Vector Devolution: Convolutive Mixture Models in Short-Time Fourier Transform Domain. In M.M. E. Davis et al. (Eds.): ICA 2007, LNCS 4666, pp. 471-479, 2007. ]

  When the reverberation time is longer than the window length of the short-time Fourier transform (STFT), the reverberation effect is not completed in one frame but extends to a plurality of frames. Since reverberation across multiple frames can be expressed as convolution in the time-frequency domain, the reverberation exceeding the STFT window length should be eliminated by the concept of “convolution mixing in the time-frequency domain” introduced in the present invention. Is possible.

  That is, a graph plotting the relationship between the window length of the STFT and the separation accuracy of the time-frequency domain ICA described above in FIG. 3 will be described as an example. Instead of a long window (2048, 4096, etc.), a short window ( 512 and 1024) and a plurality of frame taps (16 and 32) can be combined, and the time span equivalent to a long window (the number of time taps, the shift width in the frame direction, and the frame is avoided while avoiding the trade-off of long windows) Time calculated from the number of taps) can be secured.

Further, since much fewer taps are required for convolution than the time domain deconvolution (in the order of several tens), the problem of time domain deconvolution can be avoided. In the following,
The number of frame taps when the observation signal is generated from the original signal is represented by the letter L,
on the other hand,
The number of frame taps when generating the separation result from the observation signal is represented as L ′.
L is a value determined from the reverberation time of the environment and the window length and shift width of the STFT. On the other hand, L ′ can be set to a value different from L. (If L ′ = 0, it is equivalent to the conventional method.)

The number L of frame taps of the observation signal can be calculated by the following formula.
L = Tr × Fs / S
However,
Tr: reverberation time of environment Fs: sampling frequency S: STFT shift width

For example,
Reverberation time Tr = 0.3 seconds,
Sampling frequency Fs = 16000 Hz,
Shift width S = 256,
Then,
The number L of frame taps when the observation signal is generated from the original signal is
L = 18.75.
That is, it can be seen that the effect of reverberation reaches 19 frames (rounded up).

If the number L ′ of frame taps for generating the separation result Y from the observation signal X, that is, the separation result Y in FIG. 6C from the observation signal X in FIG. , If the reverberation time is known)
L ′ = αL
(Α is an appropriate positive real number).
When L is unknown, L ′ can be determined by any of the following methods, for example.

The first method is to determine a fixed value such as L ′ = 64 or L ′ = 100. Basically, the amount of calculation increases as L ′ increases, so L ′ may be determined based on the balance between the amount of calculation and the separation performance.
The second method is a method in which the reverberation time is measured by some method, and a constant multiple of the value of L obtained from the reverberation time by the above formula is set to L ′, that is, L ′ = αL. As a method for measuring the reverberation time, for example, an impulsive sound is emitted from a speaker installed in the apparatus itself, and the time until the sound is sufficiently attenuated is measured.
The third method is to perform separation under various L's on the observed signal generated from the known original signal and adopt the value of L 'that gives the best separation result. For that purpose, for example, a plurality of speakers are installed around the device, known sounds are produced from each of them, and these sounds are observed by a plurality of microphones. For the observation result, a separation result is generated with different L ′ (for example, each value of L ′ = 0 to 100). A separation performance measure called SIR (signal-interference ratio), which will be described later, is calculated from the separation result and the original signal, and L ′ that gives the highest SIR is adopted. If the environment is the same, even if the original signal is unknown, its L ′ is likely to yield the best separated signal.

  For example, L ′, that is, the number of frame taps L ′ for generating the separation result Y from the observation signal X by any of the above methods, specifically, for example, from the observation signal X in FIG. The number L ′ of frame taps for generating the separation result Y in (c) is determined, and the separation result is generated from a plurality of consecutive frames of the observation signal using this number of frame taps L ′.

As a processing method for separating the observation signal mixed in the time-frequency domain, for example, any of the following methods can be applied.
(1) Solve convolutional mixing directly in the time-frequency domain.
(2) The spectrogram is again subjected to short-time Fourier transform (STFT) in the time direction and solved as instantaneous mixing.
(3) Solve by a process combining shift stacking and instantaneous mixing ICA.
Below, each method is demonstrated.

  The above-mentioned “(3) processing combining shift stacking and instantaneous mixing ICA” is a method for realizing separation processing equivalent to “(1) solving convolutional mixing directly in the time-frequency domain”. This is a technique of applying the conventional instantaneous frequency-domain mixed ICA to the result after stacking while shifting the spectrogram. Details will be described later.

(1) Directly solving convolutional mixing in the time-frequency domain First, processing for separating observation signals mixed in the time-frequency domain by directly solving convolutional mixing in the time-frequency domain will be described.

  A description will be given with reference to FIG. 6 again. As described above, the t-th frame S (t) of the original signal spectrogram affects the t-th to t + L-th frames of the observation signal. Therefore, in order to estimate one frame of the original signal, the observation signal is required for L frames or more. Let that value be L ′.

  When the t-th frame in the separated signal is used as a reference, for example, considering Y (t) in the separated signal in FIG. 6C as a reference, at least the following L + 1 is required to estimate S (t). Data for the frame is required. Therefore, convolutional mixing from the observed signal X (t) to X (t + L ′) is performed using Y (t), which is the estimation result (= separation result) of the original signal, as shown in Equation [6.3] above. Represent as

On the other hand, when the t + L′-th frame in the separated signal is used as a reference, for example, when Y (t + L ′) in the separated signal in FIG. Data for the immediately preceding L + 1 frame is required. Therefore, the separation signal Y (t) is expressed as a convolution mixture from X (t−L ′) to X (t) as shown in Equation [6.2].
Although both equations differ in frame deviation from S (t) but are essentially equivalent, a method for estimating Y (t) from equation [6.2] will be described below.

Assuming that mixing occurs only in the same frequency bin (ie, no frequency modulation occurs in the middle of propagation), Equation [6.1], an equation for mixing all frequency bins, is It can be expressed as equation [6.4], which is a mixing equation for each bin. Under the assumption, since the separation matrix W ^[l] of the equation [6.2] can be expressed as a matrix composed of diagonal matrices as in the equation [6.5], W ^[l] is estimated. In order to do this, it is only necessary to estimate the non-zero component of equation [6.5].

  The process for obtaining the learning rule (formula of ΔW) from the formula [6.2] is performed as follows. As a measure representing independence in the entire spectrogram, the Kullback-Leiblar information amount I (Y) calculated by the formula [4.5] is considered. This method is the same processing as described in JP-A-2006-238409.

In order to make Y1 (t) components Y1 (t) to Yn (t) independent from each other, a separation matrix W ^[1 ] that minimizes the Kullback-Leiblar information amount I (Y) in equation [4.5] ^{. 0] to} W ^{[L ′]} may be obtained. Since the method shown in Japanese Patent Laid-Open No. 2006-238409 is instantaneous mixing, it is sufficient to estimate only one separation matrix. However, in the present invention, since the convolutional mixing of L ′ + 1 frames is performed, the separation matrix is also L '+1 need to be estimated.

Here, in addition to the assumption that “Y1 (t) to Yn (t) are independent from each other” (independence between channels), it is assumed that “Yk (t−L ′) to Yk (t) are also independent from each other” (frame (Independence between), the learning rule of formula [7.1] shown below is finally derived.

That is, in order to obtain the separation matrix W ^{[0] to} W ^{[L ′]} , the equations [6.2], [7, 1], and [7.8] are changed from W ^{[0] to} W ^{[L ′].} Repeat until convergence (or a fixed number of times). However, ΔW ^[l] (ω) and W ^[l] (ω) in Expression [7, 1] are submatrices obtained by extracting elements corresponding to the frequency bin ω from ΔW ^[l] and W ^[l] , respectively. Equation [6.6]) and Rω ^[l] is a cross term calculated by Equation [7.2]. Φω (Y (t)) in equation [7.2] is a vector composed of a score function (equation [7.4]), which is shown in the earlier application (Japanese Patent Laid-Open No. 2006-238409) of the present applicant. It is the same as the vector consisting of The score function is defined as a logarithmic derivative of the probability density function (formula [7.5]), and prevents the occurrence of permutation by using a multivariate score function as disclosed in JP-A-2006-238409. it can.

  A specific example of the score function may be the same as that described in JP-A-2006-238409, and for example, Equation [7.6] is used. In this equation, αk (ω), m, γk (ω) are positive real numbers, and βk (ω) is a non-negative real number. As a simple example, equation [7.7] may be applied.

In equation [7.8], η is a positive real number called a learning rate. For example, η may be a constant such as 0.1, but may be calculated as shown in Equation [7.9]. In this equation, ‖W (ω) ‖ is also the sum of squares of all elements of W ^[0] (ω) to W ^[L] (ω) (equation [7.10]), and ‖ΔW (ω) ‖. Similarly, the sum of squares of all elements of ΔW ^[0] (ω) to ΔW ^{[ L ′]} (ω), η ₀ is a positive real number representing the upper limit value of η. By using Equation [7.8], η becomes a relatively small value at the beginning of learning (because ‖ΔW (ω) 大 き い is large), and overflow of W (ω) can be avoided. On the other hand, at the end of learning, η becomes a relatively large value (because ‖ΔW (ω) ‖ is close to the zero matrix), and W (ω) converges quickly to the target value.

In addition, when the formula [6.3] is used instead of the formula [6.2], the formulas [6.3], [7, 1], and [7.8] are repeated in learning. However, equation [7.3] is used instead of equation [7.2] as Rω ^[l] in equation [7.1].

In deriving equation [7.1] above, the assumption was made that “Yk (t−L ′) to Yk (t) are also independent of each other”, but instead “Yk (t−L ′) to When the assumption that “Yk (t) depends on each other” is set, another learning rule [8.1] shown below is obtained (equation [7.1] is common).

  The difference between the formula [7.2] and the formula [8.1] is in the argument of the score function. The formula [7.2] uses only Y (t) as an argument whereas the formula [8.1] Uses Y (t−L ′) to Y (t) as arguments. This score function is defined by Equation [8.4], and P (Yk (t),..., Yk (t−L ′)) appearing in this equation is the data of adjacent L ′ + 1 frames. It represents the probability of simultaneous occurrence. Therefore, when Expression [8.1] is used, the dependency relationship between adjacent frames can be further reflected in the separation matrix. As an example of the score function, Expression [8.5] (Expression [8.6] is a specific example thereof) can be given.

  In addition, Formula [8.1] is a formula corresponding to Formula [6.2]. When equation [6.3] is used instead of equation [6.2], equation [8.2] corresponds.

  In the above description, the Kullback-Leiblar information amount is adopted as the measure of independence, but other measures may be used. The scales representing independence other than the Kullback-Leiblar information amount include non-normality and kurtosis, and the separation matrix may be updated so that these amounts are maximized or minimized.

(2) Analyzing the spectrogram once again in the time direction and solving it as instantaneous mixing Next, short-time Fourier transform (STFT) of the spectrogram in the time direction once again and solving it as instantaneous mixing, convolution mixed observation in the time frequency domain Processing for separating signals will be described.

When convolution is short-time Fourier transformed (STFT) with a window length longer than the number of taps, the convolution is converted into a simple product. The same applies to convolutional mixing in the time frequency domain. That is, when the above formula [6.4], which is convolutional mixing in the time-frequency domain, is again subjected to short-time Fourier transform (STFT) in the time direction, the following formula [9.1] is obtained. However, X ′, A ′, and S ′ are the results of short-time Fourier transform (STFT) of each element of X, A, and S in Equation [6.4].

  The above equation [9.1] is an instantaneous mixing equation, and in order to separate the observation signal into independent components, the equation [9.2] may be considered.

  Here, the conversion from the spectrogram X to X ′ (modulation spectrogram) which has been subjected to short-time Fourier transform (STFT) in the time direction again will be described with reference to FIGS. 8 and 9. For comparison, the conversion from the waveform x to the spectrogram X will also be described.

FIG. 8A shows the waveform of the observation signal (in this figure, the number of channels = 2,
The number of channels is arbitrary).
FIG. 8B is a spectrogram generated by short-time Fourier transform (STFT) of the waveform of the observation signal (FIG. 8A) (STFT is performed for each channel, and the results are arranged vertically. Displayed). N frequency components are obtained when Fourier transform is performed with window length = N. However, in the conversion of real number data, since the negative frequency component is related to the conjugate complex number of the positive frequency component (referred to as conjugate symmetry), the direct current component And only N / 2 + 1 = M frequency bins of the positive frequency component. A frequency bin 201 shown in FIG. 8B represents one of the frequency bins. Normally, the spectrogram indicates a plot of the absolute value of X, but here X is also called a spectrogram. The same applies to the original signal S and the separation result Y.

  Here, for the spectrogram X shown in FIG. 8B, a short-time Fourier transform (STFT) is performed for each frequency bin. Data generated by performing STFT on the spectrogram again is called a modulation spectrogram. When the window length of the second short-time Fourier transform (STFT) is L ′, one frequency bin, for example, L ′ bins is generated from the bin 201 in FIG. It expresses with. This is the bin 202 shown in FIG. 9C, and the result of accumulating these is the data shown in FIG. 9C.

  That is, FIG. 9C is a modulation spectrogram generated by performing short-time Fourier transform (STFT) for each frequency bin for the spectrogram X shown in FIG. 8B, as shown in FIG. 9C. It is represented by a modulation spectrogram X ′ of a rectangular parallelepiped structure. Although the depth direction is also a frequency component, it is not a waveform frequency component but an envelope frequency component. In the second short-time Fourier transform (STFT), the data before conversion is also a complex number, so the conversion result is not conjugate symmetric. Therefore, all L ′ bins must be considered.

  The newly generated bin is arranged in the vertical direction instead of the depth direction. That is, when the bin 202 shown in FIG. 9C is arranged like the bin 203 shown in FIG. 9D, the modulation spectrogram can also be expressed in a plane as shown in FIG. 9D. It should be noted that the modulation spectrogram X ′ shown in FIG. 9 (d) is similar to the spectrogram X shown in FIG. 8 (b), but the meaning of the frequency bin is different. (The number of bins per channel is L ′ for the spectrogram X shown in FIG. 8B and M × L ′ for the spectrogram X ′ shown in FIG. 9D.

Here, the mathematical formula [9. Returning to [n], the three-dimensional modulation spectrogram X ′ shown in FIG. 9C corresponds to X ′ in the equations [9.1] and [9.2], where ω is the vertical frequency. The bin is represented by ω ₂ in the depth direction. In the equation [9.2], when the pair (ω, ω ₂ ) is collectively represented by the index ω ′, the equation [9.3] is obtained. Equation [9.3] corresponds to the planar modulation spectrogram X ′ shown in FIG.

  In order to obtain the learning rule (ΔW equation) from Equation [9.2] or Equation [9.3], the following is considered. As a measure representing independence in the entire modulation spectrogram, the Kullback-Leiblar information amount calculated by Equation [9.5] is considered. This equation is almost the same as equation [4.5], but H (Yk ′) is the entropy calculated from the modulation spectrogram for one channel, and H (Y ′) is the simultaneous entropy calculated from the entire modulation spectrogram. is there. A method of calculating H (Y′k) will be described with reference to FIG.

  FIG. 10 corresponds to the three-dimensional modulation spectrogram X ′ shown in FIG. That is, for example, for the spectrogram X shown in FIG. 8B generated by performing the short-time Fourier transform (STFT) on the waveform of the observation signal (FIG. 8A), the short-time Fourier is further provided for each frequency bin. This corresponds to a modulation spectrogram generated by performing transformation (STFT). In the three-dimensional modulation spectrogram X ′ shown in FIG. 10, for example, in the entropy calculation of the first channel, the modulation spectrogram Y1 ′ (t) 221 of the first frame in FIG. 10 represents a plane, The entropy H (Y1 ′) 223 is obtained by substituting Y1 ′ (t) into the multivariate probability density function P (Y1 ′ (t)) 222 having the argument as a parameter.

  Formula [9.3] is the same as Formula [3.5] except for the difference in variable names. Therefore, in order to derive the learning rule, the variable name of the equation [5.2] may be changed, and the equation [9.5] is obtained as a result. That is, if the expressions [9.3], [9.5], and [9.6] are repeated until W ′ converges, Y1 ′ (t) to Yn ′ (t) become independent from each other.

  When inverse Fourier transform and overlap add are applied to the modulation spectrograms Y1 ′ to Yn ′ that are independent of each other, spectrograms Y1 to Yn that are independent of each other are obtained.

In the above description, the Kullback-Leiblar information amount is adopted as a measure of independence, but other measures may be used as in (1).
In addition, although an equation based on the natural gradient method is derived as an equation for updating the separation matrix, another algorithm may be used instead. Other algorithms include a gradient method with an orthonormal constraint, a fixed point method, and a Newton method, which are the same as those of the conventional instantaneous mixing ICA.

(3) Solve by a process combining shift stacking and instantaneous mixing ICA.
Next, a process for separating the observation signal mixed by convolution in the time-frequency domain by a process combining shift stacking and instantaneous mixing ICA will be described.

  This third processing method is a method that realizes a separation process substantially equivalent to the previously described [(1) method for directly solving convolutional mixing in the time-frequency domain]. This is realized by utilizing the instantaneous mixing ICA process disclosed in JP-A-2006-238409.

  In this method, for example, the observation signal spectrograms are stacked while shifting, and then the instantaneous frequency ICA is mixed with the result, that is, the instantaneous time disclosed in Japanese Patent Application Laid-Open No. 2006-238409, the applicant's earlier patent application. Realized by applying mixed ICA. By applying this third method, the permutation (replacement) problem is solved, and the mixed sound signal with various delay amounts such as direct waves and reflected waves is processed with high accuracy in consideration of the delay amount. Is realized.

  First, before describing the third method, the permutation (replacement) problem that occurs in the separation process of the observation signal, and Japanese Patent Application Laid-Open No. 2006-238409, which is an earlier patent application of the present applicant that solved this problem. The outline of the instantaneous mixing ICA will be briefly described again.

  When s1 to sn are independent original signals emitted from n sound sources and s is a vector having these elements as s, the observed signal x observed by the microphone is expressed by the equation [1. 2] is subjected to the convolution / mixing operation. Next, short-time Fourier transform is performed on the observation signal x to obtain a signal X in the time-frequency domain. If the element of X is Xk (ω, t), Xk (ω, t) takes a complex value. A diagram expressing | Xk (ω, t) |, which is the absolute value of Xk (ω, t), with color shading is, for example, a spectrogram of the observation signal shown in FIG. 6B. This spectrogram is a spectrogram X of an observation signal generated by performing short-time Fourier transform (STFT) on the observation signals acquired by the microphones 121-1 to 121-n shown in FIG.

  In the spectrogram, for example, | Xk (ω, t) |, which is the absolute value of Xk (ω, t), is expressed by color shading, where the horizontal axis is t (frame number) and the vertical axis is ω (frequency bin number). FIG. Subsequently, the separation signal Y is obtained by multiplying each frequency bin of the signal X by the separation matrix W (ω). A separation signal y in the time domain can be obtained by inverse Fourier transforming the separation signal Y.

  However, as described above, in the conventional independent component analysis in the time-frequency domain, signal separation processing is performed for each frequency bin, and the relationship between the frequency bins is not considered. Therefore, even if the separation itself is successful, there is a possibility that scaling and separation destination inconsistency may occur among the frequency bins. The inconsistency of scaling can be solved by a method of estimating the observation signal for each sound source. However, the inconsistency of the separation destination, for example, the signal derived from S1 appears in Y1 when ω = 1, whereas it becomes Y1 when ω = 2. The permutation (replacement) problem that the signal derived from S2 appears cannot be solved.

Japanese Patent Application Laid-Open No. 2006-238409, the applicant's earlier patent application, disclosed a technique for solving this permutation (substitution) problem. That is, for the expression representing the separation in the entire spectrogram, the method of obtaining the separation matrix W that maximizes the independence in the entire spectrogram using the previously described equation [4.4] shown below was adopted.

  Specifically, as the independence of the entire spectrogram, a KL (Kullback-Leiblar) information amount I (Y) represented by the formula [4.5] is introduced, and a separation matrix W that minimizes I (Y) is obtained. Ask. The KL information amount I (Y) is obtained by subtracting the simultaneous entropy of the entire spectrogram from the total entropy of each spectrogram, and is minimum (ideally 0) when all spectrograms are independent of each other. Become.

In the definition formula [4.5] of the KL information amount I (Y), H (Y _k ) represents the entropy of one spectrogram for each channel, and H (Y) represents one spectrogram for all channels. Represents simultaneous entropy.

For example, the relationship between H (Y _k ) and H (Y) when n = 2 is as described above with reference to FIG. In FIG. 2, P (Y _k (t)) is a probability density function of Y _k (t), and H (Y _k ) is an entropy for one spectrogram for each channel. The Kullback-Leiblar information amount I (Y) is obtained by subtracting the simultaneous entropy 13 of the entire spectrogram from the sum of entropies 11 and 12 for each spectrogram, and is minimum (ideal) when all spectrograms are independent of each other. 0).

In order to minimize the KL information amount I (Y) in the entire spectrogram, as described above, the following equations [5.1] to [5.3] are repeated until W and Y converge.

  Note that ΔW (ω), W (ω), and Y (ω, t) appearing in Equation [5.3] are elements corresponding to the ω-th frequency bin from ΔW, W, and Y (t), respectively. This is the extracted submatrix. By doing so, it became possible to obtain separation results without permutation problems.

  The third processing method is a method of applying the time-frequency domain instantaneous mixing ICA disclosed in JP-A-2006-238409. Specifically, the process using the time-frequency domain instantaneous mixing ICA disclosed in Japanese Patent Laid-Open No. 2006-238409 is performed as a signal separation process from a separation matrix into which an observation signal in the time-frequency domain and an initial value are substituted. Generate a separation signal in the frequency domain, and correct the separation matrix until the generated separation signal in the time-frequency domain and the separation matrix calculated by the multidimensional score function derived from the multidimensional probability density function almost converge. Then, this modified separation matrix is applied to generate a time-frequency domain separation signal. Details are as disclosed in JP-A-2006-238409.

  In the third processing method described below, that is, [(3) separation of observation signals mixed in the time-frequency domain by processing combining shift accumulation and instantaneous mixing ICA, this is disclosed in JP-A-2006-238409. Apply the instantaneous frequency-domain mixed ICA. Specifically, for example, the observation signal spectrograms are stacked while being shifted, and then the instantaneous mixing ICA in the time-frequency domain is applied to the result. Hereinafter, the third method will be described.

In this technique, vectors stacked vertically while shifting the frame number are generated for the observed signal spectrograms of the plurality of microphones that are the voice input units. For example, with respect to the observation signal spectrogram of the k-th channel corresponding to the k-th microphone, that is, the observation signal spectrogram X _k (t) of the above equation [4.1], a vector vertically stacked while shifting the frame number is obtained. Think. Furthermore, consider a vector in which it is stacked for all channels. This is a vector X ″ (t) of the equation [11.1] shown below. The vector X ″ (t) in Expression [11.1] includes vectors for n channels. The vector for each channel is indicated as X _k ″ (t).

A procedure for generating the vector X ″ (t) represented by the above mathematical expression [11.1] will be described with reference to FIG. FIG. 11 is a diagram illustrating a process of generating a vector X ″ (t) for each channel from the observed signal spectrogram X _k (t) for each channel generated based on the input signal of each microphone. Figure 11 (a) to indicate the data 301, i.e. X _k is the observed signal one channel spectrogram, i.e. the observed signal spectrograms X _k of the k-th channel corresponding to the k-th microphone, FIG. 6 described above (b ) X1 and X2.

The result of shifting _Xk to the left by l (lower-case el) frames is _{defined as Xk} ^[l] . FIG. 11B shows a structure in which a plurality of observation signal spectrograms X _k are stacked vertically while the shift amount l is sequentially changed from 1 = 0 to L ′.
Data 311-0 is the shift amount l = 0,
Data 311-1 is a shift amount l = 1 l frame:
Data 311 -L ′ is the shift amount l = L ′ frame.
Note that L ′ is the number of frame taps when generating the separation result from the observation signal, as described above.

From one observation signal spectrogram, an observation signal spectrogram shift set having a shift amount in a plurality of different frame directions is generated, and this is set as an observation signal spectrogram shift set [X ″]. When one frame is cut out from the observed signal spectrogram shift set [X ″], a mathematical expression 312 shown in FIG. 11B is obtained. This mathematical expression corresponds to a vector [X _k (t)] corresponding to one channel included in the above-described expression [11.1].

As described above, Expression [11.1] is a vector composed of observation signal spectrogram shift sets generated by vertically stacking observation signal spectrograms X _k (t) corresponding to one channel while shifting frame numbers. Is a vector composed of observation signal spectrogram shift sets corresponding to a plurality of channels generated by stacking all channels.

  As shown in FIG. 11B, a value close to zero is assigned to the gap generated during the shift, that is, the hatched portion of the data 311-0 to 311-L ′ shown in FIG. Or the values at both ends (X (1), X (T), etc.) are copied and set. Alternatively, zero may be substituted if a zero division countermeasure described later is taken. Alternatively, the gap at both ends may be removed and data for an intermediate TL ′ frame may be used. Furthermore, instead of a normal shift process, a cyclic shift of length T (the left end data protruding at the shift is copied to the right end) may be applied. In the processing example described below, a processing example to which the observation signal spectrogram shift set [X ″] generated by the cyclic shift is applied will be described.

When the observed signal spectrogram shift set [X ″] generated by the shift process and the stack process as shown in FIG. 11B is compared with the original observed signal spectrogram [X],
The observed signal spectrogram [X] is the spectrogram for n channels,
Whereas
The observed signal spectrogram shift set [X ″] apparently includes spectrograms for n × (L ′ + 1) channels. n is the number of channels corresponding to the number of microphones, and (L ′ + 1) is the number of shift data set corresponding to one channel.

  Using this observed signal spectrogram shift set [X ″] as an observed signal spectrogram of n × (L ′ + 1) channels, the instantaneous mixing ICA disclosed in Japanese Patent Application Laid-Open No. 2006-238409, which is the applicant's earlier patent application, is applied. The separation process is performed by the method described above. By this processing, it is possible to perform separation equivalent to the above-described method of “(1) directly solving convolutional mixing in the time frequency domain”. Hereinafter, the principle will be described.

Regarding the observed signal spectrogram X, an operation of generating a separation result by convolving the tl (ell) th to tl (ell) + L'th frames will be considered. That is, as shown in FIG. 12, this is an operation for generating a separation result for one frame from the L ′ + 1 frame from X (t−l (el)) to X (t−l (el) + L ′). A calculation formula for obtaining the separation signal Y ^[l] (t) by this processing is expressed by Formula [11.2].

Let the separation result be Y ^[l] (t). Since the separation signal Y ^[l] (t) is a convolution between L ′ + 1 frames, L ′ + 1 coefficient matrices are required, but the value varies depending on the number of shift frames [l (L)]. Therefore, for the separation matrix [W], two subscripts are attached,
W ^{[l, 0] to} W ^{[l, L ']}
It expresses. That is, the separation matrix [W] is set according to the number of shift frames [l (el)] and each shift spectrogram.

Equation [11.3] and Equation [11.4] are details of the submatrix appearing in Equation [11.2], and Equation [11.5] is the detail of the submatrix appearing in Equation [11.4]. Is shown.
The separation signal [Y ^[l] (t)] and the separation matrix [W ^{[l, τ]} ] are each composed of a vector or matrix corresponding to the component of each channel. Note that the subscript τ for W is τ = 0 to L ′.

Here, represented by the formula [11.6],
Separation result: Y ^[0] (t) to Y ^{[ L ′]} (t)
A vector including all of the separation result vector Y ″ (t),
A plurality of separation matrices represented by Equation [11.7],
W ^{[0, 0] to} W ^{[L ′ + 1, L ′]}
A matrix that contains all W ''
Using these vectors [Y ″ (t)] and matrix [W ″], the equation indicating the separation process is simply the equation [11.8], that is,
Y ″ (t) = W ″ X ″ (t) ... [11.8]
It can be shown in this way.

In Patent Document 1 (Japanese Patent Laid-Open No. 2006-238409) described above as a conventional method, the expression [4.4] described above is used as an expression representing separation in the entire spectrogram, that is,
Y (t) = WX (t) ... [4.4]
Although the processing using the above formula is used, when the formula [11.8] and the formula [4.4] are compared, the formula [11.8] simply increases the number of channels from n to n × (L ′ + 1). As an example, it can be considered that the formula [4.4] is applied.

That is, as shown in FIG. 13, the observation signal spectrogram shift set [X ″] for a plurality of channels is composed of X ₁ ″ to X _n ″, and these are divided into n × (L ′ + 1) individual. Considering the observed signal spectrogram corresponding to the channel, it can be considered that the formula [11.8] is simply an increase of the number of channels from n to n × (L ′ + 1) and that the formula [4.4] is applied. it can.

  Therefore, the observation signal spectrogram X for n channels is expanded to n × (L ′ + 1) channels by the method described with reference to FIG. 11, and the observation signal spectrogram shift set [X ″] as a result is expanded. Thus, when the equations [5.1] to [5.3], which are learning equations of JP-A-2006-238409, are repeatedly applied, a separation result Y ″ and a separation matrix W ″ are obtained.

  However, in the equations [5.4] to [5.7], which are details of the variables of the equations [5.1] to [5.3], n should be read as n × (L ′ + 1). Further, k is not 1 ≦ k ≦ n but an index representing 1 ≦ k ≦ n × (L ′ + 1).

The separation result Y ″ includes spectrograms for n × (L ′ + 1) channels, but since the desired one is for n channels (or less than n), the spectrogram is selected as necessary. . As a selection method, components corresponding to a specific shift amount [l (L)] such as Y ₁ ^[0] , Y ₂ ^[0] ,..., Y _n ^[0] from the separation result Y ″. It is possible to apply a method such as leaving only.

Alternatively, as in the method for determining the value of L ′ for the number of frame taps when generating the separation result from the observation signal, an optimum shift amount [l (el)] in the frame direction is obtained using a known signal. Also good. That is, after a known signal is emitted from a speaker or the like and collected and separated by the method of the present invention, each of the separation results Y _k ^{[0] to} Y _k ^{[L ′]} is a measure of separation accuracy SIR ( signal-interference-ratio). Then, it is possible to perform processing such as selecting a separation result [Y _k ^[l] ] corresponding to the number of shifts: l (el) that provides the highest separation accuracy (SIR).

  FIG. 14 is a flowchart for explaining a sequence of processing for separating the observation signal mixed by convolution in the time-frequency domain by the processing combining (3) shift stacking and instantaneous mixing ICA. Processing of each step in the flow of FIG. 14 will be described.

  First, in step S11, observation signal spectrograms are stacked while being shifted. This process is the process described with reference to FIG. 11, and the observation signal spectrogram generated from the observation signal acquired by each microphone is sequentially shifted in units of shift frames (l (el)) as frames. Shift data is generated and accumulated until the number of taps reaches a shift amount that reaches L ′, and an observed signal spectrogram shift set [X ″] is generated.

  Next, in step S12, the separation result Y ″ is obtained using the instantaneous mixing ICA (or the changed score function). That is, the learning results of Japanese Patent Application Laid-Open No. 2006-238409, [5.1] to [5.3], are repeatedly applied to the observation signal spectrogram shift set [X ″] to obtain the separation result Y ′. 'And the separation matrix W' 'are calculated. However, in the equations [5.4] to [5.7], which are details of the variables of the equations [5.1] to [5.3], n should be read as n × (L ′ + 1). Further, k is not 1 ≦ k ≦ n but an index representing 1 ≦ k ≦ n × (L ′ + 1).

  The score function is defined as a logarithmic derivative of the probability density function, and is defined by the equation [5.7]. The multivariate score function is used as described in Japanese Patent Application Laid-Open No. 2006-238409, as described in Equation [7.5] in the previous [(1) Solve convolution mixture directly in time frequency domain] method. This prevents permutation from occurring. Processing using this score function will be described later.

  Next, in step S13, a desired spectrogram is selected from the separation result Y ″ as necessary. That is, as described above, Y ″ as the separation result includes spectrograms for n × (L ′ + 1) channels, but the desired one is for n channels (or less than n), so that it is necessary. The spectrogram is selected accordingly.

As a selection method, only components corresponding to a specific shift amount [l (L)] such as Y ₁ ^[0] , Y ₂ ^[0] ,..., Y _n ^[0] are selected from the separation result Y ″. It is possible to apply a method such as leaving At this time, it is possible to perform processing such as selecting a separation result [Y _k ^[l] ] corresponding to the number of shifts: l (el) that provides the highest separation accuracy (SIR).

The method described above is almost equivalent to the method using the equations [7.2] to [7.5] described in the [(1) Solve convolutional mixture directly in the time frequency domain] method. This is equivalent to executing the process. In other words, in the present processing example, the signals of n × (L ′ + 1) channels are separated so as to be independent from each other. For example, with reference to FIG. 13, observation signal spectrogram shifts for a plurality of channels are performed. A signal Y ₁ ^[0] 341 corresponding to _one spectrogram in the signal Y ^″ 331 which is a separation result obtained as a result of applying the set [X ″] is a signal Y _n ^[0 derived from another sound source. ^] 343 and Y _n ^{[L ′]} 344 as well as Y ₁ ^{[L ′]} 342 that should be derived from the same sound source.

  On the other hand, by changing the score function used in the above (formula [5.7] etc.), the formula [8.1 described above in the [(1) Directly solve convolution mixture in the time frequency domain] method is used. ] To processing equivalent to the method using the formula [8.4] can be performed.

[(1) In the time-frequency domain, the method using the equations [8.1] to [8.4] described in the method for directly solving convolutional mixing is “Yk (t−L ′) to Yk (t ) Is a process based on the assumption of “depending on each other”. Even in this processing example [(3) processing combining shift stacking and instantaneous mixing ICA], processing in consideration of the dependence of the separation result is possible. That is, with reference to FIG. 13, Y ₁ ^[0] 341 is independent from Y _n ^[0] 343 and Y _n ^{[L ′]} 344 and has a dependency relationship with Y ₁ ^{[L ′]} 342. There can be a separation. Below, the method is demonstrated.

In order to give a dependency relationship between the separation results Y _k ^[0] ,..., Y _k ^{[L ′]} derived from the same sound source, the ΔW (ω) calculation formula [5.2] described above is used. Instead, the following formula [12.1] is used.

However, Y ″ (ω, t) and W ″ (ω) in Equation [12.1] are vectors and matrices obtained by extracting the components of the ωth frequency bin from Y ″ and W ″, respectively. Yes, and expressed as Equation [12.3] and Equation [12.4]. φ _ω (Y ″ (t)) is a vector having n × (L ′ + 1) score functions as elements, as represented by Expression [12.5]. (Specific examples of the score function will be described later.)

The difference between Expression [12.5] and Expression [5.6] is in the argument of the score function. That is, when equation [5.6] is expanded to n × (L ′ + 1) channels, n × (L ′ + 1) score functions all take different arguments, whereas equation [12.5] In this case, φ _kω ^[0] (Y _k ″ (t)) to φ _kω ^{[L ′]} (Y _k ″ (t)) takes the same argument Y _k ″ (t), and therefore there are n types of arguments. It is.

The score function φ _kω ^[l] (Y _k ″ (t)) is multi-dimensional (multi- _valued ) with Y _k ″ (t) (that is, Y _k ^[0] ,..., Y _k ^{[L ′]} ) as an argument. (Variable) defined as the logarithmic derivative of the probability density function (Equation [12.5]). In this way, when a plurality of arguments are included in one probability density function and ICA learning is performed using a score function derived therefrom, the elements that are arguments have a dependency relationship (not independent). ) Is theoretically shown. That is, referring again to FIG. 13, the signal Y ₁ ″ 351, which is a set of Y ₁ ^[0] ,..., Y ₁ ^{[L ′]} , is dependent on each element in the set. , Independent of another set of signals, eg, Y _n ″ 352.

  Here, specific examples of the multidimensional probability density function and the score function will be described. One type of multidimensional probability density function is called a spherical distribution. This is generated by substituting the L2 norm of a vector into a function that takes a scalar as an argument, as shown in Equation [13.1] below (“∝” represents proportionality).

  The L2 norm is the square root of the sum of squares (absolute value) of each element, and can be obtained by substituting 2 for m in Equation [13.2]. As an example of the spherical distribution, when an index distribution (γ is a positive real number) such as Equation [13.3] is used, Equation [13.4] is derived as a corresponding score function. This equation may be substituted into [12.5].

It should be noted that the expression [13.4] may be changed similarly to the expression [7.6] described above in the [(1) Solve convolutional mixture directly in the time frequency domain] method. . The example is shown in Formula [13.5]. Examples of changes include
1) Add a positive value β _k ^[l] (ω) to the denominator to prevent division by zero. Further, different values are used for k, l (el), and ω.
2) The Lm norm (formula [13.2]) is used instead of the L2 norm.
3) Instead of the coefficient K of the score function, a different positive value γ _k ^[l] (ω) is used for each of k, l (el) and ω.

  Equation [12.1] is an update equation based on the natural gradient method, but other algorithms can be used. For example, an update equation based on an algorithm that performs simultaneous decorrelation and separation called “Equivalent Adaptive Separation Via Independence” (EASI) is expressed as Equation [12.2]. It is. When this algorithm is used, learning can be converged with a smaller number of loops compared to the natural gradient method.

  Note that the amount of calculation can be reduced by paying attention to the symmetry of the elements of the matrix in the equations [12.1] and [12.2]. This will be described.

The inside of Et [•] in Expression [12.1] is expanded into a matrix of (L ′ + 1) n × (L ′ + 1) n as in Expression [12.7] (the upper line represents a conjugate complex number). ). When taking an average for each element of this expression, φ _kω ^[α] (Y _k ″ (t)) as the first term of each element and Y _i ^[β] (ω, as the second element) t), α and β are integers satisfying 0 ≦ α and β ≦ L ′, and the relative shift amounts are the same. That is, the relationship of Formula [12.8] is established. In particular, when the above-described cyclic shift is used as the shift, the values are exactly the same.

Using this property, the formula [12.7] {(L '+ 1) n} of the ^two elements, the need to calculate the actual value 2 (L' + 1) ⁿ only ^two For the remaining elements, the values may be reused according to equation [12.8].

  Similarly, the amount of calculation can be reduced for the formula [12.2]. Of the three terms inside Et [•], the first can perform the same calculation as in equation [12.1]. For the second, after obtaining the first term, the Hermitian transposition is simply calculated (formula [12.9]). For the third term, the amount of calculation can be reduced by modifying Expression [12.10]. However, X ″ (ω, t) in the equation [12.10] is a vector obtained by extracting the element corresponding to the ω-th frequency bin from the equation [11.1], as in the equation [12.11]. I can express.

17 Et [X ″ (ω, t) X ″ (ω, t) ^H ] is always constant during learning. Therefore, it is only necessary to calculate Et [X ″ (ω, t) X ″ (ω, t) ^H ] once before learning, and it is not necessary to perform an averaging operation every time during learning. That is, the amount of calculation can be reduced on the right side than on the left side of Equation [12.10].

Further, in the calculation of Et [X ″ (ω, t) X ″ (ω, t) ^H ], the equation [12.12] having the same symmetry as the equation [12.8] and the diagonal line The symmetry [12.13] holds. Therefore, it is only necessary to calculate (L ′ + 1) n ² values among the {(L ′ + 1) n} ² elements.

[Specific configuration example and processing example]
A configuration example of the signal separation device of the present invention is shown in FIGS. 15 and 16. FIG. 15 shows a configuration example corresponding to a signal separation device that executes a method for solving convolutional mixing in the time-frequency domain, and FIG. 16 shows a configuration example corresponding to a signal separation device that executes a method for solving instantaneous mixing after conversion to a modulation spectrogram.

(1) Configuration for Executing Method for Solving Convolution Mixing in Time Frequency Domain First, the configuration and processing of a signal separation device for executing the method for solving convolution mixing in the time frequency domain shown in FIG. 15 will be described. Note that overall control of processing described below is executed by the control unit 409. For example, the control unit 409 controls processing according to a program that executes processing described below that is stored in advance in a storage unit (not shown) of the apparatus. Hereinafter, processing of each component will be described. Independent sounds emitted by a plurality of sound sources are observed by a plurality of microphones 401, and an AD conversion unit 402 converts an input analog signal into a digital signal to obtain a digital observation signal.

  The digital observation signal is input to a short-time Fourier transform (STFT) unit 403, and short-time Fourier transform processing is performed to obtain a spectrogram of the observation signal. The process so far corresponds to, for example, the process of obtaining the spectrogram X of the observation signal shown in FIG.

  The signal separation unit 404 separates the spectrogram X of the observation signal generated by the short-time Fourier transform (STFT) unit 403 into independent components. The signal separation device shown in FIG. 15 applies a method of directly solving convolutional mixing in the time-frequency domain as separation processing of the observation signal mixed in the time-frequency domain, and the equations [6.2] and [7. 2), [7.1], and [7.8] are repeatedly performed until the separation matrix and the separation result converge sufficiently (or a fixed number of times), thereby executing the observation signal separation processing. By this separation process, a separation result Y shown in FIG. 6C is obtained.

  The process performed by the convolution operation unit 408 is a process according to the process described above with reference to FIG. That is, this is a process considering that the observation signal X (t) of the t-th frame in the observation signal is affected by the original signal of the previous L + 1 frame with the maximum delay value being L + 1. That is, the observed signal X (t) is represented by the convolutional mixture of the equation [6.1] to which the number of frame taps L described above is applied, and Y (t + L ′) in the separated signal in FIG. , S (t) is estimated, and the separated signal Y (t) is converted from X (t−L ′) to X (t) as shown in Equation [6.2] in consideration of the data for the immediately preceding L + 1 frame. This is a convolution operation that is expressed as a convolution mixture up to and performed by applying Equations [6.2] and [7.2].

As described above, the number L ′ of frame taps for generating the separation result Y from the observation signal X, that is, the separation result Y in FIG. 6C from the observation signal X in FIG. 6B is known. If there is (that is, if the reverberation time is known), L ′ = αL may be set (α is an appropriate positive real number), and if L is unknown, L ′ is, for example, one of the following Decide by method.
(A) A fixed value such as L ′ = 64 or L ′ = 100 is determined.
(B) The reverberation time is measured, and the value of L obtained from the reverberation time is defined as L ′.
(C) Perform the separation under various L ′ and adopt the value of L ′ that gives the best separation result. For example, a separation performance measure called SIR (signal-interference ratio) is calculated, and L ′ that gives the highest SIR is adopted.
By any one of the above methods, L ′, that is, the number L ′ of frame taps for generating the separation result Y from the observation signal X, specifically, for example, from the observation signal X in FIG. The frame tap number L ′ for generating the separation result Y of c) is determined, and the separation result is generated from a plurality of continuous frames of the observation signal using this frame tap number L ′.

  The rescaling unit 405 performs rescaling processing for aligning the scale for each frequency bin of the separated signal. Rescaling is a process of adjusting the scale for each frequency bin. If normalization (adjustment of average or variance) is performed on the observation signal before separation processing, it is restored here.

  The inverse Fourier transform unit 406 converts the spectrogram of the separated signal into a time domain signal by inverse Fourier transform. The converted signal is sent to the post-stage processing execution unit 407 as necessary. Subsequent processing includes reproduction from a speaker and voice recognition. Note that the inverse Fourier transform unit may be omitted depending on the subsequent processing.

  As described above, the signal separation device shown in FIG. 15 is a signal separation device that inputs a signal in which a plurality of sound signals are mixed and separates them into individual sound signals, converts the input signal into the time-frequency domain, and observe signal spectrogram A signal conversion unit (STFT unit 403) for generating a signal, and a signal separation unit (signal separation unit 404) for generating a signal separation result from the observed signal spectrogram generated by the signal conversion unit (STFT unit 403). The (signal separation unit 404) interprets the observation signal spectrogram as an observation signal mixed by convolution in the time-frequency domain, and generates a signal separation result by executing a convolution operation in the convolution operation unit 408.

  The signal conversion means (STFT unit 403) executes processing for generating an observation signal spectrogram by executing short-time Fourier transform (STFT) on the input signal to convert it to the time-frequency domain.

  The signal separation means (signal separation unit 404) sets the separation signal Y (t) of the frame number (t) as the convolution mixture of the observation signals X (t−L ′) to X (t), and the separation signal A signal separation result is generated by a process for increasing the independence of each of Y1 (t) to Yn (t), which are individual audio signal components included in Y (t). Specifically, as a process for enhancing the independence of each of the individual sound signal components Y1 (t) to Yn (t) included in the separated signal Y (t), the amount of Kullback-Leiblar information that is an independence calculation measure Applying I (Y), a signal separation result is generated by updating the separation matrix that minimizes the Kullback-Leiblar information amount I (Y).

  Note that (3) as a device configuration for performing processing for separating the observation signal mixed in the time-frequency domain by processing combining shift accumulation and instantaneous mixing ICA, for example, a convolution operation unit 408 from the configuration shown in FIG. A configuration without the above can be applied. The processing executed in the signal separation unit is different.

  (3) In an apparatus that performs processing that combines shift stacking and instantaneous mixing ICA, the STFT unit 403 functions as a signal conversion unit that converts an input signal into a time-frequency domain and generates an observation signal spectrogram. The signal separation unit 404 performs processing for generating a signal separation result from the observation signal spectrogram generated by the signal conversion means, and the signal separation unit 404 converts the observation signal spectrogram as described above with reference to FIGS. Processing that shifts in the frame direction and generates observation signal spectrogram shift sets in which data having different shift amounts are stacked, and applies instantaneous mixed ICA (Independent Component Analysis) to the generated observation signal spectrogram shift set Signal Generate separation results. Note that the processing using the instantaneous mixing ICA is the method disclosed in Japanese Patent Laid-Open No. 2006-238409, that is, the time frequency domain separation signal is generated from the time frequency domain observation signal and the separation matrix, and the generated time frequency domain Modify the separation matrix until the separation signal and the separation matrix calculated by the multidimensional score function derived from the multidimensional probability density function converge, and apply the modified separation matrix to the time-frequency domain separation signal. It is executed as a process to generate.

(2) Configuration for executing method for solving instantaneous mixing after conversion to modulation spectrogram Next, a configuration of a signal separation device for executing the method for solving instantaneous mixing after conversion to a modulation spectrogram shown in FIG. 16, and Processing will be described. Note that overall control of processing described below is executed by the control unit 461. For example, the control unit 461 controls processing according to a program for executing processing described below that is stored in advance in a storage unit (not shown) of the apparatus. Hereinafter, processing of each component will be described. Independent sounds emitted by a plurality of sound sources are observed by a plurality of microphones 451, and an analog signal is converted into a digital signal by an AD converter 452 to obtain a digital observation signal.

  The digital observation signal is input to a first short-time Fourier transform (STFT) unit 453, and short-time Fourier transform (STFT) processing is performed to obtain a spectrogram of the observation signal. The signal obtained at this stage is, for example, a spectrogram X shown in FIG. Further, the spectrogram of the observation signal obtained by the first-stage short-time Fourier transform (STFT) process is input to the second short-time Fourier transform (STFT) unit 454, and the short-time Fourier transform (STFT) is again performed for each frequency bin. ) To obtain a modulation spectrogram.

  The modulation spectrogram obtained by the short time Fourier transform (STFT) in the second short time Fourier transform (STFT) unit 454 is, for example, the modulation spectrogram X ′ shown in FIGS.

  The signal separation unit 455 receives the modulation spectrogram X ′ and separates the modulation spectrogram X ′ into independent components. This separation process is the process described above with reference to FIG. That is, FIG. 10 corresponds to the three-dimensional modulation spectrogram X ′ shown in FIG. 9C. In the three-dimensional modulation spectrogram X ′ shown in FIG. 10, for example, in entropy calculation of the first channel, The modulation spectrogram Y1 ′ (t) 221 of the first frame in FIG. 10 represents a plane, and Y1 ′ (t) is substituted into the multivariate probability density function P (Y1 ′ (t)) 222 with the argument as an argument. Thus, entropy H (Y1 ′) 223 is obtained. Formula [9.3] is the same as Formula [3.5] except for the difference in variable names. Therefore, in order to derive the learning rule, the variable name of the equation [5.2] may be changed, and the equation [9.5] is obtained as a result. That is, if the expressions [9.3], [9.5], and [9.6] are repeated until W ′ converges, Y1 ′ (t) to Yn ′ (t) become independent from each other.

  Next, the first rescaling unit 456 performs rescaling on the modulation spectrogram. Rescaling is a process of adjusting the scale for each frequency bin. Further, the first inverse Fourier transform (FT) unit 457 performs an inverse Fourier transform (FT) process on the rescaled modulation spectrogram to convert the modulation spectrogram into a spectrogram. Then, after rescaling is performed again by the second rescaling unit 458, the second inverse Fourier transform (FT) unit 459 performs inverse Fourier transform (FT) processing on the rescaled spectrogram to obtain a spectrogram. To waveform. The signal converted into the waveform is sent to the post-stage processing execution unit 461 as necessary, and the post-stage process is executed as necessary. Subsequent processing includes reproduction from a speaker and voice recognition.

  As described above, the signal separation device shown in FIG. 16 is a signal separation device that inputs a signal in which a plurality of signals are mixed and separates them into individual signals, and converts the input signal into the time-frequency domain to generate an observation signal spectrogram. First signal converting means (first STFT unit 453), and second signal converting means (first STFT unit 453) that performs data conversion on the observed signal spectrogram generated by the first signal converting means (first STFT unit 453) and generates a modulation spectrogram. 2STFT section 454) and signal separation means (signal separation section 455) for generating a signal separation result from the modulation spectrogram generated by the second signal conversion means (second STFT section 454). Part 455) interprets the modulation spectrogram as instantaneous mixing and signal separation To generate the results.

  The first signal conversion means (first STFT unit 453) performs short-time Fourier transform (STFT) on the input signal to convert it into the time-frequency domain, and generates an observation signal spectrogram. Further, the second signal conversion means (second STFT unit 454) performs a short-time Fourier transform (STFT) in the time direction on the observed signal spectrogram to generate a modulation spectrogram.

  The signal separation means (signal separation unit 455) generates a signal separation result by a process for increasing the independence of each of the signal components Y1 ′ to Yn ′ corresponding to the separation signal included in the modulation spectrogram. Specifically, as a process for increasing the independence of each of the signal components Y1 ′ to Yn ′ corresponding to the separated signal, the Kullback-Leiblar information amount that is an independence calculation measure is applied, and the Kullback-Leiblar information amount is minimized. A signal separation result is generated by matrix update processing.

  The inverse Fourier transform unit (first inverse FT unit 457) performs inverse Fourier transform on each of the signal components Y1 ′ to Yn ′ corresponding to the separated signal obtained by the signal separation unit (signal separation unit 455). Thus, spectrograms Y1 to Yn corresponding to the separated signals are generated.

  An example of the sequence of processing executed by the signal separation device of the present invention will be described with reference to the flowchart shown in FIG. In step S101, sound is observed with a microphone. For example, as described above with reference to FIG. 5, a mixed signal of sounds output from a plurality of sound sources is acquired by a microphone. Next, in step S102, a short-time Fourier transform (STFT) process is performed on the observation signal to obtain a spectrogram. The short-time Fourier transform is the process described above with reference to FIG. 7, and a spectrogram is obtained by this process. This spectrogram is, for example, the spectrogram shown in FIG.

  Next, in step S103, ICA separation processing is performed on the spectrogram of the observation signal. Details of the processing sequence of the separation processing will be described later. In step S104, an inverse Fourier transform (IFT) is performed on the separation result as necessary, and subsequent processing is performed in step S105 as necessary.

A detailed sequence of the separation process executed in step S103 will be described with reference to the flowcharts shown in FIGS.
The separation processing sequences shown in FIGS. 18 and 19 are specific sequences of the separation processing executed in the signal separation device described above with reference to FIGS. 15 and 16, respectively.
FIG. 18 illustrates a separation process in a method for solving convolutional mixing in the time-frequency domain performed by the signal separation device of FIG.
FIG. 19 shows a separation process in a system for solving instantaneous mixing after converting into a modulation spectrogram executed by the signal separation device of FIG.
These are detailed sequences.

  First, with reference to FIG. 18, a description will be given of a separation processing sequence in which a convolution mixture is solved in the time-frequency domain performed by the signal separation device in FIG.

First, in step S201, normalization is performed on the observed signal spectrogram. The normalization process in this process is a process of setting the average to 0 and setting the variance to 1 for each frequency bin of the spectrogram, or adjusting the value to a value convenient for the subsequent processes. Next, in step S202, an initial value is substituted into the separation matrix initialization process, that is, the separation matrix W ^[τ] . The initial value is the identity matrix with respect to W ^[0], may be substituted for zero matrix for tau> 0 of the separation matrix W ^[tau]. Alternatively, if there is a separation matrix obtained by the previous learning, it may be used as an initial value.

Steps S203 to S210 are a learning loop, and this loop is repeated until the separation matrix and the separation result converge. That is,
Step S203: Determination of whether or not the separation matrix has converged,
Step S204: Calculation of the separation signal Y,
Step S205: Start of frequency bin loop (ω = 1,..., M),
Step S206: Start of frame tap loop (τ = 0,..., L),
Step S207: Calculation of the increment ΔW ^[τ] corresponding to the τ-th frame tap,
Step S208: End of the frame tap loop,
Step S209: Update ΔW ^[0] (ω) to W ^{[L ′]} (ω),
Step S210: End of the frequency bin loop,
A loop consisting of these steps is repeatedly executed.

  For the calculation of the separation result Y in step S204, the previously described equation [6.2] or equation [6.3] is used. (Y = [Y (1),..., Y (T)]) Steps S205 to S210 are loops for frequency bins, where M is the number of frequency bins and 1 ≦ ω ≦ M is satisfied. Steps S206 to S209 are repeated for each frequency (ω). In addition, it is good also as a structure which performs the parallel processing for every frequency bin instead of a loop. In the method disclosed in Japanese Patent Application Laid-Open No. 2006-238409, which is an earlier patent application of the same applicant as the present application and published, only one separation matrix (or one for each frequency bin) is estimated. However, in the present invention, it is necessary to estimate the separation matrix by the number of frame taps. Therefore, the loop is rotated by the number of frame taps (steps S206 to S208).

In step S207, an increment ΔW ^[τ] (ω) corresponding to the τ-th frame tap is obtained. Formula [7.1] is used for calculation of ΔW ^[τ] (ω). As described above, Rω ^[l] in the equation [7.1] differs depending on whether the equation [6, 2] or the equation [6.3] is used to calculate the separation result Y.

When Expression [6.2] is used for calculation of separation result Y, Expression [7.2] or [8.1] is used for calculation of Rω ^[l] , and Expression [6. 3] is used, the equation [7.3] or [8.2] is used to calculate Rω ^[l] .

After exiting the frame tap loop of steps S206 to S208, the separation matrix ΔW ^[0] (ω) to W ^{[L ′]} (ω) is updated using equation [7.8] in step S209. This process may be performed for all frequency bins after step S210. (On the other hand, note that it cannot be put inside the frame tap.)

  After exiting the frequency bin loop of steps S205 to S210, the process returns to the convergence check of step S203 again. If it is determined in step S204 that the separation matrix has converged (or looped a predetermined number of times), the process proceeds to the right and proceeds to step S211.

  Note that whether or not the separation matrix has converged in step S203 is, for example, whether or not the norm of ΔW ‖ΔW‖ (the norm of the matrix is calculated by, for example, the above equation [7.10]) is below a certain value. (Or whether or not ‖ΔW‖ / ‖W‖ falls below a certain value), or a predetermined number of loops may be simply set in advance and the number of loops may be executed.

  If it is determined in step S203 that the separation matrix has not yet converged, the processes in steps S204 to S210 are repeated. If it is determined in step S204 that the separation matrix has converged (or looped a predetermined number of times), the branch is moved to the right and the process proceeds to step S211. In step S211, rescaling is performed. Rescaling is a process of adjusting the scale for each frequency bin. If the average or variance of the frequency bin is changed in the normalization processing step (S201), it is restored here as necessary.

  The rescaling coefficient to be executed in step S211 is obtained as follows. Using the equation [7.11] shown above, a scale is obtained such that the square error between the observed signal and the separation result is minimized in a certain frequency bin (specifically, the least square method or the like is used). . Then, the separation result is updated to a value multiplied by the scale (formula [7.12]). Also, if necessary, the separation matrix itself is updated in the same manner (Formula [7.13]).

Or you may carry out as follows. The observation signal is expressed as a linear sum of the separation result and a constant using Equation [7.14]. The scales α _k1 (ω) to α _kn (ω) and the constant term βk (ω) are obtained by the equation [7.15] (specifically, the least square method or the like is used). When the scale is obtained, the separation result is updated using Expression [7.16]. (If necessary, update the separation matrix.)

If all α _kj (ω) Y _j (ω, t) appearing in Equation [7.14] is output, an output in a single input multiple output (SIMO) format is obtained. The SIMA output of ICA is “decomposing the observation signal into components derived from the respective sound sources”. For example, α _kj (ω) Y _j (ω, t) is used to convert Y _j into the i-th sound source. As an estimation result, “a component derived from the i-th sound source among signals observed by the k-th microphone” is represented. This completes the description of the flowchart for the case of solving convolutional mixing in the time-frequency domain.

  Next, processing for solving instantaneous mixing in the modulation spectrogram region will be described with reference to a flowchart shown in FIG. FIG. 19 is a detailed sequence of the separation process in the method of solving the instantaneous mixing after converting into the modulation spectrogram executed by the signal separation device of FIG.

  Step S301 normalizes the observed signal spectrogram. This process is the same as the normalization process in step S201 in the flow of FIG. 18, and is convenient for the process in which the average is set to 0 and the variance is 1 for each frequency bin of the spectrogram, or the subsequent process. This is a process of adjusting to a good value. In step S302, a short-time Fourier transform (STFT) is performed for each frequency bin to generate a modulation spectrogram, that is, a modulation spectrogram X ′ shown in FIGS.

  Note that the modulation spectrogram is generated by performing a short-time Fourier transform (STFT) process in the first short-time Fourier transform (STFT) unit 453 on the digital observation signal as described above with reference to FIG. Thus, the spectrogram of the observation signal (for example, the spectrogram X shown in FIG. 8B) is obtained, and the spectrogram of the observation signal obtained by the first-stage short-time Fourier transform (STFT) processing is further obtained. It is necessary to input to the transform (STFT) unit 454 and execute short-time Fourier transform (STFT) again for each frequency bin. The modulation spectrogram obtained by the short time Fourier transform (STFT) in the second short time Fourier transform (STFT) unit 454 is, for example, the modulation spectrogram X ′ shown in FIGS.

  As shown in FIGS. 9C and 9D, the modulation spectrogram has a cubic form (corresponding to Expression [9.2]) and a planar form (corresponding to Expression [9.3]). In the following description, a planar format is used. That is, the bins in both the vertical direction and the depth direction shown in FIG. 9C are collectively expressed by an index ω ′.

  In step S303, normalization is performed again for each bin ω ′ of the modulation spectrogram. Prior to the learning loop, in step S304, initial values are substituted into the separation matrix W ′. The initial value may be a unit matrix, but may also be a separation matrix obtained in the previous learning.

  Steps S305 to S310 are a learning loop and are repeated until the separation matrix W ′ converges (or a fixed number of times). The convergence determination in step S305 is the same determination as the process in step S203 described with reference to FIG. 18, and the determination of whether or not the separation matrix has converged is, for example, the norm ‖ΔW′‖ (matrix of ΔW ′ For example, it may be determined by whether or not the value of the norm is less than a certain value (or whether or not ‖ΔW′‖ / ‖W′‖ is less than a certain value). Alternatively, a predetermined number of loops may be simply set in advance and the number of loops may be executed.

  In step S306, Y ′ which is a separation result modulation spectrogram is calculated. This calculation may be performed by performing the equation [9.3] for all ω ′ and t.

  Steps S307 to S310 are loops for each bin ω ′ of the modulation spectrogram shown in FIG. 9C, that is, both the vertical and depth bins ω ′. Note that the processing for each bin may be executed as parallel processing instead of the loop for repeated processing for each bin. In step S308, the increment of the separation matrix is calculated (formula [9.5]), and in step S309, the separation matrix is updated (formula [9.6]).

  In step S310, after exiting the loop, the process returns to the convergence determination in step S305 again. If it is determined in step S305 that the separation matrix has converged (or looped a predetermined number of times), the conditional branch is advanced to the right. In step S311, rescaling is performed. Rescaling is a process of adjusting the scale of each bin. Rescaling the resulting modulation spectrogram. The rescaling method is almost the same as the processing in step S211 described above with reference to FIG. 18, and Y, X, and W in the equations [7.11] to [7.16] are appropriately changed to Y ′. , X ′, W ′ is performed based on the replaced expression. Moreover, the process which returns the normalization of step S301 is also performed as needed.

  Next, in step S312, an inverse Fourier transform (FT) for converting the modulation spectrogram to the spectrogram is executed. At that time, WOLA or the like is performed as necessary. That is, in the inverse Fourier transform (FT), the inverse transform result (waveform) waveforms for each frame are overlapped. This is called overlap addition. The inverse transformation result may be obtained by applying a window function such as a sine window again before overlap addition, which is called weighted overlap add (WOLA). With WOLA, noise derived from discontinuity between frames can be reduced.

  Furthermore, rescaling is performed on the spectrogram in step S313. This is the same processing as the rescaling in step S311.

  In the inverse Fourier transform (FT) executed in step S104 in the flow of FIG. 17 and step S312 in the flow shown in FIG. 19, in addition to the spectrogram and the modulation spectrogram of the separation result, the separation matrix itself is necessary. In response, inverse Fourier transform (FT) is performed.

[Modification]
Next, a modified example of the above-described embodiment will be described. In the above-described embodiment, the frame tap L ′ applied when generating the separation result, that is, the frame tap L ′ when generating the separation result from the observation signal, uses a constant value at all frequencies. Instead of making this uniform for each frequency, the value of the frame tap L ′ may be changed for each frequency.

  For example, the reverberation time is short because the high frequency component attenuates more rapidly than the low frequency component. Therefore, in the frequency bin corresponding to the high frequency, the value of the frame tap L ′ may be smaller than that in the low frequency bin. By doing so, the amount of calculation can be reduced while maintaining the separation performance.

  Further, in the method described with reference to the signal separation device in FIG. 16 and the flowchart shown in FIG. 19, that is, in the separation process in which the instantaneous mixing is solved after being converted into a modulation spectrogram, the second short-time Fourier transform is performed. In the conversion (STFT), in addition to changing the number of frame taps L ′ for each frequency bin, it is also possible to change the shift width. However, if the number of frame taps and the shift width are varied for each frequency bin, the time length per frame may be different in the modulation spectrogram.

For example, in the second short time Fourier transform (STFT):
Low frequency uses tap number = 32 and shift width = 16,
For high frequency, using tap number = 16 and shift width = 8,
The time length per frame in the converted modulation spectrogram is such that the low frequency is double the high frequency. That is, the number of frames per unit time is smaller (half) at lower frequencies.

  When the time length per frame is constant, Yk ′ (t) 221 can be extracted from the modulation spectrogram and the independence can be calculated as shown in FIG. 10, but it is difficult if it is not constant. . In such a case, any one of the methods described below (methods 1 to 3) is used to deal with a frame mismatch.

(Method 1) Thinning out frame data In the generated modulation spectrogram, the number of data is matched with the bin having the smaller number of frames by thinning out the data from the bin having the larger number of frames per unit time. In the example of 32 taps, 16 shifts, and 16 taps, 8 shifts described above, when data is thinned out every other bin for 16 taps / 8 shifts short-time Fourier transform (STFT), both units are used. The number of frames per time is the same (= time per frame is the same).

(Method 2) Interpolation of frame data Contrary to the above (Method 1), a method in which the smaller number is matched with the larger number. In the above example of 32 taps · 16 shift and 16 taps · 8 shifts, data interpolation is performed on bins that have been subjected to 32 taps · 16 shifts. For example, new data is inserted in the middle by taking the average of previous and next frame data.

(Method 3) Overlapping of frame data A method of matching the smaller number with the larger one as in the above (Method 2). In the above example of 32 taps, 16 shifts and 16 taps, 8 shifts, bins with 16 taps, 8 shifts and the number of data are duplicated twice for bins that have undergone 32 taps, 16 shifts. Adjust.

Next, referring to FIGS. 11 to 14, (3) a method based on a combination of shift stacking and instantaneous mixing ICA, that is, stacking while shifting the observed signal spectrogram, and combining the signals with instantaneous mixing ICA (for example, special mixing ICA) In the process of separating the observation signal mixed by convolution in the time-frequency domain by the method described in Japanese Patent Application Laid-Open No. 2006-238409,
A modification example in which “the value of [L ′] for each frequency, that is, the value of the number of frame taps [L ′] when generating the separation result from the observation signal is varied” will be described.

  In order to realize this modified example, that is, a modified example of “variing the value of the number of frame taps [L ′] for each frequency”, the following may be performed. L ′ that differs for each frequency bin is represented as L ′ (ω). In the shift process described with reference to FIG. 11, when the shift amount exceeds [L ′ (ω)], the data of the frequency bin is replaced with [0]. The method will be described with reference to FIG.

Assume that the value of the number of frame taps [L ′ (ω)] that differs for each frequency bin is changed as follows according to the frequency bin number ω. (M is the number of frequency bins per spectrogram)
For 1 ≦ ω <M / 4, L ′ (ω) = 2
When M / 4 ≦ ω <M / 2, L ′ (ω) = 1
For M / 2 ≦ ω <M, L ′ (ω) = 0
In order to realize this, the following operation is performed on the data X _k ^[0] , X _k ^[1] , and X _k ^[2] generated by the shift processing described with reference to FIG. Do.
X _k ^[0] remains as X _k . (For all frequency bins, shift = 0 is required),
X _k ^[1] masks the frequency bin of M / 2 ≦ ω with 0. (If M / 2 ≦ ω, one or more shifts are unnecessary)
X _k ^[2] masks the frequency bin of M / 4 ≦ ω with 0. (If M / 4 ≦ ω, two or more shifts are not required)

  Specifically, as shown in FIG. 20B, a black-filled data portion 5112 is a portion masked with 0. In actual processing, it is not necessary to secure the masked portion of the memory, and if the corresponding portion of the mask is skipped when accessing the spectrogram, an increase in processing time and memory amount can be prevented.

In addition, as a pre-process of the present invention, when a conventional instantaneous frequency-domain mixed ICA (for example, JP-A-2006-238409) is combined, an increase in processing time can be suppressed to some extent. Hereinafter, the combination of both will be described. The following corner processing examples will be described sequentially.
(1) Basic two-stage separation (2) Reduction of the number of channels (3) Use as dereverberation

(1) Basic two-stage separation In the conventional instantaneous mixing ICA in the time-frequency domain, when an analysis frame (or analysis window) shorter than the reverberation is used, the interference sound over a plurality of frames cannot be removed. On the other hand, the amount of calculation is less than that of the present invention (if the analysis frame length in the first STFT is the same). Therefore, if separation is first performed in the conventional time-frequency domain ICA, and the resulting spectrogram is further separated by the method of the present invention, the same separation accuracy can be achieved in less time than when only the present invention is used from the beginning. can do.

  In particular, when using the “(1) direct solution of convolutional mixing in the time-frequency domain” according to the present invention, the conventional method and the present invention can be operated seamlessly. That is, the feature that the equation [7.2] and the equation [8.1] (or the equation [7.3] and the equation [8.2]) are equivalent to the conventional method when L ′ = 0 is used. In the learning loop of steps S203 to S210 in the flow shown in FIG. 18, L ′ = 0 when the number of loops is small, and L ′ may be set to an original value when the number of loops exceeds a certain value. Alternatively, L ′ may be increased little by little as the number of loops increases.

(2) Reduction of the number of channels Generally, the amount of calculation of ICA is proportional to the square of the number of channels. Therefore, if the number of channels can be reduced, the amount of calculation can be greatly reduced. The use of two-stage separation can also reduce the number of channels in the steps of the present invention. The method will be described.

  In the ICA in the time frequency domain, when the number of microphones is larger than the number of sound sources, a signal determined that some of the output channels do not correspond to any sound source is output. For example, when the number of microphones = 4 and the number of sound sources = 3, three of the output channels correspond to sound sources, but the remaining one does not correspond to any sound source, and background noise and reverberation sound are mixed. Such a signal is output. Such an output can be easily detected because the power is extremely small compared to other channels or there is a correlation with any other channel.

Therefore, in the two-stage separation, according to the flowchart shown in FIG. 21, first, separation processing by the time-frequency domain instantaneous mixing ICA is performed in step S501. This process can be executed as the process disclosed in JP-A-2006-238409. Thereafter, in step S502, for example, after removing “output determined not to correspond to any sound source” (unnecessary channel), the processing according to the present invention described above, that is,
(1) Solve convolutional mixing directly in the time-frequency domain.
(2) The spectrogram is again subjected to short-time Fourier transform (STFT) in the time direction and solved as instantaneous mixing.
(3) Solve by a process combining shift stacking and instantaneous mixing ICA.
If any of these processes is performed to separate the convolution mixed observation signals in the time-frequency domain, the amount of calculation in the separation process can be reduced. Since the separation is possible if the number of input channels = the number of sound sources, even if the number of channels is reduced in step S502, the separation accuracy is not affected.

  For example, when the two-stage processing is applied to the method of directly solving convolutional mixing in the above (1) time-frequency domain, the signal separation means performs instantaneous mixing ICA (Independent Component Analysis) on the observed signal spectrogram. The first signal separation result is generated by the process of applying, the unnecessary channel removal process determined to not correspond to any sound source is executed from the first signal separation result, and the observation signal remaining after the removal process A signal separation result is generated by executing a process of solving the convolutional mixture in the time-frequency domain for the spectrogram.

In addition, when (2) this two-step process is applied to a method of performing a short-time Fourier transform (STFT) once again in the time direction and solving as an instantaneous mixture,
The first signal converting means converts the input signal into the time frequency domain to generate an observed signal spectrogram, and the unnecessary channel removing means applies the instantaneous mixing ICA to the observed signal spectrogram generated by the first signal converting means. A first signal separation result is generated by the processing, and an unnecessary channel removal process that is determined not to correspond to any sound source is executed from the signal separation result. Further, the second signal conversion unit removes unnecessary channels. Data conversion is performed on the observed signal spectrogram to generate a modulation spectrogram, and the signal separation means generates a signal separation result from the modulation spectrogram.

When (3) this two-stage process is applied to a process that combines shift stacking and instantaneous mixing ICA,
A first signal separation result is generated for the observed signal spectrogram by a process using the instantaneous mixing ICA, and an unnecessary channel removal process that is determined not to correspond to any sound source from the generated first signal separation result is performed. The observed signal spectrogram remaining after the removal processing is shifted in the frame direction to generate an observed signal spectrogram shift set, and the instantaneous mixed ICA is again applied to the generated observed signal spectrogram shift set to obtain a signal separation result Is generated.

(3) Use as dereverberation separation process of the present invention, that is,
(1) Solve convolutional mixing directly in the time-frequency domain.
(2) The spectrogram is again subjected to short-time Fourier transform (STFT) in the time direction and solved as instantaneous mixing.
(3) Solve by a process combining shift stacking and instantaneous mixing ICA.
Among these processes, when the third “shift stack + conventional method” is used, the separation itself is performed by the conventional method of preprocessing, and the present invention can also share the role of performing only dereverberation. By doing so, the calculation amount is reduced from O ({n × (L ′ + 1)} ² ) to O (n × n × (L ′ + 1)). The method will be described below.

  When the “shift of the spectrogram + stacking” described above with reference to FIG. 11 is performed, the spectrogram for one channel is apparently expanded to L ′ + 1 channel. When the result is processed as an input of the L ′ + 1 channel using the conventional instantaneous frequency domain ICA, the spectrogram for the L ′ + 1 channel is generated as a result. Even if such processing is performed, it is not separated into components for each sound source, but there is an effect of removing components across a plurality of frames, that is, an effect of dereverberation removal. Therefore, a combination of performing separation by a conventional instantaneous frequency mixing ICA in the time-frequency domain, generating a separation result spectrogram for n channels, and performing the above-mentioned “reverberation removal” on each channel is conceivable. Such processing will be described with reference to the flowchart shown in FIG.

First, in step S601, separation processing is performed by instantaneous mixing ICA in the time-frequency domain. This process can be executed as the process disclosed in JP-A-2006-238409. Spectrogram _Y 1 to Y _n of the n channels are generated as a separation result. The subsequent processing is performed separately for the n-channel spectrograms Y _{1 to} Y _n . Processing for spectrogram _{Y 1} of the first channel corresponding steps S611～S613, processing for the n-channel corresponding spectrogram _{Y n} is step S621～S623. When the separation process by the instantaneous mixing ICA in step S601 is completed, a process of removing unnecessary channels (output determined not to correspond to any sound source) is performed as described above with reference to FIG. It is good also as a structure to perform.

The processing of steps S611 to S613 corresponds to the processing of steps S11 to S13 of the flow shown in FIG. 14 which is the processing sequence of the method described above [(3) Solve by processing combining shift stacking and instantaneous mixing ICA]. It is processing. However, the process of step S11 of the flow shown in FIG. 14 is a process of expanding the spectrogram for n channels to n × (L ′ + 1), whereas the process of step S611 of the flow shown in FIG. This is a process of extending the minutes to L ′ + 1 channel. Further, the dereverberation process of step S612 is the same as the process of step S12 in the flow shown in FIG. 14, but for the reason described above, the effect of the process of step S612 is not the separation of the sound source. And executed as dereverberation. The process of step S613 is a process of selecting a desired one from the dereverberation-removed spectrogram for L ′ + 1 channel, and is the same process as the process of step S13 of FIG.
Processing in step S621~S623 are, except processed is different channels corresponding signal _{Y n} are the same as steps S611～S613.

  When dereverberation and selection are completed for all output channels (however, unnecessary channels may be deleted), the remaining spectrograms are integrated in step S631. For example, a process of stacking vertically is executed. By this process, a process for removing a component across a plurality of frames, that is, a dereverberation process is realized.

[Verification of effects in signal separation processing according to the present invention]
Experiments have confirmed that the above-described method of the present invention provides separation performance exceeding the conventional time frequency domain ICA. Hereinafter, the effects of the signal separation processing according to the present invention will be described based on the experimental results.

First, experimental conditions will be described.
Recording of sound data was performed in the environment (office room) shown in FIG.
The number of microphones = 4 (interval = 7.5 cm), the number of sound sources = 3, and the sound source disclosed on the following Web page was used.
Original signal:
ICA'99 SYNTHETIC BENCHMARKS
http: // sound. media. mit. edu / ica-bench / sources /
src1: beet. wav
src2: beet9. wav
src3: Mike. wav
In addition, recording is performed in a state where each sound source is sounded independently, and later mixed on a computer.

The experiment was performed under the following conditions.
Sampling frequency: 16 kHz
STFT window length: 64, 128, 256, 512, 1024 (2048, 4096)
STFT shift width: 1/2 of window length
Window: Sine window during short-time Fourier transform (STFT), sine window again during inverse Fourier transform (FT) η0 = 0.5 (formula)
Number of loops = 200 or 400
method:
(Method 1) Equation [5.2] (equivalent to the conventional method)
(Method 2) Equation [7.1] & Equation [7.2] (hereinafter “reverse convolution”)
(Method 3) Equation [9.5] (hereinafter “Re-STFT”)
Score function: Use equation [7.7] Value of score function γ:
(Method 1 & 2) γ = sqrt (M) M: Number of frequency bins (Method 3) γ = sqrt (L′ M)
Frame tap:
(Method 2) L ′ = 4, 5, 8, 10, 15, 16, 20, 25, 30, 32
(Method 3) L ′ = 4, 8, 16, 32

  As an evaluation measure, waveform-based signal-interference-ratio (SIR) and frequency bin-based SIR are used. Below, the calculation method of SIR is demonstrated.

Consider the separation result (waveform) corresponding to the k-th channel as yk (t), and approximate yk (t) by linear combination of the original signals s ₁ (t) to s _N (t) (shown below) Formula [10.1]).

coefficients lambda ₁ to [lambda] _N of _{s 1 (t) ~s N (} t) is determined by minimizing the square error of Formula [10.2].
When yk (t) is regarded as an estimation result of the i-th sound source s _i (t), SIR is defined as a power ratio between s _i (t) and the other sound sources (formula [10.3]). ).
When the number of output channels (= number of microphones) is n, n SIRs are calculated for one sound source, and the maximum value among them is defined as the SIR of the sound source i (formula [10.4]). In the subsequent experimental results, the SIR obtained from each of the three sound sources is further averaged.
The frequency bin-based SIR is calculated by calculating the SIR for each frequency bin and then taking an average for all frequency bins (formula [10.6]).

Below, an experimental result is demonstrated. The experimental results are shown as a table below.
In each table,
Window length: STFT window length,
frm-tap is the number of frame taps,
SIR (wave) is waveform-based SIR,
SIR (bin) is the frequency bin based SIR,
Represents.

Hereinafter, in each table,
(1) Method 1 (conventional method), 200 loops (2) Method 2 (Equations [6.1], [7.1], [7.2]), 200 times loop (3) Method 3 (Equation [ 9.2], [9.5]), 200 loops (4) Method 1 (conventional method), 400 loops (5) Method 2 (Equations [6.1], [7.1], [7. 2]), 400 loops (6) Method 3 (formulas [9.2], [9.5]), 400 loops These experimental results are shown.

FIG. 24 shows evaluation data on the separation results by the following three methods.
(1) Method 1 (conventional method), 200 loops (2) Method 2 (Equations [6.1], [7.1], [7.2]), 200 times loop (3) Method 3 (Equation [ 9.2], [9.5]), 200 loops SIR data based on the result data when these three methods are executed is plotted.
(A) Waveform-based SIR (signal-interference-ratio)
(B) Frequency bin based SIR
These SIR data are plotted. The horizontal axis is the window length of the STFT, and the vertical axis is the SIR. In each graph,
* (Solid line): Method 1,
◆: method 2,
+: Is method 3,
It is.
In some settings, it can be seen that method 2 and method 3 outperform the conventional method.

Next, FIG. 25 shows evaluation data plotted using the time span calculated by the following equation as the horizontal axis.
time_span = {(frame_tap-1) × frame_shift + window_len} / rate
In addition,
frame_tap: number of frame taps (= L ′)
window_len: Window length (first STFT cut-out section length)
frame_shift: Window shift width (this time 1/2 of the window length)
rate: Sampling frequency (16 kHz)

FIG. 25 is also evaluation data about the separation results by the following three methods.
(1) Method 1 (conventional method), 200 loops (2) Method 2 (Equations [6.1], [7.1], [7.2]), 200 times loop (3) Method 3 (Equation [ 9.2], [9.5]), 200 loops SIR data based on the result data when these three methods are executed is plotted.
(A) Waveform-based SIR (signal-interference-ratio)
(B) Frequency bin based SIR
These SIR data are plotted. The horizontal axis is the window length of the time span (Time_span) described above, and the vertical axis is the SIR. In each graph,
* (Solid line): Method 1,
◆: method 2,
+: Is method 3,
It is.

  Conventionally, in order to cover a long time span, the window length of the short-time Fourier transform (STFT) must be increased, which has caused a decrease in SIR. On the other hand, in the present invention, by using a combination of a short window and a plurality of frame taps, an equivalent time span can be covered without reducing the SIR.

FIG. 26 shows evaluation data on the separation results by the following three methods.
(4) Method 1 (conventional method), 400 loops (5) Method 2 (Equations [6.1], [7.1], [7.2]), 400 loops (6) Method 3 (Equation [ 9.2], [9.5]), 400 loops SIR data based on the result data when these three methods are executed is plotted.
(A) Waveform-based SIR (signal-interference-ratio)
(B) Frequency bin based SIR
These SIR data are plotted. The horizontal axis is the window length of the STFT, and the vertical axis is the SIR. In each graph,
* (Solid line): Method 1,
◆: method 2,
+: Is method 3,
It is.

Next, the same evaluation experiment was performed by increasing the number of separation processing loops to 400.
As data corresponding to the data shown in FIG. 26, evaluation data plotted using the time span as the horizontal axis is shown in FIG. It is evaluation data about the separation result by the following three methods.
(4) Method 1 (conventional method), loop 400 times (5) Method 2 (equations [6.1], [7.1], [7.2]), loop 400 times (6) Method 3 (equation [ 9.2], [9.5]), 400 loops SIR data based on the result data when these three methods are executed is plotted.
(A) Waveform-based SIR (signal-interference-ratio)
(B) Frequency bin based SIR
These SIR data are plotted. The horizontal axis is the window length of the time span (Time_span) described above, and the vertical axis is the SIR. In each graph,
* (Solid line): Method 1,
◆: method 2,
+: Is method 3,
It is.

  Also in FIGS. 26 and 27, method 2 and method 3 have settings that exceed the conventional method. As described above, according to the present invention, it is possible to avoid the problem of “tradeoff between window length and separation performance” that the conventional time-frequency domain ICA has.

  Next, an evaluation experiment for different types of data will be described. FIG. 28 is a sketch of an office environment that is a recording environment. As shown in the figure, the experiment was conducted in a rectangular room of approximately 750 cm × 375 cm. As shown in the figure, it is not a complete rectangle, but one is a space delimited by partitions having a height of 153 cm. The reverberation time of the room is a little shorter than 0.3 seconds. (Hereafter, it is plotted as 0.275 seconds.)

The following three types of sounds were prepared as sound sources. (The spectrogram of each signal is shown in FIG. 29.)
Sound source 1 (src1): Utterance of one woman (hereinafter referred to as female voice or F)
Sound source 2 (src2): Utterance of one male (hereinafter male voice or M)
Sound source 3 (src3): Street noise (hereinafter referred to as “busy or S”) published at the following URL: http: // sound. media. mit. edu / ica-bench / sources / street. wav

The above sounds were reproduced from each of the speakers sp1 to sp4 in the figure and recorded with four microphones (mic1 to mic4) arranged at intervals of 5 cm. Next, with the eight combinations shown in FIG. 30, sound output from the sub-speakers sp1 to sp4 was performed, and data input by four microphones (mic1 to mic4) was analyzed. Assume that one female utterance is [F], one male utterance is [M], street noise is [S], and no voice output is [0].
(1) sp1 = S, sp2 = 0, sp3 = F, sp4 = M
(2) sp1 = S, sp2 = 0, sp3 = M, sp4 = F
(3) sp1 = F, sp2 = S, sp3 = 0, sp4 = M
(4) sp1 = M, sp2 = S, sp3 = 0, sp4 = M
(5) sp1 = 0, sp2 = 0, sp3 = F, sp4 = M
(6) sp1 = 0, sp2 = 0, sp3 = M, sp4 = F
(7) sp1 = F, sp2 = 0, sp3 = 0, sp4 = M
(8) sp1 = M, sp2 = 0, sp3 = 0, sp4 = M
These eight patterns.
In the experiment, for the patterns written in (1) to (8), the experiment is performed for the case where the observation signal is 4 seconds and the case where the observation signal is 8 seconds, so the variation of the observation signal is 8 × 2 = 16 in total. Exists.

An example of the observation signal is shown in FIG. This is because [Take No. in the pattern shown in FIG. = 3]. That is, the following output pattern.
(3) sp1 = F, sp2 = S, sp3 = 0, sp4 = M
The four spectrograms X _{1 to} X ₄ shown in FIG. 31A are observation signals observed by the four microphones (mic 1 to mic 4) shown in FIG. FIG. 31B shows the SIR for each frequency bin. It can be seen that the mix of the four sound sources is almost the same among the four spectrograms.

The sound source separation experiment was conducted for the following three methods. That is, (Method 2) was omitted from the above-described experiment, and “(3) Shift stack + instantaneous mixing ICA” (the first method) was performed as Method 4 instead.
(Method 1) Equation [5.2] (equivalent to the conventional method)
(Method 3) Equation [9.5] (hereinafter “Re-STFT”)
(Method 4) Formula [11.1] & Formula [5.2] (hereinafter “shift stack”)

The experimental conditions are as follows.
Common:
Sampling frequency = 16 kHz
Number of sample bits = 16
Observation signal length: 4 and 8 seconds

(Method 1)
STFT window length: 256, 512, 1024, 2048, 4096, 8192
STFT shift width: 1/4 of window length
Window: Hanning window during short-time Fourier transform (STFT), no window during inverse Fourier transform (FT).
η0 = 0.3
Loop count = 400
Γ value of the score function: γ = sqrt (M) M: number of frequency bins

  However, only when the observation signal = 4 seconds and the STFT window length = 8192, the shift width of 1/8 was used. (Because 1/4 shift makes the number of frames too small.)

(Method 3)
STFT (first time) window length: 512
STFT (first time) shift width: 1/4 of the window length
Window (first time): Hanning window during short-time Fourier transform (STFT), no window during inverse Fourier transform (FT).
η0 = 0.3
Loop count = 400
Γ value of score function: γ = sqrt (M (L ′ + 1)) M: number of frequency bins STFT (second time) window length: L ′ + 1 = 4, 8, 16, 32
STFT (second time) shift width: 1/8 of the window length (rounded up)
Window (second time): Hamming window during short-time Fourier transform (STFT), no window during inverse Fourier transform (FT).

  The reason why the Hamming window is used instead of the Hanning window in the second STFT is that the samples at both ends are used effectively even when the number of taps is small. (Since the Hanning window is zero at both ends, the number of effective samples is reduced by two.)

(Method 4)
STFT (first time) window length: 512
STFT (first time) shift width: 1/4 of the window length
Window (first time): Hanning window during short-time Fourier transform (STFT), no window during inverse Fourier transform (FT).
η0 = 0.3
Loop count = 400
Γ value of score function: γ = sqrt (M (L ′ + 1)) M: number of frequency bins frame taps: L ′ + 1 = 2, 4, 8, 12

32 and 33 show the results of processing the observation signal in FIG. 31 by the method 4. FIG. 32 shows the result of shifting and stacking (see FIG. 11) with L ′ = 1 (that is, 2 taps), and FIG. 33 shows the result of separation using 8-channel observation signals. FIG. 33A shows the spectrogram of the separation result, and FIG. 33B shows the SIR for each frequency bin. The spectrogram of the separation result in FIG. 33A corresponds to the original signal as follows.
Y ₂ ^[1] , Y ₄ ^[0] : Sound source 1
Y ₃ ^[0] , Y ₃ ^[1] : Sound source 2
Y ₁ ^[0] , Y ₁ ^[1] : Sound source 3
Y ₂ ^[0] , Y ₄ ^[1] : No correspondence

The average of the improved SIR for each frequency bin was calculated as a measure representing the degree of separation. Taking FIG. 33 as an example, SIR is calculated for the sound source that appears most strongly for each channel (Y ₁ ^{[0] to} Y ₄ ^[1] ) of the separation result spectrogram shown in FIG. In addition, averaged over all frequency bins. For example, since the sound source 1 appears most strongly in Y ₂ ^[1] , the SIR to the sound source 1 is calculated. The same calculation is performed for Y ₄ ^[0] , and the larger value of both values is set as the degree of separation of the sound source 1. The sound sources 2 and 3 are calculated in the same manner and then averaged between the three to obtain the degree of separation as a whole. By subtracting the average SIR of the observed signal from this value, that is, the average of the SIR plots for each frequency bin shown in FIG. 33B at all frequencies, the improved SIR is calculated.

  Finally, the degree of separation for one experimental parameter is calculated by taking the average between 8 takes. In addition, the case where the observation signal was 4 seconds and the case of 8 seconds were counted separately. The tabulation results are as shown in FIG. 34 and FIG. FIG. 34 shows the case where the observation signal = 4 seconds and FIG. 35 shows the case of 8 seconds. In each case, the vertical axis represents the improved SIR and the horizontal axis represents the time span (logarithmic display). The vertical broken lines in both figures represent the room reverberation time, which is 0.275 seconds. The three broken lines correspond to the conventional method (method 1), the re-STFT (method 3), and the shift stack (method 4), respectively.

  As shown in FIGS. 34 and 35, in the conventional method, even if the window length (analysis frame length) of the STFT is increased, the separation accuracy is at a certain value (observation signal = 1024 in 4 seconds, 2048 in 8 seconds). However, if it reaches a peak and is longer than that, the separation accuracy deteriorates. This is because the time resolution of the STFT result decreases if the window is too long. The decrease in time resolution is more strongly affected when the observation signal is shorter, so that the peak is reached when the observation signal is 4 seconds and the window length is shorter than when the observation signal is 8 seconds. On the other hand, when the window is short, although the temporal resolution is high, there are many components across frames (the reverberation does not fit in one frame), so that sufficient separation accuracy cannot be obtained.

  On the other hand, in the method 3 and method 4 of the present invention, the result of STFT with a short window (512 in this experiment) is further separated using a plurality of frames, so that a reduction in time resolution is suppressed while a plurality of frames are used. It can also handle components that straddle. Therefore, even higher separation accuracy can be achieved when compared with the same time span as in the conventional method, and higher separation accuracy can be achieved with a longer time span when compared with peak separation accuracy.

  The present invention has been described in detail above with reference to specific embodiments. However, it is obvious that those skilled in the art can make modifications and substitutions of the embodiments without departing from the gist of the present invention. In other words, the present invention has been disclosed in the form of exemplification, and should not be interpreted in a limited manner. In order to determine the gist of the present invention, the claims should be taken into consideration.

  The series of processes described in the specification can be executed by hardware, software, or a combined configuration of both. When executing processing by software, the program recording the processing sequence is installed in a memory in a computer incorporated in dedicated hardware and executed, or the program is executed on a general-purpose computer capable of executing various processing. It can be installed and run.

  For example, the program can be recorded in advance on a hard disk or ROM (Read Only Memory) as a recording medium. Alternatively, the program is temporarily or permanently stored on a removable recording medium such as a flexible disk, a CD-ROM (Compact Disc Read Only Memory), an MO (Magneto optical) disk, a DVD (Digital Versatile Disc), a magnetic disk, or a semiconductor memory. It can be stored (recorded). Such a removable recording medium can be provided as so-called package software.

  The program is installed on the computer from the removable recording medium as described above, or is wirelessly transferred from the download site to the computer, or is wired to the computer via a network such as a LAN (Local Area Network) or the Internet. The computer can receive the program transferred in this manner and install it on a recording medium such as a built-in hard disk.

  Note that the various processes described in the specification are not only executed in time series according to the description, but may be executed in parallel or individually according to the processing capability of the apparatus that executes the processes or as necessary. Further, in this specification, the system is a logical set configuration of a plurality of devices, and the devices of each configuration are not limited to being in the same casing.

  As described above, according to the configuration of one embodiment of the present invention, an observation signal spectrogram is generated by converting an input signal in which a plurality of sound signals are mixed into the time-frequency domain, and a signal separation result from the observation signal spectrogram. In the signal separation process that generates the A modulation spectrogram is generated by a short-time Fourier transform (STFT) of the direction, the modulation spectrogram is interpreted as instantaneous mixing, and a signal separation result is generated by executing an independence analysis to solve the instantaneous mixing. Therefore, high-accuracy separation processing considering the delay amount is realized for the mixed sound signal having various delay amounts such as a direct wave and a reflected wave.

It is a figure explaining the acquisition structural example of the sound information applied to the isolation | separation process of the sound signal with which the several signal was mixed. It is a figure which shows the relationship of the entropy H ( _Yk ) for one spectrogram about each channel, and the simultaneous entropy H (Y) for one spectrogram about all the channels. It is a figure which shows the graph which plotted the relationship between the window length of STFT, and the separation accuracy of the time frequency domain ICA. It is a figure which shows the graph showing the isolation | separation precision of the time frequency area | region ICA by making window length of a horizontal axis into an actual number of seconds. It is a figure explaining the acquisition structural example of the sound information applied to the isolation | separation process of the sound signal with which the several signal was mixed. It is a figure explaining the concept considered that the time domain convolution mixing is convolution mixing instead of instantaneous mixing in a time frequency domain. It is a figure explaining short-time Fourier transform (STFT). It is a figure explaining conversion from spectrogram X to X '(modulation spectrogram) which carried out short-time Fourier transform (STFT) again in the time direction. It is a figure explaining conversion from spectrogram X to X '(modulation spectrogram) which carried out short-time Fourier transform (STFT) again in the time direction. It is a figure explaining the calculation method of entropy H (Y'k). It is a figure explaining the process which produces | generates the vector piled up vertically, shifting a frame number with respect to an observation signal spectrogram. It is a figure explaining operation which produces | generates a separation result by convolving the tl (ell) + tl (ell) + L'th frame about the observation signal spectrogram X. FIG. It is a figure explaining the process which combined shift accumulation and instantaneous mixing ICA. It is a figure which shows the flowchart which prints explanatoryally about the sequence of the process which combined shift stacking and instantaneous mixing ICA. It is a figure explaining the structural example of the signal separation apparatus of this invention. It is a figure explaining the structural example of the signal separation apparatus of this invention. It is a figure which shows the flowchart explaining the processing sequence of the signal separation apparatus of this invention. It is a figure which shows the flowchart explaining the detailed sequence of the separation process which the signal separation apparatus of this invention performs. It is a figure which shows the flowchart explaining the detailed sequence of the separation process which the signal separation apparatus of this invention performs. It is a figure explaining the process which changes the value of the number of frame taps [L '] for every frequency. It is a figure which shows the flowchart about the process which performs signal reduction by reducing the number of channels by two-stage separation. It is a figure explaining the process to perform. It is a figure explaining the structure of the experimental apparatus for confirmation of the effect of the signal separation apparatus of this invention. It is a figure which shows the evaluation data of the experimental result for confirmation of the effect of the signal separation apparatus of this invention. It is a figure which shows the evaluation data of the experimental result for confirmation of the effect of the signal separation apparatus of this invention. It is a figure which shows the evaluation data of the experimental result for confirmation of the effect of the signal separation apparatus of this invention. It is a figure which shows the evaluation data of the experimental result for confirmation of the effect of the signal separation apparatus of this invention. It is a figure explaining the environment where the evaluation experiment of the signal separation process was performed. It is a figure explaining the sound source applied to the evaluation experiment of signal separation processing. It is a figure explaining the input-output pattern of the sound source applied to the evaluation experiment of signal separation processing. It is a figure explaining the example of the observation signal in the evaluation experiment of signal separation processing. It is a figure explaining the result of having performed the shift & accumulation (refer FIG. 11) obtained in the evaluation experiment of signal separation processing. It is a figure explaining the separation result and SIR which were obtained in the evaluation experiment of signal separation processing. It is a figure explaining the evaluation result obtained in the evaluation experiment of signal separation processing. It is a figure explaining the evaluation result obtained in the evaluation experiment of signal separation processing.

Explanation of symbols

11,12 Entropy 13 Simultaneous entropy 111 Sound source 121 Microphone 201 Frequency bin 202 Bin 203 Bin 221 Modulation spectrogram 222 Multivariate probability density function 223,224 Entropy 401 Microphone 402 AD conversion unit 403 STFT unit 404 Signal separation unit 405 Rescaling unit 406 Inverse FT unit 407 Post-stage processing execution unit 408 Convolution operation unit 409 Control unit 451 Microphone 452 AD conversion unit 453 First STFT unit 454 Second STFT unit 455 Signal separation unit 456 First rescaling unit 457 First inverse FT unit 458 Second rescaling Unit 459 second inverse FT unit 460 post-processing execution unit 461 control unit

Claims (49)

A signal separation device that inputs a signal in which a plurality of signals are mixed and separates them into individual signals,
A signal conversion means for converting the input signal into the time-frequency domain and generating an observed signal spectrogram;
Signal separation means for generating a signal separation result from the observed signal spectrogram generated by the signal conversion means,
The signal separating means includes
A signal separation device characterized in that the observation signal spectrogram is interpreted as an observation signal that is convolutionally mixed in the time-frequency domain, and a signal separation result is generated by executing processing for solving the convolutional mixture in the time-frequency domain.   2. The signal conversion means is configured to execute processing for generating an observation signal spectrogram by performing short-time Fourier transform (STFT) on the input signal to convert to the time-frequency domain. A signal separation device according to claim 1. The signal separating means includes
The separation signal Y (t) of the frame number (t) is set as a convolution mixture of the observation signals X (t−L ′) to X (t), and is an individual signal component included in the separation signal Y (t). 2. The signal separation device according to claim 1, wherein the signal separation device is configured to generate a signal separation result by a process for increasing independence of each of Y1 (t) to Yn (t). The signal separating means includes
As a process for increasing the independence of each of the individual signal components Y1 (t) to Yn (t) included in the separated signal Y (t), an independence calculation measure Kullback-Leiblar information amount I (Y) is used. 4. The signal separation device according to claim 3, wherein the signal separation device is configured to generate a signal separation result by applying a separation matrix updating process that minimizes the Kullback-Leiblar information amount I (Y). The signal separating means includes
A first signal separation result is generated by applying an instantaneous component ICA (Independent Component Analysis) to the observed signal spectrogram, and it is determined from the first signal separation result that it does not correspond to any sound source. 2. The signal separation result is generated by performing unnecessary channel removal processing, performing processing for solving convolutional mixing in a time-frequency domain on an observed signal spectrogram remaining after the removal processing, and generating a signal separation result. Signal separation device.   The processing using the instantaneous mixing ICA generates a time-frequency domain separation signal from the time-frequency domain observation signal and the separation matrix, and generates the time-frequency domain separation signal and a multi-dimensional probability density function. 6. The process according to claim 5, wherein the separation matrix is corrected until the separation matrix calculated by the dimension score function substantially converges, and the modified separation matrix is applied to generate a separation signal in the time-frequency domain. The signal separation device as described. A signal separation device that inputs a signal in which a plurality of signals are mixed and separates them into individual signals,
First signal conversion means for converting an input signal into a time-frequency domain and generating an observation signal spectrogram;
Second signal conversion means for performing data conversion on the observed signal spectrogram generated by the first signal conversion means to generate a modulation spectrogram;
Signal separation means for generating a signal separation result from the modulation spectrogram generated by the second signal conversion means;
The signal separating means includes
A signal separation device characterized in that the modulation spectrogram is interpreted as instantaneous mixing and a signal separation result is generated.   The first signal conversion unit is configured to execute a process of performing a short-time Fourier transform (STFT) on the input signal to convert the input signal into a time-frequency domain to generate an observation signal spectrogram. Item 8. The signal separation device according to Item 7. The second signal converting means includes
A configuration for generating a modulation spectrogram as a result of performing a short-time Fourier transform (STFT) in the time direction on the observed signal spectrogram;
The signal separating means includes
8. The signal separation device according to claim 7, wherein the signal separation device is configured to generate a signal separation result by a process for increasing independence of each of the signal components Y1 ′ to Yn ′ corresponding to the separation signal included in the modulation spectrogram. The signal separating means includes
As a process for increasing the independence of each of the signal components Y1 ′ to Yn ′ corresponding to the separation signal, the Kullback-Leiblar information amount, which is an independence calculation measure, is applied, and the separation matrix updating process for minimizing the Kullback-Leiblar information amount The signal separation device according to claim 9, wherein the signal separation result is generated by the method. The signal separation device further includes:
Inverse Fourier transform means for generating spectrograms Y1 to Yn corresponding to separated signals by performing inverse Fourier transform on the signal components Y1 'to Yn' corresponding to the separated signals obtained by the signal separating means. The signal separation device according to claim 7. The signal separation device further includes:
A first signal separation result is generated by a process of applying instantaneous mixed ICA (Independent Component Analysis) to the observed signal spectrogram generated by the first signal converting means, and from the first signal separation result, which It has unnecessary channel removal means for executing unnecessary channel removal processing that is determined not to correspond to a sound source,
The second signal converting means and the signal separating means are:
8. The signal separation device according to claim 7, wherein the signal separation result is generated by executing only the processing on the signal after the unnecessary channel is removed.   The processing using the instantaneous mixing ICA generates a time-frequency domain separation signal from the time-frequency domain observation signal and the separation matrix, and generates the time-frequency domain separation signal and a multi-dimensional probability density function. The processing according to claim 12, wherein the separation matrix is corrected until the separation matrix calculated by the dimension score function substantially converges, and the modified separation matrix is applied to generate a separation signal in the time-frequency domain. The signal separation device as described. A signal separation device that inputs a signal in which a plurality of signals are mixed and separates them into individual signals,
A signal conversion means for converting the input signal into the time-frequency domain and generating an observed signal spectrogram;
Signal separation means for generating a signal separation result from the observed signal spectrogram generated by the signal conversion means,
The signal separating means includes
The observed signal spectrogram is shifted in the frame direction to generate an observed signal spectrogram shift set in which data having different shift amounts are stacked, and an instantaneous mixed ICA (Independent Component Analysis) is generated with respect to the generated observed signal spectrogram shift set. ) Is applied to generate a signal separation result.   The processing using the instantaneous mixing ICA generates a time-frequency domain separation signal from the time-frequency domain observation signal and the separation matrix, and generates the time-frequency domain separation signal and a multi-dimensional probability density function. 15. The process according to claim 14, wherein the separation matrix is corrected until the separation matrix calculated by the dimension score function substantially converges, and the modified separation matrix is applied to generate a separation signal in the time-frequency domain. The signal separation device as described. The signal separating means includes
Apply the instantaneous mixing ICA to the multi-channel observation signal spectrogram shift set obtained by stacking a plurality of observation signal spectrogram shift sets generated corresponding to the observation signals of a plurality of signal input sources. The signal separation device according to claim 14, wherein the signal separation device is generated. The signal separating means includes
15. The observation signal spectrogram shift set is generated by copying and setting a gap generated during the shift by copying zero or a value close to zero or values at both ends of the observation signal spectrogram. A signal separation device according to claim 1. The signal separating means includes
The signal separation device according to claim 14, wherein a cyclic shift process is performed in which data at one end protruding from the shift is copied to the other end. The signal separating means includes
Observation with multiple shift data set as the number of frame taps [L '] when the minimum shift amount is 0 and the maximum shift amount is generated from the observation signal, and the data with different shift amounts generated is stacked 15. The signal separation device according to claim 14, wherein a signal spectrogram shift set is generated. The signal separating means includes
The signal separation device according to claim 14, wherein the observation signal spectrogram shift set is generated by changing the number of frame taps [L ′] according to a frequency. The signal separating means includes
A first signal separation result is generated by applying an instantaneous component ICA (Independent Component Analysis) to the observed signal spectrogram, and it is determined from the first signal separation result that it does not correspond to any sound source. Execute unnecessary channel removal processing, shift the observed signal spectrogram remaining after the removal processing in the frame direction to generate an observed signal spectrogram shift set, and apply instantaneous mixing ICA to the generated observed signal spectrogram shift set The signal separation device according to claim 14, wherein the signal separation result is generated. A signal separation device that inputs a signal in which a plurality of signals are mixed and separates them into individual signals,
A signal conversion means for converting the input signal into the time-frequency domain and generating an observed signal spectrogram;
Signal separation means for generating a signal separation result from the observed signal spectrogram generated by the signal conversion means,
The signal separating means includes
For the observed signal spectrogram, signal separation results Y1 to Yn are generated by processing using instantaneous mixed ICA (Independent Component Analysis),
A signal spectrogram corresponding to each of the signal separation results Y1 to Yn is shifted in the frame direction to generate an observation signal spectrogram shift set in which data having different shift amounts are stacked, and for the generated observation signal spectrogram shift set The signal is characterized in that it performs a dereverberation process by a process to which instantaneous mixing ICA (Independent Component Analysis) is applied, and generates a signal separation result from which dereverberation is removed by an integration process of the spectrogram after dereverberation. Separation device.   The processing using the instantaneous mixing ICA generates a time-frequency domain separation signal from the time-frequency domain observation signal and the separation matrix, and generates the time-frequency domain separation signal and a multi-dimensional probability density function. 23. The processing according to claim 22, wherein the separation matrix is corrected until the separation matrix calculated by the dimension score function substantially converges, and the modified separation matrix is applied to generate a separation signal in the time-frequency domain. The signal separation device as described. In the signal separation device, a signal separation method for executing a process of inputting a signal obtained by mixing a plurality of signals and separating the signal into individual signals,
A signal conversion step in which the signal conversion means converts the input signal into the time-frequency domain and generates an observation signal spectrogram;
The signal separation means includes a signal separation step of generating a signal separation result from the observed signal spectrogram generated in the signal conversion step;
The signal separation step includes
A signal separation method comprising: interpreting the observed signal spectrogram as an observation signal mixed in a convolution in the time-frequency domain, and generating a signal separation result by executing a process for solving the convolutional mixture in the time-frequency domain.   25. The signal converting step is a step of executing a process of generating an observed signal spectrogram by performing a short-time Fourier transform (STFT) on the input signal to convert it to a time-frequency domain. The signal separation method according to 1. The signal separation step includes
The separation signal Y (t) of the frame number (t) is set as a convolution mixture of the observation signals X (t−L ′) to X (t), and is an individual signal component included in the separation signal Y (t). 25. The signal separation method according to claim 24, wherein the signal separation method is a step of generating a signal separation result by processing for increasing independence of each of Y1 (t) to Yn (t). The signal separation step includes
As a process for increasing the independence of each of the individual signal components Y1 (t) to Yn (t) included in the separated signal Y (t), an independence calculation measure Kullback-Leiblar information amount I (Y) is used. 27. The signal separation method according to claim 26, wherein the signal separation result is generated by applying a separation matrix updating process that minimizes the Kullback-Leiblar information amount I (Y). The signal separation step includes
A first signal separation result is generated by applying an instantaneous component ICA (Independent Component Analysis) to the observed signal spectrogram, and it is determined from the first signal separation result that it does not correspond to any sound source. 25. A step of generating a signal separation result by executing unnecessary channel removal processing, executing processing for solving convolutional mixing in a time-frequency domain with respect to an observation signal spectrogram remaining after the removal processing. The signal separation method according to 1.   The processing using the instantaneous mixing ICA generates a time-frequency domain separation signal from the time-frequency domain observation signal and the separation matrix, and generates the time-frequency domain separation signal and a multi-dimensional probability density function. 29. The process according to claim 28, wherein the separation matrix is corrected until the separation matrix calculated by the dimension score function substantially converges, and the modified separation matrix is applied to generate a separation signal in the time-frequency domain. The signal separation method described. In the signal separation device, a signal separation method for executing a process of inputting a signal obtained by mixing a plurality of signals and separating the signal into individual signals,
A first signal converting step in which a first signal converting means converts an input signal into a time-frequency domain and generates an observed signal spectrogram;
A second signal converting step in which a second signal converting means performs data conversion on the observed signal spectrogram generated in the first signal converting step to generate a modulation spectrogram;
A signal separation step of generating a signal separation result from the modulation spectrogram generated in the second signal conversion step;
The signal separation step includes
A signal separation method comprising the step of interpreting the modulation spectrogram as instantaneous mixing and generating a signal separation result. The first signal converting step includes:
31. The signal separation method according to claim 30, wherein the signal separation method is a step of executing short-time Fourier transform (STFT) on the input signal to convert the input signal into a time-frequency domain to generate an observation signal spectrogram. . The second signal converting step includes
Generating a modulation spectrogram as a result of performing a short time Fourier transform (STFT) in the time direction on the observed signal spectrogram;
The signal separation step includes
31. The signal separation method according to claim 30, wherein a signal separation result is generated by a process of increasing the independence of each of the signal components Y1 'to Yn' corresponding to the separation signal included in the modulation spectrogram. The signal separation step includes
As a process for increasing the independence of each of the signal components Y1 ′ to Yn ′ corresponding to the separation signal, the Kullback-Leiblar information amount, which is an independence calculation measure, is applied, and the separation matrix updating process for minimizing the Kullback-Leiblar information amount 33. The signal separation method according to claim 32, wherein a signal separation result is generated by the method. The signal separation method further includes:
The inverse Fourier transform means performs inverse Fourier transform on each of the signal components Y1 ′ to Yn ′ corresponding to the separated signals obtained in the signal separating step to generate spectrograms Y1 to Yn corresponding to the separated signals. The signal separation method according to claim 30, further comprising steps. The signal separation method further includes:
An unnecessary channel removing unit generates a first signal separation result by a process in which instantaneous mixed ICA (Independent Component Analysis) is applied to the observed signal spectrogram generated by the first signal converting unit, and the first signal separation result is generated. From the signal separation result, there is an unnecessary channel removal step for performing an unnecessary channel removal process that is determined not to correspond to any sound source,
The second signal converting means and the signal separating means are:
31. The signal separation method according to claim 30, wherein a signal separation result is generated by performing only processing on the signal after unnecessary channel removal.   The processing using the instantaneous mixing ICA generates a time-frequency domain separation signal from the time-frequency domain observation signal and the separation matrix, and generates the time-frequency domain separation signal and a multi-dimensional probability density function. 36. The process according to claim 35, wherein the separation matrix is corrected until the separation matrix calculated by the dimension score function substantially converges, and the modified separation matrix is applied to generate a separation signal in the time-frequency domain. The signal separation method described. In the signal separation device, a signal separation method for inputting a signal in which a plurality of signals are mixed and separating them into individual signals,
A signal conversion step in which the signal conversion means converts the input signal into the time-frequency domain and generates an observation signal spectrogram;
The signal separation means includes a signal separation step of generating a signal separation result from the observed signal spectrogram generated in the signal conversion step;
The signal separation step includes
The observed signal spectrogram is shifted in the frame direction to generate an observed signal spectrogram shift set in which data having different shift amounts are stacked, and an instantaneous mixed ICA (Independent Component Analysis) is generated with respect to the generated observed signal spectrogram shift set. The signal separation method is a step of generating a signal separation result by the processing to which (1) is applied.   The processing using the instantaneous mixing ICA generates a time-frequency domain separation signal from the time-frequency domain observation signal and the separation matrix, and generates the time-frequency domain separation signal and a multi-dimensional probability density function. The process according to claim 37, wherein the separation matrix is corrected until the separation matrix calculated by the dimension score function substantially converges, and the modified separation matrix is applied to generate a separation signal in the time-frequency domain. The signal separation method described. The signal separation step includes
Apply the instantaneous mixing ICA to the multi-channel observation signal spectrogram shift set obtained by stacking a plurality of observation signal spectrogram shift sets generated corresponding to the observation signals of a plurality of signal input sources. The signal separation method according to claim 37, wherein the signal separation method is generated. The signal separation step includes
38. The observed signal spectrogram shift set is generated by copying and setting a gap generated during the shift by copying a value of zero or a value close to zero or values at both ends of the observed signal spectrogram. The signal separation method according to 1. The signal separation step includes
38. The signal separation method according to claim 37, wherein a cyclic shift process is performed in which data at one end protruding from the shift is copied to the other end. The signal separation step includes
Observation with multiple shift data set as the number of frame taps [L '] when the minimum shift amount is 0 and the maximum shift amount is generated from the observation signal, and the data with different shift amounts generated is stacked 38. The signal separation method according to claim 37, wherein a signal spectrogram shift set is generated. The signal separation step includes
The signal separation method according to claim 37, wherein the observation signal spectrogram shift set is generated by changing the number of frame taps [L '] according to a frequency. The signal separation step includes
A first signal separation result is generated by applying an instantaneous component ICA (Independent Component Analysis) to the observed signal spectrogram, and it is determined from the first signal separation result that it does not correspond to any sound source. Execute unnecessary channel removal processing, shift the observed signal spectrogram remaining after the removal processing in the frame direction to generate an observed signal spectrogram shift set, and apply instantaneous mixing ICA to the generated observed signal spectrogram shift set 38. The signal separation method according to claim 37, further comprising: generating a signal separation result. In the signal separation device, a signal separation method for inputting a signal in which a plurality of signals are mixed and separating them into individual signals,
A signal conversion step in which the signal conversion means converts the input signal into the time-frequency domain and generates an observation signal spectrogram;
The signal separation means includes a signal separation step of generating a signal separation result from the observed signal spectrogram generated in the signal conversion step;
The signal separation step includes
For the observed signal spectrogram, signal separation results Y1 to Yn are generated by processing using instantaneous mixed ICA (Independent Component Analysis),
A signal spectrogram corresponding to each of the signal separation results Y1 to Yn is shifted in the frame direction to generate an observation signal spectrogram shift set in which data having different shift amounts are stacked, and for the generated observation signal spectrogram shift set The signal is characterized in that it is a step of performing a dereverberation process by a process to which instantaneous mixing ICA (Independent Component Analysis) is applied, and generating a signal separation result from which the dereverberation is removed by an integration process of the dereverberation spectrogram. Separation method.   The processing using the instantaneous mixing ICA generates a time-frequency domain separation signal from the time-frequency domain observation signal and the separation matrix, and generates the time-frequency domain separation signal and a multi-dimensional probability density function. 46. The processing according to claim 45, wherein the separation matrix is corrected until the separation matrix calculated by the dimension score function substantially converges, and the modified separation matrix is applied to generate a separation signal in a time-frequency domain. The signal separation method described. In the signal separation device, a computer program for executing a signal separation process for inputting a signal obtained by mixing a plurality of signals and separating the signal into individual signals,
A signal conversion step for causing the signal conversion means to convert the input signal into the time-frequency domain and generate an observed signal spectrogram;
A signal separation step for causing the signal separation means to generate a signal separation result from the observed signal spectrogram generated in the signal conversion step;
The signal separation step includes
A computer program comprising the steps of interpreting the observed signal spectrogram as an observation signal mixed in the time-frequency domain and generating a signal separation result by executing a process for solving the convolutional mixing in the time-frequency domain. In the signal separation device, a computer program for executing a signal separation process for inputting a signal obtained by mixing a plurality of signals and separating the signal into individual signals,
A signal conversion step of causing the first signal conversion means to convert the input signal into the time-frequency domain and generate an observation signal spectrogram;
A second signal converting step for causing the second signal converting means to perform data conversion on the observed signal spectrogram generated in the first signal converting step to generate a modulation spectrogram;
A signal separation step of causing a signal separation means to generate a signal separation result from the modulation spectrogram generated in the second signal conversion step;
The signal separation step includes
A computer program comprising the step of interpreting the modulation spectrogram as instantaneous mixing and generating a signal separation result. In the signal separation device, a computer program for executing a signal separation process for inputting a signal obtained by mixing a plurality of signals and separating the signal into individual signals,
A signal conversion step for causing the signal conversion means to convert the input signal into the time-frequency domain and generate an observed signal spectrogram;
A signal separation step for causing the signal separation means to generate a signal separation result from the observed signal spectrogram generated in the signal conversion step;
The signal separation step includes
The observed signal spectrogram is shifted in the frame direction to generate an observed signal spectrogram shift set in which data having different shift amounts are stacked, and an instantaneous mixed ICA (Independent Component Analysis) is generated for the generated observed signal spectrogram shift set. The computer program is a step of generating a signal separation result by the processing to which (1) is applied.