JP5791081B2 - Sound source separation localization apparatus, method, and program - Google Patents

Description

  The present invention relates to a sound source separation localization apparatus, method, and program, and in particular, separates sound for each individual sound source from a mixed sound generated from each of a plurality of sound sources and localizes the direction of each sound source. The present invention relates to a sound source separation localization apparatus, method, and program.

  A sound source separation technique for separating an environmental sound (hereinafter referred to as a mixed sound), which is a superposition of sounds emitted from each of a plurality of sound sources, into a sound for each individual sound source has a very long history. This technique can be used, for example, for speaker separation for creating a meeting minutes from a mixed sound recording a meeting. The sound source localization technology that calculates the relative position and direction of each sound source from the positional relationship of the plurality of microphones observing the mixed sound and the sound observed by each microphone is, for example, a robot or machine that autonomously moves in the environment. A large number of techniques have been proposed as basic techniques for self-position identification and obstacle avoidance (for example, Non-Patent Documents 1 to 3).

  Non-Patent Document 1 proposes a sound source separation method using the sparseness of a sound source in which only a signal from at most one sound source is usually observed at each time and each frequency. Non-Patent Document 2 proposes a system that performs sound source separation and localization on the premise of use in a robot. Non-Patent Document 3 proposes a sound source separation method using more microphones than the number of sound sources.

  It is known that the two problems of sound source separation and sound source localization are interdependence problems that are deeply and closely related to each other. For example, when the positions of a plurality of sound sources are known, it is known that the separated sound of each sound source can be accurately restored by using a technique called a beam former. On the other hand, when the sound of each sound source is separated, it is relatively easy to determine the position of each sound source.

Sawada, H., Araki, S. and Makino, S. “Underdetermined Convolutive Blind Source Separation via Frequency Bin-Wise Clustering and Permutation Alignment”, IEEE Transactions on Audio, Speech and Language Processing, Vol. 19, No. 3, pp 516-527, 2011. Nakadai, K. Lourens, T., Okuno, H. G. and Kitano, H. “Active Audition for Humanoid”, in Proc. AAAI, 2000. Lee, I., Kim, T. and Lee, T.-W., “Fast Fixed-point Independent Vector Analysis Algorithms for Convolutive Blind Source Separation”, Signal Processing, Vol. 87, No. 8, pp.1859-1871 , 2007.

  If the above two problems of sound source separation and sound source localization can be solved at the same time, for example, an autonomous robot acts by avoiding an obstacle while receiving a command command of a specific user in a noisy environment. A very advanced intelligent system can be realized.

  However, the existing methods represented by Non-Patent Document 1 individually solve the interdependent problems of sound source separation and sound source localization. For example, the method of Non-Patent Document 1 is mainly intended for sound source separation, and assumes that each sound source is localized after the sound source separation is completed. On the other hand, the method of Non-Patent Document 2 separates the sound signal emitted from each sound source after the localization of each sound source is completed. Like these conventional methods, after solving one problem of sound source separation and sound source localization with a method that involves some prior information and strong assumptions, when solving the other problem, There is a problem that when the accuracy of a problem is poor, the accuracy of the other problem is greatly degraded.

  The present invention has been made to solve the above problems, and a sound source separation localization apparatus, method, and program capable of stably obtaining high performance with respect to both problems of sound source separation and sound source localization. The purpose is to provide.

  In order to achieve the above object, the sound source separation and localization apparatus of the present invention is a mixed sound signal obtained by observing a mixed sound of each sound emitted from each of a plurality of sound sources by a plurality of observation means arranged at different positions. Receiving means, sound source separation for separating the mixed sound signal received by the receiving means so as to correspond to each of the plurality of sound sources, and each of the plurality of sound sources based on the observation means Analysis means for analyzing the sound source localization for estimating the direction by simultaneous optimization using iterative processing using variables mutually dependent on the sound source separation and the sound source localization; and the sound source separation analyzed by the analysis means and Output means for outputting the result of sound source localization.

  According to the sound source separation and localization apparatus of the present invention, the reception unit receives a mixed sound signal obtained by observing a mixed sound of each sound emitted from each of the plurality of sound sources by a plurality of observation units arranged at different positions. . Then, the analysis unit estimates the direction in which each of the plurality of sound sources exists based on the sound source separation that separates the mixed sound signal received by the reception unit so as to correspond to each of the plurality of sound sources. Sound source localization is analyzed by simultaneous optimization using iterative processing using variables that are mutually dependent on sound source separation and sound source localization. The use of mutually dependent variables means that a variable obtained by one of sound source separation and sound source localization is used when obtaining the other variable. Finally, the output means outputs the sound source separation and sound source localization results analyzed by the analysis means.

  As described above, by performing the optimization by making the sound source separation and the sound source localization mutually depend on each other, the high performance can be stably obtained with respect to the problems of both the sound source separation and the sound source localization.

In addition, the reception unit can convert the mixed sound signal into an observation signal _xtf in a time-frequency domain including elements for each of the time frame t and the frequency bin f and pass it to the analysis unit. Further, the analyzing means represents a probability that each element of the observation signal _xtf is a signal corresponding to a kth mask of a plurality of masks that assign each element to each of a plurality of virtually set sound sources. A sound source time-frequency mask variable calculation unit that calculates a mask variable ξ _tfk for each of the plurality of masks, and a d-th component in a plurality of directions in which a sound source corresponding to the k-th mask is divided based on the observation unit. A sound source localization variable η _kd representing the probability of existing in the direction of the sound source for each of the plurality of directions, a statistic used for calculating the mask variable ξ _tfk and the sound source localization variable η _kd Statistic calculation means for calculating the sound source time frequency mask variable calculation means, the sound source localization variable calculation means, and the calculation of the statistic calculation means with a predetermined convergence A convergence condition determination means for repeated until the condition, can be configured to include, the sound source localization variable eta _kd used in the calculation of the mask variables xi] _TFK, the mask variables in the calculation of the sound source localization variable eta _kd ξ _tfk can be used. This makes it possible to efficiently perform simultaneous optimization by making sound source separation and sound source localization depend on each other.

  Further, the analysis unit can analyze the sound source separation and the sound source localization using steering vectors of the plurality of observation units measured in an anechoic environment. Thereby, it is applicable also to various reverberation environments.

  The sound source separation and localization method of the present invention is a sound source separation and localization method in a sound source separation and localization apparatus including a reception means, an analysis means, and an output means, wherein the reception means is emitted from each of a plurality of sound sources. A mixed sound signal obtained by observing the mixed sound of each sound by a plurality of observation means arranged at different positions, and the analysis means receives the mixed sound signal received by the reception means for each of the plurality of sound sources. And the sound source localization for estimating the direction in which each of the plurality of sound sources is present based on the observation means are mutually dependent on the sound source separation and the sound source localization. In this method, analysis is performed by simultaneous optimization using iterative processing using variables, and the output means outputs the result of sound source separation and sound source localization analyzed by the analysis means.

Further, the analysis means is a sound source separation localization method in a sound source separation localization apparatus including a sound source time frequency mask variable calculation means, a sound source localization variable calculation means, a statistic calculation means, and a convergence condition determination means, The accepting means converts the mixed sound signal into an observation signal _xtf in the time frequency domain composed of each element for each time frame t and frequency bin f and passes it to the analyzing means, and the sound source time frequency mask variable calculating means , A mask variable ξ _tfk representing the probability that each element of the observed signal x _tf is a signal corresponding to the k-th mask of a plurality of masks that assign each element to a plurality of virtually set sound sources, Each of the plurality of masks is calculated, and the sound source localization variable calculating unit is configured to generate a plurality of sound sources corresponding to the k-th mask divided based on the observation unit. The sound source localization variable eta _kd representing the probability that exist d th direction of direction, was calculated for each of the plurality of directions, the statistic calculation means, the calculation of the mask variables xi] _TFK and the sound source localization variable eta _kd Until the convergence condition determining means calculates the sound source time frequency mask variable calculating means, the sound source localization variable calculating means, and the statistics calculating means until a predetermined convergence condition is satisfied. cycled, the sound source localization variable eta _kd used in the calculation of the mask variables xi] _TFK, it is possible to use the mask variables xi] _TFK in the calculation of the sound source localization variable eta _kd.

  In the sound source separation localization method of the present invention, the analysis unit can analyze the sound source separation and the sound source localization using steering vectors of the plurality of observation units measured in an anechoic environment.

  The sound source separation localization program of the present invention is a program for causing a computer to function as each means constituting the sound source separation localization device.

  As described above, according to the sound source separation localization apparatus, method, and program of the present invention, both sound source separation and sound source localization are obtained by analyzing sound source separation and sound source localization by making them mutually dependent and analyzing by simultaneous optimization. With respect to the problem, it is possible to obtain an effect that high performance can be stably obtained.

It is an image figure which shows the outline | summary (sound source separation) of this Embodiment. It is an image figure which shows the outline | summary (sound source localization) of this Embodiment. It is a block diagram which shows the functional structure of the sound source separation localization apparatus which concerns on this Embodiment. It is a figure which shows the structure of a memory | storage part. It is a flowchart which shows the content of the sound source separation localization process routine in this Embodiment. It is a flowchart which shows the content of the initial value production | generation processing routine. It is a flowchart which shows the content of the sound source time frequency mask variable calculation processing routine. It is a flowchart which shows the content of the sound source localization variable calculation processing routine. It is a flowchart which shows the content of the statistics calculation processing routine. It is the schematic which shows the setup of an experiment example. It is a graph which shows the performance of the sound source localization in an experiment example. It is a graph which shows the performance of the sound source separation in an experiment example.

  Hereinafter, embodiments of the present invention will be described in detail with reference to the drawings.

<Overview>
First, an outline of the present embodiment will be described. 1 and 2 are image diagrams showing an outline of the present embodiment.

  As shown in FIG. 1, sound source separation is realized by assigning each (t, f) element in an observed signal obtained by converting the observed mixed sound into a signal in the time-frequency domain by Fourier transform, to K types of sound sources. . Note that t is an index representing a time frame, and f is an index representing a frequency bin. This method is used in Non-Patent Document 1 as a sound source separation method using the sparsity of observation signals in the time-frequency domain, and is known to exhibit good sound source separation performance. The assignment of observation signals in the time-frequency domain to each sound source is called a “sound source time-frequency mask”. With this mask, the mixed sound can be separated for each sound source, and the sound emitted from each sound source can be restored.

  As shown in FIG. 2, the sound source localization is realized by determining the direction of the K types of sound sources to be one of 360-degree directions around the microphone array. Mathematically, the direction is discretized (divided) into D types according to constraints such as the direction resolution of the microphone. Then, each sound source is localized by assigning it to one of the D types of directions, that is, by clustering in the D types of directions.

  Although these individual methods are not new, the present embodiment is characterized by a framework that solves sound source separation and sound source localization simultaneously and in an interdependent manner. That is, in the present embodiment, sound source separation is enabled by calculating a sound source time frequency mask. In addition, sound source localization can be performed by calculating the direction of each sound source cluster. Further, the present invention is characterized in that simultaneous separation and localization of a plurality of sound sources can be achieved by alternately optimizing the sound source separation and sound source localization simultaneously and converging them by an iterative calculation method.

  Furthermore, another feature of the present embodiment is that an anechoic steering vector of a microphone array is used. An anechoic steering vector is an anechoic chamber impulse response, which is an acoustically unique property of each microphone array. This information is very effective in predicting the impulse response, although the impulse response in the observation situation in an actual anechoic environment is different. In this embodiment, the anechoic steering vector is measured and input in advance, and the acoustic characteristics suitable for the actual mixed sound are estimated simultaneously with sound source separation and localization. As a result, good separation and localization performance can be obtained regardless of the reverberant environment.

<System configuration>
A sound source separation localization apparatus 10 according to the present embodiment includes a CPU (Central Processing Unit), a RAM (Random Access Memory), and a ROM (Read Only Memory) that stores a program for executing a sound source separation localization processing routine described later. ), And the CPU is formed by reading and executing a program for executing the sound source separation localization processing routine from the ROM which is an internal storage device.

  As shown in FIG. 3, this computer functionally includes a reception unit 1 that receives input of mixed sound to be analyzed and data indicating the acoustic characteristics of the microphone array, and variables necessary for sound source separation and sound source localization analysis. It can be expressed by a configuration including an analysis unit 2 that calculates and updates, a storage unit 3 that stores received data and calculated information, and an output unit 4 that outputs an analysis result.

  The receiving unit 1 can be expressed by a configuration including a mixed sound observation unit 11, a time frequency domain observation conversion unit 12, and a preset value reception unit 13.

  The mixed sound observation unit 11 receives a mixed sound signal obtained by converting the observed mixed sound into electronic data from an input device such as a storage device or a microphone array attached to the apparatus. For example, the mixed sound signal that has already been observed by the microphone array and converted into electronic data and then temporarily stored in the storage device can be received as input data by reading from the storage device. Moreover, the mixed sound observed with the microphone array attached to this apparatus can be directly converted into electronic data and accepted.

  The time frequency domain observation conversion unit 12 converts the mixed sound signal received by the mixed sound observation unit 11 into a time frequency domain signal using Fourier transform. Hereinafter, a signal obtained by converting the mixed sound signal into the time frequency domain is referred to as an observation signal.

  The preset value receiving unit 13 receives statistics from the input device such as a keyboard or a storage device, including constants necessary for a model mounted on the device, which will be described later, and anechoic steering vector information of the microphone array in which the input mixed sound is observed. A part of the initial value of the quantity and the value of the convergence determination threshold value are received.

  The analysis unit 2 further includes an initial value generation unit 21, a sound source time frequency mask variable calculation unit 22, a sound source localization variable calculation unit 23, a statistic calculation unit 24, and a convergence condition determination unit 25. Can be represented.

  The initial value generation unit 21 stores the information received by the reception unit 1 in each unit of the storage unit 3 and initializes values stored in each unit of the storage unit 3.

  The sound source time frequency mask variable calculation unit 22 uses the information stored in the storage unit 3 to calculate, store, and update the sound source time frequency mask variable.

  The sound source localization variable calculator 23 uses the information stored in the storage unit 3 to calculate, store, and update the sound source localization variable.

  The statistic calculator 24 uses the information stored in the storage unit 3 to calculate a statistic and save and update it.

  The convergence condition determination unit 25 uses the information stored in the storage unit 3 to determine whether to continue or end the calculation process of the analysis unit 2. In the case of termination, the analysis result is passed to the output unit 4.

  As shown in FIG. 4, the storage unit 3 includes a constant storage unit 31, an observation signal storage unit 32, a sound source time frequency mask variable storage unit 33, a sound source localization variable storage unit 34, a statistic storage unit 35, and a statistic initial value. Storage units 36 and a convergence determination threshold storage unit 37 are provided.

  The constant storage unit 31 stores constants necessary for the model mounted on the apparatus.

  The observation signal storage unit 32 stores the observation signal converted by the time frequency domain observation conversion unit 12.

  The sound source time frequency mask variable storage unit 33 stores a sound source time frequency mask variable that mainly represents information representing the analysis result of sound source separation.

  The sound source localization variable storage unit 34 stores a sound source localization variable that mainly represents information representing the analysis result of the sound source localization.

  The statistics storage unit 35 stores various statistics required for sound source separation and sound source localization.

  The statistic initial value storage unit 36 stores a statistic initial value that is an initial value necessary for calculating a statistic.

  The convergence determination threshold storage unit 37 stores a threshold used for determining the convergence of the analysis result.

  The output unit 4 can be expressed by a configuration including a separated sound extraction unit 41, a speech waveform restoration unit 42, a sound source direction extraction unit 43, and a final output unit 44.

  The separated sound extraction unit 41 uses the information stored in the storage unit 3 to calculate a separated sound signal in the time-frequency domain and passes it to the speech waveform restoration unit 42.

  The speech waveform restoration unit 42 uses the information stored in the storage unit 3 and the time frequency domain signal of each separated sound passed from the separated sound extraction unit 41 to inverse Fourier transform the time frequency domain signal of each separated sound. The sound is restored to a separated sound signal by conversion.

  The sound source direction extraction unit 43 uses the information stored in the storage unit 3 to calculate and localize the direction of each sound source.

  The final output unit 44 outputs sound source separation and sound source localization analysis results in a format desired by the user to an output device mounted with a display, a printer, a speaker, a magnetic disk, or the like.

<Operation of the present embodiment>
Next, the operation of the sound source separation localization apparatus 10 according to the present embodiment will be described. First, a mixed sound observed using a microphone array in which a plurality of microphones are arranged in an arbitrary arrangement is stored as a mixed sound signal in a storage device, or a mixed sound is observed by a microphone array attached to this device. In the sound source separation localization apparatus 10, a sound source separation localization processing routine shown in FIG. 5 is executed.

  In step 100, the mixed sound observation unit 11 accepts the mixed sound signal stored in the storage device by reading it, or directly converts the mixed sound observed by the microphone array into a mixed sound signal that is electronic data. Accept.

Next, in step 102, the time frequency domain observation conversion unit 12 converts the mixed sound signal received in step 100 into an observation signal that is a signal in the time frequency domain. For the conversion, short-time Fourier transform (STFT) or fast Fourier transform (FFT) can be used. The converted observation signal is represented by _xtf . Each _xtf is a converted expression of the audio signal in the time frame t (t = 1,..., T) and the f (f = 1,..., F) th frequency bin (frequency band) by Fourier transform. . Each _xtf is a vector of the number of elements corresponding to the number of microphones, and each element is a complex number. Hereinafter, this _xtf is used as an observation amount for this apparatus. This observation amount is assumed to be generated from a complex normal distribution in the present embodiment. All the input mixed sound signals are converted into the time-frequency domain, and each converted _xtf is passed to the initial value generation unit 21.

  Next, in step 104, the preset value receiving unit 13 receives constants necessary for analysis processing in the sound source separation localization apparatus 10. The constants include the total number of time frames T of the observed signal, the total number of frequency bins F of the observed signal, the number of masks K that is the maximum number of sound sources virtually set, the number of direction classes D for clustering the sound source directions, and the mixture The number of microphones M of the microphone array in which the sound is observed and the regularization constant ε that is a positive real number used when calculating the initial value of the statistic are included. The number K of masks can be set to 12, for example, but when a larger number of sound sources are expected, a number sufficiently exceeding the number of sound sources is set. Further, the regularization constant ε can be set to 0.0001, for example.

The preset value receiving unit 13 also receives part of the initial values of the statistics necessary for the analysis (β _k ⁰ , κ _d ⁰ , a _tf ⁰ , v _fd ⁰ ). β _k ⁰ is an initial parameter of the Dirichlet distribution used in the mathematical model of the sound source time frequency mask variable, and a value larger than 0 is set for k = 1,. For example, β ₁ ⁰ = β ₂ ⁰ = ... = β _K ⁰ . κ _d ⁰ is an initial parameter of the Dirichlet distribution used in the mathematical model of the sound source localization variable, and a value larger than 0 is set for d = 1,. For example, κ ₁ ⁰ = κ ₂ ⁰ = ... = κ _D ⁰ . a _tf ⁰ is an initial parameter of the gamma distribution used in the mathematical model of the time-frequency domain observation signal. v _fd ⁰ is an initial parameter of the complex Wishart distribution used in the mathematical model of the time-frequency domain observation signal. For example, it is possible to set a _tf ⁰ = 1 and v _fd ⁰ = M for t = 1,..., T, f = 1 _,. .

  Furthermore, the preset value receiving unit 13 also receives an anechoic steering vector q representing the acoustic characteristics of the microphone array. The anechoic steering vector is measured in advance in the anechoic chamber for each of the M microphones as shown in Equation (1) for each frequency bin f and direction d.

  For example, conventional methods such as Non-Patent Document 1 and Non-Patent Document 3 do not use acoustic characteristics unique to the system, such as arrangement of microphones to be used and impulse response in an anechoic room. Although this has a generality that does not depend on the setting of the system to be used, the possibility of obtaining high-precision sound source separation and localization performance in various reverberant environments is reduced because the prior information on acoustic characteristics cannot be used.

  Therefore, in the present embodiment, high-accuracy analysis that can be applied to various reverberant environments can be realized by measuring and inputting an anechoic steering vector, which is an acoustic characteristic unique to the system, in advance.

Further, the preset value receiving unit 13 also receives a convergence determination threshold value θ used for determination of convergence of the analysis process. Although the convergence determination threshold value θ varies depending on the convergence determination criterion set by the user, in the present embodiment, since it uses the change width of the sound source separation analysis, it becomes a positive real number.

  Next, in steps 200 to 600, the analysis unit 2 performs a calculation for optimizing the variables defined in the storage unit 3. Various optimization methods (for example, Non-Patent Document 4 “CM Bishop,“ Pattern Recognition and Machine Learning Up / Down ”, Springer Japan, 2007.”) are used to optimize variables defined in the storage unit 3. In this embodiment, sound source separation and sound source localization are simultaneously optimized based on the variational Bayes method.

  First, in step 200, the initial value generation unit 21 executes an initial value generation processing routine shown in FIG. 6 to set an initial value. Then, in step 300, the sound source time frequency mask variable calculation unit 22 executes the sound source time frequency mask variable calculation processing routine shown in FIG. 7, and in step 400, the sound limit variable calculation unit 23 performs the sound limit level shown in FIG. The variable calculation processing routine is executed, and in step 500, the statistic calculation unit 24 executes the statistic calculation processing routine shown in FIG. 9 to calculate each value in order, and repeatedly performs the calculation until the convergence condition is satisfied. To optimize. In calculations based on the variational Bayes method, it is guaranteed that the calculation results converge.

  Hereinafter, each process is explained in full detail. The order of execution of the sound source time frequency mask variable calculation routine, the sound limit variable calculation routine, and the statistic calculation routine may be arbitrary.

  First, in the initial value generation processing routine (FIG. 6), in step 202, the total number of time frames T, the total number of frequency bins F, the number of masks K, and the number of direction classes D received by the preset value receiving unit 13 in step 104 above. , The number of microphones M, and the regularization constant ε are stored in the constant storage unit 31.

Next, in step 204, the observation signal x _tf obtained as a result of the calculation of the time frequency domain observation conversion unit 12 in step 102 is stored in the observation signal storage unit 32.

  Next, in step 206, the convergence determination threshold θ received by the preset value reception unit 13 in step 104 is stored in the convergence determination threshold storage unit 37.

Next, in step 208, the initial value of the statistic is set. First, a part (β _k ⁰ , κ _d ⁰ , _atf ⁰ , v _fd ⁰ ) of the statistic initial value received by the preset value receiving unit 13 in step 104 is stored in the statistic initial value storage unit 36. . Further, using the anechoic steering vector q _fd received by the preset value receiving unit 13 in step 104 and the observation signal x _tf calculated in step 102, as shown in equations (2) and (3): _Next , G _fd ⁰ and b _tf ⁰ which are a part of the statistical initial value are calculated. H represents Hermitian transposition, and I _M represents an M-dimensional unit matrix.

A part (G _fd ⁰ , b _tf ⁰ ) of the statistic initial value calculated by the equations (2) and (3) is stored in the statistic initial value storage unit 36.

Next, in step 210, the initial value ξ _tfk ⁰ of the sound source time frequency mask variable is calculated by the equation (4), and the calculated ξ _tfk ⁰ is stored in the sound source time frequency mask variable storage unit 33 as ξ _tfk . Z is a normalization term for making the sum related to k 1.

Next, in step 212, the initial value η _kd ⁰ of the sound source localization variable is calculated by the equations (5) and (6), and the calculated η _kd ⁰ is stored in the sound source localization variable storage unit 34 as η _kd .

  As described above, when the setting of the initial value is completed, the initial value generation processing routine is ended, and the process returns to the sound source separation localization processing routine.

  Next, in the sound source time frequency mask variable calculation processing routine (FIG. 7), necessary information is loaded from the storage unit 3 in step 302, and then a variable t corresponding to the time frame is set to 1 in step 304. Then, in step 306, the variable f corresponding to the frequency bin is set to 1.

Next, in step 308, the sound source time frequency mask variable ξ _tfk is calculated for k = 1 _,. Note that Ψ is a digamma function.

The sound source time frequency mask variable ξ _tfk is a variable to be calculated in order to separate the observation signals. t = 1,..., T, f = 1,..., F, k = 1,. ξ _tfk represents the probability that the observed signal in time frame t and frequency bin f is a signal from the kth sound source (mask). By separating the observation signal into K sound sources according to the sound source time frequency mask variable ξ _tfk , the separated sound for each sound source can be restored. This sound source time frequency mask variable ξ _tfk is assumed to be generated from a multinomial distribution in the present embodiment, and the parameters of the multinomial distribution are determined by the Dirichlet distribution parameterized by the statistic β. .

The point of Expression (7) is that the sound source localization variable η _kd is necessary as in the fourth term on the right side. This indicates that sound source separation is improved by using information on sound source localization.

Next, in step 310, the sound source time frequency mask variable ξ _tfk calculated for k = 1,..., K in step 308 is normalized by the equation (8). Since ξ _tfk is a probability, _{normalization} is performed so that the sum for all k is always 1 for each t and f.

  Next, in step 312, f is incremented by 1, and in the next step 314, it is determined whether f exceeds the total frequency bin number F. If f has not yet reached F, Returning to 308, the processing of steps 308-312 is repeated.

  On the other hand, if f exceeds F, the process proceeds to step 316, t is incremented by 1, and in the next step 318, it is determined whether or not t exceeds the total number of time frames T. If T has not been reached, the process returns to step 306 and the processes of steps 306 to 316 are repeated.

On the other hand, if t exceeds T, the _{process proceeds} to step 320, where the calculated sound source time frequency mask variable ξ _tfk is stored and updated in the sound source time frequency mask variable storage unit 33 to calculate sound source time frequency mask variable. The processing routine is ended, and the process returns to the sound source separation localization processing routine.

  Next, in the sound source localization variable calculation processing routine (FIG. 8), necessary information is loaded from the storage unit 3 in step 402, and then a variable k corresponding to each mask is set to 1 in step 404. .

Next, in step 406, the sound source localization variable η _kd is calculated for d = 1 _,.

The sound source localization variable η _kd is a variable calculated to estimate the localization of a plurality of sound sources, that is, the direction of each sound source relative to the microphone array. k = 1,..., K, d = 1,. η _kd represents the probability that the direction of the sound source k is in the d-th discretized direction. The direction of each sound source can be estimated according to this variable. This variable is assumed to be generated from a multinomial distribution in the present embodiment, and the parameters of the multinomial distribution are determined by a Dirichlet distribution parameterized by a statistic κ.

The point of equation (9) is that the sound source time frequency mask variable ξ _tfk is required as in the third term on the right side. This indicates that sound source localization is improved using information on sound source separation.

Next, in step 408, the sound source localization variable η _kd calculated for d = 1,..., D in step 406 is normalized by equation (10). Since η _kd is a probability, it is normalized so that the sum for all d is always 1 for each k.

  Next, in step 410, k is incremented by 1, and in the next step 412, it is determined whether or not k has exceeded the set maximum number of masks K. If k has not yet reached K, Returning to step 406, the processing of steps 406 to 410 is repeated.

On the other hand, if k exceeds K, the process proceeds to step 414, where the calculated sound source localization variable η _kd is stored and updated in the sound source localization variable storage unit 34, and the sound source localization variable calculation processing routine is terminated. Return to the sound source separation localization processing routine.

  Next, in the statistic calculation processing routine (FIG. 9), each statistic used in the sound source time frequency mask variable calculation processing routine and the sound source localization variable calculation processing routine is calculated. First, in step 502, necessary information is loaded from the storage unit 3.

Next, in step 504, β _tk , which is a parameter of the Dirichlet distribution used in the mathematical model of the sound source time frequency mask variable, is set to t = 1,..., T and k = 1,. Calculated according to equation (11). Intuitively, β _tk is a parameter representing the strength of the possibility that the observation signal on each frequency bin f is explained by the signal from the sound source k in the time frame t, and is a value larger than zero. .

Next, in step 506, κ _d , which is a parameter of the Dirichlet distribution used in the mathematical model of the sound source localization variable, is calculated for d = 1,. Intuitively, κ _d is a parameter indicating the strength of the possibility that the sound source k exists in the direction d, and is a value larger than zero.

Next, in step 508, a _tfk that is a parameter of the gamma distribution used in the mathematical model of the time-frequency domain observation signal is _changed to t = 1,..., T, f = 1 _,. k = 1,..., K is calculated by the equation (13).

Next, in step 510, b _tfk , which is a parameter of the gamma distribution used in the mathematical model of the time-frequency domain observation signal, is _changed to t = 1,..., T, f = 1 _,. For k = 1,..., K, calculation is performed according to equation (14).

Next, in step 512, v _fd that is a parameter of the complex Wishart distribution used in the mathematical model of the time-frequency domain observation signal is set to f = 1,..., F and d = 1,. , Calculated by the equation (15).

Next, in step 514, G _fd that is a parameter of the complex Wishart distribution used in the mathematical model of the time-frequency domain observation signal is set to f = 1,..., F and d = 1,. , Calculated by the equation (16). G _fd is a matrix that includes acoustic information of an actual anechoic environment, incorporating information of an anechoic steering vector.

  Next, in step 516, each statistic calculated in steps 504 to 514 is stored and updated in the statistic storage unit 35, the statistic calculation processing routine is terminated, and the process returns to the sound source separation localization processing routine. .

  In the sound source separation localization processing routine, next, the process proceeds to step 600, where the convergence condition determination unit 25 monitors each value stored in the storage unit 3 to determine whether or not the calculation convergence condition is satisfied. . The convergence condition may be arbitrarily set such as the number of iterations of the iterative calculation. However, in this embodiment in which the analysis calculation based on the variational Bayes method is performed, for example, the convergence condition as shown in Expression (17) can be used. . Here, ξ ′ represents the value of ξ before update. It is known that the calculation of each value always converges when this convergence condition is used.

In Expression (17), the change width of the sound source time frequency mask variable ξ _thk is used as the convergence condition, but the change width of the sound source localization variable η _kd may be used as the convergence condition.

  If the convergence condition is not satisfied, the process returns to step 300 and the calculation of each value is repeated. On the other hand, if the convergence condition is satisfied, the process proceeds to step 700.

In step 700, it separated sound extraction unit 41, the observation signals x _tf and by utilizing the source time-frequency mask variables xi] _TFK, to calculate the separated sound signals in the time frequency region corresponding to each sound source. First, the number N of extracted sound sources that satisfies N <= K is determined with respect to the number of K sound source masks, that is, the virtual maximum number of sound sources. This may be specified in advance, or may be determined automatically or manually based on some decision rule using information in the storage unit 3 after the analysis is completed.

At the same time, the order of the indexes of the sound source time frequency mask variable ξ _tfk is changed. Specifically, the index to be replaced is k which is the last index of ξ _tfk . As shown below, the index order is switched by calculating the sum of all mask variables for each sound source index k, and switching the order of the sum to the largest.

  Intuitionally rearranging the above indexes is equivalent to replacing the sound source k in order of increasing expected sound volume. The index of the replaced sound source is represented by n.

Then, the separated sound signal y _tf ⁿ in the time frequency domain at the time frame t and the frequency bin f of the sound source with the nth loudest volume is calculated using the equation (18) under the replaced index.

The meaning of equation (18) is that the ratio of the sound of the sound source n at each (t, f) among the N sound sources is calculated by the fraction of the time frequency mask variable ξ _tfk of the first term on the right side, and this ratio Then, x _tf which is the time frequency domain representation of the mixed sound is distributed. When the calculation of equation (18) is completed for all t = 1,..., T, f = 1,. The calculation result of the separated sound signal of the region is passed to the speech waveform restoration unit 42, and the replaced sound source index information n is passed to the sound source direction extraction unit 43.

Next, in step 702, the speech waveform restoration unit 42 transforms the separated sound signal y _tf ⁿ received from the separated sound extraction unit 41 in step 700 to restore a normal speech waveform. Specifically, the inverse Fourier transform, which is the inverse transform of the time frequency domain observation transform unit 12, is performed for each sound source n.

Next, in step 704, the sound source direction extraction unit 43 uses the replaced sound source index n and the sound source localization variable η _kd received from the separated sound extraction unit 41 in step 700 to determine the directions of the N sound sources. calculate. Specifically, the calculation of Expression (19) may be performed for each n. This makes it possible to determine the index d _n in a direction that exists in the source n.

  Next, in step 706, the final output unit 44 uses the information of the storage unit 3, the separated sound extraction unit 41, the speech waveform restoration unit 42, and the sound source direction extraction unit 43 to output the analysis result in the form desired by the user. To output the sound source separation localization processing routine.

  Note that either of the processes in steps 702 and 704 may be performed first.

  As described above, according to the sound source separation localization apparatus according to the present embodiment, when observing a mixed sound of sounds emitted from each of a plurality of sound sources, sound source separation to each sound source and sound source direction localization are performed. Is solved by one statistical framework at the same time, avoiding the situation of “failure of one problem and failure of the other problem” as in the existing method, and stable against both problems. High performance.

  Further, since sound source separation and sound source localization are problems of mutual dependence, it is possible to obtain higher accuracy than solving each problem individually by solving both problems at the same time.

  Furthermore, the steering vector can be re-measured in an actual unknown anechoic environment by using the steering vector of the anechoic environment measured in advance for the microphone array that observes the mixed sound for simultaneous separation and localization of multiple sound sources. In addition, sound source separation and localization can be performed in accordance with various environments.

<Experimental example>
Next, the result of the experiment in the sound source separation localization apparatus according to this embodiment will be described.

  FIG. 10 shows the experimental setup. In this experiment, it is assumed that the number of sound sources N = 2 or 3 is known, and the performance evaluation of the separation and localization of each sound source is performed with the pattern of the number of microphones M = 2, 4, and 8. In the experiment, the number of discretized directions is D = 72, that is, the directions are divided every 5 degrees.

FIG. 11 is a graph showing the performance of sound source localization. FIG. 4A shows the case where the number of sound sources is 2, and FIG. In the figure, RT ₆₀ represents the reverberation time of the mixed sound observation environment. According to this, it can be seen that the localization performance decreases as the reverberation becomes longer. However, when the number of microphones is large, for example, when M = 8, the localization error is almost zero. That is, particularly when the number of microphones is large, the sound source separation and localization apparatus according to the present embodiment can achieve localization of a sound source with high accuracy.

  FIG. 12 is a graph showing the performance of sound source separation. Here, comparison with the conventional method was performed. The comparison target is changed depending on the combination of the number of microphones and the number of sound sources. First, when M> = N, that is, when the number of microphones is larger than that of the sound source, the IVA method of Non-Patent Document 3 is used. On the other hand, when M <N, that is, when the number of microphones is smaller than the sound source, the method (TF-perm) of Non-Patent Document 1 is used.

In FIG. 12, (a), (c), and (e) are when the number of sound sources is 2, and (b), (d), and (f) are when the number of sound sources is 3. Also, (a) and (b) are RT ₆₀ = 20 msec, (c) and (d) are RT ₆₀ = 400 msec, and (e) and (f) are RT ₆₀ = 600 msec. That is, the experimental results in an environment where the reverberation time is long each time the line moves from the top to the bottom are shown. In (b), (d), and (f), when the number of microphones is 2, the results of the method of Non-Patent Document 1 are listed.

  As is clear from the figure, the sound source separation localization apparatus according to the present embodiment can achieve better separation accuracy than the conventional method in almost all experimental environments and the number of microphones. This means that better sound source separation can be achieved by solving the problem at the same time as sound source localization, which shows the effectiveness of the present invention.

  Note that the present invention is not limited to the above-described embodiment, and various modifications and applications are possible without departing from the gist of the present invention.

  For example, the above-described sound source separation localization apparatus has a computer system inside, but the “computer system” includes a homepage providing environment (or display environment) if a WWW system is used. .

  In the present specification, the embodiment has been described in which the program is installed in advance. However, the program can be provided by being stored in a computer-readable recording medium.

DESCRIPTION OF SYMBOLS 1 Reception part 2 Analysis part 3 Storage part 4 Output part 10 Sound source separation and localization apparatus 11 Mixed sound observation part 12 Time frequency domain observation conversion part 13 Preset value reception part 21 Initial value generation part 22 Sound source time frequency mask variable calculation part 23 Sound source Localization variable calculation unit 23 Sound source separation variable calculation unit 24 Statistics calculation unit 25 Convergence condition determination unit 41 Separated sound extraction unit 42 Speech waveform restoration unit 43 Sound source direction extraction unit 44 Final output unit

Claims (5)

Receiving means for receiving a mixed sound signal obtained by observing a mixed sound of each sound emitted from each of a plurality of sound sources by a plurality of observation means arranged at different positions;
Sound source separation for separating the mixed sound signal received by the reception unit so as to correspond to each of the plurality of sound sources, and sound source localization for estimating a direction in which each of the plurality of sound sources exists with reference to the observation unit And analyzing means for analyzing by simultaneous optimization that iteratively processes using variables mutually dependent on the sound source separation and the sound source localization;
Output means for outputting the result of sound source separation and sound source localization analyzed by the analysis means ,
The accepting means converts the mixed sound signal into an observation signal _xtf in a time frequency domain composed of elements for each time frame t and frequency bin f, and _delivers it to the analyzing means,
The analysis means includes
A mask variable ξ _tfk representing a probability that each element of the observation signal x _tf is a signal corresponding to a kth mask of a plurality of masks that assign each element to each of a plurality of virtually set sound sources , A sound source time frequency mask variable calculating means for calculating each of a plurality of masks;
A sound source localization variable η _kd representing a probability that a sound source corresponding to the k-th mask exists in the d-th direction among a plurality of directions divided on the basis of the observation means is calculated for each of the plurality of directions. A sound source localization variable calculation means;
Statistic calculation means for calculating a statistic used for calculation of the mask variable ξ _tfk and the sound source localization variable η _kd ;
A convergence condition determining means for repeating the calculation of the sound source time frequency mask variable calculating means, the sound source localization variable calculating means, and the statistic calculating means until a predetermined convergence condition is satisfied,
The sound source localization variable eta _kd used in the calculation of the mask variables xi] _TFK, using said mask variables xi] _TFK in the calculation of the sound source localization variable eta _kd
Sound source separation localization apparatus. The analyzing means uses the steering vectors of the plurality of observing means measured in anechoic environment, the sound source separation and claim 1 Symbol placement of sound source separation localization apparatus for analyzing the sound source localization. A sound source separation and localization method in a sound source separation and localization apparatus including a reception unit, a sound source time frequency mask variable calculation unit, a sound source localization variable calculation unit, a statistic calculation unit, an analysis unit including a convergence condition determination unit, and an output unit. And
The reception means receives a mixed sound signal obtained by observing a mixed sound of each sound emitted from each of a plurality of sound sources by a plurality of observation means arranged at different positions;
Sound source separation in which the analysis means separates the mixed sound signal received by the reception means so as to correspond to each of the plurality of sound sources, and a direction in which each of the plurality of sound sources exists based on the observation means The sound source localization for estimating the sound source is analyzed by simultaneous optimization using iterative processing using variables mutually dependent on the sound source separation and the sound source localization,
In the sound source separation localization method in which the output means outputs the result of sound source separation and sound source localization analyzed by the analysis means ,
The reception unit converts the mixed sound signal into an observation signal x _tf in a time frequency domain composed of elements for each of the time frame t and the frequency bin f, and passes it to the analysis unit.
The sound source time frequency mask variable calculation means is a signal in which each element of the observation signal _xtf corresponds to a kth mask of a plurality of masks that assign each element to each of a plurality of virtually set sound sources. A mask variable ξ _tfk representing the probability is calculated for each of the plurality of masks;
The sound source localization variable calculation means, the sound source corresponding to the k-th mask, the sound source localization variable eta _kd representing the probability that exist d th direction of the divided plurality of directions said observation means as a reference, the Calculate for each of multiple directions,
The statistic calculating means calculates a statistic used for calculating the mask variable ξ _tfk and the sound source localization variable η _kd ;
The convergence condition determining means repeats the calculation of the sound source time frequency mask variable calculating means, the sound source localization variable calculating means, and the statistic calculating means until a predetermined convergence condition is satisfied,
The sound source localization variable eta _kd used in the calculation of the mask variables xi] _TFK, using said mask variables xi] _TFK in the calculation of the sound source localization variable eta _kd
Sound source separation localization method. 4. The sound source separation localization method according to claim 3 , wherein the analysis unit analyzes the sound source separation and the sound source localization using steering vectors of the plurality of observation units measured in an anechoic environment. The computer, the sound source separation localization program for causing to function as each means constituting the sound source separation localization apparatus according to claim 1 or claim 2, wherein.