JP7292646B2 - Sound source separation device, sound source separation method, and program - Google Patents

Description

The present invention relates to a sound source separation device, a sound source separation method, and a program.

Technologies for extracting a specific sound source from a plurality of sound sources have been developed. For example, there is a method using beamforming as a sound source separation method using position information. In beamforming, sound source separation based on direction information can be performed by using arrival time differences and phase differences of signals (see, for example, Patent Document 1).

JP 2010-152107 A

However, with the conventional technology, it is difficult to extract a desired sound source when multiple sound sources exist in the same direction.

The present invention has been made in view of the above problems, and provides a sound source separation device, a sound source separation method, and a sound source separation method capable of extracting a desired sound source even when a plurality of sound sources exist in the same direction. The purpose is to provide a program.

(1) In order to achieve the above object, a sound source separation device according to an aspect of the present invention includes a plurality of microphone arrays for picking up acoustic signals, and each sound picked up by at least two of the microphone arrays. each of the picked sounds picked up by at least two of the microphone arrays, if the signal includes a first acoustic signal of a sound source of interest and a second acoustic signal of another sound source in the same direction as the sound source of interest; an extraction unit that extracts a common component included in the signals and extracts the first acoustic signal from the picked-up acoustic signal.

(2) In the sound source separation device according to an aspect of the present invention, the extraction unit may extract the common component using a latent Dirichlet allocation method.

(3) The sound source separation device according to an aspect of the present invention further includes a classification unit that classifies topics of sounds included in the collected sound signal, wherein the extraction unit causes the classification unit to: The topics classified for each of the microphone arrays are compared, and if the topic is the same in the collected sound signals picked up by each of the plurality of microphone arrays as a result of the comparison, the same topic is the sound source of interest. , the acoustic signal corresponding to the same topic may be extracted from the picked-up acoustic signal as the first acoustic signal.

(4) Further, in the sound source separation device according to the aspect of the present invention, the classification unit converts the collected sound signals collected by the respective microphone arrays into frequency spectra, The spectrum may be segmented by dividing it into M (M is an integer equal to or greater than 2) sections in the time frame, and the frequency spectrum for each time frame included in each segment may be classified according to the topic. good.

(5) In the sound source separation device according to an aspect of the present invention, the extraction unit estimates the distribution of the topic for each time interval and the distribution of the quantized spectrum obtained by quantizing the frequency spectrum for each topic. Then, when the posterior probabilities of the distribution of the topic and the distribution of the quantized spectrum are respectively larger than a threshold value for discriminating the active state, the state is defined as the active state, and the distribution of the topic for each of the segments at the same time is compared. and the common component may be extracted by extracting the topics that are active in at least two of the microphone arrays.

(6) The sound source separation device according to an aspect of the present invention further includes a control unit that controls the microphone array to form a beam in the direction of the sound source of interest, wherein the plurality of microphone arrays may pick up the collected sound signal including the first sound signal of the sound source of interest under the control of the control unit.

(7) In the sound source separation device according to an aspect of the present invention, a sound source localization unit that performs sound source localization on the sound signals picked up by each of the microphone arrays, and the sound picked up by each of the microphone arrays. a sound source separation unit that separates a separated signal including the first sound signal from the collected sound signal based on the localization result of the sound source localization, wherein the extraction unit includes at least two of the microphone arrays. A common component included in each of the separated signals separated from each of the collected sound signals may be extracted to extract the first sound signal from the collected sound signals.

(8) In order to achieve the above object, a sound source separation method according to an aspect of the present invention includes a plurality of microphone arrays picking up acoustic signals, and an extracting unit picking up the sounds by at least two of the microphone arrays. When each sound pickup sound signal includes a first sound signal of a sound source of interest and a second sound signal of another sound source in the same direction as the sound source of interest, each picked up by the at least two microphone arrays extracts a common component included in the collected sound signals, and extracts the first sound signal from the collected sound signals.

(9) To achieve the above object, a program according to an aspect of the present invention causes a computer to pick up sound signals with a plurality of microphone arrays, and picks up sound picked up by each of at least two of the microphone arrays. each of the picked sounds picked up by at least two of the microphone arrays, if the signal includes a first acoustic signal of a sound source of interest and a second acoustic signal of another sound source in the same direction as the sound source of interest; A common component included in the signals is extracted, and the first acoustic signal is extracted from the picked-up acoustic signal.

According to the above (1) to (9), since the common component included in the collected sound signal is extracted, even if a plurality of sound sources exist in the same direction, the desired sound source can be taken out.
Further, according to the above-mentioned (2), since the common component is extracted by the latent Dirichlet allocation method, a desired sound source can be extracted with high accuracy.
In addition, according to the above-mentioned (3), since the collected sound signals are classified into sound topics and the matching topics are extracted as common components, the desired sound source can be extracted with high accuracy. can.
Further, according to (4) described above, the collected sound signal is divided into segments, each segment is classified into sound topics, and matching topics are extracted as common components. A desired sound source can be extracted.
In addition, according to (5) above, topic distributions for each segment at the same time are compared, topics that are active in at least two microphone arrays are extracted, and common components are extracted. A desired sound source can be taken out well.
Further, according to (6) above, since the common component included in the collected sound signals separated by beamforming is extracted, even if a plurality of sound sources exist in the same direction, , the desired sound source can be extracted.
In addition, according to the above-described (7), the separated signals are separated from the collected sound signals by the sound source localization processing and the sound source separation processing, and the common components contained in the separated signals are extracted. A desired sound source can be extracted even when a plurality of sound sources exist.

FIG. 4 is a diagram showing an example of the position of a separation target sound source and an example of the arrangement of a microphone array according to the embodiment; 1 is a block diagram showing a configuration example of a sound source separation device according to a first embodiment; FIG. 4 is a flow chart showing a processing procedure performed by the sound source separation device according to the first embodiment; FIG. 4 is a diagram for explaining quantization spectrum conversion of a frequency spectrum; FIG. 10 is a flowchart showing an example of a k-means processing procedure; FIG. 11 is a flow chart showing an example of a process of generating a set of LDA quantized spectra; FIG. FIG. 3 is a diagram representing a graphical model of LDA; It is an example of a variational Bayesian inference algorithm for a topic model according to the embodiment. FIG. 4 is a diagram showing an example of spectral estimation of a sound source of interest; FIG. 10 is a diagram showing an example of extracted sounds when the number of clusters K=600, the segment time interval d=4 seconds, and the number of topics L=5. FIG. 10 is a diagram showing changes in separation performance with the number of topics L when the number of clusters K=600 and the time interval d=4 seconds; FIG. 10 is a diagram showing changes in separation performance due to the number of clusters K=100, 300, 600 and the difference in segment length. FIG. 10 is a diagram showing evaluation results comparing the separation performance when the number of clusters K=600, the time interval d=4 seconds, and the number of topics L=5, with and without removing silent components and unique components. FIG. 11 is a block diagram showing a configuration example of a sound source separation device according to a second embodiment; FIG. 4 is a diagram for explaining silent intervals and speech intervals;

BEST MODE FOR CARRYING OUT THE INVENTION Hereinafter, embodiments of the present invention will be described with reference to the drawings.
First, an outline of the embodiment will be described. FIG. 1 is a diagram showing a position example of a separation target sound source and an arrangement example of a microphone array according to the embodiment.
In the example shown in FIG. 1, among the sound sources S ₀ to S ₃ of the four speakers, the sound source S ₀ is the target sound source. References MA ₁ to MA ₃ are microphone arrays. The separated sound obtained by separating the acoustic signal picked up by the microphone array _MA1 includes the sound sources _S0 and _S1 . The separated sound obtained by separating the acoustic signal picked up by the microphone array _MA2 includes the sound sources _S0 and _S2 . The separated sound obtained by separating the acoustic signal picked up by the microphone array _MA3 includes the sound sources _S0 and _S3 .

As shown in FIG. 1, the sound source of interest _S0 may be commonly included in separated sounds picked up and separated by a plurality of microphone arrays. Therefore, in each of the embodiments described below, a desired sound source is separated by extracting a common component commonly included in separated sounds collected and separated by a plurality of microphone arrays.

<First Embodiment>
In the first embodiment, an example will be described in which the direction of the sound source is known and the acoustic signal in the direction of the sound source is picked up and separated by the beamforming method.

[Configuration example of sound source separation device]
First, a configuration example of the sound source separation device 1 of this embodiment will be described.
FIG. 2 is a block diagram showing a configuration example of the sound source separation device 1 according to this embodiment. As shown in FIG. 2 , the sound source separation device 1 includes a sound pickup section 2 and a processing section 3 .
The sound pickup unit 2 includes a first microphone array 2-1, a second microphone array 2-2, and a third microphone array 2-3. In the configuration shown in FIG. 2, an example in which the sound pickup unit 2 includes three microphone arrays will be described, but the number of microphone arrays may be two or more.
The processing unit 3 includes a beamforming control unit 30 , an acquisition unit 31 , a transformation unit 34 , a classification unit 35 , a removal unit 36 , an extraction unit 37 , an inverse transformation unit 38 and an output unit 39 .

[Operation and function of the sound source separation device]
Next, an example of the operation and function of each part of the sound source separation device 1 will be described.
Each of the first microphone array 2-1, the second microphone array 2-2, and the third microphone array 2-3 forms a beam in a known sound source direction according to the beamforming control section 30 of the processing section 3. FIG. Each of the first microphone array 2-1, the second microphone array 2-2, and the third microphone array 2-3 is a microphone array having P (P is an integer equal to or greater than 2) microphones. Each of the first microphone array 2-1, the second microphone array 2-2, and the third microphone array 2-3 outputs picked-up acoustic signals to the processing unit 3. FIG. The acoustic signal output by each microphone array contains identification information for identifying the microphone array. An acoustic signal picked up by each microphone array corresponds to an acoustic signal picked up by, for example, one directional microphone with one beam formed in a known sound source direction by the beamforming method. As shown in FIG. 1, the collected sound signals collected by the microphone arrays include the first sound signal of the sound source of interest and the second sound signal of another sound source in the same direction as the sound source of interest. Suppose there is a case.

A beamforming control unit 30 controls each of the first microphone array 2-1, the second microphone array 2-2, and the third microphone array 2-3 so as to form a beam in a known sound source direction by a beamforming method. do.

The acquisition unit 31 acquires acoustic signals (collected acoustic signals) output by the first microphone array 2-1, the second microphone array 2-2, and the third microphone array 2-3. The acquisition unit 31 outputs the acquired acoustic signal for each microphone array to the conversion unit 34 .

The conversion unit 34 acquires the acoustic signal for each microphone array output by the acquisition unit 31 . The transformation unit 34 performs a short-time Fourier transform (STFT) on the acoustic signal for each microphone array to convert it into an amplitude spectrum (hereinafter also referred to as a frequency spectrum) in the time-frequency domain. The conversion unit 34 outputs the converted frequency spectrum for each microphone array to the classification unit 35 .

The classifying unit 35 acquires the frequency spectrum for each microphone array output by the transforming unit 34 . The classification unit 35 segments the frequency spectrum of each microphone array by dividing it into M (M is an integer equal to or greater than 2) sections in a time frame. The classification unit 35 regards the amplitude spectrum for each time frame as one vector, sets the frequency spectrum for each time frame included in each segment as a quantized spectrum, and counts the number of quantized spectra. Further, the classifying unit 35 classifies the quantized spectrum included in each segment for each microphone array by, for example, the k-means clustering method. Note that the classification method will be described later. The classification unit 35 outputs count information indicating the counting result and classification information indicating the classification result to the removing unit 36 for each microphone array.

The removal unit 36 acquires the count information and the classification information output by the classification unit 35 . The removal unit 36 removes noise components from the quantized spectrum. Here, since human speech includes many silent components, there is a high possibility that quantized spectra included in many time intervals are silent. Therefore, the removing unit 36 removes, for example, classification units appearing in 70% or more of all segments and quantized spectra appearing only in less than 3 segments. The removing unit 36 outputs the count information and the classification information after removing the noise component to the extracting unit 37 .

The extraction unit 37 acquires the count information and the classification information after removing the noise component output by the removal unit 36 . Using the obtained count information and classification information, the extraction unit 37 uses, for example, Latent Dirichlet Allocation (LDA) to extract the frequency spectrum for each microphone array and for each segment according to the speaker and the utterance content. Estimate topic distributions as topics based. The extraction unit 37 extracts an estimated time-frequency spectrogram of the sound source of interest by performing spectrum extraction based on topic temporal identity in a plurality of microphone arrays. Specifically, the extraction unit 37 selects the estimated topic for each time interval and extracts only the frequency spectrum present in the topic distribution of the estimated topic. Note that the estimation method will be described later. The extraction unit 37 outputs the extracted spectrum to the inverse transform unit 38 .

The inverse transforming unit 38 acquires the spectrum output by the extracting unit 37 . The inverse transform unit 38 restores the estimated signal of the sound source of interest by performing an inverse short-time Fourier transform (ISTFT) on the acquired spectrum. The inverse transform unit 38 outputs the restored acoustic signal of the sound source of interest to the output unit 39 .

The output unit 39 is, for example, a speaker. The output unit 39 reproduces the acoustic signal output by the inverse transform unit 38 .

[Processing of the sound source separation device 1]
Next, an example of processing procedures performed by the sound source separation device 1 will be described.
FIG. 3 is a flowchart showing a processing procedure performed by the sound source separation device 1 according to this embodiment.

(Step S1) The beamforming control unit 30 controls each microphone array of the sound pickup unit 2 to form a beam in a known sound source direction.

(Step S2) The sound pickup unit 2 picks up an acoustic signal with the formed beam. Thereby, the sound pickup unit 2 picks up an acoustic signal corresponding to the sound source in the direction of the sound source. Note that the collected sound signals are separated sounds, and may include sound signals from a plurality of sound sources in the same sound source direction, as shown in FIG.

(Step S3) The transform unit 34 performs short-time Fourier transform on the collected acoustic signal of each microphone array to transform it into a frequency spectrum.

(Step S4) The classification unit 35 segments the frequency spectrum of each microphone array by dividing it into M sections in the time frame. Subsequently, the classifier 35 counts the number of quantized spectra included in each segment. Subsequently, the classifying unit 35 classifies the quantized spectrum included in each segment for each microphone array by a clustering method such as the k-means method, for example.

(Step S5) The removal unit 36 removes noise components from the quantized spectrum.

(Step S6) The extraction unit 37 uses the acquired count information and classification information, for example, using the latent Dirichlet allocation method, to extract the frequency spectrum for each microphone array and for each segment as a topic based on the speaker and the utterance content. , to estimate the topic distribution.

(Step S7) The extraction unit 37 extracts the estimated time-frequency spectrogram of the sound source of interest by performing spectrum extraction based on the temporal identity of the topic in a plurality of microphone arrays.

(Step S8) The inverse transform unit 38 restores the estimated signal of the sound source of interest by performing an inverse short-time Fourier transform on the acquired spectrum. Subsequently, the output unit 39 reproduces the acoustic signal output by the inverse transform unit 38 .

[Method of extracting sound source of interest using LDA]
Next, a method of extracting a sound source of interest using LDA will be described.
In the embodiment, the sound source of interest is extracted by extracting only components common to all the separated sounds in the direction of the sound source of interest obtained by beamforming with a plurality of microphone arrays.

In the embodiment, a frequency spectrum for each time frame is treated as one quantized spectrum, and a set of frequency spectra for each time interval is treated as a segment. In this way, the frequency spectrum can be classified into groups called topics based on speakers and speech content.
If different speakers' spectra are assigned to different topics, the sound source of interest does not exist if the topics of separated sounds are different in a certain time interval. Also, if the same topic is assigned to all the separated sounds, that topic is the sound source of interest.
In the embodiment, the topic of the sound source of interest is thus estimated from the temporal identity of the topic, and the common component is extracted by extracting only the frequency spectrum of the topic.

(Preprocessing)
In an embodiment, in order to apply LDA to an acoustic signal, preprocessing is performed to quantize and spectralize the sound.
In the embodiment, the amplitude spectrum for each time frame is regarded as one quantized spectral vector. For example, using the k-means clustering method, quantized spectral vectors having similar components are grouped into several groups Divide.

First, the quantization spectrum conversion of the frequency spectrum by the k-means method will be explained.
Applying the short-time Fourier transform to the acoustic signal X _i (t) yields the amplitude spectrum Y _i (ω,t)εR ^F×T in the time-frequency domain, where R is the set of all positive real numbers. where F represents the number of frequency bins and T represents the number of time frames. As shown in FIG. 4, the amplitude spectrum y _i (t) for each time frame is regarded as one vector, and quantized spectrum conversion is performed. Furthermore, in the embodiment, the k-means method classifies y _i (t) into K classes kε{1, . . . , K}. FIG. 4 is a diagram for explaining quantization spectrum conversion of a frequency spectrum. In FIG. 4, the horizontal axis is the time frame and the vertical axis is the frequency.

Here, an example of the k-means processing procedure will be described.
FIG. 5 is a flowchart showing an example of the k-means processing procedure. Note that i is the number of the microphone array, and K is the number of clusters in the quantized spectrum. In an embodiment, a class k is assigned to each microphone array i and each time frame t based on similarity of frequency vector components.

(Step S11) The classification unit 35 randomly distributes y _i (t) to the cluster k.

(Step S12) The classification unit 35 calculates the cluster center V _k of x _it belonging to each class k.
(Step S13) The classification unit 35 redistributes _yit to the nearest cluster center _Vk .

(Step S14) The classification unit 35 determines whether or not the change has converged, and whether or not the number of times given in advance has been completed. When the classification unit 35 determines that the change has converged or the predetermined number of times has been completed (step S14; YES), the process ends. When the classification unit 35 determines that the change has not converged and the predetermined number of times has not been completed (step S14; NO), the process returns to step S12.

Next, the topic model will be explained. To extract the common component of each isolated sound, embodiments fit a topic model to the acoustic signal.
A topic model is conceived as a tool for discovering some kind of semantic information from a large amount of document data. The topic model was devised as an analysis method for document data, but due to the high versatility of its structure model, it is also used in image processing, social network analysis, and acoustic signal processing. In the field of acoustic signal processing, for example, a speaker estimation method using a topic model for signal direction of arrival (DOA) information has been devised.

In the topic model, topic distribution m=(θ _m1 , . . . , θ _mL ) is obtained for each segment m. where θ _ml =p(l|θ _m ) represents the probability that topic l is assigned to the quantized spectrum of segment m and satisfies θ _ml ≧0 and Σ _l θ _ml =1. Also, a quantized spectral distribution φ _l =(φ _l1 , . . . ,φ _lK ) is obtained for each topic l. φ _lk =p(k|φ ₁ ) is a probability representing the likelihood of appearance of value k in topic l and satisfies φ _lk ≧0 and Σ _k φ _lk =1.

Also, in the topic model, the order of quantized spectra is not taken into account, and segments are represented by how many times each quantized spectrum appears. Therefore, the separated signal of each microphone array i is divided into M sections for segmentation.
The classification unit 35 counts the number of quantized spectra k included in each of the segments m _i thus obtained. Through this operation, the classification unit 35 creates a frequency matrix WεR ^3M×K (R is a set of all positive real numbers) for LDA. The reason why the number of rows of the frequency matrix W is 3M is that the LDA is calculated for the entire segments of the three microphone arrays.

(Prediction of hot topics by LDA)
As described above, after preprocessing, the removal unit 36 removes noise components.
In the embodiment, LDA is applied to the frequency matrix W created by preprocessing. A variational Bayesian method is used for estimating LDA.

In an embodiment, LDA is used to determine the topic distribution θ _im ={θ ₁ , . . . , θ _L } (where i=1, . Estimate the quantized spectral distribution φ _l ={φ _l1 , . . . , φ _lK } (where l=1, . LDA assumes a multinomial distribution for the quantized spectrum distribution and the topic distribution, and a Dirichlet distribution for its prior distribution. Here, the multinomial distribution represents "the probability when the operation of extracting one value from K kinds of discrete values is performed N times when the probability that the value becomes k is _φk ". A multinomial distribution is represented by the following formula (1). Note that x is a separated sound.

Further, the Dirichlet distribution _{is a} probability distribution with _parameters φ=( ^φ ₁ , _. is represented as

In equation (2), Γ(•) represents the gamma function and the fractional part is the normalization term. Also, β represents a hyperparameter, and the value of β determines the probability that the parameter of the multinomial distribution is φ.

FIG. 6 is a flow chart showing an example of a process of generating a set of LDA quantized spectra.
Note that l represents a topic number, m represents a segment number, and _Nm represents the number of quantized spectra included in segment m. Also, for the nth quantized spectrum of segment m, z _mn represents the topic number and w _mn represents the word number.
In this generation process, the topic distribution and the quantized spectrum distribution are represented by multinomial distributions, and the Dirichlet distribution is used as the prior distribution. Also, α and β represent respective hyperparameters.

(Steps S21 to S23) The extraction unit 37 repeats the process of generating distributions {φ _l to Dirichlet (β)} from 1 to L for topic l (step S22).

(Steps S24 to S30) The extraction unit 37 generates topic distributions {θ _m to Dirichlet(α)} (step S25) up to 1, .

(Steps S26 to S29) The extraction unit 37 generates topics {z _mn to Multinomial(θ _m )} ₍ step S27) and generates quantized spectra { w _mn ˜Multinomial(φ _zmn )} (step S28) is repeated.

A graphical model of LDA will now be described.
A graphical model of LDA is shown in FIG. FIG. 7 is a diagram showing a graphical model of LDA. In FIG. 7, circled nodes (α, β, θ, φ, w, z) represent unknown variables, and squared portions (L, N, M) represent iterations. The graphical model visually represents the probabilistic dependencies of each node.

In embodiments, variational Bayesian methods are used to estimate LDA. A topic estimation method based on the variational Bayesian method will be described below.
In the following description, unknown variables of the topic model are topic set Z, topic distribution set Θ, and quantized spectrum distribution set Φ.
First, the lower limit of variation F of the logarithmic marginal likelihood logp(W|α, β) of the topic model is obtained by the following equation (3).

In Equation (3), Jensen's inequality is used as the third modified inequality. Also, in the fourth modification of the formula, it is assumed that the variational posterior distribution can be modified as q(Z, Θ, Φ)=q(Z)q(Θ, Φ) for simplification of calculation.

Next, the variational posterior distribution q(z) is estimated. In the estimation, the Lagrangian method of undetermined multipliers is used to maximize the lower bound of variation F under the constraint Σ _z q(z)=1 for probability distribution. In the estimation, F(q(Z)) is set as in the following equation (4), and the extremum of equation (4) is obtained.

Note that, in Equation (4), λ(•) is an undetermined multiplier.
Solving ∂F(q(Z))/q(Z)=0, q _mnl that maximizes F((q(Z)) is given by the following equation (5).

However, in Equation (5), Ψ(•) is a digamma function.
Similarly, the lower limit of variation F is maximized for q(Θ, Φ). F(q(Θ, Φ)) is given by the following equation (6), and the extremum of equation (6) is obtained.

By solving ∂F(q(Θ, Φ))/q(Θ, Φ)=0, the variational posterior distribution q(Θ) of the topic distribution is given by the following equation (7).

In Equation (7), the parameter α _ml of the variational posterior distribution q(Θ) of the topic distribution is defined as in Equation (8) below.

Furthermore, the variational posterior distribution q(Φ) of the quantized spectral distribution can be obtained by the following equation (9).

Note that in equation (9), the parameter _βlk is defined as the following equation (10).

The extracting unit 37 estimates the topic distribution and the quantized spectrum distribution by updating the parameters α _ml and β _lk with equations (8) and (10).

FIG. 8 is an example of a variational Bayesian estimation algorithm for a topic model according to this embodiment.

(Step S101) The extraction unit 37 initializes the variational posterior parameters α _ml and β _lk with random positive values.

(Steps S102 to S114) The extraction unit 37 repeats the processes of steps S102 to S114 until the termination condition is satisfied.

(Step S103) The extraction unit 37 sets the parameters α _mn ^new =α and β _lk ^new =β, thereby initializing the parameters after step S104.

(Steps S104 to S112) The extraction unit 37 repeats the processes of steps S104 to S112 M times.
(Steps S105 to S111) The extraction unit 37 repeats the processes of steps S105 to S111 N times.
(Steps S106 to S110) The extraction unit 37 repeats the processes of steps S106 to S110 L times.

(Step S107) The extraction unit 37 performs calculation of Expression (5).
(Step S108) The extraction unit 37 sets the parameter α _ml ^new =α _ml ^new +q _mnl to update the parameter of the variational posterior distribution of the topic distribution.
(Step S109) The extraction unit 37 sets the parameter β _lwmn ^new =β _lwmn ^new +q _mnl to update the parameter of the variational posterior distribution of the quantized spectral distribution.

(Step S110) After repeating the processes of steps S106 to S110 L times, the extraction unit 37 advances to the process of step S111.
(Step S111) After repeating the processes of steps S105 to S111 N times, the extraction unit 37 advances to the process of step S112.
(Step S112) After repeating the process of steps S104 to S112 M times, the extraction unit 37 proceeds to the process of step S113.

(Step S113) The extraction unit 37 sets the parameter α _ml =α _ml ^new ,
Update by setting β _lk =β _lk ^new .
(Step S114) The extraction unit 37 terminates the process after satisfying the termination condition. Note that the end condition is, for example, when the process converges within a predetermined range or when the process is performed a predetermined number of times.

Through these processes, LDA estimates a topic distribution ΘεR ^3M×L for each time interval and a quantized spectrum distribution ΦεR ^L×K for each topic. In the embodiment, the posterior probabilities of the topic distribution and the quantized spectrum distribution exceed the thresholds γ and η, respectively, as defined as the active state.
Specifically, in order to determine which topic is included in the topic distribution θ _im , it is compared with a threshold value γ to determine α _iml as shown in the following equation (11).

Also, the active parameter β _lk in the quantized spectrum distribution is compared with the threshold η to determine β _lk as shown in the following equation (12).

If α _iml =1, it indicates that topic l has a high probability of appearing in that segment. Also, when β _lk =1, it indicates that there is a high probability that cluster k is included in topic l.
If the same sound is included in different time intervals m, m', the topic l containing that sound is likely to be active in both time intervals. That is, it is highly likely that α _iml =α _im'l =1. Therefore, if the same topic is in the active state for each input sound in the same time interval, that topic is highly likely to be the sound source of interest.

FIG. 9 is a diagram showing an example of spectral estimation of a sound source of interest. In FIG. 9 , symbol g110 is an acoustic signal picked up by the microphone array i, which is an input signal to the processing unit 3 . Further, reference g111 indicates an acoustic signal picked up by the first microphone array 2-1, reference g112 indicates an acoustic signal picked up by the second microphone array 2-2, and reference g113 indicates the third microphone array 2-3. indicates the collected acoustic signal. Moreover, the code|symbol g120 represents an estimated signal. In symbols g111 to g113 and g120, the horizontal axis is the time frame and the vertical axis is the amplitude.

FIG. 9 shows the input signal with the most active topics shaded for each time interval. Focusing on the portion surrounded by the rectangle g130, all the input signals in the same time interval are colored in the same color, so that topic represents the sound source of interest.
In the embodiment, if the topic l satisfies α _iml =1 for all input signals in the same time interval m, then the topic l is extracted and the extracted topic is taken as the estimated topic.

Furthermore, the extracting unit 37 selects an estimated topic for each time interval, and extracts a quantized spectrum having an active parameter β _lk =1 of the quantized spectrum distribution of the selected estimated topic. Then, the inverse transform unit 38 restores the estimated signal e _i (t) of the sound source of interest by performing an inverse short-time Fourier transform on the extracted quantized spectrum.

In the example shown in FIG. 9, when the same topic is included in each of the three first to third microphone arrays 2-1 to 2-3 (FIG. 2), the sound source is estimated to be the sound source of interest. Although an example of extracting by
The example of FIG. 9 is an example in which the sound source of interest _S0 is included in the acoustic signals picked up by the three first to third microphone arrays 2-1 to 2-3 as shown in FIG. However, for example, the signal of interest may be included in acoustic signals picked up by two of the three microphone arrays 2-1 to 2-3. In such a case, when the same topic is included in two or more microphone arrays among the multiple microphone arrays used for picking up sound, the common topic is estimated to be the sound source of interest. good too.

As described above, in the present embodiment, attention is paid to the content (topic) of the sound source of interest. Then, in this embodiment, sounds in the direction of the sound source of interest are separated by a plurality of microphone arrays, and the topic of each sound is estimated using a topic model. is assumed to be the sound of the sound source of interest.
Thus, according to the present embodiment, it is possible to easily separate the sound source of interest.

<Evaluation results>
Next, the results of evaluation using the sound source separation device 1 of this embodiment will be described.
For the evaluation, as shown in FIG. 1, the sound sources of four persons were picked up using three microphone arrays and the sound sources were separated. As the sound source, a reading voice by a man with a sampling frequency of 16 kHz and a length of 30 seconds was used. Of these four sound sources, the speech data of the second person was set as the target sound source _S0 . Also, the first person's voice is the sound source _S3 , the third person's voice is the sound source _S1 , and the fourth person's voice is the sound source _S2 . The sound source of interest _S0 spoke in the first half 30 seconds, and the other sound sources spoke in the second half 30 seconds. Thus, three separate signals of 60 seconds total were created. Note that the evaluation was performed in a state in which the utterance time of the target sound source and other sound sources did not overlap. In addition, the evaluation was performed under the condition that the amplitude and phase of the sound source of interest are the same for all the separated signals. The sampling frequency is 16000 Hz, the window width of the short-time Fourier transform is 512, the shift width of the short-time Fourier transform is 256, and the Hamming window is used as the window function of the short-time Fourier transform.

In the evaluation, a short-time Fourier transform was performed on the separated signal X _i (t) created, and the amplitude spectrum Y _i (ω, t) obtained by the transform was quantized and spectralized by the k-means method. The number of clusters for the k-means method was K=100, 300, and 600. In the segmentation, each microphone array was divided into M = 10, 15, 20 and 25 segments, and the time intervals of each segment m were d = 6, 4, 3 and 2.4 seconds, respectively. be. Since the separation signal switches from the target sound source to other sound sources in the 30-second portion, when d = 3 seconds and 6 seconds, the churches of the segments and the churches of the sound sources match. Contains only sound sources. Also, when d=2.4 seconds and 4 seconds, the target sound source and other sound sources are simultaneously included in the segment extending over 30 seconds. After the segmentation, quantized spectra appearing in 70% or more of the whole and quantized spectra appearing only in less than 3 segments were removed to create a frequency matrix W for each segment.

The variational Bayesian method described above was used as the LDA estimation method. Dirichlet distribution is used for both the topic distribution and the prior distribution of the quantized spectrum distribution, and the initial values of the respective hyperparameters are set to 1/L and 1/K. Further, the thresholds for active determination are set to γ=1/L and η=1/K.

BssEval's Source to Distortion Ratio (SDR) was used as an evaluation index for sound source separation performance. SDR represents the energy ratio between the estimated source signal and all noise. The Bss Eval toolbox was used for the calculations. In the evaluation, how much the SDR value improved from the non-separated state was evaluated. In the evaluation, since the estimated signal is obtained for each microphone array, the SDR calculated and averaged for each microphone array was used as an index.

Here, for the true target sound source signal s _i (t) contained in the mixed sound to be separated, the estimated signal s^ _i (t) can be decomposed as shown in the following equation (13).

In equation (13), s _target (t) is the target sound source signal term, e _interf (t) is the noise term derived from other sound sources contained in the mixed sound, and e _noise (t) is the external The noise term from , e _artif (t) represents the noise term from the separation algorithm.
Moreover, the calculation formula of SDR is represented by following Formula (14).

FIG. 10 is a diagram showing an example of extracted sounds when the number of clusters K=600, the segment time interval d=4 seconds, and the number of topics L=5. In FIG. 10, symbol g201 is the waveform of the separated sound in the target sound source direction obtained by the first microphone array 2-1. Symbol g202 is the signal waveform of the correct sound source. Symbol g203 is the waveform of the estimated signal extracted from the acoustic signal picked up by the first microphone array 2-1. In symbols g201 to g203, the horizontal axis is time (seconds) and the vertical axis is amplitude.

In the evaluation results of FIG. 10, when the waveform of the correct sound source and the waveform of the estimated signal are compared, the correct sound source portion of the estimated signal can be extracted for most of the time period. Thus, according to this embodiment, it is possible to accurately extract only the sound in the same time interval as the correct signal.

Next, the results of evaluating how the separation accuracy changes when the value of each parameter is changed will be described.
FIG. 11 is a diagram showing changes in separation performance with the number of topics L when the number of clusters K=600 and the time interval d=4 seconds. The horizontal axis is the number of topics, and the vertical axis is the SDR improvement rate [dB].
In the evaluation results shown in FIG. 11, the SDR improvement rate represents the difference in SDR between when the method of this embodiment is applied and when it is not applied, and the higher the value, the higher the separation performance. . In this evaluation result, when the number of topics L=2, the SDR value hardly improved, whereas the separation performance tended to increase as the number of topics L increased. Therefore, the number of topics L may be changed according to the sound signal to be applied. Also, the number of topics may be set or changed by machine learning, for example.

FIG. 12 is a diagram showing changes in separation performance due to the number of clusters K=100, 300, 600 and the difference in segment length. Symbol g310 is a graph showing changes in separation performance, and symbol g320 is a table showing each value of the graph of symbol g310. In symbol g310, the horizontal axis is the number of clusters, and the vertical axis is the SDR improvement rate [dB]. Reference g311 indicates a time interval of 2.4 seconds, reference g312 indicates a time interval of 3 seconds, reference g313 indicates a time interval of 4 seconds, and reference g314 indicates a time interval of 6 seconds.

In this evaluation result, looking at the number of k-means clusters K, when K is small, the separation accuracy is low. The reason for this is that when the number of clusters K is small, different sounds are also assigned to the same class, resulting in degraded separation performance. Also, if K is too large, a quantized spectrum number will be assigned to each frequency spectrum on a one-to-one basis.
For these reasons, the accuracy can be further improved by setting K to an appropriate value that is neither too small nor too large. For this reason, the number of clusters K may be set according to the application environment or the like, and may be set or changed by machine learning, for example.

Further, in FIG. 12, when comparing the difference between the segments, when d=2.4 seconds and 4 seconds, the sound source of interest and other sound sources are included in the same segment around 30 seconds. The reason for this is that the topic distribution classifies words based on co-occurrence. Therefore, in this evaluation, when d=2.4 seconds and 4 seconds, there is a high possibility that the estimated topic includes a word of another sound source. When the number of clusters is K=600, the SDR value is high when d=3 seconds, while the SDR value is low when d=4 seconds.

FIG. 13 is a diagram showing the evaluation result comparing the separation performance between the case where the number of clusters K=600, the time interval d=4 seconds, and the number of topics L=5, with and without removing silent components and unique components. is. The reference g410 represents the evaluation results in a graph, and the reference g420 represents the values of the graph of the reference g410 in a table. In symbol g410, the horizontal axis is the time interval d, and the vertical axis is the SDR improvement rate [dB]. Reference g411 is the case with silence removal, and reference g412 is the case without silence removal.

As shown in FIG. 13, the SDR value deteriorates when silence is not removed in the comparative example, but the SDR value is greatly improved when silence is removed as in the present embodiment. The reason for this is that since the reading voice of the person used for the evaluation has many silent components, a topic having silent components in a plurality of time intervals was determined to be in the active state.

<Second embodiment>
In the second embodiment, an example in which the direction of each sound source is detected by sound source localization processing and sound source separation processing will be described.

[Configuration example of sound source separation device 1A]
First, a configuration example of the sound source separation device 1A of this embodiment will be described.
FIG. 14 is a block diagram showing a configuration example of the sound source separation device 1A according to this embodiment. As shown in FIG. 14, the sound source separation device 1A includes a sound pickup section 2A and a processing section 3A. Note that functional units having functions similar to those of the sound source separation device 1 of the first embodiment are denoted by the same reference numerals, and descriptions thereof are omitted.
The sound pickup section 2A includes a first microphone array 2-1, a second microphone array 2-2, and a third microphone array 2-3.
The processing unit 3A includes an acquisition unit 31A, a sound source localization unit 32, a sound source separation unit 33, a conversion unit 34A, a classification unit 35, a removal unit 36, an extraction unit 37, an inverse conversion unit 38, and an output unit 39.

[Operations and functions of the sound source separation device 1A]
Next, an example of the operation and function of each part of the sound source separation device 1A will be described.
Each of the first microphone array 2-1, the second microphone array 2-2, and the third microphone array 2-3 outputs a picked-up P-channel acoustic signal to the processing unit 3A. The P-channel acoustic signal output from each microphone array includes identification information for identifying the microphone array.

The acquisition unit 31A acquires P-channel acoustic signals output from the first microphone array 2-1, the second microphone array 2-2, and the third microphone array 2-3. The acquisition unit 31</b>A outputs the acquired P-channel acoustic signal for each microphone array to the sound source localization unit 32 and the sound source separation unit 33 .

The sound source localization unit 32 acquires the P-channel acoustic signal for each microphone array output by the acquisition unit 31A. The sound source localization unit 32 performs sound source localization processing by, for example, the beamforming method or the MUSIC method on the acquired P-channel acoustic signal for each microphone array, and estimates the sound source direction included in the acoustic signal. The sound source localization unit 32 outputs the estimated sound source localization information to the sound source separation unit 33 for each microphone array.

The sound source separation unit 33 acquires the sound source localization information output by the sound source localization unit 32 and the M-channel acoustic signals for each microphone array output by the acquisition unit 31A. The sound source separation unit 33 extracts an acoustic signal in the direction in which the sound source is localized from the M-channel acoustic signals for each microphone array. The sound source separation unit 33 performs sound source separation processing by, for example, the GHDSS (Geometric High-order Dicorrelation-based Source Separation) method. For example, in FIG. 1, when the microphone array MA ₁ is the first microphone array 2-1, the sound sources S ₀ and S ₁ are extracted as one-channel acoustic signals. Similarly, the sound source separation unit 33 extracts an acoustic signal corresponding to the sound source from the P-channel acoustic signal picked up by the second microphone array 2-2. The sound source separation unit 33 extracts an acoustic signal corresponding to the sound source from the P-channel acoustic signal picked up by the third microphone array 2-3. The sound source separation unit 33 outputs the extracted acoustic signal for each microphone array to the conversion unit 34A.

In this embodiment, the reference direction of the multiple microphone arrays may be set, for example, in the direction of the center of gravity (position of the sound source of interest _S0 ) of the multiple microphone arrays MA ₁ to MA ₃ in FIG.

In the first embodiment, the acoustic signal of the sound source of interest is separated by picking up the acoustic signal including the sound source of interest using beams formed by the beamforming method. separates the acoustic signal of the sound source of interest. After that, the processing unit 3A extracts topics, classifies them, and estimates an estimated topic by extracting common topics, as in the first embodiment.

The configuration of the sound source separation device 1A of this embodiment can also provide the same effects as those of the first embodiment.

<Modification>
In the first and second embodiments described above, an example of performing clustering by the k-means method has been described, but the present invention is not limited to this. Clustering may use other well-known techniques (for example, weighted average method, etc.).

In addition, in the above-described first and second embodiments, an example has been described in which clustering is performed first, noise components are removed by the removal unit 36 after clustering, and the sound source of interest is extracted after the noise components are removed. , but not limited to this.

FIG. 15 is a diagram for explaining silent intervals and speech intervals.
As shown in FIG. 15, the acoustic signal generally includes a silent section g501. It is also possible to remove such a silent section or extract the speech section g502 and extract the topic from each predetermined section with respect to the speech section. Silent intervals or speech intervals may be detected, for example, by comparing a threshold value for detecting speech intervals with respect to the amplitude of the acoustic signal and the acoustic signal.

A program for realizing all or part of the functions of the sound source separation device 1 (or 1A) of the present invention is recorded on a computer-readable recording medium, and the program recorded on this recording medium is transferred to a computer system. All or part of the processing performed by the sound source separation device 1 (or 1A) may be performed by reading and executing the program. It should be noted that the "computer system" here includes hardware such as an OS and peripheral devices. Also, the "computer system" includes a WWW system provided with a home page providing environment (or display environment). The term "computer-readable recording medium" refers to portable media such as flexible discs, magneto-optical discs, ROMs and CD-ROMs, and storage devices such as hard discs incorporated in computer systems. In addition, "computer-readable recording medium" means a volatile memory (RAM) inside a computer system that acts as a server or client when a program is transmitted via a network such as the Internet or a communication line such as a telephone line. , includes those that hold the program for a certain period of time.

Further, the above program may be transmitted from a computer system storing this program in a storage device or the like to another computer system via a transmission medium or by a transmission wave in a transmission medium. Here, the "transmission medium" for transmitting the program refers to a medium having a function of transmitting information, such as a network (communication network) such as the Internet or a communication line (communication line) such as a telephone line. Further, the program may be for realizing part of the functions described above. Further, it may be a so-called difference file (difference program) that can realize the above-described functions in combination with a program already recorded in the computer system.

1, 1A ... sound source separation device,
2, 2A ... sound pickup part,
3, 3A ... processing unit,
2-1 ... first microphone array,
2-2 ... second microphone array,
2-3 ... third microphone array,
30 ... beam forming control unit,
31 ... Acquisition unit,
32... Sound source localization section,
33... Sound source separation section,
34, 34A ... conversion unit,
35... Classifying section,
36 ... removal unit,
37 ... extraction part,
38 ... inverse transformation unit,
39 ... output section,
40... Sound source localization part

Claims (8)

a plurality of microphone arrays for picking up acoustic signals;
When each of the collected sound signals picked up by the at least two microphone arrays includes a first sound signal of a sound source of interest and a second sound signal of another sound source in the same direction as the sound source of interest, at least an extraction unit that extracts a common component included in each of the collected sound signals picked up by the two microphone arrays and extracts the first sound signal from the collected sound signals ;
a classification unit that classifies topics of sounds included in the collected sound signal;
The extractor is
The classifying unit compares the topics classified for each of the microphone arrays, and if the topic is the same in the collected sound signals picked up by each of the plurality of microphone arrays as a result of the comparison, the same topic is identified. A sound source separation device that presumes that the sound source is the target sound source and extracts an acoustic signal corresponding to the same topic from the collected sound signals as the first acoustic signal. The extractor is
extracting the common component using a latent Dirichlet allocation method;
The sound source separation device according to claim 1. further comprising a conversion unit that converts the collected sound signal collected by each of the microphone arrays into a frequency spectrum;
The classification unit
The frequency spectrum for each microphone array is segmented by dividing it into M (M is an integer equal to or greater than 2) sections in a time frame, and the frequency spectrum for each of the time frames included in each segment is divided into the topic. classify by
The sound source separation device according to claim 1 or 2 . The extractor is
estimating the distribution of the topic for each time interval and the distribution of the quantized spectrum obtained by quantizing the frequency spectrum for each topic, and the posterior probabilities of the distribution of the topic and the distribution of the quantized spectrum each indicate an active state; Those larger than the threshold for discrimination are assumed to be in an active state,
extracting the common component by comparing the distribution of the topics for each of the segments at the same time and extracting the topics that are active in at least two of the microphone arrays;
The sound source separation device according to claim 3 . a control unit that controls the microphone array to form a beam in the direction of the sound source of interest;
The plurality of microphone arrays pick up the collected sound signal including the first sound signal of the sound source of interest under the control of the control unit.
The sound source separation device according to any one of claims 1 to 4 . a sound source localization unit that performs sound source localization on the sound signals picked up by each of the microphone arrays;
a sound source separation unit that separates a separated signal including the first acoustic signal from the collected sound signals picked up by each of the microphone arrays based on the localization result of the sound source localization;
The extractor is
Extracting a common component included in each of the separated signals separated from the collected sound signals of each of the at least two microphone arrays to extract the first sound signal from the collected sound signals;
The sound source separation device according to any one of claims 1 to 4 . A multiple microphone array picks up the acoustic signal,
The extraction unit causes each collected sound signal picked up by the at least two microphone arrays to include a first sound signal of a sound source of interest and a second sound signal of another sound source in the same direction as the sound source of interest. extracting a common component included in each of the collected sound signals picked up by at least two of the microphone arrays to extract the first sound signal from the collected sound signals ;
a classifying unit classifying sound topics included in the collected sound signal;
When the extraction unit compares the topics classified for each microphone array by the classification unit, and as a result of the comparison, the same topic is included in the collected sound signals picked up by each of the plurality of microphone arrays estimating the same topic as the sound source of interest, and extracting an acoustic signal corresponding to the same topic from the collected sound signal as the first acoustic signal;
sound source separation method. to the computer,
Acoustic signals are picked up by multiple microphone arrays,
When the collected sound signals picked up by each of the at least two microphone arrays include a first sound signal of a sound source of interest and a second sound signal of another sound source in the same direction as the sound source of interest, at least two extracting a common component included in each of the collected sound signals picked up by the two microphone arrays, and extracting the first sound signal from the collected sound signals ;
classifying sound topics contained in the collected sound signal;
Compare the topics classified for each of the microphone arrays, and if the topic is the same in the picked-up acoustic signals picked up by each of the plurality of microphone arrays as a result of the comparison, the same topic is picked up by the sound source of interest. estimating that there is, extracting an acoustic signal corresponding to the same topic from the collected sound signal as the first acoustic signal;
program.

Images