JP2014215544A - Sound processing device - Google Patents

Abstract

PROBLEM TO BE SOLVED: To achieve a highly accurate sound source separation with a specified component of a sound signal as an arithmetic operation object of a non-negative value matrix factorization.SOLUTION: A matrix creating unit 34 generates a separation matrix Q in which an element q (m, n) corresponding to each time frequency component y (m, n) where a first sound source is dominant among sound signals SA is set to a numerical value 1, and each remaining element q (m, n) is set to 0. A matrix decomposition unit 36 calculates a coefficient matrix G corresponding to a base matrix F and a coefficient matrix U corresponding to a base matrix H of a second sound source and a coefficient matrix U corresponding to the base matrix H from an observation matrix Y having arranged each time frequency component y (m, n) of the sound signal SA by repeating an update operation of a non-negative value matrix factorization using the base matrix F of the first sound source as teacher information with each time frequency component y (m, n) corresponding to the element q (m, n) set to the numerical value 1 in the separation matrix Q as an arithmetic operation object. The matrix decomposition unit 36 executes the non-negative value matrix factorization under a constraint condition that each element of the coefficient matrix G is suppressed.

Description

  The present invention relates to a technique for separating an acoustic signal for each sound source.

  A sound source separation technique for separating a mixed sound of a plurality of sounds generated by different sound sources into sounds for each sound source has been proposed. For example, Non-Patent Document 1 discloses a sound source separation technique using supervised non-negative matrix factorization (SNMF) in which a base matrix representing acoustic characteristics of a specific sound source is used as teacher information. Yes. Non-Patent Document 2 discloses that spatial sound source separation using the spatial position of each sound source specified from acoustic signals of a plurality of channels is used in combination with supervised non-negative matrix factorization. . Specifically, each frequency component (hereinafter referred to as “temporal frequency component”) at each time point of the acoustic signal is classified (clustered) for each sound source direction, and the component in which the sound source is located in the target direction is separated from the acoustic signal. Thus, non-negative matrix factorization similar to Non-Patent Document 1 is executed.

K. Yagi, et. Al., "Music Signal Separation by Orthogonality and Maximum-Distance Constrained Nonnegative Matrix Factorization with Target Signal Information", Proc. Of Audio Engineering Society 45th International Conference Applications of Time-Frequency Processing in Audio (AES45th), March 2012 Y. Iwao, et. Al., "Stereo Music Signal Separation Combining Directional Clustering and Nonnegative Matrix Factorization". Proc. The 12th IEEE international Symposium on Signal Processing and Information Technology (ISSPIT 2012), December 2012

  However, in the configuration (hard clustering) in which each time-frequency component of the acoustic signal is selectively classified into any of a plurality of classes as in Non-Patent Document 2, the time is required for the acoustic signal after the spatial sound source separation is performed. -There is a problem in that high-precision sound source separation is difficult as a result of missing time frequency components (zero intensity) at many points in the frequency domain and deviating from the acoustic characteristics represented by the base matrix of the teacher information. In the above description, the case where a specific component to be subjected to non-negative matrix factorization is extracted by spatial sound source separation from the acoustic signal is illustrated for the sake of convenience. In the value matrix factorization, the same problem may occur regardless of the extraction method of the specific component. In view of the above circumstances, an object of the present invention is to realize high-accuracy sound source separation under a configuration in which non-negative matrix factorization is performed on a specific component of an acoustic signal as a calculation target.

  In order to solve the above problems, the acoustic processing device according to the first aspect of the present invention includes elements corresponding to each time frequency component of an acoustic signal representing a mixed sound of acoustics of a plurality of sound sources, Among them, the element corresponding to each time frequency component in which the sound of the first sound source is dominant is set to a maintenance value that maintains the time frequency component, and the element corresponding to each remaining time frequency component suppresses the time frequency component. The matrix generating means for generating the separation matrix set to the suppression value (for example, the separation matrix Q), and the sound of the first sound source with each time frequency component corresponding to the element set to the maintenance value in the separation matrix as the calculation target By repeating the update operation of the non-negative matrix factorization using a first basis matrix (for example, basis matrix F) including a plurality of basis vectors representing the spectrum of each component in the teacher information, each time of the acoustic signal From an observation matrix in which wave number components are arranged, a first coefficient matrix (for example, coefficient matrix G) including a plurality of coefficient vectors corresponding to each basis vector of the first basis matrix and the sound of a second sound source that is different from the first sound source. And a second coefficient matrix (for example, coefficient matrix) including a plurality of coefficient vectors corresponding to the respective basis vectors of the second basis matrix. U) and a matrix decomposition unit that calculates non-negative matrix factorization using a constraint condition (for example, a first constraint condition described later) to suppress each element of the first coefficient matrix. Run. In non-negative matrix factorization in which each time frequency component corresponding to an element set as a maintenance value in a separation matrix is an operation target, each element of a first coefficient matrix corresponding to a first basis matrix used as teacher information May be set to an excessively large number. In the sound processing device according to the first aspect of the present invention, since the non-negative matrix factorization is performed with the constraint that the elements of the first coefficient matrix are suppressed, each element of the first coefficient matrix is excessively large. The possibility of setting a large number is reduced. Therefore, highly accurate sound source separation can be realized.

  In a preferred aspect of the present invention, the matrix decomposing means converts each element of the inverted separation matrix (for example, the inverted separation matrix / Q) obtained by inverting the maintenance value and the suppression value for each element of the separation matrix to the first basis matrix and the first matrix. Execute non-negative matrix factorization with the constraint that the norm of the constraint target matrix (for example, the constraint target matrix {/Q.×(FG)}) multiplied by each element of the matrix product with one coefficient matrix is reduced. . In the above embodiment, non-negative matrix factorization is executed with the constraint that the norm of the constrained matrix obtained by multiplying each element of the matrix product of the first base matrix and the first coefficient matrix by the element of the inverted separation matrix is reduced. Therefore, the above-described effect of improving the accuracy of sound source separation is particularly remarkable.

  The above-described problem that each element of the first coefficient matrix can be set to an excessively large numerical value in the non-negative matrix factorization in which each time frequency component corresponding to the element set as the maintenance value in the separation matrix is an operation target. The smaller the number of maintenance values in the separation matrix (the smaller the time frequency component in which the sound of the first sound source is dominant), the more obvious it becomes. Considering the above circumstances, the configuration in which the degree of influence of the constraint condition in the update calculation is variably controlled according to the number of maintenance values in the separation matrix, and the index value according to the number of maintenance values in the separation matrix exceeds the threshold value In such a case, it is preferable that the non-negative matrix factorization is executed under the constraint condition, and the constraint condition is canceled when the index value falls below the threshold value.

  The acoustic processing device according to the second aspect of the present invention includes elements corresponding to each time frequency component of an acoustic signal representing a mixed sound of acoustics of a plurality of sound sources, and the sound of the first sound source is dominant among the plurality of sound sources. An element corresponding to each time frequency component is set to a maintenance value for maintaining the time frequency component, and an element corresponding to each remaining time frequency component is set to a suppression value for suppressing the time frequency component ( For example, matrix generation means for generating a separation matrix Q), and a plurality of time frequency components corresponding to elements set as maintenance values in the separation matrix are calculated, An observation matrix in which the time frequency components of the acoustic signal are arranged by repeating the update operation of the non-negative matrix factorization using the first basis matrix (for example, the basis matrix F) including the basis vector as the teacher information. The first coefficient matrix (for example, coefficient matrix G) including a plurality of coefficient vectors corresponding to each basis vector of the first basis matrix and the spectrum of each component of the sound of the second sound source different from the first sound source are represented. A matrix for calculating a second basis matrix (for example, basis matrix H) including a plurality of basis vectors and a second coefficient matrix (for example, coefficient matrix U) including a plurality of coefficient vectors corresponding to each basis vector of the second basis matrix. Decomposing means, and the matrix decomposing means includes a time frequency component (for example, time frequency component d) corresponding to an element set as a suppression value in the separation matrix of the matrix product of the first base matrix and the first coefficient matrix. (m, n)) Non-negative matrix factorization is performed with the constraint of suppressing temporal variation. In the above configuration, the constraint condition is to suppress temporal variation of the time frequency component corresponding to the element set as the suppression value in the separation matrix of the matrix product of the first basis matrix and the first coefficient matrix. Since the non-negative matrix factorization is performed, the possibility that each element of the first coefficient matrix is set to an excessively large numerical value (a numerical value that varies discontinuously on the time axis) is reduced. Therefore, as with the sound processing device of the first aspect, it is possible to realize high-accuracy sound source separation.

  The sound processing apparatus according to each of the above aspects is realized by hardware (electronic circuit) such as a DSP (Digital Signal Processor) dedicated to processing of an acoustic signal, or a general-purpose operation such as a CPU (Central Processing Unit). This is also realized by cooperation between the processing device and the program. The program of the present invention can be provided in a form stored in a computer-readable recording medium and installed in the computer. The recording medium is, for example, a non-transitory recording medium, and an optical recording medium (optical disk) such as a CD-ROM is a good example, but a known arbitrary one such as a semiconductor recording medium or a magnetic recording medium This type of recording medium can be included. For example, the program of the present invention can be provided in the form of distribution via a communication network and installed in a computer.

  Moreover, the acoustic processing device according to each of the above aspects includes an element corresponding to each time frequency component of the acoustic signal representing the mixed sound of the sound of the plurality of sound sources, and the sound of the first sound source is dominant among the plurality of sound sources. A separation matrix in which an element corresponding to each time frequency component is set to a maintenance value for maintaining the time frequency component, and an element corresponding to each remaining time frequency component is set to a suppression value for suppressing the time frequency component. The first basis matrix including a plurality of basis vectors representing the spectrum of each component of the sound of the first sound source is calculated with each time frequency component corresponding to the element that is generated and set as the maintenance value in the separation matrix as a calculation target. By repeating the update operation of the non-negative matrix factorization used as information, a plurality of coefficients corresponding to each basis vector of the first basis matrix from the observation matrix in which each time frequency component of the acoustic signal is arranged Corresponds to a first coefficient matrix including a spectrum, a second basis matrix including a plurality of basis vectors representing the spectrum of each acoustic component of the second sound source different from the first sound source, and each basis vector of the second basis matrix It is also specified as a method (acoustic processing method) for calculating a second coefficient matrix including a plurality of coefficient vectors. In the acoustic processing method according to the first aspect, non-negative matrix factorization is performed with the constraint that suppression of each element of the first coefficient matrix is performed, and in the acoustic processing method according to the second aspect, the first basis matrix and Non-negative matrix factorization is executed with the constraint that the temporal variation of the time-frequency component corresponding to the element set as the suppression value in the separation matrix of the matrix product with the first coefficient matrix is restrained. .

1 is a block diagram of a sound processing apparatus according to a first embodiment of the present invention. It is explanatory drawing of nonnegative matrix factorization. It is a flowchart of direction clustering. It is explanatory drawing of the problem of contrast 1. It is explanatory drawing of the problem of contrast 2. It is explanatory drawing of the problem of contrast 2. It is explanatory drawing of a separation matrix and an inversion separation matrix. It is a chart of an experimental result of a 1st embodiment and contrast 3. It is a table | surface of 1st Embodiment and the experimental result of contrast 4. It is a chart of a 1st embodiment and an experimental result of contrast 2. It is a block diagram of the sound processing apparatus which concerns on 2nd Embodiment. It is explanatory drawing of the index value in the modification of 2nd Embodiment.

<First Embodiment>
FIG. 1 is a block diagram of a sound processing apparatus 100 according to the first embodiment of the present invention. As shown in FIG. 1, a signal supply device 12 and a sound emitting device 14 are connected to the sound processing device 100. The signal supply device 12 generates a time-domain acoustic signal SA representing a waveform of a mixed sound of sounds generated by a plurality of sound sources that may have different spatial positions (directions relative to the sound collection point) or acoustic properties. 100. The acoustic signal SA is a multi-channel signal composed of acoustic signals SA [1] to SA [J] of J channels (J is a natural number of 2 or more). A sound source having a known spatial position or acoustic characteristics among a plurality of sound sources represented by the acoustic signal SA is hereinafter referred to as a first sound source, and a sound source other than the first sound source is hereinafter referred to as a second sound source. Each of the first sound source and the second sound source corresponds to a single sound source or a plurality of sound sources (single sound source or sound source group). Sound collecting equipment that picks up surrounding sounds and generates an acoustic signal SA, a playback device that acquires an acoustic signal SA from a portable or built-in recording medium and supplies the acoustic signal SA to the acoustic processing device 100, or an acoustic from a communication network A communication device that receives the signal SA and supplies it to the sound processing device 100 can be used as the signal supply device 12.

  The acoustic processing device 100 according to the first embodiment is a signal processing device (sound source separation device) that generates an acoustic signal SB by sound source separation with respect to the acoustic signal SA supplied from the signal supply device 12. The sound signal SB is a time-domain signal obtained by separating (extracting or suppressing) one sound of the first sound source and the second sound source from the sound signal SA. Specifically, for example, an acoustic signal SB obtained by extracting the sound of the sound source selected by the user from the first sound source and the second sound source is generated. That is, the acoustic signal SA is separated for each sound source. The sound emitting device 14 (for example, a speaker or a headphone) emits a sound wave corresponding to the acoustic signal SB generated by the acoustic processing device 100.

  As shown in FIG. 1, the sound processing device 100 is realized by a computer system including an arithmetic processing device 22 and a storage device 24. The storage device 24 stores a program executed by the arithmetic processing device 22 and various data (for example, a base matrix F) used by the arithmetic processing device 22. A known recording medium such as a semiconductor recording medium or a magnetic recording medium or a combination of a plurality of types of recording media can be arbitrarily employed as the storage device 24. A configuration in which the acoustic signal SA is stored in the storage device 24 (therefore, the signal supply device 12 can be omitted) is also suitable.

  The arithmetic processing device 22 executes a program stored in the storage device 24 to thereby generate a plurality of functions (frequency analysis unit 32, matrix generation unit 34, matrix decomposition unit 36) for generating the acoustic signal SB from the acoustic signal SA. , A waveform synthesis unit 38) is realized. Processing by each element of the arithmetic processing unit 22 is sequentially executed in units of N units (frames) obtained by dividing the acoustic signal SA on the time axis (N is a natural number of 2 or more). A configuration in which the functions of the arithmetic processing device 22 are distributed to a plurality of devices, or a configuration in which a dedicated electronic circuit (for example, a DSP) realizes a part of the functions of the arithmetic processing device 22 may be employed.

  The frequency analysis unit 32 performs frequency analysis such as short-time Fourier transform on the acoustic signal SA. Specifically, the frequency analysis unit 32 observes from the acoustic signal SA (for example, any one of the average and sum of the acoustic signals SA [1] to SA [J] or the acoustic signals SA [1] to SA [J]). A matrix Y is generated. As shown in FIG. 2, the observation matrix Y has M rows N in which time frequency components y (m, n) (y (1,1) to y (M, N)) are arranged in the vertical direction and the horizontal direction. A matrix of columns. The symbol m is a variable indicating any one of M frequencies (frequency bins) set on the frequency axis (m = 1 to M), and the symbol n is N numbers on the time axis. It is a variable indicating any one of the unit sections (n = 1 to N). An arbitrary one time frequency component y (m, n) means the intensity (amplitude) at the mth frequency on the frequency axis of the acoustic signal SA in the nth unit interval on the time axis. . Therefore, the sequence of M time frequency components y (1, n) to y (M, n) located in the nth column of the observation matrix Y corresponds to the amplitude spectrum of the acoustic signal SA in the nth unit interval. To do. As understood from the above description, the observation matrix Y is a non-negative matrix of M rows and N columns representing a time series (amplitude spectrogram) of the amplitude spectrum of the acoustic signal SA over N unit intervals.

Further, the frequency analysis unit 32 calculates a time frequency component xj (m, n) for each of the acoustic signals SA [1] to SA [J] of J channels. Any one time frequency component xj (m, n) of the jth channel is the mth component on the frequency axis of the acoustic signal SA [j] in the nth unit interval on the time axis. It means intensity (amplitude) at frequency. The frequency analysis unit 32 arranges observation vectors VX (m, n) (VX (m, n) = [x1 () in which the time and frequency components xj (m, n) having a common frequency and unit interval are arranged for J channels. m, n), x2 (m, n),..., xJ (m, n)] ^T ) is generated for each unit interval for each frequency. The symbol T means transposition of the matrix.

  1 generates a separation matrix Q used as a mask for extracting the sound of the first sound source from the sound signal SA. The matrix generation unit 34 of the first embodiment generates a separation matrix Q using orientation clustering. In orientation clustering, each observation vector VX (m, n) has R classes for each spatial orientation of each sound source (sound image) estimated from acoustic signals SA [1] to SA [J] of J channels. A plurality of observation vectors VX (m, n) classified into each class are approximately expressed by space representative vectors Λr (m) (Λ1 (m) to ΛR (m)) of the class ( r = 1 to R). Although orientation clustering is also described in Non-Patent Document 2, an outline will be described below in order to understand the separation matrix Q.

数式(1)から理解される通り、変数Ψ(m,n)は、観測ベクトルＶX(m,n)に最も類似するセントロイドＣr(m)に対応するクラスの番号ｒに設定される。 FIG. 3 is a flowchart of orientation clustering. When orientation clustering is started, the matrix generation unit 34 sets initial R centroids C1 (m) to CR (m) corresponding to different classes (S1). Then, the matrix generator 34 expresses an error E (VX (m, n), Cr () between each centroid Cr (m) and the observed vector VX (m, n) as expressed by the following equation (1). m)) The number Ψ (m, n) of the class to which the observation vector VX (m, n) belongs is determined so that ² is minimized (S2).

As understood from the equation (1), the variable Ψ (m, n) is set to the number r of the class corresponding to the centroid Cr (m) most similar to the observation vector VX (m, n).

The matrix generator 34 represents an error E between the plurality of observation vectors VX (m, n) classified into the r-th class and the current centroid Cr (m) as expressed by the following equation (2). The centroid Cr (m) of the rth class is updated so that the total value of (VX (m, n), Cr (m)) ² is minimized (S3). The symbol Θr in Equation (2) means a set of unit intervals in which the observation vectors VX (m, n) belonging to the rth class are generated.

  The matrix generation unit 34 determines whether the convergence condition is successful (S4). Specifically, the presence / absence of a change in centroid Cr (m) before and after the update in step S3 is suitable as the convergence condition. That is, when the centroid Cr (m) changes before and after the update, the matrix generation unit 34 determines that the convergence condition is not satisfied (S4: NO), and proceeds to step S2. On the other hand, when the centroid Cr (m) does not change before and after the update, the matrix generation unit 34 determines that the convergence condition is satisfied (S4: YES), and sets the centroid Cr (m) immediately after the update to the first The r-th class space representative vector Λr (m) is determined (S5).

When the orientation clustering of FIG. 3 described above is executed, the matrix generation unit 34 generates the separation matrix Q according to the variable Ψ (m, n) set in step S2 of FIG. As illustrated in FIG. 2, the separation matrix Q includes elements q (m, n) (q (1,1) to q corresponding to each time frequency component y (m, n) of the observation matrix Y on a one-to-one basis. (M, N)) is a non-negative matrix with M rows and N columns arranged in the vertical and horizontal directions. Each element q (m, n) of the separation matrix Q is expressed by the following mathematical formula (3).

  The symbol η in Equation (3) means the number of one class (hereinafter referred to as “target class”) corresponding to the first sound source among the R classes in the orientation clustering, and is in the known direction of the first sound source. It is set in advance accordingly. For example, the separation matrix Q is generated with the class corresponding to the front direction of the sound collection point as the target class. As understood from Equation (3), in the separation matrix Q, the variable Ψ (m, n) corresponds to the time frequency component (each time frequency component classified into the target class) y (m, n) with the number η. Element q (m, n) to be set is a numerical value 1 and the variable Ψ (m, n) is a numerical value other than the number η (a time frequency component classified into a class other than the target class) y ( The element q (m, n) corresponding to m, n) is set to the numerical value 0. That is, the element q (m, n) corresponding to each time frequency component y (m, n) in which the sound of the first sound source is dominant compared with the sound of the second sound source in the separation matrix Q is set to a numerical value 1. The element q (m, n) corresponding to each remaining time frequency component (that is, the time frequency component in which the sound of the second sound source is dominant compared to the sound of the first sound source) y (m, n) is 0. Is set. The numerical value 1 of the element q (m, n) corresponds to a numerical value (maintenance value) that maintains the time frequency component y (m, n) when multiplied by the time frequency component y (m, n). The numerical value 0 of (m, n) corresponds to a numerical value (suppression value) that suppresses the time frequency component y (m, n) when multiplied by the time frequency component y (m, n).

  As is understood from FIG. 2, the matrix decomposition unit 36 in FIG. 1 is a non-negative matrix factorization of the observation matrix Y calculated by the frequency analysis unit 32, and converts the first matrix D1 corresponding to the first sound source and the second sound source. The corresponding second matrix D2 is calculated. The first matrix D1 is an M-row × N-column non-negative matrix representing the time series of the acoustic amplitude spectrum of the first sound source (the acoustic amplitude spectrogram of the first sound source) of the acoustic signal SA, and the base matrix F And the coefficient matrix G. As illustrated in FIG. 2, the base matrix F has M rows and K columns in which K base vectors f (1) to f (K) corresponding to the components constituting the sound of the first sound source are arranged in the horizontal direction. Is a non-negative matrix. The basis vector f (k) of the k-th column (k = 1 to K) of the basis matrix F is M elements f (1, k) to f (M, k) corresponding to different frequencies on the frequency axis. ), And corresponds to the amplitude spectrum of the k-th component of the K components (for example, components of K pitches that can be generated by the first sound source) constituting the sound of the first sound source. On the other hand, the coefficient matrix G is a non-negative matrix of K rows and N columns in which K coefficient vectors g (1) to g (K) corresponding to each base vector f (k) of the base matrix F are arranged in the vertical direction. is there. The coefficient vector q (k) in the k-th row of the coefficient matrix G is composed of N elements (coefficients) g (k, 1) to g (k, N) corresponding to different unit intervals on the time axis. This corresponds to a time series of weight values (activity) for the basis vector f (k) of the basis matrix F.

  The second matrix D2 is an M-row × N-column non-negative matrix representing a time series of the amplitude spectrum of the sound of the second sound source in the sound signal SA (the amplitude spectrogram of the sound of the second sound source). And a coefficient matrix U. The base matrix H is a non-negative matrix of M rows and K columns in which K base vectors h (1) to h (K) are arranged in the horizontal direction, as illustrated in FIG. One arbitrary basis vector h (k) corresponds to the amplitude spectrum of each component of the sound of the second sound source, and M elements h (1, k) to h corresponding to different frequencies on the frequency axis. (M, k). On the other hand, the coefficient matrix U is a non-negative matrix of K rows and N columns in which K coefficient vectors u (1) to u (K) are arranged in the vertical direction. An arbitrary coefficient vector u (k) is composed of N elements u (k, 1) to u (k, N) corresponding to different unit intervals on the time axis, and the basis of the base matrix H This corresponds to a time series of weight values for the vector h (k). Note that the number of rows of the base matrix F (the number of columns of the coefficient matrix G) and the number of rows of the base matrix H (the number of columns of the coefficient matrix U) can be made different.

  The matrix decomposition unit 36 according to the first embodiment performs a coefficient matrix G, a base matrix H, and a coefficient matrix U by supervised non-negative matrix factorization using a known base matrix F prepared in advance as teacher information (prior information). Are calculated from the observation matrix Y. The base matrix F is generated in advance from the sound generated by a known first sound source alone (the sound not including the sound of the second sound source) and then stored in the storage device 24.

  By the way, in the separation matrix Q generated by the matrix generation unit 34 described above, the element q (m, n) corresponding to each time frequency component y (m, n) in which the sound of the first sound source is dominant is set to a numerical value 1. In addition, the element q (m, n) corresponding to each remaining time frequency component y (m, n) is set to a numerical value 0. Therefore, as illustrated in FIG. 4, the matrix {generated by the spatial source separation that multiplies each time frequency component y (m, n) of the observation matrix Y by each element q (m, n) of the separation matrix Q. A configuration (hereinafter referred to as “proportional 1”) that performs non-negative matrix factorization for Y. × Q} is assumed. The symbol “. ×” means multiplication (Hadamard product) for each element between the matrices.

  However, in the configuration (hard clustering) in which each time frequency component y (m, n) of the acoustic signal SA is selectively classified into any of the R classes as in the azimuth clustering described above, the sound of the second sound source The element q (m, n) corresponding to the dominant time frequency component y (m, n) is compared with the second sound source even when the time frequency component y (m, n) contains the sound of the first sound source. If it is inferior, it can be set to a numerical value 0 (suppression value). Therefore, as can be understood from FIG. 4, the matrix {Y. × Q} after the spatial sound source separation is performed includes a number of elements including elements of the sound of the first sound source before the spatial sound source separation is performed. The intensity of the element becomes 0 (missing time frequency components at many points in the time-frequency domain), and there is a possibility that the acoustic characteristics (frequency characteristics) are different from the sound of the first sound source represented by the base matrix F. is there. Therefore, with the proportionality 1, it is difficult to separate the sound sources with high accuracy.

数式(4)の記号δ(Ｙ|ＦＧ＋ＨＵ)は、観測行列Ｙと分離後の行列｛ＦＧ＋ＨＵ｝との距離（例えばユークリッド距離）を意味する。数式(4)の評価関数Φを適用した対比例２によれば、分離行列Ｑの要素ｑ(m,n)の数値０の影響が基底行列Ｆにより補償されるから、対比例１と比較して高精度な音源分離が実現される。 From the viewpoint of solving the above problems, only the time frequency component y (m, n) corresponding to the element q (m, n) set to the numerical value 1 in the separation matrix Q in the observation matrix Y is non-negative. A configuration (hereinafter referred to as “proportional 2”) that executes update calculation of value matrix factorization is assumed. In the non-negative matrix factorization of the proportional 2, for example, for evaluating the difference (distance) between the observation matrix Y before processing and the matrix after processing (matrix sum of the first matrix D 1 and the second matrix D 2). Each matrix (G, H, U) after separation so that the evaluation function Φ in Equation (4) is minimized (the distance between the two is minimized or the correlation between the two is maximized) Are updated iteratively.

The symbol δ (Y | FG + HU) in Equation (4) means the distance (for example, Euclidean distance) between the observation matrix Y and the separated matrix {FG + HU}. According to the comparative 2 to which the evaluation function Φ of the formula (4) is applied, the influence of the numerical value 0 of the element q (m, n) of the separation matrix Q is compensated by the base matrix F. Highly accurate sound source separation.

  However, in contrast 2, the number of time frequency components y (m, n) classified into the target class of the first sound source (the total number of elements q (m, n) set to 1 in the separation matrix Q) is When the number is small, the coefficient is such that each time frequency component y (m, n) of the first sound source scattered in the time-frequency domain is approximated by a multiple of each basis vector f (k) of the basis matrix F. There is a possibility that the element g (k, n) (the gain of the base vector f (k)) of the coefficient vector g (k) of the matrix G becomes an excessively large numerical value. For example, only the element q (m1, n) corresponding to the time frequency component y (m1, n) of the m1st frequency in the actual amplitude spectrum of the sound of the first sound source (part (A) in FIG. 5) is obtained. Assume that the numerical value 1 is set according to the result of the orientation clustering and the remaining element q (m, n) is set to the numerical value 0. In the above situation, only the time frequency component y (m1, n) of the m1st frequency in the amplitude spectrum of the part (A) in FIG. 5 is subject to calculation (part (B) in FIG. 5). Accordingly, as understood from the part (C) of FIG. 5, the coefficient g (k, n) of the coefficient vector g (k) is added to the element f (m1, k) of the specific basis vector f (k) of the basis matrix F. Each coefficient g (k, n) (that is, the gain of the basis vector f (k)) of the coefficient vector g (k) is excessively large so that the numerical value obtained by multiplying by the frequency frequency component y (m1, n) is approximated. Set to a numeric value. As a result, as illustrated as section σ in FIG. 6, intensity divergence may occur in the acoustic signal SB that emphasizes the sound of the first sound source.

数式(5)の右辺の第２項を最小化することが第１拘束条件に相当する。数式(5)の記号‖ ‖はノルム（例えばフロベニウスノルム）を意味し、係数λは所定の正数に設定される。また、記号/Ｑは、図７から理解される通り、分離行列Ｑの各要素ｑ(m,n)の数値０と数値１とを反転させた各要素/ｑ(m,n)（/ｑ(m,n)＝１−ｑ(m,n)）を配列したＭ行Ｎ列の行列（以下「反転分離行列」という）である。すなわち、第１実施形態の第１拘束条件は、反転分離行列/Ｑの各要素/ｑ(m,n)を基底行列Ｆと係数行列Ｇとの行列積（第１行列Ｄ1）の各要素に乗算した行列（以下「制約対象行列」という）｛/Ｑ.×(ＦＧ)｝のノルムの自乗を抑圧することに相当する。反転分離行列/Ｑでは、第１音源の音響が優勢な各時間周波数成分ｙ(m,n)に対応する要素/ｑ(m,n)が数値０に設定され、第２音源の音響が優勢な各時間周波数成分ｙ(m,n)に対応する要素/ｑ(m,n)が数値１に設定される。したがって、第１実施形態の第１拘束条件は、第１音源に対応する第１行列Ｄ1（＝ＦＧ）のうち観測行列Ｙにて第２音源の音響が優勢な時間周波数成分ｙ(m,n)に対応する要素を抑圧するための条件とも換言され得る。 In the first embodiment, each element of the coefficient matrix G (each coefficient vector g (k)) corresponding to the base matrix F is suppressed from the viewpoint of solving the problems of the comparison 1 and the comparison 2 described above. The non-negative matrix factorization of the observation matrix Y is executed under the constraint condition (hereinafter referred to as “first constraint condition”). Specifically, each matrix (G, H, U) after separation is repetitively updated so that the evaluation function Φ expressed by the following formula (5) is minimized.

Minimizing the second term on the right side of Equation (5) corresponds to the first constraint condition. The symbol ‖ の in Equation (5) means a norm (for example, Frobenius norm), and the coefficient λ is set to a predetermined positive number. Further, as understood from FIG. 7, the symbol / Q represents each element / q (m, n) (/ q) obtained by inverting the numerical value 0 and the numerical value 1 of each element q (m, n) of the separation matrix Q. (m, n) = 1−q (m, n)) are arranged in a matrix of M rows and N columns (hereinafter referred to as “inversion separation matrix”). That is, the first constraint condition of the first embodiment is that each element / q (m, n) of the inverted separation matrix / Q is changed to each element of the matrix product (first matrix D1) of the base matrix F and the coefficient matrix G. This is equivalent to suppressing the square of the norm of the multiplied matrix (hereinafter referred to as “constraint matrix”) {/Q.×(FG)}. In the inverse separation matrix / Q, the element / q (m, n) corresponding to each time frequency component y (m, n) in which the sound of the first sound source is dominant is set to a numerical value 0, and the sound of the second sound source is dominant. The element / q (m, n) corresponding to each time frequency component y (m, n) is set to a numerical value 1. Therefore, the first constraint condition of the first embodiment is that the time frequency component y (m, n) in which the sound of the second sound source is dominant in the observation matrix Y in the first matrix D1 (= FG) corresponding to the first sound source. In other words, it can also be said as a condition for suppressing the element corresponding to.

数式（6）の右辺のうち基底行列Ｆと基底行列Ｈとの相関行列｛Ｆ^TＨ｝のノルム（例えばフロベニウスノルム）‖Ｆ^TＨ‖を含む第３項（罰則項）を最小化することが第２拘束条件に相当する。第２拘束条件の導入により、基底行列Ｆと基底行列Ｈとが共通する状況は回避される。なお、第２拘束条件については特開２０１３−０３３１９６号公報にも詳述されている。 By the way, there is a possibility that the base matrix F and the base matrix H become equivalent only when the evaluation function Φ in Expression (5) is minimized (the sound of the first sound source and the sound of the second sound source are appropriately separated). May not be possible). Considering the above circumstances, in the first embodiment, the similarity between the base matrix F expressing the acoustic characteristics of the first sound source and the base matrix H expressing the acoustic characteristics of the second sound source is reduced (between the two). A constraint condition (hereinafter referred to as “second constraint condition”) is introduced in which the distance is maximized or the correlation between the two is minimized. Specifically, each matrix (G, H, U) after separation is iteratively updated so that the evaluation function Φ expressed by the following formula (6) is minimized.

Equation (6) the third term containing norm (e.g. Frobenius norm) ‖F ^T H‖ of the correlation matrix of the basis matrix F and the basis matrix H {F ^T H} among the right-hand side of minimizing (penalty term) Corresponds to the second constraint condition. By introducing the second constraint condition, a situation where the base matrix F and the base matrix H are common is avoided. Note that the second constraint condition is also described in detail in JP2013-033196A.

The non-negative matrix factorization by the matrix decomposition unit 36 of the first embodiment corresponds to a process for minimizing the evaluation function Φ of Equation (6). From the condition of minimizing the evaluation function Φ, Expression (9) is derived from Expression (7) below. Note that the symbol “.−” in Equations (7) to (9) means division for each element between the matrices.

  The matrix decomposition unit 36 calculates the coefficient matrix G, the base matrix H, and the coefficient matrix U from the observation matrix Y by repeating each update operation of Expression (7) to Expression (9). Specifically, the matrix decomposing unit 36 sets the initial value of the unknown matrix (G, H, U) in the evaluation function Φ to, for example, a random number, and then performs the update operation from Equation (7) to Equation (9). The calculation is repeated (coefficient matrix G, basis matrix H, coefficient matrix U) when the number of iterations reaches a predetermined number. The number of iterations of the update operation is selected, for example, experimentally or statistically so that the evaluation function Φ converges to a predetermined value (for example, zero). As understood from the above description, the matrix decomposing unit 36 of the first embodiment is configured so that each time frequency component y (m, n) corresponding to the element q (m, n) set to the numerical value 1 in the separation matrix Q. ) As a computation target, the first constraint condition for suppressing each element of the coefficient matrix G and the second constraint condition for reducing the similarity between the base matrix F and the base matrix H are satisfied. The non-negative matrix factorization update operation (formula (7) to formula (9)) using the base matrix F of one sound source as teacher information is repeated.

  The waveform synthesizer 38 in FIG. 1 generates the acoustic signal SB using the analysis result (G, H, U) by the matrix decomposition unit 36. Specifically, when the first sound source is designated, the waveform synthesis unit 38 multiplies the base matrix F stored in the storage device 24 and the coefficient matrix G generated by the matrix decomposition unit 36 to thereby generate the acoustic signal SA. The acoustic spectrum of the first sound source is calculated, and the time domain acoustic signal SB is generated by short-time inverse Fourier transform using the amplitude spectrum of each unit section and the phase spectrum of the acoustic signal SA in the unit section. To do. On the other hand, when the second sound source is designated, the waveform synthesizer 38 multiplies the base matrix H generated by the matrix decomposition unit 36 and the coefficient matrix U, thereby the acoustic amplitude spectrogram of the sound of the second sound source in the acoustic signal SA. And the time-domain acoustic signal SB is generated by short-time inverse Fourier transform using the amplitude spectrum of each unit section and the phase spectrum of the acoustic signal SA in the unit section. That is, an acoustic signal SB obtained by separating the acoustic signal SA by the first sound source and the second sound source is generated. The acoustic signal SB generated by the waveform synthesizer 38 is supplied to the sound emitting device 14 and reproduced as a sound wave.

  In the first embodiment described above, the coefficient matrix G and the base matrix are obtained by non-negative matrix factorization in consideration of the first constraint condition for suppressing each element of the coefficient matrix G corresponding to the base matrix F of the first sound source. H and coefficient matrix U are calculated. Therefore, the problem of contrast 2 in which each element of the coefficient matrix G is set to an excessively large numerical value and the intensity of the separated acoustic signal SB diverges (the problem described with reference to FIG. 5) is solved. That is, according to the first embodiment, it is possible to realize high-precision sound source separation as compared with the proportional 1 and the proportional 2. In the first embodiment, the restriction target matrix {/Q.×(FG)} obtained by multiplying each element of the first matrix D1 (= FG) by each element / q (m, n) of the inversion separation matrix / Q. Since the non-negative matrix factorization is executed with the first constraint condition being to reduce the norm of the sound source, the effect of improving the accuracy of sound source separation is particularly remarkable.

  8 to 10 show experimental results of sound source separation according to the first embodiment. In FIG. 8, experimental results of spatial sound source separation using only orientation clustering (hereinafter referred to as “contrast 3”) are also shown as comparison targets, and in FIG. 9, non-negative matrix factorization taking the second constraint condition into account ( An experimental result of sound source separation (hereinafter referred to as “comparative 4”) using only the evaluation function Φ, which omits the second term of Equation (6), is also shown as a comparison target. In FIG. 10, the contrast 2 in which the orientation clustering and the non-negative matrix factorization in consideration of the second constraint condition are continuously executed (the configuration in which the coefficient λ of the evaluation function Φ in Expression (6) is set to 0). The experimental results are also shown as a comparison object with the first embodiment.

  In FIG. 8 to FIG. 10, the signals are shown for a plurality of cases where the respective instruments (Ob. = Oboe, Fl. = Flute, Pf. = Piano, Tb. = Trombone) of the first sound source and the second sound source are different. Signal-to-distortion ratio (SDR), signal-to-interference ratio (SIR), and nonlinear distortion (SAR: Sources to Artifacts Ratio) are expressed as evaluation scales of experimental results. The signal-to-distortion ratio is a comprehensive measure of the accuracy of sound source separation and the quality of the separated signal, the signal-to-interference ratio is a measure of only the accuracy of sound source separation, and the nonlinear distortion is a signal that extends before and after sound source separation. It is a distortion evaluation scale. The larger the numerical value of each evaluation scale, the better the result of sound source separation. In addition, numerical values that exceed the comparison target are written in bold.

  As can be understood from FIGS. 8 to 10, according to the first embodiment, it is good in terms of both the accuracy of the sound source separation and the quality of the separated signal as compared with any of the proportional 2 to the proportional 4. Sound source separation can be realized. Note that the signal-to-interference ratio (only the accuracy of sound source separation) is sometimes seen in FIG. 10 even when the contrast 1 exceeds the first embodiment. However, the signal-to-distortion ratio, which is an overall performance index that reflects both the accuracy of sound source separation and the quality of the separated signal, exceeds 1 in the first embodiment in all cases. Therefore, it can be evaluated that the first embodiment is more advantageous than the comparative 1 from a comprehensive viewpoint.

Second Embodiment
A second embodiment of the present invention will be described below. In addition, about the element which an effect | action and function are the same as that of 1st Embodiment in each form illustrated below, the reference | standard referred by description of 1st Embodiment is diverted, and each detailed description is abbreviate | omitted suitably.

  In contrast 2, the problem that the coefficient g (k, n) of the coefficient vector g (k) becomes an excessively large numerical value is that the time frequency component y (m, n) classified into the target class corresponding to the first sound source. There is a tendency to manifest when the number is small. In consideration of the above tendency, in the second embodiment, the influence of the restriction target matrix {/Q.×(FG)} according to the number of time frequency components y (m, n) classified into the target class (first 1) The influence of the constraint condition) is variably controlled.

  FIG. 11 is a block diagram of the sound processing apparatus 100 according to the second embodiment. As shown in FIG. 11, the arithmetic processing unit 22 of the sound processing apparatus 100 of the second embodiment includes the same elements (frequency analysis unit 32, matrix generation unit 34, matrix decomposition unit 36, waveform synthesis unit as in the first embodiment. 38) and functions as an index calculation unit 42. The index calculation unit 42 calculates an index value ν corresponding to the number of time frequency components y (m, n) classified into the target class (ηth class).

数式(10)の右辺の分子は、Ｍ行Ｎ列の分離行列Ｑのうち数値１に設定された要素ｑ(m,n)の総数（すなわち、目標クラスに分類された時間周波数成分ｙ(m,n)の総数）に相当する。数式(10)の右辺の分母は、数値１の要素ｑ(m,n)の個数を０以上かつ１以下の範囲内に正規化する演算を意味する。図５を参照した前述の説明から理解される通り、目標クラスに分類された時間周波数成分ｙ(m,n)の個数が大きい場合には、係数ベクトルｇ(k)の各係数ｇ(k,n)の過度な増加は発生し難いという傾向がある。したがって、指標値νは、非負値行列因子分解の実行前の観測行列Ｙと実行後の行列（第１行列Ｄ1および第２行列Ｄ2との行列和）との相違を最小化するという基本的な条件（数式(4)）の信頼度の尺度とも換言され得る。 As in the first embodiment, each element q (m, n) corresponding to the time frequency component y (m, n) classified into the target class in the separation matrix Q generated by the matrix generation unit 34 is set to the numerical value 1. Each element q (m, n) corresponding to the time frequency component y (m, n) set and classified into a class other than the target class is set to a numerical value 0. Therefore, the total number of elements q (m, n) set to the numerical value 1 in the separation matrix Q corresponds to the number of time frequency components y (m, n) classified into the target class. In consideration of the above relationship, the index calculation unit 42 of the present embodiment calculates the index value ν by the calculation of the following formula (10).

The numerator on the right side of Equation (10) is the total number of elements q (m, n) set to the numerical value 1 in the separation matrix Q of M rows and N columns (that is, the time frequency component y (m , n)). The denominator on the right side of Equation (10) means an operation that normalizes the number of elements q (m, n) of the numerical value 1 within a range of 0 or more and 1 or less. As understood from the above description with reference to FIG. 5, when the number of time frequency components y (m, n) classified into the target class is large, each coefficient g (k, k, There is a tendency that an excessive increase in n) hardly occurs. Therefore, the index value ν minimizes the difference between the observation matrix Y before execution of non-negative matrix factorization and the matrix after execution (matrix sum of the first matrix D1 and the second matrix D2). In other words, it is a measure of the reliability of the condition (Equation (4)).

数式(11)から理解される通り、指標値νが大きい（目標クラスに分類された時間周波数成分ｙ(m,n)の個数が多い）ほど、非負値行列因子分解（評価関数Φ）における制約対象行列｛/Ｑ.×(ＦＧ)｝のノルムの影響が低減される。行列生成部３４による分離行列Ｑの算定方法や波形合成部３８による音響信号ＳBの生成方法は第１実施形態と同様である。 As understood from the above description, the larger the index value ν (the larger the number of time-frequency components y (m, n) classified into the target class), the more the constraint target matrix {/Q.×(FG)} There is a tendency that the necessity of adding the first constraint for minimizing the norm to the non-negative matrix factorization of the observation matrix Y decreases. In consideration of the above tendency, the matrix decomposition unit 36 of the second embodiment, according to the index value ν calculated by the index calculation unit 42, the constraint target matrix {/Q.× in the update operation of the non-negative matrix factorization. (FG)} is weighted. That is, the degree of influence of the first constraint condition in the update operation of non-negative matrix factorization is variably controlled according to the index value ν. Specifically, the matrix decomposing unit 36 replaces the above-described equation (6) with the coefficient of the update operation selected so that the evaluation function Φ expressed by the following equation (11) is minimized. A matrix G, a base matrix H, and a coefficient matrix U are calculated.

As understood from Equation (11), the larger the index value ν (the larger the number of time-frequency components y (m, n) classified into the target class), the more the constraints on the non-negative matrix factorization (the evaluation function Φ) The influence of the norm of the target matrix {/Q.×(FG)} is reduced. The calculation method of the separation matrix Q by the matrix generation unit 34 and the generation method of the acoustic signal SB by the waveform synthesis unit 38 are the same as in the first embodiment.

  In the second embodiment, the same effect as in the first embodiment is realized. In the second embodiment, the influence of the constraint target matrix {/Q.×(FG)} in the non-negative matrix factorization (evaluation function Φ) is variably controlled according to the index value ν. The effect of improving accuracy is particularly remarkable.

数式(12)の評価関数Φを適用した構成でも、指標値νが大きいほど制約対象行列｛/Ｑ.×(ＦＧ)｝の影響が低減されるから、第２実施形態と同様の効果が実現される。 <Modification of Second Embodiment>
(1) The relationship between the index value ν and the influence of the constraint target matrix {/Q.×(FG)} is not limited to the above example (Formula (11)). For example, it is also possible to calculate the coefficient matrix G, the base matrix H, and the coefficient matrix U by repetition of the update operation selected so that the evaluation function Φ of the formula (12) exemplified below is minimized.

Even in the configuration in which the evaluation function Φ in Expression (12) is applied, the larger the index value ν, the smaller the influence of the restriction target matrix {/Q.×(FG)}. Therefore, the same effect as the second embodiment is realized. Is done.

(2) In the above formula (10), the index value ν is calculated so as to be proportional to the total number of elements q (m, n) set to the numerical value 1 in the separation matrix Q. The method for calculating the value ν is not limited to the above examples. For example, it is also possible to calculate the index value ν by the calculation of the following formula (13).

  Equation (13) is a sigmoid function illustrated in FIG. 12, and the coefficient α corresponds to the gain of the sigmoid function. The coefficient β is a coefficient that defines the inflection point of the sigmoid function, as understood from FIG. 12, and depends on the maximum value (M × N) of the total number of elements q (m, n) set to the numerical value 1. Is set to a numerical value (for example, a numerical value of about 80% of the maximum value). As understood from the equation (13) and FIG. 12, the index value ν changes nonlinearly according to the total number of elements q (m, n) set to the numerical value 1.

  As understood from the above description, the matrix decomposing unit 36 of the second embodiment corresponds to the number of numerical values 1 in the separation matrix Q (the number of time frequency components y (m, n) classified into the target class). Expressed as an element that variably controls the influence of the constraint target matrix {/Q.×(FG)} in the update operation of the non-negative matrix factorization, the index value ν and the constraint target matrix {/Q.×(FG)} There is no limitation on the relationship between the values and the calculation method of the index value ν.

(3) In the above description, the index value ν is calculated for N unit intervals. However, after calculating the index value ν for each unit interval, the influence of the constraint target matrix {/Q.×(FG)} It is also possible to control for each unit section. For example, as shown in the following formula (14), the index value ν (n) is calculated for each unit section, and the evaluation function of the following formula (15) is applied using the index value ν (n) for each unit section. It is also possible to apply Φ to non-negative matrix factorization.

Symbol y _n of the equation (15) means the amplitude spectrum of the acoustic signal SA of the n-th unit interval (M pieces of time-frequency component y (1, n) ~y (M, a sequence of n)). Symbol g _n is a sequence of K coefficients g (1, n) to g (K, n) corresponding to the nth unit interval in the coefficient matrix G, and symbol u _n is the coefficient matrix U. Of these, a series of K coefficients u (1, n) to u (K, n) corresponding to the nth unit interval. The symbol Q _n is a sequence of elements (q (1, n) to q (M, n)) in the nth column of the separation matrix Q.

(4) In the above description, the influence of the restriction target matrix {/Q.×(FG)} is controlled in a multistage manner according to the index value ν, but the restriction target matrix {/Q.×(FG)} ( It is also possible to control whether or not to add the first constraint condition) to the non-negative matrix factorization according to the index value ν. Specifically, when the index value ν is below a predetermined threshold value νTH (when the number of time-frequency components y (m, n) classified into the target class is small), the matrix decomposition unit 36 is the same as in the first embodiment. Similarly, the non-negative matrix factorization taking the first constraint condition into consideration is executed by repeating the update operation that minimizes the evaluation function Φ of Expression (6). On the other hand, when the index value ν exceeds the threshold value νTH (when the number of time-frequency components y (m, n) classified into the target class is large), the matrix decomposition unit 36 sets the coefficient λ of Equation (6) to 0. By repeating the update operation that minimizes the set evaluation function Φ, non-negative matrix factorization with the first constraint condition released is executed. With the above configuration, the same effect as in the second embodiment is realized. Further, since the influence (presence / absence) of the restriction target matrix {/Q.×(FG)} is controlled in a binary manner according to the index value ν, the process is simplified as compared with the configuration controlled in multiple steps. There is also an advantage of being.

<Third Embodiment>
In the first embodiment, the element (constraint) corresponding to the time frequency component y (m, n) in which the sound of the second sound source is dominant in the observation matrix Y in the first matrix D1 (= FG) corresponding to the first sound source. An example of non-negative matrix factorization in which suppression of the target matrix {/Q.×(FG)} norm) is used as the first constraint condition is illustrated. The matrix decomposition unit 36 of the third embodiment suppresses temporal variation of the element corresponding to the time frequency component y (m, n) in which the sound of the second sound source is dominant in the observation matrix Y in the first matrix D1. The non-negative matrix factorization of the observation matrix Y is executed under the constraint condition (hereinafter referred to as “third constraint condition”).

Specifically, the matrix decomposing unit 36 determines each matrix (G, H, U) unknown by repetition of the update operation selected so that the evaluation function Φ expressed by the following equation (16) is minimized. Is calculated. The coefficient ε in Expression (16) is set to a predetermined positive number.

Minimizing the third term on the right side of Equation (16) corresponds to the third constraint condition. The symbol {f (m, k) g (k, n)} corresponds to the mth frequency on the frequency axis in the basis vector f (k) (amplitude spectrum of the kth component) of the acoustic signal SA. The element f (m, n) to be multiplied by the element g (k, n) corresponding to the nth unit interval on the time axis of the coefficient vector g (k) corresponding to the base vector f (k) Value. Therefore, the symbol Σ _k f (m, k) g (k, n) in Expression (16) is the nth unit interval and mth frequency of the sound of the first sound source in the acoustic signal SA. This corresponds to the intensity (amplitude) of the corresponding time frequency component d (m, n). In other words, the parentheses in the third term of Equation (16) are the time frequency component d (m, n) of the nth unit section of the sound of the first sound source of the acoustic signal SA and the immediately preceding (n-1). This corresponds to an intensity difference (error) from the time frequency component d (m, n-1) of the) th unit interval. In addition, the multiplication of the inverse separation matrix / Q element / q (m, n) in Equation (16) is the time frequency component y (m, n) (separation matrix) in which the sound of the second sound source is dominant in the observation matrix Y. This means that the time frequency component d (m, n) corresponding to the element q (m, n)) set to the numerical value 0 in Q is extracted from the first matrix D1.

  As understood from the above description, the element q (m, n) (inverted separation matrix) of the first matrix D1, which is the matrix product of the base matrix F and the coefficient matrix G, set to the numerical value 0 in the separation matrix Q. The intensity of the time-frequency component d (m, n) corresponding to the element / q (m, n)) set to the numerical value 1 by / Q approximates each other in each unit interval (intensity difference) Is smaller), the third term on the right side of Equation (16) becomes a smaller numerical value. That is, the third constraint condition is expressed as a condition for suppressing temporal variation of the time frequency component d (m, n) corresponding to the element q (m, n) having a numerical value 0 in the first matrix D1. The

数式(17)の記号Ｇ^(-)は、直前の更新後の係数行列Ｇの各係数ベクトルｇ(k)を１列分だけ右方に移動させた行列を意味し、記号Ｇ⁽⁺⁾は、直前の更新後の係数行列Ｇの各係数ベクトルｇ(k)を１列分だけ左方に移動させた行列を意味する。行列生成部３４による分離行列Ｑの算定方法や波形合成部３８による音響信号ＳBの生成方法は第１実施形態と同様である。 The matrix decomposition unit 36 of the third embodiment calculates the coefficient matrix G by repeating the update operation of the following equation (17) selected so that the evaluation function Φ of the equation (16) is minimized. The calculation formula for the update calculation for calculating the base matrix H and the coefficient matrix U is the same as that in the first embodiment (Formulas (8) and (9)).

The symbol G ^{(−) in} Equation (17) means a matrix obtained by moving each coefficient vector g (k ⁾ of the immediately updated coefficient matrix G by one column to the right, and the symbol G ⁽⁺⁾ is Means a matrix in which each coefficient vector g (k) of the immediately updated coefficient matrix G is moved to the left by one column. The calculation method of the separation matrix Q by the matrix generation unit 34 and the generation method of the acoustic signal SB by the waveform synthesis unit 38 are the same as in the first embodiment.

  In the third embodiment described above, the element corresponding to the time frequency component y (m, n) in which the sound of the second sound source is dominant in the observation matrix Y in the first matrix D1 is continuous on the time axis. The coefficient matrix G, the base matrix H, and the coefficient matrix U are calculated by non-negative matrix factorization taking the third constraint condition into consideration. Accordingly, the problem of the proportional 2 in which each element of the coefficient matrix G is set to an excessively large numerical value (that is, a numerical value that is extremely deviated from the numerical value of the immediately preceding unit section) and the intensity of the separated acoustic signal SB diverges (FIG. 5) is eliminated. That is, according to the third embodiment, as in the first embodiment, it is possible to realize high-accuracy sound source separation as compared with the proportional 1 and the proportional 2.

<Modification>
Each of the above forms can be variously modified. Specific modifications are exemplified below. Two or more aspects arbitrarily selected from the following examples can be appropriately combined.

(1) In each of the above-described embodiments, the separation matrix Q is generated using orientation clustering, but the method of generating the separation matrix Q by the matrix generation unit 34 is arbitrary. For example, the method exemplified below can be adopted for generating the separation matrix Q.

[Aspect 1]
The distribution of each frequency component of the acoustic signal SA is displayed in the localization-frequency region in which the localization axis (sound image localization direction) and the frequency axis are set, and the specified region including the localization range and frequency band of the first sound source is used, for example For example, Japanese Patent Application Laid-Open No. 2012-249048 discloses a configuration in which it is set in a localization-frequency region in response to an instruction from a person. The matrix generation unit 34 displays the distribution of each frequency component of the acoustic signal SA in the localization-frequency domain on the display device, receives an instruction from the designated region from the user, and designates the designated region in the observation matrix Y of the acoustic signal SA. A separation matrix Q is generated in which the element q (m, n) corresponding to each time frequency component y (m, n) is set to the numerical value 1 and the remaining elements q (m, n) are set to the numerical value 0. To do.

[Aspect 2]
The matrix generation unit 34 performs noise suppression processing on the acoustic signal SA. Specifically, as expressed by the following equation (18), spectral subtraction (subtraction of the estimated noise component ζ (m, n) from each time frequency component x (m, n) of the acoustic signal SA in the frequency domain ( Generalized spectral subtraction) is performed. The time frequency component x (m, n) is, for example, the average or total of the time frequency components x1 (m, n) to xJ (m, n) (or the time frequency components x1 (m, n) to xJ (m, n) The estimated noise component ζ (m, n) is, for example, a noise component (typically stationary noise) estimated from the noise interval (non-voice interval) of the acoustic signal SA.

数式(19)から理解される通り、推定雑音成分ζ(m,n)と比較して優勢な各時間周波数成分ｘ(m,n)に対応する要素ｑ(m,n)は数値１（維持値）に設定され、残余の各要素ｑ(m,n)は数値０（抑圧値）に設定される。行列分解部３６は、数式(18)の雑音抑圧処理後の各時間周波数成分ｙ(m,n)を配列した観測行列Ｙに対し、行列生成部３４が生成した分離行列Ｑを適用した非負値行列因子分解を実行する。 In the formula (18), symbol a is a subtraction coefficient, and symbol b is a flooring coefficient. Further, the power exponent p of the time frequency component x (m, n) and the estimated noise component ζ (m, n) is set to a predetermined value (typically 2 or 4). The matrix generation unit 34 calculates each element q (m, n) of the separation matrix Q according to the result of the noise suppression processing of Expression (18) as expressed by Expression (19) below.

As understood from the equation (19), the element q (m, n) corresponding to each temporal frequency component x (m, n) that is dominant compared to the estimated noise component ζ (m, n) is a numerical value 1 (maintained). Value), and each remaining element q (m, n) is set to a numerical value 0 (suppression value). The matrix decomposition unit 36 applies the separation matrix Q generated by the matrix generation unit 34 to the observation matrix Y in which the time frequency components y (m, n) after the noise suppression processing of Expression (18) are arranged. Perform matrix factorization.

[Aspect 3]
The matrix decomposition unit 36 performs a plurality of stages of non-negative matrix factorization including the first process and the second process. The first process calculates the coefficient matrix G, the base matrix H, and the coefficient matrix U by performing non-negative matrix factorization on which the separation matrix Q is not applied to the observation matrix Y. The matrix generation unit 34 calculates each element q (m, n) of the separation matrix Q according to the result of the first process, as expressed by the following formula (20).

The symbol {Σ _k h (m, k) u (k, n)} / y (m, n) in Equation (20) is the sound (Σ _k h) of the second sound source in the time frequency component y (m, n). (m, k) u (k, n)) corresponds to an index of the degree of predominance over the sound of the first sound source. Symbol TH in equation (20) is a predetermined threshold value. Acoustic second sound source predominates to the extent as is understood, the index _{{Σ k h (m, k} ) u (k, n)} / y (m, n) exceeds the threshold value TH from the formula (20) The element q (m, n) corresponding to each time frequency component y (m, n) is set to a numerical value 1 and the remaining elements q (m, n) are set to a numerical value 0. The matrix decomposition unit 36 uses the second matrix D2 (D2 = HU) specified from the result of the first processing as a new observation matrix Y, and in the same manner as in the above embodiments, the separation matrix Q generated by the matrix generation unit 34. The non-negative matrix factorization to which is applied is executed as the second process. Incidentally, the index _{{Σ k h (m, k} ) u (k, n)} / y (m, n) numerical elements q of (m, n) is set as the separation matrix (soft mask) generating a Q Is also possible.

(2) The numerical value of each element q (m, n) of the separation matrix Q (inverted separation matrix / each element / q (m, n)) is limited to the example (0,1) in each of the above-described forms. Not. That is, each element q (m, n) of the separation matrix Q has a maintenance value γ1 that maintains the time frequency component y (m, n) or a suppression value γ0 that suppresses the time frequency component y (m, n). Is set. The numerical value 1 of the element q (m, n) in each of the above-described forms is a typical example of the maintenance value γ1, and the numerical value 0 of each element q (m, n) is a typical example of the suppression value γ0.

(3) The generation method of the base matrix F used as teacher information is arbitrary. For example, the base matrix F is generated by non-negative matrix factorization for the acoustic signal of the first sound source recorded in advance. Further, since the base matrix F is composed of K amplitude spectra assumed for the sound of the first sound source, for example, an average amplitude spectrum of the sound of the first sound source is calculated for each of the K pitches. It is also possible to generate the base matrix F by arranging K amplitude spectra corresponding to each pitch.

(4) In each of the above embodiments, the non-negative matrix factorization using the Frobenius norm is exemplified, but the distance criterion applied to the non-negative matrix factorization is not limited to the Frobenius norm. Specifically, a known distance criterion such as a Kullback-Leibler pseudorange or divergence is arbitrarily adopted. In addition, non-negative matrix factorization using sparseness constraints is also employed.

(5) In each of the above-described embodiments, the acoustic signal SB obtained by extracting the sound of one of the first sound source and the second sound source is generated. However, the sound signal SB obtained by extracting the sound of the first sound source and the sound of the second sound source are extracted. It is also possible for the waveform synthesizer 38 to generate both of the acoustic signals SB that have been performed in parallel. For example, a configuration is adopted in which different acoustic processing (for example, effect application) is performed on each of the acoustic signal SB of the first sound source and the acoustic signal SB of the second sound source and then added.

(6) In each of the above-described embodiments, the sound signal SB of the first sound source is generated by multiplying the base matrix F and the coefficient matrix G, and the sound signal SB of the second sound source is generated by multiplying the base matrix H and the coefficient matrix U. However, it is also possible to generate a filter for processing the acoustic signal SA using each matrix generated by the matrix decomposition unit 36. For example, a configuration for generating a filter (for example, a Wiener filter) for suppressing or enhancing the sound of the first sound source from the matrix product (first matrix D1) of the base matrix F and the coefficient matrix G and acting on the acoustic signal SA A configuration is adopted in which a filter for suppressing or enhancing the sound of the second sound source is generated from the matrix product (second matrix D2) of the base matrix H and the coefficient matrix U and applied to the acoustic signal SA.

(7) Generation of the acoustic signal SB (waveform synthesis unit 38) using the calculation result (G, H, U) by the matrix decomposition unit 36 may be omitted. For example, the present invention can be implemented as an acoustic processing apparatus 100 that calculates an unknown matrix (G, H, U) by non-negative matrix factorization with respect to the observation matrix Y of the acoustic signal SA. In addition, the sound processing apparatus 100 can be realized by a server device that communicates with a terminal device such as a mobile phone. For example, the acoustic processing device 100 generates an acoustic signal SB from the acoustic signal SA received from the terminal device and transmits the acoustic signal SB to the terminal device. In the configuration in which the observation matrix Y of the acoustic signal SA and each observation vector VX (m, n) are received from the terminal device (for example, the configuration in which the terminal device includes the frequency analysis unit 32), the acoustic processing device 100 to the frequency analysis unit 32. Is omitted, and the waveform synthesizing unit 38 is omitted from the acoustic processing device 100 in a configuration in which the calculation result by the matrix decomposing unit 36 is transmitted to the terminal device (for example, a configuration in which the terminal device includes the waveform synthesizing unit 38).

DESCRIPTION OF SYMBOLS 100 ... Sound processing device, 12 ... Signal supply device, 14 ... Sound emission device, 22 ... Arithmetic processing device, 24 ... Memory | storage device, 32 ... Frequency analysis part, 34 ... Matrix generation part, 36 ... ... matrix decomposition unit, 38 ... waveform synthesis unit, 42 ... index calculation unit.

Claims (5)

An element corresponding to each time frequency component of the sound signal representing the mixed sound of the sound of the plurality of sound sources, and an element corresponding to each time frequency component in which the sound of the first sound source is dominant among the plurality of sound sources is the time frequency Matrix generation means for generating a separation matrix that is set to a maintenance value that maintains a component and an element corresponding to each remaining time frequency component is set to a suppression value that suppresses the time frequency component;
The first basis matrix including a plurality of basis vectors representing the spectrum of each component of the sound of the first sound source is set as teaching information, with each time frequency component corresponding to the element set as the maintenance value in the separation matrix as a calculation target. Including a plurality of coefficient vectors corresponding to each basis vector of the first basis matrix from an observation matrix in which each time frequency component of the acoustic signal is arranged by repeating the update operation of the non-negative matrix factorization used as Corresponding to a first coefficient matrix, a second basis matrix including a plurality of basis vectors representing the spectrum of each component of the sound of the second sound source different from the first sound source, and each basis vector of the second basis matrix Matrix decomposition means for calculating a second coefficient matrix including a plurality of coefficient vectors,
The matrix processing means executes a non-negative matrix factorization with a constraint condition of suppressing each element of the first coefficient matrix. The matrix decomposition means multiplies each element of a matrix product of the first base matrix and the first coefficient matrix by each element of the inverted separation matrix obtained by inverting the maintenance value and the suppression value for each element of the separation matrix. The acoustic processing device according to claim 1, wherein non-negative matrix factorization is performed with the constraint that reducing the norm of the restriction target matrix. The acoustic processing apparatus according to claim 1, wherein the matrix decomposition unit variably controls the degree of influence of the constraint condition in the update calculation according to the number of maintenance values in the separation matrix. The matrix decomposition means performs a non-negative matrix factorization under the constraint when an index value corresponding to the number of maintenance values in the separation matrix exceeds a threshold value, and the index value falls below the threshold value The acoustic processing apparatus according to claim 1, wherein the constraint condition is released. An element corresponding to each time frequency component of the sound signal representing the mixed sound of the sound of the plurality of sound sources, and an element corresponding to each time frequency component in which the sound of the first sound source is dominant among the plurality of sound sources is the time frequency Matrix generation means for generating a separation matrix that is set to a maintenance value that maintains a component and an element corresponding to each remaining time frequency component is set to a suppression value that suppresses the time frequency component;
The first basis matrix including a plurality of basis vectors representing the spectrum of each component of the sound of the first sound source is set as teaching information, with each time frequency component corresponding to the element set as the maintenance value in the separation matrix as a calculation target. Including a plurality of coefficient vectors corresponding to each basis vector of the first basis matrix from an observation matrix in which each time frequency component of the acoustic signal is arranged by repeating the update operation of the non-negative matrix factorization used as Corresponding to a first coefficient matrix, a second basis matrix including a plurality of basis vectors representing the spectrum of each component of the sound of the second sound source different from the first sound source, and each basis vector of the second basis matrix Matrix decomposition means for calculating a second coefficient matrix including a plurality of coefficient vectors,
The matrix decomposing means suppresses temporal variation of a time frequency component corresponding to an element set as a suppression value in the separation matrix among matrix products of the first base matrix and the first coefficient matrix. A sound processing device that performs non-negative matrix factorization with as a constraint.

Images