KR101076678B1 - Apparatus and method for separating source by using advanced orthogonality on mutlichannel spectrogram - Google Patents

Abstract

The present invention relates to a technique using an improved orthogonality between different signals on a spectrogram as a kind of a multichannel speech separation algorithm for realizing a noise recognition system that is robust to noise. The techniques view multiple voices as single voices or single voices as multiple voices, based on orthogonality in the time domain or assuming windowed disjoint orthogonality (WDO) for each time-frequency point on the spectrogram. There was often a case where false speech separation was regarded. Accordingly, the present invention is characterized in that voice separation is performed by spreading each channel signal on a spectrogram by using an improved orthogonality in which time-frequency points of appropriate units are summed on a time axis and a frequency axis. According to the present invention, the time domain used in the conventional ICA has been extended to the domain on the spectrogram, and can be used more realistically than the excessive orthogonality assumed in DUET or DESPRIT, thereby increasing the overall speech separation success rate.

Speech Separation, Multichannel, Spectrogram, Orthogonality, Matrix Decomposition

Description

Speech separation device and method using improved orthogonality on multi-channel spectrogram {APPARATUS AND METHOD FOR SEPARATING SOURCE BY USING ADVANCED ORTHOGONALITY ON MUTLICHANNEL SPECTROGRAM}

BACKGROUND OF THE INVENTION 1. Field of the Invention The present invention relates to a technique for performing speech separation. In particular, the present invention relates to a multi-channel speech separation algorithm for implementing a noise recognition speech recognition system. An apparatus and method for speech separation using enhanced orthogonality on a multi-channel spectrogram suitable for performing speech separation using orthogonality of time-frequency points.

The basic principle of speech separation is to use the difference component between speech signals mixed in the microphone input. Independent Component Analysis (ICA) is one of the classic speech separation techniques. This method separates the speech signal by repeatedly estimating the entropy and Gaussianity of the microphone's Mixture Inputs. This method has been studied to produce various variations until now, but if Kurtosis incorrectly estimates rain rain, it does not return satisfactory speech separation results and it takes a long time to calculate because iterative estimation method is used. Has a problem. Binary Time-Frequency Masking presented by O. Yilmaz's paper (Blind Separation of Speech Mixtures via Time-Frequency Masking-IEEE Transactions on Signal Processing, 52 (7), 1830-1847, Jul 2004). In the case of the DUET (Degenerate Unmixing Estimation Technique), voice separation is attempted by taking the WDO assumption as shown in Equation 1 below for each point on the spectrogram.

Speech separation using the assumption of Equation 1 is superior to speech separation based on ICA when the difference of spatial information between speech signals is large.

In the DUET using the binary time-frequency masking technique among the conventional speech separation techniques operating as described above, since the sparse signal model is rarely satisfied on the spectrogram, the above assumption is practical. It is difficult to use in the environment. An extension of the above method for multichannel input is described in T. Melia's paper (Underdetermined Blind Source Separation in Echoic Environments Using DESPRIT-EURASIP Journal on Advances in Signal Processing Vol 2007, 1-19). Since the method is also based on the assumption of Equation 1, there is a problem in that there is a limit in speech separation.

Accordingly, the present invention provides an apparatus and method for speech separation using improved orthogonality on a multi-channel spectrogram capable of performing speech separation using improved orthogonality between signals on a spectrogram.

In addition, the present invention provides enhanced orthogonality on a multi-channel spectrogram that can perform speech separation using the orthogonality of each channel signal spread on the spectrogram and the sum of time-frequency points of appropriate units on the time axis and the frequency axis. Provided are a voice separation device and method.

The present invention also provides a matrix of complex energy sums of time-frequency points in an appropriate unit on a multi-channel spectrogram using improved orthogonality between different signals for speech separation, and then, through matrix decomposition. Apparatus and method for improving speech using orthogonality on multichannel spectrograms that obtain a separating matrix and multiply the obtained separation matrix by multi-channel mixed inputs to restore estimated speech signals through permutation To provide.

According to an embodiment of the present invention, a multichannel microphone for receiving a plurality of voice signals and outputting time-domain multichannel data and an orthogonality between different signals for separation of voice signals transmitted from the multichannel microphones are used. The complex sum of time-frequency points of a predetermined unit on a multi-channel spectrogram is represented by a matrix, and then a separation matrix is calculated through a matrix decomposition method, and the estimated separation matrix is multiplied by the multi-channel mixed input. Voice signals.

According to an embodiment of the present invention, a method is set on a multichannel spectrogram using orthogonality between different signals for separation of a voice signal from time domain multichannel data for a plurality of voice signals input through a multichannel microphone. A process of representing a sum of complex energy of time-frequency points in units as a matrix, a separation matrix through a matrix decomposition method for the matrix, and multiplying the calculated separation matrix by a multichannel mixed input to obtain estimated speech signals. Restoration process is included.

In the present invention, the effects obtained by the representative ones of the disclosed inventions will be briefly described as follows.

The present invention is to perform speech separation using improved orthogonality between signals on a spectrogram, and the three enhanced orthogonalities assumed in the embodiments of the present invention are non-gaussianity and nigentropy (used by ICA). Computational Complexity can be lowered than iterative estimation using negentropy, and it can be used more realistically than unreasonable orthogonality assumed in DUET or DESPRIT, thereby increasing overall speech separation success rate significantly. There is.

Hereinafter, the operating principle of the present invention will be described in detail with reference to the accompanying drawings. In the following description of the present invention, if it is determined that a detailed description of a known function or configuration may unnecessarily obscure the subject matter of the present invention, the detailed description thereof will be omitted. The following terms are defined in consideration of the functions of the present invention, and may be changed according to the intentions or customs of the user, the operator, and the like. Therefore, the definition should be based on the contents throughout this specification.

According to the present invention, a complex energy sum of time-frequency points of an appropriate unit is represented as a matrix on a multi-channel spectrogram using improved orthogonality between different signals for speech separation, and then a matrix decomposition method. (Separating Matrix) is obtained, and the obtained separation matrix is multiplied by the multichannel mixed input to recover the estimated speech signals.

1 is a view showing the structure of a voice separation device according to an embodiment of the present invention.

Referring to FIG. 1, a voice separation device receives a voice signal input from a multi-channel microphone 100 and a multi-channel microphone 100 receiving voices from a plurality of speakers, and based on a multi-channel voice separation algorithm, respectively. A voice separator 110 that performs voice separation for the speaker, a speaker 120 for outputting respective voices separated from the voice separator 110, and a voice for each voice output from the speaker 120. And a voice recognizer 130 that performs voice recognition.

2 is a block diagram showing the structure of a speech separator on a multi-channel spectrogram according to an embodiment of the present invention.

Referring to FIG. 2, the voice separator 110 includes a microphone input processor 200, a separation matrix calculator 202, a voice estimator 204, and the like.

As shown in FIG. 3, the microphone input processor 200 includes a time-frequency domain converter 300, a block splitter 302, a block adder 304, and the like, and a time domain of the multichannel microphone 100. (Time Domain) Converts Mix Inputs into data in the form of speech separation. The multi-channel microphone 100 may be a time domain multi-channel microphone, through which voices from a plurality of speakers are input.

And it is assumed that this is M microphone mixed input {xi (t) | i∈ [1, M]}. The mixed input is input to the multichannel microphone input {Xi (τ, k) | i∈ [1, M], τ∈ [1, T on the spectrogram by the time-frequency domain converter 300 in the microphone input processing unit 200. ], k∈ [0, K-1]}. Here, K means the size of the window and the frequency axis on the spectrogram when converting to the time-frequency domain. T is the magnitude of the time base on the spectrogram.

Subsequently, in the block splitter 302, all points on the spectrogram are divided into block units of Topt and Kopt having appropriate sizes, thereby obtaining a multichannel spectrogram in units of blocks. Meanwhile, Topt and Kopt represent preset optimal T and K values, which will be described later. Thereafter, the block splitter 302 transmits the multichannel spectrogram in units of blocks to the separation matrix calculator 202 and the voice estimator 204.

 The block adder 304 adds complex values of a block-by-block multichannel spectrogram to all frequency and time axes of the block, and sends multi-channel data for the separation matrix calculated as the result to the separation matrix calculator 202. To pass.

First, the principle and application of the matrix separation used on the separation matrix calculation unit 202 will be described. Speech separation algorithms that view signals in the form of a convolutional mixture in the time domain, including ICA, have a one-dimensional shape. Since it is possible to deal with orthogonality in the time axis, it is advantageous in that not only the time axis but also the orthogonality of the frequency axis can be dealt with.

In order to examine the orthogonality between different signals in the time-frequency domain, which are dealt with in the embodiment of the present invention, N time-domain signals {si (t) | i∈ [1, N]} and spectrograms {Si are used. (τ, k) | i∈ [1, N], τ∈ [1, T], k∈ [0, K-1]}. The spectrogram Si (τ, k) of the signal si (t) can be expressed by Equation 2 by taking a window function having a value of 0 except [0, K-1].

Equation (2) means taking K points near τ in the time domain and taking a window function to represent the signal on the spectrogram. Assuming that all signals do not have a direct current (DC) component, any i-th signal si (t) may be represented as follows.

From Equation 3, Equation 4 below may be derived between different i th signals and j th signals.

Equation 4 may be changed to a time-frequency domain relation as shown in Equation 5 by Generalized Parseval's Theorem.

In this way, the following Equation 6 can be reached through Equation 5, which is defined as improved orthogonality with respect to the frequency axis assuming the first orthogonality.

In addition, when Equation 5 is added to both sides of time, Equation 7 may be expressed.

When the right side of Equation 7 is taken to T → ∞ as shown in Equation 4, orthogonality can be derived as shown in Equation 8 below.

Equation (8) has an important meaning that the sum of products between spectrograms of two different signals is very small when K and T are larger than a predetermined value. Here, Equation 8 is assumed as the second orthogonality, and is defined as improved orthogonality with respect to the time-frequency axis. Finally, the following Equation (9) is assumed as the third orthogonality, and is expressed as follows.

Three assumptions about orthogonality are made, and the nature of these assumptions can be used more realistically than the WDO assumptions of DUET and DESPRIT. In the following, we define three orthogonalities as enhanced orthogonality. In fact, since T → ∞ does not exist, the orthogonality assumed in <Equation 6>, <Equation 8>, and <Equation 9> should be corrected through T and K having a value larger than the preset value. . At this time, the optimal T and K setting values are indicated as Topt and Kopt, respectively.

Using the improved orthogonality described above, the simplest case is N = 2, M = 2, that is, the speech separation process in the case where the voice signals are mixed and input from two microphones on two different time domains. Shall be. The mixing model of the time-frequency domain is expressed by Equation 10 below.

Here, αi and δi represent attenuation and delay according to the difference in arrival distance and the difference in arrival time of the i-th signal between adjacent microphones. The inverse of the mixing matrix A is a separating matrix, denoted B. In order to obtain the separation matrix B, the equation is developed using the improved orthogonality as shown in Equation 11 below.

In Equation 11, the improved orthogonality with respect to the time-frequency axis is used as the expression representing three improved orthogonalities. In this case, T _opt = 1 is improved orthogonality to the frequency axis, and K _opt = 1 is improved orthogonality to the time axis. This notation also applies to the following equation. In Equation 11, B is obtained through matrix decomposition of RX. If Equation 10 is expanded to M> 2, N> 2, it can be expressed as Equation 12 below.

Similarly, Equation 11 may be expressed as Equation 13 below.

In Equation 13, the matrix B is a non-square matrix of size N ㅧ M, and since the values of Xi (τ, k) are complex, the general matrix decomposition cannot be applied. . Accordingly, in the embodiment of the present invention, a strong uncorrelating transform (SUT) is used to solve B. This transformation method assumes a complex random vector S (k) having an uncorrelated and derives two properties from the following equation (14).

The first equation of Equation 14 represents general covariance, and no information for estimating s (k) can be obtained from the matrix I. On the other hand, the second equation is defined as pseudo covariance, and the elements of Λ have only the diagonal component {λ _{i | i} = [1, p])} with only real values. Through this property, a separation matrix is obtained. The conversion method is applied to an embodiment of the present invention.

From the above-mentioned contents, the configuration of the separation matrix calculation unit 202 is shown as follows.

4 is a block diagram illustrating a structure of a separation matrix calculator according to an exemplary embodiment of the present invention. The separation matrix calculator 202 includes a whitening matrix calculator 400, a pseudo covariance matrix calculator 402, and a separation. A matrix calculator 404 and the like.

First, the whitening matrix calculator 400 obtains a whitening matrix C of the multichannel data for the separation matrix calculation from the multichannel data transferred from the microphone input processor 200. This is obtained by removing the DC component from the result transmitted from the microphone input processing unit 200 and then obtaining a covariance matrix R _X as shown in Equation 11 and then obtaining the inverse of this square root.

Next, the pseudo covariance matrix calculator 402 multiplies the whitening matrix C obtained through the whitening matrix calculator 400 of the preceding stage by the multichannel spectrogram in units of blocks transferred from the microphone input processor 200, and then adds the pseudo covariance matrix. Obtain Subsequently, a pseudo singular value decomposition (SVD) (eg, Symmetric SVD or Takagi Factorization) is applied to the obtained pseudocovariance matrix. The pseudo SVD is a method for decomposing the pseudo covariance matrix, which can be expressed as Equation 15 below.

As shown in Equation 15, the U matrix is calculated through decomposition of the pseudocovariance matrix. Finally, the separation matrix calculator 404 transmits the whitening matrix C and the U matrix transmitted through the pseudocovariance matrix calculator 402. Through the separation matrix B can be calculated as shown in Equation 16 below.

5 is a block diagram illustrating a structure of a speech estimator according to an exemplary embodiment of the present invention.

Referring to FIG. 5, the speech estimator 204 includes a block estimator 500, a block canceler 502, a time domain transformer 504, and the like. Accordingly, the block estimator 500 multiplies the separation matrix B obtained through the separation matrix calculator 202 and the multichannel spectrogram in units of blocks transferred from the microphone input processor 200. In this way, the block release unit 502 releases the block by combining all the blocks of the result value obtained through the block estimation unit 500 in the original order. In other words, it is to restore to the original state divided by the block unit of T _opt and K _opt .

In this case, there may be a case in which the order of the restored signals is not correct among the blocks. To solve this problem, the eigenvalue, energy spectrum, and estimated angle of arrival of the estimated source (DOA) of each block may be solved. The difference is calculated at a preset ratio and the restored blocks are sorted in order, which may be referred to as a permutation process in the block release unit 502.

The time domain converter 504 receives the output value of the block released through the block release unit 502, and converts the output value into time domain signals to output a voice separation result.

6 is a flowchart illustrating an operation procedure of a voice separator according to an embodiment of the present invention.

Referring to FIG. 6, in operation 600, the voice separator 110 receives a mixed voice signal, that is, time domain multi-channel data, which is simultaneously input from a plurality of speakers through the multi-channel microphone 100. In operation 602, the microphone input processing unit 200 in the voice separator 110 converts the input time domain multi-channel data into multi-channel data for voice separation and multi-channel spectrogram in units of blocks and outputs the multi-channel spectrogram.

In operation 604, the separation matrix calculator 202 obtains a whitening matrix from the multi-channel data received from the microphone input processor 200 using orthogonality between different signals for speech separation, and calculates the whitening matrix in block units. After multiplying multi-channel spectrograms to obtain a pseudo covariance matrix, the pseudo covariance matrix is decomposed through pseudo singular value decomposition. After that, the separation matrix is calculated through the decomposition value of the pseudocovariance matrix and the whitening matrix.

The calculated matrix is transferred to the speech estimator 204. In operation 606, the speech estimator 204 multiplies the separation matrix by a multi-channel mixed input to separate and output respective speech signals.

As described above, according to the present invention, matrix summation is performed after the sum of complex energy of time-frequency points in an appropriate unit on a multichannel spectrogram using enhanced orthogonality between different signals for speech separation. The separation matrix is obtained through a decomposition method, and the estimated separation signals are multiplied by the multi-channel mixed input to recover the estimated speech signals.

While the present invention has been described in connection with what is presently considered to be the most practical and preferred embodiment, it is to be understood that the invention is not limited to the disclosed embodiments, but is capable of various modifications within the scope of the invention. Therefore, the scope of the present invention should not be limited to the described embodiments, but should be defined not only by the scope of the following claims, but also by those equivalent to the scope of the claims.

1 is a view showing the structure of a voice separation device according to an embodiment of the present invention,

2 is a block diagram showing the structure of a speech separator on a multi-channel spectrogram according to an embodiment of the present invention;

3 is a block diagram showing the structure of a microphone input processor according to an embodiment of the present invention;

4 is a block diagram showing a structure of a separation matrix calculator according to an embodiment of the present invention;

5 is a block diagram showing a structure of a speech estimator according to an embodiment of the present invention;

6 is a flowchart illustrating an operation procedure of a voice separator according to an embodiment of the present invention.

<Explanation of Signs of Major Parts of Drawings>

100: multi-channel microphone 110: voice separator

200: microphone input processing unit 202: separation matrix calculation unit

204: Speech estimator 300: Time-frequency domain converter

302 block splitter 304 block adder

400: whitening matrix calculator 402: pseudocovariance matrix calculator

404 separation matrix calculation unit 500 block estimation unit

502: block release unit 504: time domain transform unit

Claims (17)

A multi-channel microphone which receives a plurality of voice signals and outputs time domain multi-channel data; Using matrix orthogonality between different signals for separation of speech signals transmitted from the multi-channel microphone, the sum of complex energy of time-frequency points in a predetermined unit on a multi-channel spectrogram is represented as a matrix, and then a matrix decomposition method is performed. Calculating a separation matrix through the multiplier and multiplying the calculated separation matrix by a multi-channel mixed input to restore estimated speech signals; The voice separator may include a microphone input processor configured to receive the time domain multi-channel data and convert the multi-channel data for voice separation into a multi-channel spectrogram in units of blocks and output the multi-channel spectrogram; A separation matrix calculator for calculating a decomposition value of a whitening matrix and a pseudo covariance matrix using values transmitted from the microphone input processor using orthogonality between the different signals, and calculating a separation matrix based on the obtained values; And a speech estimator for releasing a block estimated based on the multi-channel spectrogram and the separation matrix in units of blocks, and converting the estimated block into time-domain signals to separate each speech. The microphone input processor may include a time-frequency domain converter for converting the time domain multi-channel data into multi-channel microphone input data on a spectrogram; A block splitter for generating a multi-channel spectrogram in units of blocks by dividing points on a spectrogram into predetermined frequency and time units; Speech separation using an improved orthogonality on a multichannel spectrogram including a block adder for generating the multichannel data for the separation matrix calculation by adding complex values of the multichannel spectrogram on a block basis to a frequency axis and a time axis in a block. Device. delete The method of claim 1, Orthogonality between the different signals, After calculating at least two different time domain signals as signals on the spectrogram, the values between the signals are calculated, and then converted into a time-frequency domain relationship through the parsing theorem. An orthogonality on a frequency axis, orthogonality on a time-frequency axis, orthogonality on a time axis. delete The method of claim 1, The separation matrix calculation unit, A whitening matrix calculator configured to obtain a covariance matrix from the multichannel data, and to obtain the whitening matrix by an inverse of a square root of the covariance matrix; A pseudo covariance matrix calculator for multiplying the whitening matrix by the multi-channel spectrogram to obtain the pseudo covariance matrix, and decomposing the pseudo covariance matrix by pseudo singular value decomposition (SVD); A separation matrix calculator for obtaining the separation matrix through the decomposition value of the pseudocovariance matrix and the whitening matrix. Voice separation apparatus using enhanced orthogonality on a multi-channel spectrogram comprising a. The method of claim 1, The speech estimator, A block estimator for outputting a value calculated by multiplying the multi-channel spectrogram of the block unit and the separation matrix; A block releasing unit for dividing an area divided for each block into the calculated value; A time domain converter for converting a value of the block released by the block release unit into a time domain signal, separating the output into a respective voice, and outputting the separated voice. Voice separation apparatus using enhanced orthogonality on a multi-channel spectrogram comprising a. The method of claim 6, The block release unit, By using the orthogonality on the multi-channel spectrogram, the difference between the eigenvalues, the energy spectrum, and the angle of arrival of the estimated source between the blocks is calculated at a predetermined ratio, the reconstructed blocks are arranged in order, and then released. Voice separation device. The method of claim 1, The voice separation device, A speaker for outputting respective voices separated from the voice separator; A speech recognizer for performing speech recognition on the respective speeches Voice separation apparatus using improved orthogonality on the multi-channel spectrogram, characterized in that it further comprises. Complex energy of time-frequency points in a predetermined unit on a multi-channel spectrogram using orthogonality between different signals for separation of voice signals from time-domain multi-channel data for a plurality of voice signals input through a multi-channel microphone The process of representing the sum of Calculating a separation matrix through a matrix decomposition method for the matrix, and multiplying the calculated separation matrix by a multi-channel mixed input to restore estimated speech signals, The process of displaying the matrix may include converting the time domain multi-channel data into multi-channel data for speech separation and multi-channel spectrogram in block units, and outputting the multi-channel spectrogram; Calculating a decomposition value of a whitening matrix and a pseudo covariance matrix using the multichannel data and the multichannel spectrogram on a block-by-block basis using orthogonality between the different signals, and calculating a separation matrix based on the obtained values; The converting and outputting may include converting the time domain multichannel data into multichannel microphone input data on a spectrogram; Generating a multi-channel spectrogram in units of blocks by dividing points on the spectrogram into predetermined frequency and time units; Generating multi-channel data for the separation matrix calculation by adding complex values of the multi-channel spectrogram on a block basis to a frequency axis and a time axis in a block; Speech separation method using improved orthogonality on a multi-channel spectrogram comprising a. delete The method of claim 9, Restoring the voice signals, And releasing the estimated block based on the matrix, converting the estimated block into a time domain signal and separating the block into individual speech. The method of claim 9, Orthogonality between the different signals, After calculating at least two different time domain signals as signals on the spectrogram, the values between the signals are calculated, and then converted into a time-frequency domain relationship through the parsing theorem. Orthogonality on the frequency axis, orthogonality on the time-frequency axis, orthogonality on the time axis. delete The method of claim 9, The process of calculating the separation matrix, Obtaining a whitening matrix from the inverse of the square root of the covariance matrix after obtaining a covariance matrix from the multichannel data; Obtaining a pseudo covariance matrix by multiplying the whitening matrix by the multi-channel spectrogram in units of blocks, and decomposing the pseudo covariance matrix by pseudo singular value decomposition (SVD); A process of obtaining a separation matrix through the decomposition value of the pseudocovariance matrix and the whitening matrix Speech separation method using the improved orthogonality on a multi-channel spectrogram comprising a. The method of claim 11, Restoring the voice signals, Outputting a value calculated by multiplying the multi-channel spectrogram of the block unit and the separation matrix; Releasing an area divided for each block in the calculated value; The process of converting the value of which the area is released for each block into a time-domain signal and separating and outputting each voice. Speech separation method using the improved orthogonality on a multi-channel spectrogram comprising a. The method of claim 15, The release process, By using the orthogonality on the multi-channel spectrogram, the difference between the eigenvalues, the energy spectrum, and the angle of arrival of the estimated source between the blocks is calculated at a predetermined ratio, the reconstructed blocks are arranged in order, and then released. Voice separation method. The method of claim 9, The voice separation method, Outputting each of the restored voice signals through a speaker; Recognizing the output voices through a voice recognizer Speech separation method using the improved orthogonality on the multi-channel spectrogram, characterized in that it further comprises.

Images