JP6278294B2 - Audio signal processing apparatus and method - Google Patents

Description

  The present invention relates to an audio signal processing apparatus and method for performing audio signal processing such as synchronization compensation on a plurality of asynchronous audio signals recorded by a plurality of recording devices such as mobile phones and voice recorders.

  Asynchronous microphone arrays (see, for example, Patent Document 1 and Non-Patent Document 1) record n simultaneous recordings of a plurality of portable recording devices such as mobile phones and voice recorders brought by participants for audio enhancement of conference recordings. It is an advantage that an inexpensive and flexible configuration can be achieved by a general device rather than a dedicated large-scale recording device. However, the arrangement of the microphone elements becomes unknown (for example, see Non-Patent Documents 2 and 3), the recording start time and the sampling frequency do not match because the recording for each channel is not synchronized (for example, Non-Patent Documents 1, 4 and 5), and it is necessary to solve a problem that is not handled by normal microphone array signal processing.

JP 2007-028391 A

Z. Liu, "Sound source separation with distributed microphone arrays in the presence of clock synchronization errors," Proceedings of IWAENC, 2008. N. Ono et al., "Blind alignment of asynchronously recorded signals for distributed microphone array," Proceedings of WASPAA}, pp. 161-164, 2009. K. Hasegawa et al., "Blind estimation of locations and time offsets for distributed recording devices," Proceedings of LVA / ICA, pp. 57-64, 2010. S. Markovich-Golan et al., "Blind sampling rate offset estimation and compensation in wireless acoustic sensor networks with application to beamforming," Proceedings of IWAENC, 2012. E. Robledo-Arnuncio et al., "On dealing with sampling rate mismatches in blind source separation and acoustic echo cancellation," Proceedings of WASPAA, pp. 21-24, 2007. Shoji Makino et al., "Speech Separation", Springer, 2007. N. Ono et al., "Stable and fast update rules for independent vector analysis based on auxiliary function technique," Proceedings of WASPAA}, pp. 189-192, 2011. E. Vincent et al., "First stereo audio source separation evaluation campaign: data, algorithms and results," Proceedings of ICA, pp. 552-559, 2007. O. L. Frost et al., "An algorithm for linearly constrained adaptive array processing," Proceedings of IEEE, Vol. 60, No. 8, pp. 926-935, August 1972. Hiroshi Sawada et al., "Latest Trends in Sound Source Separation Technology," IEICE Journal, Vol. 91, No. 4, pp. 292-296, 2008.snrbf1} H. L. Van Trees, "Optimum Array Processing," Wiley, 2002. Akiko Araki et al., “Conference Speech Enhancement Based on Speaker Classification and SNR Maximizing Beamformer,” Proc. Of the Acoustical Society of Japan, pp. 571-572, March, 2007.

  One of the most important problems is that each recording device uses a separate A / D converter, so the sampling frequency between channels will be different. The signal processing performance is greatly degraded.

Among the problems inherent to the many asynchronous microphone arrays described above, the amount of sampling frequency mismatch (hereinafter referred to as mismatch) can be said to be the biggest problem in applying array signal processing to asynchronous recording. When multiple A / D converters used for simultaneous recording are not synchronized, even if the devices have the same nominal sampling frequency, the sampling frequency is mainly adjusted due to individual differences and temperature characteristics of crystal units. Only a slight mismatch of the order of 10 ppm (ppm is 10 ⁻⁶ ) occurs. A mismatch in sampling frequency between channels has an effect that a time difference of signals between channels drifts due to a time unit shift. Many array signal processing uses the property that the direction of the sound source produces a unique phase difference between the observation signals of each microphone, but even a change of just one sample has a significant effect on the analysis of the sound source direction. Therefore, the change in the time difference of several tens of ppm is large enough to break the array signal processing.

  The problem with the sampling frequency mismatch described above is that the phase difference between devices changes in the digital domain and the sound source position changes in a pseudo manner, so that each sound source does not move and has a unique phase difference. Breaks down the sound source separation method. Therefore, as a conventional research using an asynchronous microphone array, a blind alignment that simultaneously estimates a recording start time, a microphone position, and a sound source position on the assumption that there is no mismatch in sampling frequency (see, for example, Non-Patent Document 3), A method of performing resampling by interpolation for compensation under a condition where a sampling frequency mismatch is given (see, for example, Non-Patent Document 5), and an amplitude spectrum to perform blind compensation for an unknown sampling frequency mismatch A method using the correlation (see, for example, Non-Patent Document 1) has been proposed so far.

  However, in Non-Patent Documents 2 and 3, the sound source position, the microphone position, and the recording start time can be estimated simultaneously, but there is a problem that the sampling frequency mismatch cannot be compensated. In Non-Patent Document 1, each signal of the asynchronous microphone array is synchronized based on the envelope of signal energy, but there is a problem that exact time synchronization cannot be performed.

  Furthermore, in the blind compensation method for sampling frequency mismatch disclosed in Non-Patent Document 4, information in a high frequency region where aliasing occurs cannot be used for averaging in the phase region, and the accuracy of the processing result In the case of recording for a long time, the sampling frequency of the reference channel signal and the target channel signal for which the mismatch is compensated based on the reference channel signal are shifted when the frame relationship is shifted. There was a problem that mismatch could not be compensated.

  SUMMARY OF THE INVENTION An object of the present invention is to solve the above-mentioned problems, and in an apparatus and method for performing audio signal processing on each audio signal from a plurality of recording devices from an asynchronous microphone array, with higher accuracy than in the prior art and An object of the present invention is to provide an audio signal processing apparatus and method capable of compensating for a mismatch by estimating a sampling frequency mismatch in each audio signal blindly or non-blindly even in long-time recording.

In the audio signal processing device according to the first invention, there is a recording start time difference between the target channel signal and the reference channel signal, and the A / D converter of the reference channel signal and the A / D conversion of the target channel signal In an audio signal processing device that synchronizes the target channel signal with the reference channel signal when there is a sampling frequency mismatch with a device,
By performing a certain frame shift on the reference channel signal, a reference channel signal in a short-time Fourier transform expression is obtained, and on the basis of the recording start time difference and the sampling frequency mismatch, Target channel of short-time Fourier transform expression by performing integer sample shift and fractional sample shift by phase compensation method in frequency domain for the target channel signal so that the frame centers of the target channel signal correspond to each other A first signal processing means for obtaining a signal is provided.

  In the audio signal processing apparatus, assuming that the sampling frequency mismatch is 0, the entire reference channel signal is regarded as a first interval using a time difference estimation method using a time interval signal, and the reference is referred to in the first interval. A first preprocessing unit for obtaining a recording start time difference of the target channel signal with respect to the channel signal is further provided.

  In the audio signal processing device, a first section and a second section having single sound source section information corresponding to each other are selected from the reference channel signal and the target channel signal, respectively, and the two sections A second pre-processing unit for obtaining a sampling frequency mismatch and a recording start time difference by using a sampling frequency mismatch estimation method using sound source section information is further provided.

  Further, in the audio signal processing device, the target channel of the short-time Fourier transform expression with respect to the sampling frequency mismatch based on the target channel signal of the short-time Fourier transform expression and the reference channel signal of the short-time Fourier transform expression. The method further comprises second signal processing means for estimating a sampling frequency mismatch using a maximum likelihood estimation method in which the logarithmic likelihood of the observed signal compensated for the signal is maximized.

  Still further, in the audio signal processing device, the second signal processing means narrows down a sampling frequency mismatch range by a discrete value full search method and then determines a sampling frequency mismatch which is an optimal solution by a golden ratio search method. It is characterized by estimating.

  In the audio signal processing apparatus, the short-time Fourier transform expression target channel signal and the short-time Fourier transform expression using a staircase approximation that ignores a change in a frame based on the estimated sampling frequency mismatch. Using a linear phase compensation method that minimizes the time difference between the reference channel signal and the reference channel signal of the short-time Fourier transform expression so that the target channel signal of the short-time Fourier transform expression and the reference channel signal of the short-time Fourier transform expression are synchronized. A third signal processing means is further provided.

  Further, in the audio signal processing device, the target channel signal and the reference channel signal are converted by subjecting the target channel signal of the short-time Fourier transform expression and the reference channel signal of the short-time Fourier transform expression to inverse Fourier transform. A fourth signal processing means to be obtained is further provided.

In the audio signal processing method according to the second invention, there is a recording start time difference between the target channel signal and the reference channel signal, and the A / D converter of the reference channel signal and the A / D conversion of the target channel signal In an audio signal processing method executed by an audio signal processing device that synchronizes the target channel signal with the reference channel signal when there is a sampling frequency mismatch with a device,
By performing a certain frame shift on the reference channel signal, a reference channel signal in a short-time Fourier transform expression is obtained, and on the basis of the recording start time difference and the sampling frequency mismatch, Target channel of short-time Fourier transform expression by performing integer sample shift and fractional sample shift by phase compensation method in frequency domain for the target channel signal so that the frame centers of the target channel signal correspond to each other A first signal processing step for obtaining a signal is provided.

  In the audio signal processing method, assuming that the sampling frequency mismatch is 0, the entire reference channel signal is regarded as a first interval using a time difference estimation method using a time interval signal, and the reference is referred to in the first interval. A first preprocessing step for obtaining a recording start time difference of the target channel signal with respect to the channel signal is further provided.

  Further, in the audio signal processing method, a first section and a second section having single sound source section information corresponding to each other are selected from the reference channel signal and the target channel signal, respectively, and a single of the two sections is selected. The method further includes a second pre-processing step of obtaining a sampling frequency mismatch and a recording start time difference using a sampling frequency mismatch estimation method using sound source section information.

  Further, in the audio signal processing method, the target channel of the short-time Fourier transform expression with respect to the sampling frequency mismatch based on the target channel signal of the short-time Fourier transform expression and the reference channel signal of the short-time Fourier transform expression. The method further comprises a second signal processing step of estimating a sampling frequency mismatch using a maximum likelihood estimation method that maximizes the log likelihood of the observed signal compensated for the signal.

  Still further, in the audio signal processing method, the second signal processing step narrows down a sampling frequency mismatch range by a discrete value full search method, and then determines a sampling frequency mismatch which is an optimal solution by a golden ratio search method. It is characterized by estimating.

  Further, in the audio signal processing method, the target channel signal of the short-time Fourier transform expression and the short-time Fourier transform expression using a staircase approximation that ignores a change in a frame based on the estimated sampling frequency mismatch. Using a linear phase compensation method that minimizes the time difference between the reference channel signal and the reference channel signal of the short-time Fourier transform expression so that the target channel signal of the short-time Fourier transform expression and the reference channel signal of the short-time Fourier transform expression are synchronized. A third signal processing step is further provided.

  Further, in the audio signal processing method, the target channel signal and the reference channel signal are converted by performing an inverse Fourier transform on the target channel signal of the short-time Fourier transform expression and the reference channel signal of the short-time Fourier transform expression. A fourth signal processing step to be obtained is further provided.

  A computer-readable recording medium according to a third aspect of the present invention includes the steps of the audio signal processing method.

  According to the audio signal processing apparatus and method of the present invention, the reference channel signal of the short-time Fourier transform expression is obtained by performing a certain frame shift on the reference channel signal, while the difference between the recording start time and the above Based on the sampling frequency mismatch, the target channel signal is fractional by integer sample shift and frequency domain phase compensation so that the frame centers of the reference channel signal and the target channel signal correspond to each other. By performing sample shift, a target channel signal of short-time Fourier transform expression is obtained. Next, the sampling frequency mismatch is estimated using a maximum likelihood estimation method that maximizes the log likelihood of the observed signal compensated for the target channel signal of the short-time Fourier transform expression with respect to the sampling frequency mismatch, Based on the estimated sampling frequency mismatch, the time difference between the target channel signal of the short-time Fourier transform expression and the reference channel signal of the short-time Fourier transform expression is calculated using a staircase approximation that ignores changes in the frame. Using the linear phase compensation method that minimizes, the target channel signal of the short-time Fourier transform expression and the reference channel signal of the short-time Fourier transform expression are compensated so as to be synchronized. Therefore, even when recording for a long time with higher accuracy than in the prior art, the mismatch of the sampling frequency in each audio signal can be estimated blindly or non-blindly to compensate for the mismatch.

(A) is a signal waveform diagram of an observation signal observed by each microphone of the asynchronous microphone array, and (b) is a signal waveform diagram of a digital signal of each observation signal of (a). (A) is a signal waveform diagram of an observation signal observed by each microphone when the recording start time is shifted in FIG. 1, and (b) is a signal waveform diagram of a digital signal of each observation signal of (a). . It is a signal waveform diagram showing a compensation method using a linear phase model for two audio signals used in the audio signal processing method for blind compensation according to the first embodiment of the present invention. It is a graph which shows the specific example of the function value of the log likelihood function J (V, (epsilon)) with respect to the mismatching (epsilon) of the sampling frequency calculated in the audio | voice signal processing method of the blind compensation which concerns on the 1st Embodiment of this invention. A specific example of a function value of the log-likelihood function J (V, ε) with respect to a sampling frequency mismatch ε, showing a range narrowing method based on the discrete value full search method used in the blind compensation speech signal processing method according to the first embodiment It is a graph which shows. The function value of the log-likelihood function J (V, ε) with respect to the sampling frequency mismatch ε, showing the optimal solution search method using the golden ratio search method used in the blind compensation speech signal processing method according to the first embodiment. It is a graph. 7 is a table showing the calculation efficiency of the golden ratio search method of FIG. 6, where (a) is a table showing the calculation conditions, and (b) is the relationship between the number of divisions and calculation time in the case of all discrete value full search methods. (C) is a table showing the relationship between the number of divisions and the calculation time when the discrete value full search method and the golden ratio search method are used in combination. It is a flowchart which shows the audio | voice signal processing method of the blind compensation which concerns on the 1st Embodiment of this invention. 5 is a table showing experimental conditions for creating an observation signal when evaluating the estimation accuracy of sampling frequency mismatch and the contribution of sampling frequency compensation to sound source separation accuracy using the blind compensation speech signal processing method of FIG. FIG. 5 is a graph showing experimental results of measuring the estimation accuracy of sampling frequency mismatch using the blind compensation speech signal processing method of FIG. 4 and showing the mean square error (RMSE) of estimation of sampling frequency mismatch ε with respect to data length; is there. 5 is a graph showing a signal-to-distortion ratio (SDR) with respect to a signal length, which is an experimental result obtained by measuring an evaluation of the contribution of the sampling frequency compensation to the sound source separation accuracy using the blind compensation audio signal processing method of FIG. 4. FIG. 5 is a graph showing an experimental result obtained by measuring the contribution of sampling frequency compensation to sound source separation accuracy using the blind compensation speech signal processing method of FIG. 4 and showing a signal-to-interference ratio (SIR) of blind speech separation; . Reference channel showing a correction method using two single sound source section information by the same sound source separated in time as a clue used in the non-blind compensation method of sampling frequency mismatch in the asynchronous microphone array according to the second embodiment of the present invention It is a signal waveform diagram which shows the signal waveform of a signal and an object channel signal. FIG. 14 is a signal waveform diagram illustrating time differences τ _A and τ _B between a reference channel signal and a target channel signal in a non-blind compensation method for sampling frequency mismatch in the asynchronous microphone array of FIG. 13. (A) is a signal waveform diagram of each signal before compensation of the offset compensation method showing the offset compensation method of the recording start time for the reference channel signal and the target channel signal in FIG. 14, and (b) is each signal after compensation. FIG. FIG. 16 is a signal waveform diagram showing frame cutout in consideration of time drift in a target channel signal in the sampling frequency mismatch compensation method in the recording start time offset compensation method of FIG. 15. FIG. 16 is a signal waveform diagram illustrating offset compensation of a cut-out position of each frame and phase shift within the frame in the sampling frequency mismatch compensation method in the recording start time offset compensation method of FIG. 15. FIG. 10 is a signal waveform diagram of each signal showing an experimental result of speech enhancement using the S / N ratio maximum ratio beamformer and an enhancement result in the section A in the operation of sampling frequency mismatch compensation according to the second embodiment. . FIG. 10 is a signal waveform diagram of each signal showing an experiment result of speech enhancement using a maximum S / N ratio beamformer, and an enhancement result in a section B, in the sampling frequency mismatch compensation operation according to the second embodiment. . It is a block diagram which shows the structure of the audio | voice signal processing apparatus 10 which concerns on the 3rd Embodiment of this invention. It is a flowchart which shows the blind synchronous audio | voice signal process performed by the audio | voice signal processing apparatus 10 of FIG. It is a flowchart which shows the non-blind synchronous audio | voice signal process performed by the audio | voice signal processing apparatus 10 of FIG.

  Hereinafter, embodiments according to the present invention will be described with reference to the drawings. In addition, in each following embodiment, the same code | symbol is attached | subjected about the same component.

First embodiment.
In the first embodiment of the present invention, a technique for blindly estimating and compensating for sampling frequency mismatch between channels for an asynchronous microphone array will be described. Since the change in the time difference between channels due to the sampling frequency mismatch is constant in a short time, it is compensated by manipulating the phase in the frequency domain for each frame. Also, sampling frequency mismatch is estimated by maximum likelihood estimation assuming that the sound source does not move. As will be described later, it has been confirmed by experiments that the proposed method can significantly recover the performance of the array signal processing.

  In the first embodiment, a technique for estimating and correcting mismatch between sampling frequency channels for preprocessing of blind sound source separation (for example, see Non-Patent Document 6) will be described. First, the short time Fourier transform (Short Term Fourier Transformation) is focused on the property that the expansion of the time difference between channels becomes negligibly small during a short time and can be regarded as a constant delay proportional to the sample number at the center of the frame. In the following, it is proposed that the sampling frequency mismatch be compensated by linear phase compensation in the region (STFT). Here, although the short time depends on the sampling frequency, for example, when the sampling frequency is 16 kHz, it means a short time in frame units of about several milliseconds to several tens of milliseconds.

  Furthermore, in the first embodiment, the mismatch of sampling frequencies between channels produces an effect that the phase difference changes at a constant speed and the sound source position changes in a pseudo manner. Therefore, assuming that all sound sources do not move and are stationary, it is considered that stationary signal quality can be used as a measure of mismatch. Therefore, the sampling frequency mismatch is estimated so as to maximize the likelihood function of the observed signal assuming the stationarity. Although the likelihood function cannot be solved analytically, it has been empirically known to be locally convex in the vicinity of the optimum value, so the search range is narrowed down by the rough discrete value full search and the details by the golden ratio search Optimize by using simple search.

FIG. 1A is a signal waveform diagram of observation signals observed by the microphones 1 and 2 of the asynchronous microphone array, and FIG. 1B is a signal waveform diagram of digital signals of the observation signals in FIG. It is. 2A is a signal waveform diagram of an observation signal observed by each of the microphones 1 and 2 when the recording start time is shifted in FIG. 1, and FIG. 2B is a waveform diagram of FIG. It is a signal waveform diagram of a digital signal of an observation signal. As shown in FIGS. 1A and 2A, when the sampling frequency of the A / D converter of the microphone 2 is slightly higher than the sampling frequency of the A / D converter of the microphone 1, FIG. As shown in FIG. 2B, the digital waveform of the microphone 2 drifts as if the waveform was extended as compared with the digital waveform of the microphone 1. The problems in these cases are as follows.
(1) Since the sampling frequency of each A / D converter is shifted, the waveform expands and contracts.
(2) Since the recording start time is shifted, a waveform shift occurs.

  In order to solve these problems, first, a sampling frequency mismatch and compensation formulation will be described below.

The continuous signals x _O1 (t) and x _O2 (t) (t is a continuous time) of two microphones at the same time are sampled by separate A / D converters, and discrete signals x ₁ (n) and x ₂ (n ) (N is a sample number). Here, the sampling frequency of the discrete signal x ₁ (n) is represented by f _s , and the sampling frequency of the discrete signal x ₂ (n) is represented by an unknown dimensionless number ε (| ε | << 1) (1 + ε) f _s. Suppose that At this time, the relationship between the discrete signal and the continuous signal of each channel is expressed as follows.

Now, continuous signals x _O1 (t) and x _O2 (t) (t is a continuous time) of two microphones at the same time are sampled by separate A / D converters, and discrete signals x ₁ (n ₁ ), x ₂ (n ₂ ) (n ₁ and n ₂ are sample numbers) is obtained. Here, the sampling frequency of the discrete signal x ₁ (n ₁ ) is f _s , and the sampling frequency of the discrete signal x ₂ (n ₂ ) is (1 + ε) f _s represented by an unknown mismatch ε. At this time, the relationship between the discrete signal and the continuous signal is expressed as follows.

Here, the time origin of t and recording start time of the discrete signals _{x 1 (n 1), ΔT} 21 represents a delay of recording start time of the discrete signal _x 2 _{(n 1)} for discrete signals _{x 1 (n} 1) . Here, the origin of the continuous time t is the recording start time of the discrete signal x ₁ (n), and T ₂₁ is the recording start time of the discrete signal x ₂ (n). Thus, sample number _{n i} of the i-th channel to refer to the same time t (i = 1, 2) is expressed by the following equation.

_{n 1 = f s t (5} )
_{n 2 = (1 + ε)} f s (t-T 2) (6)

n ₂ can be expressed by the following formula using n ₁ .

_{n 2 = (1 + ε)} (n 1 -f s T 21) (7)

を正確に求めるためには、以下のような離散信号ｘ_２（ｎ）のｓｉｎｃ関数補間が必要になる。 In the following, when the discrete time of each channel needs to be a pair that refers to the same time, it is expressed as n ₁ , n ₂ , but instead, it simply indicates the sample number of one channel and the correspondence of the time When there is no need to discuss the above, it is expressed as n or the like. For the integer value n ₁ , since n ₂ referring to the same time is generally a non-integer, the discrete signal is corrected so that the discrete signal x ₂ (n) is synchronized with the discrete signal x ₁ (n).

In order to accurately obtain the sinc function interpolation of the discrete signal x ₂ (n) as follows.

  However, since accurate interpolation using a sinc function requires a long sinc function convolution, it is not a realistic calculation method, and some approximation is required to consider a method for efficiently compensating for sampling frequency mismatch. .

  Next, modeling of mismatch in sampling frequency within a frame will be described below.

Since much of the array signal processing is performed in the time-frequency domain, it is considered sufficient to have a method that closely approximates the signal representation in the short-time frame conversion (STFT) domain. To that end, let us first consider the time correspondence within one frame, where m−L / 2 ≦ n ₁ ≦ n + L / 2-1, where L is the STFT frame length and m is the center sample of a frame with channel 1. From the relationship of Expression (7), n ₁ near the sample m is expressed by the following expression.

n ₂ = (1 + ε) (n ₁ −m) + (1 + ε) m−f _s T ₂₁
⇔ (n ₂ −m) = (1 + ε) (n ₁ −m) + εm−f _s T ₂₁ (9)

It can be seen that the correspondence between (n ₂ -m) and (n ₁ -m) expands by εm with m. Since the frame center m increases with the recording time, this shift cannot be ignored.

On the other hand, (n ₁ −m) is always | n ₁ −m | ≧ L / 2 regardless of m in the frame. Under the condition of Lε << ₁ , ε (n ₁ −m) is 1 / in the frame. Since it is much smaller than 2 samples, it is considered negligible here. For example, if ε is about 10 ⁻⁵ to 10 ⁻⁶ and the frame length is about 10 ³ to 10 ⁴ , this can be ignored. Therefore, if ε (n ₁ −m) is ignored, the following equation is obtained.

_{(N 2 -m) = (n} 1 -m) + εm-f s T 21 (10)

Therefore, a model is obtained on the assumption that the time difference does not depend on n ₁ and is constant within the frame. Considering this, the observation signal of the second channel is τ (m) = − ε (m−M) (11)
By giving the delay amount τ (m; ε) given by (M is a constant) to channel 2 as x ₂ (n−τ (m; ε)), the following equation is obtained.

n ₂ ← n ₂ + τ (m; ε) (12)

  Equation (9) can eliminate the dependence of the time difference in the frame on m as in the following equation.

_{(N 2 -τ (m) -m} ) = (n 1 -m) + εm-f s T 21 (13)
_{⇔ (n 2 -m) = (} n 1 -m) -f s T 21 + M (14)

  Although the delay amount given by equation (11) is still a non-integer, the time shift within the frame is simplified to a linear phase in the STFT region, and hence phase compensation in the STFT region will be discussed below.

  FIG. 3 is a signal waveform diagram showing a compensation method using a linear phase model for two audio signals used in the audio signal processing method for blind compensation according to the first embodiment of the present invention. In the present embodiment, as shown in FIG. 3, a case where the time difference drifts linearly with respect to time is considered, and in the following, it is considered that the time difference is approximated in a staircase pattern ignoring drift in the frame.

  First, the short-time Fourier transform X1 (k, m) of the frame waveform centering on the mth sample is given as follows.

^{_{x 1 fr (l, m)}} = w (l) x 1 (l + m-L / 2) (15)

Here, x ₁ ^fr (l, m) is a frame waveform, w (l) is a recombinable window function of length L, −L / 2 <k ≦ L / 2 is a discrete frequency index, and F _L {} (K) represents an operation for obtaining the complex amplitude of the discrete frequency k by the discrete Fourier transform of the L point. However, discrete Fourier transform is replaced with fast Fourier transform in actual calculation.

First, the signal x ₂ (n ₂ ) is subjected to frame analysis centered on the same m-th sample as the signal x ₁ (n ₁ ), and only the change in the delay amount of each frame due to drift is compensated in the time-frequency domain. First, the frame analysis is performed by cutting out a long-time waveform using a window function with the m-th sample as the center in the same manner as the first channel.

^{_{x 2 fr (l, m)}} = w (l) x 2 (l + m-L / 2) (17)

This is subjected to a Fourier transform of the windowing to give a linear phase corresponding to the delay of τ ₂ (m; ε) samples.

Since the linear phase in the time-frequency domain corresponds to a circular time shift in the frame in the time domain, this process has a large error when the delay amount τ ₂ (m; ε) is large. Therefore, this process is effective when the delay amount τ ₂ (m; ε) is small over the entire frame. For this purpose, when the mismatch origin N ₀ is far away from the vicinity of the center of the signal or when the signal length is long. This is not suitable when L / m >> | ε | Since the position of the mismatch origin N ₀ can be moved near the center of the signal as will be described later, the former problem does not matter. Therefore, this calculation method is effective when the signal book is short.

  Next, mismatch estimation assuming spatial stationarity will be described below.

Since the microphone array is observed when | ε | << _1, even if the mismatch is not corrected, it can be assumed that the correlation between the signal x ₁ (n ₁ ) and the signal x ₂ (n ₂ ) is high. Therefore, n ₁ = 0,..., N ₁ −1 and n ₂ = 0,..., N ₂ −1 are treated as being the same, and the signal x ₂ (n ₂ ) that maximizes the correlation is as follows. A delay amount δ ₁₂ is obtained.

Then, the signal x ₂ (n ₂ ) is delayed by the delay amount δ ₁₂ and x ₂ (n ₂ ) ← x ₂ (n ₂ −δ ₁₂ ) (20)
And As a result, the origin of mismatch between the signal x ₁ (n ₁ ) and the signal x ₂ (n ₂ ) is moved to the vicinity of the center of the sample interval where the signals overlap. Then, a sample number near the center of the overlap is given to M of the delay amount τ (m).

は離散周波数ｋ毎に定常であると仮定できるため、この仮定に基づいた最尤推定によりミスマッチεを求める。上で議論した位相補償のいずれかを用いて計算した。 Here, assuming that all observed sound sources are stationary and have no movement of position, the observed signal compensates for the sampling frequency mismatch using accurate mismatch ε estimation.

Can be assumed to be stationary for each discrete frequency k, and thus the mismatch ε is obtained by maximum likelihood estimation based on this assumption. Calculated using any of the phase compensation discussed above.

より得られる

の分布を零平均、共分散行列Ｖ（ｋ）の多変量複素正規分布とおいた場合の対数尤度は次式で表される。

More obtainable

Logarithmic likelihood when the distribution of is a zero-mean and a multivariate complex normal distribution of covariance matrix V (k) is expressed by the following equation.

を用いた次式の標本推定で置き換える。

Here, {•} ^H represents a complex conjugate transpose, and V is a set {V (k) | k = −L / 2 + 1,..., L / 2} of V (k). Since the covariance matrix V (k) is unknown,

Replace with the following sample estimation using.

  This likelihood maximization cannot be solved analytically. Therefore, it is necessary to estimate the mismatch ε by a method such as a discrete value full search method of ε that maximizes the log likelihood J (V, ε).

を定式化したが、この対数尤度関数を最大化するミスマッチεは解析的に求めることができない。推定するパラメータはミスマッチεのみであり、その最適化には離散値全探索を行うことも考えられるが、一つのミスマッチεの評価のために全体域での共分散行列とその逆行列の計算が必要であるため、高い解像度の離散値全探索を行うためには計算量が膨大になる。以下では黄金分割探索法を用いた最尤推定の効率的解法について説明する。 In the present embodiment, an efficient solution for maximum likelihood estimation, which is a combination of the discrete value full search method and the golden section method, is considered. In the above, the log-likelihood function for evaluating the mismatch ε

However, the mismatch ε that maximizes the log-likelihood function cannot be obtained analytically. The only parameter to be estimated is the mismatch ε, and it may be possible to perform a discrete value full search for the optimization. However, in order to evaluate one mismatch ε, the calculation of the covariance matrix and its inverse matrix in the whole region is required. Since it is necessary, the amount of calculation is enormous in order to perform a high-resolution discrete value full search. In the following, an efficient solution of maximum likelihood estimation using the golden section search method will be described.

  Since the only parameter to be obtained in the maximum likelihood estimation problem is the mismatch ε, it is conceivable to use the golden section search method, which is a typical technique for the one-dimensional optimization problem. In the golden section search method, the maximum or minimum value of the convex function is obtained by iterative search while narrowing the search range. In the range where the function is locally convex, it converges uniquely to the optimal solution.

は最大値周辺では凸関数となることが経験的にわかっているため、適切に探索範囲を絞り込むことで黄金比探索法が利用可能となる。 FIG. 4 shows a function of the log likelihood function J (V, ε) for the sampling frequency mismatch ε, showing a range narrowing method based on the discrete value full search method used in the blind compensation speech signal processing method according to the first embodiment. It is a graph which shows the specific example of. Here, the log-likelihood function J (V, ε) is a log-likelihood function of the observed signal when the sampling frequency mismatch is compensated for the target channel signal with respect to the sampling frequency mismatch ε. As in the example shown in FIG. 4, the log likelihood function

Since it is empirically known that becomes a convex function around the maximum value, the golden ratio search method can be used by appropriately narrowing down the search range.

について、すべての対数尤度関数

の関数値を比較して最大値を与える次式のミスマッチε_ｉ＊を求める。 First, the search range is narrowed down by a discrete value full search with rough steps. Dividing search range E for mismatch ε into I points at equal intervals

For all log-likelihood functions

A mismatch ε _{i *} of the following equation that gives the maximum value is obtained by comparing the function values of.

The range E of the discrete value full search may be set to a reasonable range as a sampling frequency mismatch between recording devices. Since it is said that the sampling frequency mismatch of a general recording device is on the order of 10 ⁻⁵ , E may be set to 10 ⁻⁴ or a multiple thereof. An appropriate value for the number of divisions I of the search range depends on the search range, but it is considered sufficient to set it to about 10 to 100.

FIG. 5 shows a function of the log likelihood function J (V, ε) for the sampling frequency mismatch ε, showing a method of searching for an optimal solution by the golden ratio search method used in the blind compensation speech signal processing method according to the first embodiment. It is a graph which shows a value. That is, in the range narrowing by the discrete value full search method, as shown in FIG.
(1) Discretize sampling frequency mismatch ε by dividing the search range at equal intervals;
(2) A search is made for a logarithmic likelihood function J (V, ε) that maximizes the discretized sampling frequency mismatch ε.
(3) Estimate the adjacent range that gives the maximum value as the existence range of the optimum value.

についてのミスマッチεの黄金比分割法を以下のアルゴリズムにより行う。 Next, a likelihood function having [ε _{i * −1} , ε _{i * + 1} ] as a search range

The golden ratio division method of mismatch ε for is performed by the following algorithm.

(1) In step SS1, the initial value is determined as in the following equation.
a = ε _{* −1}
b = ε _{* + 1}

を求める。ここで
φ＝（（√５）−１）／２ （２６）
である。 (2) In step SS2,
p = b−φ (ba)
q = a + φ (ba)
As

Ask for. Where φ = ((√5) −1) / 2 (26)
It is.

なら
ａ＝ｐ
ｐ＝ｑ
ｑ＝ａ＋φ（ｂ−ａ） （２７）
とし、そうでなければ、
ｂ＝ｑ
ｑ＝ｐ
ｐ＝ｂ−φ（ｂ−ａ） （２８）
とする。 (3) In step SS3,

Then a = p
p = q
q = a + φ (ba) (27)
And if not,
b = q
q = p
p = b−φ (ba) (28)
And

としてεの最尤推定値を求めて終了する。 (4) In step SS4, if (ba) is not sufficiently small, the process returns to step SS2, and if (ba) is sufficiently small.

To obtain the maximum likelihood estimate of ε, and the process ends.

  FIG. 6 shows a function of the log likelihood function J (V, ε) for the sampling frequency mismatch ε, showing a method for searching for an optimal solution by the golden ratio search method used in the blind compensation speech signal processing method according to the first embodiment. It is a graph which shows a value. The golden ratio search method is an efficient solution to the one-dimensional maximization problem of the local unimodal function, and as shown in FIG. 6, 1: ψ or ψ: 1 (where ψ = (1 + √ (5)) / 2). ) In which the search range is narrowed down by golden ratio division (S101, S102,... In FIG. 6), and the range is narrowed to ψ-1 (≈0.62) times each time. Specifically, the search range is divided into golden ratios, the search range is narrowed down to a range that includes a larger value, and if the range is not sufficiently small, the processing returns to the golden ratio division process. On the other hand, if the search range is sufficiently small, the midpoint of the search range is ended as a solution.

回の関数評価となる。 FIG. 7 is a table showing the calculation efficiency of the golden ratio search method of FIG. 6, FIG. 7 (a) is a table showing the calculation conditions, and FIG. 7 (b) is the case of all discrete value full search methods. FIG. 7C is a table showing the relationship between the number of divisions and the calculation time when the discrete value full search method and the golden ratio search method are used in combination. Considering the calculation efficiency of the golden ratio search method, the calculation amount of the search with the accuracy of equally dividing the search range into N points is as follows.
(1) In the discrete value full search method, N function evaluations are performed.
(2) In the golden ratio search method,

Times function evaluation.

  An example of the actual measurement value (average value of 10 calculations) is shown below. After narrowing down the range by the 20-point discrete value full search method under the conditions of FIG. 7A, a search with an accuracy of 1 / N was further performed. In the case of all discrete value full search methods, the result is as shown in FIG. 7B, and when the golden ratio search method is used in combination with the discrete value full search method, the result is as shown in FIG. 7C. As is apparent from FIG. 7, it can be seen that the amount of calculation is greatly reduced when the golden ratio search method is used together.

を求める必要があるため、その計算は演算量の小さいものを用いるのが好ましい。フレーム中心ｍが小さいあるいは大きいフレームにおいても位相補償Δ（ｍ；ε）がフレーム長Ｌに対して十分小さければ、上述のフレーム切り出しの最適化を伴う位相補償ではなく、単純位相補償を用いるのがよい。その場合には、εの最尤推定値が得られたのちには位相補償を正確に行うためにフレーム切り出しの最適化を行うことが好ましい。 In the first embodiment, in the discrete value full search method and the golden section search method, a large number of phase compensation signals are used to evaluate the log likelihood function J (V, ε) for all mismatch ε candidates.

Therefore, it is preferable to use a calculation with a small calculation amount. If the phase compensation Δ (m; ε) is sufficiently small with respect to the frame length L even in a frame with a small or large frame center m, simple phase compensation is used instead of phase compensation with optimization of the frame cutout described above. Good. In that case, it is preferable to optimize the frame cutout after the maximum likelihood estimate of ε is obtained in order to accurately perform phase compensation.

  FIG. 8 is a flowchart showing a speech signal processing method for blind compensation according to the first embodiment of the present invention. In FIG. 8, after performing the time difference compensation process so that the reference channel signal and the recording start time are matched with the target channel signal in step S1, the target channel signal and the reference channel signal after the time difference compensation process are processed in step S2. The STFT process is performed on. Next, after setting an initial value of the sampling frequency mismatch ε in step S3, in step S4, the mismatch is compensated using the initial value of the mismatch ε, the degree of mismatch is evaluated, and the mismatch is equal to or less than a predetermined threshold value. Compensation processing is performed until As a result, an STFT representation of the reference channel signal and an STFT representation of the target channel signal after mismatch compensation are obtained.

  Furthermore, in order to verify the performance of the proposed method and the effectiveness of the asynchronous array signal processing, the present inventors artificially gave a sampling frequency mismatch to the observation signal by the microphone array of the mixed voice of multiple speakers, The proposed method evaluates the sampling frequency mismatch compensation accuracy and the contribution to blind source separation performance.

  First, experimental conditions will be described below.

  The observation signal used is a convolution of the measured impulse response with the speech of two speakers. The speech used was a combination of word utterances of two men and women in the ATR database for each speaker, and all combinations of selections by two speakers were evaluated. A sampling frequency mismatch was simulated by artificially changing the sampling frequency of one channel of the microphone array observation signal thus created. The sampling frequency was changed by a polyphase filter implemented in the sample function of MATLAB, and a polyphase filter having a length of 100 taps was used. The sampling frequency before the change was 16,000 Hz, and the sampling frequency was changed to six types of 16,000 ± 0.5, 16,000 ± 1, and 16,000 ± 1.5 Hz. These correspond to ± 31.25, ± 62.5, and ± 93.75 ppm, respectively, and are practically large as sampling frequency mismatches when separate A / D converters are used.

  FIG. 9 is a table showing experimental conditions for creating an observed signal when evaluating the estimation accuracy of sampling frequency mismatch and the contribution of sampling frequency compensation to the sound source separation accuracy using the blind compensation speech signal processing method of FIG. is there. That is, an auxiliary function method independent vector analysis (see, for example, Non-Patent Document 7) is used as a separation method for sound source separation evaluation. Other experimental conditions are shown in FIG.

  Next, estimation accuracy of sampling frequency mismatch will be described below.

FIG. 10 is an experimental result of measuring the accuracy of sampling frequency mismatch estimation using the blind compensation speech signal processing method of FIG. 4, and shows the mean square error (RMSE) of the estimation of the sampling frequency mismatch ε with respect to the data length. It is a graph to show. As is apparent from FIG. 10, the RMSE converges to 1/10 or less of the mismatch ε of the original sampling frequency even in the shortest observation signal of 3 seconds, and rapidly decreases as the data increases. Therefore, it can be seen that the likelihood assuming the stationarity of the observed signal is effective as an evaluation measure of the sampling frequency mismatch. Further, in this experiment, the convergence condition of the golden ratio search is determined to be that the search interval length is smaller than 10 ^−9, but the number of search iterations required for this purpose does not exceed 30 times, and the candidates are narrowed down. Such high accuracy is obtained only by evaluating the log-likelihood function at 40 points or less together with the 10-point discrete value full search.

  Further, the contribution to sound source separation will be described below. An evaluation experiment was conducted to confirm that the sound source separation performance recovered by compensating the sampling frequency mismatch by the proposed method.

  FIG. 10 is an experimental result of measuring the accuracy of sampling frequency mismatch estimation using the blind compensation speech signal processing method of FIG. 4, and shows the mean square error (RMSE) of the estimation of the sampling frequency mismatch ε with respect to the data length. It is a graph to show. FIG. 11 shows experimental results obtained by measuring the contribution of the sampling frequency compensation to the sound source separation accuracy using the blind compensation audio signal processing method of FIG. 4, and shows the signal-to-distortion ratio (SDR) with respect to the signal length. It is a graph to show. Further, FIG. 12 shows experimental results obtained by measuring the evaluation of the contribution of the sampling frequency compensation to the sound source separation accuracy using the blind compensation speech signal processing method of FIG. 4, and the blind speech separation signal-to-interference ratio (SIR). It is a graph which shows. Here, the entire observed signal was used for learning of the separation filter. As the evaluation scale, a signal-to-interference ratio (SIR) and a signal-to-distortion ratio (SDR) representing the intensity of suppression of non-target components were used (for example, Non-patent document 8). In addition, as a reference signal for calculating these evaluation values, a sound image of each sound source in a microphone without changing the sampling frequency was used.

  11 and 12, “no mismatch” indicates the sound source separation result when the sampling frequency is not changed, and represents the performance limit of the sound source separation in which the sampling frequency mismatch is compensated. “Unprocessed” indicates the performance of sound source separation without performing sampling frequency compensation. “Manual processing” indicates performance when phase compensation is performed with a correct sampling frequency mismatch ε. The “present embodiment” refers to a proposed method in which a sampling frequency mismatch is obtained by maximum likelihood estimation and compensated with a linear phase.

  First, as is apparent from FIG. 10, it can be seen that the mean square error (RMSE) in estimating the sampling frequency mismatch ε decreases as the data length increases. Next, as is clear from FIGS. 11 and 12, since the unprocessed SDR shows a very low value, this condition is a severe condition in which sound source separation cannot be performed unless the mismatch of the sampling frequency is compensated. I understand that. Phase compensation with correct parameters given by manual processing is only about 2 dB lower in both SIR and SDR than in the case where there is no sampling frequency mismatch, and is used to compensate for sampling frequency mismatch for blind speech separation (BSS). It can be seen that phase compensation in the STFT region is effective. In addition, the proposed method for blindly estimating and compensating for sampling frequency mismatch has almost no performance difference from manual processing that represents the performance limit of linear phase compensation, and shows the high accuracy of maximum likelihood estimation of the proposed method. From the above, it was confirmed that the proposed method is effective as a sampling frequency mismatch compensation for sound source separation because the proposed method can considerably recover performance degradation caused by sampling frequency mismatch.

  As described above, according to the present embodiment, a method for blindly estimating a sampling frequency mismatch between observation channels, which is a problem in an asynchronous microphone array, has been proposed. First, paying attention to the property that the expansion of the time difference between channels is negligibly small during a short time, and can be regarded as a constant delay proportional to the sample number at the center of the frame, and by linear phase compensation in the STFT region It was proposed to compensate the sampling frequency mismatch. We also formulated a likelihood function that evaluates the estimation of the sampling frequency mismatch assuming that the observed sound source is stationary and does not move. Furthermore, we proposed an efficient solution method of this likelihood function maximization problem that cannot be solved analytically, using a narrowing-down of the range by a rough discrete value full search and a fast estimation by a golden ratio search. As a result of experiments to estimate and compensate for sampling frequency mismatch and to evaluate sound source separation, the proposed method can estimate sampling frequency mismatch with high accuracy and linear phase. It was found that the sound source separation performance can be recovered to a level close to that when no sampling frequency mismatch occurred by compensation, and the effectiveness of the proposed method was confirmed.

Second embodiment.
The second embodiment assumes an application in which voice enhancement is performed by signal processing after recording, such as conference recording for creating minutes. A method of estimating the recording start time offset and the sampling frequency mismatch value from this single sound source section information by including single sound source section information, which is the time section in which only a specific sound source produces sound, in the recording signal Propose.

  FIG. 13 shows a correction method using two single sound source section information by the same sound source separated in time as a clue used in the non-blind compensation method of sampling frequency mismatch in the asynchronous microphone array according to the second embodiment of the present invention. It is a signal waveform diagram which shows the signal waveform of the reference channel signal shown and the object channel signal. In practical use, for example, the opening of a conference, the closing greeting, the introduction of the first self, a brief voice of a specific speaker, or the sound of a known sound source for synchronization that becomes a landmark such as a chirp signal, or before the performance By recording sounds such as a one-frame performance sound of a certain instrument at the beginning and end of recording, recording is performed as single sound source section information in a time section in which only a specific sound source is generated. As shown in FIG. 13, the recording start time offset and the sampling frequency mismatch can be calculated by using the recording signal of the section information by using two single sound source section information by the same sound source separated in time as a clue. It will be easier. In order to absolutely correct the sampling frequency of each recording device, a highly accurate and stable oscillator is required. Therefore, relative to one of the recording signals and the sampling frequency aligned with that signal. Consider compensation. From the single sound source section information recorded in all the devices, the relative correction with the reference signal is performed, and it is verified whether the speech enhancement using the S / N ratio maximizing beamformer becomes easy.

  First, sampling frequency mismatch between channels will be described below.

  In the second embodiment, sampling frequency mismatch due to individual differences in clocks between recording devices having the same nominal sampling frequency is handled. In the following, the influence of the sampling frequency mismatch between channels will be discussed from the relationship between the analog waveform and the digital waveform.

  As shown in FIG. 1, a difference in arrival time due to the distance between the microphones is generated in the microphone 1 and the microphone 2, and sound waves (referred to as a reference channel signal and a target channel signal, respectively) arrive. Consider the conversion of this analog waveform to a digital waveform. When the sampling frequency of the microphone 2 is slightly higher than that of the microphone 1, the number of samples of the digital waveform of the microphone 2 in the continuous time interval is larger than that of the digital waveform corresponding to the same continuous time interval of the microphone 1. As a result, when the digital waveforms of the microphone 1 and the microphone 2 are compared, the digital waveform of the microphone 2 becomes slightly longer than the microphone 1 as shown in FIG. From this, the sampling frequency mismatch between channels causes the waveform to expand and contract. Also, when an offset due to the recording start time between channels occurs, the waveform shifts as shown in FIG.

FIG. 14 is a signal waveform diagram showing the time differences τ _A and τ _B between the reference channel signal and the target channel signal in the non-blind compensation method for sampling frequency mismatch in the asynchronous microphone array of FIG. FIG. 15A is a signal waveform diagram of each signal before compensation of the offset compensation method showing the offset compensation method of the recording start time for the reference channel signal and the target channel signal of FIG. 14, and FIG. It is a signal waveform diagram of each subsequent signal. In FIG. 14, assuming that only the same sound source is included in the first and last portions of the observation signal, the time differences τ _A and τ _B are calculated from the peaks of the cross correlation function, and the time difference between the channels is estimated. Thus, the sampling frequency mismatch ε is obtained by the following equation.

ε = (τ _B −τ _A ) / D ₁

Here, D ₁ is a time difference (sample) between the same sound sources. As shown in FIG. 15A, when the recording start time exists, it is not possible to correspond to each frame between channels in the time-frequency domain, and it is necessary to compensate for a large time offset in order to perform processing in that area. . Next, as shown in FIG. 15B, the target channel signal is subject to the reference channel signal using the single sound source section information near the recording start time by performing time offset compensation by the time difference τ _A. The offset compensation of the recording start time can be performed on the channel signal.

  This time, the signal-to-noise maximization beamformer used as a speech enhancement method maximizes the power ratio of the target speaker signal (signal component) to the noise and speaker component (noise component) in the output signal. This is a technique for performing speech enhancement by forming a directivity characteristic with a blind spot facing to the center. This method does not require a steering vector necessary for forming an adaptive beamformer, and is expected to be effective even under reverberation. Since a steering vector is not required in advance, it is not necessary to use a phase difference due to the distance between microphones, which is often used in microphone array processing, and a beamformer can be applied even if the recording start time offset is corrected. It can be said that this method is suitable for speech enhancement using

  However, since it is a beamformer that forms a blind spot in the noise component direction, it is affected by the sampling frequency mismatch. Since the covariance matrix of the observation signal necessary for the design of the S / N ratio maximizing beamformer changes if there is an influence of sampling frequency mismatch, the beamformer created around the recording start time in the long recording section Not applicable in the section. For example, when a deviation occurs between one sample channel per second under the conditions of a distance between microphones of 2.5 cm, a sampling frequency of 48 kHz, and a sound speed of 340 m / s, this corresponds to a change of about 74 ° azimuth, thus forming a beam former It is conceivable that the direction angle will deviate greatly. For this reason, a mismatch in sampling frequency between channels causes a significant performance degradation in speech enhancement using the S / N ratio maximizing beamformer.

  Next, the correction of the recording start time offset and the linear phase compensation of the sampling frequency mismatch will be described below.

First, a mismatch time domain model is defined. Two microphone signals x _O1 (t) and x _O2 (t) (t is a continuous time) at the same time are sampled by separate A / D converters, and discrete signals x ₁ (n ₁ ) and x ₂ (n ₂ ) are sampled. It is assumed that (n ₁ and n ₂ are sample numbers) is obtained. Here, the sampling frequency of the signal x ₁ (n ₁ ) is f _s , and the sampling frequency of the signal x ₂ (n ₂ ) is (1 + ε) f _s represented by an unknown mismatch ε. At this time, the relationship between the discrete signal and the continuous signal is expressed as follows.

Here, T _i (i = 1, 2) represents the recording start time of the signal x _i (n _i ). Here, sample number _{n i} to refer to the same time t of the i-th channel (i = 1, 2) is expressed by the following equation.

n ₁ = (t−T ₁ ) f _s (32)
n ₂ = (1 + ε) (t−T ₂ ) f _s (33)

Here, assuming that the recording start time T ₁ = 0 of channel 1, equations (30) and (32) are expressed by the following equations.

If the recording start time difference T ₂ −T ₁ = T ₂ is set as τ ₂ samples at this time, the following equation is obtained.

From the above, n ₂ is expressed by the following equation using n ₁ .

Reference is made to sample number n ₂ corresponding to the time referred to by sample number n ₁ .

  Next, mismatch time-frequency domain modeling will be described below.

FIG. 16 is a signal waveform diagram showing frame cutout in consideration of time drift in the target channel signal in the sampling frequency mismatch compensation method in the recording start time offset compensation method of FIG. FIG. 17 is a signal waveform diagram showing offset compensation of the cut-out position of each frame and phase shift in the frame in the sampling frequency mismatch compensation method in the recording start time offset compensation method of FIG. In FIG. 16, when compensating for a sampling frequency mismatch in the time-frequency domain, frame segmentation considering time drift is performed on the target channel signal that is a compensation channel. Here, the frame extraction of the reference channel signal is simply performed at equal intervals M, and the leading sample number of the r _1st frame is the r ₁ Mth sample. Also, it cuts out of the target channel signal carried by mismatch shift width in consideration of ε (1 + ε) M, the frame top Sample number corresponding to r ₁ th frame of the reference channel signal is (1 + ε) r 1 _M-th sample. In FIG. 17, the offset compensation of the cutout position of each frame is performed in integer sample units. Therefore, STFT analysis is performed in which the shift width is round [(1 + ε) r1M], which is rounded by (1 + ε) M times. Next, the phase shift within the frame is performed as described in detail later.

  That is, the second embodiment is characterized in that the offset of the start sample number of each frame is corrected by frame cutout, and the phase shift correction in the frame after the frame cutout correction is performed in the time frequency domain.

  First, we discuss time offset correction by frame segmentation.

A frame that takes into account the time drift of the frame when cutting n ₁ (t) = N samples to N + N ′ samples of the reference channel signal channel 1 and correcting the corresponding recording signal of channel 2 of the target channel signal. It is necessary to cut out channel 2. The cut-out sample number N of channel 1 and the sample number n ₁ (r) corresponding to the start position of each frame in the correction section have a frame number r (r = 0,..., R−1) and a frame shift length M. By doing so, it is expressed by the following equation.

n ₁ (r) = N + rM (40)

If the start time of the correction section is 0, the sample number n ₁ ′ (r) corresponding to each frame in the cut-out section is expressed by the following equation.

n ₁ ′ (r) = n ₁ (r) −N = rM (41)

The sample number n ₂ ′ (r) corresponding to each frame in the correction section of channel 2 is expressed by the following equation.

n ₂ ′ (r) = (1 + ε) n ₁ (r) −τ ₂ −N (42)

  By cutting out the frame of each channel based on the sample number obtained above, the influence of the time offset at each frame cutout point is corrected.

  Next, consider the phase shift in the frame.

を以下のように求める。 Sample number n ₂ 'in each frame in (r), the sample number n _1' so as to correspond to the phase of each frame (r), it is necessary to provide a delay corresponding to εrM sample. If the time frequency domain signal of sample number n ₂ is X ₂ (k, r) (k = −L / 2 + 1,..., L / 2 are frequency numbers), sampling is performed by the phase shift of sample number n ₂ ′ (r). Time-frequency domain signal compensated for frequency mismatch

Is obtained as follows.

  Here, j is an imaginary unit, and L is a frame length.

  Further, correction of the mismatch between the recording start time offset and the sampling frequency using the single sound source section information will be described below.

In order to realize the compensation discussed above, single sound source section information is used below. The recording signal from which the single sound source section information is obtained is recorded at the beginning and the end, and the mismatch between the recording start time offset and the sampling frequency is derived using the section. First, the section information part in each channel, the beginning and the end (hereinafter, the first half of recording is A part and the latter half of recording is B part) is cut out, and the inverse spectrum transform of the cross spectrum is performed between the channels in the same section. To calculate the cross-correlation. Thereby, the time offset in the same section of each channel can be obtained. The sampling frequency mismatch (1 + ε) in the target channel signal with the reference channel signal is expressed by the following equation by setting the number of samples between the single sound source section information in the A part and the B part of each channel as D ₁ and D ₂ , respectively. expressed.

The number of samples D1 and D2 is the number of samples A and B in the reference signal channel, where n ₁ (t _A ) and n ₁ (t _B ) are the single sound source section start sample numbers, respectively. The numbers are n ₂ (t _A ), n ₂ (t _B ),
_{D 1 = n 1 (t B} ) -n 1 (t A) (17)
Is expressed by the following equation using the cross-correlation peak differences τ _A and τ _B in the same section of each channel.

  The above-described sampling frequency mismatch compensation is performed using the values obtained above.

  The present inventors confirm the above-described sampling frequency mismatch compensation operation by speech enhancement using a S / N ratio maximizing beamformer. In this embodiment, recording was performed for about one hour using two recording devices capable of stereo recording and a total of four channels. As the number of sound sources, one male voice and one female voice were used, and chirp signals were recorded in section A and section B as section information. The chirp signal is a signal that sweeps from a low frequency to a high frequency within a finite time. Since a peak is generated as an impulse when cross-correlation is taken, it is used this time because it is optimal for mismatch estimation. The S / N maximization beamformer is used for speech enhancement, and the effectiveness of mismatch compensation is confirmed by comparing the case where the waveform of the output result is aligned with only the recording start time offset and the case where the sampling frequency mismatch compensation is performed. The experiment was conducted with a sampling frequency of 48000 Hz, a frame width of 8192 samples, and a frame shift width of 4096 samples.

  FIG. 18 shows the result of speech enhancement experiment using the S / N ratio maximum ratio beamformer for the sampling frequency mismatch compensation operation according to the second embodiment, and shows the signal waveform of each signal indicating the enhancement result in the section A. FIG. FIG. 19 shows the results of the speech enhancement experiment using the S / N ratio maximum ratio beamformer for the sampling frequency mismatch compensation operation according to the second embodiment. It is a signal waveform diagram. Here, it is possible to apply to a long-time recording signal, or the same audio part of the A part and the B part are compared. FIG. 18 shows the result of cooperation in part A, and FIG. 19 shows the result of cooperation in part B. Output 1 is the result of applying only the recording start time and applying the beamformer without compensating the sampling frequency mismatch. Output 2 is the result of aligning the recording start time and compensating for the sampling frequency mismatch, It is the result of applying. The beamformer learned and created in part A was applied to FIGS.

18 and 19, as a result of comparing the waveforms, when the beamformer created in part A is applied with only the recording start time aligned, speech enhancement is possible in part A, but the beamformer is applied in part B. I understand that I can't. In this recording data, the relative mismatch is 9.45 × 10 ⁻⁷ samples per sample, and the sampling frequency mismatch between recording devices is relatively small, so at a time close to the recording start time of the A section. This is considered to be because the influence of the mismatch does not occur so far, and the influence of the mismatch becomes large at the time away from the recording start time of the B section. Further, when the beamformer created in the A part is applied after the recording start times are aligned and the sampling frequency mismatch compensation is performed, the A part and the B part can obtain a generally good speech enhancement result. From this, it is considered that the compensation of the sampling frequency mismatch is effective even in a long recording section.

  As described above, according to the present embodiment, for the purpose of performing speech enhancement using an asynchronous microphone array in advance, such as conference recording for creating minutes, a single sound source section is provided at the beginning and end of a recorded signal. We proposed a method of recording information and obtaining the value of mismatch between recording start time offset and sampling frequency from this section information. After estimating and compensating the recording start time offset from the section information and the sampling frequency mismatch, the operation was verified by applying the SN ratio maximizing beamformer created in the first half to the second half of the long recording. . As a result, it was confirmed that if the section information was recorded before and after the recording signal, it was relatively easy to synchronize the recording signal between channels using an asynchronous device.

Third embodiment.
In the third embodiment, x ₁ (n) is referred to as a reference channel signal, x ₂ (n) is referred to as a target channel signal, and signal processing is performed on the target channel signal x ₂ (n), whereby the reference channel signal x _{1 is} processed. It shall be synchronized with (n). However, n represents a discrete time. Specifically, it is intended to synchronize on the STFT region assuming application of microphone array signal processing, and when synchronization on the time domain is required, the inverse STFT is performed at the final stage.

Due to the difference in the recording start time and the sampling frequency mismatch, the n ₁ sample of the reference channel and the n ₂ sample of the target channel can be modeled as corresponding as follows:

n ₂ = (1 + ε) n ₁ + τ ₀ (49)

Here, ε represents a relative shift of the sampling frequency of the target channel with respect to the reference channel, and τ ₀ represents a discrete time of the target channel corresponding to the same continuous time as n ₁ = 0 in the reference channel. In addition, Formula (2) in 1st Embodiment is described as following Formula.
τ ₀ = −f _s T ₂₁

  The purpose of the third embodiment is to unify the blind synchronization method according to the first embodiment, the non-blind synchronization method according to the second embodiment, and the methods used for each, and to unify them. Positioning and supplementing the lack of explanation. For the sake of uniform visibility, different notation from the original embodiment is used and corrected.

  First, the basic algorithm will be described below.

を最大にする時間差τ、すなわち、

を求めればよい。 In the time difference estimation using the time interval signal, the time interval (hereinafter referred to as interval A) of [n _A1 − (N _A / 2), n _A1 + (N _A / 2) −1] of the reference channel signal x ₁ (n). Think). Here, n _A1 and N _A represent the section center and section length in the discrete time domain, respectively. In order to obtain the time difference of the time section of the target channel having the highest correlation with the same signal length with respect to the reference channel signal of the section A, the following cross-correlation function (note that the index was re-corrected from the second embodiment) )

The time difference τ that maximizes

You can ask for.

However, it should be noted that the time difference τ _A includes an average arrival time difference in the section A depending on the arrangement of the sound source and the microphone, in addition to the inter-channel time difference caused by the sampling frequency mismatch. Instead of the equation (50), it is also possible to use a generalized cross-correlation function obtained by passing φ ₁₂ (τ) through an arbitrary linear time-invariant filter. It is also possible to use it.

  Next, sampling frequency mismatch estimation using two time intervals will be described below.

[N _A1 −N _A / 2, n _A1 + N _A / 2-1] time interval of the reference channel signal x ₁ (n) (hereinafter referred to as interval A), [n _B1 −N _B / 2, n _B1 + N _B / 2-1] from the time interval (hereinafter referred to as interval B), the time delays τ _A and τ _B of the target channel with respect to the reference channel with respect to the signal waveforms in interval A and interval _B are respectively determined by the algorithm described above. Suppose that it was found. If the influence of the arrival time difference on the sound source is ignored, the following equation is obtained from the equations (49) and (50).

n _A1 + τ _A = (1 + ε) n _A1 + τ ₀ (52)
n _B1 + τ _B = (1 + ε) n _B1 + τ ₀ (53)

  From the formula (52) and the formula (53), it is expressed by the following formula.

Thereby, mismatch ε and time difference τ ₀ can be obtained. Here, Expression (53) corresponds to Expression (48) of the second embodiment.

  Next, STFT expression by unequal interval frame shift and phase compensation will be described below.

Now, it is assumed that a rough estimate of mismatch ε and time difference τ ₀ in equation (49) has been obtained, and based on this, STFT representation in which the frame centers correspond to each other in the reference channel signal and the target channel signal. I want to ask. Both frame shifts have a length of L. Now, the frame number is r, and the center samples of the r-th frame of the target channel signal and the reference channel signal are represented as m ₁ (r) and m ₂ (r), respectively. For reference channel signals,
m ₁ (r) = Mr (56)
As described above, the STFT expression is obtained by applying a certain frame shift length M.

However,
w (l) (0 ≦ l ≦ L-1)
Is a window function. On the other hand, for the target channel signal, the estimated mismatch ε and time difference τ ₀ are used,
m ₂ (r) = (1 + ε) m ₁ (r) + τ ₀
= (1 + ε) Mr + τ ₀ (58)
However, this is generally a non-integer.

Therefore,
m ₂ (r) = round [(1 + ε) Mr + τ ₀ ] (59)
δ (r) = ((1 + ε) Mr + τ ₀ ) −round [(1 + ε) Mr + τ ₀ ]
(60)
, M ₂ (r) separates the fractional part into Δ (r) with only the integer part, combines the integer sample shift and the fractional sample shift by phase compensation in the frequency domain, and the target channel signal is expressed by the following equation: Find the STFT representation of

  Note that if ε = 0, Δ (r) = 0, which results in a normal STFT with a constant frame shift.

  Next, blind estimation of sampling frequency mismatch and linear phase compensation on the STFT region will be described below.

を最尤法により最適化し、
ε←ε＋ε’ （６３）
となるサンプリング周波数推定の修正を求める手法について述べる。起点となる第ｒ_０フレームは、フレーム数をＲとして
ｒ_０＝Ｒ／２ （６４）
のように中央のフレームを与えるのが適当であろう。適切な位相補償を施したＳＴＦＴ領域の多チャンネル信号 Basically, as described in the first embodiment, here, the phase compensation of the target channel STFT expression X ₂ (k, r) starting from the r _0th frame is used.

Is optimized by the maximum likelihood method,
ε ← ε + ε '(63)
A method for obtaining correction of sampling frequency estimation is described. The starting r _0th frame is R ₀ = R / 2 (64), where R is the number of frames.
It would be appropriate to give a central frame like Multi-channel signal in STFT area with proper phase compensation

は、音源の移動がないと仮定すると定常であるとみなすことができ、確率密度関数が以下のように与えられる零平均多変量正規分布に従うと考えられる。

Can be assumed to be stationary if there is no movement of the sound source, and it is considered that the probability density function follows a zero-mean multivariate normal distribution given as follows.

  Here, V (k) represents a covariance matrix. Therefore, the log likelihood function excluding the constant term is expressed by the following equation.

  Here, by obtaining a mismatch ε ′ that maximizes the log likelihood function, a sampling frequency mismatch can be estimated as shown in Equation (63). This log-likelihood function maximization problem cannot be solved analytically, but the log-likelihood function J (ε ′) is known to be locally unimodal around the global optimal solution. Therefore, when the possible value of the mismatch ε ′ is sufficiently close to 0, the maximum likelihood solution can be efficiently searched by the golden section search. If the mismatch ε ′ can take a value away from 0, the search for the golden section search is performed in the local unimodal search range by narrowing down the candidate range of the unimodal global optimal solution by the coarse discrete value full search. Can be used.

  Further, blind synchronous audio signal processing and non-blind synchronous audio signal processing will be described below.

  FIG. 20 is a block diagram showing the configuration of the audio signal processing apparatus 10 according to the third embodiment of the present invention. The audio signal processing apparatus 10 of FIG. 20 is formed of, for example, a digital computer that is an information processing apparatus. By executing the processing programs of FIGS. 21 and 22 and their modified examples, a computer is used to generate a reference channel signal. On the other hand, the sampling channel mismatch ε generated between the recording devices 72 and 73 is estimated by blind synchronization or non-blind synchronization for the target channel signal, and the STFT expression reference is made based on the blind estimated sampling frequency mismatch ε. A linear phase compensation process is performed on the channel signal and the target channel signal, and then each signal is subjected to inverse Fourier transform to obtain a reference channel signal and a target channel signal after linear phase compensation.

  Hereinafter, the configuration and processing of the audio signal processing apparatus 10 according to the present embodiment will be described in detail.

  In FIG. 20, the audio signal processing apparatus 10 is connected to recording devices 71 and 72 having A / D converters 71a and 72a via USB interfaces 51 and 52, respectively. When recording using the recording devices 71 and 72, the user records without connecting to the audio signal processing device 10, and then connects the recording devices 71 and 72 to the USB interfaces 51 and 52 of the audio signal processing device 10. Then, the audio data recorded by the recording devices 71 and 72 is taken into the hard disk memory 23 via the USB interfaces 51 and 52, and the blind synchronous audio signal processing of FIG. 21, the non-blind synchronous audio signal processing of FIG. The audio signal processing of the modified example is executed by the audio signal processing device 10. Further, via the drive device interface 35b of the audio signal processing device 10, it is connected to an external storage device 60 which is a hard disk memory, for example, which stores and provides data necessary for audio signal processing (including audio data) in advance. The audio signal processing apparatus 10 may acquire necessary data by accessing the external storage device 60 and store it in the hard disk memory 23.

In FIG. 1, an audio signal processing apparatus 10
(A) a CPU (central processing unit) 20 of a computer that calculates and controls the operation and processing of the audio signal processing device 10;
(B) a ROM (read only memory) 21 for storing a basic program such as an operation program and data necessary for executing the basic program;
(C) A RAM (random access memory) 22 that operates as a working memory of the CPU 20 and temporarily stores parameters and data necessary for the audio signal processing;
(D) a hard disk memory 23 for storing various data (audio data, parameter data, etc.) used in the audio signal processing;
(E) A program that is configured by, for example, a hard disk memory and that is read using the CD-ROM drive device 45 and stores the processing programs of FIGS. 21 to 22 and the like (these programs are programs executable by a computer). A memory 24;
(F) a communication interface 51 that is connected to recording devices 71 and 72 such as a voice recorder via USB interfaces 51 and 52, and that transmits and receives data to and from the recording devices 71 and 72;
(G) A keyboard connected to a keyboard 41 for inputting predetermined data and instruction commands, receiving data and instruction commands input from the keyboard 41, performing predetermined interface processing such as signal conversion, and transmitting the data to the CPU 20 Interface 31;
(H) Connected to a mouse 42 for inputting instruction commands on the CRT display 43, receives data and instruction commands input from the mouse 42, performs predetermined signal conversion and other interface processing, and transmits them to the CPU 20. A mouse interface 32;
(I) Connected to a CRT display 43 that displays data processed by the CPU 20, a setting instruction screen, a generated signal waveform, signal data, and the like, and converts image data to be displayed into an image signal for the CRT display 43. A display interface 33 for outputting and displaying on the CRT display 43;
(J) Connected to a printer 44 that prints data processed by the CPU 20 and predetermined generated signal waveforms and signal data, etc., performs predetermined signal conversion of print data to be printed, and outputs it to the printer 44 A printer interface 34 for printing;
(K) Connected to a CD-ROM drive device 45 that reads program data of the program from the CD-ROM 45a in which the processing programs of FIGS. 21 to 22 are stored, and the read program data of the image processing program is a predetermined signal. A drive device interface 35a for performing conversion and transferring to the program memory 24;
(L) a drive device interface 35b that is connected to an external storage device 60 such as a hard disk memory for storing predetermined data and that performs read signal conversion on the read data and transfers it to the CPU 20 or the hard disk memory 23; With
These circuits 20 to 24, 31 to 34, 35 a, 35 b and 51, 52 are connected via the bus 30.

  In the above embodiment, it may be executed by using a CD-ROM 45a readable by a computer in which the processing programs of FIGS. 21 to 22 are stored, CD-R, CD-RW, DVD, DVD-R, Various recording media readable by a computer such as DVD-RW and DVD-RAM may be used.

  FIG. 21 is a flowchart showing blind synchronous audio signal processing executed by the audio signal processing apparatus 10 of FIG.

In step S11 in FIG. 21, pre-processing A or B is used to obtain an estimated value of mismatch ε and recording start time difference (hereinafter referred to as time difference) τ ₀ . In the pretreatment A, assuming the mismatch epsilon = 0, with a time interval signal time difference estimation technique, the entire reference channel signal is regarded as a section T _A, obtains the time difference tau ₀ in the interval T _A. Or, in the pre-treatment B, and the reference channel signal and the target channel signals, respectively select a section T _A and interval T _B have a single sound source section information corresponding to each other, a single sound source section information of the two time intervals The estimated values of the mismatch ε and the time difference τ ₀ are obtained by using the sampling frequency mismatch estimation method used.

  Next, in step S12, based on the reference channel signal and the target channel signal, the reference channel signal and the target channel signal in STFT representation in which the frame centers correspond to each other are obtained. Then, in step S13, blind estimation is performed on the sampling frequency mismatch ε in the STFT region based on the reference channel signal and target channel signal expressed in STFT. Further, in step S14, linear phase compensation processing is performed on the reference channel signal and target channel signal in STFT representation based on the blind estimated sampling frequency mismatch ε. In step S15, the reference channel signal and the target channel signal after linear phase compensation are obtained by performing inverse Fourier transform on the reference channel signal and the target channel signal in STFT representation subjected to the linear phase compensation processing. The signal waveform and data of the obtained signal are displayed on the CRT display 43, or output to the printer 44 and printed, and the process ends.

  FIG. 22 is a flowchart showing non-blind synchronous audio signal processing executed by the audio signal processing apparatus 10 of FIG.

In step S11A, an estimated value of mismatch ε and time difference τ ₀ is obtained using preprocessing B. In the pre-treatment B, the section TA and the section TB are selected from the reference channel signal, and the mismatch ε and the time difference τ ₀ are estimated using the sampling frequency mismatch estimation method using the single sound source section information of the two time sections. Find the value. Next, in step S12, based on the reference channel signal and the target channel signal, the reference channel signal and the target channel signal in STFT representation in which the frame centers correspond to each other are obtained. Further, in step S15, the reference channel signal and the target channel signal are obtained by performing inverse Fourier transform on the reference channel signal and the target channel signal of the obtained STFT expression. The signal waveform and data of the obtained signal are displayed on the CRT display 43, or output to the printer 44 and printed, and the process ends.

  However, in the preprocessing B, it is assumed that two time intervals in which only the same sound source is present can be selected as the interval A and the interval B by prior knowledge. It should be noted that the sections A and B are preferably separated from each other, and there is no information that can be particularly used in the blind synchronization. For example, a section including the beginning and the end of the reference channel signal can be selected.

Next, the positioning of the processes of step S11 or S11A, S12 and steps S13, S14 will be described below. Steps S11 or S11A and S12, and steps S13 and S14 are a pair of processes. In the process of step S11 or S11A, the sampling frequency mismatch ε and time difference τ ₀ are estimated from only a specific time interval signal, and the process of step S12 is synchronized upon conversion to the STFT region based on this estimated value. It corresponds to that. In non-blind synchronization, as long as the section A and the section B have only the same sound source, and the section A and the section B are sufficiently separated in time, only the steps S11A and S12 are performed. The process can be synchronized.

  On the other hand, in the case of blind synchronization, only time shift is considered in the preprocessing A, and in the preprocessing B, there is no guarantee that it is possible to select a time section in which only the same sound source is selected in the sections A and B. Contains errors. Therefore, the processing of only steps S11 and S12 does not compensate for the sampling frequency mismatch, so the processing of steps S13 and S14 is necessary.

  Further, when the preprocessing B is applied in the process of step S11 and the processes of steps S13 and S14 are performed, a certain mismatch ε is estimated in the process of step S11. In step S13, another mismatch different from the mismatch ε is estimated. Note that ε ′ is estimated. When the processing of step S12 is performed by applying the preprocessing B, the sampling channel is already compensated for to some extent in the target channel signal on the STFT region by steps S11 to S12. Since the processing in steps S13 to S14 is applied to the already compensated signal, not to the original target channel signal itself, in step S13, correction is performed by adding ε 'to the mismatch ε obtained in step S11. In this way, a new mismatch ε is estimated and the sampling frequency mismatch is compensated.

  Furthermore, the relationship with 1st Embodiment and 2nd Embodiment is demonstrated below. In the first embodiment, a blind synchronization method is described in which the processing in steps S11 to S14s and the preprocessing A in step S11 are performed. In the second embodiment, a non-blind synchronization method is described in which preprocessing B is performed in step S11 in steps S11 to S12.

Differences between the present invention and Non-Patent Document 4.
Hereinafter, differences between the present invention and non-patent documents will be described.

  Non-Patent Document 4 also seeks a sampling frequency mismatch, but differs from the present invention as follows.

(1) Sampling frequency mismatch is calculated differently.
Due to the sampling frequency mismatch, the time difference between the two channels drifts, but Non-Patent Document 4 directly obtains the drift slope by averaging the time differences obtained from each frame (formula (Non-Patent Document 4) 14)), the present invention obtains the maximum likelihood method based on the phase compensation in the STFT region. In the case of Non-Patent Document 4, it is not necessary to perform iterative calculation, but information in a high frequency region where aliasing occurs cannot be used for the average calculation in the phase region. On the other hand, in the present invention, although iterative is necessary, since the likelihood is obtained while operating at a sufficiently high speed and compensating, information in a high frequency region can be used, so that high accuracy can be obtained.

(2) The sampling frequency compensation method is different.
In Non-Patent Document 4, resampling is performed by Lagrange polynomial interpolation, which is the conventional method of Non-Patent Document 5. However, there are options for the sampling frequency compensation method, and in the present invention, such a conventional method can be used after estimating the mismatch.

(3) Frame shift of STFT In Non-Patent Document 4, the frame shift is constant for both the reference channel signal and the target channel signal that compensates for the mismatch. It will shift. In the present invention, it is considered that the frame shift is changed, and it is possible to cope with long-time recording.

  As described above in detail, according to the audio signal processing apparatus and method according to the present invention, while obtaining a reference channel signal of a short-time Fourier transform expression by performing a certain frame shift on the reference channel signal, Based on the recording start time difference and the sampling frequency mismatch, an integer sample shift and a frequency domain with respect to the target channel signal so that the frame centers of the reference channel signal and the target channel signal correspond to each other. The target channel signal of the short-time Fourier transform expression is obtained by performing the fractional sample shift by the phase compensation method in FIG. Next, the sampling frequency mismatch is estimated using a maximum likelihood estimation method that maximizes the log likelihood of the observed signal compensated for the target channel signal of the short-time Fourier transform expression with respect to the sampling frequency mismatch, Based on the estimated sampling frequency mismatch, the time difference between the target channel signal of the short-time Fourier transform expression and the reference channel signal of the short-time Fourier transform expression is calculated using a staircase approximation that ignores changes in the frame. Using the linear phase compensation method that minimizes, the target channel signal of the short-time Fourier transform expression and the reference channel signal of the short-time Fourier transform expression are compensated so as to be synchronized. Therefore, even when recording for a long time with higher accuracy than in the prior art, the mismatch of the sampling frequency in each audio signal can be estimated blindly or non-blindly to compensate for the mismatch.

10: Audio signal processing device,
20 ... CPU,
21 ... ROM,
22 ... RAM,
23. Hard disk memory,
24 ... Program memory,
30 ... Bus
31 ... Keyboard interface,
32 ... Mouse interface,
33 ... Display interface,
34 ... Printer interface,
35a, 35b ... drive device interface,
41 ... Keyboard,
42 ... mouse,
43 ... CRT display
44 ... Printer,
45 ... CD-ROM drive device,
45a ... CD-ROM,
51,52 ... USB interface,
60 ... External storage device,
71, 72 ... recording equipment,
71a, 72a ... A / D converter (ADC).

Claims (15)

When there is a recording start time difference between the target channel signal and the reference channel signal, and there is a sampling frequency mismatch between the A / D converter of the reference channel signal and the A / D converter of the target channel signal In the audio signal processing device for synchronizing the target channel signal with the reference channel signal,
By performing a certain frame shift on the reference channel signal, a reference channel signal in a short-time Fourier transform expression is obtained, and on the basis of the recording start time difference and the sampling frequency mismatch, Target channel of short-time Fourier transform expression by performing integer sample shift and fractional sample shift by phase compensation method in frequency domain for the target channel signal so that the frame centers of the target channel signal correspond to each other An audio signal processing apparatus comprising first signal processing means for obtaining a signal.   Assuming that the sampling frequency mismatch is 0, the time difference estimation method using the time interval signal is used to regard the entire reference channel signal as the first interval, and the target channel signal relative to the reference channel signal in the first interval is determined. 2. The audio signal processing apparatus according to claim 1, further comprising first preprocessing means for obtaining a recording start time difference.   A first section and a second section having single sound source section information corresponding to each other are selected from the reference channel signal and the target channel signal, and the sampling frequency using the single sound source section information of the two sections is selected. 2. The audio signal processing apparatus according to claim 1, further comprising second pre-processing means for obtaining a sampling frequency mismatch and a recording start time difference using a mismatch estimation method. Using the maximum likelihood estimation method, by further updating the mismatch of the sampling frequency, the mismatch of the sampling frequency log-likelihood of the observed signal compensated for the target channel signal of the short-time Fourier transform representation up and ing The audio signal processing apparatus according to claim 1, further comprising a second signal processing means for estimating   5. The second signal processing means, after narrowing a sampling frequency mismatch range by a discrete value full search method, estimates a sampling frequency mismatch which is an optimal solution by a golden ratio search method. The audio signal processing device described. Based on mismatch of the estimated sampling frequency, using a linear phase compensation method, a reference channel signal of the target channel signal and the short-time Fourier transform representation of the short-time Fourier transform representation is compensated to synchronize 6. The audio signal processing apparatus according to claim 4, further comprising third signal processing means.   Fourth signal processing means for obtaining the target channel signal and the reference channel signal by performing inverse Fourier transform on the target channel signal of the short-time Fourier transform expression and the reference channel signal of the short-time Fourier transform expression. The audio signal processing apparatus according to claim 1, 3 or 6. When there is a recording start time difference between the target channel signal and the reference channel signal, and there is a sampling frequency mismatch between the A / D converter of the reference channel signal and the A / D converter of the target channel signal In the audio signal processing method executed by the audio signal processing device for synchronizing the target channel signal with the reference channel signal,
By performing a certain frame shift on the reference channel signal, a reference channel signal in a short-time Fourier transform expression is obtained, and on the basis of the recording start time difference and the sampling frequency mismatch, Target channel of short-time Fourier transform expression by performing integer sample shift and fractional sample shift by phase compensation method in frequency domain for the target channel signal so that the frame centers of the target channel signal correspond to each other An audio signal processing method comprising a first signal processing step for obtaining a signal.   Assuming that the sampling frequency mismatch is 0, the time difference estimation method using the time interval signal is used to regard the entire reference channel signal as the first interval, and the target channel signal relative to the reference channel signal in the first interval is determined. 9. The audio signal processing method according to claim 8, further comprising a first preprocessing step for obtaining a recording start time difference.   A first section and a second section having single sound source section information corresponding to each other are selected from the reference channel signal and the target channel signal, and the sampling frequency using the single sound source section information of the two sections is selected. 9. The audio signal processing method according to claim 8, further comprising a second pre-processing step of obtaining a sampling frequency mismatch and a recording start time difference using a mismatch estimation method. Using the maximum likelihood estimation method, by further updating the mismatch of the sampling frequency, the mismatch of the sampling frequency log-likelihood of the observed signal compensated for the target channel signal of the short-time Fourier transform representation up and ing The audio signal processing method according to any one of claims 8 to 10, further comprising a second signal processing step for estimating the signal.   12. The second signal processing step is to estimate a sampling frequency mismatch which is an optimal solution by a golden ratio search method after narrowing down a sampling frequency mismatch range by a discrete value full search method. The audio signal processing method described. Based on mismatch of the estimated sampling frequency, using a linear phase compensation method, a reference channel signal of the target channel signal and the short-time Fourier transform representation of the short-time Fourier transform representation is compensated to synchronize The audio signal processing method according to claim 11, further comprising a third signal processing step.   A fourth signal processing step of obtaining the target channel signal and the reference channel signal by performing an inverse Fourier transform on the target channel signal of the short-time Fourier transform expression and the reference channel signal of the short-time Fourier transform expression; The audio signal processing method according to claim 8, 11 or 13.   15. A computer-readable recording medium comprising the steps of the audio signal processing method according to claim 8.

Images