JP2007249009A - Sound signal analysis method and sound signal synthesis method - Google Patents

Abstract

<P>PROBLEM TO BE SOLVED: To provide novel sound analysis/synthesis techniques for improving parameter estimation accuracy of a sinusoid signal having time variation thereby greatly improving accuracy of sound analysis and further sound re-synthesis. <P>SOLUTION: A sound signal analysis method is characterized in that a phase function of a first harmonic component is estimated through local variation rate encoding for an input sound signal having time variation characteristics, then the time axis of the signal is converted into its phase axis and the converted signal is analyzed again through the local variation rate encoding, and thus momentary amplitudes, momentary frequencies, and momentary phases of all components of the input signal are output as parameter functions capable of re-synthesis. A sound signal synthesis method is characterized in that the sound signal is re-synthesized by using the respective parameter functions of the sound signal analyzing method. <P>COPYRIGHT: (C)2007,JPO&INPIT

Description

  The present invention relates to a method capable of analyzing a sound signal such as a voice or a musical instrument sound with high accuracy and a method capable of re-synthesizing the analyzed sound signal into a target form for a desired purpose.

  It is known that the voice response of voices and many instrument sounds have a line spectrum structure. For example, in utterance of voiced speech, the vibration period of the vocal cords and the shape of the vocal tract change with time. These changes appear as changes in the fundamental frequency (F0) and amplitude envelope of the generated speech. Therefore, in order to accurately analyze voiced voices and musical instrument sounds, a technique for accurately analyzing time-varying patterns of the entire spectrum shape is required.

  Considering the progress of technological development in this field, Fant (1960) found that a voice signal generated by vocal fold vibration is modulated according to its shape when passing through the vocal tract, and a variety of voiced speech is generated. A source filter theory of voice generation that is generated is proposed. The sound source signal due to vocal cord vibration has a harmonic structure with energy at an integer multiple of the fundamental frequency (F0), and F0 changes from moment to moment according to the speed of vocal cord vibration. In addition, the intensity of modulation received by each harmonic component is different for each frequency, reflecting the characteristics of the vocal tract shape, and varies with time variation of the vocal tract.

  A voiced sound signal having such characteristics can be expressed as a sum of sinusoids whose instantaneous amplitude and frequency change smoothly. This idea is called Sinusoidal Modeling, and active research has continued until now (for example, Non-Patent Document 1). In this Sinusoidal Modeling, the voiced sound signal is expressed by the following equation (Equation 1). Here, x (t) represents a voiced sound signal, K represents the number of sinusoids, and ξk (t), ψk (t), and ηk correspond to the instantaneous amplitude, instantaneous frequency, and initial phase of the kth sinusoid, respectively.

  As is clear from the above equation 1, if all the sinusoid parameters (ξk (t), ψk (t), ηk) are given, it is easy to obtain a voiced sound signal x (t) therefrom. However, since this inverse transformation, that is, the process of estimating the parameters of each sinusoid component from the voiced sound signal x (t) itself is a kind of defect setting problem, it cannot be solved unless some constraint is provided.

  In Non-Patent Document 1 (1986), McAulay and Quatieri set a constraint that voiced speech is frequency analyzed at regular time intervals, and the instantaneous amplitude and frequency of the sinusoid component can be regarded as steady near each analysis time. Then, parameter estimation was performed. First, the peak amplitude and frequency corresponding to the harmonic component are detected from the spectrum at each time, and then the amplitude and frequency trajectories of the sinusoid component are calculated by connecting them based on temporal continuity. They take one spontaneous speech as an example and show that this idea works well, but because the system contains a lot of heuristics, the voiced voice in general, for example, There is no guarantee that equivalent results will be obtained in different cases.

  Further, the constraint condition that the amplitude and frequency are steady in the vicinity of the analysis time causes a deterioration in estimation accuracy at a time when the time change of these parameters is large. In order to solve this problem, Hermus (2005) proposed Exponential Sinusoidal Modeling (ESM) that calculates a spectrum using a Gaussian window function (Non-patent Document 2). He evaluates ESM's parameter estimation performance using NMR (Noise to Masking Ratio), an accuracy index based on auditory masking patterns. Although NMR is certainly improved by using ESM, the signal re-synthesized from the estimated parameters is significantly different from the original speech. In fact, his listening experiments show that the re-synthesized signal and the original sound can be easily distinguished (that is, the re-synthesized signal is very different from the original speech).

One advantage of Sinusoidal Modeling is that if the parameters of the sinusoid component included in the signal are estimated, it is easy to re-synthesize the waveform itself. Therefore, if the difference between the input signal and the recombined signal is calculated, it is possible to quantitatively evaluate the parameter estimation performance. However, to date, no studies have been reported on performance evaluation of sinusoidal modeling based on such strict indicators.
McAulay, RJ and Quatieri, TF, "Speech Analysis / Synthesis Based on a Sinusoidal Representation", IEEE trans. On Speech and Signal Processing, ASSP-34 (4), p744-754, 1986 Hermus, K., "Perceptual audio modeling with exponentially damped sinusoids", Signal Processing 85, p163-176, 2005

  Therefore, the object of the present invention is to provide a novel acoustic analysis / synthesis technique that can greatly improve the parameter estimation accuracy of a sinusoid signal with time change, thereby greatly improving the accuracy of acoustic analysis and further acoustic resynthesis. Is to provide. More specifically, we propose a new method called Local Vector Coding (LVC) in this specification, provide high-accuracy acoustic analysis technology, and freely generate sound based on the analysis. It is an object to provide a re-synthesizable technique.

In order to solve the above problem, an acoustic signal analysis method according to the present invention estimates a phase function of a first harmonic component by local change rate coding for an input acoustic signal having a time-varying characteristic, The time axis is converted to this phase axis, and the converted signal is analyzed again by local rate-of-change coding, so that the instantaneous amplitude, instantaneous frequency, and instantaneous phase of all components of the input signal are output as recombinable parameter functions. It consists of the method characterized by doing. The acoustic signals to be analyzed include not only audio signals but also instrument sound signals.
Acoustic signal analysis method.

  An acoustic signal synthesis method according to the present invention comprises a method characterized by re-synthesizes an acoustic signal using each parameter function in the acoustic signal analysis method as described above. Re-synthesis is possible not only to faithfully reproduce the original sound signal but also to convert the sound quality from the re-synthesized sound signal or when re-synthesizing the sound signal to another sound quality. (For example, morphing from male voice to female voice is also possible.

  According to the acoustic signal analyzing method and the acoustic signal synthesizing method according to the present invention, the acoustic signal can be accurately analyzed with an accuracy that cannot be obtained by the prior art, and the analyzed acoustic signal can be used for a desired purpose. It can be freely re-synthesized into the target form.

Hereinafter, embodiments of the present invention will be described in detail with reference to the drawings along with development progress up to completion of the present invention.
First, in the research for completing the present invention, the parameter estimation accuracy of a sinusoid signal accompanied by a change in time is determined using the residual signal, which is the difference between the recombined signal and the input signal, and the input signal energy ratio (S / R: We evaluated the performance of sinusoidal modeling using Singnal per Residual ratio as an evaluation index. In the following, the analysis of voiced speech will be mainly described, but the present invention can also be applied to the analysis of musical instrument sounds.

(1) Fundamental frequency (F0) detection (Pitch Determination)
The fundamental frequency (F0) of voiced speech corresponds to the frequency of the first harmonic component and is the main factor that determines the perceived voice pitch (pitch). Further, F0 is known to be a clue for expressing a speaker's emotion and identifying personality, and can be said to be a highly applicable feature quantity in acoustic analysis of speech. Usually, sinusoidal modeling estimates the parameters of all components contained in a voiced sound signal. Therefore, the parameter F0 of the first harmonic component is inevitably estimated. In the present invention, attention is also paid to aspects of sinusoidal modeling as an F0 detection technique.

  A technique for detecting F0 from an audio signal is called a pitch determination algorithm (PDA), and extensive research has been conducted so far. These can be roughly classified into the following three groups (Hess, 1983). The first is a frequency domain technique, which detects F0 using the harmonic structure that appears in the speech spectrum. A typical example of this method is the cepstrum method (Noll, 1966). The second is a time domain technique. This is to detect F0 using the quasi-periodicity of voiced speech, and the autocorrelation function method is typical (Sondhi, 1968). The third is a time-frequency domain method, which corresponds to the above two combinations. In this method, a voice is divided into a plurality of frequency channels using a band-pass filter, and F0 is detected by evaluating the periodicity of the signal for each channel.

  Early PDA studies used a single, clean voice as input. In recent years, research on PDAs under multiple voices and noise has been reported (Shimamura and Kobayashi, 2001; Wu, et. Al, 2003). In order to quantitatively evaluate the performance of a PDA for spontaneously uttered speech, F0, which serves as a reference for calculating a detection error, is required. These recent studies also calculate a reference F0 for clean single speech using cepstrum and autocorrelation before mixing with noise and other speech. However, in this calculation, the double pitch problem in which a frequency that is an integer multiple of true F0 is detected and the jump of the value at a time when the change speed of F0 is fast are often observed. It is necessary to correct the value of (Wu, etl al., 2003). Therefore, the fact that Hess (1983) pointed out at that time-“even if it was clean and single speech, the method of automatically detecting F0 from it has not been completed yet”- It must be said that it is true even now after more than a year.

  One of the factors that make it difficult to detect F0 of naturally uttered speech is the time variation of signal parameters, as in sinusoidal modeling. Since the harmonic structure of the spectrum is distorted in speech with a large F0 time change, it is not possible to accurately detect F0 with the cepstrum or autocorrelation function method that assumes stationarity near the analysis time. On the other hand, the local rate of change coding (LVC) proposed in the present invention is characterized by the ability to accurately analyze signals with temporal changes, and is a method that can be expected to be effective as a new F0 detection method. is there. In the following, a comparison experiment with existing methods will be conducted to quantitatively evaluate LVC F0 detection performance and the effectiveness of the method.

(2) Time change (hereinafter, sometimes simply referred to as “time change”) The nature of sinsuoid signal In sinusoid modeling of voiced speech, processing of time change of parameters such as amplitude and phase is an important issue come. Here, a sinusoid signal whose instantaneous amplitude and phase change with a simple function is taken as an example, and its characteristics are theoretically explained.

(2-1) Quadratic-Parameter Sinusoid signal The parameter of the sinusoid component included in voiced speech indicates a complex time-varying pattern. Paying attention to one of these components, Taylor expansion of the instantaneous logarithmic amplitudes A (t) and P (t) near the time tc and truncation to terms up to the second order yields the following equations 2 and 3. .

Where a ₀ , a ₁ , a ₂ , p ₀ , p ₁ , p ₂ are constants, a ₀ is the instantaneous logarithmic amplitude, a ₁ is the amplitude change rate, a ₂ is the amplitude change acceleration, and p ₀ is The instantaneous phase, p _{1 corresponds} to the angular frequency, and p ₂ corresponds to the change rate of the angular frequency. For example, a ₁ = a ₂ = p ₂ = 0 means a pure tone with an angular frequency p ₁ . In the following, for the sake of simplicity, consider the case of time tc = 0. A complex signal s (t) generated from these parameters is expressed by the following equation (4).

Α ₀ , α ₁ and α ₂ are complex constants. The real part of this complex signal corresponds to the approximation of the component signal near time c. The instantaneous amplitude and phase of naturally uttered speech have a pattern that is more complex than a quadratic function, but it is effective to examine the characteristics of this signal in the sense of the simplest time-varying signal. In the following, s (t) is referred to as Quadratic-Parameter Sinusoid (QPS).

(2-2) QPS frequency response
The frequency response S ₀ (ω) of the QPS signal s (t) is calculated by the following equations (5) and (6). Here, T is a constant representing the analysis time, w (t) is a Gaussian window function, and γ is a positive constant that determines the time length of the window function.

It is impossible to analytically find Equation 5 for any range of QPS parameters. However, when the amplitude envelope of s (t) * w (t) can be regarded as zero at least at the end point t = ± T of the analysis time, Equation 5 can be solved analytically. Specifically, this condition is that the amplitude envelope has a peak value Apeak between t = -T and T, and that value is Mergin () with respect to the end value Aedge = A (± T) of the analysis time. Corresponds to being larger by dB). In order to satisfy this, the parameters a ₁ and a ₂ need to be in the range of the following equations 7 and 8.

If the above condition is satisfied, the frequency response S ₀ (ω) of the QPS can be written as the following equation (9). That is, the logarithmic amplitude response and phase response of S ₀ (ω) are expressed by a quadratic function of the angular frequency ω.

FIG. 1 shows an example of S ₀ (ω). FIG. 1 (a) shows the frequency amplitude response of the QPS. The solid line shows the response of QPS with instantaneous logarithmic amplitude and instantaneous frequency changing over time (a ₀ = a ₂ = p ₀ = 0, a ₁ = 0.4 dB / ms, p ₁ = 100 Hz, p ₂ = 1.0 Hz / ms) . The thin line represents the response of the QPS corresponding to the pure tone (a ₀ = a ₁ = a ₂ = p0 = p ₂ = 0, p ₁ = 100 Hz). The arrow corresponds to the difference (F-shift) between the peak frequency of the amplitude response and the actual instantaneous frequency. (b) is the frequency phase response of QPS. Solid and thin lines are the same as (a). The difference (P-shift) between the phase at the peak frequency of the response and the actual instantaneous frequency is represented by an arrow. (c) shows the change in F-shift when the parameters p ₂ and a ₁ are changed. Each line corresponds to a QPS of a ₁ = 0.2, 0.4, 0.8 dB / ms. (d) represents the P-shift when the parameter is changed as in (c). In Fig. 1 (a) and (c), the thin line shows the frequency response of pure tone (a ₀ = a ₁ = a ₂ = p ₀ = p ₂ = 0, p ₁ = 100 Hz), and the solid line shows the amplitude and frequency. corresponding to QPS response (a0 = a2 = p0 = 0 , a1 = 0.4 dB / ms, p 1 = 100 Hz p 2 = 1.0 Hz / ms) that varies linearly. Both of these two signals have an instantaneous frequency and phase of zero at time t = 0 and an instantaneous frequency of 100 Hz, but the peak frequency of the amplitude response of the QPS including the change component is not 100 Hz, and the phase at the peak is also zero. Must not. The response peak frequency and phase shift are shown in FIGS. 1 (b) and 1 (d). The magnitude of the deviation shown here can be uniquely derived from Equation 9.

  This result suggests that, for example, a technique such as frequency response peak detection cannot accurately analyze the instantaneous frequency and amplitude of a QPS including time changes. In voiced speech with a harmonic structure, even if the instantaneous frequency of each component is an integer multiple of F0, if F0 or the amplitude changes over time, the amplitude peak on the spectrum is not necessarily an integer multiple of F0. Explain that it does not exist.

(2-3) Local vector transformation (LVT)
From the general expression 9 of the frequency response of the QPS, the first derivative S ₁ (ω) and the second derivative S ₂ (ω) with respect to the angular frequency can be calculated as the following expressions 10 and 11.

If these three types of frequency responses S ₀ (ω), S ₁ (ω), S ₂ (ω) can be obtained from the input signal s (t), the signal parameters α ₀ , α at an arbitrary angular frequency ω ₁ and α ₂ can be determined uniquely. This is expressed by the following equations (12), (13), and (14).

FIG. 2 shows the estimated parameters for voiced speech input. The input corresponds to the first vowel / i / part of the utterance / iyoiyo / of a male speaker. The speech data uses M107B-0002 with 216 phoneme balances from the ATR digital speech database. In FIG. 2A, the solid line represents the amplitude spectrum of the input signal. The input corresponds to the time of the first vowel / i / of male speaker's utterance / iyoiyo /. The thin line represents the amplitude response obtained from the signal parameters estimated by Local Vector Transform (LVT). The estimation parameter is calculated from the frequency response of an integral multiple of 140 Hz, and the first to fifth components are illustrated. In (b), the solid line indicates the phase response of the input voice. The thin line represents the phase response calculated from the parameters estimated by LVT. (c) represents the instantaneous logarithmic amplitude (parameter a ₀ ) estimated by LVT. In a region where the energy of the input sound is sufficient, almost the same value is estimated. (d) represents the instantaneous phase (p ₀ ) estimated by LVT. (e) represents the estimated instantaneous logarithmic amplitude change rate (a ₁ ). (f) represents the estimated instantaneous frequency (p ₁ ). (g) represents the estimated change in instantaneous logarithmic amplitude (a ₂ ). (h) represents the estimated change rate (p ₂ ) of the instantaneous frequency. 2A and 2B show the amplitude response and the phase response of the input, and FIGS. 2C to 2H show the parameters estimated using the equations 12, 13, and 14. FIG. Further, the frequency response of Equation 9 is re-synthesized using an estimation parameter whose frequency is an integer multiple of 140 Hz, and is shown by thin lines in FIGS. 2 (a) and 2 (b). In the vicinity of an integer multiple of 140 Hz, it can be confirmed that the input frequency response and the re-synthesized frequency response are in good agreement with both amplitude and phase. This means that parameter estimation based on QPS is effective even for actual speech such as naturally uttered speech. In this specification, this parameter estimation operation is called Local Vector Transform (LVT).

(3) Speech coding system (LVC) based on local rate of change
A sinusouidal modeling system for voiced speech is constructed based on the local change rate transformation (LVT) described above. The process of estimating the instantaneous logarithmic amplitude and phase of each component from the input speech signal is called local rate of change coding (LVC). The configuration of the LVC system is described below.

(3-1) Overview First, the general formula number 1 of sinusoidal modeling is transformed into the following formula 15. Here, Ak (t) is the instantaneous logarithmic amplitude of the kth component, and Pk (t) corresponds to the instantaneous phase. Incidentally, the instantaneous angular frequency of the component k can be calculated by time differentiation of Pk (t).

FIG. 3 shows a calculation block of the LVC system. Where x (t): input signal, P _F0 (t): instantaneous phase of the first harmonic component, Ai (t): instantaneous amplitude of the i-th harmonic component, Pi (t): i-th harmonic component Represents the instantaneous phase, y (t): recomposition waveform. The input of the system is a voiced speech signal x (t), and the output of the system is an instantaneous logarithmic amplitude function Ak (t) and an instantaneous phase function Pk (t) of each component. Once these parameter functions can be estimated, the signal waveform y (t) can be easily re-synthesized.

The estimation of the parameter function is performed by a two-stage calculation. In the first stage calculation, the instantaneous phase P _F0 (t) of the first component of the input signal is estimated. First, in each time frame of the input signal, the component parameter is calculated by the aforementioned LVT. As shown in FIG. 2, LVT estimates parameters from the frequency response at an arbitrary time, but in order to obtain the amplitude and phase functions that are system outputs, the parameter values calculated in each time frame are set in the time direction. Need to connect. This process is calculated by the following Trajectory Estimation module. At this time, the instantaneous phase function PF0 (t) of the first component of x (t) is obtained. In the F0 detection task, the value obtained by dividing the time derivative of P _F0 (t) by 2π is output.

In the calculation of the second stage, first, the time axis of the input signal x (t) is converted into a phase by P _F0 (t) = φ estimated in the first stage (Time to Phase Conversion, which will be described later). . In the higher-order harmonic component, since the frequency change speed is large, the energy peak of one component spreads to the frequency region of the other component. Time-phase conversion is introduced to reduce this inter-component interference. In the converted waveform x (φ), the temporal change of F0 is normalized, and the inter-component interference is reduced.

  This converted waveform x (φ) is subjected to LVT once again to estimate the parameters of all components including higher-order harmonic components. The LVT output is connected in the time direction using the Trajectory Estimation module, as in the first stage. At this time, the instantaneous logarithmic amplitude Ak (φ) and the instantaneous phase Pk (φ) with respect to the phase φ are obtained. Finally, the phase axis φ of these functions is inversely converted to time t to obtain the instantaneous logarithmic amplitude function Ak (t) and phase function Pk (t) of each component. Details of each calculation block will be described below.

(3-2) Local Vector Transform
In LVC, a frequency response is calculated for each input signal x (t) at a certain time step Δ. In the vicinity of each analysis time, it is assumed that each component of the input signal can be approximated by QPS, and the parameter of QPS at that time is calculated.

First, in the nth time frame, the input signal is multiplied by a window function w (t) centered at time t = nΔ. Next, the complex frequency response X ₀ (ω) of x (t) and the first and second derivatives X ₁ (ω) and X ₂ (ω) with respect to the angular frequency are calculated by the following equations (16), (17), and (18). To do.

As is clear from the above equation, the calculation of X ₁ (ω) and X ₂ (ω) is actually performed by using the signal x (t) multiplied by (-jωt) and (-jωt) ² as input. It can be implemented with a short-time Fourier transform (FFT). Therefore, three FFTs are executed for each time frame. If these frequency responses are obtained, the parameters for all the angular frequencies can be uniquely estimated with the LVTs of Equations 12 to 14.

(3-3) Trajectory Estimation: Determination of optimal path
With LVT, QPS parameters at all frequencies in each time frame can be calculated (eg, FIG. 2). This estimated parameter is effective when only a single energy exists in the vicinity of the frequency, but includes a large error in a frequency region where a plurality of components interfere. For example, in the vicinity of 210, 350, 490, and 630 Hz in FIG. 2, it can be confirmed that the estimated parameter value varies greatly. In order to obtain the instantaneous logarithmic amplitude function Ak (t) and phase function Pk (t) of each component, it is necessary to extract meaningful ones from the parameters estimated by LVT.

  This process can be considered as a problem of determining an angular frequency ωBEST (k, n) for outputting a reasonable estimation parameter as the component k in the time frame n. This calculation process is called optimum path determination. There are two constraint conditions for determining the optimum path: (1) stability of the LVT output value and (2) time continuity of the output parameter.

As shown in FIG. 2, in the frequency region where the energy of a single component is dominant, the variation in the output parameters of the LVT is small. The six parameters a _{0 to} p ₂ have substantially constant values in the vicinity of frequencies 140, 280, 420, 560, and 700 Hz. In other words, the frequency with a smaller variation in the parameter estimation value is more appropriate as the optimum path. In order to evaluate this, the standard deviation σ (n, ω) of the parameter estimated in the vicinity of the angular frequency ω is calculated.

The parameter estimated in the time frame n and the angular frequency ω includes a time change rate. Since the instantaneous amplitude and phase of voiced speech change continuously, it is possible to evaluate whether or not it is caused by a single component by comparing the time continuity of the estimated parameters in different time frames. . The time continuity C ₁₂ between the value of the parameter estimated at a certain point G1 = (n1, ω1) and the parameter of the point G ₂ = (n ₂ , ω ₂ ) uses the following equations 19 and 20. To evaluate.

Here, g (t) is a linear function of t, and χ ₁₂ corresponds to the joint probability that g (t) satisfies the normal distribution calculated from the parameter distributions of G ₁ and G ₂ . μd is a parameter estimate. This time continuity is calculated from four parameters excluding a ₀ and p ₀ among the six parameters of QPS. This is to avoid the problem of phase rotation. Usually, the instantaneous phase of the sinusoid signal is a monotonically increasing function with respect to time t, but the instantaneous phase obtained in each time frame is limited to a range of ± π. In order to obtain a correct instantaneous phase, it is necessary to use a value obtained by adding an integer multiple of 2π to this value, but it is difficult to simultaneously estimate the integer value and evaluate time continuity.

The optimal path is calculated by dynamic programming with time continuity C ₁₂ as a cost function. By limiting the calculation range in the time-frequency plane, it is possible to calculate the optimum paths of a plurality of components. In the calculation of the first stage of LVC, the optimal path of the first harmonic component is calculated from the frequency range of 50 to 500 Hz, and in the second stage, k + 0.5 to k + 1.5 (in units of component k) The path is calculated from the range of the harmonic number).

FIG. 4 shows an example of calculating the optimum path. FIG. 4 shows the optimum path (speech is the same as in FIG. 2) determined in the phoneme change part from the first / i / to / y / of the male speaker's utterance / iyoiyo /. The (a ₁ , p ₁ ) on the determined path is set as an instantaneous value, and the parameter value at each point is displayed as a three-dimensional vector using the change rate (a ₂ , p ₂ ) (solid line). It can be confirmed that the direction of each vector is smoothly continuous with neighboring time vectors.

(3-4) Trajectory Estimation: Generation of locus function The value of the QPS parameter at N points (t _{0 to} tN _N-1 ) is obtained by calculating the optimum path. Based on these values, the instantaneous logarithmic amplitude Ak (t) and instantaneous phase Pk (t) of the component k are determined. First, the problem of phase rotation described in the previous section is solved. The instantaneous phase function Pk (t) to be estimated is a monotonically increasing function with respect to time. However, since the value of p0 obtained from each time frame is limited to ± π, it does not necessarily monotonously increase. This occurs because information on phase rotation of 2π × u (u is a positive integer) is lost between time frames. To obtain the correct phase function, it is necessary to determine the rotation factor u _n between each time frame.

Phase rotation factor u _n between time frames n and n + 1 can be calculated by the following equation number 21. Where v (t) is a cubic function representing the instantaneous angular frequency, and its coefficient is uniquely calculated from p ₁ (n), p ₂ (n), p ₁ (n + 1), p ₂ (n + 1) it can. Δ is a time frame interval.

  After solving the problem of phase rotation with the above equation, the amplitude function and phase function are determined from the parameter estimates at all times. These functions are expressed as a quintic function with respect to time t between one time frame. Although the coefficients of this quintic function can be uniquely determined from the estimated parameter values, the functions estimated by this technique can often be oscillatory. This occurs because a subtle error in parameter estimation is amplified.

  In order to suppress this vibration, the following constraint condition is used for the coefficient of each quintic function. (1) Up to the fourth derivative with respect to time is continuous at the calculated connection point of each quintic function. (2) At the connection point of the calculated quintic function, the instantaneous value, first-order and second-order derivatives maximize the probability of combining a normal distribution represented by the average value of the estimated parameter and the variance value used in determining the optimal path. To do. When the coefficient of each quintic function is determined using such a constraint condition, the vibration described above can be suppressed.

  FIG. 5 shows an example of the instantaneous logarithmic amplitude function and instantaneous phase function calculated from the parameter value and variance of the optimum path. FIG. 5 shows an example of generation of a trajectory function, and shows an example of calculation of instantaneous logarithmic amplitude function and phase function (male speaker / iyoiyo /). FIG. 5 (a) shows the instantaneous logarithmic amplitude function for which the thick line is calculated. White circles correspond to parameter values on the optimum path, and error bars correspond to the variances. (b) is the instantaneous phase function (after phase rotation correction), (c) is the instantaneous logarithmic amplitude change rate function, (d) is the instantaneous frequency function, (e) is the instantaneous logarithmic amplitude change acceleration function (F) shows the instantaneous frequency change rate function. In the calculation of the trajectory function, the temporal smoothness described above is also used as the constraint condition, so that each trajectory does not completely match the estimated parameter. However, it can be confirmed that a locus with little vibration and close to the parameter value can be generated.

(3-5) Time to Phase Conversion
When the interference between the components of the input signal is small, the trajectory function of each component can be easily obtained by the above method. However, with ordinary voiced speech, interference between components cannot be ignored particularly in higher harmonic components, and thus such a method cannot be applied. FIG. 6 shows such an example. Fig. 6 shows an example of time / phase conversion. (A) is a waveform and spectrogram of the transition part from / i / to / y / of male speaker / iyoiyo /. Is displayed. F0 increases from 140 Hz to 170 Hz (see Fig. 5). (b) shows the frequency amplitude response at time 450 ms of (a). Since the rate of change of F0 is large, interference between high-order harmonic components (2.0 kHz or higher) becomes strong, and the amplitude peak cannot be approximated by a quadratic function. (c) is a waveform and spectrogram after time / phase conversion of the signal of (a), the horizontal axis of the spectrogram represents the phase of the first harmonic component, and the vertical axis represents the harmonic number. It can be confirmed that F0 is almost constant. (d) is a frequency amplitude spectrum in a phase corresponding to the time of (c), interference between components is reduced, and amplitude peaks of higher harmonic components clearly appear. FIG. 6A shows the waveform and spectrogram of the utterance / iyoiyo / of a male speaker, and the time range of this figure is the same as FIG. F0 rises rapidly from 400 ms to 500 ms. It can be seen from FIG. 5 (f) that the change rate of F0 is about 0.8 Hz / ms at the maximum. Since the frequency change speed of the nth harmonic component is n times, interference between components cannot be ignored. In fact, in the spectrum at t = 450 ms in FIG. 6 (c), a clear peak cannot be confirmed for components of 2 kHz or higher.

In order to deal with this problem, time / phase axis conversion is introduced. This conversion is realized by estimating the phase PF _F0 (t) of the first harmonic component with little influence of interference by LVT and replacing the time axis t of the input signal with this phase axis. Since x (t) is actually sampled at discrete time, the value of x (t) at an arbitrary time t is complemented by a sinc function. In the converted signal x (φ), the phase (rad) is used instead of the time axis, and if this is subjected to frequency analysis, the harmonic component number is used instead of the angular frequency. This is shown in the following equations (22) and (23).

  By such conversion, it is possible to reduce interference between components due to the temporal change of F0. Fig. 6 (b) shows the waveform and spectrogram after time / phase conversion. In the spectrogram, it can be confirmed that the time change of the component frequency (the unit is the harmonic number) is almost eliminated. FIG. 6 (d) shows a spectrum at the same time as FIG. 6 (c), but clear peaks can be confirmed up to higher order components.

The phase axes of the instantaneous amplitude function Ak (φ) and the instantaneous phase function Pk (φ) obtained in this way can be easily set to the time axis using P _F0 (t) as shown in the following equations 24 and 25. Inverse transformation is possible.

  The output of the LVC can be expressed by 3N (2K + 1) values, which is the sum of the QPS parameter (6 × N × K) of N points × K components and the basic phase parameter (3N) of N points.

(4) Performance evaluation experiment using synthesized speech Next, the performance evaluation of the LVC system will be described. First, in order to perform quantitative evaluation, a synthesized speech whose parameters are known is used as an input signal. Evaluation items are F0 detection error and parameter estimation performance.

(4-1) Input Signal A total of 400 synthesized double vowels are created as input signals for evaluation. There are three types of stimulation parameters: vowel combinations (25 patterns), F0 change patterns (8 patterns), and speakers (male and female). F0 of the input signal is expressed by the following equation (26). Here, FC and FD are constants corresponding to the average size and change width of F0. In the case of a signal simulating a male speaker, FC = 100 Hz and FD = 10 Hz. 200 Hz, FD = 20 Hz. The constant L is the signal duration and 300 ms for all signals.

  The change pattern of F0 during the duration is determined by the parameter FP. Eight FPs are prepared in increments of 0.25 from 0 to 1.75. For example, when Fp = 0, F0 decreases smoothly during the duration, and when FP = 0.5, a peak-shaped change pattern having a vertex at time 150 ms is shown. FIG. 7 (a) shows a typical change pattern of F0. Fig. 7 (b) shows the F1-F4 of male speaker's dual vowel / ia / at the formant frequency, and (c) shows the instantaneous amplitude of each harmonic component and the second to 15th order. Ingredients are displayed.

  A double vowel is a combination of two Japanese five vowels. The frequency response corresponding to each vowel is calculated with a five-formant cascade-Klatt speech synthesizer (Klatt, 1980). There are two types of vowel formant frequencies: one that simulates a male speaker and one that simulates a female speaker. Table 1 shows the formant frequency of the input signal. These values are based on values extracted from the spontaneous speech of each adult male and female.

  Cascade-Klatt speech synthesizers are usually implemented with a sampling frequency of 10 kHz, which approximates a male speaker with a vocal tract length of about 17 cm. Since female vocal tract length is generally shorter than male, it is necessary to increase the sampling frequency of the Klatt synthesizer in order to simulate a female speaker. Here, the sampling frequency for female speakers is set to 12 kHz. This corresponds to approximately 14 cm in vocal tract length.

  Further, the frequency response H (f) above the Nyquist frequency of the Klatt synthesizer is determined using the following equations (27) and (28). Here, K (f) represents the frequency response of the Klatt synthesizer, and F5 represents the fifth formant frequency. H (f) means that the amplitude response is attenuated by -48 dB / oct and the phase response becomes zero at a frequency equal to or higher than the fifth formant frequency (male: 4.5 kHz, female: 5.4 kHz). This process makes it possible to calculate a continuous response in the region above the Nyquist frequency of the Klatt synthesizer. FIG. 8 shows an example of the calculated H (f). FIG. 8 shows an example of the amplitude envelope of the synthesized vowel, where (a) is the amplitude envelope of the vowel / i /, the line thickness corresponds to the speaker, the thick line represents male, and the thin line represents female. . (b) shows the amplitude envelope of the vowel / o /.

  The formant frequency changes smoothly from the preceding vowel to the subsequent vowel. Fig. 7 (b) shows the formant frequency of the double vowel / ia /.

  Based on these pieces of information, the input signal is created by the following equations 29 and 30. The number of components K is 80 when both male and female speakers are simulated. The sampling frequency of the signal is 48 kHz and the duration is 300 ms. FIG. 7 (c) shows an example of the amplitude pattern of each harmonic component.

(4-2) F0 detection performance Much research has already been conducted on F0 detection (pitch determination) of speech, and various algorithms have been proposed. As a technique for evaluating the performance of the F0 detection algorithm, Rabiner (1977) proposes the following error rate e (%) (Equation 31). Also in this specification, performance is evaluated using this error rate.

  In addition, in order to evaluate the performance difference from the existing method, we compare it with the cepstrum method (Noll, 1966) which is a typical F0 detection method. F0 detection by cepstrum is realized as follows. First, a 40 ms Hamming window is applied to the same time frame as LVC, and the power spectrum is calculated using an 8192-point FFT. Next, the spectrum obtained by converting the power of each frequency component into a logarithm is used as an input, and a cepstrum is obtained by performing FFT again. The quefrency with the energy peak on this cepstrum corresponds to the fundamental period (1 / F0) of the signal.

  Here, in order to improve the F0 detection accuracy of the cepstrum, the range in which F0 exists is limited in advance. For the input of Fc = 100 Hz, the search range of F0 for obtaining the peak of quefrency is 80 to 120 Hz, and for the input of Fc = 200 Hz, the search range is 160 to 240 Hz. This condition is used to improve the detection performance of the cepstrum method and is not applied to LVC. LVC performs F0 detection with the same parameters for all input signals.

  FIG. 9 shows the F0 detection error of the LVC and the cepstrum method. It can be seen from FIG. 9 that the LVC F0 detection error is always smaller than the cepstrum method regardless of the type of input signal. The average error for all inputs is 0.000650% for LVC and 0.163% for the cepstrum method. This result supports the effectiveness of the LVC system for F0 detection of voiced speech with parameter changes over time.

  FIG. 9 shows the evaluation of the F0 detection performance, where the horizontal axis indicates the type of vowel of the synthesized speech and the vertical axis indicates the F0 detection error (%). The LVC F0 estimation error is represented by ○ and ●, and the detection error by the cepstrum method (cepstrum method) is represented by ◇ and ◆. Error bars correspond to the standard deviation due to the F0 change pattern.

(4-3) Parameter estimation performance Here, the performance of the entire LVC system is evaluated. Since the parameters of the input sound are already known, it is possible to calculate the parameter estimation accuracy for each component, but in order to match the case where the input parameter is unknown, the signal recombined from the estimated parameter and the input signal The overall system performance is evaluated by comparing Specifically, using the input signal x (t) and the waveform y (t) re-synthesized from the estimated parameters, the input / residual signal ratio (S / R) is calculated. The value of S / R increases as the residual between the input signal and the recombined signal decreases. Since the recombined signal y (t) is calculated from the sum of all components, an error in parameter estimation of some components is directly reflected in the S / R. In order to obtain a high S / R, it is necessary that the error is small in the parameter estimation of all components.

  FIG. 10 shows the S / R for each vowel. S / R varies depending on the type of vowel, and there is a large difference between when the formant frequency of the input synthesized speech does not change (five types) and when it changes (20 types). The average S / R of the former is 65.9 dB at Fc = 100 Hz and 69.1 dB at Fc = 200 Hz, but the average S / R of the latter is 38.4 dB at Fc = 100 Hz and 43.5 dB at Fc = 200 Hz. It is. The average S / R of all samples is 46.3 dB, and no sample has an S / R below 30 dB.

(5) Performance evaluation experiment using natural speech The performance table experiment using synthetic speech showed that LVC can analyze voiced speech whose F0 and amplitude envelope change over time with high accuracy. Here, the performance when natural speech is input is evaluated. Since F0 of naturally uttered speech is unknown, the evaluation item is only S / R.

(5-1) Input signal Input speech is extracted from 216 words of phoneme balance in the ATR digital speech database. From the words included in the database, search for samples in which all phonemes are voiced sounds, and select the 12 words shown in Table 2. The start time and end time of each word sound are determined by the acoustic event layer label in the label information attached to the database. These 12 words spoken by 17 men and 17 women (M101-M117, F101-F117) in a quiet room are used as input signals for performance evaluation. The total number of samples is 384.

(5-2) Parameter estimation performance Similar to the case of synthesized speech, the instantaneous amplitude function and instantaneous phase function of each component of naturally uttered speech are estimated by LVC, and the signal is re-synthesized from the obtained parameters. FIG. 11 (a) shows an example of an input signal, re-synthesis, and residual signal. The input is / yumoa / spoken by a male speaker, and S / R is 27.6 dB. From this figure, it can be confirmed that the recombined signal substantially coincides with the input signal. FIG. 11 (c) shows spectrograms of these signals. The F0 and amplitude envelope of the input speech have severe time changes. Further, the spectrum of the residual signal (the lower part of FIG. 11 (c)) shows that energy remains in the time = frequency region where the energy of the input amplitude envelope is strong. FIG. 11 shows an example of LVC analysis of natural speech. (A) is an example of a male speaker's utterance / yumoa /, where the upper part is an input signal, the middle part is an LVC recombination signal, and the lower part is a residual signal. Correspond to each. The ratio of input to residual signal (S / R) is 27.6dB. (b) is an example of a female speaker's utterance / yumoa /, and the format of the figure is the same as (a). S / R = 30.4 dB. (c) is a spectrogram of the signal of (a), with the upper stage representing the input signal, the middle stage representing the recombined signal, and the lower stage representing the residual signal. (d) is the spectrogram of (b), and the format of the figure is the same as (c).

  FIGS. 11B and 11D show the input / resynthesis / residual signal of / yumoa / of a female speaker in the same format. It can be confirmed that the result is almost the same as that of the male speaker. Particularly, the energy around the time 0-200 ms of the spectrum of the residual signal and the frequency of 500 Hz is important. This residual energy exists between the first harmonic component and the second harmonic component of the input signal. In other words, the voiced sound signal is energy outside the LVC assumption that can be expressed as the sum of sinusoid signals whose amplitude and frequency continuously change.

  According to Fant's sound source filter theory, vocal cord vibration is the main sound source when voiced speech is uttered, but at the same time, a small amount of turbulent noise is also generated. Since this noise sound source is also modulated by the same resonance filter and the same vocal tract filter as the vocal cord vibration sound source, the residual energy becomes strong in a region where the amplitude envelope of the input signal is strong. It is known that sound source signals resulting from vocal cord vibrations attenuate with frequency, but turbulent noise sound sources do not have such characteristics. Therefore, it can be predicted that the magnitude of the residual signal increases as the frequency increases. FIG. 12 shows examples of other speech input signals and residual signals. In particular, the results of FIG. 12 (c) support the hypothesis described above. Fig. 12 shows a comparison between the input signal spectrum (upper) and the residual signal spectrum (lower). (A) is an example of male speaker / nyuIN /, where S / R = 31.2 dB. (B) is an example of / nyuiN / for a female speaker, S / R = 33.2 dB, (c) is an example of / reNai / for a male speaker, S / R = 27.6 dB, (d) Shows an example of / reNai / for a female speaker, and shows the case of S / R = 30.5 dB.

  FIG. 13 shows the average S / R of each word. The average S / R for male spoken words is 21.2 dB, the average S / R for female speakers is 24.0 dB, and the average S / R for all samples is 22.6 dB.

(6) Speech modulation based on LVC From the above experiment, it was shown that the component parameter of voiced speech in which LVC was spoken naturally can be estimated with high accuracy. By expressing voiced speech with such parameters, various applications can be expected from the viewpoint of speech processing. Here, as an example, a voice modulation method using sound source filter theory will be described.

(6-1) Speech rate modulation
The output of the LVC is an instantaneous logarithmic amplitude function Ak (t) and a phase function Pk (t) for each component k of speech. In order to apply Fant's sound source filter theory, the amplitude envelope E (t, ω) resulting from the vocal tract filter is calculated by the following equation (33). Here, Interpolate is a function for obtaining an instantaneous amplitude value at an arbitrary angular frequency using the instantaneous angular frequency and instantaneous amplitude value of all components at a certain time. This can be realized, for example, by linear interpolation of the instantaneous amplitudes of the two components closest to ω.

Based on this expression, a method of modulating the speech rate to an arbitrary speed will be considered. In the method of changing the speech rate by changing the simple sampling frequency, modulated speech like a speaker different from the original speech is generated. In order to avoid this, it is necessary that the envelope E does not change with respect to the angular frequency axis and that the range of F0 is the same as the original voice. Specifically, the instantaneous amplitude A ′ (t) and phase P ′ (t) of the modulated speech obtained by multiplying the speech rate by M _T are given by the following equations 34 and 35.

  By the above conversion, smooth speech that can be felt as the same speaker at any utterance speed is obtained.

(6-2) Amplitude envelope and modulation of F0 The personality of a speaker included in speech is strongly influenced by the amplitude envelope related to the vocal tract filter and F0 related to vocal cord vibration. For example, in order to modulate a voice of a woman with a short vocal tract length to generate a male voice with a long vocal tract length, the frequency axis of the envelope function may be changed. If the pitch of the voice is changed, the instantaneous phase function may be changed. Specifically, the instantaneous amplitude A ′ (t) and phase P ′ (t) of the voice modulated by MV times the speaker's vocal tract length and MP times the fundamental frequency are expressed by the following equations 36 and 37: I can write.

  By the above conversion, modulated speech that can be felt as a female speaker can be generated from the speech of a male speaker. Of course, the reverse is also possible.

(6-3) Voice Morphing Applying the amplitude envelope and the F0 modulation method described above enables continuous morphing from one voice to another voice. The feature of this method is that even a voice during morphing sounds like a natural utterance. More specifically, the two audio envelope function E _A, E _B and the phase function P _A, P _B, and the mixing ratio M _X speaker vocal tract length ratio R ₁₂ and two voice is given In this case, the amplitude A ′ (t) and the phase P ′ (t) of the morphing speech can be calculated by the following equations 38 and 39.

  FIG. 14 shows an example of morphing. FIG. 14 shows an example of morphing for utterance / warauoNna /. (A) is a spectrogram of a female speaker (b) is a modulated speech obtained by morphing (a) and (f) at a mixing ratio of 20%. (C) shows a mixing ratio of 40%, (d) shows a mixing ratio of 60%, (e) shows a mixing ratio of 80%, and (f) shows a spectrogram of a male speaker. FIG. 14 (a) shows a female utterance and FIG. 14 (f) shows a male utterance. It can be confirmed that the F0 and the amplitude envelope change smoothly as the mixing ratio is changed.

  As described above, according to the present invention, it is possible to accurately analyze voiced speech accompanied by a time change, which is difficult with the existing technology. Thereby, various applications can be expected using the technology according to the present invention. In this application, speech modulation is taken as an example of one application. In addition, it is thought that the method of analyzing the temporal change of natural speech in detail can greatly contribute to the development of future speech generation research (speech synthesis research).

  For example, a configuration example of a system showing the schematic steps up to speech signal synthesis according to the present invention is shown in FIG. As described above, according to the present invention, it is possible to perform an extremely accurate speech signal analysis by performing up to the time axis conversion of the phase spectrum, and substantially freely using the analyzed signal. Sound signals can be reproduced and synthesized with high accuracy.

  In the present application, the description has been made mainly on sound, but it goes without saying that the same application and development can be applied to musical instrument sounds.

It is a characteristic view which shows the frequency response of QPS. It is a characteristic view which shows the example of calculation of LVT. It is a block diagram which shows the example of a LVC system structure. It is a characteristic view which shows the example of optimal path determination. It is a characteristic view which shows the example of a production | generation of a locus function. It is a characteristic view which shows the example of time / phase conversion. It is a characteristic view which shows the example of the input signal of performance evaluation. It is a characteristic view which shows the example of the amplitude envelope of a synthetic | combination vowel. It is a characteristic view which shows the example of F0 detection performance. It is a characteristic view showing an example of parameter estimation accuracy. It is a characteristic view which shows the example of the LVC analysis example 1 of natural speech. It is a characteristic view which shows the example of the LVC analysis example 2 of a natural voice. It is a characteristic view which shows the example of the parameter estimation performance of natural speech. It is a characteristic view which shows the example of audio | voice morphing. It is explanatory drawing which shows the structural example of the system which concerns on this invention.

Claims (5)

  For an input acoustic signal with time-varying characteristics, the phase function of the first harmonic component is estimated by local rate-of-change coding, then the time axis of the signal is converted to this phase axis, and the converted signal is again converted to the local rate of change. An acoustic signal analysis method characterized by outputting the instantaneous amplitude, instantaneous frequency, and instantaneous phase of all components of an input signal as a recombinable parameter function by performing analysis by encoding.   The acoustic signal synthesis method according to claim 1, wherein the acoustic signal is re-synthesized using each parameter function in the acoustic signal analysis method according to claim 1.   The sound signal synthesis method according to claim 2, wherein the sound quality is converted into another sound quality from the re-synthesized sound signal or when the sound signal is re-synthesized.   The method of claims 1-3, wherein the acoustic signal comprises an audio signal.   The method of claims 1-3, wherein the acoustic signal comprises a musical instrument sound signal.

Images