JP2005523478A - How to synthesize speech - Google Patents

Abstract

The present invention relates to a method for analyzing speech, the method comprising the steps of: a) inputting a speech signal, b) obtaining the first harmonic of the speech signal, c) determining the phase-difference Df between the speech signal and the first harmonic.

Description

  The present invention relates to the field of speech analysis and synthesis, and without limitation, the field of text-to-speech synthesis.

  The function of a text-to-speech (TTS) synthesis system is to synthesize speech from common text in a given language. Today, TTS systems are implemented for many applications (eg, access to databases over the telephone network or assistance to people with handicaps). One method of synthesizing speech is by concatenating elements of a set of recorded speech subunits, such as semi-syllables or polyphonic characters. Most successful commercial systems use polyphonic concatenation. A polyphonic character has a group of two (diphones), three (triphones) or more sounds from meaningless words by segmenting the desired chunk of sound into a stable spectral region. Can be determined. In synthesis based on concatenation, the transition conversation between two adjacent sounds is important to guarantee the quality of the synthesized speech. In the selection of a polyphonic character as the basic subunit, the transition between two adjacent sounds is preserved in the recorded subunit, and concatenation is performed between similar sounds.

  However, before synthesis, the sound must be modified in duration and pitch in order to satisfy the prosodic constraints of the new word that contains the sound. This processing is necessary to avoid the generation of a monotonous sounding synthesized speech. In the TTS system, this function is performed by the prosodic module. To allow modification of duration and pitch in recorded subunits, many connections based on TTS systems are time domain pitch-synchronized waveform superposition (TD-PSOLA) (Speech Commun., Vol. 9, pp 453-467, 1990 “Pitch-synchronized waveform processing technology for text-to-speech synthesis using diphones” by E. Moulines and F. Charpentier)).

  In the TD-PSOLA model, the audio signal first follows a pitch marking algorithm. This algorithm assigns marks at the peak of the signal of the voiced segment, and assigns marks 10 ms apart in the unvoiced segment. Compositing is done by superposition of Hanning windowed segments centered at the pitch mark and extending from the previous pitch mark to the next pitch mark. The duration correction is given by deleting or repeating some of the windowed segments. On the other hand, pitch period correction is provided by increasing or decreasing the overlap between windowed segments.

Despite success in many commercial TTS systems, synthesized speech produced by using a synthetic TD-PSOLA model, as outlined below, mainly under conditions of large prosodic changes Represents several shortcomings.
1. Pitch correction introduces duration corrections that need to be properly compensated.
2. Correction of duration is one pitch period resolution (α = ..., 1 / 2,2 / 3,3 / 4, ..., 4 / 3,3 / 2,2 / 1, ...) Can be performed only with the quantization method.
3. Increasing the duration of the unvoiced portion may cause segment repetition to introduce “metallic” artifacts (synthesized speech sounds metallic).

  IEEE Bulletin on Speech and Audio Processing, Vol. 5, “Hybrid model for text-to-speech synthesis” by Fabio Violaro and Olivier Boeffard in September 1998 describes a hybrid model of text-to-speech synthesis based on concatenation.

  The audio signal is decomposed into a harmonic component with a variable maximum frequency in addition to the noise component according to the pitch synchronization analysis. The harmonic components are modeled as a sum of sinusoids with a frequency that is a multiple of the pitch. The noise component is modeled as a random stimulus applied to the LPC filter. In the unvoiced segment, the harmonic component is equal to zero. If pitch correction is present, a new set of harmonic parameters is evaluated by re-sampling the spectral envelope at the new harmonic frequency. For harmonic component synthesis where there is a duration and / or pitch correction, phase correction is introduced into the harmonic parameters.

Various other so-called “superposition and addition” methods are described, for example, by PIOLA (Pitch Inflected OverLap and Add) [P. Meyer, HW Ruhl, R. Kruger, M. Kugler LLMVogten, A. Dirksen, and K. Belhoula. By PHRITTS: Text-to-speech synthesizer for German, pages 877-890 of Eurospeech '93 in Berlin, 1993], or PICOLA (Pointer Interval Controlled OverLap and Add) [Morita: compression / decompression of speech over time Research in Japan, Nagoya University Master's Thesis (1987)].
These methods differ from each other in the method of marking the pitch period position.

  None of these methods give satisfactory results when utilized as a mixer for two different waveforms. The problem is phase mismatch. The phase of the harmonic is affected by the recording device, room acoustics, distance to the microphone, vowel color, simultaneous articulation effect, and the like. Some of those factors can be kept unchanged as in the recording environment, while other factors such as simultaneous articulation effects are very difficult (if not impossible) to control. As a result, when the pitch period position is marked without considering the phase information, the synthesis quality is compromised by phase mismatch.

  Other methods such as MBR-PSOLA (multiband resynthesis pitch-synchronized waveform superposition synthesis) [T.Dutoit and H.Leich. MBR-PSOLA: Text-to-speech synthesis based on MBE resynthesis of segment database. 1993 Speech Communication] regenerates phase information to avoid phase mismatch. However, this involves special analysis-synthesis operations that reduce the naturalness of the generated speech. This composition often sounds like a mechanical sound.

  U.S. Pat. No. 5,787,398 shows an apparatus for synthesizing speech by changing the pitch. One disadvantage of this method is that phase distortion occurs because the pitch mark is centered on the excitation peak and the measured excitation peak does not necessarily have to have a synchronous phase.

  The pitch of the synthesized speech signal is changed by dividing the speech signal into a spectral component and an excitation component. The latter is a voiced sound that is multiplied by a series of overlapping window functions that are synchronized with pitch timing mark information corresponding at least approximately to the moment of voice stimulation, and is added after application of a controllable time shift. Separate into multiplied audio segments. The spectral and excitation components are then recombined. The multiplication uses at least two windows per pitch period, each having a duration shorter than one pitch period.

  US Pat. No. 5,081,681 shows several methods and related techniques for determining the phase of each harmonic from the fundamental frequency of voiced sound. Applications include speech coding, speech enhancement, and speech time scale modification. The basic method involves reproducing the phase signal from the fundamental frequency and voiced / unvoiced information and adding a random component to the reproduced phase signal to improve the quality of the synthesized speech.

  US Pat. No. 5,081,681 describes a method of phase synthesis for speech processing. As the phase is synthesized, the result of the synthesis does not sound natural in many aspects of the human voice, and the surround sound is ignored by the synthesis.

  The present invention provides a method for analysis of speech, particularly natural speech. The method for speech synthesis according to the invention is such that the phase difference between the speech signal (especially the diphone speech signal) and the first harmonic of the speech signal is essentially a constant speaker dependent parameter for different diphones. Based on the discovery that there is.

  In the preferred embodiment of the invention, this phase difference is obtained by determining the maximum value of the audio signal and determining the phase zero, ie the positive zero crossing of the first harmonic. The difference between the maximum phase and phase zero is a speaker dependent phase difference parameter.

  In one application, this parameter serves as the basis for determining the window function (eg, raised cosine or triangular window). Preferably, the window function is centered on the phase angle given by the zero phase of the first overtone plus the phase difference. Preferably, the window function has a maximum at that phase angle. For example, the window function is selected symmetrically with respect to its phase angle.

  For speech synthesis, diphone samples are windowed by a window function, where the window function and the windowed diphone sample are offset by a phase difference.

  The diphone samples that are windowed in this way are concatenated. In this way, the natural phase information is stored so that the result of speech synthesis can be heard in a pseudo-natural manner.

  According to a preferred embodiment of the present invention, control information indicating a diphone and a pitch contour is provided. For example, such control information can be provided by a language processing module of a text speech system.

  A particular advantage of the present invention compared to other time domain superposition methods is that the pitch period (or pitch pulse) position is synchronized by the phase of the first overtone.

  The pitch information can be obtained by low pass filtering the first harmonic of the original audio signal and using a positive zero crossing as a zero phase indicator. In this way, phase discontinuity artifacts are avoided without changing the original phase information.

  Application examples of the speech synthesis method and speech synthesizer of the present invention include telecommunications services, language education, assistance for disabled persons, talking books and toys, voice monitoring, multimedia, and man-machine communication.

  The following preferred embodiments of the invention will be described in more detail with reference to the drawings.

  The flowchart of FIG. 1 is an illustration of a method for speech analysis according to the present invention. In step 101, natural speech is input. A known training sequence of meaningless words can be used for natural speech input. In step 102, a diphone is extracted from natural speech. A diphone is cut out from natural speech and consists of a transition from one phoneme to another.

  In the next step 103, at least one of the diphones is subjected to a low pass filter to obtain a first overtone of the diphone. This first overtone is a speaker dependent characteristic that can be kept constant during recording.

  In step 104, the phase difference between the first overtone and the diphone is determined. This phase difference is a speaker-specific speech parameter. This parameter is useful for speech synthesis, as will be described in more detail with reference to FIGS.

  FIG. 2 is an illustration of one method for determining the phase difference between the first overtone and the diphone (see step 4 in FIG. 1). The sound wave 201 obtained from natural speech forms the basis for analysis. The sound wave 201 is applied to a low-pass filter having a cutoff frequency of about 150 Hz for the purpose of obtaining a first overtone 202 of the sound wave 201. The positive zero crossing of the first overtone 202 defines a phase angle of zero. As shown in FIG. 2, the first overtone 202 spans 19 consecutive complete cycles. In the example considered here, the duration of the period increases slightly from period 1 to period 19. For one of the periods, a local maximum value of the sound waveform 201 within the period is determined.

For example, the local maximum value of the sound wave 201 within the period 1 is the maximum value 203. In FIG. 2, the phase of the maximum value 203 within the period 1 is indicated by _jmax . The difference Δj between j _{max in} period 1 and the zero phase j ₀ is a speaker dependent speech parameter. In the example considered here, this phase difference is about 0.3π. Note that this phase difference is approximately constant regardless of which maximum value is used to determine this phase difference. However, for this measurement, it is preferable to select the period according to the characteristic maximum energy position. For example, if the maximum value 204 in period 9 is used to perform this analysis, the resulting phase difference is approximately the same as period 1.

FIG. 3 is an example of an application of the speech synthesis method of the present invention. In step 301, been made windowed by a window function having a maximum value in the diphones is j _{0 +} .DELTA.j obtained from natural speech, it is possible to select the raised cosine which is centered relative to for example the phase j _{0 +} .DELTA.j.

  Thus, at step 302, a diphone pitch bell is provided. In step 303, voice information is input. This can be information obtained from natural speech or from a text speech system (eg, a language processing module of such a text speech system).

  A pitch bell is selected according to the audio information. For example, the audio information includes diphone information and pitch contour information to be synthesized. In this case, the pitch bell is selected accordingly at step 304 so that the connection of the pitch bell at step 305 results in the desired audio output at step 306.

An application of the method of FIG. 3 is shown as an example in FIG. FIG. 4 shows a sound wave 401 consisting of several diphones. In order to obtain a zero phase j ₀ for each of the pitch intervals, an analysis as described with reference to FIGS. 1 and 2 above is applied to the sound wave 401. As in the example of FIG. 2, the zero phase j ₀ is shifted from the maximum phase j _max within the pitch interval by a substantially constant phase angle Δj.

Raised cosine 402 is used to window sound wave 401. Raised cosine 402 is centered with respect to phase j ₀ + Δj. The windowing of the sound wave 401 by the raised cosine 402 gives a continuous pitch bell 403. Thus, the diphone waveform of the sound wave 401 is divided into such continuous pitch bells 403. The pitch bell 403 is obtained from two adjacent periods by a raised cosine centered on the phase j ₀ + Δj. The advantage of using a raised cosine over a rectangular function is that the edges are so smooth. Note that this operation is reversible by adding all of the pitch bells 403 in the same order and overlapping. This creates the original sound wave 401.

  The duration of the sound wave 401 can be changed by repeating or skipping the pitch bell 403 and / or by moving the pitch bells 403 closer to or away from each other to change the pitch. The sound wave 404 is synthesized in this way by repeating the same pitch bell 403 at a pitch larger than the original pitch in order to increase the original pitch of the sound wave 401. It should be noted that the phase remains as a result of this superposition operation due to the previous windowing operation performed taking into account the characteristic phase difference Δj. Thus, the pitch bell 403 can be used as a building block to synthesize quasi-natural speech.

FIG. 5 shows one application for processing natural speech. In step 501, natural speech of a known speaker is input. This corresponds to the input of the sound wave 401 as shown in FIG. This natural speech is windowed by the raised cosine 402 (see FIG. 4) or by another suitable window function centered on the zero phase j ₀ + Δj.

  In this way, the natural sound is broken down into pitch bells provided in step 503 (see pitch bell 403 in FIG. 4).

  In step 504, the pitch bell provided in step 503 is used as a “building block” for speech synthesis. One method of processing is to omit the specific pitch bell or repeat the specific pitch bell without changing the pitch bell itself. For example, if the pitch bell is omitted every fourth, this increases the speed of the voice by 25% without changing the voice sound differently. Similarly, the voice speed can be reduced by repeating a specific pitch bell.

  Alternatively or additionally, the pitch bell distance is modified to increase or decrease the pitch.

  In step 505, the processed pitch bells are overlaid to produce a synthesized speech waveform that sounds quasi-naturally.

  FIG. 6 is an example of another application of the present invention. In step 601, audio information is provided. The voice information includes phonemes, phoneme durations and pitch information. Such speech information can be generated from text by modern text speech processing systems.

  In step 602, a diphone is extracted from this audio information provided in step 601. In step 603, the position and pitch contour of the required diphone on the time axis are determined based on the information provided in step 601.

  In step 604, a pitch bell is selected according to the timing and pitch conditions as determined in step 603. In step 605, the selected pitch bells are concatenated to provide a quasi-natural audio output.

  This procedure is further illustrated by an example as shown in FIGS.

  FIG. 7 shows a phonetic notation for the sentence “HELLO WORD!”. The first column 701 of the notation includes phonemes in the SAMPA standard notation. The second column 702 shows the duration of individual phonemes in milliseconds. The third column has pitch information. Pitch movement is indicated by two numbers: the position as a percentage of the phoneme duration, and the pitch frequency (Hz).

  Compositing begins with a search in the previously generated diphone database. A diphone is cut out from actual speech and consists of a transition from one phoneme to another. All possible phoneme combinations for a particular language must be stored in this database, along with some extra information such as phoneme boundaries. If there are multiple databases of different speakers, the selection of a specific speaker can be a separate input to the synthesizer.

  FIG. 8 shows the transition of all phonemes in the diphone for the sentence “HELLO WORLD!”, Ie column 701 in FIG.

  FIG. 9 shows the calculation results of the position of the phone boundary, the diphone boundary, and the pitch period position to be synthesized. Phoneme boundaries are calculated by adding the phoneme duration. For example, the phoneme “h” begins after 100 ms of silence. The phoneme “schwa” starts after 155 ms = 100 ms + 55 ms, and so on.

  The diphone boundary is retrieved from the database as a percentage of the phoneme duration. Both the individual phoneme locations and the diphone boundaries are shown in the upper drawing 901 of FIG. 9, showing the starting point of the diphone. This starting point is calculated based on the phoneme duration given by column 702 and the percentage of phoneme duration given by column 703.

Drawing 902 of FIG. 9 shows the pitch profile of “HELLO WORD!”. The pitch contour is determined based on the pitch information included in the column 703 (see FIG. 7). For example, if the current pitch position is 0.25 seconds, the pitch period will be 50% of the first '|' phoneme. A corresponding pitch exists between 133 Hz and 139 Hz. It can be calculated with the following linear equation:

ここで、ｆは周波数（ｋＨｚ）である。この考えは、ＥＲＢ−レートスケールにおけるピッチ変化は、線形的な変化として人間の耳で知覚されるということである。 The next pitch position is 0.2500 + 1 / 135.5 = 0.2574 seconds. It is also possible to use a non-linear function (such as ERB-rate scale) for this calculation. ERB (equivalent rectangular bandwidth) is a measure derived from psychoacoustic measurements (Glasberg and Mooore (1990)) and gives a better representation by considering the mask characteristics of the human ear. The formula for frequency to ERB conversion is:

Here, f is a frequency (kHz). The idea is that pitch changes in the ERB-rate scale are perceived by the human ear as linear changes.

  Note that the unvoiced area is also marked with the pitch period position, even if the unvoiced part has no pitch.

  The changing pitch is given by the pitch profile in drawing 902 and is also shown in drawing 901 by vertical lines 903 having changing spacing. The greater the distance between the two lines 903, the smaller the pitch. The phonemes, diphones, and pitch information given in drawings 901 and 902 are the reference for the speech to be synthesized. Diphone samples, i.e., pitch bells (see pitch bell 403 in FIG. 4) are retrieved from the diphone database. For each diphone, such a number of pitch bells for that diphone are connected, the number of pitch bells corresponding to the duration of the diphone, and the spacing between pitch bells as given by the pitch contours of 902 drawings Corresponds to the required pitch frequency.

  The result of all pitch bell connections is a quasi-natural synthesized speech. This is because the present invention prevents phase related discontinuities at the diphone boundary. This is in contrast to the prior art where such discontinuities are unavoidable due to phase mismatch of the pitch period.

  In addition, since the duration time of both sides of each diphone is appropriately adjusted, the prosody (pitch / duration time) is appropriate. The pitch is also consistent with the desired pitch contour function.

  FIG. 10 shows a device 950 (eg, a personal computer) programmed to implement the present invention. The device 950 has a speech analysis module 951 that serves to determine the characteristic phase difference Δj. For this purpose, the voice analysis module 951 has a storage unit 952 for storing one diphone voice wave. One diphone is sufficient to obtain a constant phase difference Δj.

  Further, the voice analysis module 951 has a low-pass filter module 953. The low-pass filter module 953 has a cutoff frequency of approximately 150 Hz or another appropriate cutoff frequency for the purpose of extracting the first harmonic of the diphone stored in the storage unit 952.

  Module 954 of device 950 serves to determine the distance between the maximum energy position of the diphone in the specified period and its zero overtone zero position (this distance is converted to a phase difference Δj). This can be done by determining the phase difference between the zero phase given by the positive zero crossing of the first harmonic and the maximum value of the diphone within the harmonic period, as shown in the example of FIG. it can.

  As a result of the speech analysis, speech analysis module 951 provides a characteristic phase difference Δj and thus a period position for all diphones in the database, where, for example, a raised cosine window is centered to obtain the pitch bell. . The phase difference Δj is stored in the storage unit 955.

  The device 950 further includes a speech synthesis module 956. As shown in FIG. 2, the speech synthesis module 956 includes a storage unit 957 for storing a pitch bell, that is, a diphone sample windowed by a window function. Note that the storage unit 957 does not necessarily have to be a pitch bell. All diphones can be stored with period position information, or the diphones can be monotonized to a constant pitch. In this way, it is possible to retrieve the pitch bell from the database by using the window function of the synthesis module.

  Module 958 is responsible for selecting the pitch bell and adapting the pitch bell to the required pitch. This is done based on control information supplied to module 958.

  Module 959 serves to concatenate the pitch bells selected in module 958 to provide audio output by module 960.

6 shows a flowchart of a method for obtaining a phase difference between a diphone and its first overtone. Fig. 2 shows a signal diagram illustrating an example application of the method of Fig. 1; 2 shows an embodiment of the method of the invention for synthesizing speech. Fig. 4 shows an example application of the method of Fig. 3; Figure 2 shows an application of the present invention for processing natural speech. Fig. 2 shows an application of the present invention for text speech. It is an example of the file containing audio | voice information. It is an example of the file containing the diphone information extracted from the file of FIG. The result of the process of the file of FIG.7 and FIG.8 is shown. 1 shows a block diagram of a speech analysis and synthesis apparatus according to the present invention.

Explanation of symbols

Sonic 201
First overtone 202
Maximum value 203
Maximum value 204
Sound wave 401
Raised Cosine 402
Pitch bell 403
Sonic 404
Row 701
Row 702
Row 703
Drawing 901
Drawing 902
Device 950
Speech analysis module 951
Storage unit 952
Low pass filter module 953
Module 954
Storage unit 955
Speech synthesis module 956
Storage unit 957
Module 958
Module 959
Module 960

Claims (20)

A method for speech analysis, the method comprising:
-Steps for the input of audio signals;
-Obtaining a first overtone of said audio signal;
-Determining a phase difference between the audio signal and the first overtone;
Having a method. Obtaining the phase difference comprises:
-A step for determining the position of the maximum value of the audio signal;
The method of claim 1, comprising determining the phase difference between the maximum value and a phase zero of the first overtone of the audio signal.   The method of claim 1 or 2, wherein the audio signal is a diphone signal. A method of synthesizing speech, the method comprising:
-Selecting a diphone sample windowed by a window function centered on a phase angle determined by the phase difference between the speech signal and the first harmonic of the speech signal;
-Concatenating the windowed and selected diphone samples;
Having a method.   The method of claim 4, wherein the audio signal is a diphone signal.   6. A method according to claim 4 or 5, wherein the window function is a raised cosine or a triangular window.   7. A method according to claim 4, 5 or 6, further comprising the step of inputting information representing a diphone and pitch profile and forming a basis for selection of the windowed diphone sample.   8. A method according to any one of claims 4 to 7, wherein the information is provided from a language processing module of a text speech system. -Voice input step,
-Windowing the sound with the window function to obtain a windowed diphone sample;
The method of any one of claims 4 to 8, further comprising:   Computer program for carrying out the method according to any one of claims 1 to 9. -Means for input of audio signals;
-Means for obtaining the first harmonic of the audio signal;
-Means for determining a phase difference between the audio signal and the first overtone;
A voice analysis apparatus having   12. The means for obtaining the phase difference obtains a maximum value of the audio signal and obtains a phase zero of the audio signal in order to obtain a phase difference between the maximum value of the audio signal and the phase zero. Voice analysis device.   The voice analysis apparatus according to claim 11 or 12, wherein the voice signal is a diphone signal. Means for selecting a diphone sample windowed by a window function centered on a phase angle determined by a phase difference between an audio signal and the first harmonic of the audio signal;
-Means for concatenating the windowed and selected diphone signals;
A speech synthesizer.   The speech synthesis apparatus according to claim 14, wherein the speech signal is a diphone signal.   The speech synthesizer according to claim 14 or 15, wherein the window function is a raised cosine or a triangular window. Means for inputting information representing the diphone and pitch contour;
17. The speech synthesizer according to claim 14, 15 or 16, wherein the means for selecting the windowed diphone selects based on the information. -Language processing means for providing information representing the diphone and pitch contour;
Means for selecting, based on the information, diphone samples windowed by a window function centered on a phase angle determined by a phase difference between the speech signal and the first harmonic of the speech signal Speech synthesis means comprising: and means for connecting the windowed and selected diphone samples;
A text voice system.   The text-to-speech system of claim 18 wherein the window function is a raised cosine or a triangular window. -Means for input of a signal having a natural speech signal;
-To provide a windowed diphone sample, the natural speech signal is filtered by a window function centered on the phase angle determined by the phase difference between the speech signal and the first harmonic of the speech signal. Means for windowing,
-Means for processing said windowed diphone samples;
-Means for concatenating the windowed and selected diphone signals;
A voice processing system.

Images