EP1246163B1 - Speech synthesis method and speech synthesizer - Google Patents

Speech synthesis method and speech synthesizer Download PDF

Info

Publication number
EP1246163B1
EP1246163B1 EP02252159A EP02252159A EP1246163B1 EP 1246163 B1 EP1246163 B1 EP 1246163B1 EP 02252159 A EP02252159 A EP 02252159A EP 02252159 A EP02252159 A EP 02252159A EP 1246163 B1 EP1246163 B1 EP 1246163B1
Authority
EP
European Patent Office
Prior art keywords
formant
speech
pitch
waveforms
functions
Prior art date
Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
Expired - Lifetime
Application number
EP02252159A
Other languages
German (de)
French (fr)
Other versions
EP1246163A2 (en
EP1246163A3 (en
Inventor
Masami C/O Intellectual Property Div. Akamine
Takehiko c/o Intellectual Property Div Kagoshima
Current Assignee (The listed assignees may be inaccurate. Google has not performed a legal analysis and makes no representation or warranty as to the accuracy of the list.)
Toshiba Corp
Original Assignee
Toshiba Corp
Priority date (The priority date is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the date listed.)
Filing date
Publication date
Application filed by Toshiba Corp filed Critical Toshiba Corp
Publication of EP1246163A2 publication Critical patent/EP1246163A2/en
Publication of EP1246163A3 publication Critical patent/EP1246163A3/en
Application granted granted Critical
Publication of EP1246163B1 publication Critical patent/EP1246163B1/en
Anticipated expiration legal-status Critical
Expired - Lifetime legal-status Critical Current

Links

Images

Classifications

    • GPHYSICS
    • G10MUSICAL INSTRUMENTS; ACOUSTICS
    • G10LSPEECH ANALYSIS TECHNIQUES OR SPEECH SYNTHESIS; SPEECH RECOGNITION; SPEECH OR VOICE PROCESSING TECHNIQUES; SPEECH OR AUDIO CODING OR DECODING
    • G10L13/00Speech synthesis; Text to speech systems
    • G10L13/02Methods for producing synthetic speech; Speech synthesisers
    • G10L13/04Details of speech synthesis systems, e.g. synthesiser structure or memory management
    • GPHYSICS
    • G10MUSICAL INSTRUMENTS; ACOUSTICS
    • G10LSPEECH ANALYSIS TECHNIQUES OR SPEECH SYNTHESIS; SPEECH RECOGNITION; SPEECH OR VOICE PROCESSING TECHNIQUES; SPEECH OR AUDIO CODING OR DECODING
    • G10L13/00Speech synthesis; Text to speech systems
    • G10L13/02Methods for producing synthetic speech; Speech synthesisers
    • G10L13/027Concept to speech synthesisers; Generation of natural phrases from machine-based concepts
    • GPHYSICS
    • G10MUSICAL INSTRUMENTS; ACOUSTICS
    • G10LSPEECH ANALYSIS TECHNIQUES OR SPEECH SYNTHESIS; SPEECH RECOGNITION; SPEECH OR VOICE PROCESSING TECHNIQUES; SPEECH OR AUDIO CODING OR DECODING
    • G10L25/00Speech or voice analysis techniques not restricted to a single one of groups G10L15/00 - G10L21/00
    • G10L25/27Speech or voice analysis techniques not restricted to a single one of groups G10L15/00 - G10L21/00 characterised by the analysis technique

Definitions

  • the present invention relates to a text-to-speech synthesis, particularly a speech synthesis method of generating a synthesized speech from information such as phoneme symbol string, pitch, and phoneme duration.
  • Text-to-speech synthesis means producing artificial speech from text.
  • This text-to-speech synthesis system comprises three stages: a linguistic processor, prosody processor and speech signal generator.
  • the input text is subjected to morphological analysis or syntax analysis in a linguistic processor, and then the process of accent and intonation is performed in the prosody processor, and information such as phoneme symbol string, pitch pattern (the change pattern of voice pitch), and the phoneme duration is output.
  • a speech signal generator that is, speech synthesizer synthesizes a speech signal from information such as phoneme symbol strings, pitch patterns and phoneme duration.
  • synthesis units basic characteristic parameters units such as phone, syllable, diphone and triphone are stored in a storage and selectively read out.
  • the read-out synthesis units are connected, with their pitches and phoneme durations being controlled, whereby a speech synthesis is performed.
  • PSOLA Puls-Synchronous Overlap-add
  • An alternative method involves a formant synthesis.
  • This system was designed to emulate the way humans speak.
  • the formant synthesis system generates a speech signal by exciting a filter modeling the property of vocal tract with a speech source signal obtained by modeling a signal generated from the vocal cords.
  • the phonemes (/a/, /i/, /u/, etc) and voice variety (male voice, female voice, etc.) of synthesized speech are determined by combining the formant frequency with the bandwidth. Therefore, the synthesis unit information is generated by combining the formant frequency with the bandwidth, rather than the waveform. Since the formant synthesis system can control parameters relating to phoneme and voice variety, it is advantageous in that variations in the voice variety and so on can be flexibly controlled. However, the precision of modeling lacks, which is disadvantageous.
  • the formant synthesis system cannot mimic the finely detailed spectrum of real speech signal because only the formant frequency and bandwidth are used, meaning that speech quality is unacceptable.
  • a speech synthesis method comprising the steps of:
  • the invention also provides a speech synthesizer supplied with a pitch pattern, phoneme duration and phoneme symbol string, comprising:
  • the present invention can be implemented either in hardware or on software in a general purpose computer. Further the present invention can be implemented in a combination of hardware and software. The present invention can also be implemented by a single processing apparatus or a distributed network of processing apparatuses.
  • the present invention can be implemented by software, the present invention encompasses computer code provided to a general purpose computer on any suitable carrier medium.
  • the carrier medium can comprise any storage medium such as a floppy disk, a CD ROM, a magnetic device or a programmable memory device, or any transient medium such as any signal e.g. an electrical, optical or microwave signal.
  • FIG. 1 shows a configuration of a speech synthesizer realizing a speech synthesis method according to the first embodiment of the present invention.
  • the speech synthesizer receives pitch pattern 306, phoneme duration 307 and phoneme symbol string 308 and outputs a synthesized speech signal 305.
  • the speech synthesizer comprises a voiced speech synthesizer 31 and an unvoiced sound synthesizer 32, and generates the synthesized speech signal 305 by adding the unvoiced speech signal 304 and voiced speech signal 303 output from the synthesizers, respectively.
  • the unvoiced speech synthesizer 32 generates the unvoiced speech signal 304 referring to phoneme duration 307 and phoneme symbol string 308, when the phoneme is mainly an unvoiced consonant and voiced fricative sound,
  • the unvoiced speech synthesizer 32 can be realized by a conventional technique, such as the method of exciting an LPC synthesis filter with white noise.
  • the voiced speech synthesizer 31 comprises a pitch mark generator 33, a pitch waveform generator 34 and a waveform superposing device 35.
  • the pitch mark generator 33 generates pitch marks 302 as shown in FIG. 2 referring to the pitch pattern 306 and phoneme duration 307.
  • the pitch marks 302 indicate positions at which the pitch waveforms 301 are superposed. The interval between the pitch marks correspond to the pitch period.
  • the pitch waveform generator 34 generates pitch waveforms 301 corresponding to the pitch marks 302 as shown in FIG. 2, referring to the pitch pattern 306, phoneme duration 307 and phoneme symbol string 308.
  • the waveform superposing device 35 generates a voiced speech signal 303 by superposing, at positions of the pitch marks 302, the pitch waveforms corresponding to the pitch marks 302.
  • the pitch waveform generator 34 comprises a formant parameter storage 41, a parameter selector 42 and sine wave generators 43, 44 and 45 as shown in FIG. 3.
  • the formant parameters are stored in the formant parameter storage 41 in units of a synthesis unit.
  • FIG. 4 indicates an example of formant parameters of phonemes /a/.
  • the phonemes /a/ comprise three frames each including three formants.
  • Formant frequency, formant phase and windowing functions are stored in the formant parameter storage 41 as parameters to express the characteristics of each formant.
  • the formant parameter selector 42 selects and reads formant parameters 401 for one frame corresponding to the pitch marks 302 from the formant parameter storage 41, referring to the pitch pattern 306, phoneme duration 307 and phoneme symbol string 308 which are input to the pitch waveform generator 34.
  • the parameters corresponding to the formant number 1 are read out from the formant parameter storage 41 as formant frequency 402, formant phase 403 and windowing functions 411.
  • the parameters corresponding to the formant number 2 are read out from the formant parameter storage 41 as formant frequency 404, formant phase 405 and windowing functions 412.
  • the parameters corresponding to the formant number 3 are read out from the formant parameter storage 41 as formant frequency 406, formant phase 407 and windowing functions 413.
  • the sine wave generator 43 generates sine wave 408 according to the formant frequency 402 and formant phase 403.
  • the sine wave 408 is subjected to the windowing functions 411 to generate a formant waveform 414.
  • the sine wave generator 44 outputs sine wave 409 based on the formant frequency 404 and formant phase 405. This sine wave 409 is multiplied by the windowing function 412 to generate a formant waveform 415.
  • the sine wave generator 45 outputs a sine wave 410 based on the formant frequency 406 and formant phase 407. This sine wave 410 is multiplied by the windowing functions 413 to generate a formant waveform 416.
  • Adding the formant waveforms 414, 415 and 416 generates the pitch waveform 301.
  • Examples of the sine waves, windowing functions, formant waveforms and pitch waveforms are shown in FIG. 6.
  • the power spectrums of these waveforms are shown in FIG. 7.
  • the abscissa axis expresses time and the ordinate axes express amplitude.
  • the abscissa axes express frequency and the ordinate axes express amplitude.
  • the sine wave becomes a line spectrum having a sharp peak
  • the windowing function becomes the spectrum concentrated on a low frequency domain.
  • the windowing (multiplication) in the time domain corresponds to convolution in the frequency domain.
  • the spectrum of formant waveform indicates a shape obtained by shifting the spectrum of windowing function to the position of frequency of the sine wave in parallel. Therefore, controlling the frequency or phase of the sine wave can change the center frequency or phase of the formant of the pitch waveform. Controlling the shape of the windowing function can change the spectrum shape of the formant of the pitch waveform.
  • the center frequency, phase and spectrum shape of the formant can be independently controlled for each formant, a highly flexible model can be realized. Further, since the windowing function allows the highly detailed structure of spectrum to be expressed, the synthesized speech can approximate to a high accuracy the spectrum structure of natural voice, thus producing the feeling of natural voice.
  • the pitch waveform generator 34 of the second embodiment of the present invention will be described referring to FIG. 8.
  • like reference numerals are used to designate like structural elements corresponding to those in the first embodiment. Only the portions that differ will be described.
  • the windowing functions are developed by basis functions, and a group of weighting factors is stored in the storage 51 instead of storing the windowing functions as the formant parameters.
  • the windowing function generator 56 newly added generates windowing functions from the weighting factors.
  • the windowing function is obtained by the sum of three basis functions weighted by the weighting factors.
  • a set of three factors is stored in the storage 51 as a set of windowing function weighting factors.
  • the parameter selector 42 outputs the formant frequencies 402, 404 and 406 and formant phases 403, 405 and 407 in the selected formant parameters 501 to the sine wave generators 43, 44 and 45, and outputs a set of windowing function weighting factors 517, 518 and 519 to the windowing function generator 56.
  • the basis functions may use DCT basis, and may use basis functions generated by subjecting the windowing functions to KL-expansion.
  • the basis order is set to 3, but it is not limited to 3. Developing the windowing functions to the basis functions reduces the memory capacity of the formant parameter storage.
  • the pitch waveform generator 34 of the third embodiment of the present invention will be described referring to FIG. 9.
  • like reference numerals are used to designate like structural elements corresponding to those in the first embodiment. Only the portions that differ will be described.
  • a parameter transformer 67 is newly added, and the formant parameters are varied according to the pitch pattern 306.
  • the parameter transformer 67 outputs formant frequency 720, formant phase 721, windowing function 717, formant frequency 722, formant phase 723, windowing function 718, formant frequency 724, formant phase 725, and windowing function 719 by changing the formant frequency 402, formant phase 403, windowing function 411, formant frequency 404, formant phase 405, windowing function 412, formant frequency 406, formant phase 407, and windowing function 413 according to the pitch pattern 306. All parameters may be changed, and a part of the parameters may be changed.
  • FIG. 10 shows an example of a control function when the parameter transformer 67 controls the formant frequency according to the pitch period.
  • Such control function may be set for every phoneme, every frame or every formant number.
  • the formant frequency can be controlled according to the pitch period, by inputting such control function to the parameter transformer 67.
  • a control function to control the differential value and ratio of the input/output formant frequency may be used instead of the formant frequency itself.
  • FIG. 11 shows the control function to control the power of formant by multiplying the gain corresponding to the pitch period by the windowing functions. It is possible to model the spectrum change of speech according to the change of the pitch period by inputting such a control function to the parameter transformer 67 and changing the parameters according to the pitch period. As a result, it is possible to generate high quality synthesized speech which is not dependent on the pitch of voice.
  • the formant parameters may be changed according to a kind of preceding or following phoneme. As a result, it is possible to model a variable speech spectrum based on the phoneme environment, and to improve speech quality.
  • the voice variety information 309 inputted to the parameter transformer 67 from an external device may be altered to produce different parameters. In this case, it is possible to generate synthesized speech of various voice qualities.
  • FIG. 12 shows an example of changing the voice pitch by changing the formant frequency. If all formant frequencies are converted by the control function (a), since the formant is shifted to a high frequency domain, a thin voice is generated. The control function (b) generates a somewhat thin voice. If the control function (d) is used, since the formant frequency shifts to a low frequency domain, a deep voice is generated. The control function (c) generates a deeper voice.
  • the pitch waveform generator 34 of the fourth embodiment of the present invention will be described referring to FIG. 13.
  • like reference numerals are used to designate like structural elements corresponding to those in the first embodiment. Only the portions that differ will be described.
  • the parameter smoothing device 77 is added to smooth the parameters so that the time based change of each formant parameters is smoothed.
  • the parameter smoothing device 77 outputs formant frequency 820, formant phase 821, windowing function 817, formant frequency 822, formant phase 823, windowing function 818, formant frequency 824, formant phase 825 and windowing function 819 by smoothing the formant frequency 402, formant phase 403, windowing function 411, formant frequency 404, formant phase 405, windowing function 412, formant frequency 406, formant phase 407 and windowing function 413, respectively. All parameters may be smoothed, or merely partly smoothed.
  • FIG. 14 shows an example of smoothing of formant.
  • represents the formant frequencies 402, 404 and 406 before smoothing.
  • the smoothed formant frequencies 820, 822 and 824 indicated by ⁇ are generated by performing smoothing so that a change between corresponding formant frequencies of the current frame and the preceding or following frame are smoothed.
  • the formant corresponding to the formant frequency 404 becomes extinct, as shown by ⁇ in FIG. 15A.
  • the formant frequency 822 is generated by adding formants as shown by ⁇ .
  • the power of the windowing function 818 corresponding to the formant frequency 822 is attenuated as shown in FIG. 15B, to prevent the formant power from discontinuity.
  • FIGS. 16A and 16B show examples of windowing function position smoothing. Smoothing the windowing function positions so that the peak position of the windowing function 411 varies between frames smoothly generates the windowing function 817. Further, the shape and power of the windowing function may also be smoothed.
  • the above embodiment is explained for 3 formants.
  • the number of formants is not limited to 3, and may be changed every frame.
  • the sine wave generator of the embodiments of the present invention outputs a sine wave.
  • a waveform having a near-line power spectrum may be used instead of a complete sine wave.
  • the sine wave generator comprises a table in order to reduce computation cost, for example, the complete sine wave is not obtained because of error.
  • the spectrum of formant waveform may not always indicate the peak of the spectrum of speech signal, and the spectrum of the pitch waveform, which is the sum of plural formant waveforms, expresses a spectrum of speech.
  • the above embodiment of the present invention provides a synthesizer for text-to-speech synthesis, but another embodiment of the present invention provides a decoder for speed coding.
  • the encoder obtains, from the speech signal, formant parameters such as formant frequency, formant phase, windowing function, etc. and pitch period, etc. by analysis, and encodes them and transmits or store codes.
  • the decoder decodes the formant parameters and pitch periods, and reconstructs the speech signal similarly to the above synthesizer.
  • FIG. 17A show a flowchart of the speech synthesis process
  • FIG. 17B shows a flowchart of the voiced speech generation process of the speech synthesis process
  • FIG. 17C shows a flowchart of the pitch waveform generation process of the voiced speech generation process of FIG. 17B.
  • the pitch pattern 306, phoneme duration 307 and phoneme symbol string 308 are input (S11).
  • the voiced speech signal 303 is generated based on the pitch pattern 306, phoneme duration 307 and phoneme symbol string 308 (S12).
  • the unvoiced speech signal 304 is generated referring to the phoneme duration 307 and phoneme symbol string 308 (S13).
  • the voiced speech signal and unvoiced speech signal are added to generate the synthesized speech signal 305 (S14).
  • the pitch mark 302 is generated referring to the pitch pattern 306 and phoneme duration 307 (S21).
  • the pitch waveforms 301 are generated corresponding to the pitch marks 302, referring to the pitch pattern 306, phoneme duration 307 and phoneme symbol string 308 (S22).
  • the pitch waveforms 301 are superposed in the positions indicated by the pitch marks 302 to generate a voiced speech (S23).
  • the formant parameters 401 for 1 frame corresponding to the pitch mark 302 is selected from the formant parameter storage 41 referring to the pitch pattern 306, phoneme duration 307 and phoneme symbol string 308 (S31).
  • Plural sine waves are generated according to the formant frequencies and formant phases corresponding to the formant numbers of the selected formant parameters 401 (S32).
  • the formant waveforms 414, 415 and 416 are generated by multiplying the plural sine waves by the windowing functions (S33).
  • the formant waveforms are added to generate a pitch waveform (S34).
  • the formant frequency and formant shape are independently controlled for every formant, it is possible to express the spectrum change of speech due to the pitch period variation and voice variety change between the formants, and realize highly flexibility speech synthesis. Because the shape of the windowing functions can express the detailed structure of the formant spectrum, high quality synthesized speech having a natural voice feeling can be generated.

Landscapes

  • Engineering & Computer Science (AREA)
  • Computational Linguistics (AREA)
  • Health & Medical Sciences (AREA)
  • Audiology, Speech & Language Pathology (AREA)
  • Human Computer Interaction (AREA)
  • Physics & Mathematics (AREA)
  • Acoustics & Sound (AREA)
  • Multimedia (AREA)
  • Electrophonic Musical Instruments (AREA)

Description

  • The present invention relates to a text-to-speech synthesis, particularly a speech synthesis method of generating a synthesized speech from information such as phoneme symbol string, pitch, and phoneme duration.
  • "Text-to-speech synthesis" means producing artificial speech from text. This text-to-speech synthesis system comprises three stages: a linguistic processor, prosody processor and speech signal generator.
  • At first, the input text is subjected to morphological analysis or syntax analysis in a linguistic processor, and then the process of accent and intonation is performed in the prosody processor, and information such as phoneme symbol string, pitch pattern (the change pattern of voice pitch), and the phoneme duration is output. A speech signal generator, that is, speech synthesizer synthesizes a speech signal from information such as phoneme symbol strings, pitch patterns and phoneme duration.
  • According to the operational principle of a speech synthesis apparatus for speech-synthesizing a given phoneme symbol string, basic characteristic parameters units (hereinafter referred to as "synthesis units") such as phone, syllable, diphone and triphone are stored in a storage and selectively read out.
    The read-out synthesis units are connected, with their pitches and phoneme durations being controlled, whereby a speech synthesis is performed.
  • As a method for generating a speech signal of a desired pitch pattern and phoneme duration from information of synthesis units, the PSOLA (Pitch-Synchronous Overlap-add) method is known. It is known that synthesized speech based on PSOLA reduces speech quality degradation due to pitch period variation, and improves speech quality, when the pitch period variation is small. However, PSOLA has a problem in that speech quality deteriorates when the pitch period variation is large. Further, there is a problem that distortion occurs in the spectrum due to the smoothing process performed when a discontinuous spectrum occurs when synthesis units are combined, resulting in deterioration in the speech quality. Furthermore, PSOLA makes change of voice variety difficult and lack flexibility since the waveform itself is used as a synthesis unit.
  • An alternative method involves a formant synthesis. This system was designed to emulate the way humans speak. The formant synthesis system generates a speech signal by exciting a filter modeling the property of vocal tract with a speech source signal obtained by modeling a signal generated from the vocal cords.
  • In this system, the phonemes (/a/, /i/, /u/, etc) and voice variety (male voice, female voice, etc.) of synthesized speech are determined by combining the formant frequency with the bandwidth. Therefore, the synthesis unit information is generated by combining the formant frequency with the bandwidth, rather than the waveform. Since the formant synthesis system can control parameters relating to phoneme and voice variety, it is advantageous in that variations in the voice variety and so on can be flexibly controlled. However, the precision of modeling lacks, which is disadvantageous.
  • In other words, the formant synthesis system cannot mimic the finely detailed spectrum of real speech signal because only the formant frequency and bandwidth are used, meaning that speech quality is unacceptable.
  • The paper entitled "Control of Spectral Dynamics in Concatenative Speech Synthesis" by Wouters J and Macon M.W. from IEEE Transactions on Speech and Audio Processing, vol. 9, no. 1, January 2001, pages 30 to 38 describes a system which uses sinusoidal analysis and synthesis, sinusoidal and all-pole modelling along with TD-PSOLA to control pitch and duration of diphone units. The paper entitled "Time-domain formant-wave-function synthesis" by Rodet X. from Computer Music Journal, vol. 8, 1984, pages 9 to 11 describes a time-domain formant-wave-function synthesis for directly calculating the amplitude of the waveform of a signal as a function of time.
  • It is an object of the present invention to provide a speech synthesizer, which improves a speech quality and can flexibly control voice variety.
  • According to the invention, there is provided a speech synthesis method comprising the steps of:
  • generating formant parameters;
  • selecting predetermined formant parameters from the formant parameters according to a phoneme symbol string;
  • generating a plurality of sine waves based on the formant frequency corresponding to the formant parameters selected;
  • multiplying the sine waves by the windowing functions corresponding to the selected formant parameters, respectively, to generate a plurality of formant waveforms;
  • adding the formant waveforms to generate a plurality of pitch waveforms; and
  • superposing the pitch waveforms according to a pitch period to generate a speech signal, characterized by generating the windowing functions by adding basis functions weighted by weighting factors.
  • The invention also provides a speech synthesizer supplied with a pitch pattern, phoneme duration and phoneme symbol string, comprising:
  • means (33) for generating pitch marks referring to the pitch pattern and phoneme duration;
  • means (51) for generating formant parameters,
  • means (52) for selecting the formant parameters for one frame corresponding to the phoneme symbol string,
  • means (43-45) for generating sine waves according to formant frequencies of the read formant parameters,
  • means for multiplying the sine waves by windowing functions of the selected formant parameters to generate formant waveforms,
  • means for adding the formant waveforms to generate pitch waveforms, means (35) for superposing the pitch waveforms on the pitch marks to generate a voiced speech signal;
  • means (32) for generating an unvoiced speech; and
  • means for adding the voiced speech and the unvoiced speech to generate synthesized speech, characterized by means (56) for generating the windowing functions by adding basis functions weighted by the weighting factors.
  • The present invention can be implemented either in hardware or on software in a general purpose computer. Further the present invention can be implemented in a combination of hardware and software. The present invention can also be implemented by a single processing apparatus or a distributed network of processing apparatuses.
  • Since the present invention can be implemented by software, the present invention encompasses computer code provided to a general purpose computer on any suitable carrier medium. The carrier medium can comprise any storage medium such as a floppy disk, a CD ROM, a magnetic device or a programmable memory device, or any transient medium such as any signal e.g. an electrical, optical or microwave signal.
  • This summary of the invention does not necessarily describe all necessary features so that the invention may also be a sub-combination of these described features.
  • The invention can be more fully understood from the following detailed description when taken in conjunction with the accompanying drawings, in which:
  • FIG. 1 shows a block diagram of a speech synthesizer of an embodiment of the present invention;
  • FIG. 2 shows a process of generating voiced speech by superposing pitch waveforms;
  • FIG. 3 shows a block diagram of pitch waveform generation club related to the first embodiment of the present invention;
  • FIG. 4 shows an example of formant parameters;
  • FIG. 5 shows another example of formant parameters;
  • FIG. 6 shows sine waves, windowing functions, formant waveforms and pitch waveforms;
  • FIG. 7 shows power spectrums of sine waves, windowing functions, formant waveforms and pitch waveform;
  • FIG. 8 shows a block diagram of a pitch waveform generator of the second embodiment of the present invention;
  • FIG. 9 shows a block diagram of a pitch waveform generator related to the third embodiment of the present invention;
  • FIG. 10 shows a control function of the formant frequency;
  • FIG. 11 shows a control function of the formant gain;
  • FIG. 12 shows a mapping function of the formant frequency for use in voice variety conversion;
  • FIG. 13 shows a block diagram of a pitch waveform generator of the fourth embodiment of the present invention;
  • FIG. 14 shows a diagram for explaining smoothing of the formant frequency;
  • FIGS. 15A and 15B show another diagram for explaining smoothing of the formant frequency;
  • FIGS. 16A and 16B show smoothing states of windowing functions; and
  • FIGS. 17A, 17B and 17C show flow charts for explaining processes of the speech synthesizer of the present invention.
  • There will now be described embodiments of the present invention in conjunction with accompanying drawings.
  • FIG. 1 shows a configuration of a speech synthesizer realizing a speech synthesis method according to the first embodiment of the present invention. The speech synthesizer receives pitch pattern 306, phoneme duration 307 and phoneme symbol string 308 and outputs a synthesized speech signal 305. The speech synthesizer comprises a voiced speech synthesizer 31 and an unvoiced sound synthesizer 32, and generates the synthesized speech signal 305 by adding the unvoiced speech signal 304 and voiced speech signal 303 output from the synthesizers, respectively.
  • The unvoiced speech synthesizer 32 generates the unvoiced speech signal 304 referring to phoneme duration 307 and phoneme symbol string 308, when the phoneme is mainly an unvoiced consonant and voiced fricative sound, The unvoiced speech synthesizer 32 can be realized by a conventional technique, such as the method of exciting an LPC synthesis filter with white noise.
  • The voiced speech synthesizer 31 comprises a pitch mark generator 33, a pitch waveform generator 34 and a waveform superposing device 35. The pitch mark generator 33 generates pitch marks 302 as shown in FIG. 2 referring to the pitch pattern 306 and phoneme duration 307. The pitch marks 302 indicate positions at which the pitch waveforms 301 are superposed. The interval between the pitch marks correspond to the pitch period. The pitch waveform generator 34 generates pitch waveforms 301 corresponding to the pitch marks 302 as shown in FIG. 2, referring to the pitch pattern 306, phoneme duration 307 and phoneme symbol string 308. The waveform superposing device 35 generates a voiced speech signal 303 by superposing, at positions of the pitch marks 302, the pitch waveforms corresponding to the pitch marks 302.
  • The configuration of the pitch waveform generator of FIG. 1 will be described in detail as follows.
  • The pitch waveform generator 34 comprises a formant parameter storage 41, a parameter selector 42 and sine wave generators 43, 44 and 45 as shown in FIG. 3. The formant parameters are stored in the formant parameter storage 41 in units of a synthesis unit.
  • FIG. 4 indicates an example of formant parameters of phonemes /a/. In this example, the phonemes /a/ comprise three frames each including three formants. Formant frequency, formant phase and windowing functions are stored in the formant parameter storage 41 as parameters to express the characteristics of each formant.
  • The formant parameter selector 42 selects and reads formant parameters 401 for one frame corresponding to the pitch marks 302 from the formant parameter storage 41, referring to the pitch pattern 306, phoneme duration 307 and phoneme symbol string 308 which are input to the pitch waveform generator 34.
  • The parameters corresponding to the formant number 1 are read out from the formant parameter storage 41 as formant frequency 402, formant phase 403 and windowing functions 411. The parameters corresponding to the formant number 2 are read out from the formant parameter storage 41 as formant frequency 404, formant phase 405 and windowing functions 412. The parameters corresponding to the formant number 3 are read out from the formant parameter storage 41 as formant frequency 406, formant phase 407 and windowing functions 413. The sine wave generator 43 generates sine wave 408 according to the formant frequency 402 and formant phase 403. The sine wave 408 is subjected to the windowing functions 411 to generate a formant waveform 414. The formant waveform y (t) is represented by the following equation. y (t) = w (t) * sin (ωt + ) where ω is the format frequency,  is the format phase 403, and w(t) is the windowing function 411.
  • The sine wave generator 44 outputs sine wave 409 based on the formant frequency 404 and formant phase 405. This sine wave 409 is multiplied by the windowing function 412 to generate a formant waveform 415. The sine wave generator 45 outputs a sine wave 410 based on the formant frequency 406 and formant phase 407. This sine wave 410 is multiplied by the windowing functions 413 to generate a formant waveform 416.
  • Adding the formant waveforms 414, 415 and 416 generates the pitch waveform 301. Examples of the sine waves, windowing functions, formant waveforms and pitch waveforms are shown in FIG. 6. The power spectrums of these waveforms are shown in FIG. 7. In FIG. 6, the abscissa axis expresses time and the ordinate axes express amplitude. In FIG. 7, the abscissa axes express frequency and the ordinate axes express amplitude.
  • The sine wave becomes a line spectrum having a sharp peak, and the windowing function becomes the spectrum concentrated on a low frequency domain. The windowing (multiplication) in the time domain corresponds to convolution in the frequency domain. For this reason, the spectrum of formant waveform indicates a shape obtained by shifting the spectrum of windowing function to the position of frequency of the sine wave in parallel. Therefore, controlling the frequency or phase of the sine wave can change the center frequency or phase of the formant of the pitch waveform. Controlling the shape of the windowing function can change the spectrum shape of the formant of the pitch waveform.
  • As thus described, since the center frequency, phase and spectrum shape of the formant can be independently controlled for each formant, a highly flexible model can be realized. Further, since the windowing function allows the highly detailed structure of spectrum to be expressed, the synthesized speech can approximate to a high accuracy the spectrum structure of natural voice, thus producing the feeling of natural voice.
  • The pitch waveform generator 34 of the second embodiment of the present invention will be described referring to FIG. 8. In the second embodiment, like reference numerals are used to designate like structural elements corresponding to those in the first embodiment. Only the portions that differ will be described.
  • In the present embodiment, the windowing functions are developed by basis functions, and a group of weighting factors is stored in the storage 51 instead of storing the windowing functions as the formant parameters. The windowing function generator 56 newly added generates windowing functions from the weighting factors.
  • An example of the formant parameters stored in the formant parameter storage 51 is shown in FIG. 5. In the example, the windowing function is obtained by the sum of three basis functions weighted by the weighting factors. A set of three factors is stored in the storage 51 as a set of windowing function weighting factors. The parameter selector 42 outputs the formant frequencies 402, 404 and 406 and formant phases 403, 405 and 407 in the selected formant parameters 501 to the sine wave generators 43, 44 and 45, and outputs a set of windowing function weighting factors 517, 518 and 519 to the windowing function generator 56.
  • The windowing function generator 56 generates windowing functions 511, 512 and 513 based on the windowing function weighting factors 517, 518 and 519 respectively. If the weighting factors are represented as a1, a2 and a3 and the basis functions as b1 (t), b2 (t) and b3 (t), the window function W(t) is expressed by the following equation. W(t) = a1 * b1 (t) + a2 * b2 (t) + a3 * b3 (t)
  • The basis functions may use DCT basis, and may use basis functions generated by subjecting the windowing functions to KL-expansion. In the present embodiment, the basis order is set to 3, but it is not limited to 3. Developing the windowing functions to the basis functions reduces the memory capacity of the formant parameter storage.
  • The pitch waveform generator 34 of the third embodiment of the present invention will be described referring to FIG. 9. In the third embodiment, like reference numerals are used to designate like structural elements corresponding to those in the first embodiment. Only the portions that differ will be described. In the present embodiment, a parameter transformer 67 is newly added, and the formant parameters are varied according to the pitch pattern 306.
  • The parameter transformer 67 outputs formant frequency 720, formant phase 721, windowing function 717, formant frequency 722, formant phase 723, windowing function 718, formant frequency 724, formant phase 725, and windowing function 719 by changing the formant frequency 402, formant phase 403, windowing function 411, formant frequency 404, formant phase 405, windowing function 412, formant frequency 406, formant phase 407, and windowing function 413 according to the pitch pattern 306. All parameters may be changed, and a part of the parameters may be changed.
  • FIG. 10 shows an example of a control function when the parameter transformer 67 controls the formant frequency according to the pitch period. Such control function may be set for every phoneme, every frame or every formant number. The formant frequency can be controlled according to the pitch period, by inputting such control function to the parameter transformer 67. A control function to control the differential value and ratio of the input/output formant frequency may be used instead of the formant frequency itself.
  • FIG. 11 shows the control function to control the power of formant by multiplying the gain corresponding to the pitch period by the windowing functions. It is possible to model the spectrum change of speech according to the change of the pitch period by inputting such a control function to the parameter transformer 67 and changing the parameters according to the pitch period. As a result, it is possible to generate high quality synthesized speech which is not dependent on the pitch of voice.
  • Further, by inputting phoneme symbol string 308 into parameter transformer 67, the formant parameters may be changed according to a kind of preceding or following phoneme. As a result, it is possible to model a variable speech spectrum based on the phoneme environment, and to improve speech quality.
  • Furthermore, the voice variety information 309 inputted to the parameter transformer 67 from an external device (not shown) may be altered to produce different parameters. In this case, it is possible to generate synthesized speech of various voice qualities.
  • FIG. 12 shows an example of changing the voice pitch by changing the formant frequency. If all formant frequencies are converted by the control function (a), since the formant is shifted to a high frequency domain, a thin voice is generated. The control function (b) generates a somewhat thin voice. If the control function (d) is used, since the formant frequency shifts to a low frequency domain, a deep voice is generated. The control function (c) generates a deeper voice.
  • The pitch waveform generator 34 of the fourth embodiment of the present invention will be described referring to FIG. 13. In the fourth embodiment, like reference numerals are used to designate like structural elements corresponding to those in the first embodiment. Only the portions that differ will be described. In the present embodiment, the parameter smoothing device 77 is added to smooth the parameters so that the time based change of each formant parameters is smoothed.
  • The parameter smoothing device 77 outputs formant frequency 820, formant phase 821, windowing function 817, formant frequency 822, formant phase 823, windowing function 818, formant frequency 824, formant phase 825 and windowing function 819 by smoothing the formant frequency 402, formant phase 403, windowing function 411, formant frequency 404, formant phase 405, windowing function 412, formant frequency 406, formant phase 407 and windowing function 413, respectively. All parameters may be smoothed, or merely partly smoothed.
  • FIG. 14 shows an example of smoothing of formant. × represents the formant frequencies 402, 404 and 406 before smoothing. The smoothed formant frequencies 820, 822 and 824 indicated by ○ are generated by performing smoothing so that a change between corresponding formant frequencies of the current frame and the preceding or following frame are smoothed.
  • When the formants between synthesis units do not correspond, the formant corresponding to the formant frequency 404 becomes extinct, as shown by × in FIG. 15A. In this case, since large discontinuity produces to the spectrum and the speech quality deteriorates, the formant frequency 822 is generated by adding formants as shown by ○. At this time, the power of the windowing function 818 corresponding to the formant frequency 822 is attenuated as shown in FIG. 15B, to prevent the formant power from discontinuity.
  • FIGS. 16A and 16B show examples of windowing function position smoothing. Smoothing the windowing function positions so that the peak position of the windowing function 411 varies between frames smoothly generates the windowing function 817. Further, the shape and power of the windowing function may also be smoothed.
  • The above embodiment is explained for 3 formants. The number of formants is not limited to 3, and may be changed every frame.
  • The sine wave generator of the embodiments of the present invention outputs a sine wave. However, a waveform having a near-line power spectrum may be used instead of a complete sine wave. In case that computation precision of the sine wave generator is degraded and the sine wave generator comprises a table in order to reduce computation cost, for example, the complete sine wave is not obtained because of error.
  • Further, the spectrum of formant waveform may not always indicate the peak of the spectrum of speech signal, and the spectrum of the pitch waveform, which is the sum of plural formant waveforms, expresses a spectrum of speech.
  • The above embodiment of the present invention provides a synthesizer for text-to-speech synthesis, but another embodiment of the present invention provides a decoder for speed coding. In other words, the encoder obtains, from the speech signal, formant parameters such as formant frequency, formant phase, windowing function, etc. and pitch period, etc. by analysis, and encodes them and transmits or store codes. The decoder decodes the formant parameters and pitch periods, and reconstructs the speech signal similarly to the above synthesizer.
  • The above speech synthesis can be executed by a program control according to a program stored in a computer readable recording medium. The program control will be described referring to FIG. 17A or more 17C. FIG. 17A show a flowchart of the speech synthesis process, FIG. 17B shows a flowchart of the voiced speech generation process of the speech synthesis process, and FIG. 17C shows a flowchart of the pitch waveform generation process of the voiced speech generation process of FIG. 17B.
  • In the speech synthesis process in FIG. 17A, the pitch pattern 306, phoneme duration 307 and phoneme symbol string 308 are input (S11). The voiced speech signal 303 is generated based on the pitch pattern 306, phoneme duration 307 and phoneme symbol string 308 (S12). The unvoiced speech signal 304 is generated referring to the phoneme duration 307 and phoneme symbol string 308 (S13). The voiced speech signal and unvoiced speech signal are added to generate the synthesized speech signal 305 (S14).
  • In the voiced speech generation process in FIG. 17B, the pitch mark 302 is generated referring to the pitch pattern 306 and phoneme duration 307 (S21). The pitch waveforms 301 are generated corresponding to the pitch marks 302, referring to the pitch pattern 306, phoneme duration 307 and phoneme symbol string 308 (S22). The pitch waveforms 301 are superposed in the positions indicated by the pitch marks 302 to generate a voiced speech (S23).
  • In the pitch waveform generation process in FIG. 17C, the formant parameters 401 for 1 frame corresponding to the pitch mark 302 is selected from the formant parameter storage 41 referring to the pitch pattern 306, phoneme duration 307 and phoneme symbol string 308 (S31). Plural sine waves are generated according to the formant frequencies and formant phases corresponding to the formant numbers of the selected formant parameters 401 (S32). The formant waveforms 414, 415 and 416 are generated by multiplying the plural sine waves by the windowing functions (S33). The formant waveforms are added to generate a pitch waveform (S34).
  • As described above, according to the present invention, since the formant frequency and formant shape are independently controlled for every formant, it is possible to express the spectrum change of speech due to the pitch period variation and voice variety change between the formants, and realize highly flexibility speech synthesis. Because the shape of the windowing functions can express the detailed structure of the formant spectrum, high quality synthesized speech having a natural voice feeling can be generated.

Claims (20)

  1. A speech synthesis method comprising the steps of:
    generating formant parameters;
    selecting predetermined formant parameters from the formant parameters according to a phoneme symbol string;
    generating a plurality of sine waves based on the formant frequency corresponding to the formant parameters selected;
    multiplying the sine waves by the windowing functions corresponding to the selected formant parameters, respectively, to generate a plurality of formant waveforms;
    adding the formant waveforms to generate a plurality of pitch waveforms; and
    superposing the pitch waveforms according to a pitch period to generate a speech signal, characterized by
    generating the windowing functions by adding basis functions weighted by weighting factors.
  2. A speech synthesis method as defined in claim 1, characterized in that the basis functions use DCT basis or KL-expansion basis functions.
  3. A speech synthesis method as claimed in claim 1 or 2, characterized by changing the windowing functions individually.
  4. A speech synthesis method as defined in claim 1, characterized by including changing at least one of power of at least one of the formant waveforms, shape of at least one of the windowing functions, position of at least one of the windowing functions and at least one of the format frequencies according to the pitch period.
  5. A speech synthesis method as defined in claims 1, 2 or 3, characterized in that at least one of power of at least one of the formant waveforms, shape of at least one of the windowing functions, position of at least one of the windowing functions and at least one of the formant frequencies is changed every phoneme, every frame or every formant number.
  6. A speech synthesis method as defined in claims 1, 2 or 3, characterized by including changing at least one of power of at least one of the formant waveforms, shape of at least one of the windowing functions, position of at least one of the windowing functions and at least one of the formant frequencies according to a kind of at least preceding phoneme or following phoneme.
  7. A speech synthesis method as defined in claims 1, 2 or 3, characterized by including changing at least one of power of at least one of the formant waveforms, shape of at least one of the windowing functions, position of at least one of the windowing functions and at least one of the formant frequencies according to information of given voice variety.
  8. A speech synthesis method as defined in claims 1, 2 or 3, characterized by including changing at least one of power of at least one of the formant waveforms, at least one of the formant frequencies, shape of at least one of the windowing functions, phase of at least one of the sine waves and position of at least one of the windowing functions according to at least one of power of at least one of the formant waveforms, at least one of the formant frequencies, shape of at least one of the windowing functions, phase of at least one of the sine waves and position of at least one of the windowing functions of a corresponding formant of at least a preceding pitch waveform or a following pitch waveform.
  9. A speech synthesis method as defined in claims 1, 2 or 3, characterized by including changing at least one of power of at least one of the formant waveforms, at least one of the formant frequencies, shape of at least one of the windowing functions, phase of at least one of the sine waves and position of at least one of the windowing functions according to presence of a corresponding formant of at least a preceding pitch waveform or a following pitch waveform.
  10. A speech synthesis method as defined in any one of the preceding claims, characterized by including smoothing selectively the formant frequencies, formant phases, and windowing functions.
  11. A speech synthesizer supplied with a pitch pattern, phoneme duration and phoneme symbol string, comprising:
    means (33) for generating pitch marks referring to the pitch pattern and phoneme duration;
    means (51) for generating formant parameters,
    means (52) for selecting the formant parameters for one frame corresponding to the phoneme symbol string,
    means (43-45) for generating sine waves according to formant frequencies of the read formant parameters,
    means for multiplying the sine waves by windowing functions of the selected formant parameters to generate formant waveforms,
    means for adding the formant waveforms to generate pitch waveforms,
    means (35) for superposing the pitch waveforms on the pitch marks to generate a voiced speech signal;
    means (32) for generating an unvoiced speech; and
    means for adding the voiced speech and the unvoiced speech to generate synthesized speech, characterized by
    means (56) for generating the windowing functions by adding basis functions weighted by the weighting factors.
  12. A speech synthesizer as defined in claim 11, characterized by means (51) for storing a plurality of weighting factors.
  13. A speech synthesizer as defined in claim 11 or 12 characterized by changing the windowing functions individually.
  14. A speech synthesizer as defined in claim 11, 12 or 13, characterized in that the means (56) for generating the windowing functions uses as the basis functions DCT basis or KL-expansion basis functions.
  15. A speech synthesizer as defined in any one of claims 11 to 14 characterized by including means (67) for transforming the selected formant parameters according to the pitch period.
  16. A speech synthesizer as defined in claim 15, characterized in that the transforming means (67) transforms the selected formant parameters every phoneme, every frame or every formant number.
  17. A speech synthesizer as defined in any one of claims 11 to 14 characterized by including means (67) for transforming the selected formant parameters according to information of a preceding phoneme or a following phoneme.
  18. A speech synthesizer as defined in any one of claims 11 to 14, characterized by including means (67) for transforming the selected formant parameters according to given voice variety.
  19. A speech synthesizer as defined in any one of claims 11 to 18, characterized by including means (77) for smoothing the selected formant parameters that vary in time.
  20. A carrier medium carrying computer readable instructions for controlling a computer to carry out the method of any one claims 1 to 10.
EP02252159A 2001-03-26 2002-03-26 Speech synthesis method and speech synthesizer Expired - Lifetime EP1246163B1 (en)

Applications Claiming Priority (4)

Application Number Priority Date Filing Date Title
JP2001087041 2001-03-26
JP2001087041 2001-03-26
JP2002077096 2002-03-19
JP2002077096A JP3732793B2 (en) 2001-03-26 2002-03-19 Speech synthesis method, speech synthesis apparatus, and recording medium

Publications (3)

Publication Number Publication Date
EP1246163A2 EP1246163A2 (en) 2002-10-02
EP1246163A3 EP1246163A3 (en) 2003-08-13
EP1246163B1 true EP1246163B1 (en) 2005-08-10

Family

ID=26612017

Family Applications (1)

Application Number Title Priority Date Filing Date
EP02252159A Expired - Lifetime EP1246163B1 (en) 2001-03-26 2002-03-26 Speech synthesis method and speech synthesizer

Country Status (5)

Country Link
EP (1) EP1246163B1 (en)
JP (1) JP3732793B2 (en)
KR (1) KR100457414B1 (en)
CN (1) CN1185619C (en)
DE (1) DE60205421T2 (en)

Families Citing this family (13)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
AU2003254398A1 (en) * 2002-09-10 2004-04-30 Leslie Doherty Phoneme to speech converter
JP2004294816A (en) * 2003-03-27 2004-10-21 Yamaha Corp Mobile terminal device
JP4214842B2 (en) 2003-06-13 2009-01-28 ソニー株式会社 Speech synthesis apparatus and speech synthesis method
JP2005004105A (en) * 2003-06-13 2005-01-06 Sony Corp Signal generator and signal generating method
JP2005234337A (en) * 2004-02-20 2005-09-02 Yamaha Corp Device, method, and program for speech synthesis
JP4469883B2 (en) 2007-08-17 2010-06-02 株式会社東芝 Speech synthesis method and apparatus
JP5275102B2 (en) * 2009-03-25 2013-08-28 株式会社東芝 Speech synthesis apparatus and speech synthesis method
JP5631915B2 (en) * 2012-03-29 2014-11-26 株式会社東芝 Speech synthesis apparatus, speech synthesis method, speech synthesis program, and learning apparatus
WO2017046887A1 (en) * 2015-09-16 2017-03-23 株式会社東芝 Speech synthesis device, speech synthesis method, speech synthesis program, speech synthesis model learning device, speech synthesis model learning method, and speech synthesis model learning program
JP6728843B2 (en) * 2016-03-24 2020-07-22 カシオ計算機株式会社 Electronic musical instrument, musical tone generating device, musical tone generating method and program
CN108257613B (en) * 2017-12-05 2021-12-10 北京小唱科技有限公司 Method and device for correcting pitch deviation of audio content
CN108597527B (en) * 2018-04-19 2020-01-24 北京微播视界科技有限公司 Multi-channel audio processing method, device, computer-readable storage medium and terminal
CN110189743B (en) * 2019-05-06 2024-03-08 平安科技(深圳)有限公司 Splicing point smoothing method and device in waveform splicing and storage medium

Also Published As

Publication number Publication date
DE60205421D1 (en) 2005-09-15
JP3732793B2 (en) 2006-01-11
JP2002358090A (en) 2002-12-13
EP1246163A2 (en) 2002-10-02
CN1185619C (en) 2005-01-19
CN1378199A (en) 2002-11-06
KR100457414B1 (en) 2004-11-16
EP1246163A3 (en) 2003-08-13
KR20020076144A (en) 2002-10-09
DE60205421T2 (en) 2006-04-20

Similar Documents

Publication Publication Date Title
US6304846B1 (en) Singing voice synthesis
KR940002854B1 (en) Sound synthesizing system
Tabet et al. Speech synthesis techniques. A survey
JP4705203B2 (en) Voice quality conversion device, pitch conversion device, and voice quality conversion method
US8175881B2 (en) Method and apparatus using fused formant parameters to generate synthesized speech
EP1701336B1 (en) Sound processing apparatus and method, and program therefor
EP1246163B1 (en) Speech synthesis method and speech synthesizer
JPH031200A (en) Regulation type voice synthesizing device
EP3739571A1 (en) Speech synthesis method, speech synthesis device, and program
US7251601B2 (en) Speech synthesis method and speech synthesizer
Macon et al. Speech concatenation and synthesis using an overlap-add sinusoidal model
Degottex et al. Pitch transposition and breathiness modification using a glottal source model and its adapted vocal-tract filter
Macon et al. Concatenation-based midi-to-singing voice synthesis
JP2018077283A (en) Speech synthesis method
US7596497B2 (en) Speech synthesis apparatus and speech synthesis method
EP1543497B1 (en) Method of synthesis for a steady sound signal
Mandal et al. Epoch synchronous non-overlap-add (ESNOLA) method-based concatenative speech synthesis system for Bangla.
van Santen et al. Prediction and synthesis of prosodic effects on spectral balance of vowels
JPH09179576A (en) Voice synthesizing method
JP2018077280A (en) Speech synthesis method
JP2018077281A (en) Speech synthesis method
JP2001312300A (en) Voice synthesizing device
Rodet Sound analysis, processing and synthesis tools for music research and production
JP2765192B2 (en) Electronic musical instrument
JPH0836397A (en) Voice synthesizer

Legal Events

Date Code Title Description
PUAI Public reference made under article 153(3) epc to a published international application that has entered the european phase

Free format text: ORIGINAL CODE: 0009012

17P Request for examination filed

Effective date: 20020417

AK Designated contracting states

Kind code of ref document: A2

Designated state(s): AT BE CH CY DE DK ES FI FR GB GR IE IT LI LU MC NL PT SE TR

AX Request for extension of the european patent

Free format text: AL;LT;LV;MK;RO;SI

PUAL Search report despatched

Free format text: ORIGINAL CODE: 0009013

AK Designated contracting states

Designated state(s): AT BE CH CY DE DK ES FI FR GB GR IE IT LI LU MC NL PT SE TR

AX Request for extension of the european patent

Extension state: AL LT LV MK RO SI

AKX Designation fees paid

Designated state(s): DE FR GB

17Q First examination report despatched

Effective date: 20040503

GRAP Despatch of communication of intention to grant a patent

Free format text: ORIGINAL CODE: EPIDOSNIGR1

GRAS Grant fee paid

Free format text: ORIGINAL CODE: EPIDOSNIGR3

GRAA (expected) grant

Free format text: ORIGINAL CODE: 0009210

AK Designated contracting states

Kind code of ref document: B1

Designated state(s): DE FR GB

REG Reference to a national code

Ref country code: GB

Ref legal event code: FG4D

REF Corresponds to:

Ref document number: 60205421

Country of ref document: DE

Date of ref document: 20050915

Kind code of ref document: P

ET Fr: translation filed
PLBE No opposition filed within time limit

Free format text: ORIGINAL CODE: 0009261

STAA Information on the status of an ep patent application or granted ep patent

Free format text: STATUS: NO OPPOSITION FILED WITHIN TIME LIMIT

26N No opposition filed

Effective date: 20060511

PGFP Annual fee paid to national office [announced via postgrant information from national office to epo]

Ref country code: FR

Payment date: 20120319

Year of fee payment: 11

PGFP Annual fee paid to national office [announced via postgrant information from national office to epo]

Ref country code: GB

Payment date: 20120321

Year of fee payment: 11

PGFP Annual fee paid to national office [announced via postgrant information from national office to epo]

Ref country code: DE

Payment date: 20120411

Year of fee payment: 11

GBPC Gb: european patent ceased through non-payment of renewal fee

Effective date: 20130326

REG Reference to a national code

Ref country code: FR

Ref legal event code: ST

Effective date: 20131129

REG Reference to a national code

Ref country code: DE

Ref legal event code: R119

Ref document number: 60205421

Country of ref document: DE

Effective date: 20131001

PG25 Lapsed in a contracting state [announced via postgrant information from national office to epo]

Ref country code: GB

Free format text: LAPSE BECAUSE OF NON-PAYMENT OF DUE FEES

Effective date: 20130326

Ref country code: DE

Free format text: LAPSE BECAUSE OF NON-PAYMENT OF DUE FEES

Effective date: 20131001

Ref country code: FR

Free format text: LAPSE BECAUSE OF NON-PAYMENT OF DUE FEES

Effective date: 20130402