Classifications

• • G—PHYSICS
  • G10—MUSICAL INSTRUMENTS; ACOUSTICS
  • G10L—SPEECH ANALYSIS OR SYNTHESIS; SPEECH RECOGNITION; SPEECH OR VOICE PROCESSING; SPEECH OR AUDIO CODING OR DECODING
  • G10L13/00—Speech synthesis; Text to speech systems
  • G10L13/08—Text analysis or generation of parameters for speech synthesis out of text, e.g. grapheme to phoneme translation, prosody generation or stress or intonation determination
  • G10L13/10—Prosody rules derived from text; Stress or intonation
• • G—PHYSICS
  • G10—MUSICAL INSTRUMENTS; ACOUSTICS
  • G10L—SPEECH ANALYSIS OR SYNTHESIS; SPEECH RECOGNITION; SPEECH OR VOICE PROCESSING; SPEECH OR AUDIO CODING OR DECODING
  • G10L13/00—Speech synthesis; Text to speech systems
  • G10L13/06—Elementary speech units used in speech synthesisers; Concatenation rules
  • G10L13/07—Concatenation rules
• • G—PHYSICS
  • G10—MUSICAL INSTRUMENTS; ACOUSTICS
  • G10L—SPEECH ANALYSIS OR SYNTHESIS; SPEECH RECOGNITION; SPEECH OR VOICE PROCESSING; SPEECH OR AUDIO CODING OR DECODING
  • G10L13/00—Speech synthesis; Text to speech systems
  • G10L13/06—Elementary speech units used in speech synthesisers; Concatenation rules

Definitions

• the present invention relates generally to Text-To-Speech (TTS) synthesis.
• TTS Text-To-Speech
• the invention is particularly useful for, but not necessarily limited to, determining an appropriate synthesized pronunciation of a text segment using a non-exhaustive utterance corpus.
• TTS Text to Speech
• concatenated text to speech synthesis allows electronic devices to receive an input text string and provide a converted representation of the string in the form of synthesized speech.
• a device that may be required to synthesize speech originating from a non-deterministic number of received text strings will have difficulty in providing high quality realistic synthesized speech. That is because the pronunciation of each word or syllable (for Chinese characters and the like) to be synthesized is context and location dependent. For example, a pronunciation of a word at the beginning of a sentence
• input text string may be drawn out or lengthened.
• the pronunciation of the same word may be lengthened even more if it occurs in the middle of a sentence where emphasis is required.
• the pronunciation of a word depends on at least tone (pitch), volume and duration.
• many languages include numerous possible pronunciations of individual syllables.
• a single syllable represented by a Chinese character (or other similar character based script) may have up to 6 different pronunciations.
• a large pre-recorded utterance waveform corpus of sentences is required. This corpus typically requires on average about 500 variations of each pronunciation if realistic speech synthesis is to be achieved.
• an utterance waveform corpus of all pronunciations for every character would be prohibitively large.
• the size of the utterance waveform corpus may be particularly limited when it is embedded in a small electronic device having a low memory capacity such as a radio telephone or a personal digital assistant.
• the algorithms used to compare the input text strings with the audio database also need to be efficient and fast so that the resulting synthesized and concatenated speech flows naturally and smoothly. Due to memory and processing speed limitations, existing TTS methods for embedded applications often result in speech that is unnatural or robotic sounding. There is therefore a need for an improved method for performing TTS to provide a natural sounding synthesized speech whilst using a non-exhaustive utterance corpus.
• the present invention is a method of performing speech synthesis that includes comparing an input text segment with an utterance waveform corpus that contains numerous speech samples. The method determines whether there is a contextual best match between the text segment and one speech sample included in the utterance waveform corpus. If there is not a contextual best match, the method determines whether there is a contextual phonetic hybrid match between the text segment and a speech sample included in the utterance waveform corpus.
• a contextual phonetic hybrid match requires a match of all implicit prosodic features in a defined prosodic feature group.
• the prosodic feature group is redefined by deleting one of the implicit prosodic features from the prosodic feature group so as to redefine the prosodic feature group.
• the prosodic feature group is successively redefined by deleting one implicit prosodic feature from the group until a match is found between the input text segment and a speech sample. When a match is found, the matched speech sample is used to generate concatenative speech.
• Fig. 1 is a block diagram of an electronic device upon which the invention may be implemented
• Fig. 2 is a flow chart illustrating a specific embodiment of the present invention used to generate concatenative speech in the Chinese language
• Fig. 3 is a flowchart illustrating the process of determining whether a contextual phonetic hybrid match exists by successively relaxing the constraints used to define a match.
• Fig. 1 there is illustrated a block diagram of an electronic device 10 upon which the invention may be implemented.
• the device 10 includes a processor 30 operatively coupled, by a common bus 15, to a text memory module 20, a Read only Memory (ROM) 40, a Random Access Memory (RAM) 50 and a waveform corpus 60.
• the processor 30 is also operatively coupled to a touch screen display 90 and an input of a speech synthesizer 70.
• An output of the speech synthesizer 70 is operatively coupled to a speaker 80.
• the text memory module is a store for storing text obtained by any receiving means possible such as by radio reception, internet, or plug in portable memory cards etc.
• the ROM stores operating code for performing the invention as described in Figures 2 and 3.
• the Corpus 60 is a essentially a conventional corpus as is the speech synthesizer 70 and speaker 80 and the touch screen display 90 is a user interface and allows for display of text stored in the a text memory module 20.
• Fig. 2 is a flow chart illustrating a specific embodiment of the present invention used to generate concatenative speech 110 from an input text segment 120 in the Chinese language.
• the text segment 120 is compared with an utterance waveform corpus 60, which includes a plurality of speech samples 140, to determine whether there is a contextual best match (step S110). If a contextual best match is found between a text segment 120 and a specific speech sample 140, that specific speech sample 140 is sent to a concatenating algorithm 150 for generating the concatenative speech 110. If no contextual best match is found between the text segment 120 and a specific speech sample 140, then the text segment 120 is compared again with the utterance waveform corpus 130 to determine whether there is a contextual phonetic hybrid match (step S120).
• Fig. 3 is a flowchart illustrating the process of determining whether a contextual phonetic hybrid match exists by successively relaxing the constraints used to define a match.
• a contextual phonetic hybrid match requires a match between a text segment 120 and all of the implicit prosodic features 210 included in a defined prosodic feature group 220. If no match is found, one of the implicit prosodic features 210 is deleted from the defined prosodic feature group 220 and the group 220 is redefined as including all of the previously included features 210 less the deleted feature 210 (e.g., Step 130). The redefined prosodic feature group 220 is then compared with the text segment 120 to determine whether there is a match. The process of deleting an implicit prosodic feature 210, redefining the prosodic feature group 220, and then redetermining whether there is a contextual phonetic hybrid match, continues until a match is found (Steps S130, S140, etc.
• the matched speech sample 140 which matches the text segment 120, is sent to the concatenating algorithm 150 for generating concatenative speech 110.
• the concatenating algorithm 150 for generating concatenative speech 110.
• a basic phonetic match is performed matching only pinyin (Step S180).
• the utterance waveform corpus 60 is designed so that there is always at least one syllable included with the correct pinyin to match all possible input text segments 120. That basic phonetic match is then input into the concatenating algorithm 150.
• the invention is thus a multi-layer, data-driven method for controlling the prosody (rhythm and intonation) of the resulting synthesized, concatenative speech 110.
• each layer of the method includes a redefined prosodic feature group 220.
• a text segment 120 means any type of input text string or segment of coded language. It should not be limited to only visible text that is scaned or otherwise entered into a TTS system.
• the utterance waveform corpus 130 of the present invention is annotated with information concerning each speech sample 140 (usually a word) that is included in the corpus 130.
• the speech samples 140 themselves are generally recordings of actual human speech, usually digitized or analog waveforms. Annotations are thus required to identify the samples 140.
• Such annotations may include the specific letters or characters (depending on the language) that define the sample 140 as well as the implicit prosodic features 210 of the speech sample 140.
• the implicit prosodic features 210 include context information concerning how the speech sample 140 is used in a sentence.
• a speech sample 140 in the Chinese language may include the following implicit prosodic features 210: Text context: the Chinese characters immediately preceding and immediately following the annotated text of a speech sample 140.
• Pinyin the phonetic representation of a speech sample 140. Pinyin is a standard romanization of the Chinese language using the Western alphabet.
• Tone context the tone context of the Chinese characters immediately preceding and immediately following the annotated text of a speech sample 140.
• Co-articulation the phonetic level representatives that immediately precede and immediately follow the annotated text of a speech sample 140, such as phonemes or sub-syllables.
• Syllable position the position of a syllable in a prosodic phrase.
• Phrase position the position of a prosodic phrase in a sentence. Usually the phrase position is identified as one of the three positions of sentence initial, sentence medial and sentence final.
• Character symbol the code (e.g., ASCII code) representing the Chinese character that defines a speech sample 140.
• Length of phrase the number of Chinese characters included in a prosodic phrase.
• each character's sound could represent a speech sample 140 and could be annotated with the above implicit prosodic features 210.
• the character "H” as found in the above sentence could be annotated as follows: Text context: ⁇ , nowadays ; Pinyin: guo2; Tone context: 1 , 3; Co-articulation: ong, h ; Syllable position: 2 ; Phrase position: 1 ; Character symbol: ASCII code for H ; and Length of phrase: 2.
• step S110 determines whether there is a contextual best match between a text segment 120 and a speech sample 140.
• a contextual best match is generally defined as the closest, or an exact, match of both 1 ) the letters or characters (depending on the language) of an input text segment 120 with the corresponding letters or characters of an annotated speech sample 140, and 2) the implicit prosodic features 210 of the input text segment 120 with the implicit prosodic features 210 of the annotated speech sample 140.
• a best match is being determined by identifying the greatest number of consecutive syllables in the input text segment that are identical to attributes and attribute positions in each of the waveform utterances (speech sample) in the waveform corpus 60.
• a speech sample 140 selected immediately as an element for use in the concatenating algorithm 150.
• the method of the present invention determines whether there is a contextual phonetic hybrid match between an input text segment 120 and a speech sample 140.
• a contextual phonetic hybrid match requires a match between a text segment 120 and all of the implicit prosodic features 210 included in a defined prosodic feature group 220. As shown in Fig.
• one embodiment of the present invention used to synthesize speech in the Chinese language uses a first defined prosodic feature group 220 that includes the implicit prosodic features 210 of pinyin, tone context, co- articulation, syllable position, phrase position, character symbol, and length of phrase (Step S120). If none of the annotated speech samples 140 found in the utterance waveform corpus 130 have identical values for each of the above features 210 as found in the input text segment 120, then the corpus 130 does not contain a speech sample 140 that is close enough to the input text segment 120 based on the matching rules as applied in Step S120. Therefore the constraints of the matching rules must be relaxed and thus broadened to include other speech samples 140 that possess the next most preferable features 210 found in the input text segment 120.
• the matching rules are broadened by deleting the one feature 210 found in the defined prosodic feature group 220 that is least likely to affect the natural prosody of the input text segment 120.
• the next most preferable features 210 found in the illustrated embodiment of the present invention include all of the features 210 defined above less the length of phrase feature 210.
• the order in which the implicit prosodic features 210 are deleted from the defined prosodic feature group 220 is determined empirically. When the features 210 are deleted in a proper order, the method of the present invention results in efficient and fast speech synthesis. The output speech therefore sounds more natural even though the utterance waveform corpus 130 may be relatively limited in size.
• the variable BestPitch may be determined based on a statistical analysis of the utterance waveform corpus 130.
• a corpus 130 may include five tones, each having an average pitch.
• Each annotated speech sample 140 in the corpus 130 may also include individual prosody information represented by the values of pitch, duration and energy. So the average values of pitch, duration and energy of the entire corpus 130 are available.
• the best pitch for a particular context may then be determined using the following formula: BestPitch - pitch tone — nlndexx empiricalvalue (Eq.
• pitch t one the average pitch including tone of the utterance waveform corpus
• nlndex the index of the text segment 120 in a prosody phrase
• empircalvalue an empirical value based on the utterance waveform corpus.
• the empirical value of 4 is used in one particular embodiment of the present invention that synthesizes the Chinese language; however this number could vary depending on the content of a particular utterance waveform corpus 130.
• the empirical value of 4 is used in one particular embodiment of the present invention that synthesizes the Chinese language, however this number could vary depending on the content of a particular utterance waveform corpus 130.
• the invention is suitable for many languages.
• the implicit prosodic features 210 would need to be deleted or redefined from the examples given hereinabove.
• the feature 210 identified above as tone context would be deleted in an application of the present invention for the English language because English is not a tonal language.
• the feature 210 identified above as pinyin would likely be redefined as simply a phonetic symbol when the present invention is applied to English.
• the present invention is therefore a multi-layer, data-driven prosodic control scheme that utilizes the implicit prosodic information in an utterance waveform corpus 130.
• the method of the present invention When searching for an appropriate speech sample 140 to match with a given input text segment 120, the method of the present invention employs a strategy based on multi-layer matching, where each layer is tried in turn until a sufficiently good match is found. By successively relaxing the constraints of each layer, the method efficiently determines whether the utterance waveform corpus 130 contains a match.
• the method is therefore particularly appropriate for embedded TTS systems where the size of the utterance waveform corpus 130 and the processing power of the system may be limited.

Applications Claiming Priority (2)

Publications (2)

Family

ID=34398359

Family Applications (1)

Country Status (5)

Cited By (1)

Families Citing this family (59)

Citations (1)

Family Cites Families (10)

• 2003
  • 2003-09-29 CN CNB031326986A patent/CN1260704C/zh not_active Expired - Lifetime
• 2004
  • 2004-09-17 WO PCT/US2004/030467 patent/WO2005034082A1/en active Application Filing
  • 2004-09-17 EP EP04784355A patent/EP1668628A4/de not_active Withdrawn
  • 2004-09-17 MX MXPA06003431A patent/MXPA06003431A/es not_active Application Discontinuation
  • 2004-09-17 KR KR1020067006170A patent/KR100769033B1/ko active IP Right Grant

Patent Citations (1)

Non-Patent Citations (6)

Also Published As

Similar Documents

Legal Events