EP1960996B1 - Voice synthesis by concatenation of acoustic units - Google Patents

Description

The present invention relates to a system and method for voice synthesis by concatenation of acoustic units and a computer program for implementing the method.

A speech synthesis system based on a text conventionally comprises input means of the text to be synthesized and linguistic processing means of this text to transform it into a series of phonemes accompanied by prosodic indications. These linguistic treatments include syntactic treatments, grapheme-phoneme translations as well as prosodic treatments. They rely on dictionaries as well as rulesets.

It also includes concatenation synthesis means of prerecorded elements for generating an acoustic signal according to the sequence of phonemes provided by the linguistic processing.

Such a system is explained in more detail in Gaël Richard, Olivier Cappé "Synthesis of speech from the text", Techniques of the engineer H 7 288.

Such systems seek to achieve a quality comparable to that of natural speech.

Currently, a significant limitation in the quality of these speech synthesis systems lies in linguistic processing. This limitation is related to the loss of information induced by transcription and the ambiguous nature of certain textual forms. As a result, the systematic use of synthetic speech for static recordings can only be done under the control of an operator who overcomes the inevitable defects of this linguistic treatment.

• une méthode d'enrichissement du texte par la présence de balises. Cet enrichissement du texte permet de contrôler l'analyse linguistique (phonétisation d'un mot ou son étiquette grammaticale) ou le synthétiseur (volume, hauteur de la voix, vitesse d'élocution)..L'utilisation de balises est actuellement, en cours de normalisation par l'organisme W3C. Une première version du langage de balisage SSML (Speech Synthesis Markup Language - langage de balisage pour la synthèse vocale) a été publiée en septembre 2004, via l'url http:/www.w3.org/TR/speech-synthesis/. L'enrichissement du texte d'entrée se fait grâce à un éditeur spécialisé. L'outil « TTS Director » de la société Loquendo est un exemple d'éditeur dédié à la synthèse vocale (http://www.loquendo.com/en/technology/tts director.htm).
• le paramétrage du système. Par exemple, l'outil Lexitool qui fait partie du catalogue de la société Elan Speech, permet de gérer un lexique d'exception. L'opérateur enrichit les données du système en ajoutant dans le lexique les mots que le système ne prononce pas correctement et en leur associant la prononciation attendue.
• la synthèse interactive. Celle-ci est décrite dans l'article de Peter Rutten, Justin Fackrell « The application of interactive speech unit selection in TTS systems ». Eurospeech 2003. L'intervention de l'opérateur se fait dans le processus de synthèse après l'exécution d'une étape importante du traitement et conduit à modifier le comportement global du système par la suite en modifiant les paramètres de cette étape de traitement. Par exemple, dans cet article, un opérateur peut modifier localement le paramétrage du synthétiseur, après exécution du traitement de sélection, pour produire une variante de production de synthèse plus proche de ce qui est attendu.

In the state of the art, three methods are known to allow an operator to control the result of a speech synthesis system:

• a method of enriching the text by the presence of tags. This enrichment of the text makes it possible to control the linguistic analysis (Phonetization of a word or its grammatical label) or the synthesizer (volume, pitch of voice, speed of speech) .. The use of tags is currently being standardized by W3C. A first version of the Speech Synthesis Markup Language (SSML) markup language was published in September 2004, via the url http: /www.w3.org/TR/speech-synthesis/. The enrichment of the input text is done through a specialized editor. The "TTS Director" tool from Loquendo is an example of a publisher dedicated to speech synthesis (http://www.loquendo.com/en/technology/tts director.htm).
• the setting of the system. For example, the Lexitool tool that is part of the catalog of the company Elan Speech, allows to manage an exceptional lexicon. The operator enriches the data of the system by adding in the lexicon the words that the system does not pronounce correctly and associating with them the expected pronunciation.
• interactive synthesis. This is described in the article by Peter Rutten, Justin Fackrell "The application of interactive speech unit selection in TTS systems". Eurospeech 2003. The intervention of the operator is done in the process of synthesis after the execution of an important stage of the treatment and leads to modify the global behavior of the system later by modifying the parameters of this stage of treatment. For example, in this article, an operator can locally modify the parameterization of the synthesizer, after execution of the selection process, to produce a synthetic production variant that is closer to what is expected.

These methods have the major disadvantage of the low correlation between the modification performed by the operator and the final result obtained. By the term "low correlation" is meant here that the operator does not have an intuitive manipulation of the system. This manipulation requires significant learning before the operator is able to determine the parameter or parameters to be modified to obtain a better result.

The object of the invention is therefore to overcome this drawback by proposing an interactive speech synthesis system and method that is easy to use for an operator.

The object of the invention is a voice synthesis system by concatenation of acoustic units according to claim 1.

• le paramétrage du texte comporte des balises insérées dans le texte à synthétiser ;
• la ou chaque unité de traitement élémentaire est adaptée pour effectuer l'un des traitements élémentaires de l'ensemble des traitements élémentaires de :
  1. a) - validation du texte à synthétiser,
  2. b) - découpage du texte en phrases,
  3. c) - découpage du texte en groupes de souffle,
  4. d) - découpage du texte en mots,
  5. e) - modification d'un lexique d'exceptions,
  6. f) - phonétisation des mots,
  7. g) - analyse grammaticale,
  8. h) - prosodie.
• les moyens de traitement linguistique comportent des moyens de traitement élémentaires pour effectuer la totalité des traitements élémentaires dudit ensemble de traitements élémentaires.

Other features are:

• the setting of the text includes tags inserted into the text to be synthesized;
• the or each elementary treatment unit is adapted to perform one of the elementary treatments of all the elementary treatments of:
  1. a) - validation of the text to be synthesized,
  2. b) - division of the text into sentences,
  3. c) - cutting the text into groups of breath,
  4. d) - cutting the text into words,
  5. e) - modification of a lexicon of exceptions,
  6. f) - phonetics of words,
  7. g) - grammatical analysis,
  8. h) - prosody.
• the linguistic processing means comprise elementary processing means for performing all of the elementary treatments of said set of elementary processes.

Another object is a concatenated voice synthesis method of acoustic units according to claim 5.

• la modification des paramètres consiste à créer/modifier des balises dans le texte à synthétiser ;
• l'étape de la génération de résultats intermédiaires comporte l'une des sous étapes de traitement élémentaire :
• validation du texte à synthétiser,
• découpage du texte en phrases,
• découpage du texte en groupes de souffle,
• découpage du texte en mots,
• modification d'un lexique d'exceptions,
• phonétisation des mots,
• analyse grammaticale,
• prosodie.
• ledit procédé comporte en outre une étape de sélection de la sous étape de traitement élémentaire à exécuter parmi l'ensemble des sous étapes de traitement élémentaire ;
• il est exécuté successivement 8 fois et à chaque fois, une sous étape de traitement élémentaire différente est sélectionnée dans l'ordre suivant :
• validation du texte à synthétiser,
• découpage du texte en phrases,
• découpage du texte en groupes de souffle,
• découpage du texte en mots,
• modification d'un lexique d'exceptions,
• phonétisation des mots,
• analyse grammaticale,
• prosodie.

Other features of this object are

• modifying the parameters consists of creating / modifying tags in the text to be synthesized;
• the step of generating intermediate results comprises one of the elementary treatment sub-stages:
• validation of the text to be synthesized,
• cutting the text into sentences,
• cutting the text into groups of breath,
• cutting the text into words,
• modify a lexicon of exceptions,
• phonation of words,
• grammatical analysis,
• prosody.
• said method further comprises a step of selecting the elementary treatment substep to be performed among all of the elementary treatment substeps;
• it is executed successively 8 times and each time, a different elementary treatment sub-step is selected in the following order:
• validation of the text to be synthesized,
• cutting the text into sentences,
• cutting the text into groups of breath,
• cutting the text into words,
• modify a lexicon of exceptions,
• phonation of words,
• grammatical analysis,
• prosody.

Another object is a computer program comprising program code instructions for performing the steps of the method when said program is executed on a computer.

Advantageously, the linguistic processing is decomposed for the operator into a series of elementary processes allowing him to control all the parameters having an impact on the quality of the sound flow produced.

Being able to select the elementary step on which he wishes to intervene, the operator advantageously controls the speech synthesis tool in what appears to him to be the detail of its operation.

In addition, the sequence of elementary treatments proposes a logic order of treatment well adapted to the mode of operation of the operator while it does not correspond to the internal operation of the synthesis system.

• la figure 1 est un schéma synoptique d'un système de synthèse vocale selon un mode de réalisation de l'invention ;
• la figure 2 est un ordinogramme d'un procédé de synthèse vocale selon un mode de réalisation de l'invention ;
• la figure 3 est une variante du procédé selon la figure 2 ; et
• la figure 4 est un ordinogramme d'un procédé de synthèse vocale utilisant le procédé de la figure 3 selon un ordre de présentation de traitements élémentaires.

The invention will be better understood on reading the description which follows, made solely by way of example, and in relation to the appended drawings in which:

• the figure 1 is a block diagram of a speech synthesis system according to an embodiment of the invention;
• the figure 2 is a flow chart of a speech synthesis method according to an embodiment of the invention;
• the figure 3 is a variant of the process according to the figure 2 ; and
• the figure 4 is a flow chart of a speech synthesis method using the method of the figure 3 according to an order of presentation of elementary treatments.

With reference to the figure 1 , a voice synthesis system 1 comprises means 2 for inputting a text to be synthesized. This text is stored in a buffer memory 3 in the form of a record comprising the actual coded text, for example, according to the ISO / IEC 10646 standard, as well as linguistic processing aid parameters, for example in the form of tags. SSML.

The buffer memory 3 is connected to linguistic processing means 4 of this text. These linguistic processing means 4 are connected to a second buffer 5 in which they store the result of the linguistic processing in the form of a series of phonemes accompanied by prosodic indications.

This second memory 5 is connected to synthesis means 6 by concatenation of prerecorded elements to restore an acoustic signal as a function of the sequence of phonemes.

The acoustic signal is transformed into sounds by speakers 7.

A detailed description of these various elements is contained in the document by G. Richard and O. Cappé cited above.

The voice synthesis system 1 comprises means 8 for inputting and editing. These input and edit means 8 comprise keyboard-type input means 9 and a pointing tool 10 such as a mouse. They also comprise a display screen 11 and means 12 for controlling these devices 9, 10, 11.

Advantageously, these input and edit means 8 present to an operator of the voice synthesis system 1 a user-friendly graphical interface.

The linguistic processing means 4 comprise a chain of units 4A, 4B, 4C of elementary processing, each of which draws a particular element of the linguistic processing chain such as the division of the text into sentences, the division of the sentences into words, the phonetization of words, grammatical analysis, prosody ...

Each unit 4A, 4B, 4C of elementary treatment is connected to a specialized editor 8A, 8B, 8C 8 means of input and editing allowing the operator to intervene on the elementary results of the corresponding unit 4A, 4B, 4C to modify them.

Each pair consisting of a unit 4A, 4B, 4C of elementary processing and its editor 8A, 8B, 8C, constitutes a module 13A, 13B, 13C of processing and editing for a determined stage of linguistic processing.

The voice synthesis system 1 comprises parameterization means 14 connected to the first buffer memory 3 and to the elementary processing modules 13A, 13B, 13C.

These setting means 14 add, modify or delete the linguistic processing aid parameters contained in the record stored in the buffer memory according to the modifications made by the operator on the elementary results of the unit 4A, 4B, 4C. of elementary processing so that during a subsequent processing of the recording by the same elementary processing units, the elementary result obtained at the output of each unit is the result modified by the operator. The means 14 are not suitable for acting on the actual parameter setting of the elementary processing units, nor on the synthesis means 6.

In a preferred embodiment, the speech synthesis system 1 comprises 8 modules corresponding to 8 stages of the linguistic processing of the text.

The first module deals with the text itself. It allows the operator to validate that the text to be synthesized suits him. Optionally, this module enriches the text with change of voice tags.

The technique used by this first module is described in the state of the art, for example in the standardization of the W3C SSML language.

The second module deals with the division of text into phases. The editor shows the operator which phase boundaries can be deleted, moved or inserted.

The third module deals with splitting into breath groups. The publisher highlights breath groups and break times between groups. The operator can change the placement of breaks and their durations.

The fourth module deals with the division into words. The publisher highlights the groupings of words that have a link. The operator can separate words or group others to form phrases.

The fifth module deals with the lexicon. The operator intervenes on the data by adding, modifying or deleting entries of the exception lexicon.

The sixth module deals with the phonetization of words. The editor presents to the operator the phonetic form or forms of each word on which the system is based to vocalize the text. The operator intervenes on the choice of the variants of pronunciation, the connections, the e dumb, ... It should be noted that this module differs from the preceding module on the lexicon in that it does not modify the data but the result of the phonetization process.

The seventh module deals with grammatical analysis. The editor presents the operator with the result of the grammar analysis and the rules that resulted in this result. The operator can modify the choice of grammar rules and markers associated with each word or group of words.

The eighth module is about prosody. The editor presents the operator with prosodic information in the form of curves or tables of values that the operator can modify.

The operation of each elementary processing unit and its associated interfacing module will now be explained in relation to the figure 2 .

Since the text is stored at 20 in the voice synthesis system 1, a complete speech synthesis, until the sound signal is generated, is performed at 21. The operator thus has a reference sound signal for his analysis.

This synthesis 21 comprises successively a linguistic processing step 22 and a concatenation synthesis step 23 as explained above.

During the linguistic processing step 22, one of the units 4A, 4B, 4C of elementary processing generates at 24 intermediate results. For example, the grammatical analysis means generate a grammatical analysis result accompanied by the rules used.

The sound result and the intermediate results obtained are presented to the operator at 25.

If the sound result is in accordance with 26 expectations of the operator, it is validated in 27 and the intermediate results.

If the sound result and / or intermediate results are not in accordance with the expectations of the operator, it modifies the intermediate results in 28 using the corresponding interface module.

These modifications are taken into account at 29 by the voice synthesis system 1 in the form of a modification of the linguistic processing aid parameters contained in the memorized text. Preferably, this consideration is made in the form of an enrichment or a modification of the enrichment of the text to be synthesized.

Then the voice synthesis step 21 is executed again using the new enriched text.

The improvement process loops until the operator is satisfied with the result obtained.

It is conceivable that to obtain a sound flow having all the characteristics desired by the operator, it may be necessary to intervene on several elementary treatments.

In a preferred embodiment, figure 3 , the speech synthesis method further comprises a step 30 of selecting the elementary processing module whose intermediate results will be analyzed and possibly modified by the operation.

Thus, the operator can advantageously choose the type of elementary treatment which he wishes to analyze and modify the results.

Advantageously, figure 4 the modifications are made in the order of presentation of the following elementary treatment units.

The operator starts at 40 by editing the text via the first module associated with the basic processing units of the text itself.

Then, when he has obtained a satisfactory result at this level, the operator launches at 41 the second module for cutting the text into sentences.

After obtaining a satisfactory intermediate result, it launches at 42 the third module of division into groups of breath, then in 43 the fourth word cutting module, then 44 the fifth module of the lexicon, then 45 the sixth word phonation module, then 46 the seventh grammatical analysis module, then 47 the eighth prosody module.

This embodiment is remarkable in that it follows a logical order for the operator but does not correspond to the organization of the processing within a linguistic analyzer of a conventional speech synthesis system.

The operator can also go back to modify the intermediate results of one of the modules already treated, for example because he noticed a mistake late.

Claims (10)

1. Voice synthesis system based on concatenation of acoustic units comprising:

   - means (3) of storing a text to be synthesized and linguistic processing aid parameters,

   - means (4) for the linguistic processing of said text for transforming said text into a series of phonemes accompanied by prosodic indications,

   - means (6) of synthesis by concatenation of prerecorded elements for reconstructing an acoustic signal, as a function of the series of phonemes,

   - input and editing means (8),

   characterized in that the linguistic processing means (4) comprise at least one elementary processing unit (4A, 4B, 4C) generating intermediate results of linguistic processing of said text, said elementary processing unit being associated with an editor (8A, 8B, 8C) of the input and editing means (8), allowing an operator to modify the results of the elementary processing unit (4A, 4B, 4C), and in that said voice synthesis system furthermore comprises means (14) for parametrizing the text to be synthesized which are able to modify the linguistic processing aid parameters stored in the storage means (3) as a function of the results modified by the operator, and said linguistic processing means (4) adapting the linguistic processing of the text as a function of said linguistic processing aid parameters.
2. Voice synthesis system according to Claim 1, characterized in that the parametrization of the text comprises tags inserted into the text to be synthesized.
3. Voice synthesis system according to Claim 1 or 2, characterized in that the or each elementary processing unit is adapted for performing one of the elementary processings of the set of elementary processings of:

   a) - validation of the text to be synthesized,

   b) - division of the text into sentences,

   c) - division of the text into breath groups,

   d) - division of the text into words,

   e) - modification of a lexicon of exceptions,

   f) - phonetization of the words,

   g) - grammatical analysis,

   h) - prosody.
4. Voice synthesis system according to Claim 3, characterized in that the linguistic processing means comprise elementary processing means for performing all of the elementary processings of said set of elementary processings.
5. Voice synthesis method based on concatenation of acoustic units comprising the steps of:

   - storage (20) of a text to be synthesized and of linguistic processing aid parameters,

   - linguistic processing (22) of said text to transform said text into a series of phonemes accompanied by prosodic indications,

   - generation (23,24) of a sound signal and of intermediate results on the basis of said series,

   - analysis (25) by an operator of the sound signal and of the intermediate results,

   - modification (28) by the operator of said intermediate results if said operator establishes that the quality of the sound signal is insufficient,

   - creation and/or modification (29) of linguistic processing aid parameters of the text to be synthesized,

   - looping over the linguistic processing step, the latter generating a new series of phonemes taking account of said linguistic processing aid parameters.
6. Voice synthesis method according to Claim 5, characterized in that the modification of the parameters consists in creating/modifying tags in the text to be synthesized.
7. Voice synthesis method according to Claim 5 or 6, characterized in that the step of the generation of intermediate results comprises one of the elementary processing sub-steps:

   - validation of the text to be synthesized,

   - division of the text into sentences,

   - division of the text into breath groups,

   - division of the text into words,

   - modification of a lexicon of exceptions,

   - phonetization of the words,

   - grammatical analysis,

   - prosody.
8. Voice synthesis method according to Claim 7, characterized in that it furthermore comprises a step (30) of selection of the elementary processing sub-step to be executed from among the set of elementary processing sub-steps.
9. Voice synthesis method characterized in that the method of Claim 8 is executed successively 8 times and that each time, a different elementary processing sub-step is selected in the following order:

   - validation of the text to be synthesized,

   - division of the text into sentences,

   - division of the text into breath groups,

   - division of the text into words,

   - modification of a lexicon of exceptions,

   - phonetization of the words,

   - grammatical analysis,

   - prosody.
10. Computer program comprising program code instructions for the execution of the steps of the method according to one of Claims 5 to 9 when said program is executed on a computer.

Images