EP1273003B1 - Method and device for the determination of prosodic markers - Google Patents

Description

The present invention relates to a method for determining prosodic markers and a device for implementation of the procedure.

In the preparation of unknown text for speech synthesis in a TTS system, ('text to speech' systems) or Text / speech translation systems is an essential step the preparation and structuring of the text for the subsequent ones Generation of prosody. To prosodic parameters for speech synthesis systems to produce a two-stage Approach followed. At first, in the first stage produced prosodic markers or prosodic markers, the then converted into physical parameters in the second stage become.

In particular, phrase boundaries can be used as prosodic markers and word accents (pitch-accent) serve. Be under phrases Groupings of words understood within one Textes are usually spoken together, so without intervening inserted lying pauses. pauses lie only at the respective ends of the phrases, the phrase boundaries, at. By inserting such breaks to the Phrase boundaries of synthesized speech become their intelligibility and naturalness significantly increased.

In stage 1, prepare such a two-stage approach both the stable prediction and determination of phrase boundaries as well as accents problems.

In a publication entitled "A hierarchical stochastic model for automatic prediction of prosodic boundary location "by M. Ostendorf and N. Veilleux in Computational Linguistics, 1994, a procedure has been published, in which to determine phrase boundaries "Classification and Regression Trees "(CART) .The initialization Such a procedure requires a high level of expertise. The effort increases with this method with the desired accuracy disproportionately.

At the Eurospeech 1997 conference, under the title "Assigning phase breaks from part-of-speech sequences "by Alan W. Black and Paul Taylor have been published a procedure in which the phrase boundaries with a "hidden Markov model" (HMM) can be determined. To achieve good predictive accuracy for a phrase border is a training text with considerable extent necessary. The creation of these training texts is expensive because expert knowledge is necessary.

The article "An RNN-Based Prosodic Information Synthsizer for Mandarin Text-to-Speech, by Sin-Hong Chen et al., In IEEE Transactions on Speech and Audio Processing ", US, IEEE Inc. New York, Vol. 6, No. 3, May 1, 1998, pages 226-239 a method for determining prosodic marks the markings based on linguistic categories be determined by a neural feed-forward network.

From Yuyiko Yamaguchi et al .: "A Neural Network Approach to Multi-Language Text-to-Speech System ", in:" Proceedings of the International Conference on Spoken Language Processing (ICSLP) ", JP, Tokyo, ASJ, November 18, 1990, pp. 325-328, a method is known in which by means of a networked neural feed-forward network syntactic limits determined become.

Accordingly, it is the object of the present invention, a Process for the preparation and structuring of an unknown to create spoken text with a smaller one Training text can be trained and about similar detection rates as known processes achieved with larger Texts are trained.

This object is achieved by a method according to claim 1 and a neural network according to claim 6.

Accordingly, in a method according to the invention prosodic Tags based on a neural network determined by linguistic categories. Depending on the respective language of a text are subdivisions of the words known in various linguistic categories. In the German language are used in the context of this invention, for example 14 categories, for the English language e.g. 23 Categories provided. Being aware of these categories a neural network is trained to recognize structures can and so on based on groupings of e.g. 3 to 15 consecutive words a prosodic marker predicts or determines.

Furthermore, for a method according to the invention a two-stage Approach selected, the capturing of the properties any prosodic labeling by neuronal autoassociators and evaluating the output from each of the car associators detailed source information, referred to as so-called Error vector is present in a neural classifier includes.

By the inventive use of neural networks is enabled in generating prosodic parameters for Speech synthesis systems accurately predict phrase boundaries.

The neural network according to the invention is robust against "less" or a small amount of training material (sparse training material).

The use of neural networks allows time and money saving Training method and a flexible application a method according to the invention and a corresponding Device on any languages. It is little in addition prepared information and little expert knowledge to initialize of such a system of a particular language required. The neural network according to the invention is therefore well suited to having a multilingual TTS system Synthesize texts from multiple languages. Since the invention neural networks trained without expert knowledge they can be cheaper than known ones Method for determining phrase boundaries to be initialized.

In a further development, the two-stage structure comprises several Autoassociators, each with a phrasing strength be trained for all to be evaluated linguistic classes. Thus, parts of the neural network are class specific educated. The training material is usually statistically asymmetrical, that is, many words without phrase borders, but only a few with phrase boundaries are. In contrast to the state of the art Technology becomes a dominance within a neural network thereby avoiding a class specific training of respective car associates is performed.

Advantageous developments of a method according to the invention are the subject of dependent claims.

The present process will be referred to below explained in detail on the accompanying drawings.

Fig. 1 schematisch ein neuronales Netzwerkwerk gemäß der Erfindung;

Fig. 2 eine Ausgabe bei einfacher Phrasierung anhand eines Beispieltexts;

Fig. 3 ein Beispiel für eine Ausgabe mit ternärer Bewertung der Phrasierung anhand eines Textbeispiels;

Fig. 4 schematisch eine bevorzugte Ausführungsform eines neuronalen Netzwerkes;

Fig. 5 schematisch einen Autoassoziator während des Trainings (links) und während des Betriebs (rechts);

Fig. 6 schematisch ein Blockschaltbild des neuronalen Netzwerkes nach Fig. 4 mit den mathematischen Zusammenhängen; und

Fig. 7 schematisch einen erweiterten Autoassoziator, und

Fig. 8 ein Computersystem zum Ausführen des erfindungsgemäßen Verfahrens in einem Blockschaltbild.

In the drawings shows:

Fig. 1 shows schematically a neural network according to the invention;

FIG. 2 an output with simple phrasing on the basis of an example text; FIG.

3 shows an example of an output with ternary evaluation of the phrasing based on a text example;

Fig. 4 shows schematically a preferred embodiment of a neural network;

5 schematically shows an auto-associate during exercise (left) and during operation (right);

6 shows schematically a block diagram of the neural network according to FIG. 4 with the mathematical relationships; and

Fig. 7 shows schematically an extended autoassociator, and

8 shows a computer system for carrying out the method according to the invention in a block diagram.

FIG. 1 schematically shows a neural network 1 according to the invention with an input 2, an intermediate layer 3 and an output 4 for determining prosodic markings. The input 2 is made up of nine input groups 5 for performing a part-of-speech (POS) sequence analysis. Each of the input group 5 includes in adaptation to the German language 14 neurons 6, which are not all shown in Fig. 1 for reasons of clarity. So there is one neuron 6 each for a linguistic category. For example, the linguistic categories are subdivided as follows: linguistic categories category description NUM numeral VERB verbs VPART Verbpartikel PRON pronoun PREP prepositions NOUN Nouns, proper names PART particle DET items CONJ conjunctions ADV adverbs ADJ adjectives PDET PREP + DET INTJ interjections PUNCT punctuation mark

The output 4 is by a neuron with a continuous Trained, that means that the output values all Values of a certain range of numbers, e.g. all real Numbers between 0 and 1 may include.

In the embodiment shown in Fig. 1 are nine Input groups 5 for entering the categories of the individual Words provided. To the middle input group 5a is the Created category of the word to be determined whether there is a phase boundary at the end of the word or not Phase boundary is present. To the four input groups 5b on the left side of the input group 5a are the categories of the forerunners of the word under investigation and the the right side arranged input groups 5c the successors of the word to be examined. Precursors are all words in the context immediately before the one to be examined Word are arranged. Successors are all words, in the context immediately following that to be examined Word are arranged. This is achieved with the inventive neural network 1 of Fig. 1 is a context by Max. evaluated nine words.

In the evaluation, the category of the to be examined Word applied to the input group 5a, that is, to the neuron 6, which corresponds to the category of the word that Value +1 and to the remaining neurons 6 of the input group 5a the value -1 is created. In a similar way, the Categories of the four preceding to the word to be examined or subsequent words to the input groups 5b, or 5c created. If no corresponding precursors or Successor be present as it is e.g. at the beginning and at the end of a text are sent to the neurons 6 of the corresponding Input groups 5b, 5c, the value 0 is applied.

Another input group 5d is for inputting the previous one Phrase boundaries provided. At this input group 5d The last nine phrase boundaries can be entered.

For the German language - with 14 linguistic categories - the input space has a considerable dimension m of 135 (m = 9 * 14 + 9). A convenient subdivision of the linguistic categories of the English language comprises 23 categories such that the dimension of the input space is 216. The input data form an input vector x with the dimension m.

The neural network according to the invention is equipped with a training file trains who have a text and the information too includes the phrase boundaries of the text. These phrase boundaries can contain purely binary values, that is, only information, if there is a phrase boundary or if no Phrase boundary exists. Will the neural network with a training such a training file, so the output is on Output 4 binary. The output 4 generates continuous Output values, but by means of a threshold value decision be assigned to discrete values.

In Fig. 2, an example sentence is shown behind the In each case, word and phrases have a phrase boundary having. Behind the other words of this example sentence there is no phrase boundary.

For certain applications, it is advantageous if the output contains not only binary values, but multi-level values, that is, information about the strength of the phrase boundary be taken into account. This is the neural network to train with a training file containing multi-level information to the phrase boundaries. The gradation can from two stages to any number of stages, so that a quasi-continuous output can be achieved can.

In Fig. 3 is an example sentence with a three-stage evaluation with the output values 0 for no phrase boundary, 1 for a primary phrase boundary and 2 for a secondary phrase boundary shown. After the term "secondary" is located a secondary phrase boundary and the terms "Phrase boundary" and "required" a primary phrase boundary.

In Fig. 4 is a preferred embodiment of the invention represented neural network. This neural network again comprises an input 2, which in Fig. 4 only is shown schematically as an element, but just like the input 2 of Fig. 1 is constructed. The intermediate layer 3 There are several autoassociators in this embodiment 7 (AA1, AA2, AA3) which each have a model for one represent predetermined phrasing strength. The car associates 7 are subnetworks that detect a specific Phrasing strength are trained. The output of the Autoassoziatoren 7 is connected to a classifier 8. The classifier 8 is another neural subnetwork, this also the already described with reference to FIG. 1 output includes.

The embodiment shown in Fig. 4 comprises three autoassociators, with each car associate a specific Phrasing strength can be detected, so that this embodiment for the detection of two different phrasing strengths and the absence of any phrasing limit suitable is.

Each car associate will be using the data of the class he is using represents, trains. That is, each car associator with the belonging to the phrasing strength he represents Data is trained.

The autoassociators map the m-dimensional input vector x to an n-dimensional vector z , where n << m. The vector z is mapped to an output vector x '. The mappings are done by means of matrices w ₁ ∈ R ^{n × m} and w ₂ ∈ R ^{n × m} . The entire image executed in the Autoassociators can be represented by the following formula: x '= w 2 Tanh (w 1 X), where tanh is applied element by element.

The autoassociators are trained so that their output vectors x ' match as closely as possible with the input vectors x (Figure 5 left side). As a result, the information of the m-dimensional input vector x is compressed to the n-dimensional vector z. It assumes that no information is lost and the model captures the properties of the class. The compression ratio m: n of the individual Autoassoziatoren may be different.

During training, only the input vectors x corresponding to the states at which the phrase boundaries associated with the respective autoassociators occur are applied to the input and output sides of the individual autoassociators.

During operation, an error vector e _rec = (x-x ') ^{2 is} calculated for each auto-associate (FIG. 5, right-hand side). The squaring takes place elementwise. This error vector e _rec is a "dimension" that x 'x corresponds to the distance of the vector to the input vector and is thus inversely proportional to the probability that the assigned to the respective autoassociator phrase boundary is present.

The complete the carassocators and the classifier comprehensive neural network is shown schematically in Fig. 6. It shows car associates 7 for k classes.

wobei A_i(x)=w₂ ⁽ⁱ⁾tanh(w₁ ⁽ⁱ⁾x) gilt und tanh als elementweise Operation ausgeführt wird und diag (w₁ ⁽ⁱ⁾,...,w_m ⁽ⁱ⁾) ∈ R^m×m eine Diagonalmatrix mit den Elementen (w₁ ⁽ⁱ⁾,...,w_m ⁽ⁱ⁾) darstellt.The elements p _{i of} the output vector p are calculated according to the following formula:

where A _i (x) = w ₂ ⁽ⁱ⁾ tanh (w ₁ ⁽ⁱ⁾ x) and tanh is performed as an elementwise operation and diag (w ₁ ⁽ⁱ⁾ , ..., w _m ⁽ⁱ⁾ ) ∈ R ^{m × m is} a diagonal matrix with the elements (w ₁ ⁽ⁱ⁾ , ..., w _m ⁽ⁱ⁾ ).

The individual elements p _{i of} the output vector p indicate the probability with which a phrase boundary has been detected at the autoassociator i.

If the probability p _{i is} greater than 0.5, this is evaluated as having a corresponding phrase boundary i. If the probability p _{i is} less than 0.5, this means that the phrase limit i is not present here.

If the output vector p has more than two elements p _i , it is expedient to evaluate the output vector p in such a way that the phrase boundary is present whose probability p _{i is} greatest in comparison to the other probabilities p _{i of} the output vector p .

In a development of the invention, it may be expedient, if a phrase boundary is determined whose probability p _{i is} in the range of 0.5, for example in the range of 0.4 to 0.6, to carry out a further routine with which the existence the phrase boundary is checked. This further routine can be based on both a rule-driven and a data-driven approach.

When training with a training file that includes appropriate phrasing information, in a first training phase, the individual Autoassoziatoren 7 each trained to their predetermined Phrasierungsstärke. As stated above, the input vectors x corresponding to the phrase boundary associated with the respective auto-associate are applied to the input and output sides of the individual auto-associates 7, respectively.

In a second training phase, the weighting elements of the autoassociators 7 are recorded and the classifier 8 is trained. At the input side of the classifier 8 the error vectors e _{rec of} the autoassociators and at the output side the vectors containing the values for the different phrase boundaries are applied. In this training phase, the classifier learns from the error vectors to determine the output vectors p .

In a third training phase, a fine adjustment of all Weighting elements of the entire neural network (the k car associates and the classifier).

Due to the above-described architecture of a neural Network with several each to a specific class trained models (here: the car associators) and one parent classifier, it is possible to use an input vector having a very large dimension on an output vector with small dimension or a scalar reliable correct image. This network architecture can also be beneficial in other applications are used, in which elements different classes have to be treated. That's the way it works e.g. be expedient, this network architecture also in the Speech recognition for detecting word and / or sentence boundaries use. The input data must be adapted accordingly.

The classifier 8 shown in FIG. 6 has weighting matrices GW, which are each assigned to an auto-associate 7. The weighting matrix GW associated with the i-th auto-associate 7 has weighting factors w _n in the i-th row. The remaining elements of the matrix are equal to zero. The number of weighting factors w _n corresponds to the dimension of the input vector, wherein in each case a weighting element w _{n is} related to a component of the input vector. If a weighting element w _{n has} a larger value than the remaining weighting elements w _{n of} the matrix, this means that the corresponding component of the input vector is of great importance for the determination of the phrase boundary determined by the autoassociator to which the corresponding weighting matrix GW is assigned ,

In a preferred embodiment, advanced autoassociators are used (Figure 7) that allow for better detection of nonlinearities. These enhanced auto-associates do the following: x '= w 2 tanh (·) + w 3 (tanh (·)) 2 . where (·): = ( w ₁ · x ) and the squaring (·) ² and tanh are performed element by element.

In experiments, a neural network according to the invention has been trained with a predetermined English text. The same text was used to train an HMM recognizer. The performance criteria used were the percentage of correctly recognized phrase boundaries (B-corr), the total correctly rated words, whether one or no phrase boundary follows (total), and the non-correctly recognized words without phrase boundary (NB-ncorr) determined. In these experiments, a neural network with the autoassociators of Fig. 6 and a neural network with the extended autoassociators were used. The following results were achieved: B-corr total NB-ncorr ext. Autoass. 80.33% 91.68% 4.72% Autoass. 78.10% 90.95% 3.93 HMM 79.48% 91.60% 5.57%

The results shown in the table show that the neural networks according to the invention with respect to the correctly recognized Phrase boundaries and the total correctly recognized Words yield approximately the same results as an HMM recognizer. However, the neural networks according to the invention are. the erroneously detected phrase boundaries, in places where it in itself there is no phrase limit, much better than that HMM recognizer. This kind of mistake is in the language-to-text implementation particularly serious, since these errors are the one Immediately generate striking false accentuation.

In further experiments, one of the neural networks according to the invention was trained with a fraction of the training text used in the above experiments (5%, 10%, 30%, 50%). The following results were achieved: Fraction of the training text B-corr total NB-ncorr 5% 70.50% 89.96% 4.65% 10% 75.00% 90.76% 4.57% 30% 76.30% 91.48% 4.16% 50% 78.01% 91.53% 4.44%

With fractions of 30% and 50% of the training text were achieved excellent recognition rates. With a fraction 10% and 5% of the original training text are satisfactory Detection rates have been achieved. This shows, that the inventive neural networks even at low Training range provide good recognition rates. This represents a significant advance over known ones Phrase boundary detection methods, since the processing of Training material is costly, because here expert knowledge must be used.

The embodiment described above has k autoassociators on. For a precise evaluation of the phrase boundaries It may be appropriate to have a large number of car associates to use, taking up to 20 auto-associates appropriate could be. This is a quasi-continuous course achieved the output values.

The neural networks described above are computer programs realized independently on a computer to translate the linguistic category of a text into whose prosodic marker expire. You thus stop automatically executable process.

The computer program can also be stored on an electronically readable Disk will be saved and so on another Computer system to be transferred.

A suitable for the application of the method according to the invention Computer system is shown in FIG. The computer system 9 has an internal bus 10 having a memory area 11, a central processing unit 12 and an interface 13 is connected. The interface 13 provides via a Data line 14 a data connection to other computer systems ago. On the internal bus are also an acoustic Output unit 15, a graphic output unit 16 and a Input unit 17 connected. The acoustic output unit 15 is a speaker 18, the graphical output unit 16 with a screen 19 and the input unit 17 connected to a keyboard 20. To the computer system 9 can transmitted over the data line 14 and the interface 13 text are stored in the memory area 11. The memory area 11 is divided into several areas, in which texts, audio files, application programs for Carrying out the method according to the invention and other application and utilities are stored. The as a text file stored texts are with predetermined program packages analyzed and the respective linguistic categories of words. Thereafter, with the inventive Procedures from the linguistic categories the determined prosodic markers. These prosodic markers will be again entered into another program package, the using prosodic markers to create audio files, via the internal bus 10 to the acoustic output unit 15 transmitted and from this on the speaker 18 as Language are output.

In the examples presented here is only one application of the method for the prediction of phrase boundaries Service. The method can be similar construction of a Device and adapted training but also to Evaluation of an unknown text regarding a prediction of stresses, e.g. according to the internationally standardized ToBI-labels (tones and breaks indices), and / or the sentence melody are used. These adjustments have in Dependence on the particular language of the processed Text to be done, since the prosody always language-specific is.

Claims (12)

1. Method for determining prosodic markers, phrase boundaries and word accents serving as prosodic markers,

   prosodic markers being determined by a neural network (1) on the basis of linguistic categories,

   characterized by the following steps;

   acquisition of the properties of each prosodic marker by neural autoassociators (7) which are trained to in each case one specific prosodic marker, and

   evaluation of the output information output by each of the autoassociators (7) in a neural classifier (8).
2. Method according to Claim 1,
   characterized in that, as prosodic markers, phrase boundaries are determined and preferably also evaluated and/or assessed.
3. Method according to Claim 1 and/or Claim 2,
   characterized in that the linguistic categories of at least three words of a text to be synthesized are applied to the input (2) of the network (1).
4. Method according to one of the preceding claims,
   characterized in that the autoassociators (1) are trained for a respective predetermined phrase boundary.
5. Method according to Claim 4,
   characterized in that the neural classifier (8) is trained after the training of all the autoassociators (7).
6. Neural network for determining prosodic markers, phrase boundaries and word accents serving as prosodic markers, having an input (2), an intermediate layer (3) and an output (4), the input being designed for acquiring linguistic categories of words of a text to be analysed,
   characterized in that properties of each prosodic marker can be acquired by neural autoassociators (7) which are trained to in each case one specific prosodic marker, and
   in that the output information output by each of the autoassociators (7) can be evaluated in a neural classifier (8) .
7. Neural network according to Claim 6,
   characterized in that the intermediate layer (3) has at least two autoassociators (7).
8. Neural network according to Claim 6 or 7,
   characterized in that the input (2) has input groups (5) having a plurality of neurons (6) each assigned to a linguistic category, and each input group serves for acquiring the linguistic category of a word of the text to be analysed.
9. Neural network according to one of Claims 6 to 8,
   characterized in that the network is designed for outputting a binary, tertiary or quaternary phrasing stage.
10. Neural network according to one of Claims 7 to 9,
    characterized in that the network is designed for outputting a quasi-continuous phrasing region.
11. Method according to one of Claims 1 to 5,
    characterized by
    the use of a neural network according to one of Claims 6 to 10.
12. Device for determining prosodic markers having a computer system (9), which has a memory area (11) in which a program for executing a neural network according to one of Claims 6 to 10 is stored.

Images