WO2011000934A1

WO2011000934A1 - Enabling synthesis of speech having a target characteristic

Info

Publication number: WO2011000934A1
Application number: PCT/EP2010/059416
Authority: WO
Inventors: Junichi Yamagishi; Simon Alistair King; Stephen John Renals
Original assignee: University of Edinburgh
Current assignee: University of Edinburgh
Priority date: 2009-07-02
Filing date: 2010-07-01
Publication date: 2011-01-06
Anticipated expiration: 2012-01-02
Also published as: GB0911494D0

Abstract

A method involving: inputting target speech features derived from speech having a target characteristic; inputting labelling of the speech; and adapting, using a speech feature transform, statistical parameters of a predefined statistical parametric speech model to create an adapted statistical parametric speech model configured to synthesis speech having the target characteristic.

Description

TITLE

Enabling synthesis of speech having a target characteristic.

TECHNOLOGICAL FIELD

Embodiments of the present invention relate to enabling synthesis of speech having a target characteristic. BACKGROUND

It is desirable to be able to use a machine to produce speech from non- speech input without having to record the actual speech produced. The most common application of speech synthesis is in ^'text to speech' in which the input is written language and the output is a speech signal.

Most commercial systems for synthesising speech use a method called ^'unit selection' that relies on a large database of several hours of speech from a single human speaker (the ^'target speaker'), typically recorded by a

professional in a studio on high quality equipment under carefully controlled conditions, meticulously labelled and carefully checked. To synthesise new sentences, small fragments of speech from the database are selected using an automatic procedure, concatenated together and played back. This works very well, but is expensive and inflexible. In particular, creating a new voice for such a system requires considerable expertise and labour, and therefore costs a substantial amount of money and is well beyond the reach of individual users.

BRIEF SUMMARY

An alternative to unit selection is a method which also uses a database of recorded speech, but this time to construct a statistical model of speech from which any required new sentence can be generated. The statistical model comprises a number of statistical parameters which collectively describe the characteristics of speech. The model allows the generation of new words and phrases not contained in the original database. Because the model is statistical, it is more tolerant of certain problems or shortcomings in the data. It does not require such careful labelling or checking of the data and can give good performance with a smaller database of about one hour of speech from the target speaker.

According to some but not necessarily all embodiments of the invention there is provided a method comprising: inputting target speech features derived from speech having a target characteristic; inputting labelling of the speech; and adapting, using a speech feature transform, statistical parameters of a predefined statistical parametric speech model to create an adapted statistical parametric speech model configured to synthesis speech having the target characteristic.

According to some but not necessarily all embodiments of the invention there is provided an apparatus comprising: means for producing target speech features from speech having a target characteristic; means for receiving labelling of the speech; and means for adapting, using a speech feature transform, statistical parameters of a predefined statistical parametric speech model to create an adapted statistical parametric speech model configured to synthesis speech having the target characteristic.

According to some but not necessarily all embodiments of the invention there is provided an apparatus comprising: adaption circuitry configured to adapt statistical parameters of a predefined statistical parametric speech model to create an adapted statistical parametric speech model configured to

synthesise speech having a target characteristic, wherein the adaptation of the statistical parameters uses a transform to transform speech features derived from speech having a target characteristic.

Embodiments of the invention may allow the creation of new voices using just a few minutes of recorded speech from the target speaker. Embodiments may also utilise noisy or otherwise imperfect recordings. The parameters of the statistical parametric speech model are automatically adapted, using a transform of speech features, to match the characteristics of speech from the target speaker. BRIEF DESCRIPTION

For a better understanding of various examples of embodiments of the present invention reference will now be made by way of example only to the accompanying drawings in which:

Fig 1 schematically illustrates an example of a system and method for predefining statistical parametric models of speech;

Fig 2 schematically illustrates an example of a system and method for producing synthesised speech using the predefined statistical parametric models of speech;

Fig 3 schematically illustrates an example of a system and method that adapts existing statistical parametric models of speech to create an adapted statistical parametric model of speech that is optimized to create target synthetic speech;

Fig 4 schematically illustrates an example of a system and method for producing synthesised speech using the adapted statistical parametric models of speech;

Fig 5 schematically illustrates one example of a method that may occur at the model training block of Fig 1 ;

Fig 6 schematically illustrates the operation of the model adaptation block of

Fig 3;

Fig 7 schematically illustrates one example of the method 48 that may occur at the model adaptation block Fig 3; and

Fig 8 schematically illustrates an apparatus suitable for providing the aforementioned systems and methods.

DETAILED DESCRIPTION Fig 1 schematically illustrates a system and method for predefining statistical parametric models of speech (reference models) 20.

Annotated reference speech includes the reference speech 10 and labels 12 for the reference speech that provide a linguistic specification of the speech. The reference speech 10 is converted, by parameter extraction block 14, into parameters in the feature space domain. These parameters which may also be called speech features 16 are provided as one input to a model training block 18. Another input to the model training block 18 is the labels 12.

The model training block 18, as is known in the art, determines statistical parametric models of speech (reference models) 20 that have the maximum likelihood of producing the reference speech 10 from the labels 12. Typically the model training block 18 uses machine learning or training to determine the statistical parametric models of speech (reference models) 20.

Fig 2 schematically illustrates a system and method for producing synthesised speech 34 using the predefined statistical parametric models of speech (reference models) 20.

Text 22 is analysed at block 24 to produce labels 26 that provide a linguistic specification. The labels 26 and the reference model 20 are inputs to speech generator block 28. The speech generator block 28 may use labels 26 to select appropriate reference models 20 for speech generation. The selected reference models are then used to generate parameters 30 from the labels 26. These parameters may also be called speech features 30.

The speech features are input into a speech synthesis engine 32 which generates synthetic speech 34.

Typically the block 14 that performs parameter extraction converts the reference speech 10 from the time domain to the frequency domain. It may for example be a suitable voice coder. Some voice coders produce as outputs excitation parameters and spectral parameters, for example. The reference speech 10 is typically carefully produced in a studio and is comprehensive in its inclusion of phonemes so that the reference models 20 are accurate and comprehensive. Creating and annotating the reference speech 10 may be expensive and time consuming.

One or more statistical parametric models of speech (reference models) 20 may be defined for each different phoneme. One suitable statistical parametric model is a Hidden State Statistical Model, such as a Hidden Markov Model (HMM) or Hidden Semi-Markov Model (HSMM), which is defined using statistical parameters. Each hidden state has at least one probability distribution for the possible output values of the state. The model may therefore associate each possible output vector with a probability distribution, defined using statistical parameters. If the distribution is Gaussian, the distribution can be defined by the using the mean and variance as the statistical parameters. A HMM is defined by initial state probabilities, state transition probabilities, and state output probability distributions. A Hidden Semi Markov Model (HSMM) may have state duration probability distribution instead of or in addition to state transition probabilities.

If an /V-state left-to-right HSMM λ with no skip paths is specified by a state / output probability distribution £>, and a state duration probability distribution p, then the probability distribution £>, is dependent upon the output observation vector and statistical parameters (mean vector and covariance matrix) and the state duration d is dependent upon state duration probability p, and the statistical parameters (mean and variance).

The model training block 18 may determine maximum likelihood estimates of the parameters of the Model given a target output observation vector. The model is found that maximises the observation probability of the reference speech 10 for the labels 12. This may be achieved using the Baum-Welch algorithm. Fig 5 schematically illustrates one example of a method that may occur at the model training block 18 of Fig 1. Initially at block 60, the speech features 16, labels 12 and current reference model are time aligned. Then at block 62 the expected values of probability distribution b_\ and the state duration probability pi are determined using the existing statistical parameters. bi (o) = N (o; _μi;∑i)

where

o is the output observation vector

μι is a mean vector (statistical parameter)

∑i is a covahance matrix (statistical parameter)

d is the state duration

/7?_/ is the mean duration (statistical parameter)

σ_i ² is the variance of the duration (statistical parameter)

Then at block 64, the statistical parameters are updated. For example, the expected values may be used to re-estimate the statistical parameters of the model. This may, for example, use re-estimation formula based on the Baum- Welch algorithm.

Then at block 66, the model is updated using the expected values determined at block 62 and the re-estimated statistical parameters determined at block 64.

,1 = argmax P{O where y x

i≠i

The process is then repeated until there is convergence. The current model may then be output as the reference model 20.

The inventors have realised that it would be beneficial to somehow use the existing statistical parametric models of speech (reference models) 20 for creating target synthetic speech with different target characteristics. These characteristics may include speaker identity, emotion, urgency, gravity, accent etc.

Fig 3 schematically illustrates an example of a system and method by which this may be achieved. Annotated target speech includes the target speech 40 and labels 42 for the target speech that provide a linguistic specification of the target speech. The target speech 40 is converted, by the parameter extraction block 44, into parameters in the feature space domain. These parameters which may also be called speech features 46 are provided as one input to a model adaptation block 48. Another input to the model adaptation block 48 is the labels 42.

The model adaptation block 48 determines target statistical parametric models of speech (target models) 50 that have the maximum likelihood of producing the target speech 40 from the labels 42. The target statistical parametric models of speech (target models) 50 are based upon the reference models 20.

Fig 4 schematically illustrates a method for producing synthesised speech 54 using the adapted statistical parametric models of speech (target models) 50. Text 22 is analysed at block 24 to produce labels 26 that provide a linguistic specification. The labels 26 and the reference model 20 are input to speech generator block 28. The speech generator block 28 may use a label 26 to select an appropriate one of the target models 50 for speech generation. The selected model is then used to generate parameters 52. These parameters may also be called speech features 30.

The speech features 52 are input into a speech synthesis engine 32 which generates synthetic target speech 54.

Fig 6 schematically illustrates the operation of the model adaptation block 48. The model adaptation block 48 determines speech feature transforms 76 that transform the speech features 74 defined in the reference models. Each phoneme 7O_j has its own associated reference model 2O_j. A transform 76_j transforms the speech feature 74_j of the reference model 2O_j to the target speech features 72_j. These speech feature transforms 76_j are used to create target models 5O_j as illustrated in Fig 7, The transforms 76_j are determined, in speech feature space, that adapt the reference model 20 such that the maximum likelihood is that the adapted model (target model 50) creates as output the target speech features 46.

The transforms 76 minimise the differences between the target output 72 and the model output 74 by transforming the output observation vector of the reference model 20 in feature space as part of the maximum likelihood determination that estimates the maximum likelihood statistical parameters of the target model 50 given the target output vector (target speech features 46).

As an example, if an /V-state left-to-right HSMM λ with no skip paths is specified by a state / output probability distribution £>, and a state duration probability distribution p, then the probability distribution ό, is dependent upon the output observation vector o and statistical parameters (mean vector and covahance matrix) and the state duration d is dependent upon state duration probability p, and the statistical parameters (mean and variance). Then a first transform may be applied to the output observation vector o and a second transform may be applied to the state duration d.

For example, the observation vector may undergo a linear transform such as: o -> ξ o + ε

where

ξ is a matrix that can scale, rotate and shear

ε is a vector that shifts

This results in the expected values of the probability distribution b, being expressed as b,(o)= |ξ| N(ξo +ε; _Mi; Z₁)

For example, the state duration may undergo a linear transform such as: d -> x d + v where

X is a scalar

v is a shift

This results in the expected values of the probability distribution p, being expressed as

_Pl(ό)= IxI N( xd +v; m_ι;σ²,)

Fig 7 schematically illustrates one example of the method 48 that may occur at the model adaptation block 48 of Fig 3. Initially at block 82, the original speech features 46, labels 42 and the reference model 20 are time aligned. The reference model 20 is

predetermined as previously described with reference to Fig 1 and/or Fig5. Next at block 82 the expected values of probability distribution b, and the state duration probability p, are determined using the existing statistical parameters (μ,; Σ, and m,; σ²,).

6, (o) = | ξ | N (ξ o + ε; μ_i;∑_s)

_Pl (ό) = I x I N ( x d + v ; m_ι; σ²,)

Then at block 84, the statistical parameters are updated. For example, the expected values may be used to re-estimate the statistical parameters of the model. This may, for example, use re-estimation formula based on the Baum- Welch algorithm.

The probability density functions are given by:

where F is the number of training speakers, O = {Oi, ...O_F} is all the training data, and Of={ θif,...θτf} is training data of length T_f for speaker f, where otf -> ξ_f otf + Ef and χ_f -> χ_f d + v_f

and where γf (i) is the state occupancy probability of being in state / of the

HSMM for the period of time from t-d+ϊ to t given O_f and is defined as

i ^Λ" *

Pi OAX )

Here, the observation probability of the training data O_f given the model λ, P(Of | λ) and the forward and backward probabilities, α_t(i) βt(i), can be written as .\ Λ^" t *

) 4i /)

: ~ 1 ; ~ 1 <!:::. ! <- - f - ,M- l

O_f U ) = O_f- ,i [j J *ϊ _jc JK { if) TT h_t [ O_fl )

r^ j _J = I * -t+ \ where αo(i)= TT, and βτ(i)=1 , and π, is the initial state probability of being state / at time t=1.

Then at block 88, the feature space transforms are updated using the calculated expected values and the re-estimated statistical parameters.

A = ttr giiKXX P( O\A ).

The process is then repeated until there is convergence. The target model 50 is defined by the final statistical parameters determined at block 86. Adapting existing reference statistical parametric models 20 to create adapted models 50 for creating target synthetic speech is a cost effective way of synthesising target speech using a small amount of input target speech 40.

Constraining the maximum likelihood estimate of the parameters of the statistical model using transforms in the speech feature domain allows the same transform to be applied to both the mean and variance of a Gaussian density function which reduces processing complexity and increase memory efficiency.

Constraining the maximum likelihood estimate of the parameters of the statistical model using transforms in the speech domain automatically removes dependency on non-speech content e.g. background noise etc.

Fig 8 schematically illustrates an apparatus 90 suitable for providing the aforementioned systems and methods.

A processor 92 is configured to read from and write to the memory 94. The processor 92 may also comprise an output interface 93 via which data and/or commands are output by the processor 92 and an input interface 91 via which data and/or commands are input to the processor 92.

The memory 94 stores a computer program 96 comprising computer program instructions that control the operation of the apparatus 90 when loaded into the processor 92. The computer program instructions 96 provide the logic and routines that enables the apparatus to perform the methods illustrated in the Figs. The processor 92 by reading the memory 94 is able to load and execute the computer program 96.

The computer program may arrive at the apparatus 90 via any suitable delivery mechanism. The delivery mechanism may be, for example, a computer-readable storage medium, a computer program product, a memory device, a record medium such as a CD-ROM or DVD, an article of

manufacture that tangibly embodies the computer program. The delivery mechanism may be a signal configured to reliably transfer the computer program.

Although the memory 94 is illustrated as a single component it may be implemented as one or more separate components some or all of which may be integrated/removable and/or may provide permanent/semi-permanent/ dynamic/cached storage. The blocks illustrated in the Figs may represent steps in a method and/or sections of code in the computer program and/or functional modules in a system. The illustration of a particular order to the blocks does not necessarily imply that there is a required or preferred order for the blocks and the order and arrangement of the block may be varied. Furthermore, it may be possible for some steps to be omitted.

Although embodiments of the present invention have been described in the preceding paragraphs with reference to various examples, it should be appreciated that modifications to the examples given can be made without departing from the scope of the invention as claimed.

Features described in the preceding description may be used in combinations other than the combinations explicitly described.

Although functions have been described with reference to certain features, those functions may be performable by other features whether described or not.

Although features have been described with reference to certain

embodiments, those features may also be present in other embodiments whether described or not. Whilst endeavouring in the foregoing specification to draw attention to those features of the invention believed to be of particular importance it should be understood that the Applicant claims protection in respect of any patentable feature or combination of features hereinbefore referred to and/or shown in the drawings whether or not particular emphasis has been placed thereon.

Claims

1. A method comprising:

inputting target speech features derived from speech having a target characteristic;

inputting labelling of the speech; and

adapting, using a speech feature transform, statistical parameters of a predefined statistical parametric speech model to create an adapted statistical parametric speech model configured to synthesis speech having the target characteristic.

2. A method as claimed in claim 1 wherein the target speech features are obtained by parametrically encoding speech recorded by a consumer.

3. A method as claimed in claim 1 or 2, wherein the target speech features are obtained by parametrically encoding speech of short duration that is speech of less than thirty minutes duration and preferably less than approximately ten minutes duration.

4. A method as claimed in any preceding claim, wherein the predefined statistical parametric speech model is a hidden state statistical model.

5. A method as claimed in any preceding claim, wherein the hidden state statistical model is a Hidden Markov Model or Hidden Semi-Markov Model.

6. A method as claimed in any preceding claim, wherein the statistical parameters define a distribution and where both the mean and variance of any particular Gaussian are transformed using the same transform.

7. A method as claimed in any preceding claim, wherein the statistical parameters include mean and variance or mean vector and co-variance matrix.

8. A method as claimed in any preceding claim, further comprising

determining the transform, in speech feature space, that adapts the predefined statistical parametric speech model such that the adapted statistical parametric speech model creates speech with the target

characteristic with maximum likelihood.

9. A method as claimed in any preceding claim, wherein the transforms are linear transforms.

10. A method as claimed in any preceding claim, wherein both the transforms and the adapted statistical parametric speech model are simultaneously trained using the input speech features and the input labelling.

11. A method as claimed in any preceding claim, wherein transforms simultaneously transform both a state output and a state duration.

12. A method comprising synthesising speech using an adapted statistical parametric speech model produced by the method of any one of claims 1 to 11.

13. An apparatus comprising:

means for producing target speech features from speech having a target characteristic;

means for receiving labelling of the speech; and

means for adapting, using a speech feature transform, statistical parameters of a predefined statistical parametric speech model to create an adapted statistical parametric speech model configured to synthesis speech having the target characteristic.

14. An apparatus as claimed in claim 13, further comprising menas for performing the method of any one of claims 2 to 12.

15. An apparatus comprising: adaption circuitry configured to adapt statistical parameters of a predefined statistical parametric speech model to create an adapted statistical parametric speech model configured to synthesise speech having a target characteristic, wherein the adaptation of the statistical parameters uses a transform to transform speech features derived from speech having a target characteristic.