Classifications

• • G—PHYSICS
  • G10—MUSICAL INSTRUMENTS; ACOUSTICS
  • G10L—SPEECH ANALYSIS TECHNIQUES OR SPEECH SYNTHESIS; SPEECH RECOGNITION; SPEECH OR VOICE PROCESSING TECHNIQUES; SPEECH OR AUDIO CODING OR DECODING
  • G10L13/00—Speech synthesis; Text to speech systems
  • G10L13/02—Methods for producing synthetic speech; Speech synthesisers
• • G—PHYSICS
  • G10—MUSICAL INSTRUMENTS; ACOUSTICS
  • G10L—SPEECH ANALYSIS TECHNIQUES OR SPEECH SYNTHESIS; SPEECH RECOGNITION; SPEECH OR VOICE PROCESSING TECHNIQUES; SPEECH OR AUDIO CODING OR DECODING
  • G10L25/00—Speech or voice analysis techniques not restricted to a single one of groups G10L15/00 - G10L21/00
  • G10L25/75—Speech or voice analysis techniques not restricted to a single one of groups G10L15/00 - G10L21/00 for modelling vocal tract parameters

Definitions

• the present invention generally relates to telecommunications systems and methods, as well as speech synthesis. More particularly, the present invention pertains to the formation of the excitation signal in a Hidden Markov Model based statistical parametric speech synthesis system.
• the excitation signal may be formed by using a plurality of sub-band templates instead of a single one.
• the plurality of sub-band templates may be combined to form the excitation signal wherein the proportion in which the templates are added is dynamically based on determined energy coefficients. These coefficients vary from frame to frame and are learned, along with the spectral parameters, during feature training.
• the coefficients are appended to the feature vector, which comprises spectral parameters and is modeled using HMMs, and the excitation signal is determined.
• a method for creating parametric models for use in training a speech synthesis system, wherein the system comprises at least a training text corpus, a speech database, and a model training module, the method comprising: obtaining, by the model training module, speech data for the training text corpus, wherein the speech data comprises recorded speech signals and corresponding transcriptions; converting, by the model training module, the training text corpus into context dependent phone labels; extracting, by the model training module, for each frame of speech in the speech signal from the speech training database, at least one of: spectral features, a plurality of band excitation energy coefficients, and fundamental frequency values; forming, by the model training module, a feature vector stream for each frame of speech using the at least one of: spectral features, a plurality of band excitation energy coefficients, and fundamental frequency values; labeling speech with context dependent phones; extracting durations of each context dependent phone from the labelled speech; performing parameter estimation of the speech signal, wherein the parameter estimation is performed comprising the features,
• a method for identification of sub-band Eigen pulses from a glottal pulse database for training a speech synthesis system comprising:
• decomposing determining a vector representation of each database; determining Eigen pulse values, from the vector representation, for each database; and selecting a best Eigen pulse for each database for use in synthesis.
• Figure 1 is a diagram illustrating an embodiment of a Hidden Markov Model based text to speech system.
• Figure 2 is a flowchart illustrating an embodiment of a process for feature vector extraction.
• Figure 3 is a flowchart illustrating an embodiment of a process for feature vector extraction.
• Figure 4 is a flowchart illustrating an embodiment of a process for identification of Eigen pulses.
• Figure 5 is a flowchart illustrating an embodiment of a process for speech synthesis.
• excitation is generally assumed to be a quasi-periodic sequence of impulses for voiced regions. Each sequence is separated from the previous sequence by some duration, such as i
• T 0 — , where T 0 represents pitch period and F 0 represents fundamental frequency. In unvoiced regions, it is modeled as white noise. However, in voiced regions, the excitation is not actually impulse sequences. The excitation is instead a sequence of voice source pulses which occur due to vibration of the vocal folds and their shape. Further, the pulses' shapes may vary depending on various factors such as: the speaker, the mood of the speaker, the linguistic context, emotions, etc.
• Source pulses have been treated mathematically as vectors by length normalization (through resampling) and impulse alignment, as described in European Patent EP 2242045 (granted June 27, 2012, inventors Thomas Drugman, et al.), for example.
• the final length of the normalized source pulse signal is resampled to meet the target pitch.
• the source pulse is not chosen from a database, but obtained over a series of calculations which compromise the pulse characteristics in the frequency domain. Modeling of the voice source pulses has traditionally been done using acoustic parameters or excitation models for HMM based systems, however, the models interpolate/re-sample the glottal/residual pulse to meet the target pitch period, which compromises the model pulse
• glottal pulses may be modeled by defining metrics and providing a vector representation. Excitation formation, given a glottal pulse and fundamental frequency, is also presented which does not re-sample or interpolate on the pulse.
• speech unit signals are represented by a set of parameters which can be used to synthesize speech.
• the parameters may be learned by statistical models, such as HMMs, for example.
• speech may be represented as a source-filter model, wherein source/excitation is a signal which, when passed through an appropriate filter, produces a given sound.
• Figure 1 is a diagram illustrating an embodiment of a Hidden Markov Model (HMM) based Text to Speech (TTS) system, indicated generally at 100.
• HMM Hidden Markov Model
• TTS Text to Speech
• An embodiment of an exemplary system may contain two phases, for example, the training phase and the synthesis phase, each of which are described in greater detail below.
• the Speech Database 105 may contain an amount of speech data for use in speech synthesis.
• Speech data may comprise recorded speech signals and corresponding transcriptions.
• a speech signal 106 is converted into parameters.
• the parameters may be comprised of excitation parameters, F0 parameters, and spectral parameters.
• Excitation Parameter Extraction 110a, Spectral Parameter Extraction 110b, and F0 Parameter Extraction 110c occur from the speech signal 106, which travels from the Speech Database 105.
• a Hidden Markov Model may be trained using a training module 115 using these extracted parameters and the La bels 107 from the Speech Database 105. Any number of HMM models may result from the training and these context dependent HM Ms are stored in a database 120.
• the synthesis phase begins as the context dependent HMMs 120 are used to generate parameters 135.
• the parameter generation 135 may utilize input from a corpus of text 125 from which speech is to be synthesized from. Prior to use in parameter generation 135, the text 125 may undergo analysis 130. During analysis 130, labels 131 are extracted from the text 125 for use in the generation of parameters 135. In one embodiment, excitation parameters and spectral parameters may be generated in the parameter generation module 135.
• the excitation parameters may be used to generate the excitation signal 140, which is input, along with the spectral parameters, into a synthesis filter 145.
• Filter parameters are generally Mel frequency cepstral coefficients (MFCC) and are often modeled by a statistical time series by using HMMs.
• MFCC Mel frequency cepstral coefficients
• the predicted values of the filter and the fundamental frequency as time series values may be used to synthesize the filter by creating an excitation signal from the fundamental frequency values and the M FCC values used to form the filter.
• Synthesized speech 150 is produced when the excitation signal passes through the filter.
• spectral parameters used in a statistical parametric speech synthesis system comprise MCEPS, MGC, Mel-LPC, or Mel-LSP.
• spectral parameters are mel-generalized cepstral (MGC) computed from the pre-emphasized speech signal, but the zeroth energy coefficient is computed from the original speech signal.
• MMC mel-generalized cepstral
• the fundamental frequency value alone is considered as a source parameter and the entire spectrum is considered as a system parameter.
• the spectral tilt, or the gross spectral shape, of the speech spectrum is actually a characteristic of the glottal pulse and is thus considered as a source parameter.
• the spectral tilt is captured and modeled for glottal pulse based excitation and excluded as a system parameter. Instead, pre-emphasized speech is used for computing the spectral parameter (MGC) with exception of the zeroth energy coefficient (energy of speech). This coefficient varies slowly in time and may be treated as a prosodic parameter computed directly from unprocessed speech.
• Figure 2 is a flowchart illustrating an embodiment of a process for feature vector extraction, indicated generally at 200. This process may occur during spectral parameter extraction 110b of Figure 1. As previously described, the parameters may be used for model training, such as with an HMM model.
• the speech signal is received for conversion into parameters.
• the speech signal may be received from a speech database 105.
• Control is passed to operations 210 and 220 and process 200 continues.
• operations 210 and 215 occur simultaneously with operation 220 and the determinations are all passed to operation 225.
• the speech signal undergoes pre-emphasis. For example, pre-emphasizing the speech signal at this stage prevents low frequency source information from being captured in the determination of MGC coefficients in the next operation. Control is passed to operation 215 and process 200 continues.
• spectral parameters are determined for each frame of speech.
• the MGC coefficients 1-39 may be determined for each frame.
• M FCC and LSP may also be used. Control is passed to operation 225 and process 200 continues.
• the zeroth coefficient is determined for each frame of speech. In an embodiment, this may be determined using unprocessed speech as opposed to pre-emphasized speech. Control is passed to operation 225 and process 200 continues. [0025] In operation 225, the coefficients from operations 220 and 215 are appended to 1-39 MGC coefficients to form the 39 coefficients for each frame of speech. The spectral coefficients of a frame may then be referred to as the spectral vector. Process 200 ends.
• Figure 3 is a flowchart illustrating an embodiment of a process for feature vector extraction, indicated generally at 300. This process may occur during excitation parameter extraction 110a of Figure 1. As previously described, the parameters may be used for model training, such as with an HM M model.
• the speech signal is received for conversion into parameters.
• the speech signal may be received from a speech database 105.
• Control is passed to operations 310, 320, and 325 and process 300 continues.
• pre-emphasis is performed on the speech signal. For example, pre- emphasizing the speech signal at this stage prevents low frequency source information from being captured in the determination of MGC coefficients in the next operation. Control is passed to operation 315 and process 300 continues.
• operation 315 linear predictive coding, or LPC Analysis is performed on the pre-emphasized speech signal.
• LPC Analysis produces the coefficients which are used in the next operation to perform inverse filtering. Control is passed to operation 320 and process 300 continues.
• operation 320 inverse filtering is performed on the analyzed signal and on the original speech signal. In an embodiment, operation 320 is not performed until after pre-emphasis has been performed (operation 310). Control is passed to operation 330 and process 300 continues.
• the fundamental frequency value is determined from the original speech signal.
• the fundamental frequency value may be determined using any standard techniques known in the art.
• Control is passed to operation 330 and process 300 continues.
• operation 330 glottal cycles are segmented. Control is passed to operation 335 and process 300 continues.
• the glottal cycles are decomposed.
• the corresponding glottal cycles are decomposed into sub-band components.
• the sub- band components may comprise a plurality of bands, wherein the bands may comprise lower and higher components.
• Two components in the time domain may be obtained by placing zeros in the higher band region of the spectrum before taking the inverse FFT to obtain the time domain version of the low frequency component of the glottal pulse and vice versa to obtain the high frequency component. Control is passed to operation 340 and process 300 continues.
• the energies are determined for the sub-band components.
• the energies of each sub-band component may be determined to form the energy coefficients for each frame.
• the number of sub-band components may be two.
• the determination of the energies for the sub-band components may be made using any of the standard techniques known in the art.
• the energy coefficients of a frame is then referred to as the energy vector. Process 300 ends.
• two-band energy coefficients for each frame are determined from the inverse filtered speech.
• the energy coefficients may represent the dynamic nature of glottal excitation.
• the inverse filtered speech comprises an approximation to the source signal, after being segmented into glottal cycles.
• the two-band energy coefficients comprise energies of the low and high band components of the corresponding glottal cycle of the source signal.
• the energy of the lower frequency component comprises the energy coefficient of the lower band and similarly the energy of the higher frequency component comprises the energy coefficient of the higher band.
• the coefficients may be modeled by including them in the feature vector of corresponding frames, which are then modeled by HMM-GMM in HTS.
• the two-band energy coefficients, in this non-limiting example, of the source signal are appended to the spectral parameters determined in the process 200 to form the feature stream along with the fundamental frequency values and modeled using HMMs as in a typical HMM-GMM(HTS) based TTS system.
• the model may then be used in Process 500, as described below, for speech synthesis.
• FIG. 4 is a flowchart illustrating an embodiment of a process for identification of Eigen pulses, indicated generally at 400.
• the Eigen pulses may be identified for each sub-band glottal pulse database and used in synthesis as further described below.
• a glottal pulse database is created.
• a database of glottal pulses is automatically created using training data (speech data) obtained from a voice talent.
• speech data speech data
• linear prediction analysis is performed.
• the signal s(n) undergoes inverse filtering to obtain the integrated linear prediction residual signal which is an approximation to glottal excitation.
• the integrated linear prediction residual is then segmented into glottal cycles using a technique such as zero frequency filtering, for example.
• the glottal pulses are pooled to create the database.
• pulses from the database are decomposed into sub-band components.
• the glottal pulses may be decomposed into a plurality of sub-band components, such as low and high band components, and the two band energy coefficients.
• sub-band components such as low and high band components
• the demarcation between the bands varies from pulse to pulse as does the energy ratio between these two bands. As a result, different models for both of these bands may be needed.
• the cut off frequency is determined.
• the cut off frequency is that which separates the higher and lower bands by using a Zero Frequency Resonator (ZRF) method with suitable window size, but applied on the spectral magnitude.
• ZRF Zero Frequency Resonator
• a zero crossing at the edge of the low frequency bulge results, which is taken as the demarcation frequency between lower and higher bands.
• Two components in the time domain result from placing zeros in the higher band region of the spectrum before taking the inverse FFT to obtain the time domain version of the lower frequency component of glottal pulse and vice versa to obtain the higher frequency component.
• Control is passed to operation 415 and process 400 continues.
• the pulse databases are formed.
• a plurality of glottal pulse databases such as a low band glottal pulse database and a high band glottal pulse database, for example, result from operation 410.
• the number of data bases formed correspond to the number of bands formed. Control is passed to operation 420 and process 400 continues.
• sub-band glottal pulse refers, in this context, to a component of glottal pulse, either high or low band.
• the space of sub-band glottal pulse signals may be treated as a novel mathematical metric space as follows:
• a distance metric, d may be defined over the function space M.
• R r) Jr(r) 2 + r h ( ) 2 where r h is the Hilbert transform of r.
• the metric d forms a metric space (M,d).
• metric d is a Hilbertian metric
• the space can be isometrically embedded into a Hilbert space.
• x 6 M for a given signal in a function space, may be mapped to a vector ⁇ ⁇ (. ) in a Hilbert space, denoted as:
• mapping ⁇ ⁇ ⁇ x E M represents the total in the Hilbert space.
• the mapping is isometric, meaning
• ⁇ ⁇ — ⁇ ⁇ ⁇ ⁇ d(x, y).
• the vector representation ⁇ ⁇ . ) for a given signal x of the metric space depends on the set of distances of x from every other signal in the metric space. It is impractical to determine distances from all other points of the metric space, thus, the vector representation may depend only on the distances from a set of fixed number of points ⁇ cj of the metric space which are obtained as centroids after a metric based clustering of a large set of signals from the metric space. Control is passed to operation 425 and process 400 continues.
• Eigen pulses are determined and the process 400 ends.
• a metric or notion of distance, d(x,y) between any two sub-band glottal pulses x and y is defined.
• the metric between two pulses f,g is defined as follows.
• the normalized circular cross correlation between f,g is defined as:
• the period for circular correlation is taken to be the highest of the lengths of f,g.
• the shorter signal is zero extended for the purpose of computing the metric and not modified in the database.
• the Discrete Hilbert transform R h (n) of R(n) is determined.
• H(n) (ff(n)) 2 + (3 ⁇ 40 ) 2
• the cosine of the angle ⁇ between two signals/,g may be defined as:
• sup n H(n) refers to the maximum value among all the samples of the signal H(n).
• the distance metric may be given as:
• the k-means clustering algorithm which is well known in the art, may be modified to determine k cluster centroid glottal pulses from the entire glottal pulse data base G.
• the first modification comprises replacing the Euclidean distance metric with the metric d(x,y), defined for glottal pulses as previously described.
• the second modification comprises updating the centroids of the clusters.
• the centroid glottal pulse of a cluster of glottal pulses whose elements are denoted as [g , g 2 , ... S'w) to be that element g c such that:
• the clustering iterations are terminated when there is no shift in any of the centroids of the k clusters.
• Vector representation for sub-band glottal pulses may then be determined. Given a glottal pulse Xj, and assuming c x , c 2 , . . c t , c 256 are the centroid glottal pulses determined by clustering as described in previously, let the size of the glottal pulse database be L. Assigning each one to one of the centroid clusters q based on distance metric, the total number of elements assigned to centroid Cj may be defined as n ; . Where x 0 represents a fixed sub-band glottal pulse picked from the database, the vector representation may be defined as:
• j (xi) ⁇ d 2 ( Xi , Cj ) - d 2 ⁇ x il Cj ) - d 2 ( Cj -, x 0 ) ⁇ ⁇
• V t is the vector representation for the sub-band glottal pulse x it V t may be given as:
• V t [W 1 (x i ), W 2 (x i ), W 3 (x i ), ... W j (x i ), ... W 256 (x i )]
• PCA Principal component analysis
• the mean vector of the entire vector database is subtracted from each vector to obtain mean subtracted vectors.
• the Eigen vectors of the covariance matrix of the collection of vectors are then determined.
• a glottal pulse whose mean subtracted vector has minimum Euclidean distance from the Eigen vector is associated and called the corresponding Eigen glottal pulse.
• Eigen pulses for each sub-band glottal pulse database are thus determined and one from each is selected based on listening tests and may be used in synthesis as further described blow.
• FIG. 5 is a flowchart illustrating an embodiment of a process for speech synthesis, indicated generally at 500.
• This process may be used to train the model obtained in the process 100 ( Figure 1).
• the glottal pulse used as excitation in a particular pitch cycle is formed by combining the lower band glottal template pulse and the higher band glottal template pulse after scaling each one to the corresponding two-band energy coefficient.
• the two-band energy coefficients for a particular cycle are taken to be that of the frame the pitch cycle corresponds to.
• the excitation is formed from the glottal pulse and filtered to obtain output speech.
• Synthesis may occur in the frequency domain and in the time domain.
• the corresponding spectral parameter vector is converted into a spectrum and multiplied with the spectrum of the glottal pulse.
• the result undergoes inverse Discrete Fourier Transform (DFT) to obtain a speech segment corresponding to that pitch cycle.
• DFT inverse Discrete Fourier Transform
• Overlap add is applied to all obtained pitch synchronous speech segments in the time domain to obtain the synthesized speech.
• the excitation signal is constructed and filtered using a Mel Log Spectrum Approximation (MLSA) filter to obtain the synthesized speech signal.
• the given glottal pulse is normalized to unit energy. For unvoiced regions, white noise of fixed energy is placed in the excitation signal. For voiced regions, the excitation signal is initialized with zeros. Fundamental frequency values, such as those given for every 5 ms frame, are used to compute the pitch boundaries. The glottal pulse is placed starting from every pitch boundary and overlap added onto the zero initialized excitation signal in order to obtain the signal.
• Overlap add is performed on the glottal pulse at each pitch boundary and a small fixed amount of band pass filtered white noise is added to ensure that there is a small amount of random/stochastic component present in the excitation signal.
• a stitching mechanism is applied where a number of excitation signals are formed with using right-shifted pitch boundaries and circularly left-shifted glottal pulses.
• the right-shift in pitch boundary used for constructing comprises a fixed constant and the glottal pulse used for it is circularly left shifted by the same amount.
• the final stitched excitation is the arithmetic average of the excitation signals. This is passed through the MLSA filter to obtain the speech signal.
• operation 505 text is input into the model in the speech synthesis system.
• the model which was obtained in Figure 1 context dependent HMMs 120
• receives input text and provides features which are subsequently used to synthesize speech pertaining to the input text as described below.
• Control is passed to operation 510 and operation 515 and the process 500 continues.
• the feature vector is predicted for each frame. This may be done using methods which are standard in the art, such as context dependent decision trees, for example. Control is passed to operations 525 and 540 and operation 500 continues.
• MGC are determined for each frame. For example, the 0-39 MGC are determined. Control is passed to operation 530 and process 500 continues.
• operation 550 FFT is applied. Control is passed to operation 535 and process 500 continues. [0081] In operation 535, data multiplication may be performed. For example, the data from operation
• control is passed to operation 555 and process 500 continues.

Landscapes

• Engineering & Computer Science (AREA)
• Computational Linguistics (AREA)
• Health & Medical Sciences (AREA)
• Audiology, Speech & Language Pathology (AREA)
• Human Computer Interaction (AREA)
• Physics & Mathematics (AREA)
• Acoustics & Sound (AREA)
• Multimedia (AREA)
• Signal Processing (AREA)
• Compression, Expansion, Code Conversion, And Decoders (AREA)
• Machine Translation (AREA)
• Information Retrieval, Db Structures And Fs Structures Therefor (AREA)

Applications Claiming Priority (1)

Publications (2)

Family

ID=58488102

Family Applications (1)

Country Status (6)

Families Citing this family (5)

Family Cites Families (14)

• 2015
  • 2015-10-06 EP EP15905930.2A patent/EP3363015A4/de not_active Ceased
  • 2015-10-06 CN CN201580085103.5A patent/CN108369803B/zh active Active
  • 2015-10-06 AU AU2015411306A patent/AU2015411306A1/en not_active Abandoned
  • 2015-10-06 KR KR1020187012944A patent/KR20180078252A/ko not_active Application Discontinuation
  • 2015-10-06 CA CA3004700A patent/CA3004700C/en active Active
  • 2015-10-06 WO PCT/US2015/054122 patent/WO2017061985A1/en active Application Filing

Also Published As

Similar Documents

Legal Events