JP2018013721A - Voice synthesis parameter generating device and computer program for the same - Google Patents

Abstract

PROBLEM TO BE SOLVED: To provide a voice synthesis parameter generating device capable of achieving easy control and modification, and generating highly-accurate voice synthesis parameters.SOLUTION: A voice synthesis parameter generating device 270 includes a memory which stores a program, a HMM acoustic model 136, a DNN acoustic model 280, and a processor which executes the program and outputs a voice parameter series per frame. The program executed by the processor includes the steps of: outputting the HMM parameter series corresponding to a label string 132 by the HMM acoustic model 136; outputting the DNN parameter series corresponding to the label string by the DNN acoustic model 280 per frame; integrating the HMM parameter series and the DNN parameter series by a Product-of-Experts framework per frame and outputting integrated model parameters (step 284); and generating and outputting voice synthesis parameters on the basis of the integrated model parameters output by step 284 (step 290).SELECTED DRAWING: Figure 5

Description

  The present invention relates to a statistical parametric speech synthesis (Statistical Parametric Speech Synthesis SPSS), and more particularly to a statistical parametric speech synthesis capable of improving the quality of synthesized speech over the prior art.

  SPSS is a framework for estimating parameters for generating a speech signal based on a statistical model and synthesizing speech from the estimated parameters. SPSS has various advantages such as easy implementation of flexible speech synthesis processing such as speech style control. On the other hand, in speech synthesis using SPSS, the quality of synthesized speech tends to be deteriorated for various reasons as compared to real voice. Therefore, improvement of synthesized speech is an important research topic in SPSS.

  There are two main methods for SPSS. One is a method based on a Hidden Markov Model (HMM), and the other is a method based on a deep neural network (DNN). Hereinafter, these two conventional methods will be described.

  HMM considers utterances to arise from state transitions corresponding to phonemes. State and state transition cannot be observed, but it is possible to estimate the state and its transition, that is, phoneme sequence by observing and statistically processing the information sequence obtained from the utterances output with the state transition become. As information obtained from the utterance, a voice signal is divided into frames of a predetermined shift amount with a predetermined time length (can be overlapped), and a feature amount obtained from the voice signal of each frame is used. A plurality of values are used as the feature quantity to form a feature quantity vector. In the field of speech synthesis, mel cepstrum coefficients (Mel-Cepstrum coefficients) and power, their delta (difference between adjacent frames), delta delta (difference between adjacent deltas) and the like are generally used as feature quantities.

  Here, the three-state HMM 60 shown in FIG. 1 will be described. In the following description, the same parts are denoted by the same reference numerals, and details thereof will not be repeated.

  Referring to FIG. 1, HMM 60 includes a state S12 corresponding to the phoneme represented by HMM 60, a state S11 corresponding to the phoneme located in front of the phoneme, and a state S13 corresponding to the phoneme located behind. For each phoneme that appears in the utterance, a model such as this HMM 60 is prepared according to the combination with the phonemes before and after the phoneme. Therefore, different HMMs are used when the phonemes before and after the same phoneme are different (in order to save the storage area, different combinations may be represented by one HMM). The number of states is not limited to 3, and other numbers of states may be used.

  Referring to FIG. 1, transition probability a12 is assigned to transition 80 from state S11 to state S12. Transition probability a23 is assigned to transition 82 from state S12 to state S13. A transition probability a3e is assigned to the transition 84 from the state S13 to the end of the HMM 60. State S11, state S12, and state S13 have transitions 70, 72, and 74 to themselves, respectively, and are assigned transition probabilities a11, a22, and a33, respectively. These transition probabilities are calculated in advance using learning data.

  Referring to FIG. 2, a conventional speech synthesizer 100 using HMM, when input text 102 is given, parameter generation unit 110 that outputs a parameter for generating speech according to the input text, and parameter generation And a speech synthesis unit 112 that synthesizes and outputs the speech signal 104 using the parameters generated by the unit 110.

  The parameter generation unit 110 performs a morphological analysis, a syntax analysis, and the like on the input text 102, and outputs a label sequence 132 including labels representing phonemes to be uttered and context information thereof, and a label A duration model 134 that includes a decision tree depending on the sequence 132 and outputs the duration of each phoneme corresponding to the label sequence 132; and a probability of a speech synthesis parameter for each state of the HMM based on the decision tree that depends on the label sequence 132 The HMM acoustic model 136 that outputs the density function, the HMM state sequence determination unit 138 that outputs the HMM state sequence based on the decision tree of the duration model 134 with the label sequence 132 as input, and the HMM state sequence determination unit 138 output Based on the probability density function of each state estimated from the HMM state sequence to be estimated using the HMM acoustic model 136, the F0 parameter 142, voiced / And a voice synthesis parameter calculating unit 140 outputs the parameters 144 and the spectral envelope parameter 146 for articulation. The spectrum envelope parameter 146 may be, for example, a cepstrum coefficient, a mel cepstrum coefficient, a linear prediction coefficient, or the like.

  The speech synthesizer 112 receives the F0 parameter 142 and the voiced / unvoiced parameter 144 from the speech synthesis parameter calculator 140 and generates a sound source signal, and the sound source signal generated by the sound source signal generator 150 has a spectrum. And a speech synthesis filter 152 that outputs the speech signal 104 by modulating based on the envelope parameter 146.

  At the time of speech synthesis, the HMM state sequence determination unit 138 determines the continuation length of each state of the HMM state sequence by searching the decision tree of the continuation length model 134 using the label sequence 132 as input, and adds the time information. The obtained label sequence is given to the speech synthesis parameter calculation unit 140. The speech synthesis parameter calculation unit 140 estimates the maximum likelihood speech synthesis parameter sequence for each utterance frame using the probability density function of the output corresponding to the frame, and uses the F0 parameter 142, the voiced / unvoiced parameter. 144 and the spectral envelope parameter 146 are generated and provided to the speech synthesizer 112.

  The sound source signal generation unit 150 of the speech synthesis unit 112 generates a sound source signal according to the F0 parameter 142 and the voiced / unvoiced parameter 144 from the speech synthesis parameter calculation unit 140. The voice synthesis filter 152 modulates the sound source signal with characteristics determined by the spectrum envelope parameter 146 and outputs the voice signal 104.

  As an advantage of HMM speech synthesis, many years of knowledge are accumulated, and control and operation techniques for voice quality and speech style are established. Therefore, even when abnormal noise occurs in the generated parameters, it is easy to identify and correct the problem. On the other hand, there is a problem that the quality of the synthesized speech is deteriorated due to hard clustering by a decision tree at the time of state unit modeling and HMM state sequence determination.

  On the other hand, DNN speech synthesis uses a DNN such as DNN 170 schematically shown in FIG. 3 for parameter generation. The DNN 170 shown in FIG. 3 receives a vector generated based on the label string 132 and outputs a speech parameter corresponding to the label string 132. The network weight and bias are calculated in advance using learning data.

  The DNN 170 includes an input layer 172 having a vector obtained by converting the label sequence 132 into a binary expression or a numerical value as a node, an output layer 178 having a node made up of audio parameters, and an input layer 172 and an output layer 178 in order. One or more hidden layers 174 and 176 provided. In FIG. 3, in order to simplify the drawing, the number of nodes included in each layer is the same, but the number of nodes included in the hidden layer is not limited thereto. Moreover, although there are two hidden layers in FIG. 3, there may be one or three or more. The DNN 170 shown in FIG. 3 has a form in which the input sequentially propagates from the input layer 172 to the output layer 178, but has a path for returning a part of the output of the hidden layer in the middle to the input. Other types of NN such as type NN may be used.

  Referring to FIG. 4, a conventional speech synthesizer 200 using DNN receives input text 102 and outputs parameter F0 242, voiced / unvoiced parameter 244, and spectral envelope parameter 246. 2, which receives the F0 parameter 242, the voiced / unvoiced parameter 244 and the spectrum envelope parameter 246 output from the parameter generation unit 210 and outputs the speech signal 204.

  The parameter generation unit 210 receives the input text 102 and outputs a label sequence 132, and the DNN receives the label sequence 132 as an input and outputs an average vector of the probability density function of speech parameters for each frame. DNN acoustic model 230, a normalization parameter storage unit 234 that stores a global average vector of speech parameters and a covariance matrix, which are calculated based on learning data in advance, an average vector given from DNN acoustic model 230, After calculating speech synthesis parameters that have been denormalized based on the global average vector and the covariance matrix read from the generalization parameter storage unit 234, the F0 parameter 242 that is the speech synthesis parameter that has the highest likelihood, voiced / Silent parameter 244 and spectral envelope parameters Outputs 246 series and a voice synthesis parameter calculating unit 232 to be supplied to the speech synthesis unit 112. The F0 parameter 242 and the voiced / unvoiced parameter 244 are provided to the sound source signal generation unit 150, and the spectrum envelope parameter 246 is provided to the speech synthesis filter 152, respectively. The global average vector and the covariance matrix stored in the normalization parameter storage unit 234 are calculated from learning data during DNN learning, and are used in common in all frames.

  The label sequence 132 output from the text analysis processing unit 130 is given to the input of the DNN acoustic model 230 for each frame. In response to this input, DNN acoustic model 230 outputs an average vector of output probability density functions for each frame. This vector is given to the speech synthesis parameter calculation unit 232. The speech synthesis parameter calculation unit 232 reads the normalization parameter from the normalization parameter storage unit 234, and according to the probability density function obtained in combination with the average vector from the DNN acoustic model 230, the F0 parameter 242, the voiced / unvoiced parameter 244, and A spectral envelope parameter 246 is generated and output.

  The sound source signal generation unit 150 and the speech synthesis filter 152 of the speech synthesis unit 112 synthesize and output the speech signal 204 in the same manner as shown in FIG.

  DNN speech synthesis can be modeled in units of frames based on DNN, and in addition, higher quality speech than HMM speech synthesis can be generated. However, the control and operation technology for DNN, which is the core of the system, is still limited, and it is difficult to correct abnormal sounds and the lack of knowledge due to recent application to speech synthesis. Can be mentioned.

B. Chen, Z. Chen, J. Xu and K. Yu, “An investigation of context clustering for statistical speech synthesis with deep neural network,” in Proc. ICASSP, pp. 2212-2216, 2015. H. Zen, M. Gales, Y. Nankaku and K. Tokuda, “Product of experts for statistical parametric speech synthesis.” Audio, Speech, and Language Processing, IEEE Transactions on, 20 (3) pp. 794-805, 2012 .

  Various efforts have been made to improve the quality of SPSS. One of these is an attempt to integrate different models. There are two main methods for model integration. One is a method of connecting models in series, and the other is a method of connecting models in parallel.

  As a technique for connecting in series, a technique combining DNN, which is a typical technique of SPSS, and HMM speech synthesis has been proposed (Non-Patent Document 1). In this method, the HMM estimation result is used as the DNN input. However, since the models are connected in series, it is difficult to change the model flexibly or to integrate a plurality of models.

  On the other hand, the parallel connection method is excellent in flexibility and can easily integrate a plurality of models. An example of this technique is integration using a Product-of-Experts (PoE) framework (Non-Patent Document 2). In the method described in Non-Patent Document 2, two different types of HMMs are integrated according to PoE. However, since this method still uses HMM speech synthesis, the probability density function changes only on a state-by-state basis, so there is a suspicion that performance issues remain compared to DNN, which can be modeled precisely on a frame-by-frame basis. There is.

  In HMM speech synthesis, a probability density function is estimated for each state from a label string and output as a sequence. Therefore, in the obtained probability distribution series, the average vector and the covariance matrix change for each state. In DNN speech synthesis, an average vector of a probability density function is estimated with high accuracy in units of frames, but a covariance matrix uses a fixed value calculated in advance for all frames. In order to take advantage of the advantages of HMM and DNN speech synthesis, the present invention provides means for combining both in the PoE framework.

  The speech synthesis parameter generation device according to the first aspect of the present invention receives a label sequence composed of labels representing phonemes to be uttered and context information thereof, and converts speech parameter sequences indicating speech characteristics into speech synthesis time. Output for each frame, which is a target unit, and generate speech synthesis parameters for speech synthesis from speech parameters. This apparatus stores a program, a first acoustic model composed of an HMM, and a second acoustic model composed of a neural network (NN), and executes a program stored in the memory to execute speech. And a processor for outputting the parameter series for each frame. The processor refers to the continuation model that outputs the HMM state sequence corresponding to the label sequence by the program, the state sequence determination step that outputs the HMM state sequence corresponding to the label sequence, and the first acoustic model. Then, referring to the first model parameter generation step for outputting the HMM parameter series estimated from the HMM for each state with the state series as an input, and referring to the second acoustic model, the state series and the label string are As input, a second model parameter generation step for outputting an NN parameter sequence estimated from the NN for each frame, an HMM parameter sequence output in the first model parameter generation step, and a second model parameter generation step The output NN parameter series is integrated with the PoE framework for each frame to integrate model parameters. And model integration step of outputting as being programmed to perform a method comprising a speech synthesis parameter generating step for generating and outputting a voice synthesis parameter sequence based on the integrated model parameters outputted in the model integration step.

  Preferably, each of the speech parameters constituting the speech parameter series is a vector having a plurality of elements. Both the probability density function of the output of each state of the HMM and the second probability density function modeled by the NN include a Gaussian distribution having the same number of dimensions as the speech parameter vector.

  More preferably, each of the state parameters constituting the state series output in the first model parameter generation step includes a mean vector and a covariance matrix that define a Gaussian distribution that is a probability density function of an output for each state. Each of the NN parameters constituting the NN parameter series generated by the second model parameter generation step includes an average vector that defines a Gaussian distribution for each frame. The extracted model synthesis step includes a Gaussian distribution defined by the average vector and covariance matrix output in the first model parameter generation step, an average vector output in the second model parameter generation step, and a pre-calculated global A step of multiplying the Gaussian distribution defined by the mean vector and the fixed covariance matrix by the PoE framework for each frame and outputting as an integrated model parameter.

  Each of the HMM parameters constituting the state series output in the first model parameter generation step includes a Gaussian distribution mean vector and a covariance matrix defined as a probability density function for each state. The NN parameter generated by the second model parameter generation step includes an average vector of Gaussian distribution defined for each frame. The model integration step includes a Gaussian distribution defined by the average vector and covariance matrix output in the first model parameter generation step, an average vector output in the second model parameter generation step, and a fixed fixed value calculated in advance. A step of multiplying the Gaussian distribution defined by the global mean vector and the covariance matrix by weighting by the PoE framework for each frame and outputting as an integrated model parameter.

  Each of the HMM parameters constituting the state series output in the first model parameter generation step may include an average vector and a covariance matrix that define a Gaussian distribution for each state. The NN parameter generated by the second model parameter generation step may include an average vector that defines a Gaussian distribution for each frame. The second model parameter generation step refers to the second acoustic model, and for each frame, outputs a phoneme NN parameter corresponding to the state assigned to the frame by the first model parameter generation step. May be included.

  A computer program according to the second aspect of the present invention causes a computer to function as any of the speech synthesis parameter generation devices described above.

It is a schematic diagram which shows the conceptual structure of HMM. It is a block diagram of the speech synthesizer using the conventional HMM speech synthesis method. It is a schematic diagram which shows the conceptual structure of DNN. It is a block diagram of the speech synthesizer using the conventional DNN speech synthesis method. 1 is a block diagram of a speech synthesizer according to an embodiment of the present invention. It is a graph explaining the multiplication result of the probability density function by PoE. It is a schematic flowchart of the computer program which implement | achieves the process which produces | generates a speech synthesis parameter for every flame | frame in the speech synthesizer which concerns on embodiment of this invention. It is a block diagram of the model learning apparatus which performs learning of HMM and DNN used in the speech synthesizer concerning an embodiment of the invention. It is a graph which shows the change of the root mean square error of F0 parameter by the weight at the time of integrating HMM and DNN in the model used with the speech synthesizer concerning the present invention. It is a figure which shows the external appearance of the computer system for implement | achieving the speech synthesizer which concerns on each embodiment of this invention. It is a block diagram which shows the hardware constitutions of the computer which comprises the computer system shown in FIG.

  In the following description and drawings, the same parts are denoted by the same reference numerals. Therefore, detailed description thereof will not be repeated. Each of the embodiments described below employs a technique of combining DNN and HMM using a PoE framework. Although DNN is used in the present embodiment, NN is not limited to this, and recurrent NN or the like may be used.

<Configuration>
FIG. 5 shows a schematic block diagram of a speech synthesis apparatus including a speech synthesis parameter generation apparatus according to an embodiment of the present invention. Referring to FIG. 5, the speech synthesizer 260 receives the input text 102, estimates speech synthesis parameters for the input text 102 using the HMM and DNN, and sets F0 parameter 292, voiced / unvoiced parameter 294, and spectrum A parameter generation unit 270 that generates an envelope parameter 296, and an F0 parameter 292, a voiced / unvoiced parameter 294, and a spectrum envelope parameter 296 that are received from the parameter generation unit 270 and outputs an audio signal 262. And a speech synthesizer 112 having a configuration.

  The parameter generation unit 270 performs the same text analysis processing as before on the input text 102 and outputs a label sequence 132 including phoneme information and context information, and the label sequence 132 as an input. A DNN acoustic model 280 that outputs an average vector of a probability density function of speech parameters for each frame, a duration model 134 and an HMM state sequence determination unit 138 connected in the same manner as shown in FIG. HMM acoustic model 136.

  The parameter generation unit 270 further includes an average vector output from the DNN acoustic model 280 for each frame, a probability average function defined by a global average vector and a covariance matrix calculated in advance when learning the DNN acoustic model 280, According to the HMM sequence output from the HMM state sequence determination unit 138, the average vector output for each state from the HMM acoustic model 136 and the probability density function defined by the covariance matrix are weighted for each frame according to the PoE framework. The voice parameter integration unit 284 that integrates the voice parameters and outputs the voice parameters after the integration, and the weight used when the voice parameters are integrated by the voice parameter integration unit 284 are stored in the voice parameter integration unit 284. Supports output weight storage unit 288 and DNN acoustic model 280 A normalization parameter storage unit 286 that stores a fixed global average vector and a covariance matrix, and outputs to a speech parameter integration unit 284 in order to normalize a speech parameter output from the DNN acoustic model 280, and a speech parameter integration unit 284 A speech synthesis parameter calculation unit 290 that generates and outputs a speech synthesis parameter including an F0 parameter 292, a voiced / unvoiced parameter 294, and a spectrum envelope parameter 296 using the synthesized speech parameter output from. Note that the probability density function described above is a Gaussian distribution defined by an average vector and a covariance matrix in the present embodiment.

  In this embodiment, as described above, voice parameters obtained from the HMM and DNN are integrated according to the PoE framework. Hereinafter, this integration will be described. PoE combines multiple probability density functions into one probability density function by taking the product between them.

  As shown in FIG. 6, a probability density function that simultaneously satisfies HMM and DNN is obtained by multiplication. In FIG. 6, the horizontal axis represents a random variable, and the vertical axis represents a probability density.

  In the present embodiment, the probability density function predicted by DNN and the probability density function predicted by HMM are integrated as follows. Also in this embodiment, the global average vector and covariance matrix of the DNN probability density function are the same for all frames, which are obtained in advance from learning data.

  According to this equation, on the other hand, the average vector of the probability density function by PoE varies from frame to frame together with the variation of the average vector obtained from DNN. On the other hand, the covariance matrix of the probability density function varies from state to state along with the variation of the covariance matrix accompanying the state transition of the HMM. Therefore, this probability density function is defined by an average vector that changes following the movement of the frame and a covariance matrix that changes following the movement of the HMM state. Will be combined.

  In the present embodiment, weights for the combination of both are further introduced, and the PoE model is synthesized by the following equation.

  FIG. 7 is a flowchart showing a control structure of a computer program for realizing the voice parameter integration unit 284 shown in FIG. 5 by a computer. Referring to FIG. 7, the program includes a step 330 of executing a model synthesis step 332 described below for each frame of speech synthesis.

  The model synthesis step 332 reads an HMM parameter including an average vector of a state including the time of a frame to be processed and a covariance matrix among the HMM state sequences output by the HMM state sequence determination unit 138 in FIG. 5. A DNN parameter including a mean vector output by the DNN acoustic model 280 of FIG. 5 for a frame to be processed, and a fixed global mean vector and covariance matrix stored in the normalized parameter storage unit 286 342.

  The DNN acoustic model 280 and the HMM acoustic model 136 according to the above-described embodiment may be learned independently of each other, but it is more preferable if they are optimized for integration by PoE. Hereinafter, a DNN and HMM simultaneous learning method for optimizing DNN and HMM for speech synthesis apparatus 260 according to this embodiment will be described.

  FIG. 8 shows a flowchart of this DNN and HMM simultaneous learning method. Referring to FIG. 8, this learning method 360 is similar to normal DNN and HMM learning, and includes learning data 362 including acoustic feature amounts and a label string made up of phonemes and context information corresponding to each acoustic feature amount. And a step 380 for learning the initial HMM using the learning data 362 and a step 382 for learning the initial DNN. The HMM parameter set and DNN parameter set subjected to the initial learning in this way are optimized by maximizing the likelihood function given by the following equation by PoE in step 384.

Here, the posterior probability density function calculated in the E step of the EM algorithm is given by the following equation.

  That is, referring to FIG. 8, step 384 calculates an PoE model using a given HMM and DNN, and simultaneously estimates an a posteriori probability density function of a latent variable for the HMM and DNN using the model. 400, M step 402 for estimating maximum likelihood of DNN and HMM parameters separately using the posterior probability distribution estimated for each model in E step 400 as an observation vector, and the result of M step 402 until the termination condition is satisfied Using the obtained model parameter as a new model parameter, step 404 in which the E step 400 and M step 402 are repeated, and when it is determined in step 404 that the termination condition is satisfied, the HMM and DNN parameter sets at that time Are output as an HMM acoustic model 364 and a DNN acoustic model 366. As the termination condition, whether or not the likelihood function by PoE has converged, whether or not the parameters of HMM and DNN have converged, whether or not a predetermined number of iterations have been completed, and the like are used. Both HMM and DNN parameter sets are learned until they converge once at the pre-learning stage. Therefore, even when renewed in this process, the convergence is fast and often converges by repeating the above process once or twice.

  E step 400 includes a step 420 for calculating a PoE model based on the input HMM and DNN, and a step 422 for simultaneously estimating and outputting latent variables for the HMM and DNN using the PoE model calculated in step 420. Including.

  M step 402 performs maximum likelihood estimation and update of the HMM parameter set using the posterior probability distribution using the latent variable simultaneously estimated in step 422 as an observation vector, and similarly performs maximum likelihood estimation of the DNN parameter set. Updating 442.

  The steepest descent method is used to update the HMM parameter set. Stochastic gradient descent is used to update the DNN parameter set.

<Operation>
The apparatus shown in FIGS. 5 to 7 operates as follows. The optimization of HMM and DNN is as shown in FIG. The weight when optimized is stored in the weight storage unit 288. Further, a covariance matrix of the probability density function of the DNN output is calculated during DNN learning and stored in the normalization parameter storage unit 286.

  Referring to FIG. 5, the text analysis processing unit 130 analyzes the input text 102 and outputs a label string 132. Each label of the label column 132 includes phonemes constituting the utterance and context information.

  The DNN acoustic model 280 receives the label sequence 132, outputs an average vector for each utterance frame, and provides it to the speech parameter integration unit 284. The HMM state sequence determining unit 138 determines an HMM state sequence corresponding to the input text 102 by searching the decision tree according to the label and reading the duration from the duration model 134, and the probability density function of the output in each state Are provided to the speech parameter integration unit 284 as an HMM state sequence.

  The audio parameter integration unit 284 repeats the following process (step 332 in FIG. 7) for each frame. That is, first, HMM parameters (average vector and covariance matrix) in a state including the frame are read (step 340). Subsequently, the DNN parameter (average vector) for the frame and the global average vector and covariance matrix stored in the normalized parameter storage unit 286 are read (step 342). In step 343, it is determined whether both voiced / unvoiced parameters match. If the two match, in step 344, the average vector and covariance matrix of the probability density function of the model based on PoE are calculated using equations (7) and (8). Otherwise, in step 345, the mean vector and the covariance matrix of the probability density function of the model based on PoE are calculated by Equation (7) and Equation (8), respectively, excluding the voiced / unvoiced parameters. The DNN output is output as is. In the following step 346, the average vector and covariance matrix calculated in this way are output as a probability density function in the current frame.

  The speech synthesis based on the input text 102 is performed by performing the processing in step 332 for all the frames constituting the utterance.

  We investigated the quality of speech synthesis using the PoE model obtained by changing the weights of DNN and HMM to various values. The results are shown in FIG.

  Referring to FIG. 9, the left end is when the HMM weight is 1 and the DNN weight is 0, and the right end is when the HMM weight is 0 and the DNN weight is 1. From the graph of FIG. 9, the system based on PoE clearly outperforms the performance of the single DNN and HMM system when the DNN weight is in the range of about 0.75 to 0.98. It can be seen that the performance is particularly high when the DNN weight is in the range of ± 0.05 around 0.9.

  In the above embodiment, as shown in FIG. 7, the parameters from the DNN and the parameters from the HMM corresponding to the frame are read for each frame. However, the present invention is not limited to such an embodiment. Since the HMM parameter only changes for each state, the HMM parameter may be read only when the state changes, and may not be read in the processing of each frame.

  In the above embodiment, two models DNN and HMM are synthesized by the PoE framework. However, the present invention is not limited to such an embodiment. Even if there are three or more models, the present invention can be applied to the case where the shortcomings of one model can be compensated for by other models, that is, the acoustic models are created based on different ideas. Can be applied.

[Realization by computer]
The speech synthesis apparatus 260 including the speech synthesis parameter generation apparatus according to the embodiment of the present invention can be realized by computer hardware and a computer program executed on the computer hardware. FIG. 10 shows the external appearance of this computer system 530, and FIG. 11 shows the internal configuration of the computer system 530.

  Referring to FIG. 10, the computer system 530 includes a computer 540 having a memory port 552 and a DVD (Digital Versatile Disc) drive 550, a keyboard 546, a mouse 548, and a monitor 542.

  11, in addition to the memory port 552 and the DVD drive 550, the computer 540 includes a CPU (Central Processing Unit) 556, a bus 566 connected to the CPU 556, the memory port 552, and the DVD drive 550, and a boot program. And the like, a read only memory (ROM) 558 for storing etc., a random access memory (RAM) 560 connected to the bus 566 for storing program instructions, system programs, work data and the like, and a hard disk 554. Computer system 530 further includes a network interface (I / F) 544 that provides a connection to a network 568 that allows communication with other terminals.

  A computer program for causing the computer system 530 to function as each functional unit of the speech synthesis apparatus 260 according to the above-described embodiment is stored in the DVD drive 550 or the DVD 562 or the removable memory 564 installed in the memory port 552, and further the hard disk 554. Alternatively, the program may be transmitted to the computer 540 through the network 568 and stored in the hard disk 554. The program is loaded into the RAM 560 when executed. The program may be loaded directly into the RAM 560 from the DVD 562, from the removable memory 564, or via the network 568.

  This program includes an instruction sequence including a plurality of instructions for causing the computer 540 to function as each functional unit of the speech synthesizer 260 according to the above embodiment. Some of the basic functions necessary to cause the computer 540 to perform this operation are an operating system or third party program running on the computer 540 or various dynamically linkable programming toolkits or programs installed on the computer 540. Provided by the library. Therefore, this program itself does not necessarily include all the functions necessary for realizing the system, apparatus, and method of this embodiment. The program is a system as described above by dynamically calling an appropriate program in an appropriate function or programming toolkit or program library in a controlled manner to obtain a desired result among instructions, It is only necessary to include an instruction for realizing a function as an apparatus or a method. Of course, all necessary functions may be provided only by the program.

<Operation and effect of embodiment>
In the present invention, HMM speech synthesis and DNN speech synthesis models are integrated using a Product-of-Experts framework that can satisfy both constraints. That is, while satisfying the restrictions of HMM and DNN, it is possible to use DNN with high accuracy for the average vector and HMM for dispersion. It is possible to generate a speech synthesis parameter considering the variance of HMM speech synthesis while estimating the average vector with the accuracy of DNN speech synthesis, and to generate synthesized speech with higher quality. In other words, when one of the parameters generated by a plurality of different types of models for speech synthesis such as DNN and HMM speech synthesis causes quality degradation, the other parameter can be supplemented. In the same way, it is possible to integrate multiple models. If the models to be integrated are models that can combine the advantages of different types of models, the parameters obtained from them can be integrated to improve the quality of the speech synthesis parameters. Also, as in the above embodiment, when optimizing the model to be integrated, by introducing latent variables into the PoE framework, conditional independence is created for the original model, and the EM algorithm is used. Enable learning. The EM algorithm guarantees convergence to the most likely solution. Therefore, it is possible to stably integrate speech parameters obtained from a plurality of models. Since a speech parameter sequence that satisfies the constraints of a plurality of models can be generated, quality degradation of the synthesized speech can be prevented and a synthesized speech with high quality can be generated.

  The embodiment disclosed herein is merely an example, and the present invention is not limited to the above-described embodiment. The scope of the present invention is indicated by each claim of the claims after taking into account the description of the detailed description of the invention, and all modifications within the meaning and scope equivalent to the wording described therein are included. Including.

S11, S12, S13 State 60 HMM
100, 200, 260 Speech synthesizer 102 Input text 104, 204, 262 Speech signal 110, 210, 270 Parameter generator 112 Speech synthesizer 130 Text analysis processor 132 Label sequence 134 Duration model 136, 364 HMM acoustic model 138 HMM State series determination unit 140, 232, 290 Speech synthesis parameter calculation unit 142, 242, 292 F0 parameters 144, 244, 294 Voiced / unvoiced parameters 146, 246, 296 Spectrum envelope parameter 150 Sound source signal generation unit 152 Speech synthesis filter 170 DNN
172 Input layer 174, 176 Hidden layer 178 Output layer 230, 280, 366 DNN acoustic model 234, 286 Normalization parameter storage unit 284 Speech parameter integration unit 288 Weight storage unit 332 Model synthesis step 362 Learning data 400 E step 402 M step

Claims (5)

Receives a label string consisting of a phoneme to be uttered and a label representing its context information, and outputs a speech parameter series indicating the features of the speech for each frame, which is a time unit of speech synthesis, for speech synthesis from the speech parameters. A speech synthesis parameter generation device for generating a speech synthesis parameter of
A memory for storing a program, a first acoustic model comprising a hidden Markov model (HMM), and a second acoustic model comprising a neural network (NN);
A processor that outputs the voice parameter series for each frame by executing the program stored in the memory;
The processor is executed by the program.
With reference to a continuation length model that outputs an HMM state sequence corresponding to the label sequence, a state sequence determination step for determining and outputting an HMM state sequence corresponding to the label sequence;
A first model parameter generation step of referring to the first acoustic model and outputting the HMM parameter sequence estimated from the HMM for each state, using the state sequence as an input;
A second model parameter generation step of referring to the second acoustic model and outputting the NN parameter sequence estimated from the NN for each frame with the state sequence and the label sequence as inputs;
The HMM parameter series output in the first model parameter generation step and the NN parameter series output in the second model parameter generation step are converted into a product-of-experts (PoE) framework for each frame. A model integration step that integrates and outputs as an integrated model parameter,
A speech synthesis parameter generation device programmed to execute a method including a speech synthesis parameter generation step of generating and outputting the speech synthesis parameter series based on the integrated model parameter output in the model integration step. Each of the speech parameters constituting the speech parameter series is a vector having a plurality of elements,
The probability density function of the output of each state of the HMM and the second probability density function modeled by the NN both include a Gaussian distribution having the same number of dimensions as the vector of the speech parameters. Speech synthesis parameter generator. Each of the HMM parameter series output in the first model parameter generation step includes a mean vector and a covariance matrix that define a Gaussian distribution that is a probability density function of an output for each state,
Each of the NN parameters constituting the NN parameter series generated by the second model parameter generation step includes an average vector defining a Gaussian distribution for each frame,
The model integration step includes the Gaussian distribution defined by the average vector and the covariance matrix output in the first model parameter generation step, the average vector output in the second model parameter generation step, and The speech synthesis parameter generation device according to claim 2, comprising a step of multiplying a Gaussian distribution defined by a global average and a covariance matrix determined in advance by a PoE framework for each frame and outputting the result as the integrated model parameter. . Each of the HMM parameters output in the first model parameter generation step includes a mean vector and a covariance matrix of a Gaussian distribution defined as a probability density function for each state,
The NN parameter generated by the second model parameter generation step includes a mean vector of Gaussian distribution defined for each frame,
The model integration step includes the Gaussian distribution defined by the average vector and the covariance matrix output in the first model parameter generation step, the average vector output in the second model parameter generation step, and The speech synthesis parameter according to claim 2, comprising a step of adding a Gaussian distribution defined by a fixed covariance matrix calculated in advance by weighting by a PoE framework for each frame and outputting the result as the integrated model parameter. Generator. A computer program for causing a computer to function as the speech synthesis parameter generation device according to any one of claims 1 to 4.