JP6649210B2 - Speech synthesis learning device, method, and program - Google Patents

Description

  The present invention relates to a speech synthesis learning device, a method, and a program, and particularly to a speech synthesis learning device, a method, and a program for synthesizing speech.

  Features representing vocal vocal sound source information (fundamental frequency, aperiodicity index, etc.) and vocal tract spectrum information can be obtained by speech analysis methods such as STRAIGHT and Mel-Generalized Cepstral Analysis (MGC). Can be. In many text-to-speech synthesis systems and speech conversion systems, an approach is used in which such a sequence of speech features is predicted from an input text or a conversion source speech, and a speech signal is generated according to a vocoder method.

  In the existing vocoder-based speech synthesis, speech is generated by converting a speech feature amount sequence such as vocal cord source information or vocal tract spectrum information using a vocoder. FIG. 35 shows a conceptual diagram of a vocoder-type speech synthesis process. The vocoder described here is a model of a sound generation process based on knowledge about the mechanism of human vocalization. For example, a typical model of a vocoder is a source filter model. In this model, a sound generation process is described by using a sound source (source) and a digital filter. Specifically, it is stated that voice is generated by applying a digital filter to a voice signal (represented by a pulse signal) generated from the source as needed. As described above, in the vocoder-based speech synthesis, the utterance mechanism is abstractly modeled and expressed, so that the speech can be expressed in a compact (low-dimensional) manner. On the other hand, as a result of the abstraction, the naturalness of the voice is lost, often resulting in mechanical sound quality unique to the vocoder.

Ian J. Goodfellow, Jean Pouget-Abadie, Mehdi Mirza, Bing Xu, DavidWarde-Farley, Sherjil Ozairy, Aaron Courville, Yoshua Bengio, "Generative Adversarial Nets," 2014. Emily Denton, Soumith Chintala, Arthur Szlam, Rob Fergus, "Deep Generative Image Models using a Laplacian Pyramid of Adversarial Networks," 2015.

  The problem of predicting appropriate speech features from input text and source speech is a kind of regression (machine learning) problem, especially when only a limited number of training samples are available, compact (low-dimensional) features. Representation is advantageous in statistical prediction. A vocoder method using a speech feature amount (rather than trying to directly predict a waveform or spectrum) is used in many text-to-speech synthesis systems and speech conversion systems in order to take advantage of this advantage. On the other hand, speech generated by the vocoder method often has mechanical sound quality unique to the vocoder, which imposes a potential limit on sound quality in conventional text-to-speech synthesis systems and speech conversion systems.

  The present invention has been made to solve the above problems, and has as its object to provide a speech synthesis learning apparatus, method, and program capable of learning a neural network capable of synthesizing a more natural speech. I do.

  In order to achieve the above object, a speech synthesis learning device according to a first invention is a speech synthesis learning device for learning a neural network for synthesizing speech from arbitrary speech data or a speech feature amount sequence. A neural network as a generator that receives speech data or a speech feature amount sequence and true learning speech data as input and generates synthesized speech data from the speech data or the speech feature amount sequence; A neural network is provided as a discriminator for discriminating whether or not the speech data follows the same distribution as the true speech data.The neural network as the generator and the neural network as the discriminator compete with each other. It is configured to include a learning unit that performs learning according to optimization conditions.

  Further, in the speech synthesis learning device according to the first invention, the learning unit includes a speech feature amount sequence used for a vocoder that synthesizes speech from a speech feature amount sequence obtained by speech analysis of speech data; And the neural network as the generator and the neural network as the discriminator, which generate synthesized speech data synthesized from the speech feature amount sequence, using the real speech data of The learning may be performed according to the following.

  Further, in the speech synthesis learning device according to the first invention, the learning unit inputs a speech signal or a speech spectrum sequence obtained by synthesizing a speech from a speech feature amount sequence and true speech data for learning. The neural network as the generator that generates synthesized voice data synthesized from the voice signal or the voice spectrum series, and the neural network as the discriminator perform learning according to mutually competing optimization conditions. You may.

  Further, in the speech synthesis learning device according to the first invention, the learning unit includes a speech feature amount sequence output from an Auto Encoder which is a pre-trained neural network with speech data as input, and a learning true speech. The neural network as the generator and the neural network as the discriminator that generate synthesized speech data synthesized from the speech feature amount sequence using data as input, perform learning according to the mutually conflicting optimization conditions. You may do so.

  A speech synthesis learning device according to a second invention is a speech synthesis learning device for learning a neural network for synthesizing speech from arbitrary speech data or a speech feature amount sequence. A real network for learning, and a neural network as a generator for generating synthesized voice data from the voice data or the voice feature amount sequence, wherein the neural network as the generator includes the synthesized voice data. And a learning unit that performs learning so as to optimize an objective function representing a distance from true learning speech data.

  Further, in the speech synthesis learning device according to the second invention, the learning unit includes a speech feature amount sequence obtained by analyzing speech data and used for a vocoder that synthesizes speech from the speech feature amount sequence. And the neural network as the generator that generates synthesized speech data synthesized from the speech feature amount sequence by inputting the true speech data of the above and performs learning so as to optimize the objective function. Good.

  Further, in the speech synthesis learning device according to the second invention, the learning unit inputs a speech signal or a speech spectrum sequence obtained by synthesizing a speech from a speech feature amount sequence and true speech data for learning. The neural network as the generator that generates synthesized voice data synthesized from the voice signal or the voice spectrum sequence may perform learning so as to optimize the objective function.

  Further, in the speech synthesis learning device according to the second invention, the learning unit may include a speech feature amount sequence output from an auto-encoder which is a neural network previously trained with speech data as input, and a learning true speech. A neural network as the generator that receives data as input and generates synthesized speech data synthesized from the speech feature amount sequence may perform learning so as to optimize the objective function.

  According to the speech synthesis learning apparatus, method, and program of the present invention, input speech data or a speech feature amount sequence and true speech data for learning are input, and synthesized speech is obtained from the speech data or the speech feature amount sequence. A neural network as a generator for generating data; and a neural network as a discriminator for discriminating whether or not the synthesized speech data follows the same distribution as the true speech data. By performing learning according to optimization conditions that compete with each other, the neural network as a device can learn a neural network that can synthesize more natural speech.

  According to the speech synthesis learning device, method and program of the present invention, the input speech data or speech feature amount sequence and the true speech data for learning are input, and the speech data or the speech feature amount sequence is used. It has a neural network as a generator that generates synthesized speech data, and the neural network as a generator performs learning so as to optimize the objective function that represents the distance between the synthesized speech data and the true speech data for learning. By doing so, an effect is obtained that a neural network capable of synthesizing a more natural voice can be learned.

FIG. 4 is a conceptual diagram of a process according to the first embodiment of this invention. FIG. 1 is a block diagram illustrating a configuration of a speech synthesis device according to a first embodiment of the present invention. FIG. 3 is a conceptual diagram of a learning process according to the first embodiment of this invention. 9 is a flowchart illustrating a learning processing routine in the speech synthesis device according to the first and second embodiments of the present invention. 9 is a flowchart illustrating a generation processing routine in the speech synthesis device according to the first and second embodiments of the present invention. FIG. 9 is a conceptual diagram of a process according to a second embodiment of the present invention. It is a block diagram showing the composition of the speech synthesis device concerning a 2nd embodiment of the present invention. It is a conceptual diagram of the learning process of the second embodiment of the present invention. FIG. 14 is a conceptual diagram of a process according to a third embodiment of the present invention. It is a block diagram showing the composition of the speech synthesis device concerning a 3rd embodiment of the present invention. It is a conceptual diagram of the learning process of the 3rd Embodiment of this invention. It is a flow chart which shows a learning processing routine in a speech synthesizer concerning a 3rd and a 4th embodiment of the present invention. It is a flow chart which shows a generation processing routine in a speech synthesizer concerning a 3rd and a 4th embodiment of the present invention. FIG. 14 is a conceptual diagram of a process according to a fourth embodiment of the present invention. It is a block diagram showing the composition of the speech synthesis device concerning a 4th embodiment of the present invention. It is a conceptual diagram of the learning process of the fourth embodiment of the present invention. It is a conceptual diagram of a 5th embodiment of the present invention. It is a block diagram showing the composition of the speech synthesis device concerning a 5th embodiment of the present invention. It is a flow chart which shows a learning processing routine in a speech synthesizer concerning a 5th embodiment of the present invention. It is a flow chart which shows a generation processing routine in a speech synthesizer concerning a 5th embodiment of the present invention. It is a figure showing an example of implementation of a learning method of a 3rd embodiment in an example of an experiment. It is a figure showing an example of implementation of a learning method of a 4th embodiment in an example of an experiment. It is a figure showing an example of implementation of a generation method of a 3rd embodiment in an example of an experiment. It is a figure showing an example of implementation of a generation method of a 4th embodiment in an example of an experiment. It is a figure showing the network structure of a 3rd embodiment in an example of an experiment. It is a figure showing the network structure of a 4th embodiment in an example of an experiment. FIG. 9 is a diagram illustrating an example of a waveform of an audio signal that is a source of input and output in an experimental example. It is a figure showing an experiment result of Volume change. It is a figure which shows the experimental result of Pre-emphasis. It is a figure showing an experimental result of LPC. It is a figure showing an experimental result of LPC + pulse. FIG. 3 is a diagram illustrating a network structure according to the first embodiment in an experimental example. It is a figure showing the network structure of a 2nd embodiment in an example of an experiment. FIG. 11 is a diagram illustrating a result of audio restoration by the methods of the first and second embodiments in an experimental example. FIG. 4 is a conceptual diagram of a vocoder-based speech synthesis process.

  Hereinafter, embodiments of the present invention will be described in detail with reference to the drawings.

<Overview according to First Embodiment of the Present Invention>

  First, an outline of the first embodiment of the present invention will be described.

  Existing vocoder-based speech synthesis abstractly models the sound generation process based on knowledge of the human vocalization mechanism, and converts speech data from speech feature sequences to speech data (speech signals or speech spectrum sequences. Is not directly optimized for reproducing the same.

  In the first embodiment of the present invention, this problem is solved by directly optimizing the mapping between the audio feature sequence and the audio data. FIG. 1 shows a conceptual diagram of the processing. By applying a neural network optimized for mapping of the audio feature amount sequence and the audio data to the input audio feature amount sequence, target audio data can be obtained. At this time, when an audio signal is used as the audio data, the target audio signal can be obtained as it is. On the other hand, when a voice spectrum sequence is used as voice data, the output is also a voice spectrum sequence. In that case, the target audio signal can be obtained by performing the phase restoration. As a method of the phase restoration, for example, there is Griffin Lim.

<Configuration of the speech synthesis device according to the first embodiment of the present invention>

  Next, the configuration of the speech synthesizer according to the first embodiment of the present invention will be described. As shown in FIG. 2, the speech synthesizer 100 according to the first embodiment of the present invention stores a CPU, a RAM, a program for executing a learning process routine and a generation process routine described below, and various data. And a computer including the ROM. The speech synthesizer 100 functionally includes an input unit 10, an arithmetic unit 20, and an output unit 90, as shown in FIG.

  The input unit 10 receives human voice data x as learning data. Further, the input unit 10 receives an arbitrary voice feature amount sequence f for which synthetic voice data is to be generated.

  The calculation unit 20 includes a learning unit 30, a neural network storage unit 40, and a generation unit 50.

As described below, the learning unit 30 receives a voice feature amount sequence f used for a vocoder and a true voice data x for learning obtained by voice analysis of the voice data x, and Synthesized speech data synthesized from sequence f

Is provided with a neural network as a generator for generating the synthesized speech data.

Learning is performed so as to optimize an objective function representing a distance from the real speech data x for learning.

The learning unit 30 first obtains a voice feature amount sequence f by performing voice analysis on the voice data x received by the input unit 10. The neural network is learned so that the original true voice data x is generated for the voice feature amount sequence f obtained here. Specifically, when the speech feature amount sequence f is input to the neural network, the synthesized speech data

Is output, but the synthesized voice data is output as true voice data x.

The weight of the neural network may be optimized so that the distance is minimized for a certain distance index. The distance index described here is, for example, a least square error. For the minimum square error as a distance indicator, the objective function L ₂ is expressed by the following equation (1).

... (1)

  FIG. 3 is a conceptual diagram of the learning process according to the first embodiment.

  The neural network as a generator that has been learned so as to optimize the objective function of the above equation (1) is stored in the neural network storage unit 40.

The generation unit 50 inputs an arbitrary speech feature amount sequence f received by the input unit 10 to a neural network stored in the neural network storage unit 40, and synthesizes synthesized speech data output from the neural network.

Is output to the output unit 90.

<Operation of the speech synthesizer according to the first embodiment of the present invention>

  Next, the operation of the speech synthesizer 100 according to the first embodiment of the present invention will be described. The speech synthesizer 100 executes a learning processing routine and a generation processing routine described below.

  First, the learning processing routine will be described. When the input unit 10 receives human voice data x as learning data, the voice synthesizer 100 executes a learning processing routine shown in FIG.

  First, in step S100, the voice data x received by the input unit 10 is subjected to voice analysis to obtain a voice feature amount sequence f.

Next, in step S102, the speech feature amount sequence f obtained in step S100 and the speech data x received by the input unit 10 are input, and the speech feature amount sequence f is synthesized from the speech feature amount sequence f according to the above equation (1). Audio data

Neural network as a generator to generate

Then, learning is performed so as to optimize the objective function representing the distance from the voice data x, the learned neural network is stored in the neural network storage unit 40, and the process is terminated.

  Next, a generation processing routine will be described. When the input unit 10 receives an arbitrary voice feature amount sequence f for which synthetic voice data is to be generated, the voice synthesizer 100 executes a generation processing routine shown in FIG.

In step S200, an arbitrary speech feature amount sequence f received by the input unit 10 is input to a neural network stored in the neural network storage unit 40, and synthesized speech data output from the neural network is synthesized.

Is output to the output unit 90, and the process ends.

As described above, according to the speech synthesizer according to the first embodiment of the present invention, the speech feature amount sequence f and the true speech data x for learning are input, and the speech feature sequence f is calculated according to the above equation (1). Synthesized speech data synthesized from the speech feature amount sequence f

Neural network as a generator to generate

Then, learning is performed so as to optimize an objective function indicating a distance from the true speech data x for learning, whereby a neural network capable of synthesizing a more natural speech can be learned.

  Further, by synthesizing speech using the learned neural network, a more natural speech can be synthesized from the speech feature amount sequence.

<Overview according to Second Embodiment of the Present Invention>

  Next, an outline of the second embodiment of the present invention will be described.

  In the first embodiment, the voice is reproduced only from the speech feature amount sequence. However, in the second embodiment, the speech is reproduced by adding a new natural component as an input to the neural network. Express nature. FIG. 6 shows a conceptual diagram of the processing. Note that the speech feature amount series described here is obtained by speech analysis, but the natural component is given independently (for example, a random number).

<Configuration of Speech Synthesis Device According to Second Embodiment of the Present Invention>

  Next, the configuration of a speech synthesizer according to a second embodiment of the present invention will be described. The same parts as those in the first embodiment are denoted by the same reference numerals, and description thereof is omitted.

  As shown in FIG. 7, the speech synthesizer 200 according to the second embodiment of the present invention stores a CPU, a RAM, a program for executing a learning processing routine and a generation processing routine described below, and various data. And a computer including the ROM. The speech synthesizer 200 functionally includes the input unit 10, the arithmetic unit 220, and the output unit 90 as shown in FIG.

  The operation unit 220 includes a learning unit 230, a neural network storage unit 40, and a generation unit 250.

As described below, the learning unit 230 performs a voice analysis on the voice data x and obtains a voice feature amount sequence f used for a vocoder, a given natural component z, and a real voice for learning. Data x is input and synthesized speech data synthesized from the speech feature amount series f.

A neural network as a generator for generating (synthesized speech signal or synthesized speech spectrum sequence), and synthesized speech data

Has a neural network as a discriminator for discriminating whether or not it follows the same distribution as the true voice data, and a neural network as a generator and a neural network as a discriminator compete with each other in an optimization condition. Learn according to.

  The learning unit 230 first obtains a voice feature amount sequence f for the voice data x received by the input unit 10. A neural network as a generator such that the original true speech data x is generated based on the speech feature amount sequence f obtained here, the natural component z, and the true speech data x for learning. To learn. Here, the voice feature amount series f may be partially modified. Specifically, there is a fundamental frequency as one of the representative speech feature quantity sequences, but a fundamental frequency may be randomly multiplied by a constant. The natural component z is a random number generated according to a certain distribution (for example, a uniform distribution).

Also, true voice data x and synthetic voice data generated by a neural network as a generator

, A neural network as a discriminator for discriminating whether or not the synthesized speech data is true speech data is learned. The neural network as this discriminator discriminates whether the input synthesized speech data is true or synthesized, and outputs the result.

  In the present embodiment, the evaluation functions of the neural network as the generator and the neural network as the discriminator are optimized according to the following equation (2). In the equation (2), G represents a generator, and D represents a discriminator. In equation (2), the discriminator maximizes the evaluation function so as to be able to discriminate between the true speech and the synthesized speech as much as possible, while the generator makes the discriminator discriminate the synthesized speech from the true speech as much as possible. Then, the evaluation function is minimized. Optimization proceeds while the discriminator and the generator compete.

... (2)

  FIG. 8 is a conceptual diagram of the learning process according to the second embodiment.

  The neural network as a generator and the neural network as a discriminator, which have been learned to optimize the evaluation function of the above equation (2), are stored in the neural network storage unit 40.

  It should be noted that a neural network as a generator and a neural network as a discriminator are optimized so as to optimize an evaluation function using a discriminator (Discriminator) that also considers a speech feature amount sequence f as in the following equation (3). You may learn.

... (3)

  When learning a neural network, the neural network as a generator may be pre-trained using the method of the first embodiment.

The generation unit 250 inputs an arbitrary speech feature amount sequence f received by the input unit 10 and a natural component z given in advance to a neural network stored in the neural network storage unit 40. Output synthesized speech data

Is output to the output unit 90.

<Operation of the speech synthesizer according to the second embodiment of the present invention>

  Next, the operation of the speech synthesizer 200 according to the second embodiment of the present invention will be described. The speech synthesizer 200 executes a learning processing routine and a generation processing routine described below.

  First, the learning processing routine will be described. When the input unit 10 receives human voice data x as learning data, the voice synthesizer 200 executes the learning processing routine shown in FIG.

  In the learning processing routine according to the second embodiment, in step S102, the speech feature amount sequence f obtained in step S100, the natural component z given in advance, and the speech data x received by the input unit 10 are used. As inputs, the neural network as a generator and the neural network as a discriminator perform learning according to mutually competing optimization conditions in accordance with the above equation (2), and store the learned neural network in a neural network storage unit 40. And the process ends.

In the generation processing routine according to the second embodiment, as shown in FIG. 5 described above, in step S200, an arbitrary speech feature amount sequence f received by the input unit 10 and a natural component z given in advance are calculated. Synthesized synthesized speech data input to the neural network stored in the neural network storage unit 40 and output from the neural network

Is output to the output unit 90, and the process ends.

As described above, according to the speech synthesizer according to the second embodiment of the present invention, the speech feature amount sequence f, the natural component z, and the true speech data x for learning are input, Synthesized speech data synthesized from the speech feature amount sequence f according to the above equation (2)

Neural network as a generator for generating speech and synthesized speech data

However, a neural network as a discriminator for discriminating whether or not it follows the same distribution as the true speech data x performs learning according to mutually competing optimization conditions, so that a more natural speech can be synthesized. Learn neural networks.

  Further, by synthesizing speech using a neural network as a learned generator, a more natural speech can be synthesized.

<Overview according to Third Embodiment of the Present Invention>

  Next, an outline of a third embodiment of the present invention will be described.

  The first and second embodiments perform mapping between a speech feature amount sequence and high-quality sound speech, and are techniques that can replace an existing vocoder. On the other hand, the third embodiment is a method of performing mapping between speech once synthesized from a speech feature amount sequence and high quality speech. Here, in order to synthesize speech once from the speech feature amount sequence, an existing vocoder or the first and second embodiments may be used. FIG. 9 shows a conceptual diagram of the processing.

  When a speech feature amount sequence is given, an intermediate speech signal is obtained by using a vocoder or a neural network as a generator learned by the method of the first or second embodiment. The intermediate audio signal is input to a neural network and converted to obtain target audio data.

<Configuration of Speech Synthesis Device According to Third Embodiment of the Present Invention>

  Next, the configuration of a speech synthesizer according to a third embodiment of the present invention will be described. The same parts as those in the second embodiment are denoted by the same reference numerals, and description thereof is omitted.

  As shown in FIG. 10, the speech synthesizer 300 according to the third embodiment of the present invention stores a CPU, a RAM, a program for executing a learning processing routine and a generation processing routine described below, and various data. And a computer including the ROM. The speech synthesizer 300 functionally includes the input unit 10, the arithmetic unit 320, and the output unit 90 as shown in FIG.

  The calculation unit 320 includes a learning unit 330, a neural network storage unit 40, an intermediate voice conversion unit 332, and a generation unit 350.

As described below, the learning unit 330 synthesizes the speech from the speech feature amount sequence obtained by speech analysis of the speech data x, and obtains the intermediate speech data x ′ (intermediate speech signal or intermediate speech spectrum sequence). , A natural component z, and true speech data x for learning, and synthesized speech data synthesized from the intermediate speech data x ′.

Is provided with a neural network as a generator for generating the synthesized speech data.

Learning is performed so as to optimize an objective function representing a distance from the real audio data x for learning.

The learning unit 330 first obtains a voice feature amount sequence f for the voice data x received by the input unit 10. The speech feature quantity sequence f and the natural component z obtained here are input to a neural network as a generator trained in the same manner as in the second embodiment to obtain intermediate speech data x '. Then, a neural network as a generator is learned with respect to the intermediate audio data x 'such that the original true audio data x is generated. Specifically, when the intermediate audio data x ′ is input to the neural network, the audio data

Is output, but the synthesized voice data is output as true voice data x.

The weight of the neural network may be optimized so that the distance is minimized for a certain distance index. The distance index described here is, for example, a least square error. For the minimum square error as a distance indicator, the objective function L ₂ is expressed by the following equation (1).

... (4)

  FIG. 11 is a conceptual diagram of the learning process according to the third embodiment.

  The neural network as a generator that has been learned so as to optimize the objective function of the above equation (4) is stored in the neural network storage unit 40.

  The intermediate voice conversion unit 332 inputs an arbitrary voice feature amount sequence f received by the input unit 10 to the neural network (not shown) of the second embodiment, thereby generating intermediate voice data x ′ (an intermediate voice signal or Intermediate speech spectrum sequence).

The generation unit 350 inputs the intermediate voice data x ′ obtained by the intermediate voice conversion unit 332 to the neural network stored in the neural network storage unit 40, and synthesizes the synthesized voice data.

Is output to the output unit 90.

<Operation of the speech synthesizer according to the third embodiment of the present invention>

  Next, the operation of the speech synthesizer 300 according to the third embodiment of the present invention will be described. The speech synthesizer 300 executes a learning processing routine and a generation processing routine described below.

  First, the learning processing routine will be described. When the input unit 10 receives human voice data x as learning data, the voice synthesizer 300 executes a learning process routine shown in FIG.

  First, in step S300, the voice data x received by the input unit 10 is voice-analyzed to obtain a voice feature amount sequence f.

  Next, in step S302, the speech feature amount sequence f obtained in step S300, the natural component z, and the speech data x received by the input unit 10 are input, and learning is performed in the same manner as in the second embodiment. The intermediate speech data x '(intermediate speech signal or intermediate speech spectrum sequence) is obtained by inputting the signal to a neural network as a generator.

In step S304, the intermediate voice data x 'obtained in step S302 and the voice data x received by the input unit 10 are input, and the synthesized voice data synthesized from the intermediate voice data x' according to the above equation (4).

The neural network as a generator for generating the learning performs learning so as to optimize the objective function, stores the learned neural network in the neural network storage unit 40, and ends the processing.

  Next, a generation processing routine will be described. When the input unit 10 receives an arbitrary voice feature amount sequence f for which synthetic voice data is to be generated, the voice synthesizer 300 executes a generation processing routine shown in FIG.

  In step S400, an arbitrary speech feature amount sequence f received by the input unit 10 is input to a neural network (not shown) as a learned generator in the same manner as in the second embodiment, so that the intermediate speech data x Get '.

In step S402, the intermediate voice data x 'obtained in step S400 is input to the neural network stored in the neural network storage unit 40, and the synthesized voice data is synthesized.

Is output to the output unit 90, and the process ends.

As described above, according to the speech synthesizer according to the third embodiment of the present invention, the intermediate speech data x ′ obtained by synthesizing speech from the speech feature amount sequence and the true speech for learning are used. And the synthesized speech data synthesized from the intermediate speech data x ′ according to the above equation (4).

The neural network as a generator for generating the learning can perform learning so as to optimize the objective function, thereby learning a neural network capable of synthesizing a more natural speech.

  Further, by synthesizing speech using the learned neural network, a more natural speech can be synthesized.

  Note that the case where a neural network trained in the same manner as in the second embodiment is used for conversion to intermediate voice data has been described as an example, but the present invention is not limited to this, and a vocoder or a first vocoder may be used. A speech feature amount sequence may be converted to intermediate speech data using a neural network learned in the same manner as in the embodiment.

  When a neural network that has been trained in the same manner as in the first or second embodiment is used to convert to intermediate voice data, the learning process is performed after the learning process in this embodiment is performed. The neural network may be regarded as pre-training, and the entire neural network may be optimized again.

<Overview according to Fourth Embodiment of the Present Invention>

  Next, an outline of a fourth embodiment of the present invention will be described.

  In the third embodiment, intermediate sound data is directly converted into natural sound. In the fourth embodiment, natural sound components are added to the intermediate sound data to convert the sound into genuine sound. is there. FIG. 14 shows a conceptual diagram of the processing. The natural component described here is given independently of the synthesized speech (for example, a random number).

<Configuration of Speech Synthesis Device According to Fourth Embodiment of the Present Invention>

  Next, the configuration of a speech synthesizer according to a fourth embodiment of the present invention will be described. The same parts as those in the third embodiment are denoted by the same reference numerals, and description thereof is omitted.

  As shown in FIG. 15, the speech synthesizer 400 according to the fourth embodiment of the present invention stores a CPU, a RAM, a program for executing a learning process routine and a generation process routine described later, and various data. And a computer including the ROM. The speech synthesizer 400 functionally includes the input unit 10, the operation unit 420, and the output unit 90 as shown in FIG.

  The calculation unit 420 includes a learning unit 430, a neural network storage unit 40, an intermediate speech conversion unit 332, and a generation unit 450.

The learning unit 430, as described below, synthesizes speech from a speech feature amount sequence obtained by speech analysis of the speech data x, and obtains intermediate speech data x ′ (intermediate speech signal or intermediate speech spectrum sequence). If, 'a natural component z ₂ corresponding to, and inputs the true voice data x for learning, the intermediate audio data x' intermediate audio data x from as generator for generating a synthesized synthesized speech data Neural network and synthesized speech data

Comprises a discriminator for discriminating whether or not it follows the same distribution as the true voice data x. A neural network as a generator and a neural network as a discriminator perform learning according to mutually competing optimization conditions. Do.

The learning unit 430 first obtains a voice feature amount sequence f for the voice data x received by the input unit 10. Here the audio feature amount sequence f obtained, and a natural component z ₁ corresponding to the audio feature amount sequence f, be entered into the neural network as a generator learned as in the second embodiment To obtain intermediate audio data x ′. An intermediate voice data x 'obtained here, the natural component z _2, based on the true voice data x for learning, neural as generator as true speech data x to be based is generated Learn the network. Here, the voice feature amount series f may be partially modified. Specifically, there is a fundamental frequency as one of the representative speech feature quantity sequences, but a fundamental frequency may be randomly multiplied by a constant. Also, the natural component z ₁ and natural component z ₂ are distributed (e.g., uniform distribution) a random number generated according to.

Also, true voice data x and synthetic voice data generated by a neural network as a generator

, A neural network as a discriminator for determining whether or not the distribution follows the same distribution as the true voice data x is learned. The neural network as the discriminator discriminates whether the input speech data is true or synthesized, and outputs the result.

  In the present embodiment, the evaluation functions of the neural network as the generator and the neural network as the discriminator are optimized according to the following equation (5). In the equation (5), G represents a generator, and D represents a discriminator. In equation (5), the discriminator maximizes the evaluation function so as to be able to discriminate the true speech from the synthesized speech as much as possible, while the generator makes the discriminator discriminate the synthesized speech from the true speech as much as possible. Then, the evaluation function is minimized. Optimization proceeds while the discriminator and the generator compete.

... (5)

  FIG. 16 shows a conceptual diagram of the learning processing according to the fourth embodiment.

  The neural network as a generator and the neural network as a discriminator, which have been learned so as to optimize the evaluation function of the above equation (5), are stored in the neural network storage unit 40.

  It should be noted that a neural network as a generator and a neural network as a discriminator are optimized so as to optimize the evaluation function using a discriminator (Discriminator) in consideration of the intermediate voice data x ′ as in the following equation (6). You may learn.

... (6)

  When learning a neural network, the neural network as a generator may be pre-trained by using the method of the third embodiment.

Generating unit 450 includes an intermediate audio data x 'obtained by the intermediate voice conversion unit 332 inputs the natural component z ₂ previously given, the neural network is stored in the neural network memory section 40, synthesis Synthesized speech data

Is output to the output unit 90.

<Operation of the speech synthesizer according to the fourth embodiment of the present invention>

  Next, the operation of the speech synthesizer 400 according to the fourth embodiment of the present invention will be described. The speech synthesizer 400 executes a learning processing routine and a generation processing routine described below.

  First, the learning processing routine will be described. When the input unit 10 receives human voice data x as learning data, the voice synthesizer 400 executes the learning processing routine shown in FIG.

In the learning processing routine of the fourth embodiment, and in step S304, the intermediate audio data x 'obtained in step S302, the input and natural component z _2, and the audio data x accepted by the input unit 10, According to the above equation (5), the neural network as the generator and the neural network as the discriminator perform learning according to mutually competing optimization conditions, and store the learned neural network in the neural network storage unit 40. To end the processing.

The generation processing routine of the fourth embodiment, as shown in FIG. 13, in step S402, the intermediate audio data x 'obtained in step S400, a natural component z _2, the neural network memory section 40 Synthesized speech data input to the stored neural network and synthesized

Is output to the output unit 90, and the process ends.

  The generation processing routine of the fourth embodiment is the same as that of the third embodiment, and a description thereof will not be repeated.

As described above, according to the speech synthesizer according to the fourth embodiment of the present invention, the intermediate speech data x ′ obtained by synthesizing the speech from the speech feature amount sequence, the natural component z ₂ , , Real speech data x for learning, and synthesized speech data synthesized from the intermediate speech data x ′ according to the above equation (5).

Neural network as a generator for generating speech and synthesized speech data

However, a neural network as a discriminator for discriminating whether or not it follows the same distribution as the true speech data x performs learning according to mutually competing optimization conditions, so that a more natural speech can be synthesized. Learn neural networks.

  Further, by synthesizing speech using a neural network as a learned generator, a more natural speech can be synthesized.

  Note that the case where a neural network trained in the same manner as in the second embodiment is used for conversion to intermediate voice data has been described as an example, but the present invention is not limited to this, and a vocoder or a first vocoder may be used. A speech feature amount sequence may be converted to intermediate speech data using a neural network learned in the same manner as in the embodiment.

  When a neural network that has been trained in the same manner as in the first or second embodiment is used to convert the data into intermediate voice data, the learning process is performed after the learning process in this embodiment is performed. The neural network may be regarded as pre-training, and the entire neural network may be optimized again.

<Overview according to Fifth Embodiment of the Present Invention>

  Next, an outline of a fifth embodiment of the present invention will be described.

  As the speech feature sequence used in the first to fourth embodiments, for example, a speech feature sequence obtained by existing speech analysis can be used, but a speech feature sequence obtained by a neural network is used as an input. Can also. This is because the first to fourth embodiments learn the mapping of the audio feature amount sequence and the audio signal in a data-driven manner.

<Configuration of Speech Synthesis Device According to Fifth Embodiment of the Present Invention>

  Next, the configuration of a speech synthesizer according to a fifth embodiment of the present invention will be described. Note that portions having the same configuration as in the second embodiment are denoted by the same reference numerals, and description thereof is omitted.

  As shown in FIG. 18, the speech synthesizer 500 according to the fifth embodiment of the present invention stores a CPU, a RAM, a program for executing a learning processing routine and a generation processing routine described below, and various data. And a computer including the ROM. This speech synthesizer 500 functionally includes an input unit 510, a calculation unit 520, and an output unit 90 as shown in FIG.

  The input unit 510 receives human voice data x as learning data. Further, input unit 510 accepts arbitrary audio data to be a target of generating synthetic audio data.

  The calculation unit 520 is configured to include a voice feature value generation unit 528, a learning unit 530, the neural network storage unit 40, a voice feature value conversion unit 532, and a generation unit 250.

  The audio feature generation unit 528 inputs the audio data x received by the input unit 510 to an Auto Encoder, which is a neural network that has been learned in advance, and outputs the audio feature sequence f output from the Auto Encoder to the learning unit 530. I do. The neural network used here may be a previously learned Variational Auto Encoder.

The learning unit 530 receives the speech feature amount sequence f output from the speech feature amount generation unit 528, the naturalness component z, and the true speech data x for learning, and is synthesized from the speech feature amount sequence f. Neural network as a generator for generating synthesized speech data, and synthesized speech data

Has a neural network as a discriminator for discriminating whether or not it follows the same distribution as the true voice data x, and a neural network as a generator and a discriminator as a discriminator by the same processing as in the second embodiment. May be learned in accordance with optimization conditions that compete with each other.

  FIG. 15 shows a conceptual diagram of the learning processing according to the fifth embodiment.

  Like the speech feature generation unit 528, the speech feature generation unit 532 inputs arbitrary speech data to be subjected to generation of the synthesized speech data received by the input unit 510 to the Auto Encoder which is a pre-trained neural network. Then, the audio feature amount sequence f output from the Auto Encoder is output to the generation unit 250.

  Note that the other configuration of the fifth embodiment is the same as that of the second embodiment, and a description thereof will not be repeated.

Further, in the fifth embodiment, the case where the learning unit 530 performs the same processing as that of the second embodiment has been described, but the present invention is not limited to this. For example, the learning unit 530 receives the speech feature amount sequence f output from the speech feature amount generation unit 528 and the true speech data x for learning, and generates the same by the same processing as in the first embodiment. A neural network as a vessel may be learned. Further, from the audio feature amount sequence f output from the audio feature amount generation unit 528, the intermediate audio data x 'is obtained by the same processing as in the third embodiment, and the obtained intermediate audio data x' and learning May be used as input, and a neural network as a generator may be learned. Further, the audio feature amount sequence f output from the audio feature amount generating unit 528, from the natural components z ₁ Prefecture, to obtain intermediate speech data x 'by the fourth same processing as the embodiment of the resulting an intermediate voice data x ', a natural component z _2, and inputs the real audio data x for training the neural network as a generator, or a neural network as a generator and classifier so as to learn You may.

<Operation of Speech Synthesis Device According to Fifth Embodiment of the Present Invention>

  Next, the operation of the speech synthesizer 500 according to the fifth embodiment of the present invention will be described. The speech synthesizer 500 executes a learning processing routine and a generation processing routine described below. The same parts as those in the second embodiment are denoted by the same reference numerals, and description thereof is omitted.

  First, the learning processing routine will be described. When the input unit 510 receives human voice data x as learning data, the voice synthesizer 500 executes a learning processing routine shown in FIG.

  In step S500, the audio data x received by the input unit 510 is input to an Auto Encoder, which is a neural network learned in advance, and the audio feature amount sequence f output from the Auto Encoder is output to the learning unit 530.

  In step S102, the speech feature value sequence f obtained in step S500, the natural component z given in advance, and the speech data x received by the input unit 510 are input, and the generator is generated according to the above equation (2). And the neural network as a discriminator perform learning in accordance with mutually competing optimization conditions, store the learned neural network in the neural network storage unit 40, and end the process.

  Next, a generation processing routine will be described. When the input unit 510 receives voice data to be a target of generating synthesized voice data, the voice synthesizer 500 executes a generation processing routine shown in FIG.

  In step S500, the audio data received by the input unit 510 is input to an Auto Encoder, which is a neural network learned in advance, and the audio feature amount sequence f output from the Auto Encoder is output to the generation unit 250.

  Note that other operations of the fifth embodiment are the same as those of the second embodiment, and thus the description is omitted.

As described above, according to the speech synthesizer according to the fifth embodiment of the present invention, the speech data x is input to the Auto Encoder, which is a neural network learned in advance, and the speech output from the Auto Encoder The feature amount sequence f is output, and the output speech feature amount sequence f, the natural component z, and the true speech data x for learning are input, and the speech feature amount sequence f is obtained from the speech feature amount sequence f according to the above equation (2). Synthesized voice data synthesized

Neural network as a generator for generating speech and synthesized speech data

Learns a neural network capable of synthesizing a more natural speech by performing learning according to optimization conditions that compete with each other, and a discriminator that determines whether or not the distribution follows the same distribution as the true speech data x. it can.

  Further, by synthesizing speech using a neural network as a learned generator, a more natural speech can be synthesized.

<Experimental result 1>

  To demonstrate the effectiveness of the third and fourth embodiments, an experiment was performed using one realization method.

  Experimental data As the data for the experiment, 115 conversation sentences of one speaker in ATR Speech Data were used. 90% of this data was used for training the model, and the remaining 10% was used for testing. Note that the sampling frequency of the audio signal is 16,000 Hz.

  In the third and fourth embodiments, a speech signal or a speech spectrum sequence generated by a vocoder or a neural network having equivalent inputs and outputs is used as the input of the generator. In this experiment, a voice signal was generated using the Vocoder, and a voice spectrum sequence x 'obtained by performing preprocessing described below was input thereto. As a specific analytical synthesis method, LPC analytical synthesis was used. The purpose of a generator composed of a neural network is to convert the sound generated by the analysis and synthesis into a real voice such as an original voice signal.

  The pre-processing described above is the following processing. First, a short-time Fourier transform (STFT) was applied to each audio signal to convert it into a complex spectrum sequence. At this time, the window width of the Fourier transform was 512, and the shift width was 128. A Blackman window was used as the window function. Next, the absolute value of the complex spectrum series was taken and converted to an amplitude spectrum series. The amplitude spectrum was converted to a logarithmic spectrum of amplitude by taking a logarithmic spectrum with a base of 10 and multiplying by 20. Finally, a certain frame was cut out from the spectrum sequence obtained by this processing and used as an input to the generator. In the experiment, the cut-out length of the frame was set to 21.

  In addition, since the logarithmic spectrum having the same amplitude as the input was obtained as the output of the generator, the following processing was performed to return it to the audio signal. First, the logarithmic spectrum of the amplitude was first divided by 20, and the obtained value was used as a multiplier to find a power of 10, thereby converting the spectrum into an amplitude spectrum. On the other hand, the phase was restored using Griffin Lim and converted to an audio signal.

  FIG. 21 shows an implementation example of the learning method according to the third embodiment, and FIG. 22 shows an implementation example of the learning method according to the fourth embodiment.

  FIG. 23 shows an implementation example of the generation method according to the third embodiment, and FIG. 24 shows an implementation example of the generation method according to the fourth embodiment.

  As the network structure, both the generator and the discriminator of the third and fourth embodiments have three hidden layers, the number of units in each layer is 500, and the connection method is Fully Connected. FIGS. 25 and 26 show specific network structures of the third and fourth embodiments, respectively.

  The purpose of this method is to convert the synthesized synthesized sound into a sound close to the real voice. To demonstrate the effectiveness of the proposed framework, the following four synthesized sounds were assumed.

1.Volume change: n original sound halved
2.Pre-emphasis: sound with treble emphasis on the original sound
3.LPC: LPC analysis synthesized sound
4.LPC + pulse: Sound generated by combining LPC obtained by LPC analysis with pulse signal generated at regular intervals (every 128 samples)

  FIG. 27 shows an example of a waveform of an audio signal that is a source of input / output.

  FIG. 28 shows the experimental results of Volume change. Focusing on the magnitude of the amplitude of the waveform data of the audio signal, the synthesized sound is half of the original sound, but using the methods of the third and fourth embodiments, the original sound It can be seen that the same amplitude as that of can be reproduced.

  FIG. 29 shows the experimental results of Pre-emphasis. Focusing on the voice spectrum series, the synthesized sound has a lower value in the low frequency region than the original sound, but using the methods of the third and fourth embodiments, the same value as the original sound is obtained. It turns out that it has returned to the size order. In addition, when focusing on the amplitude of the waveform data of the audio signal, the synthesized sound is generally smaller than the original sound. It can be seen that the same amplitude as the original sound can be reproduced.

  FIG. 30 shows the experimental results of LPC. Focusing on the speech spectrum series, the synthesized sound has a characteristic that the value is lower in the lowest frequency region (0) and is wider in the higher frequency region as compared with the original sound. It can be seen that using the method of the fourth embodiment returns to the same shape as the original sound. Also, when focusing on the amplitude of the waveform data of the audio signal, the synthesized sound is generally louder than the original sound, but using the methods of the third and fourth embodiments, It can be seen that the same amplitude as the original sound can be reproduced.

  FIG. 31 shows the experimental results of LPC + pulse. Paying attention to the audio spectrum series, the synthesized sound has a characteristic that it is an evenly spaced striped power as compared with the original sound, but using the methods of the third and fourth embodiments, It can be seen that the shape returns to a shape closer to the original sound as compared to the original synthesized sound.

<Experimental result 2>
Next, experimental results of the first and second embodiments will be described.

  Experimental Data First, as a data set, the same data set as used in the experiments relating to the third and fourth embodiments was used. In the first and second embodiments, a speech feature amount sequence is used as input, but in this experiment, a speech feature amount obtained by LPC analysis was used. Specifically, pitch and LPC were used. The pitch is a one-dimensional feature amount per frame, and the LPC is a 26-dimensional feature amount because the order at the time of LPC analysis is 25. Therefore, the sum of the two becomes a 27-dimensional feature amount per dimension. As an output, a logarithmic spectrum of the amplitude was used. In this experiment, the frame length of the data to be actually processed was 1. Ultimately, the purpose is to obtain an audio signal, and for that purpose, it is necessary to restore the audio signal from the logarithmic spectrum of the amplitude obtained as an output. As the method, a method similar to the method described in the experimental section regarding the third and fourth embodiments was used.

  FIGS. 32 and 33 show implementation examples of the network structure of the first and second embodiments. Regarding the network structure, the first and second embodiments both have three hidden layers, the number of units in each layer is 500, and the way of connecting the layers is Fully Connected.

  FIG. 34 shows the result of audio restoration by the methods of the first and second embodiments. From now on, the speech feature amount used for input is 27-dimensional. On the other hand, by using the networks of the first and second embodiments, a harmonic structure having features similar to the original sound ( It can be seen that the spectrum stripe pattern) was reproduced.

  The present invention is not limited to the above-described embodiment, and various modifications and applications can be made without departing from the gist of the present invention.

  For example, in the above-described embodiment, the speech synthesizer is configured to include the learning unit that learns the neural network and the generation unit that synthesizes the speech. However, the present invention is not limited to this. , And a speech synthesis device including a generation unit.

  Moreover, CNN, RNN, or the like can also be used for the neural network in the above-described embodiment.

10, 510 Input unit 20, 220, 320, 420, 520 Operation unit 30, 230, 330, 430, 530 Learning unit 40 Neural network storage unit 50, 250, 350, 450 Generation unit 90 Output unit 100, 200, 300, 400, 500 voice synthesizer 332 intermediate voice converters 528, 532 voice feature generator

Claims (4)

A speech synthesis learning device for learning a neural network that synthesizes speech from arbitrary speech data or a speech feature amount sequence,
A neural network as a generator that receives input voice data or voice feature amount sequence and true voice data for learning and generates synthesized voice data from the voice data or voice feature amount sequence; The synthesized speech data further includes a neural network as a discriminator for discriminating whether or not the speech data follows the same distribution as true speech data, and the neural network as the generator and the neural network as the discriminator, A speech synthesis learning device that includes a learning unit that performs learning according to mutually competing optimization conditions.   The learning unit receives, as input, a speech feature amount sequence used for a vocoder that synthesizes speech from a speech feature amount sequence obtained by speech analysis of speech data and true speech data for learning. The speech synthesis learning device according to claim 1, wherein the neural network as the generator that generates synthesized speech data synthesized from the sequence and the neural network as the discriminator perform learning according to mutually competing optimization conditions. .   The learning unit receives a speech signal or a speech spectrum sequence obtained by synthesizing speech from a speech feature amount sequence and true speech data for learning as inputs, and synthesizes the speech signal or speech spectrum sequence. The speech synthesis learning device according to claim 1, wherein the neural network as the generator that generates speech data and the neural network as the discriminator perform learning according to mutually competing optimization conditions.   The learning unit receives a speech feature amount sequence output from an Auto Encoder, which is a neural network previously learned by inputting speech data, and a true speech data for learning, and is synthesized from the speech feature amount sequence. The speech synthesis learning device according to claim 1, wherein the neural network as the generator that generates the synthesized speech data and the neural network as the discriminator perform learning according to mutually competing optimization conditions.