JP6722810B2 - Speech synthesis learning device - Google Patents

Description

The present invention relates to a speech synthesis learning device, and more particularly to a speech synthesis learning device for synthesizing speech.

The vocal chord sound source information (fundamental frequency, aperiodic index, etc.) and vocal tract spectrum information are obtained by a voice analysis method such as STRAIGHT or Mel-Generalized Cepstral Analysis (MGC). You can In many text-to-speech synthesis systems and voice conversion systems, an approach is used in which such a sequence of voice feature quantities is predicted from an input text or a source voice and a voice signal is generated according to a vocoder method.

In the existing vocoder-based speech synthesis, a vocoder is used to convert a speech feature amount sequence such as vocal cord source information and vocal tract spectrum information to generate speech. FIG. 35 shows a conceptual diagram of a vocoder-based speech synthesis process. The vocoder described here is a model of the sound generation process based on the knowledge of the mechanism of human vocalization. For example, as a typical model of a vocoder, there is a source filter model. In this model, a sound generation process is described by two sources, that is, a source and a digital filter. Specifically, it is said that a voice is generated by applying a digital filter to a voice signal (represented by a pulse signal) generated from a source as needed. As described above, in the vocoder-based speech synthesis, the utterance mechanism is abstractly modeled and expressed, so that the speech can be expressed compactly (low dimension). On the other hand, as a result of abstraction, the naturalness of the voice is often lost, resulting in mechanical sound quality peculiar to the vocoder.

Ian J. Goodfellow, Jean Pouget-Abadie, Mehdi Mirza, Bing Xu, DavidWarde-Farley, Sherjil Ozairy,Aaron Courville, Yoshua Bengio, "Generative Adversarial Nets," 2014. Emily Denton, Soumith Chintala, Arthur Szlam, Rob Fergus, "Deep Generative Image Modelsusing a Laplacian Pyramid of Adversarial Networks," 2015.

The problem of predicting appropriate speech features from input text or source speech is a kind of regression (machine learning) problem, and compact (low-dimensional) features especially when only a limited number of learning samples are obtained. The expression is more advantageous in statistical prediction. In many text-to-speech synthesis systems and voice conversion systems, a vocoder method using voice features (rather than trying to directly predict a waveform or spectrum) is used to take advantage of this advantage. On the other hand, the voice generated by the vocoder system often has a mechanical sound quality peculiar to the vocoder, which gives a potential limit to the sound quality in the conventional text-to-speech synthesis system and voice conversion system.

The present invention has been made to solve the above problems, and an object of the present invention is to provide a speech synthesis learning device capable of learning a neural network capable of synthesizing more natural speech.

In order to achieve the above object, a speech synthesis learning device according to the present invention is a speech synthesis learning device for learning a neural network for synthesizing speech from arbitrary speech data or speech feature quantity sequence, wherein the input speech data Alternatively, the first learned in advance so as to accept the voice feature amount sequence and the true voice data for learning and generate intermediate voice data from the voice data or the voice feature amount sequence and the true voice data for learning. A neural network as a generator and a neural network as a second generator that is learned to generate synthetic speech data from the intermediate speech data and the true speech data for learning. Alternatively, the intermediate feature data is obtained by using the voice feature quantity sequence as an input to the neural network as the first generator, and the obtained intermediate voice data is supplied to the neural network as the second generator. The synthetic voice data is generated as an input, and the objective function representing the distance between the generated synthetic voice data and the true voice data for learning is optimized, or as the second generator. The neural network and the neural network as a discriminator that determines whether or not the generated synthetic speech data follows the same distribution as the true speech data for learning, so that the neural network complies with optimization conditions competing with each other, A learning unit that learns a neural network as a second generator, and the neural network as the first generator has an objective function that represents a distance between the intermediate speech data and the true speech data for learning. According to the above optimization, or a neural network as the first generator, and a neural network as a discriminator for determining whether or not the intermediate voice data follows the same distribution as the true voice data for learning. Are previously learned according to the optimization conditions that compete with each other.

According to the speech synthesis learning device of the present invention, it is possible to obtain the effect that a neural network capable of synthesizing a more natural speech can be learned.

It is a conceptual diagram of a process of the 1st Embodiment of this invention. It is a block diagram which shows the structure of the speech synthesizer which concerns on the 1st Embodiment of this invention. It is a conceptual diagram of the learning process of the 1st Embodiment of this invention. It is a flowchart which shows the learning process routine in the speech synthesizer which concerns on the 1st and 2nd embodiment of this invention. It is a flowchart which shows the production|generation processing routine in the speech synthesis apparatus which concerns on the 1st and 2nd embodiment of this invention. It is a conceptual diagram of the process of the 2nd Embodiment of this invention. It is a block diagram which shows the structure of the speech synthesizing|combining apparatus which concerns on the 2nd Embodiment of this invention. It is a conceptual diagram of the learning process of the 2nd Embodiment of this invention. It is a conceptual diagram of a process of the 3rd Embodiment of this invention. It is a block diagram which shows the structure of the speech synthesizing|combining apparatus which concerns on the 3rd Embodiment of this invention. It is a conceptual diagram of the learning process of the 3rd Embodiment of this invention. It is a flowchart which shows the learning process routine in the speech synthesis apparatus which concerns on the 3rd and 4th embodiment of this invention. It is a flowchart which shows the production|generation processing routine in the speech synthesis apparatus which concerns on the 3rd and 4th embodiment of this invention. It is a conceptual diagram of a process of the 4th Embodiment of this invention. It is a block diagram which shows the structure of the speech synthesizing|combining apparatus which concerns on the 4th Embodiment of this invention. It is a conceptual diagram of the learning process of the 4th Embodiment of this invention. It is a conceptual diagram of the 5th Embodiment of this invention. It is a block diagram which shows the structure of the speech synthesizing|combining apparatus which concerns on the 5th Embodiment of this invention. It is a flowchart which shows the learning process routine in the speech synthesizer which concerns on the 5th Embodiment of this invention. It is a flowchart which shows the production|generation processing routine in the speech synthesis apparatus which concerns on the 5th Embodiment of this invention. It is a figure which shows the implementation example of the learning method of 3rd Embodiment in an experiment example. It is a figure which shows the implementation example of the learning method of 4th Embodiment in an experiment example. It is a figure which shows the implementation example of the production|generation method of 3rd Embodiment in an experiment example. It is a figure which shows the implementation example of the production|generation method of 4th Embodiment in an experiment example. It is a figure which shows the network structure of 3rd Embodiment in an experimental example. It is a figure which shows the network structure of 4th Embodiment in an experiment example. It is a figure which shows the example of the waveform of the audio|voice signal used as the input/output origin in an experiment example. It is a figure which shows the experimental result of Volume change. It is a figure which shows the experimental result of Pre-emphasis. It is a figure which shows the experimental result of LPC. It is a figure which shows the experimental result of LPC+pulse. It is a figure which shows the network structure of 1st Embodiment in an experimental example. It is a figure which shows the network structure of 2nd Embodiment in an experimental example. It is a figure which shows the result of the audio|voice restoration by the method of the 1st and 2nd embodiment in an experimental example. It is a conceptual diagram of a vocoder-type speech synthesis process.

Hereinafter, embodiments of the present invention will be described in detail with reference to the drawings.

<Outline of First Embodiment of the Present Invention>

First, the outline of the first embodiment of the present invention will be described.

The existing vocoder-based speech synthesis is an abstract modeling of the sound generation process based on the knowledge of the human vocalization mechanism.From the speech feature quantity sequence to speech data (speech signal or speech spectrum sequence, It is not a direct optimization to reproduce

In the first embodiment of the present invention, this problem is solved by directly optimizing the mapping of the audio feature amount sequence and the audio data. A conceptual diagram of the processing is shown in FIG. The target voice data can be obtained by applying a neural network optimized for the mapping of the voice feature amount sequence and the voice data to the input voice feature amount sequence. At this time, when the voice signal is used as the voice data, the target voice signal is obtained as it is. On the other hand, when the voice spectrum series is used as the voice data, the output is also the voice spectrum series. In that case, the target voice signal can be obtained by performing the phase restoration. As a method of phase restoration, there is Griffin Lim, for example.

<Structure of speech synthesizer according to first embodiment of the present invention>

Next, the configuration of the speech synthesizer according to the first embodiment of the present invention will be described. As shown in FIG. 2, the speech synthesis apparatus 100 according to the first embodiment of the present invention stores a CPU, a RAM, a program for executing a learning processing routine and a generation processing routine described later, and various data. And a computer including the ROM. This speech synthesizer 100 functionally includes an input unit 10, a calculation unit 20, and an output unit 90 as shown in FIG.

The input unit 10 receives human voice data x as learning data. In addition, the input unit 10 receives an arbitrary voice feature amount sequence f for which synthetic voice data is to be generated.

The calculation unit 20 includes a learning unit 30, a neural network storage unit 40, and a generation unit 50.

As will be described below, the learning unit 30 receives the voice feature amount series f used for the vocoder and the true voice data x for learning obtained by voice analysis of the voice data x, and inputs the voice feature amount. Synthesized voice data synthesized from the series f

A neural network as a generator for generating the

And learning is performed so as to optimize the objective function representing the distance from the true voice data x for learning.

The learning unit 30 first obtains a voice feature amount sequence f by performing voice analysis on the voice data x received by the input unit 10. The neural network is learned so that the original true voice data x is generated for the voice feature amount sequence f obtained here. Specifically, when the speech feature amount series f is input to the neural network, the synthesized speech data

Is output, but true voice data x and synthetic voice data to be output

The weights of the neural network may be optimized so that and are minimized with respect to a certain distance index. The distance index described here is, for example, the least square error. When the least square error is used as the distance index, the objective function L ₂ is represented by the following equation (1).

...(1)

FIG. 3 shows a conceptual diagram of the learning process of the first embodiment.

The neural network as a generator that has been learned to optimize the objective function of the equation (1) is stored in the neural network storage unit 40.

The generation unit 50 inputs the arbitrary speech feature amount sequence f received by the input unit 10 into the neural network stored in the neural network storage unit 40, and the synthesized synthetic speech data output from the neural network.

Is output to the output unit 90.

<Operation of the speech synthesizer according to the first embodiment of the present invention>

Next, the operation of the speech synthesizer 100 according to the first embodiment of the present invention will be described. The speech synthesizer 100 executes a learning processing routine and a generation processing routine described below.

First, the learning processing routine will be described. When the input unit 10 receives human voice data x as learning data, the voice synthesizing apparatus 100 executes a learning processing routine shown in FIG.

First, in step S100, the voice data x received by the input unit 10 is voice analyzed to obtain a voice feature amount sequence f.

Next, in step S102, the speech feature quantity sequence f obtained in step S100 and the speech data x received by the input unit 10 are input, and synthesis is performed from the speech feature quantity sequence f according to the above equation (1). Voice data

Neural network as a generator that generates

Then, learning is performed so as to optimize the objective function representing the distance from the voice data x, the learned neural network is stored in the neural network storage unit 40, and the processing is ended.

Next, the generation processing routine will be described. When the input unit 10 receives an arbitrary speech feature amount sequence f for which synthetic speech data is to be generated, the speech synthesis device 100 executes the generation processing routine shown in FIG.

In step S200, the synthesized speech data synthesized by inputting the arbitrary speech feature amount sequence f received by the input unit 10 into the neural network stored in the neural network storage unit 40 and output from the neural network.

Is output to the output unit 90, and the process ends.

As described above, according to the speech synthesizer according to the first embodiment of the present invention, the speech feature amount series f and the true speech data x for learning are input and the expression (1) is followed. , Synthetic speech data synthesized from the speech feature sequence f

Neural network as a generator that generates

Then, by performing learning so as to optimize the objective function representing the distance from the true voice data x for learning, a neural network capable of synthesizing a more natural voice can be learned.

Further, by synthesizing the voice using the learned neural network, it is possible to synthesize a more natural voice from the voice feature quantity sequence.

<Overview of Second Embodiment of the Present Invention>

Next, the outline of the second embodiment of the present invention will be described.

In the first embodiment, the voice is reproduced only from the voice feature amount series, but in the second embodiment, a new natural component is added as an input of the neural network, and Express the naturalness. A conceptual diagram of the processing is shown in FIG. The speech feature quantity sequence described here is obtained by speech analysis, but the naturalness component is given independently of it (for example, a random number).

<Structure of speech synthesizer according to second embodiment of the present invention>

Next, the configuration of the speech synthesizer according to the second embodiment of the present invention will be described. The same parts as those in the first embodiment are designated by the same reference numerals and the description thereof will be omitted.

As shown in FIG. 7, a speech synthesis apparatus 200 according to the second embodiment of the present invention stores a CPU, a RAM, a program for executing a learning processing routine and a generation processing routine described later, and various data. And a computer including the ROM. This speech synthesizer 200 functionally includes an input unit 10, a calculation unit 220, and an output unit 90, as shown in FIG. 7.

The calculation unit 220 includes a learning unit 230, a neural network storage unit 40, and a generation unit 250.

As will be described below, the learning unit 230 uses a voice feature amount sequence f used for a vocoder, obtained by voice analysis of the voice data x, a natural component z given in advance, and a true voice for learning. Synthesized voice data that is synthesized from the voice feature series f by inputting the data x

Neural network as a generator for generating (synthetic speech signal or synthetic speech spectrum sequence), and synthetic speech data

Is a neural network as a discriminator for discriminating whether or not to follow the same distribution as the true voice data, and the neural network as a generator and the neural network as a discriminator compete with each other for optimization conditions. Learn according to.

The learning unit 230 first obtains the audio feature amount sequence f for the audio data x received by the input unit 10. A neural network as a generator so that the original true voice data x is generated based on the obtained voice feature amount sequence f, the naturalness component z, and the true voice data x for learning. To learn. Here, as the audio feature quantity sequence f, a partially modified version may be used. Specifically, the fundamental frequency is one of the typical ones of the speech feature amount series, but a fundamental frequency may be randomly multiplied by a constant. The naturalness component z is a random number generated according to a certain distribution (for example, uniform distribution).

In addition, true voice data x and synthetic voice data generated by a neural network as a generator

Based on and, the neural network as a discriminator for discriminating whether or not the synthesized voice data is the true voice data is learned. The neural network as the discriminator discriminates whether the input synthetic speech data is true or synthesized, and outputs the result.

In this embodiment, the evaluation functions of the neural network as the generator and the neural network as the discriminator are optimized according to the following equation (2). In the equation (2), G represents a generator and D represents a discriminator. In equation (2), the classifier maximizes the evaluation function so that the true speech and the synthetic speech can be discriminated from each other as much as possible, while the generator discriminates the synthetic speech from the true speech as much as possible. Then, the evaluation function is minimized. Optimization progresses while the discriminator and the generator compete.

...(2)

FIG. 8 shows a conceptual diagram of the learning process of the second embodiment.

The neural network as a generator and the neural network as a discriminator learned so as to optimize the evaluation function of the equation (2) are stored in the neural network storage unit 40.

As shown in the following equation (3), a neural network as a generator and a neural network as a discriminator are used so as to optimize an evaluation function using a discriminator that also considers the speech feature amount sequence f. You may learn.

...(3)

Further, when learning the neural network, the neural network as the generator may be pre-trained by using the method of the first embodiment.

The generation unit 250 inputs the arbitrary speech feature amount sequence f accepted by the input unit 10 and a natural component z given in advance to a neural network stored in the neural network storage unit 40, and outputs the neural network from the neural network. Output synthesized voice data

Is output to the output unit 90.

<Operation of speech synthesis apparatus according to second embodiment of the present invention>

Next, the operation of the speech synthesizer 200 according to the second embodiment of the present invention will be described. The speech synthesizer 200 executes a learning processing routine and a generation processing routine described below.

First, the learning processing routine will be described. When the input unit 10 receives human voice data x as learning data, the voice synthesizer 200 executes the learning processing routine shown in FIG.

In the learning processing routine according to the second embodiment, in step S102, the voice feature amount sequence f obtained in step S100, the natural component z given in advance, and the voice data x received by the input unit 10 are input. Using the input as an input, the neural network as the generator and the neural network as the discriminator perform learning according to the optimization conditions competing with each other according to the above equation (2), and the learned neural network is stored in the neural network storage unit 40. And the process ends.

In the generation processing routine of the second embodiment, as shown in FIG. 5, in step S200, an arbitrary audio feature amount sequence f accepted by the input unit 10 and a natural component z given in advance are Synthesized synthetic voice data input to the neural network stored in the neural network storage unit 40 and output from the neural network.

Is output to the output unit 90, and the process ends.

As described above, according to the speech synthesis device according to the second embodiment of the present invention, the speech feature amount sequence f, the naturalness component z, and the true speech data x for learning are input, Synthesized voice data synthesized from the voice feature quantity sequence f according to the above equation (2).

Neural network as a generator to generate the

However, a neural network as a discriminator for discriminating whether or not to follow the same distribution as the true voice data x learns according to competing optimization conditions to synthesize a more natural voice. Can learn neural networks.

Further, a more natural voice can be synthesized by synthesizing the voice by using the learned neural network as the generator.

<Outline of Third Embodiment of the Present Invention>

Next, an outline of the third embodiment of the present invention will be described.

The first and second embodiments are for performing mapping between a voice feature amount sequence and high-quality voice, and are technologies that replace existing vocoders. On the other hand, the third embodiment is a method of performing mapping between a voice synthesized once from a voice feature quantity sequence and high quality voice. Here, in order to synthesize the voice once from the voice feature quantity sequence, an existing vocoder or the first and second embodiments may be used. A conceptual diagram of the processing is shown in FIG.

Given a voice feature quantity sequence, an intermediate voice signal is first obtained by using a vocoder or a neural network as a generator learned by the method of the first or second embodiment. By inputting this intermediate voice signal into the neural network and converting it, the target voice data is obtained.

<Configuration of speech synthesizer according to third embodiment of the present invention>

Next, the configuration of the speech synthesizer according to the third embodiment of the present invention will be described. The same parts as those in the second embodiment are designated by the same reference numerals and the description thereof will be omitted.

As shown in FIG. 10, a speech synthesis apparatus 300 according to the third embodiment of the present invention stores a CPU, a RAM, a program for executing a learning processing routine and a generation processing routine described later, and various data. And a computer including the ROM. This speech synthesizer 300 functionally includes an input unit 10, a calculation unit 320, and an output unit 90, as shown in FIG.

The calculation unit 320 includes a learning unit 330, a neural network storage unit 40, an intermediate voice conversion unit 332, and a generation unit 350.

The learning unit 330, as described below, obtains the intermediate voice data x′ (intermediate voice signal or intermediate voice spectrum sequence) obtained by synthesizing voice from a voice feature amount sequence obtained by voice analysis of the voice data x. , The natural component z, and the true voice data x for learning are input, and the synthesized voice data is synthesized from the intermediate voice data x′.

A neural network as a generator for generating the

And learning is performed so as to optimize the objective function representing the distance from the true voice data x for learning.

The learning unit 330 first obtains the audio feature amount sequence f for the audio data x received by the input unit 10. The intermediate feature data x′ is obtained by inputting the voice feature amount sequence f and the naturalness component z obtained here to a neural network as a generator learned as in the second embodiment. Then, a neural network as a generator is trained so that the original true voice data x is generated for the intermediate voice data x′. Specifically, when the intermediate voice data x′ is input to the neural network, the voice data

Is output, but true voice data x and synthetic voice data to be output

The weights of the neural network may be optimized so that and are minimized with respect to a certain distance index. The distance index described here is, for example, the least square error. When the least square error is used as the distance index, the objective function L ₂ is represented by the following equation (1).

...(4)

FIG. 11 shows a conceptual diagram of the learning process of the third embodiment.

The neural network as a generator learned so as to optimize the objective function of the equation (4) is stored in the neural network storage unit 40.

The intermediate voice conversion unit 332 inputs the arbitrary voice feature amount sequence f accepted by the input unit 10 to the neural network (not shown) of the second embodiment, thereby generating the intermediate voice data x′ (intermediate voice signal or Intermediate speech spectrum sequence).

The generation unit 350 inputs the intermediate voice data x′ obtained by the intermediate voice conversion unit 332 to the neural network stored in the neural network storage unit 40 and synthesizes the synthesized voice data.

Is output to the output unit 90.

<Operation of speech synthesis device according to third embodiment of the present invention>

Next, the operation of the speech synthesizer 300 according to the third embodiment of the present invention will be described. The speech synthesizer 300 executes a learning processing routine and a generation processing routine described below.

First, the learning processing routine will be described. When the input unit 10 receives human voice data x as learning data, the voice synthesizer 300 executes a learning processing routine shown in FIG. 12.

First, in step S300, the voice data x received by the input unit 10 is voice analyzed to obtain a voice feature amount sequence f.

Next, in step S302, the speech feature amount sequence f obtained in step S300, the naturalness component z, and the speech data x received by the input unit 10 are input, and learning is performed as in the second embodiment. The intermediate speech data x′ (intermediate speech signal or intermediate speech spectrum sequence) is obtained by inputting the neural network as a generator.

In step S304, the synthetic speech data x′ obtained in step S302 and the speech data x accepted by the input unit 10 are input, and synthetic speech data synthesized from the intermediate speech data x′ according to the equation (4).

A neural network as a generator that performs learning performs learning so as to optimize the objective function, stores the learned neural network in the neural network storage unit 40, and ends the processing.

Next, the generation processing routine will be described. When the input unit 10 receives an arbitrary speech feature amount sequence f for which synthetic speech data is to be generated, the speech synthesis device 300 executes the generation processing routine shown in FIG. 13.

In step S400, the arbitrary speech feature amount sequence f accepted by the input unit 10 is input to a learned neural network (not shown) as a generator as in the second embodiment, so that the intermediate speech data x is input. Get'

In step S402, the synthetic speech data synthesized by inputting the intermediate speech data x′ obtained in step S400 to the neural network stored in the neural network storage unit 40.

Is output to the output unit 90, and the process ends.

As described above, according to the speech synthesis device according to the third embodiment of the present invention, the intermediate speech data x′ obtained by synthesizing speech from the speech feature quantity sequence and the true speech for learning are used. Data and data as input, and synthesized voice data synthesized from the intermediate voice data x′ according to the equation (4).

By learning so that the neural network as a generator for generating the objective function optimizes the objective function, a neural network capable of synthesizing more natural speech can be learned.

Further, by synthesizing the voice using the learned neural network, a more natural voice can be synthesized.

Note that the case of using the learned neural network in the same way as in the second embodiment to convert the intermediate voice data has been described as an example, but the present invention is not limited to this, and the vocoder or the first The speech feature quantity sequence may be converted into intermediate speech data by using the learned neural network as in the embodiment.

Further, in the case of using the neural network learned in the same manner as in the first or second embodiment to convert the intermediate voice data, the learning process in the present embodiment is performed and then the learning is performed. The neural network may be regarded as pre-training and the entire neural network may be optimized again.

<Outline of Fourth Embodiment of the Present Invention>

Next, the outline of the fourth embodiment of the present invention will be described.

In the third embodiment, the intermediate voice data is directly converted into a natural voice, but in the fourth embodiment, the natural voice component is added to the intermediate voice data to convert the voice into a genuine voice. is there. A conceptual diagram of the processing is shown in FIG. The naturalness component described here is a component (for example, a random number) that is given independently of the synthesized voice.

<Structure of speech synthesizer according to fourth embodiment of the present invention>

Next, the configuration of the speech synthesizer according to the fourth embodiment of the present invention will be described. The same parts as those in the third embodiment are designated by the same reference numerals and the description thereof will be omitted.

As shown in FIG. 15, a speech synthesis apparatus 400 according to the fourth embodiment of the present invention stores a CPU, a RAM, a program for executing a learning processing routine and a generation processing routine described later, and various data. And a computer including the ROM. This speech synthesizer 400 functionally includes an input unit 10, a calculation unit 420, and an output unit 90, as shown in FIG.

The calculation unit 420 includes a learning unit 430, a neural network storage unit 40, an intermediate voice conversion unit 332, and a generation unit 450.

As will be described below, the learning unit 430 synthesizes speech from a speech feature quantity sequence obtained by speech analysis of the speech data x, and obtains intermediate speech data x′ (intermediate speech signal or intermediate speech spectrum series). , A natural component z ₂ corresponding to the intermediate voice data x′, and the true voice data x for learning are input, and as a generator that generates synthesized voice data synthesized from the intermediate voice data x′. Neural network and synthetic speech data

, A discriminator that discriminates whether or not to follow the same distribution as the true voice data x, and the neural network as the generator and the neural network as the discriminator perform learning according to competing optimization conditions. To do.

The learning unit 430 first obtains the audio feature amount sequence f for the audio data x received by the input unit 10. Input the speech feature amount sequence f and the naturalness component z ₁ corresponding to the speech feature amount sequence f into a neural network as a generator learned as in the second embodiment. To obtain intermediate voice data x′. Based on the intermediate voice data x′ obtained here, the naturalness component z _2, and the true voice data x for learning, the neural voice as a generator is generated so as to generate the true voice data x to be the original. Learn the network. Here, as the audio feature quantity sequence f, a partially modified version may be used. Specifically, the fundamental frequency is one of the typical ones of the speech feature amount series, but a fundamental frequency may be randomly multiplied by a constant. The natural component z ₁ and the natural component z ₂ are random numbers generated according to a certain distribution (for example, uniform distribution).

In addition, true voice data x and synthetic voice data generated by a neural network as a generator

A neural network as a discriminator that discriminates whether or not to follow the same distribution as the true voice data x is learned based on and. The neural network as the discriminator discriminates whether the input voice data is true or synthesized, and outputs the result.

In the present embodiment, the evaluation function of the neural network as the generator and the evaluation function of the neural network as the discriminator are optimized according to the following equation (5). In the equation (5), G represents a generator and D represents a discriminator. In equation (5), the classifier maximizes the evaluation function so that the true speech and the synthesized speech can be discriminated from each other as much as possible, while the generator discriminates the synthesized speech from the true speech as much as possible. Then, the evaluation function is minimized. Optimization progresses while the discriminator and the generator compete.

...(5)

FIG. 16 shows a conceptual diagram of the learning process of the fourth embodiment.

The neural network as a generator and the neural network as a discriminator learned so as to optimize the evaluation function of the equation (5) are stored in the neural network storage unit 40.

As shown in the following equation (6), a neural network as a generator and a neural network as a discriminator are used so as to optimize an evaluation function using a discriminator that also considers intermediate speech data x′. You may learn.

...(6)

Further, when learning the neural network, the neural network as the generator may be pre-trained by using the method of the third embodiment.

The generation unit 450 inputs the intermediate voice data x′ obtained by the intermediate voice conversion unit 332 and the natural component z ₂ given in advance to the neural network stored in the neural network storage unit 40, and synthesizes the neural network. Synthesized voice data

Is output to the output unit 90.

<Operation of Speech Synthesis Device According to Fourth Embodiment of the Present Invention>

Next, the operation of the speech synthesizer 400 according to the fourth embodiment of the present invention will be described. The speech synthesizer 400 executes a learning processing routine and a generation processing routine described below.

First, the learning processing routine will be described. When the input unit 10 receives human voice data x as learning data, the voice synthesizer 400 executes the learning processing routine shown in FIG.

In the learning processing routine of the fourth embodiment, in step S304, the intermediate voice data x′ obtained in step S302, the naturalness component z _2, and the voice data x received by the input unit 10 are input, According to the above equation (5), the neural network as the generator and the neural network as the discriminator perform learning according to the optimization conditions competing with each other, and the learned neural network is stored in the neural network storage unit 40. Ends the process.

In the generation processing routine of the fourth embodiment, as shown in FIG. 13, in step S402, the intermediate voice data x′ obtained in step S400 and the naturalness component z ₂ are stored in the neural network storage unit 40. Synthesized voice data synthesized by inputting to the stored neural network

Is output to the output unit 90, and the process ends.

The generation processing routine of the fourth embodiment is the same as that of the third embodiment, so the description thereof is omitted.

As described above, according to the voice synthesizing apparatus according to the fourth embodiment of the present invention, the intermediate voice data x′ obtained by synthesizing the voice from the voice feature amount sequence, and the naturalness component z ₂ are obtained. , The true speech data x for learning are input, and the synthesized speech data synthesized from the intermediate speech data x′ according to the above equation (5).

Neural network as a generator to generate the

However, a neural network as a discriminator for discriminating whether or not to follow the same distribution as the true voice data x learns according to competing optimization conditions to synthesize a more natural voice. Can learn neural networks.

Further, a more natural voice can be synthesized by synthesizing the voice by using the learned neural network as the generator.

Note that the case of using the learned neural network in the same way as in the second embodiment to convert the intermediate voice data has been described as an example, but the present invention is not limited to this, and the vocoder or the first The speech feature quantity sequence may be converted into intermediate speech data by using the learned neural network as in the embodiment.

Further, in the case of using the neural network learned in the same manner as in the first or second embodiment to convert the intermediate voice data, the learning process in the present embodiment is performed and then the learning is performed. The neural network may be regarded as pre-training and the entire neural network may be optimized again.

<Outline of Fifth Embodiment of the Present Invention>

Next, an outline of the fifth embodiment of the present invention will be described.

As the speech feature amount sequence used in the first to fourth embodiments, for example, a speech feature amount sequence obtained by existing speech analysis can be used, but the speech feature amount sequence obtained by a neural network is used as an input. Can also This is because the first to fourth embodiments are data driven learning of the mapping of the audio feature amount sequence and the audio signal.

<Structure of speech synthesizer according to fifth embodiment of the present invention>

Next, the configuration of the speech synthesizer according to the fifth embodiment of the present invention will be described. In addition, the same reference numerals are given to the portions having the same configurations as those in the second embodiment, and the description thereof will be omitted.

As shown in FIG. 18, a speech synthesis apparatus 500 according to the fifth embodiment of the present invention stores a CPU, a RAM, and a program and various data for executing a learning processing routine and a generation processing routine described later. And a computer including the ROM. This speech synthesizer 500 functionally includes an input unit 510, a calculation unit 520, and an output unit 90, as shown in FIG.

The input unit 510 receives human voice data x as learning data. The input unit 510 also receives arbitrary voice data that is a target for generating synthetic voice data.

The calculation unit 520 includes a voice feature amount generation unit 528, a learning unit 530, a neural network storage unit 40, a voice feature amount conversion unit 532, and a generation unit 250.

The voice feature amount generation unit 528 inputs the voice data x received by the input unit 510 into Auto Encoder that is a neural network that has been learned in advance, and outputs the voice feature amount sequence f output from the Auto Encoder to the learning unit 530. To do. The neural network used here may be a Variational Auto Encoder learned in advance.

The learning unit 530 receives the voice feature amount sequence f output from the voice feature amount generation unit 528, the naturalness component z, and the true voice data x for learning, and synthesizes the voice feature amount sequence f. Neural network as a generator for generating synthesized speech data, and synthesized speech data

Is provided with a neural network as a discriminator that discriminates whether or not to follow the same distribution as the true voice data x. The neural network and the neural network may perform learning according to optimization conditions that compete with each other.

FIG. 15 shows a conceptual diagram of the learning process of the fifth embodiment.

The voice feature amount generation unit 532 inputs arbitrary voice data, which is the target of generation of the synthesized voice data received by the input unit 510, to the Auto Encoder which is a preliminarily learned neural network similarly to the voice feature amount generation unit 528. Then, the audio feature amount sequence f output from the Auto Encoder is output to the generation unit 250.

The rest of the configuration of the fifth embodiment is the same as that of the second embodiment, so the explanation is omitted.

Further, in the fifth embodiment, the case where the learning unit 530 performs the same processing as in the second embodiment has been described, but the present invention is not limited to this. For example, the learning unit 530 receives the voice feature amount sequence f output from the voice feature amount generating unit 528 and the true voice data x for learning as input, and generates the voice feature amount sequence f by the same process as in the first embodiment. You may make it learn the neural network as a container. Further, from the audio feature amount sequence f output from the audio feature amount generation unit 528, intermediate voice data x′ is obtained by the same processing as in the third embodiment, and the obtained intermediate voice data x′ and learning It is also possible to learn the neural network as the generator by inputting the true voice data x for. Further, the intermediate feature data x′ is obtained and obtained from the feature feature sequence f output from the feature feature generation unit 528 and the naturalness component z ₁ by the same processing as in the fourth embodiment. The intermediate voice data x′, the naturalness component z _2, and the true voice data x for learning are input, and a neural network as a generator or a neural network as a generator and a discriminator is learned. May be.

<Operation of Speech Synthesis Device According to Fifth Embodiment of Present Invention>

Next, the operation of the speech synthesizer 500 according to the fifth embodiment of the present invention will be described. The speech synthesizer 500 executes a learning processing routine and a generation processing routine described below. The same parts as those in the second embodiment are designated by the same reference numerals and the description thereof will be omitted.

First, the learning processing routine will be described. When the input unit 510 receives human voice data x as learning data, the voice synthesizer 500 executes a learning processing routine shown in FIG.

In step S500, the voice data x received by the input unit 510 is input to Auto Encoder, which is a neural network learned in advance, and the voice feature amount sequence f output from the Auto Encoder is output to the learning unit 530.

In step S102, the speech feature amount sequence f obtained in step S500, the naturalness component z given in advance, and the speech data x received by the input unit 510 are input, and the generator is generated according to the above equation (2). And the neural network as the discriminator perform learning in accordance with the optimization conditions competing with each other, and the learned neural network is stored in the neural network storage unit 40, and the process ends.

Next, the generation processing routine will be described. When the input unit 510 receives the voice data to be the target of generating the synthetic voice data, the voice synthesizing device 500 executes the generation processing routine shown in FIG.

In step S500, the voice data received by the input unit 510 is input to Auto Encoder, which is a neural network learned in advance, and the voice feature amount sequence f output from the Auto Encoder is output to the generation unit 250.

Note that the other operations of the fifth embodiment are similar to those of the second embodiment, and therefore description thereof will be omitted.

As described above, according to the speech synthesizer of the fifth embodiment of the present invention, the speech data x is input to the Auto Encoder which is a pre-learned neural network and the speech output from the Auto Encoder is input. A feature amount sequence f is output, and the output voice feature amount sequence f, the naturalness component z, and true voice data x for learning are input, and the feature amount sequence f is converted from the voice feature amount sequence f according to the above equation (2). Synthesized voice data

Neural network as a generator to generate the

, A discriminator for discriminating whether or not to follow the same distribution as the true voice data x, learns a neural network capable of synthesizing a more natural voice by performing learning according to optimization conditions competing with each other. it can.

Further, a more natural voice can be synthesized by synthesizing the voice by using the learned neural network as the generator.

<Experiment result 1>

To show the effectiveness of the third and fourth embodiments, an experiment was conducted using one implementation method.

Experimental data As the data for the experiment, 115 conversation sentences of one speaker in ATR Speech Data were used. Of this data, 90% was used for model training and the remaining 10% was used for testing. The sampling frequency of the audio signal is 16,000 Hz.

In the third and fourth embodiments, a voice signal or a voice spectrum sequence generated by a Vocoder or a neural network having an input/output equivalent thereto is used as an input of the generator. In this experiment, a voice signal was generated using Vocoder among these, and the voice spectrum sequence x′ obtained by performing the preprocessing described below was used as the input. LPC analysis synthesis was used as a specific analysis synthesis method. Converting the sound generated by this analysis and synthesis into a real voice like the original voice signal is the role of the generator composed of neural networks.

The above-mentioned pre-processing is the following processing. First, short-time Fourier transform (STFT) was applied to each speech signal to transform it into a complex spectrum sequence. At this time, the window width of the Fourier transform was 512 and the shift width was 128. A Blackman window was used as the window function. Next, the absolute value of the complex spectrum series was taken and converted into an amplitude spectrum series. Furthermore, a logarithmic spectrum with a base of 10 was taken for this amplitude spectrum and multiplied by 20 to convert it into a logarithmic spectrum of amplitude. Finally, a certain frame was cut out from the spectrum sequence obtained by this process and used as the input of the generator. In the experiment, the length to cut out the frame was set to 21.

Further, as the output of the generator, a logarithmic spectrum having the same amplitude as that of the input is obtained, and therefore the following processing was performed in order to return it to a voice signal. First, the logarithmic spectrum of the amplitude was divided by 20, and the value obtained there was used as a multiplier to find the power of 10 to convert into the amplitude spectrum. On the other hand, we performed phase restoration using Griffin Lim and converted it into an audio signal.

FIG. 21 shows an implementation example of the learning method of the third embodiment, and FIG. 22 shows an implementation example of the learning method of the fourth embodiment.

FIG. 23 shows an implementation example of the generation method of the third embodiment, and FIG. 24 shows an implementation example of the generation method of the fourth embodiment.

As the network structure, both the generator and the discriminator of the third and fourth embodiments have three hidden layers, the number of units in each layer is 500, and the coupling method is that of Fully Connected. 25 and 26 respectively show specific network structures of the third and fourth embodiments.

The purpose of this method is to convert the analyzed and synthesized sound into a sound close to a real voice, but in order to show the effectiveness of the proposed framework, the following four synthesized sounds were assumed.

1.Volume change:n The original sound is halved.
2.Pre-emphasis: High-pitched sound of the original sound
3.LPC: LPC analysis and synthesis sound
4.LPC+pulse: Sound generated by combining the LPC obtained by the LPC analysis and the pulse signal generated at regular intervals (every 128 samples).

FIG. 27 shows an example of the waveform of the audio signal which is the source of the input/output.

FIG. 28 shows the experimental results of Volume change. Focusing on the magnitude of the amplitude of the waveform data of the audio signal, the synthesized sound is half of the original sound. However, using the methods of the third and fourth embodiments, the original sound is It can be seen that the amplitude equivalent to is reproduced.

The experimental results of Pre-emphasis are shown in FIG. Focusing on the speech spectrum series, the synthesized sound has a smaller value in the low frequency region than the original sound. However, when the methods of the third and fourth embodiments are used, the synthesized sound is equivalent to the original sound. You can see that it has returned to the size. Further, if attention is paid to the amplitude of the waveform data of the audio signal, the synthetic sound is generally smaller than the original sound. However, in any case, the methods of the third and fourth embodiments are used. It can be seen that the same amplitude as the original sound can be reproduced.

FIG. 30 shows the experimental result of LPC. Focusing on the speech spectrum sequence, the synthesized speech has a value in the lowest frequency region (0) and spreads in the high frequency region as compared with the original speech. By using the method of the fourth embodiment, it can be seen that the sound has returned to a shape equivalent to the original sound. Further, if attention is paid to the amplitude of the waveform data of the audio signal, the synthesized sound is larger than the original sound as a whole. However, by using the methods of the third and fourth embodiments, in any case. It can be seen that the same amplitude as the original sound can be reproduced.

FIG. 31 shows the experimental result of LPC+pulse. Focusing on the speech spectrum sequence, the synthesized speech has a feature that it is a striped pattern with equal intervals as compared with the original speech. However, when the methods of the third and fourth embodiments are used, in either case It can be seen that the shape has returned to a shape close to the original sound compared to the synthetic sound.

<Experiment result 2>
Next, experimental results of the first and second embodiments will be shown.

Experimental Data First, as the data set, the same data set as the experiment related to the third and fourth embodiments described above was used. In the first and second embodiments, the speech feature quantity sequence is used as the input, but in this experiment, the speech feature quantity obtained by the LPC analysis was used. Specifically, pitch and LPC were used. The pitch is a one-dimensional feature amount per frame, and the LPC is a 26-dimensional feature amount because the order at the time of LPC analysis is 25. Therefore, when the two are combined, the feature quantity is 27 dimensions per dimension. As the output, the logarithmic spectrum of the amplitude was used. Also, in this experiment, the frame length of the data to be actually processed was set to 1. Ultimately, the goal is to obtain a speech signal, which requires the reconstruction of the speech signal from the logarithmic spectrum of the amplitude obtained as output. As the method, the same method as the method described in the experimental section regarding the third and fourth embodiments was used.

32 and 33 show implementation examples of the network structure according to the first and second embodiments. As the network structure, in both the first and second embodiments, three hidden layers, the number of units in each layer is 500, and the method of connecting the layers is Fully Connected.

FIG. 34 shows the result of voice restoration by the methods of the first and second embodiments. From now on, although the voice feature quantity used for input is 27 dimensions, by using the networks of the first and second embodiments, the harmonic structure of features similar to the original sound ( It can be seen that the striped pattern of the spectrum) has been reproduced.

The present invention is not limited to the above-described embodiments, and various modifications and applications can be made without departing from the spirit of the present invention.

For example, in the above-described embodiment, the speech synthesizer includes the learning unit that performs the learning of the neural network and the generation unit that synthesizes the speech. However, the present invention is not limited to this, and the learning unit is not limited thereto. The speech synthesis learning device including the above and the speech synthesis device including the generation unit may be separately configured.

Moreover, CNN, RNN, etc. can also be used for the neural network in the above-mentioned embodiment.

10, 510 input unit 20, 220, 320, 420, 520 arithmetic unit 30, 230, 330, 430, 530 learning unit 40 neural network storage unit 50, 250, 350, 450 generation unit 90 output unit 100, 200, 300, 400, 500 voice synthesizer 332 intermediate voice converter 528, 532 voice feature quantity generator

Claims (7)

A speech synthesis learning device for learning a neural network for synthesizing speech from arbitrary speech data or a speech feature quantity sequence,
Accepts the input voice data or voice feature amount series, and true voice data for learning,
A neural network as a first generator that is pre-learned to generate intermediate voice data from the voice data or the voice feature amount sequence and the true voice data for learning;
A neural network as a second generator that is learned to generate synthetic speech data from the intermediate speech data and the true speech data for learning;
The voice data or the voice feature quantity sequence is used as an input to a neural network as the first generator to obtain the intermediate voice data,
The obtained intermediate voice data is used as an input to a neural network as the second generator to generate the synthetic voice data,
To optimize the objective function representing the distance between the generated synthetic speech data and the true speech data for learning, or a neural network as the second generator, and the generated synthetic speech data Learns the neural network as the second generator so that it follows an optimization condition in which the neural network as a discriminator for determining whether or not follows the same distribution as the true speech data for learning. Including a learning section to
The neural network as the first generator is
According to the optimization of the objective function representing the distance between the intermediate voice data and the true voice data for learning, or the neural network as the first generator, and the intermediate voice data is the true voice for learning. The neural network as a discriminator for determining whether or not to follow the same distribution as the voice data of the
Speech synthesis learning device. The neural network as the first generator obtains the intermediate voice data from the voice data or the voice feature amount sequence, the natural component given independently of the voice data sequence, and the true voice data for learning. The speech synthesis learning device according to claim 1, which has been learned in advance. The intermediate speech data and a naturalness component given independently of the intermediate speech data are generated as the input to the neural network as the second generator to generate the synthetic speech data,
The learning unit optimizes an objective function representing a distance between the generated synthetic voice data and the true voice data for learning, or a neural network as the second generator, A neural network as the second generator so that the synthetic speech data and a neural network as a discriminator for discriminating whether or not the true speech data for learning follow the same distribution are subject to optimization conditions competing with each other. The speech synthesis learning device according to claim 1, which learns a network. The neural network as the first generator discriminates whether or not the neural network as the first generator and the intermediate voice data and the true voice data for learning follow the same distribution. The speech synthesis learning device according to any one of claims 1 to 3, wherein the neural network serving as a training device is preliminarily trained according to competing optimization conditions. The neural network as the second generator discriminates whether or not the neural network as the second generator and the intermediate voice data and the true voice data for learning follow the same distribution. The speech synthesis learning device according to any one of claims 1 to 3, wherein the neural network as a training device learns according to optimization conditions competing with each other. The learning of the neural network as the first generator and the learning of the neural network as the second generator in the learning unit are pre-learned, and the neural network as the first generator and the second The speech synthesis learning device according to any one of claims 1 to 5, which optimizes an entire neural network including a neural network as a generator. The neural network as the first generator is pre-learned, and the entire neural network including the neural network as the first generator and the neural network as the second generator is optimized. The speech synthesis learning device according to any one of claims 1 to 5.