JP7228998B2 - speech synthesizer and program - Google Patents

Description

Application of Article 30, Paragraph 2 of the Patent Act Acoustical Society of Japan 2018 Autumn Research Presentation Meeting CD-ROM containing lecture proceedings Issue date August 29, 2018 Meeting name Acoustical Society of Japan 2018 Fall Research Presentation Date September 12, 2018

The present invention relates to a speech synthesizer and program.

In recent years, due to advances in speech synthesis technology using statistical models, techniques for synthesizing speech from text are known. For example, techniques have been developed for learning features such as a speaker's voice using a Deep Neural Network (DNN) and performing speech synthesis from text (see, for example, Non-Patent Documents 1, 2, and 3). ). A technique has also been developed for estimating a mel-spectrogram from a character string written in English and generating a speech waveform from this mel-spectrogram (see Non-Patent Document 4).

A conventional statistical speech synthesizer generates speech by a statistical model using a phoneme label file in order to calculate acoustic features and synthesize speech. This phoneme label file contains labels such as phonemes, durations of phonemes, parts of speech, etc., and labels are given from acoustic features of speech.

Kiyoshi Kurihara et al, "Automatic generation of audio descriptions for sports programs", International Broadcasting Convention [IBC 2017], 2017 Kiyoshi Kurihara, Nobumasa Kiyoyama, Atsushi Imai, Toru Miyakogi, "Study of DNN speech synthesis method that can control speaker's characteristics and emotional expression", The Institute of Electronics, Information and Communication Engineers General Conference, 2017, D-14-10, p .150 Hojo, Ijima, Miyazaki, "Study of DNN speech synthesis using speaker code", Proceedings of Acoustical Society of Japan, September 2015, pp.215-218 Shen et al., [online], February 2018, "Natural TTS Synthesis by Conditioning WaveNet on Mel Spectrogram Predictions", arXiv:1712.05884, [searched July 11, 2018], Internet<URL: https://arxiv .org/pdf/1712.05884.pdf>

As mentioned above, a statistical speech synthesizer uses a phoneme label file. It was sometimes difficult to recognize correctly, and sometimes the labels described above were not given correctly. In addition, when it is difficult to correctly determine the boundaries of phonemes, manual correction is required to generate a correct phoneme label file, which poses problems of human cost and time cost. Furthermore, in the case of Japanese, a large amount of learning data is required in order to cover sentences mixed with kana and kanji, which are made up of various combinations of kanji, hiragana, and katakana. There was also the problem that it could not be performed correctly. Therefore, it is difficult to directly apply the technique disclosed in Non-Patent Document 4 to Japanese sentences containing kana and kanji characters.

The present invention has been made in consideration of such circumstances, and provides a speech synthesizer and program capable of synthesizing high-quality speech at low cost.

According to one aspect of the present invention, a sentence representing the content of a Japanese utterance is formed by a character string using characters or character strings representing how to read the content of the utterance, a prosody symbol representing a prosody, and an utterance style symbol representing a characteristic given to the utterance. inputting the described first text data to a first acoustic feature quantity generation model for generating an acoustic feature quantity from the first text data, and estimating an acoustic feature quantity of speech corresponding to the speech content; Estimation processing , or a second acoustic feature for generating an acoustic feature amount from the second text data described by the character or character string representing the reading and the character string using the prosody symbol from the second text data A second estimation process for estimating the acoustic feature value of the speech corresponding to the utterance content by inputting it into a quantity generation model, or described by a character string or a character string representing the reading and a character string using the utterance style symbol A third estimation process of inputting the third text data to a third acoustic feature value generation model that generates an acoustic feature value from the third text data, and estimating the acoustic feature value of the speech corresponding to the utterance content. and the acoustic feature value estimated by the acoustic feature value estimating unit by either the first estimation process, the second estimation process, or the third estimation process. and a vocoder unit that estimates a speech waveform using a deep neural An encoder and a decoder using a network are provided, and the encoder uses a recurrent neural network to consider character strings before and after the utterance content in the text for the utterance content indicated by the text data. and the decoder uses a recursive neural network to generate an acoustic feature of speech corresponding to the utterance content indicated by the text data based on the feature amount generated by the encoder and the acoustic feature amount generated in the past. A speech synthesizer characterized by generating a quantity.

One aspect of the present invention is the speech synthesis device described above , wherein the characters representing the reading are katakana, hiragana, the alphabet, or phonetic symbols, and the first acoustic feature value generation model, the second and the third acoustic feature generation model further include an attention network using a deep neural network, wherein the attention network weights the features output by the encoder and weighting the feature amount with the generated weight and inputting it to the decoder, and the decoder uses a recurrent neural network to combine the feature amount input from the attention network with the past and generating an acoustic feature amount of speech corresponding to the utterance content indicated by the text data.

One aspect of the present invention is the above-described speech synthesizer, wherein the prosodic symbols include a symbol that designates an accent position, a symbol that designates a break of a phrase or a phrase, a symbol that designates intonation at the end of a sentence, and a symbol that designates an intonation at the end of a sentence. and a symbol that specifies the size .

One aspect of the present invention is the speech synthesizer described above, wherein the feature given to the speech is an emotion, a speech style, or a speaker.

One aspect of the present invention is the above-described speech synthesizer, wherein the utterance to be given the feature is the entire utterance of one or more sentences to which the utterance style symbol is added at a predetermined position, surrounded by the utterance style symbol. or the utterance of one or more phrases surrounded by the utterance style symbols.

One aspect of the present invention is a program for causing a computer to function as any of the speech synthesizers described above.

According to the present invention, high-quality speech can be synthesized at low cost.

1 is a diagram showing an outline of a speech synthesizer according to a first embodiment of the present invention and a conventional speech synthesizer; FIG. It is a functional block diagram which shows the structural example of the speech synthesizer by the same embodiment. It is a figure which shows the prosody mark used for the intermediate language by the same embodiment. FIG. 4 is a flowchart showing learning processing of the speech synthesizer according to the same embodiment; FIG. 4 is a flowchart showing speech synthesis processing of the speech synthesizer according to the same embodiment; It is a figure which shows the acoustic feature-value generation model and learning algorithm by the same embodiment. It is a figure which shows the example of the encoder by the same embodiment. It is a figure which shows the example of the decoder by the same embodiment. It is a figure which shows the speech-synthesis algorithm using the acoustic feature-value generation model by the same embodiment. It is a figure which shows the result of the evaluation experiment by the same embodiment. FIG. 11 is a functional block diagram showing a configuration example of a speech synthesizer according to a second embodiment; FIG. It is a figure which shows the outline|summary of the speech synthesizing process of the speech synthesizer by the same embodiment. It is a figure which shows the acoustic feature-value generation model and learning algorithm by the same embodiment. It is a figure which shows the speech-synthesis algorithm using the acoustic feature-value generation model by the same embodiment. It is a figure which shows the example of the encoder by the same embodiment. It is a figure which shows the result of the evaluation experiment by the same embodiment. It is a figure which shows the result of the evaluation experiment by the same embodiment.

Hereinafter, embodiments of the present invention will be described in detail with reference to the drawings.

[First embodiment]
FIG. 1 is a diagram showing an overview of a speech synthesizer 1 according to this embodiment and a speech synthesizer 9 according to the prior art. In the speech synthesizer 9 according to the prior art, the first language processing unit 91 estimates the kana (for example, katakana) notation and prosodic symbols of Japanese sentences mixed with kana and kanji, and the second language processing unit 92 uses the estimation result as Add labels such as phoneme labels and phoneme lengths to generate a phoneme label file. The acoustic feature amount estimation unit 93 uses the manually modified phoneme label file to estimate the frequency waveform as an acoustic feature amount by, for example, a DNN (Deep Neural Network), and the vocoder unit 94 estimates the The speech waveform is estimated from the obtained frequency waveform.

On the other hand, the speech synthesizer 1 of this embodiment includes a language processing unit 41 , an acoustic feature amount estimation unit 42 and a vocoder unit 43 . The language processing unit 41 converts a Japanese sentence containing a mixture of kana and kanji into an intermediate language using kana and prosodic symbols. In this embodiment, katakana is used as kana, but hiragana, the alphabet, or phonetic symbols may be used. It is also possible to use symbols representing phonemes instead of kana. The prosody symbols used in the intermediate language are letters representing prosody. The acoustic feature amount estimating unit 42 uses text data describing an intermediate language as input data to estimate an acoustic feature amount by DNN. A mel spectrogram, for example, is used as the acoustic feature quantity. The vocoder unit 43 uses a DNN such as WaveNet to estimate a speech waveform from acoustic features. WaveNet is described, for example, in reference 1 "A. van den Oord, S. Dieleman, H. Zen, K. Simonyan, O. Vinyals, A. Graves, N. Kalchbrenner, A. Senior and K. Kavukcuoglu," WaveNet: A Generative Model for Raw Audio, ”arXiv:1609.03499, 2016”.

As described above, the speech synthesizer 1 according to the present embodiment does not require a full context label that defines phonemes and the positions of the phonemes in detail. directly generate acoustic features. Therefore, it is easy to create data used for learning of DNN that generates acoustic features, and for example, it becomes easy to utilize existing voice data as learning data. As a result, it is possible to improve the accuracy of speech synthesis by performing learning using a large amount of data while reducing human cost and time cost.

FIG. 2 is a functional block diagram showing a configuration example of the speech synthesizing device 1 according to this embodiment, in which only functional blocks related to this embodiment are extracted. The speech synthesizer 1 includes a storage unit 20 , a learning unit 30 and a speech synthesis unit 40 .

The storage unit 20 stores an acoustic feature generation model 20-1 and a speech waveform generation model 20-2. The acoustic feature value generation model 20-1 is a DNN that inputs text data and outputs data representing acoustic feature values. The speech waveform generation model 20-2 is a DNN that inputs acoustic feature data and outputs a speech waveform.

The learning unit 30 updates the acoustic feature value generation model 20-1 stored in the storage unit 20 using the learning data. The learning data is a set of learning voice data representing a voice waveform of an utterance and learning text data describing the contents of the utterance in a mixture of kana and kanji. The learning unit 30 includes a correct acoustic feature value calculation unit 31 and a model update unit 32 .

The correct acoustic feature amount calculation unit 31 calculates an acoustic feature amount from the speech waveform of the learning speech data included in the learning data. The model update unit 32 updates the acoustic feature amount calculated from the learning speech data by the correct acoustic feature amount calculation unit 31 and the acoustic feature amount estimated by the speech synthesis unit 40 based on the learning text data included in the learning data. Based on the difference, the acoustic feature value generation model 20-1 stored in the storage unit 20 is updated.

The speech synthesizing unit 40 receives intermediate language text data described in katakana and prosodic symbols, executes the acoustic feature quantity generation model 20-1, and obtains data representing the acoustic feature quantity of the speech content of the utterance. The speech synthesizing unit 40 includes a language processing unit 41 , an acoustic feature quantity estimating unit 42 and a vocoder unit 43 .

The language processing unit 41 converts the text data of sentences mixed with kana and kanji into an intermediate language using katakana and prosody symbols. This conversion can be done by existing techniques such as morphological analysis. The language processing unit 41 outputs text data representing the intermediate language to the acoustic feature amount estimation unit 42 . The acoustic feature value estimating unit 42 inputs text data in the intermediate language input from the language processing unit 41 to the acoustic feature value generation model 20-1 stored in the storage unit 20, so that the text data described in the intermediate language Estimate the acoustic features of the utterance content. The vocoder unit 43 receives as input the acoustic feature quantity estimated by the acoustic feature quantity estimation unit 42 and generates a speech waveform using the speech waveform generation model 20 - 2 stored in the storage unit 20 .

During learning of the acoustic feature value generation model 20-1, the language processing unit 41 and the acoustic feature value estimation unit 42 operate as the learning unit 30. FIG. The language processing unit 41 converts text data for learning included in the learning data into an intermediate language, and the acoustic feature value estimation unit 42 converts the text data representing the converted intermediate language to the acoustic feature value generation model 20-1. The acoustic feature amount is estimated by inputting, and the estimation result is output to the model updating unit 32 .

Note that the speech synthesizer 1 can be realized by one or more computer devices. When the speech synthesizer 1 is implemented by a plurality of computer devices, it is possible to arbitrarily decide which functional unit is implemented by which computer device. For example, the storage unit 20 and the learning unit 30 may be implemented by one or more server computers, and the speech synthesis unit 40 may be implemented by a client terminal. Also, the same functional unit may be realized by a plurality of computer devices.

FIG. 3 is a diagram showing prosody symbols used in the intermediate language of this embodiment. The prosody symbols shown in Fig. 3 are taken from Reference 2, "Speech Input/Output Method Standardization Committee, JEITA Standard IT-4006 Symbols for Japanese Text Speech Synthesis, Japan Electronics and Information Technology Industries Association, 2010, p.4-10. ” is information obtained by modifying the prosodic symbols described in The prosody information includes types such as designation of accent position, designation of phrase/phrase break, designation of intonation at the end of sentence, designation of pause , and the like. The prosody mark "'" is used to specify the accent position, and indicates that the mora immediately preceding the prosody mark has an accent kernel. For specifying the accent position, the prosody symbol "_" representing the accent descending position may be used. A prosodic symbol "/" representing an accent phrase delimiter and a prosodic symbol "#" representing a phrase delimiter are used to specify phrase/phrase delimiters. The prosody symbol “=” representing the end of a normal sentence and the prosody symbol “?” representing the end of an interrogative sentence are used to designate the end-of-sentence intonation. A prosody symbol “$%” representing a pause is used to specify the pause. Note that it is not necessary to use phrase/phrase delimiter designations.

For convenience, the above symbols are assigned to these prosody symbols. It is possible to have the same function as the above by replacing the prosody symbols, prosody symbols that indicate the end of sentences, prosody symbols that indicate the end of interrogative sentences, and prosody symbols that indicate pauses with other symbols. .

FIG. 4 is a flowchart showing learning processing of the speech synthesizer 1. As shown in FIG.
First, in step S110, the speech synthesizer 1 inputs learning data. In step S120, the correct acoustic feature amount calculation unit 31 selects one unselected learning speech data included in the learning data, and calculates an acoustic feature amount from the speech waveform indicated by the selected learning speech data. In step S130, the language processing unit 41 acquires learning text data in which the utterance content of the selected learning voice data is described, performs morphological analysis and the like, and converts sentences representing the utterance content into reading kana. and a prosody symbol. The user may modify the intermediate language as needed. In step S140, the acoustic feature value estimation unit 42 stores the intermediate language data, which is text data representing the intermediate language generated by the language processing unit 41 in step S130, in the acoustic feature value generation model 20-1 read from the storage unit 20. Input and estimate acoustic features.

In step S150, the model updating unit 32 performs the The acoustic feature value generation model 20-1 stored in the storage unit 20 is updated. Specifically, the model updating unit 32 calculates this error by MSE (least squares method), and uses ADAM of the stochastic gradient descent method so that the calculated difference becomes small. -1 to update the weight of the input to each unit (node). MSE, for example, Reference 3 "GitHub, Inc, [online], " Spectrogram Feature prediction network", [searched August 24, 2018], Internet <URL: https://github.com/Rayhane-mamah/ Tacotron-2/wiki/Spectrogram-Feature-prediction-network#training>”. In addition, ADAM is described in, for example, reference 4 "Diederik P. Kingma, Jimmy Ba, [online], 2017," ADAM: A Method for Stochastic Optimization ", arXiv:1412.6980v9, [searched on August 24, 2018] , Internet <URL: https://arxiv.org/pdf/1412.6980.pdf>”.

In step S160, the learning unit 30 determines whether or not the model update has ended. For example, when the mean square error between the acoustic feature quantity calculated by the correct acoustic feature quantity calculating unit 31 and the acoustic feature quantity estimated by the acoustic feature quantity estimating unit 42 is equal to or less than a predetermined value, it is determined that the model update is completed. . When the learning unit 30 determines that the model update has not ended (step S160: NO), the processing from step S120 is repeated. Then, when the learning unit 30 determines that the model update is finished (step S160: YES), the learning process is finished.

FIG. 5 is a flowchart showing the speech synthesizing process of the speech synthesizer 1. As shown in FIG.
First, in step S210, the speech synthesizing unit 40 inputs text data of sentences containing kana and kanji representing the content of the speech. A sentence representing the contents of the utterance may be one sentence or a plurality of sentences. In step S220, the language processing unit 41 performs morphological analysis on the input text data, and converts sentences representing the contents of the utterance into an intermediate language described by character strings using reading kana and prosodic symbols. The user may modify the intermediate language as needed.

In step S230, the acoustic feature value estimation unit 42 inputs the intermediate language data, which is text data representing the intermediate language generated in step S220, to the acoustic feature value generation model 20-1 read from the storage unit 20, and Estimate features. In step S240, the vocoder unit 43 inputs the acoustic features generated in step S230 to the speech waveform generation model 20-2 read from the storage unit 20, and estimates the speech waveform. The vocoder unit 43 outputs the estimated voice waveform as voice data or from a voice output unit (not shown) such as a speaker.

FIG. 6 is a diagram showing an acoustic feature value generation model and a learning algorithm used by the speech synthesizer 1. As shown in FIG. First, the acoustic feature value generation model 60 will be described. The acoustic feature quantity generation model 60 shown in FIG. 6 is an example of the acoustic feature quantity generation model 20-1, and is a DNN to which the technology disclosed in Non-Patent Document 4 is applied. The acoustic feature quantity generation model 60 has an encoder 61 and a decoder 65 . FIG. 7 is a diagram showing an example of the encoder 61, and FIG. 8 is a diagram showing an example of the decoder 65. As shown in FIG. Note that the attention network 651 of the decoder 65 is shown in FIG. The encoder 61 and decoder 65 will be described with reference to FIGS. 6 to 8. FIG.

The encoder 61 uses a CNN (Convolutional Neural Network) and an RNN (Recurrent Neural Network) to convert the utterance content in the sentence indicated by the input intermediate language text data to the text data. It is possible to generate a feature amount of a character string in consideration of the context before and after the utterance content in the sentence. The decoder 65 uses the RNN to generate, based on the feature amount generated by the encoder 61 and the acoustic feature amount generated in the past, an acoustic feature amount for predicting speech corresponding to the utterance content indicated by the input text data for one frame. generated one by one.

The encoder 61 is composed of a character string conversion process 611 , a convolutional network 612 and a bidirectional LSTM network 613 . Character string conversion processing 611 converts each character used in the description of the intermediate language into a numerical value, and converts the intermediate language into a vector representation.

The convolutional network 612 is a neural network in which multiple (eg, three) convolutional layers are connected. Each convolution layer performs convolution processing on the vector representation of the intermediate language using a plurality of filters having a size corresponding to a predetermined number of characters, and further performs batch normalization and ReLU (Rectified Linear Units) activation. This models the context of the utterance content. For example, the filter size of the three convolution layers is [5, 0, 0] and the number of filters is 512. The output of the convolutional network 612 is input to a bi-directional LSTM network 613 in order to generate character string features for input to the decoder 65 . Bidirectional LSTM network 613 is a single bidirectional LSTM of 512 units (256 units in each direction). The bi-directional LSTM network 613 makes it possible to generate character string feature amounts that take into consideration the contexts before and after the sentences described in the input text data. LSTM is one of RNNs (Recurrent Neural Networks).

Decoder 65 is an autoregressive RNN. The decoder 65 comprises an attention network 651, a preprocessing network 652, an LSTM network 653, a first linear transformation process 654, a post-processing network 655, an addition process 656, and a second linear transformation process 657. .

The attention network 651 is a network obtained by adding an attention function to the autoregressive RNN, and outputs a fixed-length context vector summarizing the entire output from the encoder 61 for each frame. Attention network 651 inputs the output (encoder output) from bidirectional LSTM network 613 . For each frame, the weights in extracting data from the encoder output to generate the summary are different depending on the data position in the encoder output. The attention network 651 uses the data extracted from the encoder output to add features using the context vector generated at the timing of the previous decoding, and converts the context vector (attention network output) to be the output of the current frame. Generate.

The preprocessing network 652 inputs the data output by the first linear transformation process 654 at the previous time step. The preprocessing network 652 is a neural network containing multiple (eg, two) fully connected layers of 256 hidden ReLU units each. A layer consisting of ReLU units outputs zero if the value of each unit is less than zero, and outputs the value as is if it is greater than zero. The LSTM network 653 is a neural network in which multiple (for example, two layers) one-way LSTMs having 1024 units are combined, and the data obtained by combining the output from the preprocessing network 652 and the output from the attention network 651 is input. do. Since the acoustic features of a frame are affected by the acoustic features of the previous frame, by combining the features of the current frame output from the attention network 651 with the output from the preprocessing network 652, A feature based on the acoustic feature amount of the frame is added. (See Non-Patent Document 4 for details.)

A first linear transformation process 654 linearly transforms the data output from the LSTM network 653 to generate a context vector, which is mel-spectrogram data for one frame. First linear transformation process 654 outputs the generated context vector to pre-processing network 652 , post-processing network 655 and summation process 656 .

The post-processing network 655 is a neural network that combines multiple layers (eg, 5 layers) of convolutional networks. For example, a 5-layer convolutional network has a filter size of [5,0,0] and 1024 filters. Each convolutional network performs convolution processing and batch normalization and tanh activations except for the last layer. The output from post-processing network 655 is used to improve the overall quality after wavelength conversion. Addition operation 656 adds the context vector generated by first linear transformation operation 654 and the output from post-processing network 655 .

In parallel with spectrogram frame prediction above, a second linear transform operation 657 projects the concatenation of the output of the LSTM network 653 and the attention context to a scalar followed by sigmoidal activation to determine if the output sequence is complete. Outputs the Stop Token used for

Next, the learning algorithm will be explained. In step S120 of the learning process shown in FIG. 4, the correct acoustic feature value calculation unit 31 applies ABS (absolute value calculation processing), and further mel filter bank processing is performed to obtain MFCCs (Mel-Frequency Cepstrum Coefficients). The correct acoustic feature amount calculator 31 calculates the mel spectrogram A2 from the MFCC as an acoustic feature amount.

On the other hand, in step S140, the acoustic feature value estimation unit 42 inputs learning intermediate language data B1, which is intermediate language data generated from learning text data, to the acoustic feature value generation model 60, and estimates a mel-spectrogram B2. get as a result. In step S150, the model update unit 32 calculates the difference between the mel-spectrogram A2 calculated by the correct acoustic feature quantity calculation unit 31 and the mel-spectrogram B2 estimated by the acoustic feature quantity generation model 60 as an error. The model updating unit 32 updates the acoustic feature quantity generation model 60 based on the calculated error.

The learning unit 30 uses a plurality of learning data so that the difference between the mel-spectrogram calculated from the learning speech data and the mel-spectrogram estimated from the learning intermediate language data by the acoustic feature value generation model 60 becomes small. The acoustic feature value generation model 60 is updated.

FIG. 9 is a diagram showing a speech synthesis algorithm using the acoustic feature value generation model 60. As shown in FIG. In step S230 of FIG. 5, the acoustic feature value estimation unit 42 inputs the intermediate language data C1 generated based on the text data including kana and kanji characters to the trained acoustic feature value generation model 60, and calculates the acoustic feature value for each frame. A mel-spectrogram C2, which is a quantity, is generated and output to the vocoder unit 43. FIG. In step S240, the vocoder unit 43 inputs the frame-by-frame mel-spectrogram C2 to the speech waveform generation model 20-2 stored in the storage unit 20, and inversely transforms it into a time domain waveform to generate a speech waveform C3. WaveNet using a multi-layered convolutional network, for example, is used for the speech waveform generation model 20-2. Note that this processing may be implemented using a vocoder section of a type other than the above.

Next, the results of evaluation experiments on the accuracy of mel-spectrogram estimation by the speech synthesizer 1 of this embodiment will be described. A speech corpus of 12,518 sentences (18 hours) uttered by one female narrator was used for the evaluation experiment. Audio data is PCM (pulse code modulation) with a sampling frequency of 22050 [Hz] and quantization of 16 [bits]. 12,452 sentences from the speech corpus were used for learning the acoustic feature value generation model, and 10 sentences randomly extracted from the remaining data were used for the evaluation experiment. The number of times of learning is 535,000.

Four types of 10 sentences were used for the voice stimuli to the subjects. These four types are synthesized speech generated by the speech synthesizer 1 using intermediate language data described by kana and prosody symbols as input (this embodiment), and speech synthesized by analyzing and synthesizing the original speech by conventional technology (analysis synthesis). , synthesized speech (kana only) generated by the speech synthesizer 1 using only kana as input data, and original speech.

The subjects were six speech research professionals. Each subject listened to the speech stimuli at a volume that was easy for each subject to hear through headphones and rated them. Subjects rated the overall sound quality on a 5-point scale for randomly presented speech stimuli. A mean opinion score (MOS) was obtained from the evaluation results of all subjects.

FIG. 10 is a diagram showing the results of evaluation experiments. FIG. 10 shows MOS values and 95% confidence intervals. The speech synthesized by the speech synthesizer 1 of this embodiment is inferior to the original speech, but has a quality comparable to that of the analysis synthesis, and was evaluated higher than using only kana as input data. This is thought to be due to the effective functioning of prosody symbols.

According to the speech synthesizer 1 of this embodiment, it is possible to generate acoustic features directly from intermediate language text data described using kana and prosody symbols, and to learn a model used for the generation. In this embodiment, input expressions suitable for end-to-end speech synthesis can be obtained by limiting the types of character strings used for input while ensuring the diversity and accuracy of Japanese speech expressions. Since Japanese kanji characters have multiple readings, the character strings do not necessarily match the speech. It is possible to synthesize natural voices while doing so, and it is possible to control the accent position and pose position.

In the above-described embodiment, the language processing unit 41 generates the intermediate language data in which sentences representing the content of the utterance are described by a character string using the kana of the content of the utterance and prosody symbols representing the prosody. Intermediate language data may be generated manually. In this case, the speech synthesizer 1 does not have to include the language processing section 41 .

Note that the notation method of the intermediate language used for Japanese speech synthesis in this embodiment is not limited to the speech synthesis method of the encoder/decoder model described in Non-Patent Document 4, and can also be applied to other encoder/decoder models. is. For example, Reference 5 "Wei Ping et al., [online], February 2018, "Deep Voice 3: Scaling Text-to-Speech with Convolutional Sequence Learning", arXiv:1710.07654, Internet <URL: https:// arxiv.org/pdf/1710.07654.pdf>”.

Since the speech synthesizer 1 of the present embodiment does not require full context labels that specify phonemes, positions of phonemes, etc. in detail, it is easy to create training data. Therefore, it becomes easier to utilize existing speech data as learning data. In order to obtain high-quality synthesized speech with the conventional method, it was necessary to perform complicated tasks such as manually assigning phoneme boundaries to the training data. determines the boundaries for the yomigana and prosodic symbols. Therefore, the cost per phoneme is reduced to about ⅓ compared to the case of using conventional HTS-compliant full-context labels. Furthermore, since the work time can be greatly reduced, a large amount of learning data can be created to improve the accuracy of the acoustic feature value generation model.

In addition, by utilizing the existing notation, it is easy to connect with the existing front end, and the use of the existing system becomes easy. In addition, even if the speech synthesizer 1 does not have phoneme boundaries as data in advance, it does not perform forced alignment using HMMs (Hidden Markov Models) or the like, and whether alignment is performed only from the intermediate language. You can learn phonemes like

[Second embodiment]
In order to achieve broadcast-quality speech synthesis that meets the intentions of program production, it is important to control the utterance style according to the performance requirements of the program. For example, different speaking styles are required depending on programs such as news, live sports, and documentaries. In this embodiment, the characteristics given to the entire utterance can be controlled by utterance style symbols such as tags represented by character strings. The features given to the entire utterance are, for example, utterance style (play-by-play, news style), emotion (sad, happy, etc.), and speaker. The following description focuses on differences from the first embodiment.

FIG. 11 is a functional block diagram showing a configuration example of the speech synthesizing device 1a according to this embodiment, in which only functional blocks related to this embodiment are extracted. In FIG. 11, the same parts as those of the speech synthesizing apparatus 1 according to the first embodiment shown in FIG. The speech synthesizer 1a includes a storage unit 20, a learning unit 30, and a speech synthesis unit 40a.

The speech synthesis unit 40a differs from the speech synthesis unit 40 of the first embodiment in that it includes a language processing unit 41a instead of the language processing unit 41. FIG. Like the language processing unit 41, the language processing unit 41a converts the text data of sentences mixed with kana and kanji into an intermediate language using katakana and prosodic symbols. Furthermore, the language processing unit 41a adds symbols representing features given to the entire utterance to the intermediate language using katakana and prosody symbols. In the following, symbols representing features given to the entire utterance are referred to as "utterance style symbols". The utterance style symbols use characters or character strings that are different from kana (characters representing readings) and also different from characters or character strings representing prosody symbols.

Note that the speech synthesizer 1a can be realized by one or more computer devices. When the speech synthesizing device 1a is realized by a plurality of computer devices, it is possible to arbitrarily decide which functional unit is to be realized by which computer device. For example, the speech synthesizing unit 40a may be realized by a client terminal, and the storage unit 20 and the learning unit 30 may be realized by one or more server computers. Alternatively, the language processing unit 41a may be implemented by the client terminal, and the other functional units may be implemented by the server computer. Also, the same functional unit may be realized by a plurality of computer devices. The speech synthesizer 1a may also include a display unit and an input unit (not shown).

FIG. 12 is a diagram showing the flow of speech synthesizing processing by the speech synthesizing device 1a. The description will be continued below with reference to FIG. The text D1 is text data of sentences mixed with kana and kanji representing the content of the speech, and is input to the speech synthesizing unit 40a. The language processing unit 41a obtains the text D2 by, for example, morphologically analyzing the text D1. The text D2 is an intermediate language used in the first embodiment, and is a character string using reading kana and prosody symbols. Text D2 may be corrected manually. Subsequently, the language processing unit 41a adds an utterance style symbol to the text D2 to obtain the text D3, which is the intermediate language in this embodiment. In FIG. 12, the utterance tag "<tag>" is used as the utterance style symbol.

A character string representing the type of feature to be given to the entire utterance can be used for the 'tag' portion of the utterance style symbol '<tag>'. The number of characters in the character string representing the speech style symbol may be changed. For example, "<sad>" is used when the feature given to the whole utterance is a sad emotion, "<news>" is used when it is news-like, and "<spkerA>" is used when speaker A is present. Also, in FIG. 12, the sentence to which the feature given to the entire utterance is to be added is surrounded by the utterance style symbol, but the utterance style symbol may be added only to the beginning of the sentence. The sentences enclosed by the utterance style symbols may be one sentence or plural sentences. Also, when a feature is given to a phrase in a sentence, the phrase giving the feature is surrounded by utterance style symbols. In this way, the utterance to which features are to be assigned is the entire utterance of one or more sentences with utterance style symbols added at predetermined positions, the entire utterance of one or more sentences surrounded by utterance style symbols, or the utterance style symbols. can be the entire utterance of the portion of one or more clauses enclosed by .

Here, as an utterance style symbol, an utterance tag "<tag>" that emphasizes human readability like XML (extensible markup language) is used. or combinations thereof may be used. These symbols may be half-width or full-width.

The language processing unit 41a may input the text D1 from a sentence generation system that automatically generates sentences used for a predetermined purpose, such as sports commentary sentences. In this case, the text generation system inputs text D1 describing an automatically generated document and information indicating characteristics given to the entire utterance according to the purpose of the text to the language processing unit 41a.

Also, the user may input features to be given to the utterance. In this case, the display unit (not shown) displays the text D1 or text D2 and a list of icons corresponding to features given to the entire utterance (icon corresponding to each emotion, icon corresponding to each utterance style, icon corresponding to each speaker). corresponding icon, etc.). The user selects an icon representing the feature to be added with the pointing device. The language processing unit 41a adds utterance style symbols corresponding to the selected icon before and after sentences included in the text D2 to generate the text D3. It should be noted that the user may select a partial sentence or clause of the displayed text D1 or text D2 using an input unit (not shown). The language processing unit 41a adds utterance style symbols before and after the portion of the text D2 corresponding to the selected sentence or phrase. The language processing unit 41a outputs the generated text D3 to the acoustic feature estimation unit .

Alternatively, the user may manually enter the speech style symbols. Specifically, the user designates the input position of the utterance style symbol on the text D2 displayed on the display unit (not shown) using a pointing device such as a mouse. Further, the user inputs utterance style symbols according to the characteristics given to the entire utterance using a keyboard or the like.

The acoustic feature estimation unit 42 and vocoder unit 43 perform the same processing as in the first embodiment. That is, the acoustic feature quantity estimating unit 42 uses the techniques described in Non-Patent Document 4 and Reference Document 5, etc., and uses weights (attention ) to estimate acoustic features. The encoder vectorizes and encodes the text D3, which is a character string written in an intermediate language. The decoder weights the output of the encoder and generates acoustic features of the mel-spectrogram by autoregressive RNN. The vocoder unit 43 uses the technique described in reference 1 or the like to estimate the speech waveform from the acoustic feature amount.

By using prosodic symbols, it is possible to control local acoustic features such as prosody (high and low accent), rise and fall at the end of sentences, and pauses. On the other hand, by using utterance style symbols, it is possible to control the tone, tone, emotion, and speaker of the entire utterance or part of the utterance in speech synthesis. By using an intermediate language using speech style symbols, it is possible to model speech corresponding to program production such as live commentary and news style with a small amount of training data. Further, the speech synthesizer 1a may learn a plurality of features using a single acoustic feature value generation model 20-1. In this case, the speech synthesizer 1a can synthesize speech having the features used for learning, using the trained acoustic feature value generation model 20-1.

The learning process of the speech synthesizer 1a is the same as that of the first embodiment shown in the flow chart of FIG. 4 except for the process of step S130. In step S130, the language processing unit 41a of the speech synthesizer 1a converts the learning text data into a character string using reading kana and prosodic symbols, like the language processing unit 41 of the first embodiment. The language processing unit 41a generates an intermediate language by adding, to the character string after conversion, an utterance style symbol representing a feature given to the utterance of the speech data for learning.

FIG. 13 is a diagram showing a learning algorithm of the speech synthesizer 1a. The speech synthesizer 1a enables style control only by setting utterance style symbols in intermediate language data for learning without changing the configuration of the acoustic feature quantity generation model 60 of the first embodiment. For example, a speech corpus containing only sad speech is used for learning the acoustic feature quantity generation model 60 . The data of each speech included in this speech corpus is set as learning speech data A4. The language processing unit 41a of the speech synthesizer 1a morphologically analyzes the utterance content of the speech data for learning A4, encloses the result of the morphological analysis with an utterance tag "<sad>" representing sad emotion, and converts it into intermediate language data for learning B4. to generate The speech synthesizer 1a uses, as learning data, a pair of training speech data A4 obtained from a speech corpus and training intermediate language data B4 generated from the utterance content of this training speech data A4 to generate acoustic features. The quantity generation model 60 is learned. Further, the speech synthesizer 1a learns, for example, the voice of speaker A using the utterance tag "<spkerA>", and learns the voice of speaker B using the utterance tag "<spkerB>". The learning algorithm of the speech synthesizer 1a is as shown in FIG. It is the same as the learning algorithm of the speech synthesizer 1 according to the first embodiment shown.

The speech synthesizing process of the speech synthesizer 1a is the same as that of the first embodiment shown in the flow chart of FIG. 5 except for the process of step S220. In step S220, the language processing unit 41a converts the text data of the sentence containing kana and kanji representing the content of the utterance into a character string using reading kana and prosody symbols in the same manner as the language processing unit 41 of the first embodiment. do. The language processing unit 41a generates an intermediate language by adding an utterance style symbol representing a desired utterance style to the converted character string.

FIG. 14 is a diagram showing a speech synthesis algorithm using the acoustic feature amount generation model 60 of the speech synthesizer 1a. The speech synthesis algorithm shown in FIG. 14 differs from the speech synthesis algorithm of the first embodiment shown in FIG. 9 in that intermediate language data C4 is input instead of intermediate language data C1. The intermediate language data C4 is described using utterance tags (utterance style symbols), prosody symbols, and katakana. The speech synthesis algorithm shown in FIG. 14 is the same as the speech synthesis algorithm of the first embodiment shown in FIG. 9 except that the intermediate language data C4 is input. The acoustic feature quantity generation model 60 is a model learned by the learning algorithm shown in FIG.

FIG. 15 is a diagram showing an example of the encoder 61 of this embodiment. The intermediate language data input to the encoder 61 corresponds to the learning intermediate language data B4 input in FIG. 13 in the case of learning processing, and corresponds to the intermediate language data C4 input in FIG. 14 in the case of speech synthesis processing. do. Character string conversion processing 611 converts each character or symbol used in the description of the intermediate language into a numerical value, and converts the intermediate language into a vector representation. For example, in the character string conversion processing 611, the portion of the utterance tag "<tag>" is converted into values representing "<", "t", "a", "g", and ">". Character string conversion processing 611 and subsequent steps are the same as those of the encoder 61 of the first embodiment shown in FIG. Also, the decoder 65 of this embodiment is the same as that of the first embodiment shown in FIG.

As mentioned above, the structure of the encoder 61 remains unchanged from the first embodiment. However, the intermediate language speech style symbols (speech tags) converted to vector representation by the string conversion process 611 are convolved with nearby strings in the convolution network 612 . Furthermore, in bi-directional LSTM network 613, speech style symbols affect the entire utterance. Therefore, in the attention network 651, the layer that receives the output from the encoder 61 receives speech style control. The structure of the attention network 651 is also unchanged from the first embodiment. Then, when the decoder 65 estimates the acoustic feature amount by RNN, the speech having the same features as the speech corpus having the feature corresponding to the utterance style symbol described in the intermediate language data, specifically "<sad> , which has the characteristics of sad emotional speech, and the speech corpus of "<spkerA>," which has the characteristics of speaker A's speech, can be reproduced.

As described above, since the encoder 61 uses the bidirectional LSTM network 613, in this embodiment, the utterance style symbols are placed before and after the prosody symbols and sentences written in katakana.

In the above-described embodiment, the intermediate language data is generated in the language processing unit 41a. You can enter 1a. In this case, the speech synthesizer 1a does not have to include the language processing section 41a.

Next, the results of evaluation experiments performed by the speech synthesizer 1a of this embodiment will be described. A speech corpus of 12,518 sentences (18 hours) uttered by one female narrator was used for the evaluation experiment. The speech data included in this speech corpus is classified into 2,596 sentences (3 hours and 40 minutes) for sports commentary (hereinafter referred to as "commentary"), 633 sentences (50 minutes) for sadness, and normal reading (hereinafter referred to as " 9,222 sentences (13 hours). Audio data is PCM with a sampling frequency of 22,050 [Hz] and quantization of 16 [bits]. The technique of Non-Patent Document 4 is used for the acoustic feature value generation model 60, and the technique described in Reference Document 1 is used for the vocoder unit 43. FIG. The mel-spectrogram used in the model learning process and the speech synthesis process is 80 [dimensions], the window function is 1,024 [points], and the frame shift is 11.6 [ms].

For learning of the acoustic feature quantity generation model 60, learning speech data A4, which is speech data contained in the speech corpus of the female narrator, and learning intermediate language data B4 created from the kana-kanji mixed sentences of this speech corpus. We used training data paired with The learning intermediate language data B4 used in the experiment was generated by manually correcting the kana and prosody symbols obtained by linguistically analyzing the kana-kanji mixed sentences of the speech corpus and adding the utterance style symbols. is. The number of times of learning is 310,000. For the learning of the vocoder unit 43, mel spectrograms calculated from speech data of 12,451 sentences (18 hours) were directly used. The number of times of learning is 1,220,000.

In the evaluation experiment, intermediate language data was generated by adding 3 types of utterance style symbols, ie commentary, serenity, and sorrow, to 10 sentences of kana and prosodic symbols that are not included in the speech corpus. By inputting the mel-spectrogram estimated by the acoustic feature amount estimation unit 42 using the generated intermediate language data to the vocoder unit 43, 30 voices were synthesized. The volume of these synthesized voices (hereinafter also referred to as "synthetic voice with utterance style") adjusted based on the average loudness value was used as a speech stimulus. The experiment was conducted in a soundproof room at a volume comfortable for each subject to hear with headphones. There are 13 subjects. The experiment was conducted in a soundproof room at a volume comfortable for each subject to hear with headphones. Subjects rated randomly presented audio stimuli.

FIG. 16 is a diagram showing 5-level DMOS values (degradation mean opinion scores) and 95% confidence intervals obtained as evaluation results of the reproducibility of the utterance style for synthesized speech with an utterance style synthesized according to the present embodiment. . For DMOS, for example, Reference 6 "Nippon Telegraph and Telephone Corporation, [online]," Voice Quality Evaluation Method 3. Subjective Evaluation Method of Voice Quality 3.2. DMOS (Degradation Mean Opinion Score)", Internet <URL: http: //www.ntt.co.jp/qos/technology/sound/03_2.html>”. In the experiment to see if the utterance style is reproduced, the reference voice (recorded voice with utterance style) and the evaluation target voice (synthetic voice with utterance style) synthesized by the speech synthesizer 1a of the present embodiment are continuously reproduced. Then, the similarity of their utterance styles (whether it is a sad tone or a commentary tone) was evaluated on a 5-point scale, and the average value was summarized. Five reference speeches were prepared for each of the 10 sentences not included in the speech corpus, in order to rate each sentence five times for three speech styles: live, calm, and sad. A total of 150 speech stimuli per subject were used for evaluation by combining 5 types of reference speech with each of the 30 synthetic speeches with utterance styles. Subjects rated the similarity of speech style to the speech stimuli on a 5-point scale. As shown in FIG. 16, high reproducibility was obtained for each utterance style, but the commentary was evaluated significantly higher. There was no significant difference between sad and calm. The commentary has a fast speaking speed, and the characteristics of clear speech are easier to understand than those of calmness and sadness. The reason for this is thought to be that this was reproduced with high accuracy.

FIG. 17 is a diagram showing the MOS value and the 95% confidence interval obtained as the evaluation of the naturalness of the utterance style for synthetic speech with utterance style synthesized according to this embodiment. There are 13 subjects. A total of 30 sentences of speech stimuli were used for evaluation, with 10 sentences for each of the three utterance styles of commentary, calmness, and sadness. The subjects evaluated the naturalness of each speech stimulus five times, for a total of 150 times per person. As shown in FIG. 17, naturalness was evaluated in the order of calmness, liveliness, and sadness. This is probably because the amount of data in the speech corpus for each utterance style affected the evaluation results.

In the first embodiment, the control of the local acoustic feature quantity called prosody is realized, and the acoustic features of Japanese accents other than reading kana are reproduced by symbols. In this embodiment, the control of the "overall" acoustic feature amount of voice utterance is realized, and it is possible to reproduce the feature over the entire utterance.

According to the speech synthesizer 1a of the present embodiment, both text data for training and text data to be input when performing speech synthesis are expressed in a simple notation, so that the emotions of the synthesized speech, the utterance style, and the control of the speaker can be expressed. It is possible.

This embodiment can be applied not only to Japanese but also to other languages. In this case, instead of Japanese kana, characters or character strings representing the reading of the language are used. In addition, in the present embodiment, kana characters are used as characters representing readings in order to synthesize Japanese speech, and prosody symbols are used. column) itself may serve as both a reading and a prosody mark. In the case of such a language, the text data of the intermediate language describing the sentences representing the contents of the utterance using the characters or character strings representing the reading and the characters or character strings representing the characteristics given to the entire utterance is used for acoustic feature estimation. It is sufficient to input it to the part 42 .

Alternatively, text data containing kana and utterance style symbols but not including prosody symbols may be input to the acoustic feature estimation unit 42 . By using such an intermediate language, the accuracy of local features at the word level is reduced, but the features given to utterances can be controlled with high accuracy.

Conventionally, it has been necessary to rearrange the acoustic feature value generation model for each feature given to the utterance, or to provide an encoder with an input for controlling switching according to the feature given to the utterance. According to the speech synthesizer 1a of the present embodiment, an intermediate language in which utterance style symbols are described is used to learn speech of a plurality of features (emotion, utterance style, speaker) by one acoustic feature amount generation model. , it is possible to synthesize speech of arbitrary utterance contents having features represented by the utterance style symbols used during learning.

Note that the speech synthesizers 1 and 1a described above have a computer system therein. The process of operation of the speech synthesizers 1 and 1a is stored in a computer-readable recording medium in the form of a program, and the computer system reads out and executes the program to perform the above processing. The computer system here includes hardware such as a CPU, various memories, an OS, and peripheral devices.

The "computer system" also includes the home page providing environment (or display environment) if the WWW system is used.
The term "computer-readable recording medium" refers to portable media such as flexible discs, magneto-optical discs, ROMs and CD-ROMs, and storage devices such as hard discs incorporated in computer systems. Furthermore, "computer-readable recording medium" means a medium that dynamically retains a program for a short period of time, like a communication line when transmitting a program via a network such as the Internet or a communication line such as a telephone line. It also includes those that hold programs for a certain period of time, such as volatile memories inside computer systems that serve as servers and clients in that case. Further, the program may be for realizing part of the functions described above, or may be capable of realizing the functions described above in combination with a program already recorded in the computer system.

Reference Signs List 1, 1a Speech synthesizer 20 Storage unit 20-1 Acoustic feature value generation model 20-2 Speech waveform generation model 30 Learning unit 31 Correct acoustic feature value calculation unit 32 Model updating unit 40, 40a Speech Synthesis units 41, 41a...Language processing unit 42...Acoustic feature amount estimation unit 43...Vocoder unit 60...Acoustic feature amount generation model

Claims (6)

First text data in which a sentence representing the contents of a Japanese utterance is described by a character string using characters or character strings representing how to read the contents of the utterance, prosody symbols representing prosody, and utterance style symbols representing characteristics given to the utterance. is input to a first acoustic feature generation model that generates an acoustic feature from the first text data, and a first estimation process for estimating an acoustic feature of the speech corresponding to the utterance content, or the reading The second text data described by a character or character string representing and the character string used with the prosody symbol is input to a second acoustic feature value generation model that generates an acoustic feature value from the second text data, a second estimation process for estimating acoustic features of speech corresponding to the utterance content , or third text data described by a character string or a character string representing the reading and a character string using the utterance style symbol, Acoustic features that are input to a third acoustic feature value generation model that generates acoustic feature values from the third text data, and perform any of a third estimation process of estimating acoustic feature values of speech corresponding to the utterance content an amount estimator;
a vocoder for estimating a speech waveform using the acoustic features estimated by the acoustic feature quantity estimating unit through any one of the first estimation process, the second estimation process, and the third estimation process;
with
The first acoustic feature value generation model , the second acoustic feature value generation model, and the third acoustic feature value generation model have encoders and decoders using deep neural networks,
The encoder uses a recursive neural network to generate a character string feature amount in consideration of character strings before and after the utterance content in the text in the utterance content indicated by the text data,
The decoder uses a recursive neural network to generate an acoustic feature amount of speech corresponding to the utterance content indicated by the text data based on the feature amount generated by the encoder and the acoustic feature amount generated in the past. ,
A speech synthesizer characterized by: The characters representing the reading are katakana, hiragana, alphabet or phonetic symbols,
The first acoustic feature value generation model, the second acoustic feature value generation model, and the third acoustic feature value generation model further have an attention network using a deep neural network,
The attention network generates a weight for weighting the feature quantity output from the encoder, weights the feature quantity with the generated weight, and inputs the feature quantity to the decoder;
The decoder uses a recurrent neural network to generate an acoustic feature quantity of speech corresponding to the utterance content indicated by the text data based on the feature quantity input from the attention network and acoustic feature quantity generated in the past. generate,
2. The speech synthesizer according to claim 1, wherein: The prosodic symbols include any one of a symbol that designates an accent position, a symbol that designates a phrase or phrase delimiter, a symbol that designates the intonation at the end of a sentence, and a symbol that designates a pause .
3. The speech synthesizer according to claim 1, wherein: wherein the feature imparted to speech is emotion, speech style, or speaker;
4. The speech synthesizer according to any one of claims 1 to 3, characterized by: The utterance to which the feature is given is the entire utterance of one or more sentences to which the utterance style symbol is added at a predetermined position, the entire utterance of one or more sentences surrounded by the utterance style symbol, or the utterance style symbol. is an utterance of one or more phrases enclosed by
5. The speech synthesizer according to any one of claims 1 to 4, characterized in that: A program for causing a computer to function as the speech synthesizer according to any one of claims 1 to 5.

Images