JP7339151B2 - Speech synthesizer, speech synthesis program and speech synthesis method - Google Patents

Description

The present invention relates to a speech synthesizer, a speech synthesis program, and a speech synthesis method for synthesizing speech from images.

A speech synthesizer is known that synthesizes speech by inputting information such as characters and phonemes into a speech synthesis model. In addition to characters and phonemes, by inputting the characteristics of the speaker obtained from the speech uttered by the speaker into the speech synthesis model, it synthesizes speech that looks like the speaker uttered according to the characters and phonemes. A speech synthesizer that does this is also known (Non-Patent Document 1).

Also disclosed is a technique for extracting the feature amount of a face image of a target speaker based on subjective evaluation, and generating a voice that sounds like the speaker uttered according to the feature amount based on a statistical model. (Non-Patent Document 2).

”Transfer Learning from Speaker Verification to Multi-speaker Text-To-Speech Synthesis”: https://arxiv.org/abs/1806.04558 ”A Comparative Study of Statistical Conversion of Face to Voice Based on Their Subjective Impressions”: https://www.isca-speech.org/archive/Interspeech_2018/pdfs/2005.pdf

However, there has not been sufficient research on a speech synthesizer capable of objectively and automatically synthesizing a speech that is uttered by a target speaker based on facial information and text information of the target speaker.

One aspect of the present invention is a voice constructed by machine learning using a data set in which image data of an object, audio data of a voice uttered by the object, and content information indicating the content of the audio data are associated. Input of an image encoder that receives an input of image data and outputs a feature vector for the image data, a feature vector generated by the image encoder, and content information indicating the content of the sound to be generated. a voice synthesizer for synthesizing and outputting a voice as if the object indicated by the image data uttered the content corresponding to the content information, wherein the image encoder receives the voice data; Using a speech encoder machine-learned so as to output a feature vector indicating the object associated with the audio data, the feature vector output when the image data of the object is input is the image data Machine learning is performed so as to match the feature vector output from the speech encoder when voice data associated with is input, and the voice synthesizer is configured to perform the speech when voice data of an object is input. When the feature vector output from the encoder and the content information associated with the audio data are input, the audio data of the audio synthesized and output matches the audio data input to the speech encoder. It is a speech synthesizer characterized by being machine-learned as follows.

Another aspect of the present invention is a speech synthesis program using a data set in which image data of an object, audio data of a sound uttered by the object, and content information indicating the content of the audio data are associated, the program comprising: an input of an image encoder that receives an input of image data and outputs a feature vector for the image data, a feature vector generated by the image encoder, and content information indicating the content of the generated sound, and a speech synthesizer that synthesizes and outputs a speech as if the object indicated by the image data uttered the content corresponding to the content information, and the image encoder receives the speech data and outputs the speech. Using a speech encoder machine-learned to output a feature vector indicating the object associated with the data, when image data of the object is input, the output feature vector is associated with the image data. is machine-learned so as to match the feature vector output from the speech encoder when the target speech data is input, and the speech synthesizer is output from the speech encoder when the target speech data is input and the content information associated with the speech data are inputted so that the speech data of speech synthesized and outputted matches the speech data inputted to the speech encoder. A speech synthesis program characterized by being learned.

Another aspect of the present invention is a speech synthesis method using a data set in which image data of an object, audio data of a speech uttered by the object, and content information indicating the content of the audio data are associated with each other, wherein the image An image encoder that receives data input and outputs a feature vector for image data; a feature vector generated by the image encoder; and a speech synthesizer that synthesizes and outputs speech as if the object indicated by is uttering the content corresponding to the content information, and the image encoder inputs the speech data by inputting the speech data. Using a speech encoder machine-learned to output a feature vector indicating an object associated with audio data, the feature vector output when image data of the object is input corresponds to the image data. Machine learning is performed to match the feature vector output from the speech encoder when the attached speech data is input, and the speech synthesizer receives the speech data from the speech encoder when the speech data of the target object is input. When the feature vector to be output and the content information associated with the speech data are input, the voice data of the voice synthesized and output is matched with the voice data input to the speech encoder. This speech synthesis method is characterized by being machine-learned.

Here, the object is a person, and the speech synthesizer receives the input of the feature vector generated by the image encoder and the content information indicating the content of the speech to be generated. It is preferable to synthesize and output a voice as if a person uttered the content corresponding to the content information.

Also, the speech data used for machine learning of the speech synthesizer is preferably cleaner than the speech data used for machine learning of the speech encoder.

The audio data used for machine learning of the speech synthesizer is audio data included in VCTK or LibriTTS, and the audio data used for machine learning of the speech encoder is audio data extracted from the VoxCeleb2 video site. is preferred.

Further, the image encoder receives an average value of feature vectors output from the speech encoder when a plurality of audio data associated with the same object is input, and image data of the object. It is preferable that machine learning is performed so that the difference between the characteristic vector and the output characteristic vector is small.

Further, the speech encoder receives an average value of feature vectors output when a plurality of audio data associated with the same object is input, and other audio data associated with the object. Machine learning is preferably performed so that the difference between the feature vector output when the

One object of the embodiments of the present invention is to provide a speech synthesizer, a speech synthesis program, and a speech synthesis method for synthesizing speech from an image. Other objects of embodiments of the present invention will become apparent by reference to the specification as a whole.

1 is a diagram showing the configuration of a speech synthesizer according to an embodiment of the present invention; FIG. It is a flowchart which shows the speech-synthesis method in embodiment of this invention. FIG. 3 is a diagram showing a speech synthesis model of the speech synthesizer according to the embodiment of the present invention; FIG. It is a figure for demonstrating the face image data in embodiment of this invention. FIG. 4 is a diagram for explaining a method of constructing a speech encoder according to an embodiment of the present invention; FIG. It is a figure for demonstrating the construction method of the image encoder in embodiment of this invention. FIG. 4 is a diagram for explaining a method of constructing a multi-speaker TTS according to the embodiment of the present invention;

A speech synthesizer 100 according to the embodiment of the present invention includes a processing unit 10, a storage unit 12, an input unit 14, an output unit 16, and a communication unit 18, as shown in FIG.

The speech synthesizer 100 can be configured with a general computer. The processing unit 10 includes a CPU and the like, and integrally performs processing in the speech synthesizer 100 . The processing unit 10 executes the speech synthesis program stored in the storage unit 12 to perform speech synthesis processing according to the present embodiment. The storage unit 12 stores speech synthesis models (speech encoder, image encoder, multi-speaker TTS (Text-To-Speech)) used in speech synthesis processing, face image data, speech data, text data, etc. necessary for model generation, Stores information necessary for speech synthesis processing. The storage unit 12 can be composed of, for example, a semiconductor memory, a hard disk, or the like. The storage unit 12 may be provided inside the speech synthesizer 100, or may be provided outside so as to be accessible from the processing unit 10 using an information network such as wireless or wired. Input unit 14 includes means for inputting information to speech synthesizer 100 . The output unit 16 includes means for displaying information processed by the speech synthesizer 100 . The communication unit 18 includes an interface for exchanging information with an external device (server or the like). The communication unit 18 enables communication with external devices by being connected to an information communication network such as the Internet.

[Construction of speech synthesizer]
A method for configuring the speech synthesizer according to the present embodiment will be described below with reference to the flowchart of FIG. The speech synthesizer 100 is configured by performing machine learning for speech synthesis models (speech encoder, multi-speaker TTS, image encoder) by executing a speech synthesis program. By using the speech synthesizer 100, processing for automatically synthesizing speech based on a speech synthesis model can be performed.

The speech synthesis model of the speech synthesizer 100 includes an image encoder 102 and a multi-speaker TTS 104, as shown in FIG. The speech synthesizer 100 is configured by combining a speech encoder, an image encoder and a multi-speaker TTS by machine learning.

A set of text data, face image data, and voice data is used to construct the speech synthesis model of the present embodiment. The text data is data in which the content of the speech of the speaker is represented by characters or phonemes. The text data is used as content information indicating the content of speech generated by the speech synthesizer 100 . The face image data is an image representing the speaker's face. Voice data is voice data corresponding to characters and phonemes included in text data. Here, the voice data is the data of the voice uttered by the speaker, but it is assumed that the data of the sound uttered by some object is also included. For machine learning of the speech synthesis model, face image data of a speaker and speech data corresponding to text data uttered by the speaker are used as a set.

In this embodiment, Vox-Celeb2 and VGGFace2 are used as a combination of face image data and voice data. VoxCeleb2 is a data set obtained by extracting utterances (audio data) of more than 6000 celebrities from video sites. With VoxCeleb2, it is possible to obtain a data set in which face image data and voice data correspond to speakers of various genders and nationalities. However, in the present embodiment, face image data cut out from a moving image has a low resolution. Therefore, face image data is prepared from VGGFace2, which is a data set of images containing the same person as VoxCeleb2, and is combined with voice data of VoxCeleb2. used in combination.

Audio data may be used after being down-sampled to a sampling frequency of 16 kHz, for example. However, the sampling frequency is not limited to this, and other sampling frequencies may be used.

FIG. 4 is a diagram showing a comparison between a face image of a speaker cut out from a moving image in VoxCeleb2 and a corresponding face image of the speaker in VGGFace2. As shown in FIG. 4, the face image in VGGFace2 has a higher resolution than the face image in VoxCeleb2.

VCTK and LibriTTS were used as data sets of text data and voice data. VCTK contains a dataset of over 90,000 utterances by over 100 speakers. LibriTTS contains a dataset of over 18000 utterances by over 800 speakers. The voice data in both the VCTK and LibriTTS data sets are cleaner voices with less background noise than the voice data in VoxCeleb2.

Note that in the present embodiment, text data expressing the content of speech as characters is used, but the present invention is not limited to this. Instead of text data, or in addition to text data, data representing voice content in other ways may be used. For example, it may be phoneme data in which the content of the voice is represented by phonemes, or moving image data in which the content of the voice is represented by changes in facial expressions of the speaker. Also, voice data may be used instead of text data.

Moreover, in the present embodiment, face image data of an existing person is used to represent the speaker, but the present invention is not limited to this. For example, instead of face image data of an existing person, face image data of a character in an animation or the like or face image data of a three-dimensional human model may be used.

At step S10, a speech encoder is constructed by machine learning. Through the processing in this step, the speech synthesizer 100 functions as speech encoder construction means. As shown in FIG. 5, the speech encoder 106 receives speech data by machine learning using a plurality of data sets of speech data associated with each speaker, and determines the speaker who uttered the speech data. is constructed to output the speaker feature vector shown.

Specifically, a plurality of speech data for the same speaker are treated as a mini-batch, and a speaker feature vector output when one speech data included in the mini-batch is input to the speech encoder 106 is included in the mini-batch. Machine learning is performed so as to approach the average vector of speaker feature vectors output when other speech data is input to the speech encoder 106 .

In this embodiment, it is preferable to use voice data extracted from the VoxCeleb2 video site for machine learning of the speech encoder 106 . Audio data included in VoxCeleb2 includes data mixed with noise such as background noise and BGM, and clean audio that is not mixed with them. In the learning of the speech encoder 106, by using speech data mixed with noise and BGM, speaker feature vectors that do not depend on the cleanliness of the speech are obtained as compared to the case of using speech data containing only clean speech. An output speech encoder 106 can be obtained.

For example, the audio data may have window length, hop length, and FFT length of 400 samples (25 ms), 160 samples (10 ms), and 512 samples, respectively. A Hann window may be used as the window function. A 40-dimensional log-Mel spectrogram having a length of 160 frames may be input as speech data, and a speaker feature vector output from the speech encoder 106 may be a 256-dimensional vector. The hidden layer of the neural network that constitutes the speech encoder 106 should be a combination of a 768-dimensional three-layer LSTM (Long Short-Term Memory) and a linear layer that converts from 768 dimensions to 256 dimensions in the final frame. . The output obtained for the calculation of the error function may be L2 normalized. Also, the learning rate is set to 10 ⁻⁵ and Adam is used as the optimization function. However, the configuration of the speech encoder 106 is not limited to these conditions, as long as it is configured to output an appropriate speaker feature vector indicating the speaker who uttered the speech data by inputting the speech data. good.

In step S12, the image encoder 102 is constructed by machine learning. Through the processing in this step, the speech synthesizer 100 functions as image encoder construction means. The image encoder 102 is constructed so as to output an appropriate speaker feature vector when face image data of a speaker is input. That is, the image encoder 102 functions as an image encoder that outputs a feature vector corresponding to the speaker by inputting face image data of the speaker.

In this embodiment, as shown in FIG. 6, when pairs (data sets) of voice data and face image data associated with each speaker are input to the speech encoder 106 and the image encoder 102 respectively, the speech encoder 106 Machine learning is performed so that the speaker feature vector output from the image encoder 102 matches the speaker feature vector output from the image encoder 102 as much as possible.

For example, the speech encoder 106 constructed by machine learning in step S10 is input with speech data in VoxCeleb2 for each speaker, and the average vector of the speaker feature vectors obtained is used as a teacher vector to obtain face image data of the speaker. are combined to form a dataset for supervised learning. Then, when face image data is input, the speaker feature amount vector output from the image encoder 102 becomes as close as possible to the teacher vector (average speaker feature vector) of the speaker corresponding to the face image data, and the The image encoder 102 is machine-learned so as to be as far away from the teacher vector as possible. More specifically, for example, machine learning may be performed so that GE2E loss (generalized end-to-end loss) to which Softmax loss is applied is minimized.

Note that the output of the image encoder 102 may be, for example, a 256-dimensional feature vector. Also, as the neural network forming the image encoder 102, for example, a convolutional neural network such as VGG19 may be applied. As the error function, a supervised GE2E loss in which the center of gravity sequentially calculated by GE2E, which is the error function of speaker features, is replaced with a feature vector learned by a speech encoder may be used.

Here, in the machine learning of the image encoder 102, it is preferable to use face image data corresponding to the speaker in VGGFace instead of the face image data of the speaker extracted from the VoxCeleb2 video site. That is, the voice data used for machine learning of the image encoder 102 is the voice data of each speaker extracted from the video site in VoxCeleb2, and the face image data used for the machine learning of the image encoder 102 is the face corresponding to the speaker in VGGFace. Image data is preferable. This is because the face image resolution of VGGFace2 is higher than that of VoxCeleb2, as shown in FIG. This makes it possible to configure the image encoder 102 that generates a speaker feature vector that more appropriately captures the features of the face image. The face image data may be image data of 160 dots×160 dots, for example.

In step S14, a multi-speaker TTS 104 is constructed by machine learning. Through the processing in this step, the speech synthesizer 100 functions as multi-speaker TTS constructing means. The multi-speaker TTS 104 synthesizes speech as if the speaker uttered the content of the text data when the speaker feature vector indicating the feature of the speaker and the text data indicating the content of the speech to be synthesized were input. Built to output. That is, the multi-speaker TTS 104 functions as a speech synthesizer that receives input of feature vectors and text data, synthesizes speech corresponding to the feature vectors and the text data, and outputs the synthesized speech.

In this embodiment, as shown in FIG. 7, multi-speaker TTS 104 is constructed by performing machine learning using a data set of speech data and text data corresponding to the speech data as learning data. Speech data included in the learning data is input to the speech encoder 106 constructed in step S10, and speaker feature vectors output from the speech encoder 106 for the speech data are input to the multi-speaker TTS 104. FIG. Also, text data corresponding to the voice data input to the speech encoder 106 is input to the multi-speaker TTS 104 . Here, the text data is converted into linguistic features (for example, part of speech, reading, number of moras, accent type/phrase, etc.) using existing text analysis means, and is input to multi-speaker TTS 104. You may make it Upon receiving these inputs, machine learning is performed so that the same voice data as the voice data input from the multi-speaker TTS 104 is output.

For example, multi-speaker TTS 104 combines duration estimation and acoustic feature estimation. We apply a bidirectional LSTM for both duration estimation and acoustic feature estimation. The input in duration estimation is a value obtained by concatenating the linguistic feature amount for each phoneme and the feature vector for each utterance. The output in duration estimation is the number of frames corresponding to the phoneme duration. The input for acoustic feature quantity estimation is a value obtained by concatenating the language feature quantity for each frame and the feature quantity vector for each utterance. The output of the acoustic feature quantity estimation is the acoustic feature quantity for each frame (F0 indicating the pitch of the voice, the spectral envelope (e.g. mel-cepstrum) indicating the feature quantity of the vocal tract, and the aperiodicity index indicating the degree of hoarseness of the voice). do. Also, the squared error may be used as the error function of the models for duration estimation and acoustic feature quantity estimation.

Here, it is preferable to use VCTK and LibriTTS for the data sets of text data and voice data. That is, since the speech data in the VCTK and LibriTTS data sets is cleaner speech with less noise than the speech data in VoxCeleb2, it is possible to construct a multi-speaker TTS 104 that synthesizes and outputs cleaner speech. .

At step S16, a speech synthesis model for the speech synthesizer 100 is constructed. Through the processing in this step, the speech synthesizer 100 functions as speech synthesis model building means. The speech synthesis model in the speech synthesizer 100 is configured by combining the image encoder 102 constructed in step S12 and the multi-speaker TTS 104 constructed in step S14. That is, as shown in FIG. 3, the text data and the speaker feature vector output from the image encoder 102 constructed in step S12 are input to the multi-speaker TTS 104. FIG. The multi-speaker TTS 104 synthesizes speech according to the input text data and speaker feature vectors. In this way, it is possible to configure the speech synthesizer 100 that synthesizes and outputs speech as if the content corresponding to the text data was uttered by the speaker corresponding to the face image data input to the image encoder 102 . .

[Speech synthesis processing]
Processing for synthesizing speech by the speech synthesizing device 100 will be described below with reference to FIG. When synthesizing speech, face image data of a speaker is input to the image encoder 102 in the speech synthesizing device 100 . As a result, the image encoder 102 generates and outputs a speaker feature vector corresponding to the input face image data. The output speaker feature vector is input to multi-speaker TTS 104 . Also, text data indicating the content of the voice to be synthesized is input to the multi-speaker TTS 104 in the voice synthesizer 100 . As a result, the multi-speaker TTS 104 synthesizes speech according to the input text data and speaker feature vectors. In this way, a voice is synthesized and output as if the speaker corresponding to the face image data input to the image encoder 102 uttered the content corresponding to the text data.

As described above, the speech synthesizing apparatus 100 according to the present embodiment can receive text data and face image data of a speaker as input, synthesize and output an appropriate speech as if it were uttered by the speaker.

However, the scope of application of the present embodiment is not limited to human speech synthesis. For example, the speech synthesizer 100 that synthesizes animal voices can be constructed by performing machine learning using image data of animals, voices uttered by animals, and information indicating the content of the voices. In addition, for example, machine learning can be performed using image data of moving objects such as automobiles and trains, sounds generated by the moving objects, and video information showing the movement of the moving objects indicating the content of the sounds. A speech synthesizer 100 that synthesizes sounds can also be constructed.

Also, the speech synthesizer 100 may be constructed using speech data instead of text data. In this case, the content of the input voice data is synthesized with the voice of the person of the face image data input to the image encoder 102 and output. Therefore, it can be used like a voice changer that changes and outputs input data.

Note that the speech synthesizing apparatus 100 according to the present embodiment is configured such that each component is realized by one device, but each component may be realized by different devices or different execution entities. For example, some of the components may be shared by a plurality of computers.

10 processing unit, 12 storage unit, 14 input unit, 16 output unit, 18 communication unit, 100 speech synthesizer, 102 image encoder, 104 multi-speaker TTS, 106 speech encoder.

Claims (7)

A speech synthesizer constructed by machine learning using a data set in which image data of an object, audio data of a sound emitted by the object, and content information indicating the content of the audio data are associated,
an image encoder that receives input of image data and outputs a feature vector for the image data;
In response to the input of the feature vector generated by the image encoder and the content information indicating the content of the generated sound, the object indicated by the image data generates a sound as if the content corresponding to the content information was emitted. a speech synthesizer for synthesizing and outputting;
with
The image encoder is input with image data of an object using a speech encoder machine-learned to output a feature vector indicating an object associated with the audio data by inputting the audio data. Machine learning is performed so that the feature vector that is output when the speech encoder matches the feature vector that is output from the speech encoder when the audio data associated with the image data is input,
The speech synthesizer synthesizes and outputs the feature vector output from the speech encoder when the speech data of the object is input and the content information associated with the speech data when the speech synthesizer is input. 1. A speech synthesizer, wherein machine learning is performed so that the speech data of the speech to be generated matches the speech data inputted to the speech encoder. The speech synthesizer according to claim 1,
the object is a person,
The voice synthesizer receives input of the feature vector generated by the image encoder and content information indicating the content of the voice to be generated, and the person indicated by the image data utters content corresponding to the content information. A speech synthesizer characterized by synthesizing and outputting speech such as The speech synthesizer according to claim 1 or 2,
A speech synthesizer according to claim 1, wherein speech data used for machine learning of said speech synthesizer is cleaner than speech data used for machine learning of said speech encoder. The speech synthesizer according to any one of claims 1 to 3,
The image encoder outputs an average value of feature vectors output from the speech encoder when a plurality of audio data associated with the same object is input, and when image data of the object is input, A speech synthesizer characterized in that machine learning is performed so that the difference between output feature vectors and feature vectors is reduced. The speech synthesizer according to any one of claims 1 to 4,
The speech encoder receives an average value of feature vectors output when a plurality of audio data associated with the same object is input, and other audio data associated with the object. A speech synthesizer characterized in that machine learning is performed so that the difference between a feature vector that is output from time to time and a feature vector that is output from time to time is reduced. A speech synthesis program using a data set in which image data of an object, audio data of a sound uttered by the object, and content information indicating the content of the audio data are associated,
the computer,
an image encoder that receives input of image data and outputs a feature vector for the image data;
In response to the input of the feature vector generated by the image encoder and the content information indicating the content of the generated sound, the object indicated by the image data generates a sound as if the content corresponding to the content information was emitted. a speech synthesizer for synthesizing and outputting;
function as
The image encoder is input with image data of an object using a speech encoder machine-learned to output a feature vector indicating an object associated with the audio data by inputting the audio data. Machine learning is performed so that the feature vector that is output when the speech encoder matches the feature vector that is output from the speech encoder when the audio data associated with the image data is input,
The speech synthesizer synthesizes and outputs the feature vector output from the speech encoder when the speech data of the object is input and the content information associated with the speech data when the speech synthesizer is input. 1. A speech synthesis program characterized in that machine learning is performed so that the speech data of the speech to be input matches the speech data input to the speech encoder. A speech synthesis method using a data set in which image data of an object, audio data of a speech uttered by the object, and content information indicating the content of the audio data are associated,
an image encoder that receives input of image data and outputs a feature vector for the image data;
In response to the input of the feature vector generated by the image encoder and the content information indicating the content of the generated sound, the object indicated by the image data generates a sound as if the content corresponding to the content information was emitted. a speech synthesizer for synthesizing and outputting;
Synthesize the speech using
The image encoder is input with image data of an object using a speech encoder machine-learned to output a feature vector indicating an object associated with the audio data by inputting the audio data. Machine learning is performed so that the feature vector that is output when the speech encoder matches the feature vector that is output from the speech encoder when the audio data associated with the image data is input,
The speech synthesizer synthesizes and outputs the feature vector output from the speech encoder when the speech data of the object is input and the content information associated with the speech data when the speech synthesizer is input. 1. A speech synthesis method, wherein machine learning is performed so that the speech data of the speech to be generated matches the speech data inputted to the speech encoder.

Images