JP2021148942A - Voice quality conversion system and voice quality conversion method - Google Patents

Abstract

To provide a voice quality conversion system and a voice quality conversion method capable of stably performing voice quality conversion with high voice quality in performing voice quality conversion.SOLUTION: A voice quality conversion system includes a voice quality conversion data creation device and a voice quality conversion device. The voice quality conversion data creation device generates a dictionary with prosody information from a word dictionary in a PPG conversion model learning, applies morpheme analysis to text included in a voice corpus, generates a phoneme array with prosody information on the basis of a result of the morpheme analysis and a dictionary with dictionary prosody information, generates an acoustic model from the phoneme array with prosody information and a voice feature amount outputted as a result of a feature amount analysis of voice information included in the voice corpus, learns the acoustic model, and generates a PPG conversion model including the prosody information. Also, it generates a voice parameter generation model by using the PPG conversion model including the prosody information. The voice conversion device generates a voice parameter to input voice, and performs voice quality conversion on the basis of the voice parameter generation model.SELECTED DRAWING: Figure 5

Description

The present invention relates to a voice quality conversion system and a voice quality conversion method, and more particularly to a voice quality conversion system and a voice quality conversion method that enable stable and high-quality sound quality conversion when performing voice quality conversion of voice.

In recent years, with the dramatic improvement in technologies such as voice recognition, machine translation, and dialogue generation, the practical application of voice communication by artificial intelligence such as voice translation, voice dialogue service, and service robot has rapidly progressed. .. Among them, voice conversion (VC) technology is attracting attention as one of the important technologies. Voice conversion is a technology that edits the voice of one speaker (source speaker) so that it can be heard by another speaker (target speaker) without changing the content of the utterance and the way of speaking. ..

In recent years, prototypes of service robots of each company have been developed one after another, and PoC (proof of concept) is being carried out. In such service robots, speech recognition and speech synthesis techniques are indispensable. However, in the actual environment (especially at airports and train stations), the accuracy of voice recognition is poor, and there is a problem that the success rate of dialogue is very low. As a result, the actual deployment of service robots has been postponed, and real data cannot be accumulated, which is one of the causes that hinder the market growth of service robots. Therefore, in order to improve the quality of customer service by robots, a hybrid voice dialogue service is being conceived in which automatic response and operator response are linked in parallel with research on improving the accuracy of voice recognition and intention understanding.

In order to realize this concept, the automatic response voice generated by TTS (Text To Speech) and the operator's real voice can be seamlessly switched, so a voice quality conversion technology that converts the operator's voice into the robot's voice is indispensable. ..

Regarding such a voice quality conversion technique, for example, Non-Patent Document 1 discusses that voice quality conversion is performed using a phonetic posterior probability (PPG).

L. Sun, K. Li, H. Wang, S. Kang, and H. Meng, “PHONETIC POSTERIORGRAMS FOR MANY-TO-ONE VOICE CONVERSION WITHOUT PARALLEL DATA TRAINING” Multimedia and Expo (ICME), 2016.

The conventional voice quality conversion technology has a problem that the performance of voice quality conversion is remarkably deteriorated depending on the recording environment of the input voice, but the voice quality conversion technology described in Non-Patent Document 1 solves such a problem. Is intended to be. The technique described in Non-Patent Document 1 removes the speaker nature of the input voice and the recording environment sound, and uses an acoustic model learned by voice recognition to convert the voice feature amount into a PPG containing only information related to the utterance content. By converting, it is intended to realize stable voice quality conversion.

However, the PPG generated from the acoustic model used in Japanese speech recognition is the phoneme posterior probability of Japanese phonemes, and it is said that it does not include information such as prosodic information and aperiodic component information that are not related to the tone structure. There is. Therefore, it is difficult to estimate the fundamental frequency (F0) (one of the acoustic features, the reciprocal of the interval of the impulse train of the voice, which corresponds to the pitch of the voice) only from the Japanese PPG. In conventional research, only MCEP (merkeptrum coefficient, merkeptrum is a method of finely sampling the low frequency region according to human auditory characteristics) related to vocal tract structure is converted by PPG, and prosodic information (pitch, etc.) is linearly converted. Convert by the method of. Then, the audio is reconstructed using the separately converted parameters. However, it is generally known that when voice parameters generated separately are used in voice quality conversion, it tends to lead to deterioration of voice quality conversion sound quality. In particular, since the extraction of F0 was very unstable, stable voice quality conversion could not be performed.

An object of the present invention is to provide a voice quality conversion system and a voice quality conversion method that enable stable and high-quality sound quality conversion in performing voice quality conversion of voice.

The configuration of the voice quality conversion system of the present invention is preferably a voice quality conversion system that outputs a synthetic voice obtained by converting voice quality from input voice by an information processing device, and comprises a voice quality conversion data creation device and a voice quality conversion device. The voice quality conversion data creation device includes a PPG (speech post-probability) model learning unit that generates a PPG conversion model by inputting a word dictionary, a voice corpus that associates text with voice information, and a voice corpus and PPG. It consists of a voice parameter generation model learning unit that inputs a conversion model and generates a voice parameter generation model. The PPG conversion model learning unit generates a dictionary with rhyme information from a word dictionary and morphologically analyzes the text contained in the voice corpus. Then, based on the result of the morphological analysis and the dictionary with the rhyme information, the phonetic sequence with the rhyme information is generated, and the phonetic sequence with the rhyme information and the feature quantity analysis of the speech information contained in the speech corpus are output. An acoustic model is generated from the voice feature amount, the sound model is learned, and a PPG conversion model is generated. The voice parameter generation model learning unit uses the PPG conversion model and the voice corpus to generate a PPG corresponding to the PPG conversion model. Generates, extracts voice parameters from voice information contained in the voice corpus, learns the generated PPG and voice parameters, generates a voice parameter generation model, and the voice quality conversion device is an input voice and PPG conversion model learning unit. Input the PPG conversion model generated by the voice parameter generation model and the voice parameter generation model generated by the learning unit, and the voice feature amount and the PPG conversion model output as a result of the feature amount analysis of the voice information included in the voice corpus. Based on the above, a PPG is generated, a voice parameter is generated based on the generated PPG and a voice parameter generation model, a voice waveform based on the voice parameter is generated, and the voice is output as an output voice.

According to the present invention, it is possible to provide a voice quality conversion system and a voice quality conversion method that enable stable and high-quality sound quality conversion in performing voice quality conversion of voice.

It is a figure which showed the structure and data flow of a voice quality conversion system. It is a hardware block diagram of the voice quality conversion apparatus. It is a hardware configuration diagram of a voice quality conversion system consisting of a client-server system. It is a figure which shows the functional structure and data flow of the data creation apparatus for voice quality conversion. It is a figure which shows the functional structure and data flow of the PPG conversion model learning part. It is a table showing the result of morphological analysis of the text "This is a bridge." It is a table showing the result of morphological analysis of the text "This is chopsticks." It is a figure which shows the functional structure and data flow of a voice model learning part. It is a figure which shows the functional structure and data flow of a voice quality conversion apparatus. It is a figure which shows the processing and data flow of the speech feature learning system using the hostile generation network learning. It is a figure which shows the process flow and data flow of constructing the multilingual speech synthesis system using the speech corpus converted by the speech quality conversion. It is a figure which shows the process flow and data flow which constructs the system which synthesizes speech with respect to the input text using the speech corpus converted by the speech quality conversion.

Hereinafter, each embodiment of the present invention will be described with reference to FIGS. 1 to 11.

[Embodiment 1]
First, the configuration of the voice quality conversion system will be described with reference to FIGS. 1 and 3.
As shown in FIG. 1, a general voice quality conversion system includes a voice quality conversion data creation device 200 and a voice quality conversion device 100. The voice quality conversion data creation device 200 is a device that generates voice quality conversion data 20 from the voice corpus 10. The voice quality conversion device 100 is a device that converts the input voice 30 into a synthetic voice 40 having a desired voice quality and outputs the voice quality conversion data 20 using the voice quality conversion data 20.

The voice corpus 10 is data in which a voice file and a text are associated with each other. The voice quality conversion data 20 is a PPG conversion model 1 to a PPG conversion model N and a voice parameter generation model (details will be described later).

In the following, the functional configuration of each device and the processing by the functional configuration will be mainly described, but the functional configuration unit may be realized as hardware or as a software program.

Further, in the following explanation, the Japanese voice corpus is taken as an example at the time of learning, but processing can also be performed by using another natural language or a voice corpus in which a plurality of languages are mixed. However, in that case, the program data corresponding to the language must be used.

Further, in the following description, for example, DNN (Deep Neural Network) is used as the voice quality conversion method, but other statistics-based methods may be used.

Next, the hardware configuration of the voice quality conversion device will be described with reference to FIG.
The voice quality conversion device 100 can be realized by a general information processing device, and as shown in FIG. 2, for example, an auxiliary storage device 101, a voice input I / F (InterFace) 102, a CPU 103, and a main memory 104. , Audio output I / F 105, and they are connected by a bus 107.

The CPU 103 controls each part of the voice quality conversion device 100, loads a program required for the main storage device 104, and executes the program.
The main memory 104 is usually composed of a volatile memory such as a RAM, and stores a program executed by the CPU 102 and data to be referred to.

The voice input I / F 102 is an interface for inputting a voice signal by being connected to a microphone or the like.
The audio output I / F 103 is an interface for inputting an audio signal by being connected to a speaker or the like.

For audio input / output, for example, encoded audio data such as a WAVE file or an MP3 file may be input / output.

The auxiliary storage device 101 is a storage device having a large storage capacity such as an HDD (Hard Disk Drive) or an SSD (Solid State Drive).
Although not shown in the auxiliary storage device 101, a feature quantity analysis program, a PPG extraction program, a merge program, a voice parameter generation program, and a waveform generation program, which are programs for executing the functions of the voice quality conversion device 100 of the present embodiment, are not shown. The program is installed.

The feature quantity analysis program, PPG extraction program, merge program, voice parameter generation program, and waveform generation program are programs that execute the functions of the feature quantity analysis section, PPG extraction section, merge section, voice parameter generation section, and waveform generation section, respectively. be. The details of the processing of these functional units will be described later.

Further, the auxiliary storage device 101 stores various data used in the voice quality conversion device 100. Various data used in the voice quality conversion device 100 include voice features, PPG, voice parameter generation model, and voice parameters, as will be described later.

Similarly, the voice quality conversion data creation device 200 can also be realized by an information processing device having the same configuration as the voice quality conversion device 100 of FIG.

Although not shown in the auxiliary storage device 101 of the voice quality conversion data creation device 200, the PPG conversion model learning program and the voice parameters, which are programs for executing the functions of the voice quality conversion data creation device 200 of the present embodiment, are shown in the auxiliary storage device 101. The generative model learning program is installed.

The PPG conversion model learning program and the voice parameter generation model learning program are programs that execute the functions of the PPG conversion model learning unit and the voice parameter generation model learning unit, respectively. The details of the processing of these functional units will be described later.

Further, the auxiliary storage device 101 stores various data used in the voice quality conversion data creation device 200. Various data used in the voice quality conversion data creation device 200 include a word dictionary, a voice corpus, and a voice parameter generation model, as will be described later.

The voice quality conversion device 100 is incorporated as a voice quality conversion unit in a device such as a car navigation device, a mobile phone, or a personal computer. Therefore, each hardware shown in FIG. 2 may be realized by a device in which the voice quality conversion device 100 is incorporated, or separately from the voice quality conversion data creation device 200 and the device in which the voice quality conversion device 100 is incorporated. It may be provided.

All the functions related to voice quality conversion may be realized by only one or two devices, but as shown in the modified example shown in FIG. 3, the server 330 and the client terminal 400 (denoted as 400A and 400B in FIG. 3) , It can also be realized in a system interconnected by the network 5.

In this case, the client terminal 400 has a voice input I / F 402, a voice output I / F 405, and a communication I / F 406, receives voice on the client terminal 400 side, and performs some or all of the functions related to voice quality conversion. The server 300 may be in charge, and necessary data may be exchanged between the communication I / F 306 on the server 300 side and the communication I / F 405 on the client terminal 400.

Next, the functions and processing of the voice quality conversion system according to the first embodiment will be described with reference to FIGS. 4 to 8.
First, the functional configuration and data flow of the voice quality conversion data device will be described with reference to FIG.
As shown in FIG. 4, the voice quality conversion data creation device 200 has a PPG conversion model learning unit 210 and a voice parameter generation model learning unit 220 as functional configurations.

The PPG conversion model learning unit 210 is a functional unit that generates a PPG conversion model (details will be described later) 700 by learning using the word dictionary 500 and the voice corpus 510.

PPG (phoneme posterior probability) is, as defined in Non-Patent Document 1, the posterior probability for each phoneme class in a certain utterance (in Non-Patent Document 1, the expression matrix of the time-phoneme class posterior probability). be.

The voice parameter generation model learning unit 220 is a functional unit that generates a voice parameter generation model from the generation voice corps (same data structure as the voice corps 510) 520 and the PPG conversion model 600.

Next, the function of the PPG conversion model learning unit and the details of the data flow will be described with reference to FIG.
The PPG conversion model learning unit 210 generates the PPG conversion model 700 by learning using the word dictionary 500 and the voice corpus 510 as described above. This PPG conversion model 700 is a model that converts the input voice 30 into a PPG including the utterance content and the utterance style information. As shown in FIG. 5, the PPG conversion model learning unit 210 includes a morphological analysis unit 211, a language model learning unit 212, a feature quantity analysis unit 213, a dictionary reading extension unit 214, a morpheme arrangement expansion unit 215, a phoneme arrangement & sound. It is composed of a sub-functional unit of a model learning unit 216 and a language model-considered acoustic model learning unit 217.

The morphological analysis unit 211, the language model learning unit 212, the feature quantity analysis unit 213, the dictionary reading extension unit 214, the morphological element arrangement expansion unit 215, the phonetic array & acoustic model learning unit 216, and the language model-considered acoustic model learning unit 217 are respectively. Execute morphological analysis program, language model learning program, feature quantity analysis program, dictionary reading extension program, morphological sequence extension program, phonetic array & acoustic model learning program, language model-considered acoustic model learning program as subroutines of PPG conversion model learning program. It can be realized by doing.

The morphological analysis unit 211 is a functional unit that divides text into morpheme units using a word dictionary 500 prepared in advance. Here, the morpheme is the smallest unit of expression having meaning in linguistics. In order to realize the function of the morphological analysis unit 211, OSS (Open Source Software) morphological analysis tools such as MeCab and chasen, which are generally used, can be used.

It is assumed that the word dictionary 500 always has a reading in that language. Then, the morphological analysis unit 211 generates a morpheme array 600 with reading information for the input text.

In the description of this embodiment, it is assumed that the "word dictionary" is input to the morphological analysis unit 211, but the unit of the dictionary is not necessarily a word but a phrase or a sentence.

Here, to give an example, the result of morphological analysis for the text "This is chopsticks" is shown in Fig. 6A, while the text "This is a bridge" is morphologically analyzed. The result of the analysis is shown in FIG. 6B.

The language model learning unit 212 is a functional unit that performs language model learning and creates a language model using the morpheme array 600 with reading information generated from the morphological analysis unit 211. In language model learning, a language model generally called N-gram is often used. N-gram is a method of dividing an arbitrary character string or document into N consecutive characters. In recent years, a language model using an RNN (Recurrent Neural Network) has also come to be used.

The feature amount analysis unit 213 is a functional unit that extracts a feature amount from the voice included in the voice corpus 520. As shown in FIG. 5, the voice corpus 520 is data in which the utterance text 521 and the voice 522 are associated one-to-one. Generally, MFCC is often used as a feature of speech, but there are some studies using prosodic information such as LF0. MFCC (Mel Frequency Ceastral Coefficient) is a feature quantity in which the low-order components of logarithmic cepstrum (expressing frequency characteristics derived from voice tract components) are weighted in consideration of human frequency perception characteristics. Is. LF0 is the logarithm of the fundamental frequency F0.

Any feature amount may be used as the feature amount used in the present embodiment, but at a minimum, it is necessary to include tone information and prosodic information. That is, when MFCC is used, it is recommended to use all dimensions (40 dimensions in the case of 16 kHz voice) because prosodic information is not included when only low dimensions are used.

Here, articulation is a new sound that controls the flow of air in the vocal organs by the shape and movement of organs above the larynx, changes the way the voices that are generated in the vocal organs resonate, and so on. To generate or add various vowels and consonants. Prosody is a phonetic property that appears in speech and is not predicted from general clerk records of the language, such as intonation or tone, emphatic consonant, length, and rhythm.

The dictionary reading extension unit 214 adds a prosody symbol to the reading information (phoneme information) given to the word dictionary 500 for morphological analysis prepared in advance, expands it into a phoneme with prosody information, and prosody. Generates a phoneme dictionary 602 with information. In the case of Japanese, syllables may be given as reading information, so even if the term "phoneme" is simply used, it may refer to "syllables". Here, in linguistics, a phoneme is a group of sounds that are considered to be the same in a certain individual language, and a syllable is a kind of syllable unit that separates continuous speech sounds. ..

This phoneme with prosody information needs to be matched to the characteristics of each language, and it is necessary to define the prosodic information that bears the linguistic information. For example, in a pitch-accent language such as Japanese, the relative position of F0 between syllables is an important clue to distinguish accents, so all vowels mean "H", which means High Pitch, and Low Pitch. "L" can be added. On the other hand, in a tone language such as Chinese, the F0 pattern in a syllable plays an important role in understanding the meaning, so the number 1 is used as four tone symbols for vowel phonemes (in the case of so-called putonghua). ~ 5 (light voice, 1st to 4th voices) can be added. Furthermore, by registering a plurality of prosodic patterns for the same word in consideration of the deformation of the accent (modulation in Chinese), it is possible to actually accurately capture the prosodic change of the voice.

As an example, for the word "bridge", before expansion, it is assumed that "notation = bridge; reading = / ha / + / shi /", and after expansion, the prosodic symbol in the above Japanese case. Is added and expanded to "Notation = Bridge; Reading 1 = / Ha L / + / Shi H /: Reading 2 = / Ha H / + / Shi H /" to list all the accent types that can be spoken. On the other hand, for the word "chopsticks", "notation = chopsticks; reading = / ha / + / shi /" before expansion, whereas "notation = chopsticks; reading 1 = /" after expansion. C H / + / C Chopsticks L /: Reading 2 = / C Chopsticks / + / Chopsticks H / ”.

That is, conventionally, only the phoneme sequence is generated from the word dictionary 500, but in the present embodiment, the phoneme sequence 603 with prosody information is generated by the dictionary 602 with prosody information. Therefore, even homonyms that cannot be uniquely identified by phoneme arrangements in the past can be identified by the difference in accent. That is, by introducing a phoneme with prosody information, the prosody information is taken into consideration at the time of speech recognition, and the PPG which is the output of the acoustic model includes the prosody information.

The morpheme array expansion unit 215 is a functional unit that expands morpheme readings into a plurality of morphemes, prepares all readable patterns, and converts them into phoneme sequences 603 with prosody information.

For example, in contrast to "This is a bridge." It is expanded to "/ co L / + / re H / + / wa H / + / ha H / + / shi H / + / de H / + / sl /". The number of phoneme sequences is a combination of the total number of readings registered for each word.

The phoneme arrangement determination & acoustic model learning unit 216 determines the most probable combination from the phoneme arrangements 603 with a plurality of prosodic information generated by the morpheme arrangement extension unit 215, and then determines the characteristics of each phoneme (MFCC, which is a voice feature amount). It is a functional part that calculates the average and variance of) and generates an acoustic model 620. In general, HMM (Hidden Markov Model) is often used to determine the optimum series, but DNN is the mainstream for learning acoustic models.

The language model consideration acoustic model learning unit 213 uses the language model 610 generated by the language model learning unit 212 and the acoustic model 620 generated by the phoneme arrangement determination & acoustic model learning unit 210, and has an error rate with respect to the voice corpus 520. It is a functional part that relearns based on the standard of minimization and generates a PPG conversion model 700. Since the PPG conversion model 700 learned in this way can express the prosodic information necessary for transmitting the linguistic information, the prosodic information that can be expressed differs depending on the characteristics of the language.

For example, in the case of Japanese, which is a pitch-accent language (a language in which the relative position of F0 between syllables contributes to the distinction of words), F0 fluctuation over a wide area (a range that spans multiple syllables) and tonal language (the shape of the F0 pattern within a syllable) In Chinese, where differences contribute to the distinction of words), local (intrasyllable) F0 fluctuations can be captured. On the other hand, in English, which is a strong and weak accent language, it is thought that the strength of sound can be expressed.

Next, the function of the voice parameter generation model learning unit and the details of the data flow will be described with reference to FIG. 7.
As described above, the voice parameter generation model learning unit 220 is a functional unit that generates the voice parameter generation model 1000 from the generation voice corps 520 and the PPG conversion model 700.

As shown in FIG. 7, the voice parameter generation model learning unit 220 includes a PPG extraction unit 221 and a PPG merge unit 222, a voice parameter extraction unit 223, a voice model learning unit 224, and a sub-function unit of the feature amount analysis unit 225. Has been done.

The PPG extraction unit 221 and the PPG merge unit 222, the voice parameter extraction unit 223, the model learning unit 224, and the feature quantity analysis unit 225 are the subroutines of the voice parameter generation model learning program, respectively, the PPG extraction program, the PPG merge program, and the voice parameter. This can be achieved by executing an extraction program, a voice model learning program, and a feature quantity analysis program.

The PPG extraction unit 221 is the PPG conversion model 700 obtained by the PPG conversion model learning unit 210 described with reference to FIG. 5 (in FIG. 7, it is expressed as PPG conversion model 1: 700-1 to PPG conversion model N: 700-N). Is used to extract PPG800 (indicated as PPG1: 800-1 to PPGN: 800-N in FIG. 7) with respect to the voice feature amount 640 extracted by the feature amount analysis unit 225 from the generation voice corpus 511. It is a functional part. The feature amount analysis unit 225 is the same as the feature amount analysis unit 213 shown in FIG. Here, by using a plurality of PPG conversion models, accurate prosodic expression becomes possible. Specifically, by combining a Japanese PPG that can express the movement of F0 that slowly changes across syllables and a Chinese PPG that can capture local F0 changes within a syllable, the F0 pattern is fully expressed. can do. That is, by combining a plurality of languages having different characteristics, it is possible to design which characteristic of the input voice is desired to be left in the output voice. In this respect, in the demonstration by the present inventor, it was confirmed that the strength of pronunciation and the intonation of utterance can be accurately reproduced by using the three languages of Japanese, Chinese, and English.

The PPG merging unit 222 is a functional unit that merges a plurality of PPG 800s obtained from the PPG extraction unit 221 into one vector. Here, it is conceivable to simply connect a plurality of vectors to form a vector having a large number of dimensions, but it is also possible to compress the vector into a small vector by using a dimension compression technique such as AutoEncoder.

The voice parameter extraction unit 223 extracts the voice parameter 223 for voice synthesis from the voice of the voice corpus 511 for generation. This part is a technique generally used for speech synthesis, and high-quality synthetic speech can be obtained by using OSS such as Straight or World.

The voice model learning unit 224 uses the voice parameter generation model 1000 by learning the conversion DNN for the PPG 800 extracted from the same voice and merged by the PPG merge unit 222 and the voice parameter 900 extracted by the voice parameter extraction unit 223. To generate. That is, the input is PPG 800 and the voice parameter 900, and as the output, the voice parameter generation model 1000 of the input voice parameter 900 is obtained. Generally, for time-series signals such as voice, higher performance can be obtained by using Bi-LSTM (Bidirectional Long Short Term Memory).

Next, the function and data flow of the voice quality conversion device will be described with reference to FIG.
As shown in FIG. 8, the voice quality conversion device 100 includes a feature amount analysis unit 110, a PPG extraction unit 111, a PPG merging unit 112, a voice parameter generation unit 113, and a waveform generation unit 114.

The feature amount analysis unit 110 is the same functional component unit as the feature amount analysis unit 213 of the PPG conversion model learning unit 210 of the voice quality conversion data creation device 200 shown in FIG. The PPG extraction unit 111 and the PPG merging unit 112 are the same functional components as the PPG extraction unit 221 and the PPG merging unit 222 of the voice parameter generation model learning unit 210 of the voice quality conversion data creation device 200 shown in FIG. 7, respectively. ..

The voice parameter generation unit 113 inputs the voice parameter generation model 1000 obtained by the voice parameter generation model learning unit 210 of the voice quality conversion data creation device 200, and the input voice 30 and the PPG 800 obtained from the PPG conversion model 700. Generate voice parameter 900. The voice parameter 900 has, for example, a fundamental frequency corresponding to the pitch of the voice, a spectral envelope corresponding to the timbre, and an aperiodicity index (Aperiodicity) corresponding to the faint voiced sound.

The waveform generation unit (also referred to as a vocoder) 114 generates a voice waveform using the generated voice parameter 900, and outputs the converted voice as the output voice 40.

According to the voice quality conversion system of the present embodiment, PPG is generated by a PPG conversion model including prosodic information, and voice parameters are generated using the PPG. Therefore, the voice quality conversion based on the voice parameters takes into consideration the prosody peculiar to the speaker's language, and enables stable and high-quality sound conversion.

[Embodiment 2]
Hereinafter, Embodiment 2 of the present invention will be described with reference to FIG.
In the present embodiment, as one of the applications using the voice quality conversion device shown in the first embodiment, the processing of the voice feature learning system using the hostile generative network (GAN) learning and its data flow are described. explain.

The voice feature learning system of the present embodiment performs learning based on a hostile generation network, and comprises two stages of a Generator Training Stage, a Generator Training Stage, and a Discriminator Training Stage.

The voice feature learning system of the present embodiment is a system capable of extracting voice features with a unified voice quality from a multilingual voice corpus 2010 recorded from different speakers. This makes it possible to construct a multilingual speech synthesis system using the optimized speech features.

First, in the voice feature amount learning system, the voice quality conversion of the first embodiment is performed from the multilingual voice corpus 2010 (indicated as voice corpus 2010-1, voice corpus 2010-2, ... In FIG. 9) having different languages and voice qualities. The voice quality conversion process 150 is executed by the device 100, and the multilingual voice corpus 2060 having the target voice quality Z for each language (indicated as voice corpus 2060-1, voice corpus 2060-2, ... In FIG. 9). To generate.

On the other hand, in the voice feature learning system, the Generator Training Stage generates the language feature 2020 from the multilingual voice corpus 2010 by the language feature analysis process 2002, and the voice feature analysis process 2003 generates the voice feature of the recorded voice. Generate 2030.

Next, the language feature amount 2020 and the voice feature amount 2030 of the recorded voice are input to perform the model learning process 2000, and the result is output by the process 2001 by the Generator (generation network). In the processing 2001 by the Generator, the synthetic speech feature amount 2040 based on the data including noise is output. So to speak, it generates data that deceives the Discriminator (identification network) described below.

Further, in the Discriminator Training Stage, the voice feature amount 2050 of the converted voice is generated from the multilingual voice corpus 2060 having the target voice quality Z by the language feature amount analysis process 2006. Then, the feature amount 2040 of the synthetic voice generated by the Generator 2001 and the voice feature amount 2050 of the converted voice are input to execute the model learning process 2005, and the result is output to the Discriminator. In the process 2004 by the Discriminator, the authenticity is identified, a label for determining the authenticity is generated, and the label is fed back to the model learning process 2000 of the Generator Training Stage. Thereby, the processing 2001 by the Generator can generate a voice feature amount that is natural and close to the voice quality of the speaker of voice quality Z.

That is, by repeating this Generator Training Stage and the Discriminator Training Stage and looping each other's learning processing, it is possible to obtain the voice feature amount of the synthetic voice of the target voice quality Z that maintains high sound quality, and that voice. It is possible to construct a multilingual speech synthesis system for target speakers that supports multiple languages using feature quantities.

In the embodiment of the book, it is desirable that the PPG used for the voice quality conversion device 100 is a PPG that merges PPGs of all languages included in the multilingual speech corpus 2010. For example, when constructing a Japanese-Chinese-English three-language bilingual speech synthesis system, it is desirable to extract Japanese PPG, Chinese PPG, and English PPG in the PPG extraction unit shown in FIG.

In the description of the present embodiment, it is assumed that the input multilingual voice corpus 2010 includes a plurality of languages and a plurality of voice qualities, but a single language single voice quality may be used. In that case, the effect of voice quality customization of voice synthesis can be obtained.

[Embodiment 3]
Hereinafter, the third embodiment according to the present invention will be described with reference to FIG.
In this embodiment, a process of generating a multilingual speech synthesis system will be described as one of the applications using the voice quality conversion device 100 of the first embodiment.
First, from the multilingual voice corpus 3010 (denoted as voice corpus 3010-1, voice corpus 3010-2, ... In FIG. 10) having different languages and voice qualities, the voice quality conversion process 150 by the voice quality conversion device 100 of the first embodiment 150. Is executed to generate a multilingual voice corpus 3020 (denoted as voice corpus 3020-1, voice corpus 3020-2, ... In FIG. 9) having a target voice quality Z for each language.

Next, from the multilingual voice corpus 3020, the language feature amount analysis process 3001 generates the language feature amount 3030, and the voice feature amount analysis process 3002 generates the voice feature amount 3040 of the recorded voice.
Then, the multilingual speech synthesis system 3050 is constructed by the synthesis system construction process 3000 that uses them as inputs.

In the construction process of this speech synthesis system, the speech synthesis system can be constructed regardless of the synthesis method of the synthesis system construction process 3000.

In the description of the present embodiment, it is assumed that the input multilingual voice corpus 2010 includes a plurality of languages and a plurality of voice qualities, but a single language single voice quality may be used. In that case, the effect of voice quality customization of voice synthesis can be obtained.

[Embodiment 4]
Hereinafter, the fourth embodiment according to the present invention will be described with reference to FIG.
FIG. 11 is a diagram showing a process flow and a data flow for constructing a system for synthesizing speech with respect to input text using a speech corpus converted by voice quality conversion.
The process of this embodiment includes two stages of a speech synthesis system construction process and a speech synthesis process.

First, in the speech synthesis system construction process, the language feature analysis process 4001 is performed from the multilingual speech corpus 4010 (denoted as speech corpus 4010-1, voice corpus 4010-2, ... In FIG. 10) having different languages and voice qualities. The language feature amount 4020 is generated by the above, and the voice feature amount 4030 of the recorded voice is generated by the voice feature amount analysis process 4002.
Then, the multilingual speech synthesis system 5000 is constructed by the synthesis system construction process 4000 that uses them as inputs.

Next, in the speech synthesis processing, the input text 5010 is input to the speech synthesis system 5000, the synthetic speech 5020 is generated, and the synthetic speech 5020 is input, and the target is performed by the voice quality conversion process 150 of the voice quality conversion device 100 of the first embodiment. The synthetic voice 5030 of the voice quality Z to be output is output.

In the present embodiment, it is possible to output a synthetic voice of voice quality Z having a unified voice quality corresponding to the input text from the multilingual voice corpus 4010 having different languages and voice qualities.

In the description of the present embodiment, it is assumed that the input multilingual voice corpus 2010 includes a plurality of languages and a plurality of voice qualities, but a single language single voice quality may be used. In that case, the effect of voice quality customization of voice synthesis can be obtained.

Claims (8)

A voice quality conversion system that outputs synthetic voice that is obtained by converting voice quality from input voice using an information processing device.
A data creation device for voice quality conversion and
Equipped with a voice conversion device
The voice quality conversion data creation device is
A PPG (phoneme posterior probability) model learning unit that generates a PPG conversion model by inputting a word dictionary, a voice corpus that associates text with voice information, and
It consists of a voice corpus and a voice parameter generation model learning unit that inputs the PPG conversion model and generates a voice parameter generation model.
The PPG conversion model learning unit
A dictionary with prosodic information is generated from the word dictionary,
The text contained in the speech corpus is morphologically analyzed, and a phoneme sequence with prosody information is generated based on the result of the morphological analysis and the dictionary with prosody information.
An acoustic model is generated from the phoneme sequence with prosodic information and the speech features output as a result of the feature analysis of the speech information included in the speech corpus.
The acoustic model is trained to generate the PPG conversion model, and the PPG conversion model is generated.
The voice parameter generation model learning unit
From the PPG conversion model and the voice corpus, a PPG corresponding to the PPG conversion model is generated.
Voice parameters are extracted from the voice information included in the voice corpus, and
By learning the generated PPG and the voice parameter, a voice parameter generation model is generated.
The voice quality conversion device inputs the input voice, the PPG conversion model generated by the PPG conversion model learning unit, and the voice parameter generation model generated by the voice parameter generation model learning unit.
PPG is generated based on the voice feature amount output as a result of the feature amount analysis of the voice information included in the voice corpus and the PPG conversion model.
A voice parameter is generated based on the generated PPG and the voice parameter generation model.
A voice quality conversion system characterized in that a voice waveform based on the voice parameters is generated and output as output voice. The voice quality conversion system according to claim 1, wherein there are a plurality of PPG conversion models generated by the PPG conversion model learning unit for each language. The voice quality conversion system according to claim 2, wherein a prosodic symbol is added based on the characteristics of each language in generating the phoneme sequence with prosody information in the PPG conversion model learning unit. The PPG conversion model learning unit morphologically analyzes the text of the voice corpus, performs language model learning, generates a language model, and then generates a language model.
The voice quality conversion system according to claim 1, wherein the PPG conversion model is generated by learning the acoustic model and the language model. It is a voice quality conversion method that performs voice quality conversion by a voice quality conversion system that outputs synthetic voice obtained by converting voice quality from input voice by an information processing device.
The voice quality conversion system
A data creation device for voice quality conversion and
Equipped with a voice conversion device
The PPG (phoneme posterior probability) model learning step in which the voice quality conversion data creation device inputs a word dictionary and a voice corpus in which text and voice information are associated to generate a PPG conversion model, and
The voice quality conversion data creation device has a voice corpus and a voice parameter generation model learning step for inputting the PPG conversion model and generating a voice parameter generation model.
The PPG conversion model learning step
A step in which the voice quality conversion data creation device generates a dictionary with prosodic information from the word dictionary,
A step in which the voice quality conversion data creation device performs morphological analysis of the text contained in the voice corpus and generates a phoneme sequence with prosody information based on the result of the morphological analysis and the dictionary with prosody information.
A step in which the voice quality conversion data creation device generates an acoustic model from the phoneme array with prosodic information and the voice feature amount output as a result of the feature amount analysis of the voice information included in the voice corpus.
The voice quality conversion data creation device includes a step of learning the acoustic model and generating the PPG conversion model.
The voice parameter generation model learning step
A step in which the voice quality conversion data creation device generates a PPG corresponding to the PPG conversion model from the PPG conversion model and the voice corpus.
A step in which the voice quality conversion data creation device extracts voice parameters from voice information included in the voice corpus, and
The voice quality conversion data creation device includes a step of learning the generated PPG and the voice parameter to generate a voice parameter generation model.
A step in which the voice quality conversion device inputs the input voice, a PPG conversion model generated by the PPG conversion model learning step, and a voice parameter generation model generated by the voice parameter generation model learning step.
A step in which the voice quality conversion device generates a PPG based on a voice feature amount output as a result of a feature amount analysis of voice information included in the voice corpus and the PPG conversion model.
A step of generating a voice parameter based on the generated PPG and the voice parameter generation model by the voice quality conversion device.
A voice quality conversion method, characterized in that the voice quality conversion device includes a step of generating a voice waveform according to the voice parameter and outputting it as an output voice. The voice quality conversion method according to claim 5, wherein there are a plurality of PPG conversion models generated by the PPG conversion model learning step for each language. The voice quality conversion method according to claim 6, wherein a prosodic symbol is added based on the characteristics of each language in generating the phoneme sequence with prosody information in the PPG conversion model learning step. The PPG conversion model learning step includes a step of morphologically analyzing the text of the voice corpus, a step of performing language model learning based on the result of the morphological analysis, and a step of generating a language model.
The voice quality conversion method according to claim 5, further comprising a step of generating the PPG conversion model by learning the acoustic model and the language model.

Images