JP2020160319A - Voice synthesizing device, method and program - Google Patents

Abstract

To provide a voice synthesizing device capable of estimating voice characteristics of a specific speaker while ensuring a predetermined accuracy without necessarily using a large amount of clean voice which is restricted to be collected in a large amount.SOLUTION: A voice synthesizing device includes a first analysis unit 4 which analyzes a first voice which is determined to be clean by a specific speaker to obtain first speaker property information, a second analysis unit 5 which analyzes a second voice of the specific speaker to obtain second speaker property information, and a prediction unit 7 for predicting voice characteristics when the specific speaker utters a specific text by applying a learning model to the first speaker property information, the second speaker property information, and an intermediate expression for voice synthesis of the specified text.SELECTED DRAWING: Figure 1

Description

The present invention relates to a voice synthesizer, a method and a program capable of estimating the voice characteristics of a specific speaker with a predetermined accuracy without necessarily using a large amount of clean voice, which is restricted to collect a large amount. ..

Speech synthesis technology is a technique for artificially synthesizing speech. A typical usage method is text-to-speech. For example, in Japanese, the text input for TTS is usually a kanji-kana mixed sentence, for example, a voice to be synthesized with a character. It is difficult to directly map the features of the above because the structure of the relationship is extremely complicated. Therefore, using an abstracted intermediate expression, through two-step conversion of (1) text to intermediate expression and (2) intermediate expression to speech characteristics, a speech waveform that matches the information of speech features is signal-processed. A synthetic voice waveform can be obtained by selecting an appropriate waveform from the generation or the accumulation of the waveform prepared in advance.

(1) Intermediate representation As this intermediate representation, a speech synthesis symbol is assumed below. There may be various forms of speech synthesis symbols, but for example, information on phonemes constituting a series of speech and prosodic information mainly expressed as a pose or pitch of voice can be considered at the same time. That is, it is a symbol that describes a spoken language. As an example of such a symbol for speech synthesis, there is JEITA (Japan Electronics and Information Technology Industries Association) standard IT-4006 “symbol for Japanese text speech synthesis” (see Non-Patent Document 1). The voice synthesizer described below refers to a device that generates a voice waveform corresponding to the input by such a voice synthesis symbol.

(2) Obtaining voice features from intermediate representation Consider reproducing the voice of a specific speaker by such a voice synthesizer (hereinafter, the speaker who wants to reproduce is referred to as a target speaker). That is, it is considered to generate a synthetic speech having the characteristics of the speech of a specific speaker from the above-mentioned abstracted intermediate representation. Since the abstracted intermediate expression itself does not contain sufficient information to describe the characteristics of the speaker, simply, in advance, the physical characteristics of the target speaker's speech (waveform itself or signal processing from the waveform). The correspondence between the cepstrum (one of the expression methods of frequency spectrum information) and the voice feature amount such as the fundamental frequency) extracted by the above and the intermediate expression corresponding to the voice is investigated, and for each target speaker, A conversion rule from the intermediate expression to the speech feature may be defined, the speech feature may be predicted from the intermediate representation using the rule, and the speech synthesis waveform corresponding to the prediction result may be synthesized by signal processing. In general, the amount of information of only the intermediate representation that directly corresponds to each time is limited because the intermediate representation is abstract and simple, but it is before and after the time (still adjacent in time). More complex conversion rules can be defined by taking into account the intermediate representations (but not limited to). Such information other than the target time is generally also called a context. In the following, the intermediate representation shall include cases where such a context is also taken into consideration.

(3) Learning to automatically determine conversion rules Since it is difficult to manually determine all complicated conversion rules when context is taken into consideration, conversion from intermediate representation to voice features based on decision trees and neural networks. A method of defining rules by machine learning methods is widely used. In order to automatically determine the conversion rule (also called learning) with high accuracy, it is necessary to use a large amount of sets of voice data and an intermediate representation that describes it in consideration of the context as learning data. That is, in order to reproduce the voice of a specific speaker in speech synthesis, a large amount of voice data of the target speaker is collected, intermediate representations are manually added to all of the voice data, and the relationship between the two is learned by a machine learning method. Just do it. For example, in HMM speech synthesis, which is a speech synthesis method based on the Hidden Markov Model, the parameters of the HMM that models the speech corresponding to the intermediate expression are predicted from the intermediate expression by a decision tree, and the HMM of the prediction result is used for the speech. Is a method for synthesizing (see Non-Patent Document 2). Further, in many speech synthesis methods using a neural network (DNN) based on deep learning, the corresponding speech features are directly predicted from the intermediate representation, and speech is synthesized.

"Symbols for Japanese Text and Speech Synthesis" JEITA Standard IT-4006, Japan Electronics and Information Technology Industries Association, March 2010 Takashi Masuko, Keiichi Tokuda, Takao Kobayashi, Sei Imai, "Speech Synthesis Based on HMM Using Dynamic Features", IEICE Transactions (D-II), J79-D-II, 12, pp.2184- 2190, Dec. 1996. Najim Dehak, Patrick J. Kenny, Reda Dehak, Pierre Dumouchel, and Pierre Ouellet, "Front-End Factor Analysis for Speaker Verification," IEEE Transactions on Audio Speech and Language Processing 19 (4), pp.788-798, June 2011 ..

However, in the learning of the existing method as described above, there is a problem that a large amount of voice data is required for each specific speaker who wants to reproduce the voice, and the time and effort for learning becomes excessive. That is, in reality, it is often difficult to collect a large amount of voice data of a specific speaker to be reproduced by voice synthesis.

The following can be considered as a coping method for the problem of this prior art.

That is, the speaker property is described by some method that can be determined from a small amount of voice data, and the relationship between the speaker property information, the set of intermediate expressions, and the voice feature is machine-learned for the voice of a large number of speakers in advance. If a model learned by the method is prepared, it is possible to create a synthetic voice that simulates the voice of the speaker with at least the amount of voice data that can predict the speaker character information. (Although it is possible to call it an "intermediate representation" that includes speaker information, here it is assumed that the intermediate representation is basically composed of information obtained only from text analysis.)

Of course, in this method, it is necessary to model the complicated relationship between the speaker information, the intermediate representation, and the speech feature in advance, which makes it more difficult to learn the model prepared in advance. However, since it is not essential to include the voice of the synthetic target speaker in the model data prepared in advance, it is sufficient to learn a highly accurate model in advance.

On the other hand, in order to estimate speaker information with high accuracy, clean voice (with less background noise and reverberation) is generally required. In the extraction of speaker information, especially in the analysis of the spectrum of voice, the influence of background noise and echo is large, and if the speaker information is estimated from the voice including large background noise and echo, the accuracy of the estimation result is lowered. Is. And the effect will eventually reduce the speaker reproducibility.

The voice used for pre-learning also needs to be clean, but since this is the voice used when building a system, it can be collected relatively easily by recording the voice in a soundproof room or the like. On the other hand, it is generally not desirable to request a large amount of clean voice for the voice of the target speaker because it limits the voice recording environment of the user.

In view of the above problems of the prior art, the present invention does not necessarily need to use a large amount of clean voice, which is restricted to collect a large amount, and estimates the voice characteristics of a specific speaker while ensuring a predetermined accuracy. It is an object of the present invention to provide a voice synthesizer, a method and a program capable of providing a voice synthesizer.

In order to achieve the above object, the present invention is a voice synthesizer, the first analysis unit which analyzes the first voice determined to be clean of a specific speaker and obtains the first speaker character information, and the above. The second analysis unit that analyzes the second voice of a specific speaker to obtain the second speaker character information, the first speaker character information, the second speaker character information, and the voice synthesis of the designated text. It is characterized in that it includes an intermediate expression for use, and a prediction unit that predicts a voice feature when the specific speaker utters the text by applying a learning model to the text. Further, it is characterized in that it is a method and a program corresponding to the voice synthesizer.

According to the present invention, it is possible to predict the voice characteristics of a specific speaker with a predetermined accuracy even if the first voice is not always available in a large amount.

It is a functional block diagram of the voice synthesis apparatus which concerns on one Embodiment. It is a flowchart of the operation of the voice synthesizer which concerns on one Embodiment. It is a functional block diagram of the learning device which learns the learning model necessary for the prediction part to make a prediction. It is a functional block diagram of the speech synthesizer whose prediction part was composed of the 1st modification. It is a functional block diagram of the speech synthesizer whose prediction part was composed of the 2nd modification. It is a figure which shows the structure of a general computer.

FIG. 1 is a functional block diagram of the speech synthesizer 10 according to the embodiment. As shown in the figure, the speech synthesizer 10 includes a collection unit 1, an identification unit 2, a designation unit 3, a first analysis unit 4, a second analysis unit 5, an analysis unit 6, a prediction unit 7, and a generation unit 8. FIG. 2 is a flowchart of the operation of the voice synthesizer 10 according to the embodiment. Hereinafter, the outline of the processing contents of each part of the speech synthesizer 10 of FIG. 1 will be described while explaining each step of FIG.

When the flow of FIG. 2 is started, first, in step S1, a large amount of voice data as human utterance data is collected by the collection unit 1, and the speaker identification and speaker identification by the identification unit 2 with respect to the collected voice data. After performing the identification processing of the voice environment, the identification unit 2 saves the identified voice data, and proceeds to step S2.

The collection unit 1 is realized, for example, as a smart speaker (a type of smart home appliance) that can search information and operate network home appliances (smart home appliances) by voice operation on the dialogue side, and stores operation voices and the like. Alternatively, the voice recorded by such a smart speaker may be acquired via a network.

The identification unit 2 detects each section D (i) (i = 1,2, ...) In which a person is speaking from a large amount of voice data collected by the collection unit 1, and then each section. In the speaker x (x = a, b, c, ...) and the voice environment e (e = 1,2) of each section, e = 1 is "clean voice" and e = 2 is "unclean voice". ), And the identification results x = x (i) and e = e (i) are associated with the interval D (i) and saved.

The detection of the utterance section D (i) in the identification unit 2, the identification of the speaker x = x (i) in the section D (i), and the voice environment e = e (i) (clean voice) in the section D (i). Any existing method may be used for identification (whether or not). For example, a threshold value determination based on zero intersection, amplitude, or the like may be used to detect the utterance section. Alternatively, the user may be made to input whether or not the voice is clean by operating a switch or the like. For speaker identification, existing speaker recognition technology using i-vector or the like may be used, and for speaker x = a, b, c, ... As identification candidates, i-vector or the like is obtained in advance. All you have to do is register the voice of. The i-vector is disclosed in Non-Patent Document 3 described above. At this time, in order to avoid the influence of temporary voice quality change due to the influence of physical condition and the influence of apparent voice quality change due to the influence of background noise, a stricter judgment standard than the general speaker identification purpose is set. Therefore, the audio section that does not meet the criteria may be excluded from the processing target. As this determination criterion, a method similar to the method for estimating speaker information described later may be used.

Regarding the automatic identification of the voice environment e = e (i), the noise level other than the speaker's utterance due to the living environment sound or other causes is identified as e = 1 (clean voice). Those that do not satisfy the threshold value and are determined to have a noise level above a certain level may be identified as e = 2 (unclean voice). Any existing method such as spectrum analysis can be used to determine whether the noise is above the threshold.

The section where it is determined that a plurality of persons are uttering at the same time may not be included in the section D (i) saved by the identification unit 2, and the speaker x having the loudest volume may be included in the speaker identification result. And, the environment identification result may be given as e = 2 (unclean voice).

In one embodiment, the identification unit 2 may further identify the utterance content text t (i) in the section D (i) and save it in association with the section D (i). Any existing speech recognition method may be used to identify the text t (i).

In step S2, after receiving the designation of the speaker x who desires speech synthesis in the designation unit 3 and the designation of the text T which desires speech synthesis in the analysis unit 6 from the user who uses the speech synthesis device 10, step S3 Proceed to. Hereinafter, for the sake of explanation, it is assumed that the designation of the speaker x who desires speech synthesis is "x = a" (a is a speaker identifier).

In step S3, first, the designated unit 3 identifies the speaker x (i) and the voice environment e (i) in which the voice data of the speaker x = a specified in step S2 is stored in the identification unit 2. A large amount of utterance sections D (i) are searched and acquired, the first voice V1 (a) of these is output to the first analysis unit 4, and the second voice V2 (of these) is output to the second analysis unit 5. a) is output.

Specifically, among a large number of utterance sections D (i), it is identified that "x (i) = a", that is, the utterance is made by the designated speaker a, and the voice environment is "e". (i) = 1 ”, that is, all the voices identified as clean voices without noise are searched for as the first voice V1 (a), acquired, and output to the first analysis unit 4. Further, among a large number of utterance sections D (i), it is identified that "x (i) = a", that is, the utterance is made by the designated speaker a, and the voice environment is "e (i)". ) = 2 ”, that is, all the voices that are identified as unclean voices mixed with noise are searched for as the second voice V2 (a), acquired, and output to the second analysis unit 5. The first voice V1 (a) and the second voice V2 (a) are expressed by the set notation formula as follows.
V1 (a) = {D (i) | x (i) = a, e (i) = 1}
V2 (a) = {D (i) | x (i) = a, e (i) = 2}

In step S3, the first analysis unit 4 then analyzes the acquired first voice V1 (a), obtains the first speaker information s1 (a) of the speaker a, and outputs the first voice V1 (a) to the prediction unit 7. In addition, the acquired second voice V2 (a) is analyzed by the second analysis unit 5, the second speaker information s2 (a) of the speaker a is obtained, and the second voice V2 (a) is output to the prediction unit 7. Proceed to step S4.

If the identification unit 2 also acquires and links the information of the spoken text t (i) to each section D (i), the first analysis unit 4 and the second analysis unit 5 may use the information. When analyzing the first voice V1 (a) and the second voice V2 (a), by also using the linked text information, the first speaker information s1 (a) and the second voice V2 (a) can be used. Speaker information s2 (a) can be obtained respectively.

In addition, in order to suppress the influence of changes in voice quality, if the first speaker information s1 (a), which is the analysis result of the first analysis unit 4, is significantly different from the previous result, the first voice V1 (a) is discarded. It may be a procedure to end the process. The same applies to the second speaker information s2 (a), which is the analysis result of the second analysis unit 5. That is, under the premise that voice collection is continuously performed in the collecting unit 1, the first analysis unit 4 and the second analysis unit 5 continuously perform each period m (m = 1,2, ..., For example, 1). The first voice V1 (a) _[m] and the second voice V2 (a) _[m] newly collected in the weekly period, etc.) are also added to the first speaker information and the second speaker information. May be analyzed. At this time, the first speaker information s1 (a) _[m-1] and the second speaker have already been analyzed using all the voices up to m-1 in the past period (excluding those discarded) _. The analysis was performed using only the information s2 (a) _[m-1] and the first voice V1 (a) _[m] and the second voice V2 (a) _[m] newly collected during the current period m. When the 1-speaker information s1 (a) _{[m only]} and the 2nd speaker information s2 (a) _{[m only]} are compared and it is determined that the difference is greater than or equal to the threshold value, a new period m is used. The first voice V1 (a) _[m] and the second voice V2 (a) _[m] collected in the _above have a temporary change in voice quality (for example, a change in voice quality due to having a cold). As a matter of fact, the first speaker information and the second speaker information may be deleted from the objects used for analysis.

The details of the processing of the first analysis unit 4 and the second analysis unit 5 as the processing on the latter half of step S3 described above will be described later.

In step S4, the analysis unit 6 analyzes the text T specified in step S2, obtains the intermediate representation im (T), outputs the text T to the prediction unit 7, and then proceeds to step S5. For the analysis processing by the analysis unit 6, a rule-based method for text can be used, and the intermediate representation im (T) is previously described as, for example, "symbol for Japanese text speech synthesis" in Non-Patent Document 1 described above. The defined predetermined format may be used.

The order of step S3 and step S4 may be reversed, or both steps may be performed in parallel.

In step S5, the first speaker information s1 (a), the second speaker information s2 (a), and the intermediate representation im (T) obtained in steps S3 and S4 are used as inputs by the speaker a. After the prediction unit 7 predicts the voice feature f (a, T) when the text T is uttered and outputs it to the generation unit 8, the process proceeds to step S6.

For the voice features f (a, T) predicted by the prediction unit 7, any existing acoustic features based on the HMM model or the like may be used. For example, the static features based on the fundamental frequency and the mer cepstrum coefficient of a predetermined dimension and the static features A voice feature f (a, T) can be obtained as a frame time series along the text T of a combination (vector) of dynamic features such as a delta parameter.

Specific prediction by the prediction unit 7 can be performed using a model learned in advance, and the details will be described later.

In step S6, the synthetic voice waveform W in which the generation unit 8 performs the synthesis processing using the voice feature f (a, T) obtained in the prediction unit 7 in step S5 as an input and the speaker a utters the text T. (a, T) is output, and the flow of FIG. 2 ends. The synthetic speech waveform generation process may be performed by performing a predetermined waveform generation process according to the type of the speech feature f (a, T) predicted by the prediction unit 7, for example, by combining the fundamental frequency and the mer cepstrum coefficient of a predetermined dimension. When the speech characteristics are represented, a synthetic speech is created by creating a digital filter with a spectrum wrapping characteristic determined by the mer cepstrum coefficient and driving the above-mentioned digital filter with an impulse train having the fundamental frequency as its drive frequency. Can generate waveforms.

Each step of FIG. 2 has been described above. Hereinafter, the processing of the first analysis unit 4 and the second analysis unit 5 on the latter half side of step S3, which will be described in detail later, and the model learning for enabling the prediction unit 7 to make a prediction in step S5. The explanation will be given in this order.

Regarding the first analysis unit 4 and the second analysis unit 5, the concept of using these two separately will be described first. As a premise, it is difficult to acquire the clean first voice V1 (a) as an input to the first analysis unit 4 due to the restriction that it needs to be acquired in a quiet environment without noise, and it takes time and cost to acquire it. It is assumed that the amount of data is small due to the existence, while the unclean second voice V2 (a) as the input to the second analysis unit 5 can be acquired because there is no restriction that the acquisition environment is quiet. It is assumed that it is easy and the amount of data is abundant.

It is assumed that the second voice can be easily acquired in large quantities by the collecting unit 1, for example, as an operation voice for a smart home appliance in a daily noisy environment.

In this embodiment, such a precondition of available voice data is positively utilized, and the speaker character information that can be obtained only from the first voice V1 (a), which is only a small amount, is the first speaker character. The speech characteristics of the text T of the target speaker a can be efficiently obtained by obtaining the speaker information that can be obtained as information and can be obtained in large quantities from the second voice V2 (a) as the second voice information. It predicts f (a, T).

Here, the following first and second considerations can be made with respect to speaker information.

(First consideration)
Of the speaker information, for example, the spectral characteristics of voice are strongly affected by noise, reverberation, etc. in the voice recording environment, and therefore clean voice is required for highly accurate estimation. However, many of the spectral features are constrained by the shape of the human vocal organs and physiological motor conditions, so the model of the acoustic features of human speech generation is high from the voices of many different speakers. If the target speaker can be learned accurately, it is expected that the spectral characteristics of the target speaker can be estimated with high accuracy even if the target speaker voice (first speech set V1 (a)) is small.

(Second consideration)
On the other hand, the mode of power change of the speaker's voice for a long time (several syllables or more), the mode of fundamental frequency change, and the mode of change of characteristics in the time axis direction such as the duration of syllables (these are summarized below). , Syllable features) can be estimated from short-term audio data because the object is originally a long-term feature and the generation mechanism has a relatively large degree of freedom in how to change it. difficult. However, power change and fundamental frequency information can be extracted even in an environment containing some noise by comparing with the spectrum analysis required for voice reproduction. In addition, time information of phonemes and syllables by voice recognition technology is required to extract the change mode of the characteristics of the time axis method, but voice recognition itself does not require the spectral accuracy required for voice quality reproduction. That is, the prosodic feature can be estimated from the input voice information of the voice dialogue system (that is, the voice acquired in a noise environment such as the second voice V2 (a)).

Based on the above first and second considerations, in the first analysis unit 4, a small amount of clean first voice V1 (a) is converted into the text of the speaker a as the first speaker information s1 (a). Obtain a vector in which the generalized spectral features are described in an independent manner (or, based on the first discussion, a text-independent feature vector for features having characteristics similar to the spectral features). Further, in the second analysis unit 5, a large amount of unclean second voice V2 (a) was generalized as the second speaker information s2 (a) in a text-independent manner of the speaker a. Obtain a vector in which the prosodic features are described (or, based on the second consideration, a text-independent feature vector for features having characteristics similar to the prosodic features).

The first speaker information and the second speaker information are each defined by physical feature data (mean cepstrum, average fundamental frequency, and other physical quantities) corresponding to a plurality of impressions defined manually in advance. It may be directly configured with a multidimensional vector), or it has an i-vector or a bottleneck layer (an intermediate layer composed of fewer units than the number of units in the input layer and output layer) as a network structure. Statistical processing such as a value in the bottleneck layer (hereinafter referred to as a bottleneck feature quantity) for an input in a neural network that estimates a speaker (here, speaker ID information represented by a one-hot vector) from a feature. It may be a multidimensional vector defined by the method. Alternatively, it may be a vector obtained by combining both of them.

When the speaker information is obtained by the statistical processing method as described above, the first analysis unit 4 and the second analysis unit 5 (and the feature extraction unit 18 in FIG. 3 to be described later) that perform the processing are respectively. Is considered as a predictor that predicts speaker information from a relatively small amount of voice, and those predictors are learned in advance from another large amount of voice data. For example, in error evaluation in i-vector and bottleneck feature estimation (in the case of bottleneck feature, error evaluation when learning the entire neural network estimated by the speaker), the spectrum is heavy and the fundamental frequency is lightened. By learning, the predictor used by the first analysis unit 4 is learned, and conversely, by learning by making the spectrum lighter and the fundamental frequency heavier, the predictor used by the second analysis unit 5 is learned. Is possible.

In this way, in the first analysis unit 4 and the second analysis unit 5, any of the above-mentioned spectrum and the fundamental frequency (for example, any of the above spectrum and the fundamental frequency) according to the predefined first speaker information and the second speaker information. The error evaluation at the time of learning differs depending on the emphasis on, and the first voice V1 (a) and the second voice V2 (a) are analyzed by the method using different predictors obtained as a result, respectively, and the first analysis unit 4 In, the obtained first speaker information may be output to the prediction unit 7, and in the second analysis unit 5, the second speaker information may be output to the prediction unit 7.

FIG. 3 is a functional block diagram of the learning device 20 that learns the learning model M required for the prediction unit 7 to make a prediction. The learning device 20 outputs a learning model M by using a large amount of learning voice V [n] (n = 1,2, ...) As learning data as its overall operation. As shown in the figure, the learning device 20 includes a processing unit 13, a learning first analysis unit 14, a learning second analysis unit 15, a learning analysis unit 16, a learning unit 17, and a feature extraction unit 18.

Of these, the processing unit 13, the first analysis unit 14 for learning, the second analysis unit 15 for learning, the analysis unit 16 for learning, and the feature extraction unit 18 are the learning voice V [n] (n = 1,2, This is a configuration for preparing learning data L [n] (n = 1,2, ...) Used in the learning unit 17 from each of (...). The learning data L [n] includes the learning first speaker information s1 [n], the learning second speaker information s2 [n], the learning intermediate representation im (T [n]), and the learning. Voice features for f [n] (L [n] = {s1 [n], s2 [n], im (T [n]), f [n]} (n = 1,2,… )).

Each of the learning voices V [n] (n = 1,2, ...) is determined to be in a clean state (e = 1 (clean voice) when the voice environment is identified by the identification unit 2 of the voice synthesizer 10). It shall be prepared as the one (state to be done). The learning voice V [n] is read into the processing unit 13, the learning analysis unit 16, and the feature extraction unit 18, respectively.

The processing unit 13 outputs the learning voice V [n] as it is to the learning first analysis unit 14 as the learning first voice V1 [n], while superimposing noise on the learning voice V [n]. The applied voice is output to the learning second analysis unit 15 as the learning second voice V2 [n]. Here, noise superimposition may be performed to the extent that e = 2 (unclean voice) is determined when the voice environment of the second voice V2 [n] for learning is identified by the identification unit 2 of the voice synthesizer 10. However, for example, when a feature in which the tendency of error due to the influence of noise is random is used for speaker information estimation as seen in the estimation of the fundamental frequency, the processing unit 13 does not perform noise superimposition or performs processing. The clean voice may be used as it is as the second voice V2 [n] for learning without providing the processing unit 13.

The learning first analysis unit 14 performs the same processing as the learning first analysis unit 4 on the learning first voice V1 [n], and the same speaker among the learning first voice V1 [n]. The first speaker character information s1 [n] for learning is obtained by analyzing each set of voices of, and is output to the learning unit 17. The learning second analysis unit 15 performs the same processing as the learning second analysis unit 5 on the learning second voice V2 [n], and is the same speaker among the learning second voice V2 [n]. By analyzing each set of voices of, the second speaker character information s2 [n] for learning is obtained and output to the learning unit 17. These processes are performed for each speaker, and the same s1 [n] and s2 [n] are output for the voice of the same speaker, respectively. (That is, in the learning voice V [n], the speaker n is pre-identified by the label n by manual assignment and / or automatic identification, and the collection of the voices of the speaker n is the learning voice V [n]. Used as n], the learning first analysis unit 14 and the learning second analysis unit 15 output the learning first speaker information s1 [n] and the learning second speaker information s2 [n], respectively. You can do it.)

The learning analysis unit 16 recognizes the learning voice V [n] by voice, obtains the text T [n], and further performs the same processing as the analysis unit 6 of the speech synthesizer 10 to perform an intermediate expression im of this text. Find (T [n]) and output it to the learning unit 17. Any existing method may be used for speech recognition in the learning analysis unit 16. When the text T [n] is associated with the learning voice V [n] in advance by a voice corpus or the like, the learning analysis unit 16 may omit the acquisition of the text T [n] by voice recognition. Alternatively, in order to avoid the influence of speech recognition errors, an intermediate representation im (T [n]) corresponding to the output of the analysis unit 6 of the speech synthesizer 10 is manually created based on the manual listening of the learning speech. You may.

The feature extraction unit 18 analyzes the learning voice V [n], obtains the learning voice feature f [n] as data along the frame time series of this voice, and outputs it to the learning unit 17.

A feature similar to the learning voice feature f [n] output by the feature extraction unit 18 is the voice feature f (a, T) output by the prediction unit 7 of the speech synthesizer 10 described above. (Because the prediction unit 7 uses the model M learned by the learning device 20.) Therefore, as already explained in the prediction unit 7, this learning voice feature f [n] is also used by the existing method as the mer cepstrum coefficient or the like. You just have to ask.

The learning unit 17 has the learning data L [n] = {s1 [n], s2 [n], im (T [n]), f [n]} (n = 1,2,… obtained as described above. ) Is used to perform learning, and the model M used by the prediction unit 7 is output. Specifically, the first speaker information for learning s1 [n], the second speaker information for learning s2 [n], and the intermediate representation im (T [n]) for learning are input. The model M used by the predictor 7 in the speech synthesizer 10 that outputs the speech feature f [n] for learning is learned. In this model M, a decision tree using a set of the first speaker information s1 [n], the second speaker information s2 [n], and the intermediate representation im (T [n]) as explanatory variables, and similarly DNN (Deep Neural Network) that takes them as input can be used. By learning the model M based on a large amount of learning speech V [n] (and a corresponding large number of intermediate representations im (T [n])) for a large number of speakers n, the prediction unit 7 can also be arbitrary. It is expected that predictions about speaker a (and any text T) will be possible.

Various variations are possible in the composition of the prediction by the prediction unit 7. Hereinafter, a modified example will be described with reference to FIGS. 4 and 5.

FIG. 4 is a functional block diagram of the speech synthesizer 10 in which the prediction unit 7 is composed of the first modification, and the prediction unit 7 is composed of the first predictor 71 and the second predictor 72, and performs the corresponding processing. It is the same as the voice synthesizer 10 of FIG. 1 except that it is performed, and it is possible to operate according to the flow of FIG. That is, since the operations are the same as those described in FIGS. 1 and 2 except for the first predictor 71 and the second predictor 72 of the prediction unit 7, the configuration for inputting / outputting data to / from the first predictor 71, and the second predictor 72. Duplicate description is omitted.

In the first predictor 71, the speaker a utters the text T using the first speaker information s1 (a) obtained from the first analysis unit 4 and the intermediate representation im (T) obtained from the analysis unit 6. The first voice feature f1 (a, T) as a feature quantity frame time series is predicted and output to the generation unit 8. In the second predictor 72, the speaker a utters the text T using the second speaker information s2 (a) obtained from the second analysis unit 5 and the intermediate representation im (T) obtained from the analysis unit 6. The second voice feature f2 (a, T) as a feature quantity frame time series is predicted and output to the generation unit 8.

Here, the first voice feature f1 (a, T) output by the first predictor 71 is a time series of features having a high correlation with the first speaker information s1 (a) along the text T. Can be. For example, it is a feature quantity related to a spectrum. Further, the second voice feature f2 (a, T) output by the second predictor 72 is a time series of features having a high correlation with the second speaker information s2 (a) along the text T. can do. For example, it is a feature quantity related to the fundamental frequency.

In the generation unit 8, the first voice feature f1 (a, T) and the second voice feature f2 (a, T) are combined to obtain the voice feature f (a, T), and (the first voice feature f1 (a, T) If each frame of T) is P-dimensional and each frame of second voice feature f2 (a, T) is Q-dimensional, these are integrated to obtain a P + Q-dimensional vector.) This voice feature f (a) , T), the voice synthesis waveform W (a, T) can be obtained by the same processing as in the embodiment of FIG.

Regarding learning, the model M1 used by the first predictor 71 and the model M2 used by the second predictor 72 can be learned and obtained as follows, almost in the same manner as in the case of FIG. That is, using a large amount of clean or unclean learning voice V [n] (and corresponding text T [n]), the first speaker information s1 [n] and the second speaker information s2 are used as learning data. [n], intermediate expression im (T [n]), first voice feature f1 [n] and second voice feature f2 [n] are prepared, and model M1 uses the first speaker information s1 [n] and Learning is based on the clean learning voice V [n] as the first voice feature f1 [n] is output by inputting the intermediate expression im (T [n]), and the model M2 is the second speaker. It is possible to learn based on the unclean learning voice V [n] as the output of the second voice feature f2 [n] by inputting the information s2 [n] and the intermediate expression im (T [n]). Machine learning (including deep learning) may be used for learning as well. This method assumes that the first speaker information s1 [n] and the second voice feature f2 [n], and the second speaker information s2 [n] and the first voice feature f1 [n] are independent. Therefore, it is considered disadvantageous if there is a large amount of training data, but the amount of training data is limited because models M1 and M2, which are smaller than the model M in the predictor 7 in FIG. 1, can be trained independently. If so, it is valid.

FIG. 5 is a functional block diagram of the speech synthesizer 10 in which the prediction unit 7 is composed of the second modification, and the prediction unit 7 is composed of the third predictor 73 and the fourth predictor 74, and performs the corresponding processing. It is the same as the voice synthesizer 10 of FIG. 1 except that it is performed, and it is possible to operate according to the flow of FIG. That is, the operations are the same as those described in FIGS. 1 and 2 except for the third predictor 73 and the fourth predictor 74 of the prediction unit 7, the configuration for inputting / outputting data to and from the third predictor 73, and the fourth predictor 74. Duplicate description is omitted.

There are two possible embodiments of FIG. 5 with and without the flow of the line L1 (the flow of outputting the intermediate representation im (T) obtained from the analysis unit 6 to the fourth predictor 74). Hereinafter, the case where the line L1 is present will be described.

In the third predictor 73, the speaker a utters the text T using the second speaker information s2 (a) obtained from the second analysis unit 5 and the intermediate representation im (T) obtained from the analysis unit 6. The second voice feature f2 (a, T) as a feature quantity frame time series is predicted and output to the fourth predictor 74. The fourth predictor 74 includes the first speaker information s1 (a) obtained from the first analysis unit 4, the second voice feature f2 (a, T) obtained from the third predictor 73, and the analysis unit 6. Using the intermediate representation im (T) obtained from, the speech feature f (a, T) as the feature quantity frame time series when the speaker a utters the text T is predicted, and the text T is sent to the generator 8. Is output.

Here, the second voice feature f2 (a, T) output by the third predictor 73 is a time series of features highly correlated with the second speaker information s2 (a) along the text T. Can be. On the other hand, the speech feature f (a, T) output by the fourth predictor 74 needs to have a feature amount including all features necessary for speech synthesis as a time series along the text T.

Regarding learning, the model M3 used by the third predictor 73 and the model M4 used by the fourth predictor 74 can be learned and obtained as follows, almost in the same manner as in the case of FIG. That is, using a large amount of clean or unclean learning speech V [n] (and corresponding text T [n]), the first speaker information s1 [n] and the second speaker information s2 are used as learning data. Prepare [n], intermediate representation im (T [n]), first voice feature f1 [n], and second voice feature f2 [n].

The model M3 outputs the second speech feature f2 [n] by inputting the second speaker information s2 [n] and the intermediate expression im (T [n]), and outputs the unclean learning speech V [n]. Learning based on, the model M4 has the first speaker information s1 [n], the second speech feature f2 [n] (the second speech feature f2 [n] obtained from unclean speech) and the intermediate expression im. With (T [n]) as the input, as a vector element of the voice feature f [n] (the first voice feature f1 [n] obtained from clean voice and the second voice feature f2 [n] obtained from unclean voice. Can be learned as an output (combination of). Machine learning (including deep learning) may be used for learning as well. In this method, when the second speech feature f2 [n] includes the fundamental frequency of speech, the influence of the fundamental frequency of speech on the spectrum of speech is indirectly expressed via the intermediate representation im (T [n]). It is a structure that aims to model not only information but also directly.

Further, even when the line L1 is omitted in the embodiment of FIG. 5, it is possible to operate as follows.

In the third predictor 73, the speaker a utters the text T using the second speaker information s2 (a) obtained from the second analysis unit 5 and the intermediate representation im (T) obtained from the analysis unit 6. The second voice feature f2 (a, T) as a feature quantity frame time series is predicted and output to the fourth predictor 74. The fourth predictor 74 uses the first speaker information s1 (a) obtained from the first analysis unit 4 and the second voice feature f2 (a, T) obtained from the third predictor 73. , The voice feature f (a, T) as a feature amount frame time series when the speaker a utters the text T is predicted and output to the generation unit 8.

Here, in addition to the voice feature f2 (a, T) output by the fourth predictor 74, the second voice feature f2 (a, T) output by the third predictor 73 is also all the features required for speech synthesis. It is necessary to make the feature quantity including the above into a time series along the text T.

Regarding learning, the model M3 used by the third predictor 73 and the model M4 used by the fourth predictor 74 can be learned and obtained as follows, almost in the same manner as in the case of FIG. That is, using a large amount of clean or unclean learning speech V [n] (and corresponding text T [n]), the first speaker information s1 [n] and the second speaker information s2 are used as learning data. Prepare [n], intermediate representation im (T [n]), first voice feature f1 [n], and second voice feature f2 [n].

The model M3 outputs the second speech feature f2 [n] by inputting the second speaker information s2 [n] and the intermediate expression im (T [n]), and outputs the unclean learning speech V [n]. Learning based on this, the model M4 uses the first speaker information s1 [n] and the second voice feature f2 [n] (the second voice feature f2 [n] obtained from the unclean voice) as inputs. Learn as outputting feature f [n] (combination of first speech feature f1 [n] obtained from clean speech and second speech feature f2 [n] obtained from unclean speech as vector elements) be able to. Machine learning may be used for learning as well. The intermediate representation im (T [n]] is used when a larger amount of unclean voice is obtained, or when, for example, background noise level is low and spectrum-related speaker information can be obtained with high accuracy from unclean voice. ) Can be used to predict voice features with high accuracy using only the third predictor 73, and the input of the fourth predictor 74 does not include the intermediate representation im (T [n]). The input / output relationship becomes simpler, and the training of model M4 becomes easier.

In the above explanation, it is assumed that the speaker character information is requested for each speaker, but here the speaker does not have to be in line with the actual speaker. For example, utterances by the same person but with different emotional expressions and styles may be treated as voices of different speakers. By doing so, it is possible to synthesize sounds of various emotions and styles in the same way. On the contrary, by treating different speakers who are similar in the speaker property information as the same speaker, it is possible to increase the data for estimating the speaker property information and improve the estimation accuracy. Here, the similarity of speaker information can be defined, for example, in the form that the distance between vectors representing the speaker information is equal to or less than a predetermined value.

FIG. 6 is a diagram showing a hardware configuration in a general computer device 50, and the speech synthesizer 10 can be realized as one or more computer devices 50 having such a configuration. The computer device 50 is a CPU (central processing unit) 51 that executes a predetermined instruction, and a dedicated processor 52 (GPU (graphic calculation device)) that executes a part or all of the execution instructions of the CPU 51 on behalf of the CPU 51 or in cooperation with the CPU 51. RAM53 as the main storage device that provides a work area for the CPU 51 and the dedicated processor 52, ROM 54 as the auxiliary storage device, communication interface 55, display 56, microphone 57 and speaker 58, keyboard, mouse, etc. It is provided with an input interface 59 composed of a touch panel or the like and receiving operation input from a user, and a bus B for exchanging data between them.

Each part of the speech synthesizer 10 can be realized by a CPU 51 and / or a dedicated processor 52 that reads and executes a predetermined program corresponding to the function of each part from the ROM 54. Here, when display-related processing is performed, the display 56 operates in conjunction with the display 56, and when communication-related processing related to data transmission / reception is performed, the communication interface 55 further operates in conjunction with the voice recording. When the processing related to the above is performed, the microphone 57 operates in conjunction with each other, and when the processing related to audio reproduction is performed, the speaker 58 operates in conjunction with each other. For example, the synthetic voice obtained by the generation unit 8 may be reproduced and output from the speaker 58.

When the voice synthesizer 10 is realized as a system by two or more computer devices 50, the information required for each process may be transmitted and received via the network.

10 ... Speech synthesizer, 4 ... 1st analysis unit, 5 ... 2nd analysis unit, 7 ... Analysis unit, 8 ... Generation unit

Claims (10)

The first analysis unit that analyzes the first voice that is judged to be clean for a specific speaker and obtains the first speaker information,
A second analysis unit that analyzes the second voice of the specific speaker to obtain second speaker information, and
The specific speaker utters the text by applying a learning model to the first speaker information, the second speaker information, and the intermediate representation for voice synthesis of the designated text. A voice synthesizer including a prediction unit that predicts voice characteristics at the time of learning. The prediction unit
By applying a learning model to the first speaker information and the intermediate representation, the first voice feature when the specific speaker utters the text is predicted.
By applying the learning model to the second speaker character information and the intermediate representation, the second voice feature when the specific speaker utters the text is predicted.
The voice synthesizer according to claim 1, wherein the voice feature is obtained by combining the first voice feature and the second voice feature. The prediction unit
By applying the learning model to the second speaker character information and the intermediate representation, the second voice feature when the specific speaker utters the text is predicted.
The voice synthesizer according to claim 1, wherein the voice feature is obtained by applying a learning model to the first speaker information and the second voice feature. The prediction unit
By applying the learning model to the second speaker character information and the intermediate representation, the second voice feature when the specific speaker utters the text is predicted.
The voice synthesizer according to claim 1, wherein the voice feature is obtained by applying a learning model to the first speaker information, the intermediate representation, and the second voice feature. .. The speech synthesizer according to any one of claims 1 to 4, wherein the first speaker information includes information regarding spectral features, and the second speaker information includes information regarding prosodic features. Further equipped with a collecting unit for collecting operation voices of the specific speaker for smart home appliances,
The voice synthesizer according to any one of claims 1 to 5, wherein the second voice includes the operation voice. The voice synthesizer according to any one of claims 1 to 6, further comprising a generation unit that synthesizes voice using the voice feature. The first analysis unit and / or the second analysis unit continuously add and acquire new first voice and / or second voice, respectively, and acquire first speaker character information and / or second speaker. The sexual information is obtained, and the first speaker information and / or the second speaker information obtained by using only the newly acquired first voice and / or the second voice has already been obtained. If it is determined that the first speaker information and / or the second speaker information is different from the second speaker information in the threshold determination, the newly acquired first voice and / or the second voice is used as the first speaker information and / or. The voice synthesizer according to any one of claims 1 to 7, wherein the voice synthesizer is excluded from the target used for obtaining the second speaker information. The first analysis stage to analyze the first voice judged to be clean of a specific speaker and obtain the first speaker character information, and
The second analysis stage of analyzing the second voice of the specific speaker to obtain the second speaker information, and
The specific speaker utters the text by applying a learning model to the first speaker information, the second speaker information, and the intermediate representation for voice synthesis of the designated text. A voice synthesis method comprising a prediction stage for predicting voice characteristics at the time of learning. A program characterized in that a computer functions as the voice synthesizer according to any one of claims 1 to 8.

Images