JP2021189402A - Voice processing program, voice processing device and voice processing method - Google Patents

Abstract

To appropriately convert a sound uttered by any speaker to a sound quality of an uttered voice of a target speaker.SOLUTION: A voice processing device 100 includes: a voice analysis unit 20 which receives a first voice; a phoneme conversion unit 24 which extracts a first phoneme post-probability based on the first voice; a normalization unit 26 which generates a second phoneme post-probability obtained by normalizing the first phoneme post-probability for each hour; a feature amount conversion unit 28 which generates a feature amount of the first voice based on the second phoneme post-probability; and a voice generation unit 30 which generates a second voice based on the feature amount.SELECTED DRAWING: Figure 2

Description

The present invention relates to a voice processing program, a voice processing device, and a voice processing method.

A voice processing device has been developed that converts a voice uttered by an arbitrary speaker into a voice having the voice quality of another speaker.

For example, a technique for creating a synthetic sound by performing model learning using parallel data having the same or similar utterance content as the voice signal of the target speaker is disclosed (Patent Document 1). Further, a technique for converting the voice of an arbitrary speaker into the voice quality of a target speaker by applying machine learning is disclosed (Non-Patent Document 1).

Japanese Unexamined Patent Publication No. 2017-151230

L.Sun, et. Al. "Phonetic Posterior for many-to-one voice conversion without parallel data training" in Proc. ICME, Seattle, USA, Jul. 2016.

By the way, in the prior art, the voice quality and wording of the synthesized voice may depend on the original speaker. That is, one example of the problem is that it was not possible to synthesize a voice sufficiently similar to the voice quality of the target speaker from the voice data of any speaker.

One aspect of the present invention is a first step of receiving a first voice from a computer, a second step of extracting a first phoneme posterior probability based on the first voice, and a time of the first phoneme posterior probability. The third step of generating the second phoneme posterior probability normalized for each, the fourth step of generating the feature amount of the first voice based on the second phoneme posterior probability, and the second step based on the feature amount. It is a voice processing program characterized by executing a fifth step of generating voice.

Here, in the third step, the second phoneme posterior probability is generated so that at least one of the maximum value, the average value, and the variance value of the phoneme probability becomes a constant value for the first phoneme posterior probability every hour. Is preferable.

Further, it is preferable that the feature amount is a spectral envelope.

Further, it is preferable to include a sixth step of extracting the first fundamental frequency from the first voice, and the fourth step preferably generates the feature amount based on the second phoneme posterior probability and the first fundamental frequency. Is.

Further, it is preferable to calculate the second fundamental frequency from the first fundamental frequency based on the gradient of the first fundamental frequency and generate the feature amount based on the second fundamental posterior probability and the second fundamental frequency. Is.

One aspect of the present invention is a voice acquisition means for receiving the first voice, a phoneme post-probability extraction means for extracting the first phoneme post-probability based on the first voice, and the first phoneme post-probability for each hour. A normalization means for generating the second phoneme post-probability normalized to, a feature amount generating means for generating the feature amount of the first voice based on the second phoneme post-probability, and a second feature amount based on the feature amount. It is a voice processing apparatus including a voice generation means for generating voice.

One aspect of the present invention is a first step of receiving a first voice, a second step of extracting a first phoneme posterior probability based on the first voice, and a normal first phoneme posterior probability for each hour. The third step of generating the converted second phoneme posterior probability, the fourth step of generating the feature amount of the first voice based on the second phoneme posterior probability, and the second voice generation based on the feature amount. This is a voice processing method characterized by comprising the fifth step.

INDUSTRIAL APPLICABILITY According to the present invention, it is possible to provide a voice processing program, a voice processing device, and a voice processing method for appropriately converting a voice uttered by an arbitrary speaker into the sound quality of a voice uttered by a target speaker. Other objects of the embodiments of the present invention will become apparent by reference to the entire specification.

It is a figure which shows the structure of the voice processing apparatus in embodiment of this invention. It is a functional block diagram which shows the structure of the voice processing apparatus in embodiment of this invention. It is a figure which shows the extraction example of the 1st fundamental frequency in embodiment of this invention. It is a figure which shows the extraction example of the utterance information in embodiment of this invention. It is a figure explaining the normalization process of the phoneme posterior probability in embodiment of this invention.

As shown in FIG. 1, the voice processing device 100 according to the embodiment of the present invention includes a processing unit 10, a storage unit 12, an input unit 14, an output unit 16, and a communication unit 18. The processing unit 10 includes means for performing arithmetic processing such as a CPU. The processing unit 10 realizes the function related to voice processing in the present embodiment by executing the voice processing program stored in the storage unit 12. The storage unit 12 includes storage means such as a semiconductor memory and a memory card. The storage unit 12 is accessiblely connected to the processing unit 10 and stores a voice processing program and information necessary for the processing. The input unit 14 includes means for inputting information. The input unit 14 includes, for example, a keyboard, a touch panel, buttons, and the like that receive input of information from the user. Further, the input unit 14 includes voice input means for receiving voice input of an arbitrary speaker and a target predetermined speaker. The voice input means may be configured to include, for example, a microphone, an amplifier circuit, or the like. The output unit 16 includes a user interface screen (UI) for receiving input information from the administrator and means for outputting the processing result. The output unit 16 includes, for example, a display for presenting an image. Further, the output unit 16 includes a voice output means for outputting the synthetic voice generated by the voice processing device 100. The audio output means may be configured to include, for example, a speaker, an amplifier, and the like. The communication unit 18 includes an interface for communicating information with an external terminal (not shown) via the network 102. The communication by the communication unit 18 may be wired or wireless. The voice information used for voice processing may be acquired from an external terminal via the communication unit 18.

In the present embodiment, voice processing is performed to convert the voices emitted by a plurality of speakers into the sound quality of the voices of a predetermined speaker (target speaker). FIG. 2 is a functional block diagram showing the configuration of the voice processing device 100. The voice processing device 100 functions as a voice analysis unit 20, an intonation extraction unit 22, a phoneme conversion unit 24, a normalization unit 26, a feature amount conversion unit 28, and a voice generation unit 30.

The voice analysis unit 20 performs a process of acquiring voice data. That is, the processing unit 10 of the voice processing device 100 functions as the voice analysis unit 20. The voice data may be acquired by converting the voice of the speaker into data using a microphone constituting the input unit 14. Further, the voice data recorded in advance in an external computer or the like may be received via the communication unit 18. The acquired voice data is stored in the storage unit 12.

The voice data acquisition process is performed on both the voice emitted by an arbitrary speaker and the voice emitted by the target speaker. The voice from any speaker and the voice from the target speaker do not have to have the same content (so-called parallel training data). However, if the voice from any speaker and the voice from the target speaker have the same content (parallel training data), there is a high possibility that the voice conversion process will be performed more appropriately.

In addition, the voice analysis unit 20 further performs voice analysis necessary for voice processing. For example, the voice analysis unit 20 performs voice cepstrum analysis based on the frequency characteristics of the input voice, and has a meta-frequency cepstrum coefficient including spectral envelopes (information indicating voice thickness and the like) and fine structure information. (MFCC), voice data such as basic voice frequency and resonance frequency (information indicating voice pitch, voice faintness, etc.) are obtained. Hereinafter, the fundamental frequency of the voice data obtained by the voice analysis unit 20 will be referred to as the first fundamental frequency F0.

The intonation extraction unit 22 calculates the slope ΔF0 of the first fundamental frequency F0 obtained from the voice by the voice analysis unit 20. FIG. 3 shows an example of the first fundamental frequency F0 extracted from the voice for each time frame. The intonation extraction unit 22 calculates the slope ΔF0 of the first fundamental frequency F0 extracted from the voice. The slope ΔF0 of the first fundamental frequency F0 is input to the feature amount conversion unit 28. The slope ΔF0 of the first fundamental frequency F0 is used as one of the calculation methods of the second fundamental frequency F1 in the feature amount conversion unit 28 described later.

The phoneme conversion unit 24 performs a process of extracting utterance information from voice data. That is, the processing unit 10 of the voice processing device 100 functions as a phoneme conversion unit 24 by performing a process of extracting utterance information from the voice data output from the voice analysis unit 20. Here, the utterance information is information including phoneme posterior probabilities (PPG: Phonetic Posterior Grams) or phonemes (Phoneme).

The processing unit 10 performs machine learning so as to be a phoneme conversion unit 24 that extracts and outputs phoneme posterior probabilities or phonemes from voice data, and constitutes a phoneme conversion unit 24. More specifically, the learning device uses voice data, which is time-series information that changes according to time, as input data, and extracts and outputs phoneme posterior probabilities or phonemes of the voice data as illustrated in FIG. To learn. As shown in FIG. 4, the phoneme posterior probability is information indicating the phoneme probability for each time frame.

For example, a convolutional neural network (CNN) is applied to configure a sound element conversion unit 24 that extracts and outputs sound element posterior probabilities or sound elements from input voice data. A convolutional neural network expresses processing equivalent to image convolution by using a technique called weight sharing, which limits the coupling well without fully coupling the perceptrons, within the framework of the neural network. .. Further, for example, a recurrent neural network (RNN) may be applied to configure a phoneme conversion unit 24 that extracts phoneme posterior probabilities or phonemes from input voice data and outputs them. Further, another neural network such as a pooling layer may be further incorporated in the phoneme conversion unit 24.

Reference 1: "Yuki Saito, Kei Akuzawa, Kentaro Tachibana," Simultaneous hostile learning of speech recognition and generative models for many-to-one speech conversion using phoneme posterior probabilities, "Acoustical Society of Japan 2019 Fall Study" Presentation Proceedings, 2-4-2, pp. 963--966, September 2019 "http: // sound. org / papers / ASJ / saito2019 asja_dena. The network architecture described in "pdf" may be applied. That is, it is learned to apply a Dave neural network (DNN) to predict a phoneme label sequence from an input speech feature quantity sequence included in speech data such as MFCC. The pair of input voice features and phoneme labels is extracted from a voice corpus containing a large number of speakers. The phoneme posterior probability series represents the phoneme posterior probabilities given the input speech features. Phoneme converter 24 is learned phoneme recognition error defined as softmax cross-entropy L _SCE phoneme labels and the phoneme posterior probabilities to minimize. At this time, it is preferable to apply a learning method for improving the speaker invariance of the phoneme posterior probability input to the phoneme conversion unit 24 based on the Domain-adversarial training (DAT). That is, it is preferable to define the voice corpus used for learning of the phoneme conversion unit 24 as a majority speaker domain and a target speaker domain, respectively, and to alleviate the difference between these two domains by DAT. Specifically, the DNN architecture of the speech recognition model shown in Table 1 of Reference 1 may be applied. (Reference 2: CLDNN: Tara N. Sainath, Oriol Vinyals, Andrew Senior, and Hasim Sak. Convolutional, Long ShortTerm Memory, Fully Connected Deep Neural Networks. In IEEE International Conference on Acoustics, Speech and Signal Processing, 2015. References 3: LAS: Chan, W., Jaitly, N., Le, QV, & Vinyals, O. (2015). Listen, attend and spell. ArXiv preprint arXiv: 1508.01211. Reference 4: Transformer-ASR: Karita, S ., Chen, N., Hayashi, T., Hori, T., Inaguma, H., Jiang, Z., ... & Watanabe, S. (2019). A comparative study on transformer vs rnn in speech applications. arXiv preprint arXiv: 1909.06317.)

It is preferable to apply the activation function to convolutional neural networks, recurrent neural networks, or combinations thereof. As the activation function, for example, ReLU, sigmoid function, softmax function, polynomial and the like can be applied. Further, the loss function may be applied according to the activation function. As the loss function, a softmax function or a connectionist temporal classification (CTC) loss function may be used.

The normalization unit 26 performs a process of normalizing the utterance information extracted by the phoneme conversion unit 24. That is, the processing unit 10 of the voice processing device 100 functions as the normalization unit 26 by normalizing the utterance information output from the phoneme conversion unit 24.

The processing unit 10 generates a second phoneme posterior probability by normalizing a phoneme posterior probability (PPG) indicating a probability for each phoneme in each unit as a first phoneme posterior probability. FIG. 5 shows an example of first phoneme posterior probabilities extracted from the voices of two different females. The maximum value of the first phoneme posterior probability in each time frame is different. That is, as shown in the figure, the maximum value of the first phoneme posterior probability obtained from the voice emitted by female 1 and the maximum value of the first phoneme posterior probability obtained from the voice emitted by female 2 are different. In this way, because the posterior probabilities are not aligned in each time frame, the quality of the voice deteriorates when the voice emitted by a plurality of speakers is converted into the sound quality of the voice of a predetermined speaker (target speaker). It is considered to be a factor that causes.

Therefore, in the present embodiment, the processing for normalizing the utterance information extracted by the phoneme conversion unit 24 is performed. For example, it is preferable to normalize so that the maximum value of the posterior probability of a phoneme for each time frame becomes a predetermined constant value. Specifically, by dividing the posterior probability of each phoneme by the maximum value for each time frame, it can be normalized so that the maximum value of the phoneme for each time frame becomes 1. Further, for example, it is preferable to normalize so that the average value of the posterior probabilities of phonemes for each time frame becomes a predetermined constant value. Further, for example, it is preferable to normalize so that the variance value of the posterior probability of the phoneme for each time frame becomes a predetermined constant value. Specifically, for each time frame, the posterior probability Y is calculated by applying the transformation of the equation (1) to the posterior probability X of each phonetic element using the average value μ and the standard deviation σ. It can be normalized so that the mean value of Y is 0 and the variance value is 1.

(Number 1)
Y = (X−μ) / σ ・ ・ ・ (1)

In this way, by normalizing the first phoneme posterior probability in the normalization unit 26 and calculating the second phoneme posterior probability, the feature quantity conversion of the second phoneme posterior probability similarly normalized regardless of the speaker. It can be input to the unit 28. That is, it is possible to generate a voice based on the second phoneme posterior probability in which the posterior probability distributions are similar among different speakers.

The feature amount conversion unit 28 performs a process of generating voice data of the target speaker using the slope ΔF0 of the first fundamental frequency F0 obtained by the intonation extraction unit 22 and the utterance information generated by the phoneme conversion unit 24 as input data. .. That is, the processing unit 10 of the voice processing device 100 receives speech data including phoneme post-probability (PPG: Phonetic PosterGrams) or phoneme (Phoneme) generated by the phoneme conversion unit 24 and voice data from the inclination ΔF0 of the first basic frequency F0. Functions as a feature amount conversion unit 28 by performing a process of reconstructing.

In addition to the slope ΔF0, a standardized first fundamental frequency F0 or a quantized first fundamental frequency F0 may be applied so that the average value is 0 and the variance value is 1. For the quantization process, an existing coding algorithm such as μ-law may be used, or the algorithm may be divided based on the cumulative density function of the first fundamental frequency F0. Further, the slope ΔF0 indicating a dynamic value is not limited, and only the first fundamental frequency F0 indicating a static value may be used, or both may be used.

The processing unit 10 performs machine learning so as to generate voice data of the voice quality of the target speaker based on the time-series information of the gradient ΔF0 of the first fundamental frequency F0 and the second phoneme posterior probability, and performs the feature quantity conversion unit 28. Configure. For example, by applying a convolutional neural network (CNN), time-series information of the slope ΔF0 of the first basic frequency F0 and the second sound posterior probability is used as input data to obtain voice data having the voice quality of the target speaker. A feature amount conversion unit 28 that is generated and output is configured. A convolutional neural network expresses processing equivalent to image convolution by using a technique called weight sharing, which limits the coupling well without fully coupling the perceptrons, within the framework of the neural network. .. Further, for example, a recurrent neural network (RNN) may be applied to configure the feature amount conversion unit 28. Further, another neural network such as a pooling layer may be further incorporated in the feature amount conversion unit 28. It is preferable to apply the activation function to convolutional neural networks, recurrent neural networks, or combinations thereof. As the activation function, for example, ReLU, sigmoid function, softmax function, polynomial and the like can be applied.

Further, the feature quantity conversion unit 28 may be configured by applying the technique described in Reference 1. Specifically, the architecture of the speech generation model shown in Table 1 of Reference 1 may be applied.

The voice loss function used for learning is a boolean value indicating whether or not the voice data generated by the feature amount conversion unit 28 is voice data generated from the voice of the target speaker (for example, the voice of the target speaker). If it is data, it may be 1, and if it is not the voice data of the target speaker, it may be 0). It may be a value.

The processing unit 10 uses the generated voice data and the voice data extracted in advance from the voice of the target speaker as input data, and the voice data generated by the feature amount conversion unit 28 is extracted from the voice of the target speaker. Machine learning of the feature amount conversion unit 28 is performed using a voice loss function indicating whether the data is voice data. A convolutional neural network (Conv) can be applied to generate a voice loss function. The convolutional neural network layer is a layer that expresses processing equivalent to image convolution by using a technique called weight sharing, which limits the coupling well without fully coupling the perceptrons together, within the framework of the neural network. be. Further, for example, a recurrent neural network (RNN) may be applied. Further, another neural network such as a pooling layer may be incorporated. It is preferable to apply the activation function to convolutional neural networks, recurrent neural networks, or combinations thereof. As the activation function, for example, ReLU, sigmoid function, softmax function, polynomial and the like can be applied. Further, as the voice loss function, it is preferable to use the L1 norm or the L2 norm of the error between the generated feature amount and the feature amount extracted from the real voice. Further, when GAN is applied, the hostile loss function may be based on the least squares error or may be a sigmoid function. At this time, the architecture of the speaker authentication model shown in Table 1 of Reference 1 may be applied.

It should be noted that the feature quantity conversion unit is a combination of the voice data generated based on the time series information of the second phoneme posterior probability and the second fundamental frequency F1 generated based on the gradient ΔF0 of the fundamental frequency F0 or the first fundamental frequency F0. The process is defined as a process of outputting a voice loss function indicating whether the voice data generated by 28 is voice data generated from the voice from the target speaker, but the present invention is not limited to this. That is, even if the truth value or the likelihood value is calculated from the difference between the voice data generated based on the time series information of the second phoneme posterior probability and the voice data generated from the voice from the target speaker. good. Then, the feature amount conversion unit 28 receives the feedback of the voice loss function, uses the voice loss function as one of the input data, and bases the time series information of the second sound element posterior probability so that the voice loss function becomes smaller. Machine learning of the feature quantity conversion unit 28 may be performed so as to generate voice data according to the voice quality of the target speaker. Further, the truth value and the likelihood value are obtained from the difference between the second fundamental frequency F1 generated based on the slope ΔF0 of the fundamental frequency F0 or the first fundamental frequency F0 and the fundamental frequency F0 obtained from the voice from the target speaker. May be calculated. Then, the feature amount conversion unit 28 receives the feedback of the voice loss function indicating the degree of matching output from the speaker matching identification unit 32, and the voice loss function is used as one of the input data, and the voice loss function becomes smaller. As described above, the feature quantity conversion unit 28 performs machine learning so as to generate voice data including information of the second basic frequency F1 according to the voice quality of the target speaker based on the inclination ΔF0 of the first basic frequency F0. May be good. As described above, the voice data including the voice data obtained from the time-series information of the second phoneme posterior probability and the information of the second fundamental frequency F1 according to the voice quality of the target speaker based on the inclination ΔF0 of the first fundamental frequency F0. May be combined and output to the voice generation unit 30.

In addition, the second fundamental frequency F1 is arithmetically calculated by adding the amount of change based on the slope ΔF0 of the first fundamental frequency F0 extracted from the voice of an arbitrary speaker to the fundamental frequency of the target speaker's voice. It may be calculated. Further, instead of the slope ΔF0, a standardized first fundamental frequency F0 or a quantized first fundamental frequency F0 may be used so that the average value is 0 and the variance value is 1. For the quantization process, an existing coding algorithm such as μ-law may be used, or the algorithm may be divided based on the cumulative density function of the first fundamental frequency F0. Further, the slope ΔF0 indicating a dynamic value is not limited, and only the first fundamental frequency F0 indicating a static value may be used, or both may be used. Then, the voice data of the target speaker may be generated by inputting the second fundamental frequency F1 as input data to the feature amount conversion unit 28 together with the time series information of the second phoneme posterior probability.

The voice generation unit 30 converts the voice data generated by the feature amount conversion unit 28 into voice and outputs it. The feature amount conversion unit 28 is learned to convert the utterance information extracted by the phoneme conversion unit 24 by the hostile generation network (GANs) with the speaker matching identification unit 32 into the voice data of the target speaker's voice quality. Therefore, the voice generated by the voice generation unit 30 is a voice having the voice quality of the target speaker.

As described above, according to the voice processing device 100 of the present embodiment, the voice processing device and the voice processing program appropriately convert the voice uttered by an arbitrary speaker into the sound quality of the voice uttered by the target speaker. Can be provided.

10 Processing unit, 12 Storage unit, 14 Input unit, 16 Output unit, 18 Communication unit, 20 Speech analysis unit, 22 Inflection extraction unit, 24 Phoneme conversion unit, 26 Normalization unit, 28 Feature quantity conversion unit, 30 Speech generation unit , 100 voice processors, 102 networks.

Claims (7)

On the computer
The first step of receiving the first voice and
The second step of extracting the first phoneme posterior probability based on the first voice, and
The third step of generating the second phoneme posterior probability obtained by normalizing the first phoneme posterior probability for each hour,
The fourth step of generating the feature amount of the first voice based on the second phoneme posterior probability, and
The fifth step of generating the second voice based on the feature amount, and
A voice processing program characterized by executing. The voice processing program according to claim 1.
The third step is characterized in that the second phoneme posterior probability is generated so that at least one of the maximum value, the average value, and the variance value of the phoneme probability becomes a constant value for the first phoneme posterior probability every hour. Voice processing program. The voice processing program according to claim 1 or 2.
The feature quantity is a speech processing program characterized by being a spectral envelope. The voice processing program according to any one of claims 1 to 3.
A sixth step of extracting the first fundamental frequency from the first voice is provided.
The fourth step is a speech processing program characterized in that the feature amount is generated based on the second phoneme posterior probability and the first fundamental frequency. The voice processing program according to claim 4.
It is characterized in that a second fundamental frequency is calculated from the first fundamental frequency based on the gradient of the first fundamental frequency, and the feature amount is generated based on the second phoneme posterior probability and the second fundamental frequency. Voice processing program. The voice acquisition means for receiving the first voice,
A phoneme posterior probability extraction means for extracting a first phoneme posterior probability based on the first voice,
A normalization means for generating a second phoneme posterior probability in which the first phoneme posterior probability is normalized for each hour,
A feature amount generating means for generating a feature amount of the first voice based on the second phoneme posterior probability, and a feature amount generating means.
A voice generation means for generating a second voice based on the feature amount,
A voice processing device characterized by being provided with. The first step of receiving the first voice and
The second step of extracting the first phoneme posterior probability based on the first voice, and
The third step of generating the second phoneme posterior probability obtained by normalizing the first phoneme posterior probability for each hour,
The fourth step of generating the feature amount of the first voice based on the second phoneme posterior probability, and
The fifth step of generating the second voice based on the feature amount, and
A voice processing method characterized by comprising.

Images