KR102526338B1 - Apparatus and method for synthesizing voice frequency using amplitude scaling of voice for emotion transformation - Google Patents

Abstract

Disclosed is technology which relates to an apparatus for synthesizing a voice frequency using amplitude scaling of a voice for emotion transformation. The apparatus for synthesizing a voice frequency extracts a spectrum signal (SP) and a fundamental frequency signal (f0) from a source voice signal using an input vocoder, generates a spectrum signal and a fundamental frequency signal with emotion using an encoder and a generator trained in parallel using a VAW-GAN neural network model for each signal, and outputs an emotion-transformed voice signal using an output vocoder. Continuous wavelet transform (CWT) is applied to the fundamental frequency signal (f0) so as to reflect prosodic information, and amplitude scaling which reflects the amplitude of original emotion is applied to the spectrum signal with emotion so as to obtain a voice with highly-transformed emotional similarity.

Description

Apparatus and method for synthesizing voice frequency using amplitude scaling of voice for emotion transformation}

Data processing technology, in particular, a data processing technology related to a voice frequency synthesis device that converts emotion of a source voice using a generator trained using a deep learning neural network is disclosed.

The present invention is derived from research conducted as part of the Gyeonggi-do Regional Cooperation Research Center project in Gyeonggi-do. [GRRC Gyeonggi 2020-B03, Industrial Statistics and Data Mining Research]

With the development of artificial intelligence, research on deep learning models related to speech conversion is being actively conducted. Deep learning-based generative models such as generative adversarial networks (GANs) and variable auto-encoders (VAEs) are frequently used in speech conversion and have greatly improved speech conversion quality.

By improving the loss function of the existing GAN, the style of the original (Ground Truth) is changed, but CycleGAN, modified to maintain the characteristics of the original, showed good performance in speech conversion. However, since only one-to-one voice conversion is possible, there is a speaker-dependent disadvantage. The VAE-based voice conversion method can perform independent speaker voice conversion in an n-to-m manner, but has a disadvantage in that the quality of sound quality is poor.

On the other hand, if text-to-speech (TTS) is used, a machine can automatically create sound waves corresponding to speech sounds, but existing speech conversion models do not consider prosody and use spectrum mapping for speech conversion. emotions were not included. Recently, research has been conducted to synthesize voices closer to human voices through emotional voice conversion that adds other emotions without loss of language information. In particular, since prosody contains emotional information of speech as well as linguistic information, it is helpful in accurately analyzing and expressing speech in speech conversion.

Paper published in November 2020 K. Zhou et al. (arXiv:2011.02314 [cs.SD]) is about a method for decomposing and reconstructing emotional elements of speech using VAW-GAN (Variational Autoencoding Wasserstein Generative Adversarial Network), a neural network model combining VAE and GAN. For this purpose, a method for training with an encoder by separating emotion and prosody (F0) information from spectral features is disclosed.

Patent Publication No. 10-2021-0032235 published on March 24, 2021 relates to an apparatus for synthesizing emotional voice, building a synthesis model for generating synthesized voice data and an emotion model for generating emotional voice data, Disclosed is an apparatus capable of generating user synthesized emotion voice data in which emotion appropriate to the request sentence is reflected by inputting a user's request sentence and emotion information into a user synthesis model learned based on the constructed synthesis model and emotion model. .

However, a voice conversion method capable of effectively reflecting prosody information, supporting speaker-independent voice conversion, and improving voice quality has not yet been specifically disclosed.

An object of the proposed invention is to provide a voice frequency synthesizer capable of independently converting emotion through n-to-m voice conversion instead of 1-to-1 voice conversion.

Furthermore, an object of the proposed invention is to provide a voice frequency synthesis device capable of accurately transforming according to emotion by extracting and learning prosody information of voice.

Furthermore, an object of the proposed invention is to provide a voice frequency synthesis device capable of increasing the emotion similarity of the synthesized voice by correcting the difference between the target data and the emotion spectrum.

According to one aspect of the proposed invention, an audio frequency synthesis apparatus for emotion conversion using amplitude scaling of voice includes an input vocoder, an emotion spectrum generator, an emotion fundamental frequency generator, an amplitude scaler, and an output vocoder. include

The input vocoder receives a source voice signal and outputs a source spectrum signal and a source fundamental frequency signal. The emotion spectrum generator generates an emotion spectrum signal having emotion information from the source spectrum signal. The emotion fundamental frequency generator generates an emotion fundamental frequency signal including prosody information from the source fundamental frequency signal. The amplitude scale unit adjusts the amplitude of the emotion spectrum signal to output an amplitude-scaled emotion spectrum signal. The output vocoder receives the amplitude-scaled emotion spectrum signal and the emotion fundamental frequency signal and outputs an emotion-converted emotion-converted voice signal.

According to an additional aspect, the emotion spectrum generation unit generates an emotion spectrum signal using a spectrum encoder learned with a generative adversarial network (GAN) to calculate an emotion-independent latent vector from a source spectrum signal, and the emotion-independent latent vector. It includes a spectrum generator that has been trained with a generative adversarial network (GAN) to generate

According to an additional aspect, the spectrum generator includes a fundamental frequency input unit that receives a source fundamental frequency signal, and generates an emotion spectrum signal using the emotion-independent latent vector and the source fundamental frequency signal.

According to an additional aspect, the emotion fundamental frequency generator uses a basic frequency encoder learned with a generative adversarial network (GAN) to calculate an emotion-independent feature vector from a source fundamental frequency signal, and the emotion-independent feature vector. It includes a fundamental frequency generator trained with a generative adversarial network (GAN) to generate a fundamental frequency signal.

According to an additional aspect, the emotion fundamental frequency generator further includes a continuous wavelet transformer for applying continuous wavelet transform (CWT) to the source fundamental frequency signal.

According to an additional aspect, the amplitude scale unit may include an amplitude scale input unit that receives the emotion spectrum signal and a random spectrum signal generated by inputting an arbitrary latent vector to the spectrum generator, and an average and standard deviation of the random spectrum signal. and an amplitude control unit for adjusting the amplitude of the emotion spectrum signal using

According to the proposed invention, it is possible to provide a voice frequency synthesizer capable of n-to-m voice conversion using a VAW-GAN-based generator for voice frequency synthesis and capable of speaker-independent emotion conversion.

Furthermore, the proposed invention enables emotion-dependent transformation more effectively by learning using prosody information using continuous wavelet-transformed fundamental frequencies.

Furthermore, the proposed invention can increase the emotion similarity of the synthesized voice by correcting the difference between the regenerated emotion spectrum and the emotion spectrum obtained during learning using amplitude scaling.

1 is a graph showing a comparison of spectrums (SP) and fundamental frequencies (f0) of voices having different emotions for the same sentence.
2 is a configuration diagram showing the configuration of a VAW-GAN learning device for a voice frequency synthesis device for emotion conversion using amplitude scaling of voice according to an embodiment.
3 is a configuration diagram showing the configuration of a voice frequency synthesis device for emotional conversion using amplitude scaling of voice according to an embodiment.
4 is a spectrogram showing a result of converting neutral emotion into angry emotion and dislike emotion by using a voice frequency synthesizer for emotion conversion using amplitude scaling of voice according to an embodiment.
5 is a flowchart illustrating a VAW-GAN learning method for a voice frequency synthesizer for emotion conversion using voice amplitude scaling according to an embodiment.
6 is a flowchart illustrating a voice frequency synthesis method for emotion conversion using amplitude scaling of voice according to an embodiment.

The foregoing and additional aspects are embodied through embodiments described with reference to the accompanying drawings. It is understood that the elements of each embodiment can be combined in various ways within one embodiment or with elements of another embodiment without contradiction with each other or other references. Based on the principle that the inventor can properly define the concept of terms in order to explain his/her invention in the best way, the terms used in this specification and claims have meanings consistent with the description or proposed technical idea. and should be interpreted as a concept. In this specification, a module or part may be implemented as a set of program instructions executable by a computer or processor, or a set of electronic components or circuits capable of executing these instructions. Also, the operation of each module or part may be performed by one or a plurality of processors or devices.

Hereinafter, preferred embodiments of the present invention will be described in detail with reference to the accompanying drawings.

1 is a graph showing a comparison of spectrums (SP) and fundamental frequencies (f0) of voices having different emotions for the same sentence.

People have different voices, and voices can be analyzed in frequency and amplitude. Emotions can be analyzed by deep learning models by visualizing signals extracted from speech with a computer in the form of speech waveforms, spectrums, and spectrograms. A waveform is a form of sound oscillatory wave that is created over time. Spectra represent the frequencies and amplitudes of different types of speech. Using a spectrum, it is possible to analyze the waveform of sound and express the components that make up sound. The spectrogram is a visual representation of the sound spectrum as a graph, and the characteristics of the waveform and the spectrum are combined and displayed as time, frequency, and amplitude on the X-axis, Y-axis, and Z-axis, respectively.

Sentiment analysis is used to analyze a person's tendencies, opinions, and attitudes in a conversation or text. For voice-based emotion analysis, voice signals can be extracted as spectrograms and applied to deep learning models. For emotional conversion of voice data, a vocoder can be used to extract two audio characteristics, a spectrum (SP) and a fundamental frequency (f0). Spectrum SP is an extracted smoothed spectrogram representing the harmonic spectral envelope. The fundamental frequency f0 may be defined as the lowest frequency of a periodic waveform. The fundamental frequency (f0) is the most basic prosody element appearing in voice data, and f0 representing prosody along with vocabulary and words can be used as an element expressing emotion.

As shown in FIG. 1, the spectrum (SP) and the fundamental frequency (f0) of the same sentence show different characteristics according to emotions. The spectrum (SP) shows frequency on the x-axis and amplitude on the y-axis. Emotional voice data and neutral voice data have different amplitude envelopes and fundamental frequencies (f0). These voice features are preprocessed and used as learning inputs, and voices can be synthesized with converted emotions.

2 is a configuration diagram showing the configuration of a VAW-GAN learning device for a voice frequency synthesis device for emotion conversion using amplitude scaling of voice according to an embodiment.

According to an embodiment, a learning device for a voice frequency synthesis device for emotional conversion using voice amplitude scaling includes an input vocoder 220, a spectrum encoder 235, a spectrum generator 240, and a spectrum discriminator. 245, a fundamental frequency encoder 260, a fundamental frequency generator 265, and a fundamental frequency discriminator 275.

The learning device according to an embodiment divides the source spectrum signal (SP) and the source fundamental frequency signal (f0), and proceeds with learning to convert emotions for each. The input vocoder 220 is used to output the source spectrum signal SP and the source fundamental frequency signal f0 from the source audio signal 210. In addition to neutral, the source audio signal 210 data used for learning may use voices having various emotions such as happiness, calm, sadness, fear, anger, surprise, and disgust expressions. The input vocoder 220 may use a WORLD vocoder or a STRAIGHT vocoder capable of receiving the source voice signal 210 and outputting a source spectrum signal (SP) and a source fundamental frequency signal (f0). As the input vocoder, it is preferable to use a WORLD vocoder capable of high-quality voice synthesis and fast calculation of the source spectrum signal and the source fundamental frequency signal from the source voice signal.

The learning apparatus according to an embodiment configures neural network models for learning the source spectrum signal SP and the source fundamental frequency signal f0, respectively. Each neural network is arranged in parallel, each encoder in each neural network is trained separately so as not to include emotion elements in the input signal, and a generator, which is a decoder, has an emotion ID representing the emotion type of the source speech signal 210. It can be trained to generate an output signal corresponding to the emotion ID by being conditioned on the source fundamental frequency f0 and the source fundamental frequency f0. As a neural network model used for learning, various artificial neural network models based on generative adversarial networks (GANs) can be used. Among these neural network models, it is preferable to use a VAW-GAN (Variational Autoencoding Wasserstein Generative Adversarial Network) neural network model in which a variable autoencoder (VAE) is combined with a generative adversarial network (GAN) model. The VAW-GAN neural network model corresponds to a combined model of VAE and GAN using a generator of a generative adversarial network (GAN) in the decoder of a variable autoencoder (VAE).

For the source spectrum signal SP, the spectrum encoder 235 and the spectrum generator 240 serve as a pair of encoder and decoder. The spectrum encoder 235 calculates a latent vector, which is an emotion-independent feature vector, from the source spectrum signal SP. The spectrum generator 240 generates an emotion spectrum signal using the emotion-independent latent vector. The spectrum generator 240 may additionally receive the source fundamental frequency signal f0 and the emotion ID. The spectrum generator 240 generates an emotion spectrum signal conditioned by the source fundamental frequency signal f0 and the emotion ID from the source spectrum signal SP, and transfers it to the spectrum discriminator 245. The spectrum discriminator 245 determines whether the generated emotion spectrum signal is real or fake. The spectrum encoder 235 is trained to generate emotion-independent latent vectors, the spectrum generator 240 is trained to reduce loss between the source spectrum signal SP and the generated emotion spectrum signal, and the spectrum discriminator 245 is learned adversarially to maximize the loss between the source spectrum signal (SP) and the generated emotion spectrum signal.

For the source fundamental frequency signal f0, the fundamental frequency encoder 260 and the fundamental frequency generator 265 serve as a pair of encoder and decoder. The fundamental frequency encoder 260 calculates a latent vector independent of emotion from the source fundamental frequency signal f0, and the fundamental frequency generator 265 uses the emotion-independent feature vector to calculate emotion close to the source fundamental frequency signal f0. To generate the fundamental frequency signal, the fundamental frequency discriminator 275 may be trained with an artificial neural network model to discriminate the generated emotion fundamental frequency signal from the source fundamental frequency signal f0. The basic frequency generator 265 may additionally receive an emotion ID and generate an emotion fundamental frequency signal conditioned by the emotion ID from the source fundamental frequency signal f0.

The first continuous wavelet transformer 255, which applies continuous wavelet transform (CWT) to the source fundamental frequency signal f0, is placed in front of the fundamental frequency encoder 260, and the continuous wavelet transformed signal is input to the fundamental frequency encoder 280. can do. Continuous Wavelet Transformation (CWT) is used to learn prosodic information from voice characteristics of source data. Equation 1 represents the continuous wavelet transform (CWT).

[Equation 1]

In Equation 1, a represents a scale. b is the time ψ _ab denotes a parent wavelet function.

Continuous wavelet transform (CWT) can perform multi-scale modeling on the source fundamental frequency signal f0 and divide it into various time scales. Thus, the continuous wavelet transform (CWT) can represent short rhymes of phrases made up of a few words and long rhymes of entire speeches. Prosody information can be efficiently learned through continuous wavelet transform (CWT) on the source fundamental frequency signal f0.

3 is a configuration diagram showing the configuration of a voice frequency synthesis device for emotional conversion using amplitude scaling of voice according to an embodiment.

According to one aspect of the proposed invention, an apparatus for synthesizing voice frequencies for emotion conversion using amplitude scaling of voice includes an input vocoder 320, an emotion spectrum generator 330, an emotion fundamental frequency generator 350, , an amplitude scale unit 380 and an output vocoder 390.

The input vocoder 320 receives the source voice signal 310 and outputs a source spectrum signal SP and a source fundamental frequency signal f0. It is preferable to use the same input vocoder as the input vocoder used for learning as the input vocoder. The input vocoder 320 may use a WORLD vocoder or a STRAIGHT vocoder capable of receiving the source voice signal 310 and outputting the source spectrum signal SP and the source fundamental frequency signal f0. As the input vocoder, it is preferable to use a WORLD vocoder capable of high-quality voice synthesis and fast calculation of the source spectrum signal and the source fundamental frequency signal from the source voice signal.

The emotion spectrum generator 330 generates an emotion spectrum signal (eSP) having emotion information from the source spectrum signal (SP). The emotion fundamental frequency generator 350 generates an emotion fundamental frequency signal ef0 including prosody information from the source fundamental frequency signal f0. The amplitude scale unit 380 adjusts the amplitude of the emotion spectrum signal eSP to output an amplitude-scaled emotion spectrum signal AeSP. The emotion spectrum generator 330 and the emotion fundamental frequency generator 350 are each composed of an artificial neural network model composed of a pair of encoders and decoders (generators). That is, the voice frequency synthesis apparatus according to an embodiment includes two artificial neural networks arranged in parallel. In each neural network, an encoder is trained to remove emotion elements from an input signal, and a generator, which is a decoder, is conditioned on the emotion ID and fundamental frequency (f0) representing the target emotion type to be generated to generate an output signal corresponding to the emotion ID. can be trained As an artificial neural network model, various artificial neural network models based on generative adversarial networks (GANs) can be used. Among these artificial neural network models, it is preferable to use a VAW-GAN neural network model using a generator of a generative adversarial network (GAN) for a decoder of a variable autoencoder (VAE).

The output vocoder 390 receives the amplitude-scaled emotion spectrum signal AeSP and the emotion fundamental frequency signal ef0 and outputs an emotion converted voice signal 395. The output vocoder 390 can use a WORLD vocoder, a STRAIGHT vocoder, etc. that can output a voice signal by receiving a spectrum signal and a fundamental frequency signal. It is preferable to use the same vocoder as the input vocoder for the output vocoder, and it is preferable to use the same WORLD vocoder as the input vocoder for the output vocoder.

According to an additional aspect, the emotion spectrum generator 330 includes a spectrum encoder 335 and a spectrum generator 340 . The spectrum encoder 335 is trained with an artificial neural network to calculate a latent vector, which is an emotion-independent feature vector, from the source spectrum signal SP. The spectrum generator 340 is trained as an artificial neural network to generate an emotion spectrum signal (eSP) using the emotion-independent latent vector. As an artificial neural network model used for learning, various artificial neural network models based on generative adversarial networks (GANs) can be used. Among these neural network models, it is preferable to use a VAW-GAN neural network model using a generator of a generative adversarial network (GAN) in a decoder of a variable autoencoder (VAE). The spectrum encoder 335 and the spectrum generator 340 serve as a pair of encoder and decoder.

According to an additional aspect, the spectrum generator 340 includes a fundamental frequency input unit that receives the source fundamental frequency signal f0, and generates an emotion spectrum using the emotion-independent latent vector and the source fundamental frequency signal f0. Generates the signal eSP. The spectrum generator 340 may further include an emotion selection unit that receives a target emotion ID to be converted. The spectrum generator 340 may generate the source fundamental frequency signal f0 and the emotion spectrum signal eSP conditioned by the target emotion ID to be converted from the source spectrum signal SP.

According to an additional aspect, the emotion fundamental frequency generator 350 includes a fundamental frequency encoder 360 and a fundamental frequency generator 365 . The fundamental frequency encoder 360 is trained with an artificial neural network to calculate an emotion-independent feature vector from the source fundamental frequency signal f0. The fundamental frequency generator 365 is trained with an artificial neural network to generate an emotion fundamental frequency signal ef0 using the emotion-independent feature vector. As an artificial neural network model used for learning, various artificial neural network models based on generative adversarial networks (GANs) can be used. Among these neural network models, it is preferable to use a VAW-GAN neural network model using a generator of a generative adversarial network (GAN) in a decoder of a variable autoencoder (VAE). The fundamental frequency encoder 360 and the fundamental frequency generator 365 serve as a pair of encoder and decoder. The basic frequency generator 365 may further include an emotion selection unit that receives a target emotion ID to be converted. The fundamental frequency generator 365 may generate the emotion fundamental frequency signal ef0 conditioned by the emotion ID from the source fundamental frequency signal f0.

According to an additional aspect, the emotion fundamental frequency generator 350 further includes a first continuous wavelet transformer 355 for applying continuous wavelet transform (CWT) to the source fundamental frequency signal f0. The first continuous wavelet transformer 355 is disposed in front of the fundamental frequency encoder 360 and inputs the continuous wavelet-transformed signal to the fundamental frequency encoder 360. The emotion fundamental frequency generator 350 may further include a second continuous wavelet transformer 370 after the generator. The second continuous wavelet transformer 370 may perform an inverse transform of the continuous wavelet transform performed by the first continuous wavelet transformer 355 to generate an emotion fundamental frequency signal ef0.

According to an additional aspect, the amplitude scale unit 380 includes an amplitude scale input unit and an amplitude control unit. The amplitude scale input unit receives the emotion spectrum signal eSP and a random spectrum signal eSPz generated by inputting an arbitrary random latent vector z to the spectrum generator. The amplitude controller adjusts the amplitude of the emotion spectrum signal eSP by using the mean and standard deviation of the random spectrum signal eSPz. The emotion spectrum signal (eSP) contains the amplitude information of the source voice signal, so that a difference occurs in amplitude compared to the voice of the emotion to be converted. In order to compensate for this difference, standard scaling may be performed by obtaining a random spectrum signal (eSPz) from the same spectrum generator 340. Since the random latent vector z is irrelevant to the input signal, the random spectrum signal eSPz represents the amplitude of the emotion selected as the emotion ID regardless of the emotion of the source signal.

According to an additional aspect, the amplitude adjusting unit obtains the average (μ) and standard deviation (σ) from the amplitude data of the random spectrum signal (eSPz), and converts the amplitude data (X) of the emotion spectrum signal (eSP) into the following Equation 2 Amplitude data (X') of the normalized and amplitude-scaled emotion spectrum signal (AeSP) is obtained.

[Equation 2]

In Equation 2, X is the emotion spectrum signal (eSP) data, μ is the average of the random spectrum signal (eSPz) data, and σ is the standard deviation of the random spectrum signal (eSPz) data.

The amplitude scale unit 380 may further include a log scale change unit for changing the amplitude data of the random spectrum signal eSPz and the amplitude data of the emotion spectrum signal eSP transmitted to the amplitude control unit into a log scale. Since human hearing responds in proportion to the logarithmic scale of amplitude, it is preferable to change the amplitude data to logarithmic scale and transmit the data to the amplitude control unit.

Dynamic frequency warping (DFW) can be used to adjust amplitude characteristics in voice conversion. Dynamic frequency warping (DFW) is a method of distorting a source spectrum signal to approximate a target spectrum signal. However, since dynamic frequency warping (DFW) has a limitation in not properly reflecting speaker characteristics, a separate Gaussian Mixture Model (GMM)-based conversion process is required.

According to an embodiment of the present invention, the VAW-GAN-based amplitude conversion method performed in a voice frequency synthesizer interpolates two spectra generated by a generator that has learned a latent distribution according to an emotion ID, and separate A Gaussian Mixture Model (GMM)-based conversion process is not required. As a result, emotion conversion to target emotion ID is possible without distorting prosodic characteristics and speaker characteristics by amplitude conversion.

Hereinafter, an experimental example in which emotion conversion of voice is performed using the voice frequency synthesis apparatus for emotion conversion according to an embodiment of the present invention will be disclosed.

In one embodiment of the present invention, the emotion conversion device for voice data was implemented in a hardware experiment environment of Intel Skylake Xeon, NVIDIA Tesla V100 X 2 (20 RFLOPS), and 128 GB of RAM. For software, the Tensorflow backend engine was used for the design and implementation of the model on the Ubuntu operating system.

The training data for the voice frequency synthesizer for emotion conversion was based on RAVDESS (Ryerson Audio-Visual Database of Emotional Speech and Song) consisting of 7,356 voice data files (24.8 GB). RAVDESS contains audio data of 24 speakers (12 female, 12 male pronounced with a neutral North American accent). The emotion classes represented by the voice are happiness, calm, sadness, fear, anger, surprise, and disgust expressions.

WORLD vocoder was used as the input vocoder and output vocoder, and voice features were extracted from voice-only data (16-bit, 48kHz .wav) using the WORLD vocoder, which was the input vocoder, and converted into a spectrogram that could be used as input data for the model. did In the continuous wavelet transform, the mother wavelet function was MexicanHat.

Mel Cepstral Distortion (MCD) and Log-Spectral Distortion (LSD) were used as performance evaluation indicators of the emotion conversion model. Mel Cepstral Distortion (MCD) is a measure of how different the original voice is from the converted voice in terms of mel-cepstral. The lower the MCD value, the more similar the quality of the converted voice is to the original voice. Log Spectrum Distortion (LSD) means the distance between the original voice spectrum and the converted voice spectrum, and the lower the LSD value, the more similar the converted voice quality is to the original voice.

Table 1 shows the results of comparing the average MCD and LSD of the existing speech conversion model and the model disclosed in an embodiment of the present invention.

Case Model MCD LSD Comparative Example 1 LG-Cycle-GAN 5.251 dB 6.897 dB Comparative Example 2 LG-VAW-GAN 4.656 dB 6.488 dB Comparative Example 3 CWT-VAW-GAN 4.458 dB 6.245 dB Example 1 CWT-AS-VAW-GAN 4.451 dB 6.239 dB

In Table 1, Comparative Example 1 used Cycle-GAN as an artificial neural network model, and Comparative Example 2, Comparative Example 3, and Example 1 all used VAW-GAN as an artificial neural network model. In Comparative Example 1 and Comparative Example 2, the fundamental frequency (f0) was converted using a log-Gaussian-based linear transformation, and in Comparative Example 3 and Example 1, the basic frequency (f0) was converted using continuous wavelet transform (CWT). The frequency (f0) was converted. In Example 1, amplitude scaling was additionally applied to the generated emotion spectrum signal.

4 is a spectrogram showing a result of converting neutral emotion into angry emotion and dislike emotion by using a voice frequency synthesizer for emotion conversion using amplitude scaling of voice according to an embodiment.

4(a) is a spectrogram of source voice data, and emotion indicates neutral. The left side of FIG. 4(b) shows spectrograms of voice data for Angry and Disgust emotions. As can be seen from the spectrogram, the spectrum containing prosody and volume information differs according to emotion.

The right side of FIG. 4(b) shows a spectrum obtained by converting source voice data having neutral emotions into anger and disgust emotions using the voice frequency synthesizer according to the embodiment. Looking at the converted spectrum, it can be confirmed that the converted spectrum is similar to the target emotion spectrum (Angry, Disgust) compared to the spectrum of the source voice data.

Hereinafter, a voice frequency synthesizing method for emotion conversion of voice performed in the voice frequency synthesis apparatus for emotion conversion according to an embodiment of the present invention will be disclosed.

5 is a flowchart illustrating a VAW-GAN learning method for a voice frequency synthesizer for emotion conversion using voice amplitude scaling according to an embodiment.

According to an embodiment, in the learning method for a voice frequency synthesizer for emotional conversion using voice amplitude scaling, an input vocoder converts a source spectrum signal (SP) and a source fundamental frequency signal (f0) from a source voice signal (210). After extracting (S510), learning the source spectrum signal (SP) and the source fundamental frequency signal (f0), respectively, confirming that all learning has been completed for the data for learning (S590), the data to be learned yet If remains, it returns to the input step and repeats, and if there is no data to learn, learning ends. Source voice signal data used for learning may use voices having various emotions, such as happiness, calm, sadness, fear, anger, surprise, and disgust expressions, in addition to neutral. The source spectrum signal (SP) and the source fundamental frequency signal (f0) are input to each neural network arranged in parallel and used for learning. Each encoder is separately trained not to include an emotion element in the input signal, and a generator, which is a decoder, is trained to generate an output signal corresponding to an emotion ID by being conditioned on an emotion ID representing an emotion type and a source fundamental frequency (f0). can As a neural network model used for learning, various artificial neural network models based on generative adversarial networks (GANs) can be used. Among these neural network models, it is preferable to use a VAW-GAN neural network model using a generator of a generative adversarial network (GAN) in a decoder of a variable autoencoder (VAE).

The method of learning the source spectrum signal (SP) includes a spectrum encoding step (S535) in which the spectrum encoder calculates a latent vector, which is an emotion-independent feature vector, from the source spectrum signal (SP), and the spectrum generator calculates the emotion. A step of generating an emotion spectrum signal using an independent latent vector (S540), and a step of determining whether the generated emotion spectrum signal is real or fake by the spectrum discriminator (S545). In the step of generating the emotion spectrum signal (S540), the spectrum generator further receives the source fundamental frequency signal f0 and the emotion ID, and the source fundamental frequency signal f0 and the emotion ID from the source spectrum signal SP. An emotion spectrum signal can be generated and transmitted to the spectrum discriminator. Through the learning process, the spectrum encoder learns to generate an emotion-independent latent vector, the spectrum generator learns to reduce the loss between the source spectrum signal SP and the generated emotion spectrum signal, and the spectrum discriminator learns to generate the source spectrum signal SP. ) and the generated emotion spectrum signal is learned adversarially to maximize the loss.

The method of learning the source fundamental frequency signal (f0) includes a fundamental frequency encoding step (S560) in which the fundamental frequency encoder calculates an emotion-independent latent vector from the source fundamental frequency signal (f0), and the fundamental frequency generator calculates the emotion-independent latent vector. Generating an emotion fundamental frequency signal using the latent vector (S565), and determining whether the generated emotion fundamental frequency signal is real or fake by the fundamental frequency discriminator (S575). In the step of generating the emotion fundamental frequency signal (S565), the fundamental frequency generator may further receive an emotion ID, generate an emotion fundamental frequency signal conditioned by the emotion ID from the source fundamental frequency signal f0, and transmit it to the fundamental frequency discriminator. there is. Through the learning process, the fundamental frequency encoder learns to generate an emotion-independent latent vector, the fundamental frequency generator learns to reduce the loss between the source fundamental frequency signal f0 and the generated emotion fundamental frequency signal, and the fundamental frequency discriminator It is adversarially learned to maximize the loss between the source fundamental frequency signal f0 and the generated emotion fundamental frequency signal.

The method for learning the source fundamental frequency signal f0 further includes a first continuous wavelet transformation step (S555) of applying continuous wavelet transform (CWT) to the source fundamental frequency signal f0 before the fundamental frequency encoding step (S560). can do. Through continuous wavelet transform (CWT) on the source fundamental frequency signal f0, it is possible to efficiently learn prosody information of speech.

6 is a flowchart illustrating a voice frequency synthesis method for emotion conversion using amplitude scaling of voice according to an embodiment.

According to another aspect of the proposed invention, a voice frequency synthesis method for emotion conversion using amplitude scaling of voice includes a voice signal separation step in which an input vocoder receives a source voice signal and outputs a source spectrum signal and a source fundamental frequency signal ( S620), an emotion spectrum generation step of generating an emotion spectrum signal having emotion information from the source spectrum signal (S630), and an emotion fundamental frequency generation step of generating an emotion fundamental frequency signal including prosody information from the source fundamental frequency signal (S630) S650), an amplitude scaling step (S680) of adjusting the amplitude of the emotion spectrum signal and outputting an amplitude-scaled emotion spectrum signal, and an output vocoder receives the amplitude-scaled emotion spectrum signal and the emotion fundamental frequency signal, and the emotion is and a voice signal synthesis step (S690) of outputting the converted emotion conversion voice signal. It is desirable to use the same WORLD vocoder as the input vocoder used for learning as the input vocoder and output vocoder.

In the emotion spectrum generation step (S630) and the emotion fundamental frequency generation step (S650), the emotion spectrum signal (esP) and the emotion fundamental frequency are generated by using the pair of encoders and generators (decoders) trained with two artificial neural network models arranged in parallel, respectively. Generates signal ef0. As an artificial neural network model, various artificial neural network models based on generative adversarial networks (GANs) can be used. Among these artificial neural network models, it is preferable to use a VAW-GAN neural network model using a generator of a generative adversarial network (GAN) for a decoder of a variable autoencoder (VAE).

According to an additional aspect, the emotion spectrum generation step (S630) includes a spectrum encoding step (S635) in which the spectrum encoder learned by the artificial neural network calculates an emotion-independent latent vector from the source spectrum signal, and the spectrum learned by the artificial neural network. A spectrum generation step (S640) in which the generator generates an emotion spectrum signal using the emotion-independent latent vector. As an artificial neural network model used for learning, various artificial neural network models based on generative adversarial networks (GANs) can be used. Among these neural network models, it is preferable to use a VAW-GAN neural network model using a generator of a generative adversarial network (GAN) in a decoder of a variable autoencoder (VAE).

According to an additional aspect, the spectrum generating step (S640) includes a basic frequency input step in which the spectrum generator receives the source fundamental frequency signal, and an emotion spectrum signal is generated using the emotion-independent latent vector and the source fundamental frequency signal. It includes the steps of creating The spectrum generation step (S640) may further include an emotion ID input step of receiving an emotion ID. In the spectrum generation step (S640), an emotion spectrum signal conditioned by the additionally input source fundamental frequency signal and emotion ID is generated using the encoded source spectrum signal (latent vector).

According to an additional aspect, the emotion fundamental frequency generation step (S650) includes the basic frequency encoding step (S660) of calculating an emotion-independent feature vector from the source fundamental frequency signal by the basic frequency encoder learned by the artificial neural network, and the artificial neural network and a basic frequency generation step (S665) of generating an emotion fundamental frequency signal by using the emotion-independent feature vector by the basic frequency generator learned as . As an artificial neural network model used for learning, various artificial neural network models based on generative adversarial networks (GANs) can be used. Among these neural network models, it is preferable to use a VAW-GAN neural network model using a generator of a generative adversarial network (GAN) in a decoder of a variable autoencoder (VAE).

According to an additional aspect, the emotion fundamental frequency generation step (S650) further includes a first continuous wavelet transform step (S655) of applying continuous wavelet transform (CWT) to the source fundamental frequency signal. The first continuous wavelet transformation step (S655) is performed prior to the fundamental frequency encoding step (S660), and the continuous wavelet-transformed fundamental frequency signal is transferred to the fundamental frequency encoding step (S660). The emotion basic frequency generating step (S650) may further include a second continuous wavelet transform step (S670) after performing the basic frequency generating step (S665). In the second continuous wavelet transform step (S670), an inverse transform of the continuous wavelet transform performed in the first continuous wavelet transform step (S655) may be performed to generate the emotion fundamental frequency signal ef0.

According to an additional aspect, the amplitude scale step (S680) includes an amplitude scale input step of receiving the emotion spectrum signal (eSP) and a random spectrum signal (eSPz) generated by inputting an arbitrary latent vector to the spectrum generator; and an amplitude adjusting step of adjusting the amplitude of the emotion spectrum signal eSP by using the mean and standard deviation of the random spectrum signal eSPz.

According to an additional aspect, the amplitude adjusting step obtains an average (μ) and a standard deviation (σ) from the amplitude data of the random spectrum signal (eSPz), and converts the amplitude data (X) of the emotion spectrum signal (eSPz) to Equation 2 Amplitude data (X') of the amplitude-scaled emotion spectrum signal (AeSP) is obtained by normalizing with .

The amplitude scaling step (S680) may further include a log scale changing step of changing the amplitude data of the random spectrum signal (eSPz) and the amplitude data of the emotion spectrum signal (eSP) delivered to the amplitude adjusting step to a log scale. there is. Since human hearing reacts in proportion to the logarithmic scale of amplitude, it is preferable to convert the amplitude data into a logarithmic scale and transfer it to the amplitude control step.

An embodiment of the present invention discloses an apparatus and method for synthesizing voice frequencies using VAW-GAN-based amplitude scaling for emotion conversion. Emotions are converted more effectively by learning the spectrum (SP) and the fundamental frequency (f0) respectively. In particular, by applying the continuous wavelet transform (CWT) to the fundamental frequency (f0), speaker-independent emotional transformation in which prosody characteristics are reflected is possible. Meanwhile, amplitude scaling is applied to the converted emotion spectrum to create similar emotions by adjusting the difference in amplitude from the original emotion spectrum. Using such a voice frequency synthesis device, voice conversion with emotions is possible and can be applied to various fields such as artificial intelligence speakers.

In the above, the present invention has been described through embodiments with reference to the accompanying drawings, but is not limited thereto, and should be interpreted to cover various modifications that can be obviously derived by those skilled in the art. The claims are intended to cover these variations.

210: source audio signal 220: input vocoder
235: spectrum encoder 240: spectrum generator
245: spectrum discriminator 255: first continuous wavelet transformer
260: fundamental frequency encoder 265: basic frequency generator
275: fundamental frequency discriminator
310: source audio signal 320: input vocoder
330: emotion spectrum generator
335: spectrum encoder 340: spectrum generator
350: emotion basic frequency generator 355: first continuous wavelet transformer
360: fundamental frequency encoder 365: basic frequency generator
370: second continuous wavelet transformer 380: amplitude scale unit
390: output vocoder 395: emotion conversion voice signal

Claims (12)

In the voice frequency synthesis device for emotion conversion using amplitude scaling of voice,
an input vocoder receiving a source voice signal and outputting a source spectrum signal and a source fundamental frequency signal;
an emotion spectrum generating unit generating an emotion spectrum signal having emotion information from the source spectrum signal;
an emotion fundamental frequency generator for generating an emotion fundamental frequency signal including prosody information from the source fundamental frequency signal;
an amplitude scaler configured to output an amplitude-scaled emotion spectrum signal by adjusting the amplitude of the emotion spectrum signal; and
an output vocoder which receives the amplitude-scaled emotion spectrum signal and the emotion fundamental frequency signal and outputs an emotion-converted voice signal;
including,
The emotion spectrum generator,
a spectrum encoder trained with a generative adversarial network (GAN) to calculate an emotion-independent latent vector from the source spectrum signal; and
a spectrum generator trained with a generative adversarial network (GAN) to generate an emotion spectrum signal using the emotion-independent latent vector;
including,
The amplitude scale part,
an amplitude scale input unit that receives the emotion spectrum signal and a random spectrum signal generated by inputting an arbitrary latent vector to the spectrum generator; and
an amplitude adjusting unit adjusting the amplitude of the emotion spectrum signal using the mean and standard deviation of the random spectrum signal;
Including, voice frequency synthesis device. delete The method according to claim 1, wherein the spectrum generator,
Including; a fundamental frequency input unit for receiving the source fundamental frequency signal;
A voice frequency synthesis apparatus for generating an emotion spectrum signal using the emotion-independent latent vector and the source fundamental frequency signal. The method according to claim 1, wherein the emotion fundamental frequency generator,
a fundamental frequency encoder learned with a generative adversarial network (GAN) to calculate an emotion-independent feature vector from the source fundamental frequency signal; and
a basic frequency generator learned with a generative adversarial network (GAN) to generate an emotion fundamental frequency signal using the emotion-independent feature vector;
Including, voice frequency synthesis device. The method according to claim 4, wherein the emotion fundamental frequency generator,
a continuous wavelet transformer for applying continuous wavelet transform (CWT) to the source fundamental frequency signal;
Further comprising a voice frequency synthesis device. delete In the voice frequency synthesis method for emotion conversion using amplitude scaling of voice,
A voice signal separation step in which an input vocoder receives a source voice signal and outputs a source spectrum signal and a source fundamental frequency signal;
an emotion spectrum generating step of generating an emotion spectrum signal having emotion information from the source spectrum signal;
an emotion fundamental frequency generating step of generating an emotion fundamental frequency signal including prosody information from the source fundamental frequency signal;
an amplitude scaling step of outputting an amplitude-scaled emotion spectrum signal by adjusting the amplitude of the emotion spectrum signal; and
a voice signal synthesizing step in which an output vocoder receives the amplitude-scaled emotion spectrum signal and the emotion fundamental frequency signal and outputs an emotion converted voice signal in which emotion is converted;
including,
In the emotion spectrum generation step,
a spectrum encoding step in which a spectrum encoder trained with a generative adversarial network (GAN) calculates an emotion-independent latent vector from the source spectrum signal; and
a spectrum generation step in which a spectrum generator learned with a generative adversarial network (GAN) generates an emotion spectrum signal using the emotion-independent latent vector;
including,
The amplitude scale step,
an amplitude scale input step of receiving the emotion spectrum signal and a random spectrum signal generated by inputting an arbitrary latent vector to the spectrum generator; and
an amplitude adjusting step of adjusting the amplitude of the emotion spectrum signal using the mean and standard deviation of the random spectrum signal;
Including, voice frequency synthesis method. delete The method according to claim 7, wherein the spectrum generation step,
a fundamental frequency input step in which the spectrum generator receives the source fundamental frequency signal; and
generating an emotion spectrum signal using the emotion-independent latent vector and the source fundamental frequency signal;
Including, voice frequency synthesis method. The method according to claim 7, wherein the emotion fundamental frequency generation step,
A fundamental frequency encoding step of calculating a feature vector independent of emotion from the source fundamental frequency signal by a fundamental frequency encoder learned with a generative adversarial network (GAN); and
a basic frequency generation step of generating an emotion fundamental frequency signal using the emotion-independent feature vector by a fundamental frequency generator learned by a generative adversarial network (GAN);
Including, voice frequency synthesis method. The method according to claim 10, wherein the emotion fundamental frequency generation step,
a continuous wavelet transform step of applying continuous wavelet transform (CWT) to the source fundamental frequency signal;
Further comprising, voice frequency synthesis method. delete

Images