KR20210059586A - Method and Apparatus for Emotional Voice Conversion using Multitask Learning with Text-to-Speech - Google Patents

Abstract

Disclosed are an emotional voice conversion method using multitask learning to preserve learning stability by the multitask learning and text-to-speech (TTS) conversion and an emotional voice conversion apparatus thereof. According to the present invention, the emotional voice conversion method comprises the following steps: performing voice conversion (VC) when a pair of input voices is a log Mel spectrogram of a language transferring language content and a log Mel spectrogram of a style reference voice; performing TTS conversion when the pair of input voices are a log Mel spectrogram of a one-hot representative text and a style reference voice; mapping both the log Mel spectrogram and the one-hot representative text of the language transferring the language content to the same space and being decoded into the Mel spectrogram; and acquiring a linear spectrum from the decoded Mel spectrogram through a preprocessor.

Description

Method and Apparatus for Emotional Voice Conversion using Multitask Learning with Text-to-Speech}

The present invention relates to an emotional speech conversion method and apparatus using multitask learning with text-to-speech conversion.

Voice can be regarded as the composition of what should be conveyed-the content of the language-and the method of delivery-a style. Voice Conversion (VC) is an operation of changing a voice style while maintaining language content. This is a difficult task as language content may be lost or style information may not be changed during the conversion process.

In the prior art, speech conversion has been performed using a frame-based approach. When considering the source and target voices, an alignment between the two voices is obtained, and then the acoustic properties of the source voice are transformed into the target voice. Various methods have been applied to model acoustic functions such as Gaussian Mixture Model (GMM), Deep Neural Network (DNN), and Recurrent Neural Network (RNN). The success of a sequence-to-sequence (seq2seq) model, which has been noted in the recent prior art, is also applied to speech conversion. During the training of the seq2seq speech conversion model, problems such as incorrect pronunciation and learning instability were observed. In this prior art, text supervision is added to each stage of the decoder output to improve the quality of speech conversion. However, this approach is limited because it requires clear alignment by human or Dynamic Time Wrapping (DTW).

Traditionally, frame-based models have been used to solve VC. Given the source and target voices, their alignment is found by the DTW. Then, the transformation between the acoustic properties of the aligned frames is modeled. Recently, the seq2seq model was proposed in VC. In this model, the model jointly learns the alignment and frame transformation by attention mechanisms without explicit temporal alignment. However, the performance of VC is not sufficient because the converted voice may lose language information. For example, the same word may be repeatedly generated, some words may be deleted, or mispronounced. Text supervision is added to the VC to prevent this speech phenomenon, but explicit alignment is required. Unlike previous work, TTS was used to guide language information to the VC. No explicit alignment is required for this approach.

On the other hand, emotional voice conversion mainly performed frame-based conversion or rule-based approach. This approach has limitations because DTW does not guarantee that exact sorting and rule-based approaches have limitations in speech conversion modeling. This can be improved by using models that do not rely on explicit alignment.

An object of the present invention is to provide a speech conversion method and apparatus using multitask learning using text-to-speech (TTS). Through the proposed emotional voice conversion method and apparatus, the entire network is learned to minimize the loss of VC and TTS, and the VC captures more language information and tries to preserve learning stability by multitask learning.

In one aspect, the emotional voice conversion method using multitask learning along with text-to-speech conversion proposed in the present invention is a log mel spectrogram of a language in which a pair of input voices convey language content and a style reference voice. In the case of a log mel spectrogram, performing a voice conversion (VC), a pair of input voices is a one-hot representative text and a log mel spectrogram of the style reference voice. spectrogram), the step of performing text-to-speech (TTS), the log mel spectrogram of the language delivering the language content, and the one-hot representative text are all mapped to the same space. It includes a step of decoding a post-mel spectrogram and obtaining a linear spectrum from the decoded mel spectrogram through a preprocessor.

If the pair of input voices is the log mel spectrogram of the language that conveys the language content and the log mel spectrogram of the style reference voice, the step of performing the speech conversion is the log mel spectrogram of the language conveying the language content, using the content encoder. Through the embedding, the log mel spectrogram of the style reference voice is embedded through the style encoder.

If the pair of input voices is the log mel spectrogram of the one-hot representative text and the style reference voice, the step of performing text-to-speech is to embed the one-hot representative text through the text encoder, and to the log mel spectrogram of the style reference voice. Embed gram via style encoder

Both the log mel spectrogram of the language conveying the language content and the one-hot representative text are mapped to the same space and then decoded into the mel spectrogram is performed in the log mel spectrogram of the style reference voice in each decoding step. The extracted style vector is connected to the attention RNN and the decoder RNN.

Both the log mel spectrogram and the one-hot representative text of the language delivering the language content are mapped to the same space and then decoded into the mel spectrogram. When performing speech conversion, a context vector is used for all repetitions of the attention RNN.

When considering style reference speech, the style encoder is designed to extract only emotion information and remove linguistic content, and to extract emotion regardless of linguistic content to process multiple input style domains, and when the extracted emotion is injected into the decoder, It processes many-to-many emotional voice transformations by generating various emotions.

In another aspect, the emotional voice conversion apparatus using multitask learning along with text-to-speech conversion proposed in the present invention refers to a log mel spectrogram and a style of a language in which a pair of input voices convey language content. Voice conversion via Style Encoder, Content Encoder and Text Encoder depending on whether it is a log mel spectrogram of speech or a one-hot representative text and style reference log mel spectrogram of speech Alternatively, the conversion unit performing text-to-speech conversion, the log mel spectrogram of the language delivering the language content, and the one-hot representative text are all mapped to the same space, and then a neural network network that is decoded into the mel spectrogram, and the decoded It includes a preprocessor for obtaining a linear spectrum from the mel spectrogram.

According to embodiments of the present invention, through the emotional voice conversion method and apparatus using multitask learning using text-to-speech, the entire network learns to minimize the loss of VC and TTS, and the VC captures more language information and multi-tasks. Learning stability can be preserved by task learning.

1 is a flowchart illustrating an emotional voice conversion method using multitask learning along with text-to-speech conversion according to an embodiment of the present invention.
2 is a diagram illustrating a configuration of an emotional speech conversion apparatus using multitask learning along with text-to-speech conversion according to an embodiment of the present invention.
3 is a diagram illustrating a congestion matrix of a style encoder according to an embodiment of the present invention.
4 is a diagram illustrating an example of a voice conversion result according to an embodiment of the present invention.

In the present invention, an approach using text-to-speech (TTS) is adopted. TTS is an operation that converts text or voice information into voice waveforms, and abundant seq2seq-based research has been actively conducted. TTS is a task very closely related to Voice Conversion (VC). VC and TTS differ only in the input domain, and the role of the decoder for converting voice information into an acoustic shape is very the same. TTS embedding space is highly correlated with voice information, and VC is expected to learn embedding space close to TTS through multitask learning. In the present invention, the TTS is used to provide voice information to the VC to improve performance.

According to an embodiment of the present invention, this task is extended to emotional speech conversion. When considering the style reference voice, the style encoder extracts only the emotion information and removes the linguistic content. Style encoders are designed to extract emotions regardless of linguistic content, so they can process multiple input style domains. In addition, when the extracted emotion is injected into the decoder, various emotions can be generated. Therefore, the proposed model can handle many-to-many emotional speech conversion.

The contribution of the model according to the embodiment of the present invention is as follows: Multitasking learning using TTS can improve the performance of VC. Many emotional speech transformations were initially performed by the seq2seq model. The style reference voice may determine a target area for voice conversion.

Many-to-many VC refers to the number of source domains and the number of target domains. In the prior art, Cycle-GAN converted a speaker, not a dataset, into a target speaker, and vice versa. An i-vector based VC system is proposed to generate the linguistic features of speakers that are not in the learning set. Compared to other many-to-many VC schemes, the proposed model can convey the speaker's emotional knowledge and enables conversion between different emotions.

TTS is one of the most active research fields in the voice field. Various types of Seq2seq models have been proposed and expressive synthesis has also been studied. The style vector extracted from the style encoder generates an emotional voice by injecting the input emotion into the network. In the present invention, the emotion label is not utilized during learning, and no emotion label is explicitly accepted as an input to the network. Hereinafter, embodiments of the present invention will be described in detail with reference to the accompanying drawings.

1 is a flowchart illustrating an emotional voice conversion method using multitask learning along with text-to-speech conversion according to an embodiment of the present invention.

Voice conversion (VC) is the work of changing a person's voice into a different style while preserving the linguistic content. The conventional state-of-the-art technology for VC is based on a sequence-to-sequence (seq2seq) model capable of misleading language information. Attempts have been made to overcome this using text watchdog, and a clear alignment is required, which loses the benefits of using the seq2seq model. In the present invention, speech conversion using multitask learning using text-to-speech (TTS) is proposed. The seq2seq-based TTS embedding has a wealth of information about text. The role of the TTS decoder is to convert the built-in space into voice such as VC. In the proposed model, the entire network is trained to minimize the loss of VC and TTS. VC is expected to capture more linguistic information and preserve learning stability by multitasking learning. VC's experiments were conducted on a Korean male emotional text-to-speech dataset, and show that multitask learning helps to maintain linguistic content in VC.

The proposed text-to-speech and emotional speech conversion method using multitask learning is the log mel spectrogram of the language in which the pair of input speech conveys the language content and the log mel spectrogram of the style reference speech. In the case of, the step 110 of performing voice conversion (VC), when the pair of input voices is a log mel spectrogram of one-hot representative text and style reference voice, Step 120 of performing text-to-speech (TTS), the log mel spectrogram of the language delivering the language content, and the one-hot representative text are all mapped to the same space and then the mel A step 130 of decoding into a spectrogram and a step 140 of acquiring a linear spectrum from the decoded mel spectrogram through a preprocessor.

In step 110, when the pair of input voices is a log Mel spectrogram of a language delivering language content and a log Mel spectrogram of a style reference voice, Voice Conversion (VC) Perform. The log mel spectrogram of the language delivering the language content is embedded through the content encoder, and the log mel spectrogram of the style reference voice is embedded through the style encoder.

In step 120, when the pair of input voices is a log mel spectrogram of one-hot representative text and style reference voice, text-to-speech (TTS) is performed. Carry out. One-hot representative text is embedded through the text encoder, and the log mel spectrogram of the style reference voice is embedded through the style encoder.

In step 130, both the log mel spectrogram and the one-hot representative text of the language delivering the language content are mapped to the same space and then decoded into the mel spectrogram. In each decoding step, the style vector extracted from the log mel spectrogram of the style reference voice is connected to the attention RNN and the decoder RNN. In this case, when text-to-speech is performed through the text encoder, the attention RNN, the decoder RNN, and the preprocessor, the context vector is used for all repetitions of the attention RNN.

In step 140, a linear spectrum is obtained from the decoded mel spectrogram through a preprocessor.

According to an embodiment of the present invention, when considering the style reference voice, the style encoder is designed to extract only emotion information and remove linguistic content, and to extract emotion regardless of linguistic content to process a plurality of input style domains, When the extracted emotions are injected into the decoder, many-to-many emotional voice conversion is processed by generating various emotions.

In more detail, the proposed emotional voice conversion method can perform VC and TTS in a single model. The network is played in _{VC when the input pair is (x c} , x _s ) or in TTS when the input pair is (y _t , x _{s ).} Here, x _c , y _t , and x _s are the log mel spectrogram of the language conveying the language content, the one-hot representative text, and the log mel spectrogram of the style reference voice. Both x _c and y _t are mapped to the same space h _l and then decoded into a mel spectogram m. In each decoding step, the style vector h _{s extracted from x s} is connected to the attention RNN and the decoder RNN. The linear spectrum l is obtained by the preprocessor. A detailed network architecture will be described with reference to FIG. 2 and the following equation.

2 is a diagram illustrating a configuration of an emotional speech conversion apparatus using multitask learning along with text-to-speech conversion according to an embodiment of the present invention.

An emotional speech conversion apparatus using multitask learning along with the proposed text-to-speech conversion includes a conversion unit 210, a neural network 220, and a preprocessor 230.

_{The conversion unit 210 is a log mel spectrogram (Mel spectrogram) (x c} ) of a language that delivers the language content and a log mel spectrogram (Mel spectrogram) of the style reference voice or one-hot (one). -hot) Voice conversion through the style encoder 211, the content encoder 212 and the text encoder 213 depending on whether the representative text (y _t ) and the log mel spectrogram (x _{s) of the style reference voice are} Or perform text-to-speech.

The conversion unit 210 performs speech conversion when the pair of input voices is a log mel spectrogram (x _c ) of a language delivering language content and a log mel spectrogram (x _s ) of a style reference voice. A log mel spectrogram (x _c ) of a language delivering language content is embedded through the content encoder 212, and a log mel spectrogram (x _s ) of a style reference voice is embedded through the style encoder (211).

The conversion unit 210 performs text-to-speech when the pair of input voices is a one-hot representative text (y _t ) and a log mel spectrogram (x _s ) of a style reference voice. One-hot representative text (y _t ) is embedded through the text encoder 213, and the log mel spectrogram (x _s ) of the style reference voice is embedded through the style encoder 211.

The neural network network 220 maps both the log mel spectrogram (x _c ) and the one-hot representative text (y _t ) of the language delivering the language content to the same space and then converts it into a mel spectrogram (m). Is decoded. The neural network 220 connects the style vector extracted from the log mel spectrogram of the style reference voice to the attention RNN 221 and the decoder RNN 222 in each decoding step.

The preprocessor 230 obtains a linear spectrum l from the decoded mel spectrogram m. The linear spectrum (l) outputs _{the converted voice (x o) through a vocoder (240).}

According to an embodiment of the present invention, when considering the style reference voice, the style encoder is designed to extract only emotion information and remove linguistic content, and to extract emotion regardless of linguistic content to process a plurality of input style domains, When the extracted emotions are injected into the decoder, many-to-many emotional voice conversion is processed by generating various emotions.

In more detail, VC and TTS can be performed in a single model of the proposed emotional speech conversion device. The network is played in _{VC when the input pair is (x c} , x _s ) or in TTS when the input pair is (y _t , x _{s ).} Here, x _c , y _t , and x _s are the log mel spectrogram of the language conveying the language content, the one-hot representative text, and the log mel spectrogram of the style reference voice. Both x _c and y _t are mapped to the same space h _l and then decoded into a mel spectogram m. In each decoding step, the style vector h _{s extracted from x s} is connected to the attention RNN and the decoder RNN. The linear spectrum l is obtained by the preprocessor.

Here, h _t , h _c , and h _s are embeddings of the text encoder 213, the content encoder 212, and the style encoder 211, respectively. h _att ^(t) and h _dec ^(t) are hidden representations of the attention RNN and decoder RNN at time step t. m, l, c ^(t) and o ^(t) are the log mel spectrogram at time step t, the log linear spectrogram of the object, the context vector achieved by the attention mechanism, and the attention RNN output at time step t. XOR stands for exclusive OR operator.

In the case of the TTS part including the text encoder, decoder, attention RNN and preprocessor, the overall architecture is based on Tacotron, and the context vector c ^(t) is used for all iterations of the attention RNN, and the residual connection is Convolution Bank + Highway + bi-GRU (CBHG) added to the connection.

The text encoder consists of a character embedding layer, a prenet, and a CBHG, where the freenet consists of two FC ReLU-Dropout layers. The LSTM stack is used in the content encoder, and there is no reduction in temporal resolution because the temporal reduction can lose local time information. This means that _{the length of h c} is equal to x _c. In the case of a style encoder, x _{s is mapped to h s} with a fixed dimension by embedding a fully connected layer following the last step of the LSTM to reduce the dimension. The attention module consists of the attention RNN and attention mechanism. Note The RNN takes h _s , c ^(t-1) and m ^(t-1) as inputs. Then its output o ^(t) and h _l are used to note the mechanism for generating the context vector c ^(t).

The learning loss is Im-mgtI+II-IgtI, where mgt and lgt are the ground truth log mel spectrogram and log linear spectrogram.

According to an embodiment of the present invention, a Korean Emotional Text-to-speech (mKETTS) data set was constructed. For example, a 30-year-old man, measured for the experiment, pronounced 3,000 sentences with 7 different emotions (neutral, happy, sad, anger, fear, surprise, disgust), so the total number of utterances is 20,000. It reaches a thousand. The text of every sentence is the same throughout the emotion. All recordings are done in a silent studio and recorded at a sampling rate of 44.1 kHz. The total time after finishing the static is 29.2 hours.

For pre-processing, the statics at the first and end of each waveform are trimmed using a voice activity detection (VAD) algorithm and resampled at 16 kHz. Then, the log mel spectrogram is extracted with a window size of 50 ms, shift 12.5 ms, nfft 2048, 80 Mel bins and Hanning windows. Its size was standardized to [0, 1]. Since the text encoder uses a character-based representation, Korean characters are decomposed into onset, nucleus, and coda, and then converted into a one-hot representation.

Detailed parameter settings are as follows. According to an embodiment of the present invention, 256-character embedding and 32-dimensional hs were used. The initial learning rate was 1e-3 for Adam's optimal period. Gradient clipping was used as 1, and m was created with a reduction factor of 5. The batch size was 32, and Bahdanau attention was used. The content encoder uses two layers, the bidirectional LSTM and the style encoder, and the content encoder consists of two layers, the unidirectional LSTM. The output of the last step is only used by the content encoder. The parameters of the content encoder and style encoder are not shared. In the learning stage, teacher forcing was used to prevent the cumulative loss of the predicted products. When creating a mini-batch, each sample is zero-padding to the longest length of the sample. Thereafter, the loss of the zero-faded region is also reverse transcribed for inference stability. For every iteration, the network's work is randomly determined by VC or TTS. For each sample, the source and target emotions are chosen differently.

Three different models were trained and evaluated to verify the linguistic consistency of the proposed model. VCTS is a model that combines TTS and VC as described above. VC refers to a voice converter without a TTS path. TTS refers to a TTS model that does not include a VC path. VCTTS-V is a voice conversion inference model using VCTTS, and VCTTS-T is a model when the TTS path of VCTTS is activated.

The word error rate (WER) was calculated to measure how the proposed model improves the consistency of the transformed language. In fact, morphemes were used instead of words because they were regarded as the unit of recognition in Korean. The Google Cloud Voice-to-Text API recorded the converted voices, and the transcript was divided into a series of morphemes by KoNLPy's Komoran Morphological Analyzer. Thereafter, the average WER between the two morpheme sequences of the actual record and the automatically recognized record was calculated and displayed in Table 1. As a result, VCTTS-V was superior to VC, and WER of TTS was superior to VCTTS-T.

<Table 1>

After learning, 8 native speakers participated in the subjective evaluation. Twenty sentences were generated by the VCTTS-V model. The emotion of x _c was set to neutral, the emotion of the object was set to happiness, and the sentence of _{x s was fixed.} We blindly test that subjects never know which model produces which voice. Subjects were asked to rate their clarity on a scale of 1 to 5. In addition, the ABX test of preference between the two models was conducted. Given two voices with no information on which model produced which sample, subjects were asked to choose a clearer voice. Subjects cannot choose any of the two samples if they are perceived to be similar. The results are shown in Table 2. It shows that VCTTV-V has a high MOS and ABX preference score, which means that the multitask learning of VC and TTS helps to maintain language information.

<Table 2>

3 is a diagram illustrating a congestion matrix of a style encoder according to an embodiment of the present invention.

To investigate emotional speech transformation, the VCTTS model mentioned above is used for inference. After training, 20 samples were randomly selected for each emotion, and the samples were fed to the model. Then, h _s can be obtained per sample, and the cosine similarity between each sample is measured and shown in FIG. 3. The average value of cosine similarity between all emotion pairs is also displayed.

In FIG. 3, it is shown that the sample of the same emotion has very high cosine similarity, while the emotion pair other than the diagonal line has low similarity. It implies that the style encoder can extract the emotional style regardless of the linguistic content. Except for the diagonal emotion pair, the emotion pair (disgusting-anger) exhibits a relatively high similarity, which means that the intrinsicity of these two emotions is closer than the other emotions. (Sad-Fear) The same voice is also observed in the pair.

4 is a diagram illustrating an example of a voice conversion result according to an embodiment of the present invention.

Emotional voice conversion should reflect emotions while maintaining the linguistic content. 4 shows an example of voice conversion. Given a neutral voice, the given x _s transforms into 6 different emotions. The content of x _s is fixed during this experiment. The first row of FIG. 4 is a log mel spectrogram of an input voice, and a log mel spectrogram of the converted voice from the second row to the seventh row. While the overall shape of the spectrogram is similar to the input, it can be seen that some changes such as time shift, frequency shift, and pause time have been made. Within a single model, it can create multiple language domains.

The apparatus described above may be implemented as a hardware component, a software component, and/or a combination of a hardware component and a software component. For example, the devices and components described in the embodiments include, for example, a processor, a controller, an arithmetic logic unit (ALU), a digital signal processor, a microcomputer, a field programmable array (FPA), It may be implemented using one or more general purpose or special purpose computers, such as a programmable logic unit (PLU), a microprocessor, or any other device capable of executing and responding to instructions. The processing device may execute an operating system (OS) and one or more software applications executed on the operating system. Further, the processing device may access, store, manipulate, process, and generate data in response to the execution of software. For the convenience of understanding, although it is sometimes described that one processing device is used, one of ordinary skill in the art, the processing device is a plurality of processing elements and/or a plurality of types of processing elements. It can be seen that it may include. For example, the processing device may include a plurality of processors or one processor and one controller. In addition, other processing configurations are possible, such as a parallel processor.

The software may include a computer program, code, instructions, or a combination of one or more of these, configuring the processing unit to behave as desired or processed independently or collectively. You can command the device. Software and/or data may be interpreted by a processing device or, to provide instructions or data to a processing device, of any type of machine, component, physical device, virtual equipment, computer storage medium or device. Can be embodyed. The software may be distributed over networked computer systems and stored or executed in a distributed manner. Software and data may be stored on one or more computer-readable recording media.

The method according to the embodiment may be implemented in the form of program instructions that can be executed through various computer means and recorded in a computer-readable medium. The computer-readable medium may include program instructions, data files, data structures, and the like alone or in combination. The program instructions recorded on the medium may be specially designed and configured for the embodiment, or may be known and usable to those skilled in computer software. Examples of computer-readable recording media include magnetic media such as hard disks, floppy disks, and magnetic tapes, optical media such as CD-ROMs and DVDs, and magnetic media such as floptical disks. -A hardware device specially configured to store and execute program instructions such as magneto-optical media, and ROM, RAM, flash memory, and the like. Examples of program instructions include not only machine language codes such as those produced by a compiler, but also high-level language codes that can be executed by a computer using an interpreter or the like.

Although the embodiments have been described by the limited embodiments and drawings as described above, various modifications and variations can be made from the above description to those of ordinary skill in the art. For example, the described techniques are performed in a different order from the described method, and/or components such as systems, structures, devices, circuits, etc. described are combined or combined in a form different from the described method, or other components Alternatively, even if substituted or substituted by an equivalent, an appropriate result can be achieved.

Therefore, other implementations, other embodiments, and those equivalent to the claims also fall within the scope of the claims to be described later.

Claims (12)

If the pair of input voices is a log mel spectrogram of a language delivering language content and a log mel spectrogram of a style reference voice, performing voice conversion (VC);
When the pair of input voices is a log mel spectrogram of a one-hot representative text and a style reference voice, performing text-to-speech (TTS);
Mapping both a log mel spectrogram and a one-hot representative text of a language delivering language content to the same space and then decoding them into a mel spectrogram; And
Acquiring a linear spectrum from the decoded mel spectrogram through a preprocessor
Emotional voice conversion method comprising a. The method of claim 1,
When the pair of input voices is a log mel spectrogram of a language delivering language content and a log mel spectrogram of a style reference voice, the step of performing speech conversion includes:
The log mel spectrogram of the language delivering the language content is embedded through the content encoder, and the log mel spectrogram of the style reference voice is embedded through the style encoder.
Emotional speech conversion method. The method of claim 1,
When the pair of input voices is a log mel spectrogram of one-hot representative text and style reference voice, the step of performing text-to-speech conversion includes:
One-hot representative text is embedded through the text encoder, and the log mel spectrogram of the style reference voice is embedded through the style encoder.
Emotional speech conversion method. The method of claim 1,
Both the log mel spectrogram and the one-hot representative text of the language conveying the language content are mapped to the same space and then decoded into the mel spectrogram.
In each decoding step, the style vector extracted from the log mel spectrogram of the style reference voice is connected to the attention RNN and the decoder RNN.
Emotional speech conversion method. The method of claim 2,
Both the log mel spectrogram and the one-hot representative text of the language conveying the language content are mapped to the same space and then decoded into the mel spectrogram.
When text-to-speech is performed through the text encoder, the attention RNN, the decoder RNN, and the preprocessor, the context vector is used for all repetitions of the attention RNN.
Emotional speech conversion method. The method of claim 1,
When considering style reference speech, the style encoder is designed to extract only emotion information and remove linguistic content, and to extract emotion regardless of linguistic content to process multiple input style domains, and when the extracted emotion is injected into the decoder, To process many-to-many emotional speech transformations by creating a variety of emotions.
Emotional speech conversion method. Whether the pair of input voices is the log Mel spectrogram of the language conveying the language content and the log Mel spectrogram of the style reference voice, or a one-hot representative of the text and style reference voice. A conversion unit that performs speech conversion or text-to-speech conversion through a style encoder, a content encoder, and a text encoder according to whether it is a log mel spectrogram;
A neural network network in which both log mel spectrograms and one-hot representative texts of a language delivering language content are mapped to the same space and then decoded into mel spectrograms; And
Preprocessor to obtain a linear spectrum from the decoded mel spectrogram
Emotional speech conversion device comprising a. The method of claim 7,
The conversion unit,
If the pair of input voices is the log mel spectrogram of the language that conveys the language content and the log mel spectrogram of the style reference voice, voice conversion is performed, and
The log mel spectrogram of the language delivering the language content is embedded through the content encoder, and the log mel spectrogram of the style reference voice is embedded through the style encoder.
Emotional speech conversion device. The method of claim 7,
The conversion unit,
If the pair of input speech is a log mel spectrogram of one-hot representative text and style reference speech, text-to-speech is performed, and
One-hot representative text is embedded through the text encoder, and the log mel spectrogram of the style reference voice is embedded through the style encoder.
Emotional speech conversion device. The method of claim 7,
Neural network network,
In each decoding step, the style vector extracted from the log mel spectrogram of the style reference voice is connected to the attention RNN and the decoder RNN.
Emotional speech conversion device. The method of claim 8,
Neural network network,
When text-to-speech is performed through the text encoder, the attention RNN, the decoder RNN, and the preprocessor, the context vector is used for all repetitions of the attention RNN.
Emotional speech conversion device. The method of claim 7,
When considering style reference speech, the style encoder is designed to extract only emotion information and remove linguistic content, and to extract emotion regardless of linguistic content to process multiple input style domains, and when the extracted emotion is injected into the decoder, To process many-to-many emotional speech transformations by creating a variety of emotions.
Emotional speech conversion device.

Images