KR20230075340A - Voice synthesis system and method capable of duplicating tone and prosody styles in real time - Google Patents

Abstract

Disclosed are a voice synthesis system and a method capable of duplicating one and prosody styles in real time which can rapidly and accurately synthesize voices according to various prosody and tone styles of a speaker by using deep learning-based end-to-end voice synthesis technology using a global style token. The voice synthesis method comprises the steps of: extracting a basic frequency from an input reference audio to transmit the same as an input of a reference encoder; encoding the basic frequency by the reference encoder to generate a prosody embedding; generating a first style embedding from the prosody embedding; converting the reference audio into a reference Mel Spectrogram by Fourier transform; encoding the reference Mel Spectrogram to generate a speaker embedding; generating a second style embedding from the speaker embedding; inputting, into attention of a text to speech (TTS) model, an integrated style embedding summing the first style embedding and the second style embedding along with an output of an encoder of the TTS model; and generating an audio of voice synthesis having tone and prosody combined for an input text by the TTS model.

Description

Voice synthesis system and method capable of replicating real-time tone and rhyme style

The present invention relates to a speech synthesis system that quickly and accurately synthesizes speech according to various prosody and speaker timbre styles, and deep learning-based end-to-end speech synthesis using Global Style Token (GST) A voice synthesis system and method capable of replicating real-time timbre and prosody using technology.

The ultimate goal of Text to Speech (TTS) is a technology that reads input text with an accurate and natural voice, and can be applied to all areas where people directly vocalize. Typical applications of speech synthesis include navigation, audio books, and artificial intelligence assistants.

In addition, the quality of speech synthesized using rapidly developing deep learning technology has been greatly improved, and as end-to-end speech synthesis models have been proposed, relatively simple model learning has become possible without additional knowledge or work.

Despite these developments, the current deep learning-based end-to-end speech synthesis system has limitations in that it cannot follow the prosodic information of voice that can be easily heard in everyday conversation, such as emotion, intonation, tone, etc., or the tone of the speaker. Research is required.

The present invention was derived to meet the above-mentioned needs of the prior art, and an object of the present invention is a speech synthesis system and method capable of replicating real-time timbre and prosody style capable of replicating various prosody elements and timbres of speakers in real time. is providing

Another object of the present invention is to provide a voice synthesis system and method capable of replicating and synthesizing voices and prosody styles in real time using a global style token (GST) based module.

Another object of the present invention is to provide a voice synthesis system and method capable of emotion control together with real-time tone and prosody style replication.

A voice synthesis method according to an aspect of the present invention for achieving the above technical problem is a voice synthesis method capable of replicating real-time timbre and prosody style, and extracts a fundamental frequency from an input reference audio (reference waveform) and uses it as an input of a reference encoder. conveying; generating, by the reference encoder, a prosody embedding by encoding the fundamental frequency; generating a first style embedding from the prosody embedding; converting the reference audio into a reference mel spectrogram by Fourier transform; encoding the reference mel spectrogram to generate a speaker embedding; generating a second style embedding from the speaker embedding; inputting an integrated style embedding obtained by combining the first style embedding and the second style embedding into attention of a text to speech (TTS) model; and generating audio in which a tone and a prosody synthesized by the unified style embedding are synthesized with respect to the input text by the TTS model.

In one embodiment, in the step of extracting the fundamental frequency, the reference audio is cut in units of a sliding window of a certain section, and a pitch contour is calculated through frequency decomposition of the reference audio using a normalized cross-correlation function. process may be included. Here, the pitch may correspond to the pitch of the reference audio.

In one embodiment, the step of generating the first style embedding uses a hierarchical GST of three layers consisting of a first global style token (GST) layer to a third global style token layer. to generate the first style embedding for the prosody of the reference audio.

In one embodiment, each of the first to third GST layers includes a multi-head attention that transfers an input embedding to a plurality of tokens of each GST layer, and a plurality of tokens connected to the multi-head attention, , It may be configured to measure the similarity between the input embedding and each token and generate a style embedding as a weighted sum of each token.

In one embodiment, the first to third GST layers may have a residual connection connecting the plurality of tokens of the current GST layer and the plurality of tokens of the previous GST layer.

In one embodiment, the generating of the second style embedding may be configured to generate the second style embedding for the timbre of the reference audio using a fourth GST layer.

In one embodiment, the generating of the audio may include extracting, by an encoder of the TTS model, feature information from characters of text input to the encoder; mapping the feature information to attention alignment information to be used in the decoder of the TTS model at each time point by the attention, and transmitting the mapped attention alignment information and the unified style embedding to the decoder; and generating, by the decoder, a mel spectrogram of the current view synthesized with the unified style embedding using the attention information of the attention and the mel spectrogram of a previous view.

In one embodiment, generating the audio may further include generating audio from the Mel spectrogram of the current view by Mel generative adversarial networks (MelGAN).

In one embodiment, the voice synthesis method compares a target mel spectrogram corresponding to the reference mel spectrogram with a predicted mel spectrogram generated from the TTS model for the input text and the reference audio, and then the target mel spectrogram. obtaining a difference between a spectrogram and the predicted MEL spectrogram or a loss corresponding to the difference; and training the TTS model until the difference or loss is equal to or less than a preset level or a reference value.

A voice synthesis system according to another aspect of the present invention for achieving the above technical problem is a voice synthesis system capable of replicating real-time timbre and prosody style, and includes an extractor for extracting a fundamental frequency from an input reference audio (reference waveform); a first reference encoder that receives the fundamental frequency and encodes the fundamental frequency to generate a prosody embedding; hierarchical global style token layers for generating a first style embedding for a prosody of the reference audio from the prosody embedding; a transform unit for transforming the reference audio into a reference mel spectrogram by Fourier transform; a second reference encoder for encoding the reference mel spectrogram to generate a speaker embedding; a single global style token layer for generating a second style embedding for a timbre of the reference audio from the speaker embedding; and text-to-speech, which receives the unified style embedding obtained by combining the first style embedding and the second style embedding as attention, and generates audio in which the tone and rhyme synthesized by the unified style embedding are synthesized for the input text. TTS) model.

In one embodiment, the extraction unit cuts the reference audio in units of a sliding window of a predetermined interval, classifies an effective frame using a normalized cross-correlation function, and performs frequency decomposition of the reference audio within the effective frame. It can be configured to calculate pitch contours. Here, the pitch may correspond to the pitch of the reference audio.

In one embodiment, the extractor may estimate the fundamental frequency corresponding to the pitch contour using the YIN method. Here, YIN is a compound word of Yin and Yang in Eastern philosophy, implying the interaction between autocorrelation and cancellation.

In one embodiment, the hierarchical global style token layers may be composed of three hierarchical GST layers consisting of a first global style token (GST) layer to a third global style token layer. .

In one embodiment, each GST layer of the hierarchical global style token layers may include multiple heads of attention and a plurality of tokens connected to the multiple heads of attention. Here, the first GST layer measures the similarity between the rhyme embedding and each token and generates a style embedding as a weighted sum of each token, and the style embedding generated in the first GST layer is the second GST layer and the second GST layer. It can be generated as a first style embedding by sequentially passing through 3 GST layers.

In one embodiment, in the hierarchical global style token layers, the first pair of the first GST layer and the second GST layer and the second pair of the second GST layer and the third GST layer are a current layer The tokens of may have residual connections connected to tokens of the previous layer.

In one embodiment, the speech synthesis system compares a target mel spectrogram corresponding to the reference mel spectrogram with a predicted mel spectrogram generated from the TTS model for the input text and the reference audio, and compares the target mel spectrogram. The method may further include a learning management unit that obtains a difference between the MEL spectrogram and the predicted MEL spectrogram or a loss corresponding to the difference. The learning management unit may train the TTS model until the difference or loss is less than or equal to a preset level or a reference value.

In one embodiment, the TTS model may include a spectrogram predictor and a vocoder. Here, the spectrogram predictor may generate a mel spectrogram based on the input text and the integrated style embedding, and the vocoder may generate a waveform sample corresponding to a synthesized waveform from the mel spectrogram.

In one embodiment, the spectrogram predictor may include an encoder, an attention and a decoder. The encoder may extract feature information from characters of the input text. The attention may map the feature information into attention alignment information to be used by the decoder at each time point and deliver the mapped attention alignment information and the unified style embedding to the decoder. Also, the decoder may generate a current view MEL spectrogram synthesized with the unified style embedding using the attention information of the attention and the MEL spectrogram of a previous view.

In one embodiment, the spectrogram predictor may further include a preprocessor located at an input terminal of the encoder. The pre-processing unit may be configured to divide input text into syllable units and express the separated syllables as integers through one-hot encoding.

In one embodiment, the encoder includes a character embedding unit, 3 convolution layers connected to the character embedding generation unit, and a bidirectional LSTM connected to the three convolution layers. (bidirectional long short term memory) may be provided. The character embedding generating unit may convert the integer sequence received from the pre-processing unit into a matrix form. The convolutional layers may condense matrix-type information. And, the bidirectional LSTM can convert information in a reduced matrix form into encoder feature information. Here, the encoder characteristic information may include a context vector compressed to one fixed size.

In one embodiment, the Attention sets the Attention Align information to the encoder characteristic information according to the time-step of the decoder so that the voice pronunciation to be output through the vocoder of the TTS model proceeds in the sequential order of the input text. can be configured to add

In one embodiment, the vocoder may be Mel generative adversarial networks (MelGAN).

In one embodiment, a voice synthesis system may include a third reference encoder for encoding the reference audio to generate emotion embedding; and another global style token layer generating a third style embedding for emotion information included in the reference audio from the emotion embedding. The third style embedding may be included in the integrated style embedding along with the first style embedding and the second style embedding and input to the attention of the TTS model.

In one embodiment, the third reference encoder includes three convolutional neural networks (3 CNNs) to which a mel spectrogram (hereinafter referred to as 'reference mel spectrogram') converted from the reference audio through SFTF is input, the It can be composed of 2 residual blocks connected to 3 convolutional neural networks and a modified AlexNet of 4 layers connected to the 2 residual blocks.

In one embodiment, the TTS model combined with the third reference encoder and the another power style token layer is configured to adjust the style transfer of reference audio and the strength of the style transfer, and learns only with reconstruction loss. can proceed.

A voice synthesis system according to another aspect of the present invention for achieving the above technical problem is stored in a computer-readable recording medium for implementing a voice synthesis method capable of replicating real-time tone and prosody style of any one of the above-described embodiments. It may contain a computer program.

A voice synthesis system according to another aspect of the present invention for achieving the above technical problem is a computer-readable record recording a program for implementing a voice synthesis method capable of replicating real-time tone and prosody style of any one of the above-described embodiments. media may be included.

According to the configuration of the above-described voice synthesis system and method, a global style token (GST) based module can be used to reproduce the timbre or prosody style of the reference audio and effectively reflect it in the voice synthesis result for the input text. .

In addition, according to the present invention, it is possible to effectively generate voice synthesized waveforms reflecting various styles of timbre, prosody, and emotion based on global style tokens with respect to input text.

In addition, according to the present invention, a speech synthesis system and method capable of replicating various prosodic elements of reference audio, a speaker's timbre, or a speaker's emotions in real time and adding them to a speech synthesis output for input text in real time are provided. can

1 is a schematic block diagram of the entire configuration of a voice synthesis system capable of replicating real-time timbre and prosody style (hereinafter referred to as 'speech synthesis system') according to an embodiment of the present invention.
2 is a block diagram of a voice synthesis system of a comparative example.
FIG. 3 is a configuration diagram of a prosody module that can be employed in the voice synthesis system of FIG. 1 .
FIG. 4 is a detailed configuration diagram of an entire structure that can be employed in the voice synthesis system of FIG. 1. Referring to FIG.
5 is an overall configuration diagram of a voice synthesis system according to another embodiment of the present invention.
FIG. 6 is a block diagram for explaining an emotion module that can be employed in the voice synthesis system of FIG. 5 .
FIG. 7 is a detailed configuration diagram of a reference encoder that can be employed in the emotion module of FIG. 6 .
8 is a schematic configuration diagram of a voice synthesis system according to another embodiment of the present invention.
FIG. 9 is a block diagram for explaining main operations applicable to the speech synthesis system of FIG. 8 .

Since the present invention can have various changes and various embodiments, it will be described in detail by exemplifying specific embodiments in the drawings. However, this is not intended to limit the present invention to specific embodiments, and should be understood to include all modifications, equivalents, and substitutes included in the spirit and scope of the present invention. Like reference numerals have been used for like elements throughout the description of each figure.

Terms such as first and second may be used to describe various components, but the components should not be limited by the terms. These terms are only used for the purpose of distinguishing one component from another. For example, a first element may be termed a second element, and similarly, a second element may be termed a first element, without departing from the scope of the present invention. The term 'and/or' includes a combination of a plurality of related recited items or any one of a plurality of related recited items.

In embodiments of the present application, 'at least one of A and B' may mean 'at least one of A or B' or 'at least one of combinations of one or more of A and B'. Also, in the embodiments of the present application, 'at least one of A and B' may mean 'at least one of A or B' or 'at least one of combinations of one or more of A and B'.

It is understood that when a component is referred to as being 'connected' or 'connected' to another component, it may be directly connected or connected to the other component, but other components may exist in the middle. It should be. On the other hand, when a component is referred to as being 'directly connected' or 'directly connected' to another component, it should be understood that no other component exists in the middle.

Terms used in this application are only used to describe specific embodiments, and are not intended to limit the present invention. Singular expressions include plural expressions unless the context clearly dictates otherwise. In this application, terms such as 'comprise' or 'having' are intended to designate that there is a feature, number, step, operation, component, part, or combination thereof described in the specification, but one or more other features It should be understood that the presence or addition of numbers, steps, operations, components, parts, or combinations thereof is not precluded.

Unless defined otherwise, all terms used herein, including technical or scientific terms, have the same meaning as commonly understood by one of ordinary skill in the art to which the present invention belongs. Terms such as those defined in commonly used dictionaries should be interpreted as having a meaning consistent with the meaning in the context of the related art, and unless explicitly defined in the present application, they should not be interpreted in an ideal or excessively formal meaning. don't

Hereinafter, with reference to the accompanying drawings, preferred embodiments of the present invention will be described in more detail. In order to facilitate overall understanding in the description of the present invention, the same reference numerals are used for the same components in the drawings, and redundant descriptions of the same components are omitted.

1 is a schematic block diagram of the entire configuration of a voice synthesis system capable of replicating real-time timbre and prosody style (hereinafter referred to as 'speech synthesis system') according to an embodiment of the present invention.

Referring to FIG. 1, the speech synthesis system includes a prosody module 100, a timbre module 200, a spectrum predictor 600 and a vocoder 700, and a first style of the prosody module 100 By applying an integrated style embedding that combines the style embedding and the second style embedding of the tone module 200, a raw waveform with a specific rhyme and tone added to the input text can be output. .

The prosody module 100 includes an extractor 110, a first reference encoder (RE, 120), a first global style token layer (GST1, 130), and a second GST layer (GST2, 140). ) and a third GST layer (GST3, 150).

The extraction unit 110 extracts a fundamental frequency (f0) from reference audio, which is an input reference waveform. The extraction unit 110 may be referred to as a feature extractor (FE) that extracts features related to prosody from reference audio. Here, the rhyme may refer to a pitch of reference audio at each reference point or to a change in pitch and stress.

The first reference encoder 120 encodes the reference audio of the fundamental frequency received from the extractor 110 to generate prosody embedding. The first reference encoder 120 filters the sliding window units of a certain period through three convolutional neural networks having a batchNorm, calculates weights using previously learned information through two residual blocks, and then , can be made to optimize the weights with a modified alexnet with 4 layers. Here, the prosody embedding is a parameter describing the prosody of the reference audio, and the prosody embedding generated after learning is completed may be an optimal parameter that best describes the reference audio.

The first GST layer (GST1, 130), the second GST layer (GST2, 140), and the third GST layer (GST3, 150) form a hierarchical GST structure. That is, the first to third GST layers 130, 140, and 150 have a structure in which embedding is sequentially processed in the order described. Each GST layer includes multiple heads of attention and a plurality of tokens connected to the multiple heads of attention. These hierarchical GST layers (hereinafter simply referred to as 'hierarchical GST') are intended to appropriately and effectively process prosody embedding input from the first reference encoder 120 .

In the hierarchical GST, the first GST layer 130 measures the similarity between the prosody embedding and each token thereof and creates a style embedding as a weighted sum of the tokens. That is, the first GST layer 130 may express the prosody embedding by changing it to a weighted sum of k (any natural number greater than or equal to 5) global style tokens (GSTs). In this process, k GSTs are learned so that their combinations can express various prosody styles. In particular, in this embodiment, the style embedding learned in the first GST layer 130 is re-learned in the second GST layer 140 and the third GST layer 150, thereby reducing processing time and Considering the system complexity, it is configured to learn as effectively as possible. The learning result of the hierarchical GST is output as a first style embedding.

Moreover, in this embodiment, for more effective learning of prosodic embeddings, the hierarchical GST consists of a first pair of first GST layer 130 and second GST layer 140, a first pair of second GST layer 140 and a third GST layer 140. It may be configured to have a residual connection for each of the second pair of GST layers 150 .

When learning various expressions of prosody embedding using the three GST layers 130, 140, and 150, the current GST layer placed relatively later in the data processing flow is arranged relatively earlier. This is to make the previous GST layer use the learning result, and can contribute to maximizing the learning effect of prosody embedding.

On the other hand, in this embodiment, it has been described that the hierarchical GST is composed of three GST layers, but the present invention is not limited to such a configuration, and the hierarchical GST is composed of two, four, or five GST layers. Even in the case of configuration, of course, effects similar to those of the present embodiment can be obtained. Configuration of more than five GST layers may be limited in consideration of processing time and system complexity.

The tone module 200 includes a second reference encoder (RE, 220) and a fourth GST layer (GST4, 240), and converts the speaker embedding converted by the second reference encoder (220) into the fourth GST layer (240). learn and output second style embeddings.

As an input of the second reference encoder 220, a spectrogram generated through Fourier transform by cutting the reference audio into sliding window units having a specific length may be used. The process of creating such a spectrogram may be briefly referred to as short-time fourier transform (STFT). The spectrogram may be converted into a Mel spectrogram through log transform using a Mel-filter bank.

The fourth GST layer 240 is used to effectively learn speaker embedding. The fourth GST layer 240 measures the similarity between the speaker embedding and each of its own tokens, and generates a style embedding (second style embedding) as a weighted sum of the tokens. That is, the fourth GST layer 240 can express the speaker embedding by transforming it into a weighted sum of k (any natural number greater than or equal to 5) global style tokens (GSTs). In this process, the k GSTs are a combination of these It is learned to express various timbre styles.

A spectrum predictor (600) and a vocoder (700) form a text to speech (TTS) model (500). In this embodiment, the TTS model 500 receives the unified style embedding that combines the first style embedding and the second style embedding, and when generating a voice synthesis result for the input text, that is, the input text, the unified style embedding It is configured to reflect the timbre and prosody corresponding to.

Here, the spectrum predictor 600 generates a mel spectrogram for the input text reflecting the unified style embedding. And, the vocoder 700 outputs a raw waveform from the mel spectrogram. The initial voice waveform may have a specific format such as a synthesized waveform or waveform audio format (WAV) as waveform samples, but is not limited thereto.

The spectrum predictor 600 may include an encoder into which text is input, an attention connected to the encoder, and a decoder connected to the attention.

The vocoder 700 is a module that generates a waveform signal or audio signal from a spectrogram or a MEL spectrogram. The vocoder 700 may use a WaveNet vocoder, a Mel generative adversarial network (MelGAN), or the like. Using MelGAN, it is possible to generate a voice signal that better expresses the timbre or rhyme of the reference audio than other vocoders such as the Wavenet vocoder.

2 is a block diagram of a voice synthesis system of a comparative example.

Referring to FIG. 2, the speech synthesis system of the comparative example is composed of a spectrum predictor 10, a reference encoder 20, and a WaveNet vocoder 90. The spectrum predictor 10 includes an encoder 30 ), attention (50), and decoder (70).

When compared with the voice synthesis system of the present embodiment described above (see FIG. 1), the voice synthesis system of the comparative example includes the reference embedding generated from the reference audio in the reference encoder 20 and the characteristic information of the encoder 30. When combined and input to the attention 50, the attention 50 and the decoder 70 generate a mel spectrogram through their collaboration, and the wavenet vocoder 90 generates a speech synthesis result from the mel spectrogram, speech) is configured to generate.

In comparison with the voice synthesis system of the above-described comparative example, the voice synthesis system of the present embodiment uses the fundamental frequency of the reference audio as an input to the first reference encoder in order to effectively express the prosody of the reference audio, and in the first reference encoder It can be seen that the prosody is configured to learn the prosody by inputting the generated prosody embedding to a plurality of GST layers (in particular, first to third GST layers) of the hierarchical GST.

In addition, the speech synthesis system of the present embodiment uses the spectrogram of the reference audio as an input of the second reference encoder in order to effectively represent the timbre of the reference audio, and uses the speaker embedding generated by the second reference encoder as a single GST layer ( It can be seen that it is configured to learn the timbre by inputting to the fourth GST layer).

In addition, the speech synthesis system of the present embodiment combines the first style embedding obtained as a result of prosody learning and the second style embedding obtained as a result of timbre learning, and adds them to the encoder output of the spectrum predictor or inputs the attention of the spectrum predictor together with the output of the encoder. It can be seen that it is configured so that

Moreover, as will be described later, the speech synthesis system of the present embodiment uses the spectrogram of the reference audio as an input of the third reference encoder to effectively express the emotion of the reference audio, and embedding the emotion generated by the third reference encoder. It can be seen that it is configured to learn the emotion by inputting to a single GST layer (fifth GST layer). Here, the third style embedding output from the fifth GST layer may be combined with the first and second style embeddings and transferred to the TTS model including the spectrum predictor.

In addition, it can be seen that the voice synthesis system of the present embodiment is configured to more effectively generate voice synthesis results in which timbres and rhymes or timbres, rhymes, and emotion styles are reproduced more effectively by using MelGAN as a vocoder.

FIG. 3 is a configuration diagram of a prosody module that can be employed in the voice synthesis system of FIG. 1 .

Referring to FIG. 3, the prosody module 100 includes an extractor (FE, 110), a reference encoder (120), a first GST layer (GST1 or GST layer 1, 130) to an Nth GST layer (GST layer). N, 160). Here, the reference encoder 120 corresponds to the first reference encoder, and N may be any one natural number from 2 to 5.

The prosody module 100 extracts a fundamental frequency f0 from a reference waveform or reference audio input by the extractor 110, and prosody is embedded from the fundamental frequency by the first reference encoder 120. embedding), learn prosody embedding by the hierarchical GST of the first to Nth GST layers 130 and 160, and output a first style embedding.

The first GST layer 130 includes a multi-head attention 132 and a plurality of tokens (token 1 to token k) 134 connected to the multi-head attention 132 . Here, k may be any natural number greater than or equal to 5. The first GST layer 130 measures the similarity between the rhyme embedding and each token (token 1 to token k) and creates a style embedding as a weighted sum of the tokens. The style embedding generated in the first GST layer 130 may be transferred as an input of the second GST layer.

Similar to the above case, the Nth GST layer 160 includes a multi-head attention 162 and a plurality of tokens (token 1 to token k) 164 connected to the multi-head attention 162 . The Nth GST layer 160 measures the similarity between the style embeddings received from other GST layers located at the input terminal and each token (token 1 to token k), and creates a style embedding as a weighted sum of these tokens. This style embedding becomes the first style embedding finally output to the prosody module 100 .

On the other hand, when N is 2, the second GST layer measures the similarity between the style embedding received from the first GST layer 130 and each of its plurality of tokens, and generates a style embedding as a weighted sum of the tokens can do. In this case, the plurality of tokens 134 of the first GST layer 130 may be connected to the plurality of tokens of the second GST layer through a residual connection 170, whereby the second GST layer may be connected to the first Learning may be performed in parallel based on weights of the plurality of tokens 134 of the GST layer 130 .

In addition, when N is 3 and N is 2, the third GST layer measures the similarity between the style embedding received from the second GST layer and each of its own tokens, You can create style embeddings with a weighted sum. In this case, the plurality of tokens of the second GST layer may be connected to the plurality of tokens of the third GST layer through another residual connection (170), whereby the third GST layer is connected to the plurality of tokens of the second GST layer. Based on weights, learning can be performed effectively.

FIG. 4 is a detailed configuration diagram of an entire structure that can be employed in the voice synthesis system of FIG. 1. Referring to FIG.

Referring to FIG. 4, the speech synthesis system includes a prosody module 100, a timbre module 200, a spectrum predictor 600, and MelGAN 700a as a vocoder, and the prosody module 100 1 style embedding and the 2nd style embedding of the voice module 200 are combined to apply the integrated style embedding to output waveform samples with a specific rhyme and timbre added to the input text. there is.

The prosody module 100 includes an extractor (FE, 110), a first reference encoder (120), a first style token layer (style token layer 1, 130), a second style token layer (style token layer 2, 140) and a third style token layer (style token layer 3, 150). Each style token layer corresponds to a global style token (GST) layer.

In the prosody module 100, the extraction unit 110 extracts a fundamental frequency (f0) from reference audio, which is a reference waveform. Reference audio may be audio in a pre-designated dataset. In addition, the reference audio may be original, target, or ground truth (GT) representing a true value during training, and may be audio including a desired style during inference. The fundamental frequency is used as an input of the first reference encoder 120.

The first reference encoder 120 generates a prosody embedding from the fundamental frequency. The first reference encoder 120 may include six 2D convolutional neural networks (CNNs) and one gated recurrent unit (GRU). The prosody embedding is a result of passing the basic frequency through the first reference encoder, is an embedding of a fixed length, and is used as an input of the hierarchical GST.

A hierarchical GST composed of three style token layers 130, 140, and 150 may learn a prosody embedding and generate a first style embedding. Each style token layer (130, 140, 150) has multi-head attention. In the hierarchical GST, when the similarity between the prosody embedding output from the first reference encoder 120 and each style token is measured through multihead attention, the style embedding is generated as a weighted sum of the style tokens.

Unlike the style embedding of the voice module 200, the style embedding of the prosody module 100 is different from the style embedding of the tone module 200, when the first style embedding additionally passes through two style token layers, the first style token layer and the second style Residual connections may be additionally used between the token layer and between the second style token layer and the third style token layer, respectively, to maximize the learning effect of the prosody characteristics of the reference audio.

The timbre module 200 includes a STFT unit 210, a second reference encoder 220, and a fourth style token layer (style token layer 4, 240). The STFT unit 210 may be omitted when a spectrogram or a Mel spectrogram separately processed from reference audio is prepared as input data of the second reference encoder 220 . The fourth style token layer 240 corresponds to the fourth GST layer.

In the tone module 200, the STFT unit 210 provides a Mel spectrogram obtained by transforming the reference audio by Fourier transform to the second reference encoder 220 as an input. The second reference encoder 220 receives a mel spectrogram rather than a fundamental frequency and generates a speaker embedding. The speaker embedding that has passed through the fourth GST layer 240 is output as a second style embedding.

The fourth style token layer 240 may generate a second style embedding by learning the speaker embedding. The fourth style token layer 240 includes multi-head attention 242 and a plurality of tokens 244 . The plurality of tokens 244 may include a first token (token 1) to a fifth token (token 5), and each token may be referred to as a style token.

The fourth style token layer 240, when the similarity between the speaker embedding output from the second reference encoder 220 and each style token is measured through the multi-head attention 242, the weight of the style tokens 244 The sum can generate a style embedding (second style embedding).

The style embedding (first style embedding) passing through a total of three style token layers in the tone module 100 and the style embedding (second style embedding) in the tone module 200 are concatenated, and the spectrum predictor of the speech synthesis system Combined with the output of the encoder at 600.

The voice synthesis system of the present embodiment may be composed of a spectrum predictor 600 and MelGAN 700a as a vocoder in a narrow sense, and may further include a tone module 100 and a tone module 200 in a broad sense. . Here, the spectrum predictor 600 includes an encoder 610, an attention 620, and a decoder 630.

That is, the spectrum predictor 600 includes an encoder 610 receiving input text, an attention 620 connected to an output terminal of the encoder 610, and a decoder 630 connected to the attention 620. . This spectrum predictor 600 may be implemented as Tacotron2.

In the above case, the speech synthesis system is a seq2seq speech synthesis system and can synthesize text into speech through two steps. In the first step, a mel spectrogram is generated from the input text, and in the second step, a voice is synthesized from the mel spectrogram using a vocoder.

Compared to the general Tacotron2, the spectrum predictor 600 of the present embodiment is input to the attention 620 by combining the combined style embedding of the first and second style embeddings with the output of the encoder 610. There is a difference in points.

More specifically, in the spectrum predictor 600, the encoder 610 encodes input text and converts it into a 512-dimensional feature (hereinafter referred to as 'feature information'). The encoder 610 includes a character embedding generation unit 613 into which text is input, three convolution layers (3 conv layers, 615) connected to the character embedding generation unit 613, and three convolution It may have a bidirectional long short term memory (LSTM) 617 coupled to the layers 615 . The three convolutional layers correspond to three convolutional neural network layers and may be referred to as 'encoder convolutional layers'.

In addition, the spectrum predictor 600 may further include a preprocessor 611 for preprocessing input text at an input terminal of the encoder 610 . The pre-processor 611 may be configured to divide the input text into syllable units and express the separated syllables as integers through one-hot encoding. Meanwhile, the pre-processing unit 611 may be omitted when the input text is separately processed by an independent pre-processing unit or an independent component performing a function corresponding thereto. However, this pre-processing unit 611 is not included in the spectrum predictor 600 but may be included in the TTS model.

In the encoder 610, the character embedding generation unit 613 converts the integer sequence received from the preprocessor 611 into a matrix form. Encoder convolutional layers 615 reduce information in matrix form. And, the bidirectional LSTM 617 converts the reduced matrix-type information into encoder feature information. Here, the encoder feature information may include a context vector compressed to one fixed size.

Attention 620 encodes attention alignment information (AAI) according to the time-step of decoder 630 so that speech pronunciation to be output through MelGAN 700a proceeds in the sequential order of the input text. Add to feature information. That is, the attention 620 may refer to a process of extracting, from the encoder 610, alignment or mapping information to be used by the decoder 630 at each point in time. More specifically, the attention 620 extracts and allocates information to be used by the decoder 630 at a predetermined point in time from the encoder 610, and the information used at each point is configured to use the attention alignment information of the previous point in time. It can be. According to this configuration, the alignment graph of the attention 620 can continue in an upward direction while the TTS model is being trained. Such attention 620 may be implemented using location sensitive attention.

For example, location-sensitive attention is calculated by applying an arctangent (tanh) to a weighted sum of attention and encoder information at a previous point in time and then multiplying by a weight in the calculation of an attention score. Here, attention is normalized in the range of 0 to 1 by applying softmax to the attention score, and corresponds to information for determining how much encoder information to allocate. The attention extraction result is referred to as a context, and corresponds to the sum of products of attention and encoder up to that point in time.

The decoder 630 generates a mel spectrogram of the next time-step by using an alignment feature, which is attention information, and a mel spectrogram generated in the previous time-step through the attention 620 . The generated mel spectrogram includes timbre and prosody information by integrated style embedding. The previous time-step may correspond to the previous time-step, and the next time-step may correspond to the next time-step or the current time point, respectively.

The decoder 630 includes a pre-Net (631), two decoder LSTMs (632), a first fully connected layer (633), a second fully connected layer (634) and a post-Net , 635). The free net 631 may be a two-layer pre-Net, each decoder LSTM 632 may be a unidirectional long short term memory (LSTM), and the fully connected layer may be a linear projection ), and the postnet 635 may be a 5 convolution layer post-Net composed of 5 convolution layers.

In the decoder 630, the freenet 631 extracts information from the mel vector of the previous point in time. The decoder LSTM (632) extracts information of the current point in time using the operation results of the attention (620) and the freenet (631). The first fully connected layer 633 generates a Mel spectrogram using information of a current view. The second fully connected layer 634 is combined with sigmoid to calculate the end probability of the current time based on the information output from the decoder LSTM 632 and outputs a stop token. The termination probability may be calculated in the range of 0 to 1. In addition, the postnet 635 improves the quality of the mel spectrogram through 5 one-dimensional (1D) convolutional layers and batch normalization. The postnet 635 may be applied after all of the mel spectrograms are generated through the decoder 630, and the quality of the generated mel spectrograms may be corrected in a smooth manner using the remaining connections.

In the aforementioned spectrum predictor 600, the total loss of the loss function may be calculated as the sum of mean squared error (MSE) and binary cross entropy (BCE). MSE represents the difference between the original mel spectrogram and the estimated mel spectrogram, and BCE represents the difference between the actual termination probability and the estimated termination probability. The original mel spectrogram may correspond to the target mel spectrogram.

MelGAN 700a receives a Mel spectrogram and generates a waveform signal or audio signal. Melgen 700a is a GAN-based vocoder composed of a generator and a discriminator, and is a one-dimensional convolution (conv1d) model composed of several layers.

The constructor of Melgen 700a consists of several layers of conv1d and a residual stack. Inside the rest stack is another conv1d, configured to go through the rest connections 3 times. The constructor takes as input the mel spectrogram in batches. Therefore, the size of the input tensor can be [batch size, 80, frame length]. Here, 80 becomes the degree of the Mel spectrogram, and the frame length is a variable that the user determines how many intervals to input. The output is an audio signal and its size is [batch size, 1, frame length × hop size).

The discriminator of Melgen 700a consists of three multi-scale structures. One multi-scale has a total of 7 outputs with 6 feature maps and 1 output. A discriminator block may consist of a total of 7 conv1d.

The last output of Melgen 700a is basically a discriminator output, and the remaining outputs can be used as a loss.

5 is an overall configuration diagram of a voice synthesis system according to another embodiment of the present invention. FIG. 6 is a block diagram for explaining an emotion module that can be employed in the voice synthesis system of FIG. 5 . And Figure 7 is a detailed configuration diagram of a reference encoder that can be employed in the emotion module of Figure 6

Referring to FIG. 5, the voice synthesis system includes a prosody module 100, a timbre module 200, an emotion module 300, a spectrum predictor 600, and a MelGAN 700a, and a prosody module ( 100) of the first style embedding, the voice module 200 of the second style embedding, and the emotion module 300 of the third style embedding of the combined style embedding is applied to a specific prosody of the input text , waveform samples to which timbre and emotion are added can be output. The spectrum predictor 600 and MelGAN 700a constitute a text to speech (TTS) model 500.

Since the prosody module 100, the tone module 200, the spectrum predictor 600, and the MelGAN 700a are substantially the same as the corresponding components of the voice synthesis system described above with reference to FIG. 4, detailed descriptions thereof are given. will be omitted.

The emotion module 300 includes a STFT unit 310, a third reference encoder 320, and a fifth style token layer (style token 5, 350). The STFT unit 310 may be omitted when a spectrogram or a Mel spectrogram separately processed from reference audio is prepared as input data of the third reference encoder 320 . The fifth style token layer 350 corresponds to the fifth GST layer.

In the emotion module 300, the STFT unit 310 may provide a spectrogram or a Mel spectrogram obtained by transforming the reference audio by Fourier transform to the third reference encoder 320 as an input. The third reference encoder 320 receives a spectrogram or a MEL spectrogram that is not a fundamental frequency and generates an emotional embedding.

The emotion embedding that has passed through the fifth GST layer 350 is output as a third style embedding. That is, the fifth style token layer 350 may generate a third style embedding by learning emotion embedding. This fifth style token layer 350 includes multi-head attention 352 and a plurality of tokens 354 . The plurality of tokens 354 may include a first token (token 1) to a k-th token (token k), and each token may be referred to as a style token. In order to effectively learn emotion embedding, k is preferably 7. That is, the fifth GST layer 350 of this embodiment may include seven style tokens to correspond to seven classified emotional styles.

The seven emotions included in the emotion style are Anger, Disappointment, Fear, Surprise, Sad, Neutral, and Happy, which are MOS (mean When summarized together with the opinion score, it is shown in Table 1 below.

As shown in FIG. 6 , the above-described fifth style token layer 350 is similar to the emotion embedding output from the third reference encoder 320 by the multi-head attention 352 and each of the seven style tokens. When is measured (see 360), a weighted sum of these style tokens 244 can generate a style embedding (a third style embedding).

The third style embedding generated by the emotion module 300 is concatenated with the first style embedding passed through a total of three style token layers in the tone module 100 and the second style embedding of the tone module 200, and voice It may be combined with the output of the encoder (encoder, 610) of the spectrum predictor (600) of the synthesis system and input to the attention (620) of the spectrum predictor (600).

Meanwhile, referring to FIG. 7, the aforementioned third reference encoder 320 will be described in more detail. In order to convert an input reference spectrogram and output an emotional embedding, the third reference encoder 320 includes three convolutional neural networks (3 CNNs) 322, two residual blocks (324) connected to the three convolutional neural networks 322, and two residuals. It may consist of a modified AlexNet 326 connected to blocks 324 . The reference spectrogram refers to a spectrogram obtained by transforming an input of reference audio through Fourier transform. The reference spectrogram may be replaced with the reference mel spectrogram.

The TTS model coupled to the emotion module including the third reference encoder 320 is configured to adjust the style transfer and the strength of the style transfer in order to apply the emotion expression of the reference audio to speech synthesis, and the reconstruction loss loss) alone.

8 is a schematic configuration diagram of a voice synthesis system according to another embodiment of the present invention, and FIG. 9 is a block diagram for explaining main operations applicable to the voice synthesis system of FIG. 8 .

Referring to FIG. 8 , a voice synthesis system 1000 may include a processor 1100 and a memory 1200. In addition, the speech synthesis system 100 may be configured to further include a transceiver 1300, a storage device 1400, or an input interface device 1500 and an output interface device 1500. can

In the speech synthesis system 1000, the processor 1100 may be a central processing unit (CPU), a graphics processing unit (GPU), or a dedicated processor in which methods according to embodiments of the present invention are performed. can mean a processor.

The memory 1200 or the storage device 1400 may store at least one command executed by the processor 1100 . The at least one command includes a first command to implement or operate a tone module, a second command to implement or operate a tone module, a third command to implement or operate an emotion module, and a TTS model to implement or operate. A fifth command may be included.

In addition, at least one command includes a command to extract a fundamental frequency from the input reference audio, a command to transfer the fundamental frequency to the input of the reference encoder, a command to encode the fundamental frequency to generate a prosody embedding, and a first style from the prosody embedding. Instructions for generating an embedding, instructions for transforming the reference audio into a reference mel spectrogram by Fourier transform, instructions for generating a speaker embedding by encoding the reference mel spectrogram, instructions for generating a second style embedding from the speaker embedding, first A command to input the combined style embedding, which is a combination of style embedding and second style embedding, to the attention by combining the output of the encoder of the text to speech (TTS) model, a command to create a mel spectrogram through the attention of the TTS model and the decoder, It may include a command for generating voice synthesized audio in which tones and prosody are combined by integrated style embedding from input text through a vocoder of the TTS model.

Each of the above-described memory 1200 and storage device 1400 may include at least one of a volatile storage medium and a non-volatile storage medium. For example, the memory 1200 or the storage device 1400 may include at least one of a read only memory (ROM) and a random access memory (RAM).

The transceiver 1300 may include at least one sub-communication system supporting communication with an external device through a wireless network, a wired network, a satellite network, or a combination thereof.

The input interface device 1500 includes at least one input unit selected from input units such as a keyboard, a microphone, a touch pad, and a touch screen, and an input signal processor that maps or processes a signal input through the at least one input unit with a pre-stored command. can include

The output interface device 1600 includes an output signal processing unit that maps or processes a signal output under the control of the processor 1100 into a pre-stored signal type or level, and a signal in the form of vibration or light according to the signal of the output signal processing unit. or at least one output means for outputting information. The output signal processing unit may generate a network topology or voltage state estimation result of the wiring system in the form of an image, audio, or a combination thereof. The at least one output means may include a speaker, a display device, a printer, an optical output device, a vibration output device, and the like.

The above-described voice synthesis system 1000 is, for example, a desktop computer, a laptop computer, a notebook, a smart phone, a tablet PC, a mobile phone ( mobile phone), smart watch, smart glass, e-book reader, PMP (portable multimedia player), portable game device, navigation device, digital camera, DMB (digital multimedia) broadcasting) player, digital audio recorder, digital audio player, digital video recorder, digital video player, personal digital assistant (PDA), network server, It may be integrally combined with or mounted on a web server, mail server, service server for a specific service, and the like.

Meanwhile, as shown in FIG. 9 , the speech synthesis system 1000 may further include a first module 1700 for learning management and a second module 180 for a style replication TTS service.

The first module 1700 uses a first audio waveform having a desired tone and rhyme or a second audio waveform having a desired tone, rhyme, and emotion as a reference audio of ground truth (GT), thereby forming a hierarchy of prosody modules It may be configured to train the type GST and the fourth GST layer of the voice module, or further train the fifth GST layer of the emotion module.

In addition, the speech synthesis system 1000 provides a speech synthesis (TTS) service in which the timbre and prosody style of the reference audio are duplicated by the second module 1800, or the timbre, prosody and emotion style of the reference audio are reproduced. It may be configured to provide a voice synthesis service.

The first module 1700 and/or the second module 1800 may be implemented as program commands or software modules, stored in the memory 1200 or the storage device 1400, and executed by the processor 1100. there is. Of course, depending on the implementation, at least a part of the first module 1700 and/or the second module 1800 may be configured to include a hardware configuration for at least a part of the function.

As described above, the speech synthesis system 100 is configured to include a prosody module and a timbre module together with the TTS model, or optionally further include an emotion module, each of which is a desired timbre and prosody based on the global style token. There is an advantage in that voice synthesis can be implemented according to the style of the reference audio if there is only the reference audio including the style of the reference audio, the emotion, or a combination thereof.

The operation of the method according to the embodiment of the present invention can be implemented as a computer readable program or code on a computer readable recording medium. A computer-readable recording medium includes all types of recording devices in which data that can be read by a computer system is stored. In addition, computer-readable recording media may be distributed to computer systems connected through a network to store and execute computer-readable programs or codes in a distributed manner.

In addition, the computer-readable recording medium may include hardware devices specially configured to store and execute program instructions, such as ROM, RAM, and flash memory. The program command may include high-level language codes that can be executed by a computer using an interpreter or the like as well as machine code generated by a compiler.

Although some aspects of the present invention have been described in the context of an apparatus, it may also represent a description according to a corresponding method, where a block or apparatus corresponds to a method step or feature of a method step. Similarly, aspects described in the context of a method may also be represented by a corresponding block or item or a corresponding feature of a device. Some or all of the method steps may be performed by (or using) a hardware device such as, for example, a microprocessor, programmable computer, or electronic circuitry. In some embodiments, one or more of the most important method steps may be performed by such an apparatus.

In embodiments, a programmable logic device, such as a field programmable gate array, may be used to perform some or all of the functions of the methods described herein. In embodiments, a field programmable gate array may operate in conjunction with a microprocessor to perform one of the methods described herein. Generally, methods are preferably performed by some hardware device.

Although the above has been described with reference to preferred embodiments of the present invention, those skilled in the art can variously modify and change the present invention without departing from the spirit and scope of the present invention described in the claims below. You will understand that you can.

Claims (20)

As a voice synthesis method capable of replicating real-time tone and prosody style,
extracting a fundamental frequency from input reference audio and transferring the fundamental frequency to an input of a reference encoder;
generating, by the reference encoder, a prosody embedding by encoding the fundamental frequency;
generating a first style embedding from the prosody embedding;
converting the reference audio into a reference mel spectrogram by Fourier transform;
encoding the reference mel spectrogram to generate a speaker embedding;
generating a second style embedding from the speaker embedding;
inputting an integrated style embedding obtained by combining the first style embedding and the second style embedding into attention of a text to speech (TTS) model; and
and generating audio in which a tone and a prosody are synthesized by the integrated style embedding with respect to the input text by the TTS model. The method of claim 1,
In the step of extracting the fundamental frequency, the reference audio is cut in units of a sliding window of a certain period, and a pitch contour is calculated through frequency decomposition of the reference audio using a normalized cross-correlation function, where pitch is A voice synthesis method corresponding to the pitch of a sound. The method of claim 1,
The generating of the first style embedding may include using a hierarchical GST of three layers including a first global style token (GST) layer to a third global style token layer to generate the reference audio. Speech synthesis method for generating the first style embedding for prosody. The method of claim 3,
Each of the first to third GST layers includes a multi-head attention that transfers an input embedding to a plurality of tokens of each GST layer and a plurality of tokens connected to the multi-head attention, and the input embedding Measure the similarity between each token and generate a style embedding as a weighted sum of each token,
The first to third GST layers include a residual connection connecting the plurality of tokens of a current GST layer and the plurality of tokens of a previous GST layer. The method of claim 3,
In the generating of the second style embedding, the second style embedding for the timbre of the reference audio is generated by using a fourth GST layer. The method of claim 1,
To generate the audio,
extracting, by the encoder of the TTS model, feature information from text input to the encoder;
mapping the feature information to attention alignment information to be used in the decoder of the TTS model at each time point by the attention, and transmitting the mapped attention alignment information and the unified style embedding to the decoder; and
and generating, by the decoder, a mel spectrogram of a current view synthesized with the integrated style embedding using attention information of the attention and a mel spectrogram of a previous view. The method of claim 6,
The generating of the audio further comprises generating the audio from the current Mel spectrogram by Mel generative adversarial networks (MelGAN). The method of claim 1,
The target mel spectrogram corresponding to the reference mel spectrogram is compared with the predicted mel spectrograms for the input text and the reference audio generated by the TTS model, and the target mel spectrogram and the predicted mel spectrogram are matched. obtaining a difference or a loss corresponding to the difference; and
Further comprising the step of training the TTS model until the difference or loss is less than or equal to a preset level or a reference value. As a voice synthesis system capable of replicating real-time tone and prosody style,
an extraction unit for extracting a fundamental frequency from input reference audio;
a first reference encoder that receives the fundamental frequency and encodes the fundamental frequency to generate a prosody embedding;
hierarchical global style token layers for generating a first style embedding for a prosody of the reference audio from the prosody embedding;
a transform unit for transforming the reference audio into a reference mel spectrogram by Fourier transform;
a second reference encoder for encoding the reference mel spectrogram to generate a speaker embedding;
a single global style token layer for generating a second style embedding for a timbre of the reference audio from the speaker embedding; and
Speech synthesis (text to speech, TTS) that receives the unified style embedding, which is the combination of the first style embedding and the second style embedding, as attention, and generates audio synthesized with tones and prosody by the unified style embedding for the input text. ) speech synthesis system containing the model. The method of claim 9,
The extractor cuts the reference audio in units of a sliding window of a certain section, divides valid frames using a normalized cross-correlation function, and calculates a pitch contour through frequency decomposition of the reference audio within the valid frame. wherein the pitch corresponds to the pitch of the reference audio. The method of claim 9,
The hierarchical global style token layers are composed of three layers of hierarchical GST consisting of a first global style token (GST) layer to a third global style token layer. The method of claim 11,
Each GST layer of the hierarchical global style token layers includes a multi-head attention and a plurality of tokens connected to the multi-head attention;
The first GST layer measures a similarity between the rhyme embedding and each token and generates a style embedding as a weighted sum of the plurality of tokens;
The speech synthesis system of claim 1 , wherein the style embedding generated in the first GST layer sequentially passes through the second GST layer and the third GST layer and is output as a first style embedding. The method of claim 11,
In the hierarchical global style token layers, a first pair of the first GST layer and the second GST layer and a second pair of the second GST layer and the third GST layer are tokens of a current layer and a previous layer. A speech synthesis system having a residual connection connected with tokens of . The method of claim 9,
The target mel spectrogram corresponding to the reference mel spectrogram is compared with the predicted mel spectrograms for the input text and the reference audio generated by the TTS model, and the target mel spectrogram and the predicted mel spectrogram are Further comprising a learning management unit for obtaining a difference in or a loss corresponding to the difference,
The learning management unit trains the TTS model until the difference or loss is less than or equal to a preset level or a reference value. The method of claim 9,
The TTS model includes a spectrogram predictor and a vocoder,
The spectrogram predictor generates a mel spectrogram based on the input text and the integrated style embedding;
Wherein the vocoder generates a waveform sample corresponding to a synthesized waveform from the mel spectrogram. The method of claim 15
The spectrogram predictor includes an encoder, an attention and a decoder,
The encoder extracts feature information from the input text,
The Attention maps the feature information to attention alignment information to be used by the decoder at each point in time and transfers the mapped attention alignment information and the unified style embedding to the decoder;
The speech synthesis system of claim 1 , wherein the decoder generates a mel spectrogram of a current view synthesized with the integrated style embedding using the attention information of the attention and a mel spectrogram of a previous view. The method of claim 16
The spectrogram predictor further includes a preprocessor located at an input terminal of the encoder, wherein the preprocessor divides the input text into syllable units and expresses the separated syllables as integers through one-hot encoding, voice synthesis system. The method of claim 17
The encoder includes a character embedding generating unit, 3 convolution layers connected to the character embedding generating unit, and a bidirectional long short term memory (LSTM) connected to the 3 convolution layers. ) is provided,
The character embedding generation unit converts the integer sequence received from the pre-processing unit into a matrix form;
The 3 convolutional layers reduce information in matrix form,
The bidirectional LSTM converts information in a reduced matrix form into encoder characteristic information, wherein the encoder characteristic information includes a context vector compressed to one fixed size. The method of claim 16
The Attention synthesizes speech by adding the Attention Alignment information to the encoder characteristic information according to the time-step of the decoder so that the speech pronunciation to be output through the vocoder of the TTS model proceeds in the sequential order of the input text. system. The method of claim 15
The vocoder is a Mel generative adversarial networks (MelGAN) speech synthesis system.

Images