KR102272554B1 - Method and system of text to multiple speech - Google Patents

Abstract

The present invention relates to a text-multi-speech conversion method and system, and more particularly, generating a character embedding vector for a sentence to be voiced on a character-by-character basis, generating a voice spectrum from the embedding vector, and the voice spectrum to a text-to-speech conversion method and system, comprising the step of outputting a voice spectrum, wherein the voice spectrum generating step receives multiple voice data, converts it into a voice embedding vector, and uses it for voice spectrum merging.

Description

Text-to-speech conversion method and system {Method and system of text to multiple speech}

The present invention relates to a text-to-multiple speech conversion method and system.

Speech is the most natural means of communication for humans, a means of information transfer, and a meaningful sound made by humans as a means of realizing language.

Attempts to realize communication between humans and machines through voice have been steadily developed since the past. Moreover, recently, speech information technology (SIT) for effectively processing voice information has made remarkable progress. As such, it is being applied one after another in real life.

When such speech information processing technologies are broadly classified, they may be classified into categories such as speech recognition, speech synthesis, speaker identification and verification, and speech coding.

Speech recognition is a technology that recognizes the spoken voice and converts it into a character string. Speech synthesis is a technology that converts a character string into an original voice using data or parameters obtained from voice analysis. It is a technique for estimating or authenticating, and voice coding is a technique for effectively compressing and encoding a voice signal.

Among them, looking at the development process of speech synthesis technology, most of the early speech synthesis adopts a structure that mimics a human vocal organ using a mechanical device or an electronic circuit. For example, in the 18th century, Wolfgang von Kem pelen devised a bellows speech synthesis machine that had a rubber mouth and nostrils and could mimic changes in the vocal tract. After that, it developed into a speech synthesis technology using an electrical analysis method, and in the 1930s, Dudley introduced an early form of a vocoder.

Today, thanks to the rapid development of computers, computer-based speech synthesis has become the mainstream of speech synthesis, and system model methods (articulary synthesis, etc.) or signal model methods (rule-based formant synthesis or unit A variety of methods such as negative bond synthesis) are being developed.

Speech synthesis technology can be roughly divided into two types according to the actual application method. A limited lexical synthesis or Automatic Response System (ARS) that synthesizes only sentences with a limited number of words and a syntactic structure, and an unlimited lexical synthesis or text-to-speech (TTS; Text-to) system that receives arbitrary sentences and synthesizes speech -Speech) system.

Among them, a text-to-speech (TTS) system generates a voice for an arbitrary sentence using a small synthesized unit voice and language processing. By using language processing, the input sentence is matched with a combination of appropriate compound units, and the appropriate intonation and duration are extracted from the sentence to determine the prosody of the synthesized sound. Since speech is synthesized by the combination of phonemes, syllables, etc., which are the basic units of language, there is no limitation on the target vocabulary, and it is mainly applied to TTS (Text-to-Speech) devices and CTS (Context-to-Speech) devices.

Conventional speech synthesis technology has a problem in that it is inefficient because it needs to collect a large amount of data to generate a new human voice, and it takes time to learn a model on the data.

The object of the present invention is

In implementing a plurality of voices in the existing text-to-speech conversion method and system, a text-to-multi-speech conversion method and system for reducing the time consumed for voice implementation and shortening the pre-data collection step for synthesizing a plurality of voices is to provide

In order to achieve the above object, a text-multi-speech conversion method according to an embodiment of the present invention comprises the steps of: generating a character embedding vector for a sentence to be voiced in character units; generating a voice spectrum from the embedding vector; and outputting the voice spectrum as voice, wherein the generating of the voice spectrum may receive multiple voice data and convert it into a voice embedding vector to be used for voice spectrum merging.

According to an embodiment of the present invention, the multi-voice data may be a speaker-id in the form of a one-hot vector.

According to an embodiment of the present invention, the multi-voice data may be log-mel-spectrogram data.

According to an embodiment of the present invention, the log-mel-spectrogram may be converted into a speech embedding vector through an embedder network.

According to an embodiment of the present invention, the speech embedding vector may be input to the decoder RNN and the lumped RNN of the spectrum merging unit.

In order to achieve the above object, a text-to-multi-speech conversion system according to an embodiment of the present invention includes a text-to-multi-speech conversion module for outputting the voice of the text from the text, and the text-to-speech conversion module converts the voice An encoder for generating a text embedding vector in a character-by-character unit of a sentence to be generated, a decoder for generating a speech spectrum from the embedding vector, and a speech output unit for outputting the speech spectrum as speech, wherein the decoder receives multiple speech data and provides speech It can be converted into an embedding vector and used for merging speech spectrum.

According to an embodiment of the present invention, the multi-voice data may be a speaker-id in the form of a one-hot vector.

According to an embodiment of the present invention, the multi-voice data may be log-mel-spectrogram data.

According to an embodiment of the present invention, the log-mel-spectrogram may be converted into a speech embedding vector through an embedder network.

According to an embodiment of the present invention, the speech embedding vector is

It may be input to the decoder RNN and the lumped RNN of the spectrum merging unit.

In the text-to-speech conversion method of the present invention, a new voice can be implemented with only a small amount of data, and text-to-speech conversion is possible without separately learning a model,

By receiving IDs corresponding to a plurality of voices together with text information, it is possible to shorten the time taken for voice conversion per multi-party conversation.

In addition, it is a method of synthesizing voices by multiple speakers with only a small amount of data using log-mel-spectogram data for multiple voices, which can reduce model training time and reduce the time and cost consumed in pre-data preparation. There is an effect that can make it happen.

1 is a schematic diagram illustrating a general text-to-speech conversion method.
2 is a schematic diagram illustrating a Gated Recurrent Unit (GRU) according to an embodiment of the present invention.
3 is a flowchart of a text-multiple speech conversion method according to an embodiment of the present invention.
4 is a block diagram of an RNN encoder-decoder network according to an embodiment of the present invention.
5 is a block diagram of a content-based centralized mechanism RNN encoder-decoder network according to an embodiment of the present invention.
6 is a schematic diagram of tacotron.
Fig. 7 is a graph showing an ideal graph of concentrated sorting and a problem of actual centralized sorting.
8 is a schematic diagram of a CBHG module;
9 is a graph illustrating parameters of tacotron according to an embodiment of the present invention.
10 is a schematic diagram of a text-multi-speech conversion tacotron according to an embodiment of the present invention.
11 is a schematic diagram of a speech embedding prediction network according to an embodiment of the present invention.
12 is a table showing a voice profile in a test set according to an embodiment of the present invention.
13 is a parameter table of a multi-voice tacotron according to an embodiment of the present invention.
14 is an example of a generated voice spectogram according to an embodiment of the present invention.
15 is a graph illustrating a speech embedding result according to a theoretical element according to an embodiment of the present invention.
16 is a graph illustrating a spectogram according to a change in a theoretical element according to an embodiment of the present invention.
17 is a graph illustrating a spectogram by mixed voice according to an embodiment of the present invention.

A text-to-speech conversion method according to an embodiment of the present invention is a method of receiving a text input and synthesizing and outputting the spoken voice of the corresponding content.

That is, the text-to-speech conversion method according to an embodiment of the present invention is a method of outputting a read voice signal when a character of content to be uttered is input.

Hereinafter, a text-to-speech conversion method according to an embodiment of the present invention will be described in detail for each step with reference to the drawings.

A text-to-speech conversion method according to an embodiment of the present invention includes generating a speech spectrum corresponding to each partial band from a text.

The text-to-speech conversion method according to an embodiment of the present invention divides characters by band and generates a voice spectrum for each signal divided by band, so that a single voice spectrum is generated for an input character without distinction by band. Compared to the conventional method, the amount of calculation can be significantly reduced, and there is an advantage in that difficult problems can be solved in a divide and conquer method.

According to an embodiment of the present invention, the step of generating the voice spectrum corresponding to each subband from the text may generate the voice spectrum in parallel for each subband at the same time.

In the generating of the voice spectrum, signals for each band may be calculated using different methods or systems, and methods or systems for calculating signals for each band may be independent of each other. Accordingly, in order to generate a voice spectrum corresponding to each signal for each band in the above step, a plurality of text-to-speech systems may be used, and as the text-to-speech system, a Tacotron or Wavenet algorithm this can be used

Here, the tacotron generates a character as a voice spectrum by the following method.

Tacotron is a 'sequence-to-sequence' model using a Recurrent Neural Network (RNN) encoder-decoder. It can be divided into a decoder unit for synthesizing.

In the encoder unit, as an input value of the encoder network, character embedding made in a vector form by decomposing a sentence into character units is used, and a text encoding vector is used through a neural network. vector) as output.

As the neural network, a CBHG module, that is, a neural network in which a convolutional neural network, a highway network, and a bi-directional recurrent neural network are sequentially stacked may be used.

In the decoder unit, the weighted sum of text encoding vectors and the last decoder output value of the previous t-1 time step are combined and used as an input value of the decoder network at the t time step. The output value of the decoder is a mel-scale spectrogram, and r vectors are given out for every starting step. Only the last vector among r vectors is used as the decoder input value for the next time step. Mel-scale spectrogram vectors generated by r for each time step are combined in the decoder time step direction to form a melt-scale spectrogram of the synthesized speech. This spectrogram is converted into a linear-scale spectrogram through an additional neural network. is converted Thereafter, the linear-scale spectrogram is converted into a wave form through a 'Griffin-Lim reconstruction' algorithm, and when this is written as a '~.wav' file, an audio file is generated.

The step of generating a voice spectrum corresponding to each partial band from a character according to an embodiment of the present invention may be implemented using the tacotron.

That is, the output values of the decoder in the tacotron are r spectrograms. If you think of this in the form of a matrix, you can see it as a matrix of the S x R form. Here, s is the size of the spectrogram, and an 80-dimensional vector may be used for the mel-scale spectrogram.

The speech spectrum is converted into a linear spectrum, and the linear spectrum is converted into a waveform, and finally output as audio.

Meanwhile, the generating of the voice spectrum may include generating the voice spectrum by giving a utterance condition of at least one of tone, age, gender, and emotion of a speaker.

In the above step, when the utterance condition is added, a voice signal reflecting the condition may be generated.

Outputting the merged spectrum as a voice may include converting the voice spectrum into a linear spectrum and converting the linear spectrum into a waveform.

Also, the present invention

A text-to-speech conversion module for generating a signal for each partial band from a character and outputting a voice;

The text-to-speech conversion module,

A text-to-speech conversion unit that generates a speech spectrum corresponding to each partial band from the text;

a spectrum merging unit for merging voice spectra corresponding to each band generated by the text-to-speech conversion unit; and

It provides a text-to-speech conversion system including a voice output unit for outputting the spectrum merged by the spectrum merging unit as a voice.

A text-to-speech conversion system according to an embodiment of the present invention is a system for receiving a text input and synthesizing the spoken voice of the corresponding content.

That is, the text-to-speech conversion system according to an embodiment of the present invention is a system in which, when a character of content to be uttered is input, a voice signal reading it is output.

A text-to-speech conversion system according to an embodiment of the present invention synthesizes speech using an artificial neural network, and converts text into speech by dividing and creating one or more bands for a speech signal to be synthesized in speech synthesis and combining them. It is a text-to-speech conversion system.

A text-to-speech conversion system according to an embodiment of the present invention includes a text-to-speech conversion module for outputting voice by calculating the signals for each band generated by the band dividing module.

According to an embodiment of the present invention, in the text-to-speech conversion module, the partial bands may include subbands having a length shorter than the length of the entire band.

According to an embodiment of the present invention, the length of the audio waveform of the partial band may be less than or equal to 1/2 of the length of the audio waveform of the entire band.

According to an embodiment of the present invention, when a wavelet transform having four sub-bands is applied when the length of the original voice waveform is 100, each of the four subbands having a length of {50, 25, 12.5, 12.5} can be divided into signals.

In this case, the text-to-speech conversion module includes a text-to-speech conversion unit that generates a speech spectrum corresponding to each partial band from the text.

According to an embodiment of the present invention, the step of generating the voice spectrum corresponding to each subband from the text may generate the voice spectrum in parallel for each subband at the same time.

The text-to-speech converter generates a corresponding voice spectrum for each band.

In this case, each band may generate a voice spectrum using a different method or system, and thus, the method or system may be independent of each other. Accordingly, the text-to-speech conversion unit may use a plurality of text-to-speech conversion systems to generate a speech spectrum corresponding to a signal for each band, and as the text-to-speech conversion system, Tacotron or Wavenet ( Wavenet) algorithm may be used.

In addition, the text-to-speech conversion module may include a spectrum merging unit for merging voice spectra corresponding to each band generated by the text-to-speech converting unit, and outputting the merged spectrum from the spectrum merging unit as voice. It may further include a voice output unit.

In this case, the voice output unit may convert the merged voice spectrum into a linear spectrum and convert the linear spectrum into a waveform to output a final voice.

1 is a schematic diagram illustrating a voice user interface.

Referring to FIG. 1 , the configuration of an existing voice user interface can be confirmed. Speech is one of the most basic and effective communication tools for humans. In everyday life, people exchange information by using words such as conversations and presentations. Because voice-based communication is intuitive and convenient for users, some devices use voice user interfaces to interact with voice using voice. A simple way to implement a voice response in a voice user interface is audio recording. However, recording has a limitation in that only the recorded voice can be spoken. The flexibility of using the device is reduced because it cannot respond to situations where the device is not ready. For example, artificial intelligence (AI) agents such as Apple Siri and Amazon Alexa require the use of different sentences as a user's query can be arbitrary. Having to log all possible responses from these applications can be a burden on companies in terms of time and money. Therefore, many researchers have tried to create a natural and fast speech synthesis model. In this sense, speech synthesis, also known as text-to-speech (TTS), is an important topic as it can generate arbitrary speech.

Speech synthesis technology can provide more value to AI agents. These days, in particular, the demand for interactive agents and AI assistants is increasing with the advancement of machine learning technology. As a series of modules, as depicted in Fig. 1, we formulated the framework of the pictorial model. For these human-like agents, the response part of the front-end system as well as the back-end algorithm of the dialog system is important. People get the first impression of an agent through their looks and voice. Agents can benefit from improving the naturalness of speech synthesis models. In this regard, if each of the different AI agents could speak arbitrary sentences in their own voice, they would look more natural and user-friendly.

2 is a schematic diagram illustrating a Gated Recurrent Unit (GRU) according to an embodiment of the present invention.

Referring to Figure 2, mapping a sequence to another (not necessarily the same type of sequence) is a widespread problem. For example, a speech recognition task should map a series of waveform samples to a series of characters, and a speech synthesis task should perform the reverse mapping of speech recognition. Machine translation is also a mapping from text to another text in another language. Sequences - To solve a sequence problem, you need to create one sequence from another and know how to align the elements of both sequences. Long-term dependencies must be considered to include contextual information during mapping between sequences. One of the best choices when we implement this kind of network is a recurrent neural network (RNN). A recurrent neural network can receive sequential input and accumulate contextual information in a hidden state. In particular, we showed that LSTMs and GRUs can efficiently maintain relevant context information using their own gating mechanisms (GRU, GRUEMP, LSTM). Specifically, a GRU with a reset gate r and an update gate z can be described by the following Equation 1.

[Equation 1]

를 적용함으로써 (0,1) 사이의 값을 갖도록 강제할 수 있다. 도 2는 GRU의 아키텍처를 보여준다. 본 발명의 일 실시 예에서는 GRU를 RNN의 기본 단위로 사용할 수 있다.where x, h, t and j are input, hidden state, time step index and hidden dimension index, respectively. Sigmoid function as nonlinearity for gating variable

You can force it to have a value between (0,1) by applying . 2 shows the architecture of the GRU. In an embodiment of the present invention, the GRU may be used as a basic unit of the RNN.

3 is a flowchart of a text-multiple speech conversion method according to an embodiment of the present invention.

Referring to FIG. 3 , in the text-multi-voice conversion method according to an embodiment of the present invention, a step ( S110 ) of distinguishing an entire frequency band for generating a voice into a plurality of sub-bands, a text corresponding to each sub-band from the text It may include generating a voice spectrum (S120), merging voice spectra corresponding to the respective bands (S130), and outputting the merged spectrum as a voice (S140).

In step S130, multiple voice data may be received and converted into a voice embedding vector to be used for voice spectrum merging.

The multi-voice data may be a speaker-id in the form of a one-hot vector.

The multi-voice data may be log-mel-spectrogram data.

The log-mel-spectrogram may be converted into a speech embedding vector through an embedder network.

The speech embedding vector may be input to a decoder RNN and a lumped RNN of the spectrum merging unit.

4 is a block diagram of an RNN encoder-decoder network according to an embodiment of the present invention.

Referring to FIG. 4 , an inter-sequence network may be formed using an RNN function capable of modeling long-term dependencies. In FIG. 2 , the last hidden state h _T of the encoder RNN may include summary information of the input sequence. The decoder RNN can then take this context vector and produce an output. The generated output can then be used as input along with the _{context vector h T in the next time step.} A disadvantage of this architecture is that the context vector h _T is fixed at every time step of the decoder RNN. It doesn't matter if the context vector doesn't corrupt the entire information in the input sequence. However, in many cases, if the sequence is longer than its capacity, the fixed capacity of the context vector cannot contain the entire information of the input sequence. This problem can be solved by the mechanism of interest.

A conventional convergence mechanism, called a soft window, multiplies a portion of an input sequence by a intensive weight. A mixture of K Gaussian functions can be used to compute the concentrated weights. The fixed number K is the number of mixtures. Since we assumed a Gaussian function to compute the concentrated weights, we could not have any flexibility in the shape of the concentrated weights.

5 is a block diagram of a content-based centralized mechanism RNN encoder-decoder network according to an embodiment of the present invention.

는 신경망에 의해 모델링 된 정렬 모델에 의해 계산될 수 있다. 길이가 T_enc인 입력 시퀀스를 취하는 콘텐츠 기반주의 메커니즘을 갖는 RNN 인코더 - 디코더 네트워크는 다음 수학식 2와 같이 정의 될 수있다 :Referring to FIG. 5 , a content-based concentration mechanism can be identified. The context vector of the network may be calculated as the weighted sum of the hidden states of the encoder. Since the context vector can be flexibly changed at each decoder's time step, the hidden state of the encoder does not need to remember all previous information. concentration weight vector

can be calculated by an alignment model modeled by a neural network. RNN length encoder has a content based on a mechanism that takes care of the input sequence T _enc - decoder network may be defined as the following equation (2):

[Equation 2]

Here, c is a context vector, and d and h are the hidden states of the decoder and encoder, respectively. Since there is a bidirectional RNN for the encoder, the encoder hidden state h _i is defined by concatenating the back and front hidden states. The matrices W _h , W _d and the bias vector b are learnable parameters. The softmax function normalizes the input vector, so the sum of the resulting vectors is 1. Therefore, it can have a context vector of the same size as the hidden state of the encoder. Consequently, the context vector c and _{the decoder hidden state of the previous time step d t-1} are concatenated to make the input of the decoder RNN.

)를 모델링하기위한 매개 변수를 학습할 수 있다. 여기서 o, l 및

는 각각 보코더 매개 변수, 언어 기능 및 모델 매개 변수다. 훈련 단계에서 매개 TTS는 다음 수학식 3에 설명 된대로 모델 매개 변수를 학습할 수 있다.A common TTS system may be composed of two main parts: a text encoding part and a speech generating part. Using prior knowledge of the target language, the system of the present invention can define useful functions of the language and extract them from the input text. This process is called the text encoding part and many natural language processing techniques can be used in this step. For example, a phoneme-phoneme model may be applied to input text to obtain a phoneme sequence, and a part-of-speech tagging machine may be applied to obtain part-of-speech information. In this way, the text encoding unit may take text input and output various language features. In addition, the following voice generator may generate a waveform of the voice by taking linguistic features. Two general approaches to speech generators are concatenated TTS and mediated TTS. The concatenated TTS may generate speech by concatenating short units of speech. A short unit has a scale of a phoneme or a sub-phoneme, and the order of the short units may be determined by a state transition network. This process may be similar to HMM-based speech recognition. On the other hand, intermediary TTS may use generative models and vocoders. The generative model is the joint probability p(o, l,

) can learn parameters for modeling. where o, l and

are vocoder parameters, language features, and model parameters, respectively. In the training phase, the parametric TTS can learn the model parameters as described in Equation 3 below.

[Equation 3]

Vocoder parameters can be configured with voice-related functions such as Mel frequency cepstral coefficient (MFCC), fundamental frequency (F _{0 ), aperiodicity, etc.} A pre-trained feature extractor is available to obtain the vocoder parameters. Language functions are available in the output of the text encoding part. In the generation step, vocoder parameters can be obtained from language features and model parameters as shown in Equation 4 below.

[Equation 4]

The obtained vocoder parameters are then processed by a vocoder such as Vocaine or WORLD. Eventually we can get the waveform of the voice. (unrelated content)

The Tacotron is an end-to-end TTS model consisting of three modules: encoder, decoder and post processor. Although Tacotron has a more complex structure than the vanilla sequence-to-sequence model, Tacotron basically follows a sequence-to-sequence framework with an attention mechanism that transforms sequences of characters into their corresponding waveforms. More specifically, the encoder takes a character sequence as input and produces a text encoding sequence that is the same length as the character sequence. The decoder generates mel-scale spectra in an autoregressive manner. The decoder output from the previous time-step is used as the input of the decoder in the current time-step. At the decoder module, attention RNNs usually predict attention alignments just like normal sequence-to-sequence models. Combining attention alignment with text encoding gives a context vector and the decoder RNN takes as input the context vector and the output of the attention RNN. The decoder RNN predicts the mel-scale spectrogram, and the post-processor module consequently generates a linear scale spectrogram from the mel-scale spectrogram. The linear scale spectrogram itself has no phase information, so it cannot be reverted directly to the waveform. The Griffin-Lim reconstruction algorithm applies an inverse Fourier transform to estimate the phase of a linear scale spectrogram {GLRECON}. Tacotron used the Griffin-Lim reconstruction in the final stage of the post-processor module to obtain the waveforms from the linear scale spectrogram.

6 is a schematic diagram of tacotron.

Referring to FIG. 6 , the tacotron is an end-to-end TTS model composed of three modules: an encoder, a decoder, and a post processor. Although the tacotron has a more complex structure than the vanilla sequence-to-sequence model, the tacotron can basically follow a sequence-to-sequence framework using a convergence mechanism that transforms character sequences into corresponding waveforms. More specifically, the encoder may take a character sequence as input and generate a text encoding sequence that is the same length as the character sequence. The decoder may generate a mel-scale spectrum in a self-circulating manner. The decoder output from the previous time-step can be used as the input of the decoder in the current time-step. Attention RNNs in the decoder module can usually predict the alignment of attentions just like a normal sequence-to-sequence model. Combining attention alignment with text encoding can provide a context vector and the decoder RNN can take as input the context vector and the output of the attention RNN. The decoder RNN predicts the mel-scale spectrogram, and the post-processor module can consequently generate a linear scale spectrogram from the mel-scale spectrogram. Since the linear scale spectrogram itself has no phase information, it cannot be reverted directly to the waveform. The Griffin-Lim reconstruction algorithm can estimate the phase of a linear scale spectrogram by applying an inverse Fourier transform. The tacotron can use the Griffin-Lim reconstruction at the last stage of the post-processor module to obtain a waveform from the linear scale spectrogram.

An important network architecture used in the tacotron may be the CBHG module. CBHGs may include convolutional banks, highway networks, and bidirectional Gated Recurrent Units (GRUs). The GRU may be a type of RNN. Conceptually, CBHG's convolutional layer captures local patterns, whereas bidirectional GRUs can capture long-term dependencies in a given sequence. To encode the features of the sequence, CBHG is used in the encoder and the post-processor of the tacotron.

와Y_mel이고 선형 스케일 스펙트로 그램은

와Y_linear이다. 다음과 같이 목적 함수인 수학식 5를 계산하기 위해 추가될 수 있다.For training, the model has two targets in the objective function. (1) Mel-scale spectroscopy target Y _mel and (2) linear-scale spectrogram target Y _linear . The L1 distance of each mel-scale spectrogram is

and Y _mel and the linear scale spectrogram is

and Y _linear . It can be added to calculate Equation 5, which is an objective function as follows.

[Equation 5]

와

는 상기 타코트론의 출력이고, 그라운드 진실 스펙트로그램이다.here

Wow

is the output of the tacotron, and is the ground truth spectrogram.

Fig. 7 is a graph showing an ideal graph of concentrated sorting and a problem of actual centralized sorting.

)와 인코더의 텍스트 인코딩이라는 두 가지 소스에서 입력을 받을 수 있다. 상기 두 가지 소스를 바탕으로 집중 정렬의 예측을 개선하기 위해 다음과 같이 개량할 수 있다.Referring to Figure 7, when generating long voices, the focused alignment showed irregularities in the middle part of the speech. In addition, as a result of observing the correlation between the clarity of the attention alignment and the quality of the generated speech when comparing the focused alignment with the corresponding generated speech, intuitively, the decoder of the tacotron generates a spectrogram based on the focused context vector, so You should get a spectrogram that is obscuredly predicted by the blurred focus. In an embodiment of the present invention, it was decided to examine the flow of information in the tacotron to improve the focused alignment prediction. In Figure 6, the concentration module of the tacotron is the current hidden state (

) and the encoder's text encoding. Based on the above two sources, it can be improved as follows to improve the prediction of concentrated alignment.

Since the pronunciation of a character usually requires more than one spectrogram frame, the focused part of the existing takotron model often focuses on similar parts of the text input for several decoder time steps. Even if the model needs to change the focus weight, the next part to focus on can be the adjacent part of the currently focused text. Therefore, when determining the concentration weight, the model can have information _{of the previously focused text c t-1 , which is the weighted sum of text encodings.} However, the existing tacotron does not utilize the _{above c t-1 information.} Concentrated RNN can only use spectrograms of previous time-steps that contain little information _{of c t-1 .} Based on the above, we connected _{c t-1 to the input x t} of the concentrated RNN. _{The concentrated RNN may take one more input c t-1} as shown in Equation 6 below.

[Equation 6]

where x _t and h _t are the input and hidden states of the concentrated RNN at time-step t, respectively.

8 is a schematic diagram of a CBHG module.

Referring to FIG. 8 , a second idea of improving focused prediction could change the way text input is encoded. Altered text input is also a source of information for determining focused alignment. The encoding may be generated by the CBHG module as in FIG. 8 . A CBHG may contain a bank of convolutional filters, each filter capable of explicitly extracting local and context information. The final stage of CBHG may include a bidirectional RNN that can capture long-term dependencies on text input. As the RNN reads its inputs sequentially, long-term dependencies can accumulate in the hidden state of the RNN. The problem here is that the size of the hidden state is fixed. If the sequence is long enough, the hidden state cannot contain the entire information of the sequence. In addition, the bidirectional RNN of CBHG should include text input information of the current time step and long-term dependency information. This doesn't put more strain on the hidden state. Referring to FIG. 7( a ), if the input alignment is long in the original tacotron, the central alignment may include a gap or a blurred part in the middle of the sequence. This irregularity is due to the lack of capacity of the hidden state of the bidirectional RNN in CBHG. Therefore, we add a residual connection connecting the input and output of the bidirectional RNN. The output of CBHG was changed to have an additional term of _{x t} as shown in Equation 7 below.

[Equation 7]

where x _t , h _{t ,} and y _t are the input, hidden state, and output of the bidirectional RNN, respectively. The remaining connections now carry information from the current time step, so the hidden state of a bidirectional RNN does not need to contain information from the current time step. The above connection makes the hidden state less cluttered and helps to encode textual information.

9 is a graph showing results of concentrated sorting according to each embodiment of the present invention.

Referring to FIG. 9 , an effective method for solving the problem was confirmed by comparing the four models. The four models can include baseline, (baseline + CI), (baseline + RC) and (baseline + CI + RC). The image in Fig. 9 is the attention alignment obtained during text generation of the four models. It is intuitive that text and speech should have a monotonic alignment in nature. Thus, the ideal attentional alignment should decrease continuously and monotonically (or increase along the y-axis of the focused alignment plot).

It was predicted that context insertion and residual connection could allow the network to easily predict the focused alignment, resulting in a sharp and clear focused alignment. The basic model selected sentences with problems in speech synthesis. In Figure 9 (a), it can be seen that the opaque part and discontinuity in the central alignment of the baseline model. In contrast, the concentrated alignment of the proposed model (baseline + CI + RC) showed monotonic and continuous lines in Fig. 9(b). When the generated results were examined, the focused alignment of the proposed model was sharp and clear, and the quality of the generated speech of the proposed model was better than that of the reference model. However, a significant improvement could not be seen by applying only one of the proposed methods (see FIGS. 9 (c) and 9 (d)). Applying each of these approaches individually, these results may indicate that the two information flows from the concentrated RNN and the encoder output are equally important. A model may not be able to generate speech using only one information source.

It is expected that the performance will be further improved if the monotonous concentration mechanism and anti-teacher compulsory education are implemented. Naturally, in the alignment of text and speech, using existing monotony, the model can learn focused alignment more easily than before and the model can use more capacity to predict spectrograms. The anti-teacher compulsory education can reduce the inconsistency of the input distribution of the decoder module in the training phase and the test phase. This discrepancy, called exposure-bias, can occur because the autoregressive network takes an input from a previous time step in the test phase while taking a ground truth input in the training phase. We solved this problem with constant sampling by randomly replacing ground truth inputs with generated inputs, and class-teacher forced training can expose the model to samples generated during the training phase. You can apply these tricks, but I didn't because the proposed method can be used orthogonally to these tricks.

10 is a schematic diagram of a text-multi-speech conversion tacotron according to an embodiment of the present invention.

Referring to FIG. 10 , as an input of the present invention, a speaker-id is additionally received in addition to text to generate a voice. Speaker-id is usually 0,1,2,… It can be specified as an integer of , and when entering the input of the actual model, it can be entered in the form of a one-hot vector. This one-hot vector is converted into a dense speech embedding vector through a lookup table (the same concept as one linear layer), and it is used by concatenating the decoder RNN input and the concentrated RNN input of the tacotron. . Therefore, the output voice is different depending on who the voice input is. See Fig. 10(a). In the case of the multi-voice tacotron, it was confirmed that voice synthesis for multiple speakers was possible if the data for each speaker was collected for about 30 minutes and the total of all the speakers had more than 20 hours of data.

과 표적 희박성 매개 변수

사이의 Kullback-Leibler 발산을 최소화함으로써 얻을 수 있다. 일반적으로 희박 매개 변수는 0에 가까운 값 (즉, 0.05)을 취한다. Kullback-Leibler divergence term을 원래 목적 함수에 추가하면 다음과 같은 수정 된 목적 함수 수학식 8을 얻을 수 있다.Speaker insertion is the only variable that changes the voice of the generated voice. In other words, by adjusting the speaker built-in in an appropriate way, you can get a new voice. However, we do not know what characteristics each dimension of the speaker built-in vector represents. By setting the space of the speaker built-in vector to be non-linear and tuning the value of the speaker built-in vector, it is possible to prevent intuitively changing the voice. Creating a new voice would be much easier and more intuitive if we could represent each dimension of the speaker built-in vector as a rectangular feature. By adding a Sparsity Constraint, we can give orthogonality to the voice generation of multiple speakers. In this sense, a sparsity constraint is added to the training objective function. Each dimension d of the loudspeaker containing e _d is assumed to follow the Bernoulli distribution, so there is a value in the closed interval [0,1]. If there is a low probability of having a random variable in a speaker intrinsic, the speaker intrinsic becomes rare by the definition of sparsity. This property is a sample average of each speaker built-in dimension.

and target sparsity parameters

This can be achieved by minimizing the Kullback-Leibler divergence between In general, the sparse parameter takes a value close to zero (i.e., 0.05). By adding the Kullback-Leibler divergence term to the original objective function, the following modified objective function Equation 8 can be obtained.

[Equation 8]

및 KL()는 각각 음성 임베딩 차원, 미니 배치(batch) 크기, 정규화 계수 및 Kullback-Leibler 분기 함수의 차원을 나타낸다.where D, m,

and KL( ) denote the dimensions of the negative embedding dimension, the mini-batch size, the normalization coefficient and the Kullback-Leibler branching function, respectively.

We analyzed how the learned speech interpolation formulates manifolds in vector space through two-dimensional visualization of points obtained by principal component analysis (PCA). We also compared how the sparsity constraint affects the learned speaker insertion vectors.

Referring to FIG. 10(b), the multi-voice tacotron is capable of voice synthesis for several speakers, and requires 30 minutes of voice for each speaker, which is an improvement compared to the conventional tacotron, but Obtaining a clear voice is also a task that requires a lot of effort. In addition, in a situation where one multi-voice tacotron is trained, if a model that generates a new speaker's voice is needed, a process of re-training the model for the new voice is required, and this process takes several hours. do. The model according to an embodiment of the present invention has the advantage that very little data (6 seconds in the experiment) is required to generate a new voice, and a new voice can be immediately generated without additional training of the model.

The model is a voice imitative TTS and has a structure quite similar to a multi-voice tacotron. One difference is that if the multi-voice tacotron receives (text, speaker_id) as input, Voice imitative TTS can receive (text, log-mel-spectrogram of the speaker's voice) pair as input. The log-mel-spectrogram received as an input can be converted into a speech embedding vector through an embedder network. The generated speech embedding vector is input to the decoder RNN and the lumped RNN in the same way as in the multi-voice TTS.

The current multi-speaker TTS model, which uses a hot speaker ID vector as input, cannot be easily extended to new voices for which voice data is not present in the training data. Since the model only learned embeddings for the speech represented by a single hot vector, there is no way to immediately get the embeddings of the new speech. Generating new voices requires retraining the entire TTS model or fine-tuning the embedded layer of the TTS model. This is a time-consuming process, even on computers with GPUs. You can tune the speaker embeddings to get the voice you want, but it will be difficult to find the exact combination of values for the speaker embedding vectors. The aforementioned approach is not only inaccurate, but also labor intensive. To solve the above problem more efficiently, we can include a new TTS architecture that can generate speech for new speakers on the fly without further training the TTS model or manually retrieving the speaker built-in vectors.

11 is a schematic diagram of a speech embedding prediction network according to an embodiment of the present invention.

Referring to FIG. 11 , a subnetwork for predicting a speaker built-in vector may be added. Figure 11 may input a subnetwork speaker insert comprising a convolutional layer followed by a fully connected layer. The network can predict a fixed-dimensional speaker embedding vector using a log-Mel-spectrogram as an input. Since there are no restrictions on using spectrograms, arbitrary spectrograms can be inserted into this network, which allows the network to immediately adapt to generate a new speaker's voice. The input spectrogram can have various lengths, but the maximum timed pooling layer located at the end of the convolutional layer can output the input as a fixed dimension vector of length 1 with respect to the time axis. The voice inserter can replace a single hot vector input of a multi-voice tacotron as described in Fig. 10(b). The speech mock model can be trained using the same objective function Equation (5).

12 is a table showing a voice profile in a test set according to an embodiment of the present invention, and FIG. 13 is a table showing parameters of a multi-voice tacotron according to an embodiment of the present invention.

According to the parameters of FIGS. 12 and 13 , text-multi-speech conversion can be performed, and the result can be confirmed in the drawings to be described below.

14 is an example of a generated voice spectogram according to an embodiment of the present invention.

Referring to FIG. 14 , in an embodiment of the present invention, multiple negative tacotrons of 140 epochs are trained, and 1 epoch is equal to 1289 repetitions with or without sparsity constraint on the negative embedding vector. The generated negative samples of both models showed similar quality. The gender and accent of each speaker can be clearly distinguished from the sample. 14 shows sample spectrograms of three different voices. The text corresponding to the spectrogram is the same for each voice and does not appear in the training data. It can be seen that the pitch and speed of each sample is different from the other samples. 14 (a), (b) and (c) are results of (female, high pitch), (female, low pitch), and (male, low pitch), respectively.

15 is a graph illustrating a speech embedding result according to a theoretical element according to an embodiment of the present invention.

Referring to Figure 15, we used PCA to see how the speaker built-in was learned. By selecting two dimensions with second-order eigenvalues, you can check which factors cause the most variance in the data. Figure 15 is color-coded to highlight the linear separability by gender or accent. 15(a), (c), where the model is trained without sparsity constraints, all embeddings appear to be entangled. In contrast, Figs. 15(b), (d), where the model was trained with a sparsity constraint, show clear segregation in terms of gender. Drawing a line intersecting the points (0,1) and (3,0) in Fig. 15(d), it can be seen that the line can distinguish between English accents and American accents. This result does not necessarily imply a sparsity constraint that can better represent the negative characteristics. Because we chose the two-dimensionality that gave the largest variance, the model without the sparsity constraint may have discovered other factors that could distinguish each individual's voice. However, if the voice embedding is distributed across the manifold, it can be intuitively understood, so it may be much easier for people to tune their voice to produce the voice they want. You can control the voice characteristics by directly manipulating the voice embedding values. In particular, in a model trained with sparsity constraints, each principal component can represent gender, intonation, speed, and amplitude.

16 is a graph illustrating a spectogram according to a change in a theoretical element according to an embodiment of the present invention.

Referring to Fig. 16, to demonstrate the speech intrinsic tuning approach, speech was generated by changing the values of one main component at a time while keeping the other components fixed. When the main component is selected, the maximum and minimum values of the selected component are checked. Then five values are chosen between the maximum and the minimum, and in the selected component we can get 5 speech embeddings that differ only by the same values as the other components. Since the obtained speech embeddings are in the space transformed by PCA, we applied an inverse transform to the speech embeddings to be compatible with the multi-voice tacotron. In the end, 5 voice embeddings were used to generate voice samples.

The first graph in Fig. 16 shows that the pitch increases as the value of the first principal component increases. As we expected in Fig. 15 (c), it can be seen that the voice changes from male to female as the value increases according to listening to the sample voice.

For the second major component, looking at Fig. 15(d), the speaker's accent is expected to change. By generating and listening to the sentence "I need some water from fast river.", we can identify changes in accents in the words "water," "fast," and "river." Unfortunately, there is no effective way to visualize this change, so figures from this experiment have been omitted.

You cannot get an intuition from the PCA plots of the third and fourth principal components, but you can see trends by changing the values of each component. Looking at the width of the voiced sound of each spectrogram in the second graph of FIG. 16 , the width decreases as the value of the component moves to the median value. As you listen to the sample, you will notice that the speed of the voice increases as the width decreases. However, it does not show an obvious trend compared to the first and second principal components.

Finally, the fourth major component appears to represent the amplitude of speech. The spectrogram has a stronger pattern when the sound is loud. In the third graph of FIG. 16 , the region of the brightest color is the smallest in the rightmost spectrogram, and the right spectrogram is generally darker than the left spectrogram. Also, the voice on the right may be perceived as louder than the voice on the left.

Since there were only 4 dimensions for negative embeddings, we could observe only 4 features affected by negative embedding vectors. It can be assumed that by increasing the dimension of the speech embedding vector, we will be able to discover more meaningful features. In the analysis of the third and fourth major components, we observed that speed and loudness could be one of the main factors for the variability of speech. To show the distinct differences for the last two key components, we can use data augmentation at varying speeds and volumes of existing data to use the speaker built-in tuning approach. Most data have average speed and average loudness because no data sets are collected to distinguish between speed and loudness. Therefore, a clear distinction between speed and loudness may not be sufficient. There are several ways to manipulate audio files, including the program Sound eXchange (SOX). Although the manipulated sound may have different characteristics than the original data, these tools can be used to increase the number of samples that differ in speed and loudness.

17 is a graph illustrating a spectogram by mixed voice according to an embodiment of the present invention.

Referring to Figure 17, when performing the same analysis as the multi-negative tacotron without sparsity limitation, no trend could be recognized in the sample, unlike the previous analysis. Although no intuition can be gained from this analysis and the two-dimensional visualization of speech embeddings, it is worth noting that interpolated speech embeddings can be used as input to the model to generate speech. We averaged two speech embedding vectors and used them to generate speech. It can be seen that the generated sample has a pitch level between the original two voices as shown in FIG. 17 . Further investigation is required to determine the characteristics each component exhibits.

We speculate that the sparsity constraint may further increase the problem complexity, which may lead the model to converge faster than the model without the sparsity constraint. The complexity of the model may be too high for the complexity of the problem. Therefore, the model is prone to exaggeration. Here sparsity constraint is used as a regularization term and the complexity of the problem and model can match. Assuming this scenario, we can benefit from reducing the regularization coefficient ρ after the constrained model has converged to some extent. This not only improves convergence in the early stages, but also helps to find the correct optimal parameters in the later stages. The model with the sparsity constraint may not be sufficiently trained.

It should be understood that the present invention may be embodied in many forms of hardware, software, firmware, special purpose processor, or combinations thereof. Preferably, the present invention is implemented as a combination of hardware and software. Furthermore, the software is preferably implemented as an application program tangibly embodied on a program storage device. This application program can be uploaded to and executed by a machine comprising any suitable structure. Preferably, the machine is implemented on a computer platform comprising one or more central processing units (CPUs) and random access memory (RAM) and hardware such as input/output (I/O) interface(s). . This computer platform also includes an operating system and micro-instruction code. The various processes and functions described herein may be part of microinstruction code or part of an application program (or a combination thereof) executed through an operating system. Furthermore, several other peripheral devices may be connected to this computer platform, such as additional data storage devices and printing devices.

Since some of the method steps and system components of the components shown in the accompanying drawings are preferably implemented in software, the actual connections between the system components (or method steps) may vary depending on the manner in which the present invention is programmed. You will have to understand more. According to the disclosure herein, those of ordinary skill in the related art will be able to think of these embodiments or configurations of the present invention and similar embodiments or configurations.

Claims (6)

generating, by the encoder, a character embedding vector for a sentence to generate a voice in character units;
generating, by a decoder, a speech spectrum from the embedding vector; and
Including; outputting the voice spectrum as voice by an audio output unit;
The decoder generating the voice spectrum comprises:
It receives multiple voice data, converts it into a voice embedding vector, and uses it for voice spectrum merging.
The multi-voice data is
It is a speaker-id in the form of a one-hot vector and is converted into a speech embedding vector through a lookup table,
The speech embedding vector is input to the decoder RNN and the concentrated RNN,
and the decoder adjusts the speaker embedding vector by setting the space of a speaker embedding vector (speaker embedded vector or speaker embedding vector) to be non-linear.
According to claim 1,
The multi-voice data is
A text-multi-speech conversion method that is data in log-mel-spectrogram format.
3. The method of claim 2,
The log-mel-spectrogram is converted into a speech embedding vector through an embedder network.
Including; a text-multi-speech conversion module for outputting the voice of the text from the text;
The text-multi-speech conversion module,
An encoder that generates a character embedding vector for a sentence to generate a voice character by character;
a decoder for generating a speech spectrum from the embedding vector; and
and a voice output unit for outputting the voice spectrum as voice;
The decoder is
It receives multiple voice data, converts it into a voice embedding vector, and uses it for voice spectrum merging.
The multi-voice data is
It is a speaker-id in the form of a one-hot vector and is converted into a speech embedding vector through a lookup table,
The speech embedding vector is input to the decoder RNN and the concentrated RNN,
and the decoder adjusts the speaker embedding vector by setting the space of a speaker embedding vector (speaker embedded vector or speaker embedding vector) to be non-linear.
5. The method of claim 4,
The multi-voice data is
A text-multi-speech conversion system that is data in the form of log-mel-spectrogram.
6. The method of claim 5,
The log-mel-spectrogram is converted into a speech embedding vector through an embedder network.

Images