KR102463570B1 - Method and tts system for configuring mel-spectrogram batch using unvoice section - Google Patents

Abstract

A method of generating voice data using speaker information and text includes the steps of: receiving speaker information and generating a speaker embedding vector based on the speaker information; receiving text and generating a text embedding vector based on the text; generating a mel-spectrogram based on the embedding vector and the text embedding vector, and generating speech data from the mel-spectrogram.
The step of generating the voice data includes determining a silent part in the mel-spectrogram, dividing the mel-spectrogram into a plurality of sub mel-spectrograms based on the silent part, and a plurality of sub mel-spectrograms. post-processing, and generating voice data from a plurality of post-processed sub-mel-spectrograms.

Description

Method and voice synthesis system for arrangement of Mel spectrogram through silent section detection

The present disclosure relates to a method for configuring a Mel spectrogram arrangement through silent section detection and a speech synthesis system.

Recently, with the development of artificial intelligence technology, interfaces using voice signals are becoming common. Accordingly, research on a speech synthesis technology capable of uttering a synthesized voice according to a given situation is being actively conducted.

Speech synthesis technology is being applied to many fields such as virtual assistants, audiobooks, automatic interpretation and translation, and virtual voice actors by combining speech recognition technology based on artificial intelligence.

As a conventional speech synthesis method, there are various methods such as unit selection synthesis (USS) and statistic-based parameter synthesis (HMM-based speech synthesis, HTS). The USS method cuts out speech data into phoneme units and stores them, and finds a sound piece suitable for utterance during speech synthesis. The HTS method creates a statistical model by extracting parameters corresponding to speech characteristics and converts text into speech based on the statistical model. as a way to reconstruct it. However, the conventional voice synthesis method described above has many limitations in synthesizing a natural voice reflecting the speaker's speech style or emotional expression.

Accordingly, recently, a speech synthesis method for synthesizing speech from text based on an artificial neural network is attracting attention.

The goal is to provide artificial intelligence-based speech synthesis technology that can implement natural voices as if they were being spoken by a real speaker. In addition, the purpose of the present invention is to provide an artificial intelligence-based speech synthesis technology capable of realizing a natural voice as if speaking by an actual speaker. In addition, it is to provide a highly efficient artificial intelligence-based speech synthesis technology using a small amount of learning data.

The technical problem to be solved is not limited to the technical problems as described above, and other technical problems may be inferred.

As a technical means for achieving the above technical problem, a first aspect of the present disclosure provides a method for generating voice data using speaker information and text, receiving speaker information and based on the speaker information, a speaker embedding vector creating a; receiving text and generating a text embedding vector based on the text; generating a Mel-spectrogram based on the speaker embedding vector and the text embedding vector; and generating voice data from the mel-spectrogram, wherein the generating of the voice data includes: determining a silence part in the mel-spectrogram; dividing the mel-spectrogram into a plurality of sub-mel-spectrograms based on the silent portion; post-processing the plurality of sub-mel-spectrograms; and generating the voice data from the plurality of post-processed sub-mel-spectrograms.

In addition, the post-processing may include calculating a length of each of the plurality of sub-mel-spectrograms; and post-processing the plurality of sub-mel-spectrograms so that the lengths of the plurality of sub-mel-spectrograms are equal to the reference batch length.

In addition, in the post-processing of the plurality of sub-mel-spectrograms, the sub-mel having a length less than the reference batch length so that the length of each of the plurality of sub-mel-spectrograms is equal to the preset batch length -Applying zero-padding to the spectrogram; including; a method may be provided.

According to the above-described problem solving means of the present disclosure, by dividing the Mel-spectrogram into sub-mel-spectrograms based on the silent part of the Mel-spectrogram and generating voice data from the sub-mel-spectrogram, higher accuracy Voice data can be generated.

According to one of the other problem solving means of the present disclosure, by generating the voice data using the silent Mel-spectrogram, it is possible to generate more accurate voice data.

1 is a diagram schematically illustrating an operation of a speech synthesis system.
2 is a diagram illustrating an embodiment of a speech synthesis system.
3 is a diagram illustrating an embodiment of a synthesizer of a speech synthesis system.
4 is a diagram illustrating an embodiment of outputting a Mel spectrogram through a synthesizer.
5 is a diagram illustrating a volume graph corresponding to a Mel-spectrogram according to an exemplary embodiment.
6A to 6B are diagrams for explaining a process of dividing a mel-spectrogram into a plurality of sub-mel-spectrograms according to an exemplary embodiment.
7 is a diagram for explaining a process of generating voice data from a plurality of sub-mel-spectrograms according to an embodiment.
8 is a diagram for explaining an example of dividing a text sequence according to an embodiment.
FIG. 9 is a diagram for explaining a process of generating a final Mel-spectrogram by adding a silent Mel-spectrogram between sub-mel-spectrograms according to an exemplary embodiment.
10 is a flowchart of a method for determining a silent portion of a mel-spectrogram according to an embodiment.

The terms used in the present embodiments have been selected as currently widely used general terms as possible while considering the functions in the present embodiments, but this may vary depending on the intention or precedent of a person skilled in the art, the emergence of new technology, etc. have. In addition, in certain cases, there are also terms arbitrarily selected by the applicant, and in this case, the meaning will be described in detail in the relevant part. Therefore, the terms used in the present embodiments should be defined based on the meaning of the term and the contents throughout the present embodiments, rather than the simple name of the term.

Since the present embodiments may have various changes and may have various forms, some embodiments will be illustrated in the drawings and described in detail. However, this is not intended to limit the present embodiments to a specific disclosure form, and it should be understood that all modifications, equivalents and substitutes included in the spirit and scope of the present embodiments are included. The terms used herein are used only for description of the embodiments, and are not intended to limit the present embodiments.

Unless otherwise defined, terms used in the present embodiments have the same meanings as commonly understood by those of ordinary skill in the art to which the present embodiments belong. Terms such as those defined in a commonly used dictionary should be interpreted as having a meaning consistent with the meaning in the context of the related art, and unless explicitly defined in the present embodiments, have an ideal or excessively formal meaning. should not be interpreted.

DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS [0012] DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS [0014] DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS [0016] Reference is made to the accompanying drawings, which show by way of illustration specific embodiments in which the present invention may be practiced. These embodiments are described in sufficient detail to enable those skilled in the art to practice the present invention. It should be understood that the various embodiments of the present invention are different but need not be mutually exclusive. For example, certain shapes, structures, and characteristics described herein may be implemented with changes from one embodiment to another without departing from the spirit and scope of the present invention. In addition, it should be understood that the location or arrangement of individual components within each embodiment may be changed without departing from the spirit and scope of the present invention. Accordingly, the following detailed description is not to be taken in a limiting sense, and the scope of the present invention should be taken as encompassing the scope of the claims and all equivalents thereto. In the drawings, like reference numerals refer to the same or similar elements throughout the various aspects.

On the other hand, in the present specification, technical features that are individually described within one drawing may be implemented individually or at the same time.

Hereinafter, various embodiments of the present invention will be described in detail with reference to the accompanying drawings in order to enable those of ordinary skill in the art to easily practice the present invention.

1 is a diagram schematically illustrating an operation of a speech synthesis system.

A speech synthesis device is a device that converts text into human speech.

For example, the speech synthesis system 100 of FIG. 1 may be an artificial neural network-based speech synthesis system. An artificial neural network refers to an overall model that has problem-solving ability by changing the strength of synaptic bonding through learning in which artificial neurons that form a network through synaptic bonding.

The voice synthesis system 100 may be implemented in various types of devices such as a personal computer (PC), a server device, a mobile device, an embedded device, and the like, and as a specific example, a smartphone or tablet that performs voice synthesis using an artificial neural network. It may correspond to a device, an Augmented Reality (AR) device, an Internet of Things (IoT) device, an autonomous vehicle, robotics, a medical device, an e-book terminal, and a navigation device, but is not limited thereto.

Furthermore, the speech synthesis system 100 may correspond to a dedicated hardware accelerator mounted on the above device. Alternatively, the speech synthesis system 100 may be a hardware accelerator such as a neural processing unit (NPU), a tensor processing unit (TPU), or a neural engine, which is a dedicated module for driving an artificial neural network, but is not limited thereto.

Referring to FIG. 1 , the speech synthesis system 100 may receive a text input and specific speaker information. For example, the speech synthesis system 100 may receive "Have a good day!" as shown in FIG. 1 as a text input, and may receive "Speaker 1" as a speaker information input.

"Speaker 1" may correspond to a voice signal or a voice sample indicating the preset speech characteristics of Speaker 1. For example, the speaker information may be received from an external device through a communication unit included in the speech synthesis system 100 . Alternatively, the speaker information may be input from the user through the user interface of the speech synthesis system 100 and may be selected from among various speaker information pre-stored in the database of the speech synthesis system 100, but is limited thereto. not.

The speech synthesis system 100 may output a speech based on a text input received as an input and specific speaker information. For example, the speech synthesis system 100 may say "Have a good day!" and "Speaker 1" may be received as an input, and a voice for "Have a good day!" in which the speech characteristics of speaker 1 are reflected may be output. The speech characteristics of the first speaker may include at least one of various factors such as the speaker's voice, rhyme, pitch, and emotion. That is, the output voice may be a voice as if the speaker 1 naturally pronounces “Have a good day!”. A detailed operation of the speech synthesis system 100 will be described later with reference to FIGS. 2 to 4 .

2 is a diagram illustrating an embodiment of a speech synthesis system. The speech synthesis system 200 of FIG. 2 may be the same as the speech synthesis system 100 of FIG. 1 .

Referring to FIG. 2 , the speech synthesis system 200 may include a speaker encoder 210 , a synthesizer 220 , and a vocoder 230 . Meanwhile, in the speech synthesis system 200 illustrated in FIG. 2 , only components related to an embodiment are illustrated. Accordingly, it is apparent to those skilled in the art that the speech synthesis system 200 may include other general-purpose components in addition to the components shown in FIG. 2 .

The speech synthesis system 200 of FIG. 2 may receive speaker information and text as inputs and output a speech.

For example, the speaker encoder 210 of the speech synthesis system 200 may receive speaker information as an input and generate a speaker embedding vector. The speaker information may correspond to a speaker's voice signal or a voice sample. The speaker encoder 210 may receive the speaker's voice signal or voice sample, extract the speaker's utterance characteristics, and represent this as an embedding vector.

The speaker's speech characteristics may include at least one of various factors such as speech speed, pause period, pitch, tone, rhyme, intonation, or emotion. That is, the speaker encoder 210 may represent the discontinuous data values included in the speaker information as a vector composed of continuous numbers. For example, the speaker encoder 210 includes a pre-net, a CBHG module, a deep neural network (DNN), a convolutional neural network (CNN), a recurrent neural network (RNN), a long short-term memory network (LSTM), a BRDNN ( A speaker embedding vector may be generated based on at least one or a combination of two or more of various artificial neural network models such as Bidirectional Recurrent Deep Neural Network).

For example, the synthesizer 220 of the speech synthesis system 200 may receive text and an embedding vector representing the speaker's utterance characteristics as inputs, and output a spectrogram.

3 is a diagram illustrating an embodiment of a synthesizer of a speech synthesis system. The synthesizer 300 of FIG. 3 may be the same as the synthesizer 220 of FIG. 2 .

Referring to FIG. 3 , the synthesizer 300 of the speech synthesis system 200 may include an encoder and a decoder. On the other hand, it is apparent to those skilled in the art that the synthesizer 300 may further include other general-purpose components.

The embedding vector representing the speech characteristic of the speaker may be generated from the speaker encoder 210 as described above, and the encoder or decoder of the synthesizer 300 receives the embedding vector representing the speech characteristic of the speaker from the speaker encoder 210. can

The encoder of the synthesizer 300 may receive text as input and generate a text embedding vector. The text may include a sequence of characters in a particular natural language. For example, the sequence of characters may include alphabetic characters, numbers, punctuation marks, or other special characters.

The encoder of the synthesizer 300 may separate the inputted text into a unit of a alphabet, a unit of a letter, or a unit of a phoneme, and may input the separated text into the artificial neural network model. For example, the encoder of the synthesizer 300 may generate a text embedding vector based on at least one or a combination of two or more of various artificial neural network models such as pre-net, CBHG module, DNN, CNN, RNN, LSTM, BRDNN. .

Alternatively, the encoder of the synthesizer 300 may divide the input text into a plurality of short texts and generate a plurality of text embedding vectors for each of the short texts.

The decoder of the synthesizer 300 may receive a speaker embedding vector and a text embedding vector from the speaker encoder 210 as inputs. Alternatively, the decoder of the synthesizer 300 may receive a speaker embedding vector from the speaker encoder 210 as an input, and receive a text embedding vector from the encoder of the synthesizer 300 as an input.

The decoder of the synthesizer 300 may generate a spectrogram corresponding to the input text by inputting the speaker embedding vector and the text embedding vector to the artificial neural network model. That is, the decoder of the synthesizer 300 may generate a spectrogram of the input text in which the speaker's speech characteristics are reflected. For example, the spectrogram may correspond to a mel-spectrogram, but is not limited thereto.

A spectrogram is a graph that visualizes the spectrum of a voice signal. The x-axis of the spectrogram represents time and the y-axis represents frequency, and the value of each time frequency can be expressed as a color according to the size of the value. The spectogram may be a result of performing short-time Fourier transform (STFT) on a continuously given speech signal.

STFT is a method in which a speech signal is divided into sections of a certain length and a Fourier transform is applied to each section. At this time, since the result of performing STFT on the speech signal is a complex value, it is possible to take an absolute value to the complex value to lose phase information and to generate a spectrogram including only magnitude information.

On the other hand, the Mel spectrogram is a readjustment of the frequency interval of the spectrogram to the Mel scale. The human auditory organ is more sensitive in the low frequency band than in the high frequency band, and the Mel Scale expresses the relationship between the physical frequency and the frequency perceived by humans by reflecting these characteristics. The Mel spectrogram may be generated by applying a filter bank based on the Mel scale to the spectrogram.

Meanwhile, although not shown in FIG. 3 , the synthesizer 300 may further include an attention module for generating the attention alignment. The attention module is a module that learns which output of a specific time-step of the decoder is most related to the output of all the time-steps of the encoder. A higher quality spectrogram or Mel spectrogram can be output by using the attention module.

4 is a diagram illustrating an embodiment of outputting a Mel spectrogram through a synthesizer. The synthesizer 400 of FIG. 4 may be the same as the synthesizer 300 of FIG. 3 .

Referring to FIG. 4 , the synthesizer 400 may receive a list including input texts and speaker embedding vectors corresponding thereto. For example, the synthesizer 400 includes the input text 'first sentence', the corresponding speaker embedding vectors embed_voice1, the input text 'second sentence', and the corresponding speaker embedding vectors embed_voice2 and the input text 'third sentence'. and a list 410 including embed_voice3, which is a speaker embedding vector corresponding thereto, may be received as an input.

The synthesizer 400 may generate as many Mel spectrograms 420 as the number of input texts included in the received list 410 . Referring to FIG. 4 , it can be seen that Mel spectrograms corresponding to input texts of 'first sentence', 'second sentence' and 'third sentence' are generated.

Alternatively, the synthesizer 400 may generate as many Mel spectrograms 420 and attention alignment as the number of input texts together. Although not shown in FIG. 4 , for example, attention alignment corresponding to each input text of 'first sentence', 'second sentence' and 'third sentence' may be additionally generated. Alternatively, the synthesizer 400 may generate a plurality of Mel spectrograms and a plurality of attention alignments for each of the input texts.

Returning to FIG. 2 again, the vocoder 230 of the speech synthesis system 200 may generate the spectrogram output from the synthesizer 220 as actual speech. As described above, the output spectrogram may be a Mel spectrogram.

In an embodiment, the vocoder 230 may generate the spectrogram output from the synthesizer 220 as an actual speech signal by using Inverse Short-Time Fourier Transform (ISFT). Since the spectrogram or Mel spectrogram does not include phase information, when a voice signal is generated using ISFT, the phase information of the spectrogram or Mel spectrogram is not considered.

In another embodiment, the vocoder 230 may generate the spectrogram output from the synthesizer 220 as an actual speech signal using a Griffin-Lim algorithm. The Griffin-Rim algorithm is an algorithm for estimating phase information from size information of a spectrogram or Mel spectrogram.

Alternatively, the vocoder 230 may generate a spectrogram output from the synthesizer 220 as an actual voice signal, for example, based on a neural vocoder.

A neural vocoder is an artificial neural network model that generates a voice signal by receiving a spectrogram or a Mel spectrogram as an input. A neural vocoder can learn the relationship between a spectrogram or a Mel spectrogram and a voice signal from a large amount of data, and can generate a high-quality real voice signal.

A neural vocoder may correspond to a vocoder based on an artificial neural network model such as, but not limited to, WaveNet, Parallel WaveNet, WaveRNN, WaveGlow, or MelGAN.

For example, the WaveNet vocoder consists of several dilated causal convolution layers and is an autoregressive model that uses sequential features between speech samples. The WaveRNN vocoder is an autoregressive model that replaces the dilated causal convolution layer of WaveNet with a Gated Recurrent Unit (GRU). The WaveGlow vocoder can learn to obtain a simple distribution such as a Gaussian distribution from the spectrogram dataset (x) by using an invertible transform function. After learning, the WaveGlow vocoder can output a voice signal from a sample of a Gaussian distribution using the inverse function of the transform function.

5 is a diagram illustrating a volume graph corresponding to a Mel-spectrogram according to an exemplary embodiment.

The Mel-spectrogram 520 may include a plurality of frames. Referring to FIG. 5 , the Mel-spectrogram 520 may include 400 frames. The processor may generate the volume graph 510 by calculating the average energy of each frame. In the frame of the Mel-spectrogram 520 , a dark-colored portion (eg, a yellow portion) has a high volume value. The volume graph 510 generated from the Mel-spectrogram 520 has a maximum value of 4.0 and a minimum value of -4.0.

The larger the average energy of the frame, the larger the volume value, and the smaller the average energy of the frame, the smaller the volume value. That is, a frame having a small average energy may correspond to a silent portion.

The processor may determine a silent portion in the mel-spectrogram 520 . The processor may generate the volume graph 510 by calculating a volume value for each of a plurality of frames constituting the mel-spectrogram 520 .

The processor may select at least one frame having a volume value equal to or less than the first threshold value 511 from among the plurality of frames as the first sections 521a to 521f.

In an embodiment, the processor may determine the first sections 521a to 521f as the silent part of the Mel-spectrogram 520 . For example, the first threshold value 511 may be -3.0, -3.25, -3.5, -3.75, etc., but is not limited thereto. The first threshold value 511 may be set differently according to how much noise is included in the Mel-spectrogram 520 . In the case of the noisy Mel-spectrogram 520 , the first threshold 511 may be set to a larger value.

In another embodiment, the processor may select a section in which the number of frames is equal to or greater than the second threshold value from among the first sections 521a to 521f as the second sections 521c and 521e. The processor may determine the second sections 521c and 521e of the Mel-spectrogram 520 as the silent part. For example, the second threshold value may be 3, 4, 5, 6, 7, etc., but is not limited thereto. The second threshold value may be determined based on the overlap value and the hop size set in WaveRNN, which is one of the vocoders, when making a voice with the Mel-spectrogram 520 . overlap means the length of crossfading between batches when creating voice data in WaveRNN. For example, if the overlap value is 1200 and the hop size is 300, the second threshold value may be set to 4 or 5 because it is preferable that the volume value of four consecutive frames is less than or equal to the first threshold value (511).

Referring to FIG. 5 , the processor may determine a section of [[123, 132, 141], [280, 283, 286]] of the Mel-spectrogram 520 as a silent part. The above list means [the starting point of the silent part, the middle value, and the ending point of the silent part]. On the other hand, even if the volume value of the frames included in the first first section 521a is less than or equal to the first threshold value 511 and the number of frames is equal to or greater than the second threshold value, the first first section 521a is a voice start This section can be excluded from the silent section. However, when the processor later generates voice data from the Mel-spectrogram 520 , the processor may set the voice start point after the first first section 521a or later.

6A to 6B are diagrams for explaining a process of dividing a mel-spectrogram into a plurality of sub-mel-spectrograms according to an exemplary embodiment.

The Mel-spectrogram 620 of FIG. 6A may correspond to the Mel-spectrogram 520 of FIG. 5 .

The processor may divide the mel-spectrogram 620 into a plurality of sub mel-spectrograms 631 , 632 , and 633 based on the second sections 521c and 521e determined as the silent part in FIG. 5 . If the first second interval 521c is [123, 132, 141], and the second second interval 521e is [280, 283, 286], the processor determines the median value of the first second interval 521c may be determined as the first dividing point 641 , and an intermediate value of the second second section 521e may be determined as the second dividing point 642 .

The processor may generate a plurality of sub-mel-spectrograms 631 , 632 , and 633 by dividing the mel-spectrogram 620 based on the first dividing point 641 and the second dividing point 642 .

The processor may calculate the length of each of the plurality of sub-mel-spectrograms 631 , 632 , and 633 . Since the processor divided the mel-spectrogram 620 into a plurality of sub-mel-spectrograms 631, 632, and 633 based on the silent part of the mel-spectrogram 620, a plurality of sub-mel-spectrograms 631 , 632, 633) may have different lengths.

Referring to FIG. 6A , the first sub-mel-spectrogram 631 has a length of 132 corresponding to the interval [0, 132], and the second sub-mel-spectrogram 632 is in the interval [133, 283]. It has a corresponding length of 150, and the third sub Mel-spectrogram 633 has a length of 114 corresponding to [284, 398].

Referring to FIG. 6B , the processor performs a plurality of sub-mel-spectrograms 631, 632, and 633 such that the lengths of the plurality of sub-mel-spectrograms 631, 632, and 633 are equal to the reference batch length. can be post-processed. In an embodiment, the reference arrangement length may be a preset value.

In another embodiment, the reference arrangement length may be set as the length of the sub-mel-spectrogram having the longest length among the plurality of sub-mel-spectrograms 631 , 632 , and 633 . For example, the length of the first sub-mel-spectrogram 631 is 132, the length of the second sub-mel-spectrogram 632 is 150, and the length of the third sub-mel-spectrogram 633 is 114. In this case, the reference arrangement length may be set to 150.

The processor performs zero-padding on sub-mel-spectrograms having a length less than the reference batch length, such that the lengths of the plurality of sub-mel-spectrograms 631, 632, and 633 are equal to the reference batch length. -padding) can be applied. For example, when the reference batch length is set to 150, the processor may apply zero-padding to the first sub-mel-spectrogram 631 and the third sub-mel-spectrogram 633 .

Referring to FIG. 6B , the plurality of post-processed sub-mel-spectrograms 651 , 652 , and 653 all have the same length (eg, 150 ).

7 is a diagram for explaining a process of generating voice data from a plurality of sub-mel-spectrograms according to an embodiment.

Referring to FIG. 7 , the plurality of post-processed sub-mel-spectrograms 751 , 752 , and 753 all have the same length (eg, 150 ). Each of the plurality of post-processed sub-mel-spectrograms 751, 752, and 753 of FIG. 7 may correspond to the plurality of post-processed sub-mel-spectrograms 751, 752, and 753 of FIG. 6 .

The processor may generate a plurality of sub-voice data 761 , 762 , and 763 from each of the plurality of post-processed sub-mel-spectrograms 751 , 752 , and 753 . For example, the processor may generate a plurality of sub-voice data 761, 762, 763 from each of the plurality of sub-mel-spectrograms 751, 752, and 753 post-processed using ISFT or a Griffin-Rim algorithm. have.

In one embodiment, the processor, based on the length of each of the plurality of sub-mel-spectrograms 631, 632, and 633 before post-processing, a reference interval for each of the plurality of sub-voice data 761, 762, 763 ( 771, 772, 773) can be determined.

For example, the plurality of post-processed sub-mel-spectrograms 751, 752, and 753 all have a length of 150, but a first sub-mel-spectrogram corresponding to the first post-processed sub-mel-spectrogram 751 Since the spectrogram 631 has a length of 132, the first post-processed sub-mel-spectrogram 751 includes valid data up to a length of 132 . For the same reason, the third post-processed sub-mel-spectrogram 753 includes valid data only up to a length of 114, whereas, in the case of the second post-processed sub-mel-spectrogram 752, data valid for all lengths of 150 may include.

The processor determines the length of the first reference section 771 of the first sub-voice data 761 generated from the first post-processed sub-mel-spectrogram 751 to be 132, and uses the second post-processed sub-mel-spectrum as 132. The length of the second reference section 772 of the second sub-voice data 762 generated from the gram 752 is determined to be 150, and the third sub-voice generated from the third post-processed sub-mel-spectrogram 753 is The length of the third reference section 773 of the data 763 may be determined to be 114 .

The processor may generate the voice data 780 by connecting the first reference section 771 , the second reference section 772 , and the third reference section 773 .

The processor may generate voice data from the plurality of sub-mel-spectrograms based on a length of each of the plurality of sub-mel-spectrograms and a preset hop size. Specifically, the processor multiplies the length of each of the plurality of sub-mel-spectrograms by the hop size (eg, 300) corresponding to the length of voice data covered by one frame of the mel-spectrogram, thereby Reference sections 771 , 772 , and 773 for each of the sub-voice data 761 , 762 , and 763 may be determined.

Meanwhile, the processor of FIGS. 5, 6A, 6B, and 7 may be hardware included in the vocoder of the speech synthesis system and/or separate and independent hardware.

8 is a diagram for explaining an example of dividing a text sequence according to an embodiment.

8 shows an example of a text sequence constructed in a specific natural language. Also, the text sequence includes punctuation marks. For example, punctuation marks are '.', ',', '?', '!' etc.

The speech synthesis systems 100 and 200 identify characters and punctuation marks included in the text sequence. Then, the speech synthesis systems 100 and 200 check whether the punctuation marks included in the text sequence are predetermined punctuation marks.

If the punctuation marks are predetermined punctuation marks, the speech synthesis systems 100 and 200 may divide the text sequence based on the punctuation marks. For example, when the punctuation marks '.', ',', '?', and '!' are predetermined punctuation marks, the speech synthesis systems 100 and 200 divide the text sequence based on the preset punctuation marks. , subsequences may be generated.

FIG. 9 is a diagram for explaining a process of generating a final Mel-spectrogram by adding a silent Mel-spectrogram between sub-mel-spectrograms according to an exemplary embodiment.

Referring to FIG. 9 , the speech synthesis systems 100 and 200 may divide a text sequence 910 into a plurality of subsequences 911 , 912 , and 913 . Since the method of dividing the text sequence 910 into the plurality of subsequences 911, 912, and 913 has been described above with reference to FIG. 8, it will be omitted herein.

The speech synthesis systems 100 and 200 may generate a plurality of sub-mel-spectrograms 921 , 922 , and 923 using the plurality of sub-sequences 911 , 912 , and 913 . Specifically, the speech synthesis systems 100 and 200 may generate a plurality of sub-sequences 911, 912, and 913 corresponding to text and a plurality of sub-mel-spectrograms 921, 922, and 923 based on speaker information. have. Also, the speech synthesis systems 100 and 200 may generate speech data from the plurality of sub Mel-spectrograms 921 , 922 , and 923 . Specific details thereof have been described above with reference to FIGS. 1 to 4 , and thus will be omitted herein.

In one embodiment, the speech synthesis system 100, 200 adds the silent Mel-spectrograms 931 and 932 between the plurality of sub-mel-spectrograms 921, 922, 923 to obtain a final Mel-spectrogram 940. can create The speech synthesis systems 100 and 200 may generate speech data from the final Mel-spectrogram 940 .

Specifically, the speech synthesis system 100, 200 identifies the last character of the plurality of sub-sequences 911, 912, 913 (ie, text) corresponding to the plurality of sub-mel-spectrograms 921, 922, 923 can do. When the last character is the first group character, the speech synthesis systems 100 and 200 may generate the final Mel-spectrogram by adding the silent Mel-spectrogram having the first time to the sub-mel-spectrogram. Also, when the last character is the second group character, the speech synthesis systems 100 and 200 may generate the final Mel-spectrogram by adding a silent Mel-spectrogram having a second time to the sub-mel-spectrogram. .

For example, the first group character may include ',' and ' ' as characters corresponding to a short idle period. Also, the second group character may include '.', '?', and '!' as characters corresponding to a long idle period. In this case, the first time may be set as the reference time, and the second time may be set to three times the reference time.

On the other hand, the speech synthesis systems 100 and 200 can classify the characters of the plurality of subsequences 911, 912, and 913 into two or more groups, and the time of the silent Mel-spectrogram corresponding to each group is also the above-described example. not limited

In another embodiment, the speech synthesis systems 100 and 200 may generate a final Mel-spectrogram by adding a breath sound-mel-spectrogram between the plurality of sub-mel-spectrograms 921 , 922 , and 923 . To this end, the speech synthesis systems 100 and 200 may acquire breath sound data as speaker information.

In another embodiment, the speech synthesis system 100, 200 extracts the last character of the plurality of sub-sequences 911, 912, 913 (ie, text) corresponding to the plurality of sub-mel-spectrograms 921, 922, 923. can be identified. When the last character is the first group character, the speech synthesis systems 100 and 200 may generate the final mel-spectrogram by adding a silent mel-spectrogram having a predetermined time to the sub mel-spectrogram. Also, when the last character is the second group character, the speech synthesis systems 100 and 200 may generate the final mel-spectrogram by adding the breath sound mel-spectrogram to the sub mel-spectrogram. For example, the first group character may include ',' and ' ' as characters corresponding to a short idle period. Also, the second group character may include '.', '?', and '!' as characters corresponding to a long idle period.

In another embodiment, the speech synthesis system 100 , 200 generates a final mel-spectrogram by adding a silent mel-spectrogram having an arbitrary time between the plurality of sub mel-spectrograms 921 , 922 , 923 . may be

10 is a flowchart of a method for determining a silent portion of a mel-spectrogram according to an embodiment.

Referring to FIG. 10 , in step 1010, the processor may receive speaker information and generate a speaker embedding vector based on the speaker information.

The speaker information may correspond to a speaker's voice signal or a voice sample. The processor may receive the speaker's voice signal or voice sample, extract the speaker's utterance characteristics, and represent this as an embedding vector.

The speaker's speech characteristics may include at least one of various factors such as speech speed, pause period, pitch, tone, rhyme, intonation, or emotion. That is, the processor may represent the discontinuous data values included in the speaker information as a vector composed of continuous numbers. For example, the processor includes pre-net, CBHG module, Deep Neural Network (DNN), Convolutional Neural Network (CNN), Recurrent Neural Network (RNN), Long Short-Term Memory Network (LSTM), Bidirectional Recurrent Deep Neural (BRDNN). Network) and the like, a speaker embedding vector may be generated based on at least one or a combination of two or more of various artificial neural network models. In one embodiment, step 1010 may be performed by the speaker encoder 210 of FIG. 2 .

In operation 1020, the processor may receive the text and generate a text embedding vector based on the text.

The text may include a sequence of characters in a particular natural language. For example, the sequence of characters may include alphabetic characters, numbers, punctuation marks, or other special characters.

The processor may divide the inputted text into a unit of a alphabet, a unit of a letter, or a unit of a phoneme, and may input the separated text into the artificial neural network model. For example, the processor may generate a text embedding vector based on at least one or a combination of two or more of various artificial neural network models such as pre-net, CBHG module, DNN, CNN, RNN, LSTM, and BRDNN.

Alternatively, the processor may divide the input text into a plurality of short texts and generate a plurality of text embedding vectors for each of the short texts. In one embodiment, step 1020 may be performed by the synthesizer 220 of FIG. 2 .

In operation 1030, the processor may generate a Mel-spectrogram based on the speaker embedding vector and the text embedding vector.

The processor may receive a speaker embedding vector and a text embedding vector as inputs. The processor may generate a spectrogram corresponding to the input text by inputting the speaker embedding vector and the text embedding vector into the artificial neural network model. That is, the processor may generate a spectrogram of the input text in which the speaker's speech characteristics are reflected. For example, the spectrogram may correspond to a mel-spectrogram, but is not limited thereto.

A spectrogram is a graph that visualizes the spectrum of a voice signal. The x-axis of the spectrogram represents time and the y-axis represents frequency, and the value of the frequency per time can be expressed as a color according to the size of the value. The spectogram may be a result of performing short-time Fourier transform (STFT) on a continuously given speech signal. The Mel spectrogram is a readjustment of the frequency interval of the spectrogram to the Mel Scale. In one embodiment, step 1030 may be performed by the synthesizer 220 of FIG. 2 .

In operation 1040, the processor may determine a silence portion in the mel-spectrogram.

The processor may generate a volume graph by calculating a volume value for each of a plurality of frames constituting the mel-spectrogram. The processor may select at least one frame having a volume value equal to or less than the first threshold value from among the plurality of frames as the first period. In an embodiment, the processor may determine the first interval as a silent part of the Mel-spectrogram.

In another embodiment, the processor may select a section in which the number of frames is equal to or greater than the second threshold value from among the first section as the second section. The processor may determine the second section of the mel-spectrogram as a silent part. For example, the second threshold value may be determined based on the overlap value and the hop size set in WaveRNN, which is one of the vocoders, when making a voice with the Mel-spectrogram 520 .

In operation 1050, the processor may divide the mel-spectrogram into a plurality of sub-mel-spectrograms based on the silent portion.

The processor may calculate a length of each of the plurality of sub-mel-spectrograms. The processor may post-process the plurality of sub-mel-spectrograms so that the lengths of the plurality of sub-mel-spectrograms are equal to the reference batch length. In an embodiment, the reference arrangement length may be a preset value. In another embodiment, the reference batch length may be set as the length of the sub-mel-spectrogram having the longest length among the plurality of sub-mel-spectrograms.

The processor may generate the voice data from a plurality of post-processed sub-mel-spectrograms. The processor applies zero-padding to the sub-mel-spectrograms having a length less than the reference batch length so that the length of each of the plurality of sub-mel-spectrograms is equal to the preset batch length, thereby forming a plurality of sub-mel-spectrograms. Sub-mel-spectrograms can be post-processed.

In operation 1060, the processor may generate voice data from the plurality of sub-mel-spectrograms.

The processor may generate a plurality of sub-voice data from each of the plurality of post-processed sub-mel-spectrograms. The processor may determine a reference section for each of the plurality of sub-voice data based on the lengths of each of the plurality of sub-mel-spectrograms.

The processor may generate voice data from the plurality of sub-mel-spectrograms based on a length of each of the plurality of sub-mel-spectrograms and a preset hop size. Specifically, the processor multiplies the length of each of the plurality of sub mel-spectrograms by a hop size corresponding to the length of the voice data covered by one frame of the mel-spectrogram, thereby referencing each of the plurality of sub-voice data. section can be determined.

The processor may generate voice data by concatenating the reference sections.

In this specification, "unit" may be a hardware component such as a processor or circuit, and/or a software component executed by a hardware component such as a processor.

The description of the present specification described above is for illustration, and those of ordinary skill in the art to which the content of this specification pertains will understand that it can be easily modified into other specific forms without changing the technical spirit or essential features of the present invention. will be able Therefore, it should be understood that the embodiments described above are illustrative in all respects and not restrictive. For example, each component described as a single type may be implemented in a dispersed form, and likewise components described as distributed may be implemented in a combined form.

The scope of the present embodiment is indicated by the claims to be described later rather than the above detailed description, and it should be construed as including all changes or modifications derived from the meaning and scope of the claims and their equivalents.

Claims (3)

A method of generating voice data using speaker information and text, the method comprising:
receiving speaker information and generating a speaker embedding vector based on the speaker information;
receiving text and generating a text embedding vector based on the text;
generating a Mel-spectrogram based on the speaker embedding vector and the text embedding vector; and
Including; generating voice data from the Mel-spectrogram;
The generating of the voice data comprises:
determining a silence portion in the Mel-spectrogram;
dividing the mel-spectrogram into a plurality of sub-mel-spectrograms based on the silent portion;
calculating a length of each of the plurality of sub-mel-spectrograms;
post-processing the plurality of sub-mel-spectrograms so that the lengths of the plurality of sub-mel-spectrograms are equal to a reference batch length; and
generating the voice data from the post-processed plurality of sub-mel-spectrograms;
including,
The reference batch length is set to the length of the sub-mel-spectrogram having the longest length among the plurality of sub-mel-spectrograms,
Way. delete The method of claim 1,
Post-processing the plurality of sub-mel-spectrograms,
applying zero-padding to the sub-mel-spectrograms having a length less than the reference batch length so that the length of each of the plurality of sub-mel-spectrograms is equal to the reference batch length;
A method comprising

Images