KR102057926B1 - Apparatus for synthesizing speech and method thereof - Google Patents

Abstract

Provided is a voice synthesis device with a beat, tone, or intensity adjustment function. According to embodiments of the present invention, the voice synthesis device comprises: a preprocessing unit preprocessing input text; and a voice synthesis unit inputting the preprocessed text and beat information to a neural network-based voice synthesis model to synthesize a target voice reflecting the beat information on the input text. The voice synthesis model comprises: an embedding module converting the preprocessed text into a text-embedded vector; an aggregating module aggregating the beat information and the text-embedded vector to generate an input vector forming an input sequence; an encoder neural network encoding the input sequence to output an encoded vector; and a decoder neural network decoding the encoded vector to output an output sequence related to the target voice.

Description

Speech synthesizer and its method {APPARATUS FOR SYNTHESIZING SPEECH AND METHOD THEREOF}

The present disclosure relates to a speech synthesis apparatus and a method thereof. More specifically, the present invention relates to a speech synthesis apparatus equipped with pitch, sound intensity, and time signature control, a speech synthesis method performed in the apparatus, and a method for constructing a neural network-based speech synthesis model equipped with the above-described adjustment functions. .

Speech synthesis is a technique for synthesizing sounds similar to human speech from input text, and is also known as a text-to-speech technique. In recent years, the development and dissemination of personal portable devices such as smart phones, e-book readers, and vehicle navigation systems have been actively performed, and the demand for speech synthesis technology for voice output is rapidly increasing.

As the demand for speech synthesis technology increases, the requirements have also been subdivided. Recently, there has been a demand for pitch or time control functions beyond simply synthesizing speech for a given text.

In the related art, a technique of inferring a formant between syllables from text input through machine learning and adjusting a pitch of a synthesized voice based on the inferred formant has been utilized. However, the voice synthesized in the conventional manner has a problem in that the natural sound is degraded, such as a machine sound is mixed.

Therefore, there is a need for a new method capable of synthesizing natural voices while being able to adjust pitch and time signature.

Korean Patent Publication No. 10-2011-0021944 (published Mar. 04, 2011)

SUMMARY An object of the present disclosure is to provide a speech synthesizing apparatus and a method performed in the apparatus capable of performing pitch control together in synthesizing speech for a given text.

Another technical problem to be solved by some embodiments of the present disclosure is to provide a speech synthesizing apparatus and a method performed in the apparatus that can perform the time adjustment in synthesizing speech for a given text.

Another technical problem to be solved by some embodiments of the present disclosure is to provide a speech synthesis apparatus and a method performed in the apparatus that can perform the sound intensity control in synthesizing speech for a given text. will be.

The technical problems of the present disclosure are not limited to the technical problems mentioned above, and other technical problems that are not mentioned will be clearly understood by those skilled in the art from the following description.

In order to solve the above technical problem, a speech synthesis apparatus according to some embodiments of the present disclosure may include a preprocessor for performing preprocessing on input text and inputting the preprocessed text and time signature information to a neural network-based speech synthesis model. And a speech synthesizer for synthesizing the target speech with the time information reflected on the input text, wherein the speech synthesis model includes: an embedding module for converting the preprocessed text into a character embedding vector, and the beat information and the character embedding vector. An aggregator module for generating an input vector constituting an input sequence by aggregating the encoder, an encoder neural network for encoding the input sequence to output an encoded vector, and an output sequence associated with the target speech by decoding the encoded vector. It may include a decoder neural network for outputting.

In some embodiments, the speech synthesis model may further comprise an attention module located between the encoder neural network and the decoder neural network and determining an portion of the decoder neural network to concentrate in the encoded vector.

In some embodiments, the encoder neural network and the decoder neural network may be implemented based on a Recurrent Neural Network (RNN) or a self-attention technique.

In some embodiments, the aggregator module may concatenate the character embedding vector and the time signature to generate the input vector.

In some embodiments, the time signature information may be duration information of notes set for each phoneme or syllable for the input text.

In some embodiments, the output sequence is composed of data in a spectrogram form, and the speech synthesis unit may further include a vocoder unit for converting the output sequence into the target voice.

In some embodiments, the output sequence is composed of data in the form of spectrogram, and the speech synthesis unit inputs the training text and correct time signature information preprocessed by the preprocessing unit into the speech synthesis model, and is obtained as a result. The predicted spectrogram data may be compared with the correct answer spectrogram data to calculate an error value, and the speech synthesis model may be trained by back-propagation of the calculated error value.

According to another aspect of the present disclosure, there is provided a speech synthesis apparatus, which includes a preprocessor for performing preprocessing of input text, and inputs the preprocessed text and rhyme information into a neural network-based speech synthesis model. And a speech synthesizer configured to synthesize a target voice in which the rhyme information is reflected on the input text, wherein the speech synthesis model includes: an embedding module for converting the preprocessed text into a character embedding vector, and the rhyme information and the character embedding. An aggregator module that generates an input vector constituting an input sequence by aggregating a vector, an encoder neural network that encodes the input sequence to output an encoded vector, and an output associated with the target speech by decoding the encoded vector. May contain a decoder neural network that outputs a sequence have.

According to another aspect of the present disclosure, there is provided a speech synthesis apparatus, which includes a preprocessor for performing preprocessing of input text and the preprocessed text and rhyme information in a neural network-based speech synthesis model. And a speech synthesizer configured to synthesize a target voice in which the rhyme information is reflected on the input text, wherein the speech synthesis model comprises: an embedding module for converting the preprocessed text into a character embedding vector, and for the character embedding vector. An encoder neural network that encodes an input sequence and outputs an encoded vector and a decoder neural network that outputs an output sequence associated with the target speech by decoding the encoded vector using the rhyme information.

1 is a diagram illustrating an input and an output of a speech synthesis apparatus according to some embodiments of the present disclosure.
2 is an exemplary block diagram illustrating data flow in a speech synthesis apparatus and a learning process according to some embodiments of the present disclosure.
3 is an exemplary block diagram illustrating a preprocessor in accordance with some embodiments of the present disclosure.
4 is an exemplary diagram for describing an operation of a text preprocessor according to some embodiments of the present disclosure.
5 is an exemplary diagram for describing an operation of a voice analyzer according to an exemplary embodiment of the present disclosure.
6 is an exemplary block diagram illustrating a speech synthesis unit according to some embodiments of the present disclosure.
7 is a diagram for describing a neural network structure of a speech synthesis model according to some embodiments of the present disclosure.
8 is an exemplary diagram for describing an operation of an aggregator module according to some embodiments of the present disclosure.
9 is an exemplary diagram illustrating an LSTM cyclic neural network that may be used in a speech synthesis model according to some embodiments of the present disclosure.
10 is an exemplary diagram for describing a learning process of a speech synthesis model according to some embodiments of the present disclosure.
11 is an exemplary block diagram illustrating a data flow in a synthesis process with a speech synthesis device according to some embodiments of the disclosure.
12 and 13 are diagrams for describing a neural network structure of a modified speech synthesis model according to various embodiments of the present disclosure.
14 is an exemplary flowchart illustrating a speech synthesis method according to some embodiments of the present disclosure.
15 is an exemplary flowchart for further explaining a learning process for a speech synthesis model according to some embodiments of the present disclosure.
16 is an exemplary flowchart for further describing a synthesis process using a speech synthesis model, according to some embodiments of the present disclosure.
FIG. 17 is an exemplary diagram for describing a user interface (UI) indicating a speech synthesis result that may be referred to in various embodiments of the present disclosure.
FIG. 18 is a diagram for describing an example computing device that may implement a speech synthesis device according to some embodiments of the present disclosure.

Hereinafter, exemplary embodiments of the present disclosure will be described in detail with reference to the accompanying drawings. Advantages and features of the present disclosure, and methods of accomplishing the same will become apparent with reference to the embodiments described below in detail in conjunction with the accompanying drawings. However, the technical spirit of the present disclosure is not limited to the following embodiments, and may be implemented in various forms, and only the following embodiments make the technical spirit of the present disclosure complete and in the technical field to which the present disclosure belongs. It is provided to fully inform those skilled in the art the scope of the present disclosure, and the technical spirit of the present disclosure is only defined by the scope of the claims.

In adding reference numerals to the components of each drawing, it should be noted that the same reference numerals are assigned to the same components as much as possible even though they are shown in different drawings. In addition, in describing the present disclosure, when it is determined that a detailed description of a related known configuration or function may obscure the subject matter of the present disclosure, the detailed description thereof will be omitted.

Unless otherwise defined, all terms used in the present specification (including technical and scientific terms) may be used in a sense that can be commonly understood by those skilled in the art to which the present disclosure belongs. In addition, the terms defined in the commonly used dictionaries are not ideally or excessively interpreted unless they are specifically defined clearly. The terminology used herein is for the purpose of describing particular embodiments only and is not intended to be limiting of the disclosure. In this specification, the singular also includes the plural unless specifically stated otherwise in the phrase.

In addition, in describing the components of the present disclosure, terms such as first, second, A, B, (a), and (b) may be used. These terms are only for distinguishing the components from other components, and the nature, order or order of the components are not limited by the terms. If a component is described as being "connected", "coupled" or "connected" to another component, that component may be directly connected to or connected to that other component, but there may be another configuration between each component. It is to be understood that the elements may be "connected", "coupled" or "connected".

As used herein, “comprises” and / or “comprising” refers to a component, step, operation and / or element that is mentioned in the presence of one or more other components, steps, operations and / or elements. Or does not exclude additions.

Prior to the description herein, some terms used herein will be clarified.

In this specification, the prosody information may include all kinds of information related to the rhythm of the synthesized sound, such as pitch (or pitch, pitch), loudness (ie, strength and weakness) with respect to the input text. For example, the rhyme information may be expressed as tone information or tone intensity information corresponding to unit text or unit time, but the technical scope of the present disclosure is not limited thereto. The unit text may be a phoneme, a syllable, a word, or a word, but the technical scope of the present disclosure is not limited thereto. The pitch may be expressed, for example, in the form of a frequency (e.g. fundamental frequency F 0), but the technical scope of the present disclosure is not limited thereto. The sound intensity may be expressed in the form of decibels (dB), for example, but the technical scope of the present disclosure is not limited thereto.

In the present specification, the time signature information may include all kinds of information associated with the time signature of the synthesized sound, such as the duration information of the input text. For example, the time signature information may be represented by unit length text or sound length information corresponding to a unit time, but the technical scope of the present disclosure is not limited thereto. The unit text may be a phoneme, a syllable, a word, or a word, but the technical scope of the present disclosure is not limited thereto. The negative length may be expressed in the form of a single value, such as a duration, and may be expressed in the form of a range value, such as (start time, end time), but the technical scope of the present disclosure is limited thereto. no.

In the present specification, the target voice literally means a synthesized sound that is a target to be generated from a given text.

In the present specification, an instruction is a series of computer readable instructions grouped by function and refers to a component of a computer program and executed by a processor.

Hereinafter, some embodiments of the present disclosure will be described in detail with reference to the accompanying drawings.

1 is an exemplary diagram illustrating input and output of a speech synthesis apparatus 10 according to some embodiments of the present disclosure.

As shown in FIG. 1, the speech synthesis apparatus 10 receives at least one of the text 1, the rhyme information 3, and the time signature information 5, and computes a synthesized voice 7 corresponding thereto. Device. The voice 7 at this time means a target voice (ie, target synthesized sound) in which the input rhyme information 3 and / or time signature information 5 is reflected. The computing device may be a laptop, a desktop, a laptop, or the like, but is not limited thereto and may include any kind of device having a computing function. An example of the computing device further refers to FIG. 18.

Although FIG. 1 illustrates that the speech synthesis apparatus 10 is implemented as a single computing device, the first function of the speech synthesis apparatus 10 is implemented in the first computing device, and the second function is the second computing device. Of course it can also be implemented in.

According to various embodiments of the present disclosure, the speech synthesizing apparatus 10 uses a neural network-based speech synthesis model to synthesize a target speech reflecting rhyme information 3 and / or time signature information 5. . A detailed description of the neural network structure and the learning method of the speech synthesis model will be described in detail with reference to the drawings of FIG. 2.

Hereinafter, the description will be continued on the assumption that the speech synthesizing apparatus 10 synthesizes the target speech reflecting the rhyme information 3 and the time signature information 5. However, the speech synthesizing apparatus 10 may operate to reflect only the rhyme information 3 or may operate to reflect only the beat information 50.

2 is an exemplary block diagram illustrating a speech synthesis apparatus 10 according to some embodiments of the present disclosure. In particular, FIG. 2 also shows a data flow for the process in which the speech synthesis model 63 is trained.

For convenience of understanding, the operation of each component 21 to 27 in the learning process of the speech synthesis model 63 will first be described, and the operation of each component 21 to 27 in the speech synthesis process. Will be described later with reference to FIG.

As illustrated in FIG. 2, the speech synthesis apparatus 10 may include an input unit 21, a preprocessor 23, a storage unit 25, and a speech synthesis unit 27. However, only components that are related to the embodiment of the present disclosure are shown in FIG. 2. Accordingly, it will be appreciated by those skilled in the art that the present disclosure may further include other general purpose components in addition to the components shown in FIG. 2. In addition, each component of the speech synthesis apparatus 10 illustrated in FIG. 2 represents functionally divided functional elements, and it is noted that a plurality of components may be implemented in a form in which they are integrated with each other in an actual physical environment. . Hereinafter, each component is demonstrated in detail.

The input unit 21 receives a training data set including training text and correct answer voice data. The correct answer voice data is voice data corresponding to the learning text, and the voice includes a rhyme (ie, pitch and intensity) and a rhythm characteristic according to the tone of the speaker. Therefore, when machine learning the rhyme information and time signature information reflected in the correct answer voice data, it is possible to synthesize the natural voice while controlling the rhyme or time signature.

The learning text and the correct answer voice data are input to the preprocessor 23 for preprocessing.

Next, the preprocessor 23 performs preprocessing on the input learning text and the correct answer voice data. The preprocessor 23 according to some embodiments may include a text preprocessor 31, a voice analyzer 33, and a voice preprocessor 35 as shown in FIG. 3.

The text preprocessor 31 performs preprocessing on the input text. The preprocessing may be performed in various ways such as dividing the input text into sentence units, parsing the sentence text into units of words, words, letters, phonemes, and the like, and converting numbers and special characters into characters. The specific pretreatment scheme may vary depending on the embodiment. Some examples of pretreatment procedures are shown in FIG. 4.

As shown in FIG. 4, the text preprocessor 31 converts the number of the input text 41 into a character to generate a text 43 in the form of a character, and converts the text 43 into phoneme-based text 45 Can be converted to). However, this is only an example for explaining the operation of the text preprocessor 31, and the text preprocessor 31 may perform a natural language preprocessing function in various ways.

Referring back to FIG. 3, the voice analyzer 33 extracts beat information and rhyme information about the correct answer voice data through voice analysis on the correct answer voice data. The speech analyzer 33 may further receive a text for learning together with the correct answer voice data in order to increase the accuracy of extraction.

As illustrated in FIG. 3, the voice analyzer 33 may include a time signature information extractor 33-1 and a rhyme information extractor 33-2. The beat information extractor 33-1 extracts beat information from the correct answer voice data (eg wav format audio) in the audio format, and the rhyme information extractor 33-2 extracts the rhyme information from the correct answer speech data in the audio format. Extract. For example, the beat information extractor 33-1 may extract the length of a sound corresponding to each phoneme or syllable (or word, sentence, etc.) by analyzing the correct answer voice data. In addition, the rhyme information extracting unit 33-2 may analyze the correct answer voice data to extract a height or a loudness (e.g. frequency) of a sound corresponding to each phoneme or syllable (or word, sentence, etc.).

In order to extract the beat information and the rhyme information, one or more speech analysis algorithms well known in the art may be used. For example, speech analysis and annotation tools that are well known in the art may be used, such as the SPPAS tool. Currently, the SPPAS tool provides the ability to analyze the audio speech and extract the duration and pitch information in units of phonemes.

An example of extracting beat information using the SPPAS tool is shown in FIG. 5. As illustrated in FIG. 5, the SPPAS tool may analyze voice data 51 of an audio format and extract sound length information 53 for each phoneme.

Referring back to FIG. 3, the speech preprocessor 35 converts the correct answer speech data (e.g. wav format audio) in audio format into spectrogram data. For example, the speech preprocessor 35 may perform short time fourier transform (STFT) signal processing to convert speech data into STFT spectrogram data or to convert the STFT spectrogram data to mel-scale. . The spectrogram data can be used to train the speech synthesis model 63.

Data preprocessed by the preprocessor 23 (e.g. preprocessed text, rhyme information, beat information, spectrogram data, etc.) may be stored in the storage 25. The data stored in the storage 25 may be used to perform repetitive learning on the speech synthesis model 63, to build another speech synthesis model, or to reconstruct the speech synthesis model 63.

Referring back to FIG. 2, the storage unit 25 stores and manages various data such as text, rhyme information, time signature information, voice data, spectrogram data, and the like. For effective management of the data, the storage 25 may use a database.

Next, the speech synthesis unit 27 receives the preprocessed learning text and the correct answer spectrogram data, and uses the same to construct a neural network-based speech synthesis model 63. At this time, the rhyme information and the beat information are used for learning the speech synthesis model 63 in order to have pitch, sound intensity and / or time adjustment functions. For example, when a pitch adjustment function is desired, the speech synthesis model 63 may be constructed by further learning pitch information. In addition, if it is desired to have a beat control function, the speech synthesis model 63 may be constructed by further learning the beat information.

As illustrated in FIG. 6, the speech synthesizer 27 according to some embodiments may include a learner 61, a speech synthesized model 63, a synthesizer 65, and a vocoder 67. Hereinafter, the detailed components of the speech synthesis unit 27 will be described in detail.

The learner 61 trains the speech synthesis model 63 using the training data set. That is, the learner 61 may build the speech synthesis model 63 by updating the weight of the speech synthesis model 63 so that the prediction error of the speech synthesis model 63 is minimized using the training data set. The learning data set may be provided from the preprocessor 23 or the storage 25. For convenience of understanding, the neural network structure of the speech synthesis model 63 will be described first, and then the operation of the learner 61 will be described in detail.

The speech synthesis model 63 is a neural network based model that receives preprocessed text, rhyme information, and / or time signature information, and synthesizes speech corresponding thereto. As shown in FIG. 7, the speech synthesis model 63, according to some embodiments of the present disclosure, includes an embedding module 71, an aggregator module 73, an encoder neural network 75, an attention module. 77 and a decoder neural network 79.

The embedding module 71 is a module that converts the preprocessed text into a character embedding vector through an embedding technique. In this case, the embedding module 71 may generate a character embedding vector in phoneme units, or may generate a character embedding vector in units of syllables, words, and words. For example, the embedding module 71 may generate a character embedding vector using a fasttext embedding technique, an auto-encoder embedding technique, a self-attention embedding technique, or the like. The technical scope is not limited thereto.

Next, the aggregator module 75 is a module that aggregates the input information to generate an input vector constituting an input sequence for the encoder neural network 75. For example, when a character embedding vector, rhyme information, and time signature information are input, an aggregator module 75 aggregates the character embedding vector, the rhyme information, and the time signature information to generate an input vector for the encoder neural network 75. can do. In addition, the generated sequence of input vectors may be input to the encoder neural network 75.

The specific way in which the aggregating is performed may vary depending on the embodiment.

In some embodiments, the aggregator module 75 may perform aggregation in a manner that concatenates the input information. An example of this is shown in FIG. 8. In particular, FIG. 8 illustrates an example of generating an input sequence by connecting the time signature information 53 of the phoneme unit illustrated in FIG. 5 to the corresponding character embedding vectors 81 and 84. As shown in FIG. 8, the aggregator module 73 connects the first beat information 82 for the phoneme p with the first character embedding vector 81 for the phoneme p to form a first input vector. (83) and a second input vector (86) can be generated by concatenating the second time signature information (85) for the next phoneme (r) with the second character embedding vector (84) for the phoneme (r). have. The generated input vectors 83 and 86 form an input sequence of the encoder neural network 75. Rhyme information can be linked to a character embedding vector in a similar manner.

In some other embodiments, the information entered through the artificial neural network (positioning a fully connected layer in front of the e.g. encoder neural network 75) may be aggregated. Alternatively, aggregating may be performed using a self-attention or auto encoder technique.

In some other embodiments, aggregating may be performed through a linear model in addition to non-linear models such as artificial neural networks. For example, by using the linear model, the character embedding vectors e.g. 81 and 84 and the time signature information e.g. 82 and 85 are aggregated to generate an input vector (or sequence) of the encoder neural network 75. However, the technical scope of the present disclosure is not limited to the examples listed above.

Another component of the speech synthesis model 63 will be described with reference to FIG. 7 again.

The encoder neural network 75 is a neural network that receives an input sequence consisting of one or more vectors, encodes the input sequence, and outputs the encoded vector. As learning progresses, the encoder neural network 75 understands the context contained in the input sequence and outputs an encoded vector representing the understood context. The encoded vector may also be referred to in the art as the context vector.

In some embodiments, encoder neural network 75 and decoder neural network 79 may be implemented with a Recurrent Neural Network (RNN) to be suitable for inputting and outputting sequences. For example, the encoder neural network 75 and the decoder neural network 79 may be implemented with a long short-term memory model (LSTM) neural network 90 as shown in FIG. 9. However, the present invention is not limited thereto, and at least some of the encoder neural network 75 and the decoder neural network 79 may be implemented through self-attention, a transformer network, or the like. Those skilled in the art will be able to clearly understand self-attention, transformer network, so a detailed description of the technique will be omitted.

Referring again to FIG. 7, the attention module 77 provides attention information indicating which part to focus on (or which part to focus on) when learning / predicting an output sequence for an encoded vector in the decoder neural network 79. This module is provided. As learning progresses, the attention module 77 may learn the mapping relationship between the encoded vector and the output sequence and provide attention information indicating portions that should be concentrated during decoding and portions that are not. The attention information may be provided in the form of a weight vector (or weight matrix), but the technical scope of the present disclosure is not limited thereto. Those skilled in the art will clearly understand the attention mechanism, and further description thereof will be omitted.

The decoder neural network 79 receives the encoded vector and the attention information and outputs an output sequence corresponding to the encoded vector. More specifically, the decoder neural network 79 uses the encoded vector and the attention information to predict an output sequence for speech in which specific rhythm and time signature information is reflected. In this case, the output sequence may be composed of spectrogram data in a frame unit, but the technical scope of the present disclosure is not limited thereto.

When the decoder neural network 79 is implemented as a cyclic neural network, the decoder neural network 79 may further input spectrogram data of a previous frame and sequentially output spectrogram data of a current frame to configure an output sequence.

The spectrogram data is data representing a spectrogram of a voice signal, and may be STFT spectrogram data or mel-spectrogram data. However, the technical scope of the present disclosure is not limited thereto.

For reference, the reason why the decoder neural network 79 is configured to output spectrogram data instead of a speech signal is that learning by spectrogram data can more accurately calculate a prediction error than speech signals. In addition, since accurate prediction error can be calculated, a better speech synthesis model can be constructed.

So far, the neural network structure and operating principle of the speech synthesis model 63 according to some embodiments of the present disclosure have been described with reference to FIGS. 7 to 9. Hereinafter, a process of learning the speech synthesis model 63 by the learner 61 will be described with reference to FIG. 10 based on the above description.

As illustrated in FIG. 10, each of the learning data 100 may include the text for learning 101 and the correct answer voice data 102. Of course, the learning data 100 may be composed of at least one of the learning time signature information and the learning rhyme information, the learning text 101 and the correct answer spectrogram data.

The correct voice data 102 is audio data of an audio format corresponding to the text 101. Before the learning is performed, the correct answer voice data 102 is converted into correct answer spectrogram data 104 through the voice preprocessor 35, and the text 101 is subjected to appropriate preprocessing by the text preprocessor 31. Can be. In addition, as described above, the speech analyzer 33 may extract the beat information and / or the rhyme information by analyzing the correct answer voice data 102.

The process of learning the speech synthesis model 63 by the learner 61 is as follows. First, when the preprocessed text 101 is input to the embedding module 71, the preprocessed text 101 by the embedding module 71 is converted into a character embedding vector. The character embedding vector and the time signature and / or rhyme information are converted by the aggregator module 73 into a single input vector. The sequence of input vectors is input to an encoder neural network 75, and as a result, an output sequence consisting of prediction spectrogram data 103 is output from the decoder neural network 79.

The learner 61 compares the prediction spectrogram data 103 and the correct answer spectrogram data 104 to calculate a prediction error 105, and back propagates the prediction error 105 to weight the speech synthesis model 63. Update the. In this case, the weights of the encoder neural network 75, the attention module 77, and the decoder neural network 79 may be updated at once through the backpropagation. If the embedding module 71 and / or the aggregator module 73 are implemented with some layers of the neural network, the weights of the embedding module 71 and / or the aggregator module 73 may also be updated. The learning unit 61 may construct the speech synthesis model 63 by repeating the learning process for a plurality of learning data.

Meanwhile, according to some embodiments of the present disclosure, the prediction error 105 may further include an attention error 106 value. Attention error 106 is the attention information (i.e., prediction information) provided by the attention module 77 and the beat information (i.e., the correct time signature information) or the rhyme information (i.e., correct rhyme information) extracted by the speech analyzer 33 Information). For example, if learning is performed based on the beat information, the attention information should have a value similar to the beat information because the decoding will be performed with more emphasis on the longer notes. Therefore, when the weight of the speech synthesis model 63 is updated so that the difference between the attention information and the beat information (that is, the attention error 106) is minimized, a more accurate time adjustment function can be provided through the speech synthesis model 63. do. In another example, when learning is performed based on the pitch information of the rhyme information, the attention information should have a value similar to the pitch information because decoding will be performed with a higher focus on the higher pitch. Therefore, when the weight of the speech synthesis model 63 is updated so that the difference between the attention information and the pitch information (that is, the attention error 106) is minimized, a more accurate pitch control function can be provided through the speech synthesis model 63. do.

In some embodiments, a parameter may be further used to control the degree to which the above-described attention error 106 is reflected in the prediction error 105. The parameter is a kind of hyper-parameter, and may be set to adjust the degree to which the attention error 106 is reflected in the prediction error 105 before the model 63 is trained. For example, the value of the set parameter is reflected in the attention error 106 (eg multiplication, addition, etc.), the magnitude of the attention error 107 is changed, and the attention error 107 of the changed magnitude is added to the prediction error 105. May be reflected (eg addition). In some examples, learning about speech synthesis model 63 may be performed by changing the value of the parameter. For example, learning the first speech synthesis model may be performed after setting the parameter to a first value, and learning about the second speech synthesis model may be performed after setting the parameter to a second value. In addition, the speech synthesis model to be used in the actual synthesis process may be determined according to a performance evaluation result of the first speech synthesis model and the second speech synthesis model.

So far, the learner 61 and the speech synthesis model 63 have been described with reference to FIGS. 7 to 10. The other components 65 and 67 of the speech synthesizer 27 are those used in actual speech synthesis, which will be described with reference to FIG. 11.

FIG. 11 is also an exemplary block diagram illustrating a speech synthesis apparatus 11 in accordance with some embodiments of the present disclosure. FIG. 11 together illustrates a data flow in a speech synthesis process. Hereinafter, a description will be given with reference to FIGS. 6 and 11.

Once the speech synthesis model 63 is constructed, it can be provided to the speech synthesis function for the text for synthesis. Hereinafter, the operation of each component 21 to 27 in the speech synthesis process will be described.

The input unit 21 receives the synthesis information 111. The synthesis information 111 includes synthesis text, synthesis rhyme information, and synthesis beat information. Among them, the text for synthesis is input to the text preprocessor 31, and preprocessing is performed by the text preprocessor 31. The synthesizing rhyme information and the synchronizing time signature information are input to the speech synthesizing unit 27, and precisely, to the synthesizing unit 65.

The synthesizing unit 65 inputs the preprocessed synthesis text, the synthesis rhyme information, and the synthesis beat information into the speech synthesis model 63, and as a result, obtains an output sequence associated with the target speech. As described above, the output sequence may be composed of, for example, spectrogram data in units of frames. The output sequence is input to the vocoder section 67 to be converted into a target voice in audio format.

The vocoder section 67 converts the output sequence into audio data (that is, target voice) in audio format. If the conversion function can be performed, the vocoder section 67 may be implemented in any manner. For example, the vocoder section 67 may be implemented with one or more vocoder modules (e.g. WaveNet, Griffin-lim) well known in the art. In order not to obscure the subject matter of the present invention, a detailed description of the vocoder section 67 will be omitted.

The target voice output by the vocoder section 67 is a voice in which the synthesis rhyme information and the synthesis beat information are reflected.

For reference, when the rhyme of the target voice is to be changed, the voice may be synthesized by changing the rhyme information. In the same vein, if you want to change the beat of the target voice, you can change the beat information and synthesize the voice again. As such, by using the speech synthesis model 63 according to various embodiments of the present disclosure, the rhythm or time signature of the final synthesized target voice may be adjusted by adjusting the input rhyme or time signature information.

Meanwhile, it should be noted that not all components illustrated in FIGS. 2, 3, 6, and 11 may be essential components for implementing the speech synthesis apparatus 10. That is, the speech synthesis apparatus 10 according to some other embodiments of the present disclosure may be implemented by some of the components shown in FIGS. 2, 3, 6, and 11.

Each component shown in FIGS. 2, 3, 6, and 11 may refer to software or hardware such as a field programmable gate array (FPGA) or an application-specific integrated circuit (ASIC). have. However, the components are not limited to software or hardware, and may be configured to be in an addressable storage medium, or may be configured to execute one or more processors. The functions provided in the above components may be implemented by more detailed components, or may be implemented as one component that performs a specific function by combining a plurality of components.

So far, the speech synthesis apparatus 10 according to some embodiments of the present disclosure has been described with reference to FIGS. 2 to 11. As described above, since the time synthesis information and the rhyme information reflected in the actual voice data of the speaker are synthesized, a neural network-based voice synthesis model is constructed, so that the natural time can be generated while the time and rhyme control is possible. For example, the rhythm or rhythm contained in the target voice synthesized by adjusting the intensity or pitch of the sound on the rhyme information may be adjusted.

Hereinafter, a neural network structure of a modified speech synthesis model according to various embodiments of the present disclosure will be described with reference to FIGS. 12 and 13. In the following embodiments, the description of the overlapping part with the above-described speech synthesis model 63 will be omitted.

12 illustrates a neural network structure of the modified speech synthesis model 120 according to the first embodiment of the present disclosure.

As shown in FIG. 12, the components 121 to 125 of the speech synthesis model 120 according to the first embodiment are similar to the speech synthesis model 63 described above. However, the aggregator module 122 generates the input sequence based on the beat information 126 and the character embedding vector, and the rhyme information 127 is provided to the decoder neural network 125. ) Is different.

In some embodiments, when rhyme information 127 is provided to decoder neural network 125, rhyme information 127 may be transformed according to the decoding interval of decoder neural network 125. For example, when the rhyme information 127 is pitch information set in phoneme units, the pitch information may be provided to the decoder neural network 125 by being divided into frame units of spectrogram data instead of phoneme units.

In the first embodiment, the decoder neural network 125 decodes the encoded vector and the attention information and outputs an output sequence. In this case, decoding is performed using the rhyme information 127 together. The operation of the other modules 121, 123, 124 is similar to that described above.

According to the first embodiment, it may be adjusted based on rhyme information input at the time of rhythm decoding of the synthesized speech. That is, since the rhyme adjustment is performed based on the rhyme information input immediately before decoding, there is an advantage that the rhyme adjustment can be performed more precisely. The effects mentioned above may be equally applied to the time signature information.

13 illustrates a neural network structure of a modified speech synthesis model 130 according to a second embodiment of the present disclosure.

As shown in FIG. 13, the components 131 to 135 of the speech synthesis model 130 according to the second embodiment are similar to the speech synthesis model 63 or 130 described above. However, the rhyme information 137 is input to the aggregator module 123 and is also input to the decoder neural network 135, which is different from the speech synthesis model 63 described above.

As in the first embodiment, the decoder neural network 135 decodes the encoded vector and the attention information and outputs an output sequence. In this case, the decoder neural network 135 performs decoding using the rhyme information 137 together. The operation of the other modules 131, 133, 134 is similar to that described above.

When the rhyme information is input to the encoder neural network, since the rhyme adjusting function is learned along with the text information, the rhythm of the synthesized voice is adjusted more intuitively (ie, more naturally) for the user to feel. Therefore, according to the second embodiment, natural and fine rhyme control can be performed on the synthesized voice. The effects mentioned above may be equally applied to the time signature information.

So far, the modified speech synthesis models 120 and 130 according to various embodiments of the present disclosure have been described with reference to FIGS. 12 and 13. Hereinafter, a voice synthesis method according to some embodiments of the present disclosure will be described in detail with reference to FIGS. 14 to 17.

Each step of the speech synthesis method may be performed by a computing device. In other words, each step of the speech synthesis method may be implemented with one or more instructions executed by a processor of the computing device. All steps included in the speech synthesis method may be performed by one physical computing device, but the first steps of the method are performed by a first computing device and the second steps of the method are second computing devices. It may also be performed by. In the following description, it is assumed that each step of the speech synthesis method is performed by the speech synthesis apparatus 10. However, for convenience of description, the description of the operation subject of each step included in the speech synthesis method may be omitted.

14 is an exemplary flowchart illustrating a speech synthesis method according to some embodiments of the present disclosure. However, this is only a preferred embodiment for achieving the object of the present disclosure, of course, some steps may be added or deleted as necessary.

As shown in FIG. 14, the speech synthesis method includes a learning process of constructing a speech synthesis model and a synthesis process of synthesizing a speech using the speech synthesis model.

The learning process begins at step S100 of obtaining a learning dataset. In this case, each of the learning data included in the learning data set is composed of learning text and correct answer voice data.

In step S200, a neural network based speech synthesis model is constructed using the training dataset. Since the structure of the speech synthesis model has already been described above, a further description thereof will be omitted, and details of this step S200 will be described later with reference to FIG. 15.

The synthesis process starts at step S300 of acquiring synthesis data. The synthesis data is composed of synthesis text, synthesis rhyme information, and synthesis time signature information. Of course, if the rhyme of the voice to be synthesized is not adjusted, the synthesis rhyme information may be excluded from the synthesis data. Likewise, if the time signature of the voice to be synthesized is not controlled, the time signature for synthesis may be excluded from the synthesis data.

In step S400, a target speech for the synthesis text is synthesized and output using the speech synthesis model. More specifically, an output sequence consisting of spectrogram data may be output from a speech synthesis model, and the target voice may be output by vocoding the output sequence. In this case, the target voice is a voice in which the synthesis rhyme information and the synthesis time information are reflected. See FIG. 17 for an example of the synthesized target voice.

In this step S400, by adjusting the pitch or the intensity of the sound on the rhyme information, the rhyme of the synthesized target voice can be adjusted. Also, by adjusting the length of the sound on the beat information, the beat of the target voice to be synthesized can be adjusted.

For reference, among the above-described steps S100 to S400, steps S100 and S200 are performed by the input unit 21, the preprocessor 23, and the learner 61, and steps S300 and S400 are the input unit 21 and the preprocessor ( 23), by the combining section 65 and the vocoder section 67.

So far, the speech synthesis method according to some embodiments of the present disclosure has been described with reference to FIG. 14. Hereinafter, a method for constructing a speech synthesis model that can be performed in step S200 will be described in more detail with reference to FIG. 15.

15 is an exemplary flowchart illustrating a learning process of a speech synthesis model according to some embodiments of the present disclosure. However, this is only a preferred embodiment for achieving the object of the present disclosure, of course, some steps may be added or deleted as necessary.

As shown in FIG. 15, the learning process starts at step S210 of preprocessing the learning text and the correct answer voice data. The correct answer voice data may be converted into correct answer spectrogram data through preprocessing. Since the details of the preprocessing are as described above, further description thereof will be omitted.

In step S220, the time information and the rhyme information for the text for learning are extracted by analyzing the correct answer voice data.

In step S230, the text preprocessed through embedding is converted into a character embedding vector. The character embedding vector may be generated for each phoneme or for each syllable, which may vary depending on the embodiment. The embedding may be performed in the embedding module (e.g. 71 in FIG. 7) constituting the speech synthesis model (e.g. 73 in FIG. 7), but may be performed in a separate embedding module.

In step S240, the aggregator module of the speech synthesis model (eg, 73 in FIG. 7) aggregates the character embedding vector, the rhyme information, and the beat information to form an input sequence of the encoder neural network (eg 75 in FIG. 7). Is generated. For example, the input vector may be generated by connecting the rhyme information and the time signature information to a character embedding vector, and the sequence of the input vector may be an input sequence of the encoder neural network. However, the technical scope of the present disclosure is not limited thereto.

In step S250, encoding is performed on the input sequence in the encoder neural network (e. G. 75 in FIG. 7). Through this, the input sequence is converted into an encoded vector, and the encoded vector is output from the encoder neural network.

In step S240, decoding of the encoded vector is performed in the decoder neural network (e. G. 79 of FIG. 7) of the speech synthesis model. Through this, the encoded vector is converted into an output sequence consisting of prediction spectrogram data, and the output sequence is output from the decoder neural network.

In some embodiments, the decoder neural network may further receive attention information from an attention module (e. G. 77 of FIG. 7) located between the encoder neural network and the decoder neural network.

In addition, in some embodiments, the decoder neural network may receive the prediction spectrogram data of the previous frame, and may further output the prediction spectrogram data of the current frame.

In step S250, the error value between the correct answer spectrogram data and the predicted spectrogram data is backpropagated to update the weight of the speech synthesis model. In this case, weights of the encoder neural network and the decoder neural network may be updated at one time through the error backpropagation. If the embedding module is included in the speech synthesis model, the weight of the embedding module may also be updated.

In some embodiments, the error value may further include an attention error. Since the attention error is the same as described above, further description thereof will be omitted.

By performing the above-described steps S210 to S270 on the plurality of training data, a speech synthesis model can be constructed. In addition, among the above-described steps S210 to S270, step S210 is performed by the text preprocessor 31 and the voice preprocessor 35, step S220 is performed by the voice analyzer 33, and the remaining steps S230 to S270 are It may be performed by the learner 61 and the speech synthesis model 63.

So far, the method for constructing the speech synthesis model according to some embodiments of the present disclosure has been described. According to the above-described method, a speech synthesis model capable of controlling rhyme and time signature can be constructed.

Hereinafter, a speech synthesis process that may be performed in step S400 will be described in detail with reference to FIG. 16.

16 is an exemplary flowchart illustrating a speech synthesis process based on a speech synthesis model according to some embodiments of the present disclosure. However, this is only a preferred embodiment for achieving the object of the present disclosure, of course, some steps may be added or deleted as necessary.

As shown in FIG. 16, the speech synthesis method starts at step S410 of preprocessing synthesis text.

In step S420, the synthesis text preprocessed through embedding is converted into a character embedding vector. The character embedding vector may be generated for each phoneme or for each syllable, which may vary depending on the embodiment. The embedding may be performed in the embedding module (e.g. 71 in FIG. 7) constituting the speech synthesis model (e.g. 73 in FIG. 7), but may be performed in a separate embedding module.

In step S430, the aggregator module of the speech synthesis model (eg, 73 in FIG. 7) is aggregated to construct an input sequence of the encoder neural network (eg 75 in FIG. 7) by aggregating the character embedding vector, the synthesis rhyme information, and the synthesis beat information. An input vector is generated. For example, the input vector may be generated by connecting the rhyme information and the time signature information to a character embedding vector, and the sequence of the input vector may be an input sequence of the encoder neural network. However, the technical scope of the present disclosure is not limited thereto.

In step S440, encoding is performed on the input sequence in the encoder neural network (e. G. 75 in FIG. 7). Through this, the input sequence is converted into an encoded vector, and the encoded vector is output from the encoder neural network.

In step S450, decoding of the encoded vector is performed in a decoder neural network (e. G. 79 of FIG. 7) of the speech synthesis model. Through this, the encoded vector is converted into an output sequence consisting of spectrogram data in units of frames, and the output sequence is output from the decoder neural network.

In some embodiments, the decoder neural network may further receive attention information from an attention module (eg, 77 of FIG. 7) located between the encoder neural network and the decoder neural network, and perform decoding by further using the attention information. .

In addition, in some embodiments, the decoder neural network may receive spectrogram data of a previous frame and further output spectrogram data of the current frame.

In step S450, the spectrogram data in units of frames included in the output sequence is vocoded to synthesize a target voice in an audio format. At this time, the target voice is a voice in which the synthesis rhyme information and the synthesis beat information are reflected.

In some embodiments, the target voice may be provided visually via a graphical user interface (GUI). An example of such a GUI is shown in FIG. 17. The table 171 shown at the top of FIG. 17 displays text / rhyme / beat information for synthesis in syllable units, and the GUI 175 shown at the bottom of FIG. 17 is synthesized from the information for synthesis in the table 171. The selected target voice in the form of a voice waveform. In particular, the illustrated speech waveform shows actual results synthesized through a speech synthesis model constructed in accordance with the above-described embodiments.

As shown in FIG. 17, it can be confirmed that the target voice is accurately synthesized according to the synthesis rhyme information and the synthesis time information. For example, since the pitch information 173 of the syllable ("yi") is "660Hz" and the pitch information (174) of the syllable ("yi") is "663Hz", it is set to a higher pitch value than other syllables. It can be seen that the speech waveforms 177 and 178 of the syllable ("Y") and the syllable ("Rum") are synthesized with high pitch. In addition, the beat information 172 of the syllable ("Yo") is set to "0.5sec", and the length of the note is relatively longer than other syllables. It can be confirmed that the synthesis.

Among the above-described steps S410 to S460, step S410 is performed by the text preprocessor 31, steps S420 to S450 are performed by the synthesizer 65 and the speech synthesis model 63, and step S460 is performed by the vocoder unit ( 67).

So far, the speech synthesis method according to some embodiments of the present disclosure has been described with reference to FIGS. 14 to 17. Hereinafter, an exemplary computing device 180 that can implement the speech synthesis device 10 according to some embodiments of the present disclosure will be described.

18 is a hardware diagram illustrating an example computing device 180 that can implement a speech synthesis device 10 in accordance with some embodiments of the present disclosure.

As shown in FIG. 18, the computing device 180 may include a memory that loads one or more processors 181, a bus 183, a communication interface 184, and a computer program executed by the processor 181. 182 and storage 185 for storing computer programs 186. However, FIG. 18 illustrates only the components related to the embodiment of the present disclosure. Accordingly, it will be appreciated by those skilled in the art that the present disclosure may further include other general purpose components in addition to the components illustrated in FIG. 18.

The processor 181 controls the overall operation of each component of the computing device 180. The processor 181 includes a central processing unit (CPU), a micro processor unit (MPU), a micro controller unit (MCU), a graphics processing unit (GPU), or any type of processor well known in the art. Can be. In addition, the processor 181 may perform an operation on at least one application or program for executing a method according to embodiments of the present disclosure. Computing device 180 may have one or more processors.

The memory 182 stores various data, commands, and / or information. The memory 182 may load one or more programs 186 from the storage 185 to execute the speech synthesis method according to embodiments of the present disclosure. For example, if the computer program 186 is loaded into the memory 182, a module as shown in FIG. 2 may be implemented on the memory 182. The memory 182 may be implemented as a volatile memory such as a RAM, but the technical scope of the present disclosure is not limited thereto.

The bus 183 provides communication between components of the computing device 180. The bus 183 may be implemented as various types of buses such as an address bus, a data bus, and a control bus.

The communication interface 184 supports wired and wireless Internet communication of the computing device 180. In addition, the communication interface 184 may support various communication methods other than Internet communication. To this end, the communication interface 184 may comprise a communication module well known in the art of the present disclosure.

According to some embodiments, communication interface 184 may be omitted.

The storage 185 may non-temporarily store the one or more programs 186 and various data. For example, when the speech synthesis apparatus 10 is implemented through the computing device 180, the various data may include data managed by the storage unit 25.

Storage 185 is well known in the art for non-volatile memory, hard disks, removable disks, or the like to which the present disclosure pertains, such as Read Only Memory (ROM), Eraseable Programmable ROM (EPROM), Electrically Erasable Programmable ROM (EEPROM), flash memory, and the like. It may comprise any known type of computer readable recording medium.

Computer program 186 may include one or more instructions that, when loaded into memory 182, cause processor 181 to perform methods / operations in accordance with various embodiments of the present disclosure. That is, the processor 181 may perform methods / operations according to various embodiments of the present disclosure by executing the one or more instructions.

For example, the computer program 186 may be configured to acquire a training data set, to build a speech synthesis model using the training data set, to obtain synthesis data, and to perform the synthesis using the speech synthesis model. Instructions for performing an operation of synthesizing a target voice for the application data. In this case, the speech synthesis apparatus 10 according to some embodiments of the present disclosure may be implemented through the computing device 180.

So far, the exemplary computing device 180 that can implement the speech synthesis apparatus 10 according to the embodiment of the present disclosure has been described with reference to FIGS. 1 to 18.

So far, various embodiments of the present disclosure and effects according to the embodiments have been described with reference to FIGS. 1 to 18. Effects according to the technical spirit of the present disclosure are not limited to the above-mentioned effects, other effects not mentioned will be clearly understood by those skilled in the art from the following description.

The technical idea of the present disclosure described above with reference to FIGS. 1 to 18 may be implemented as computer readable codes on a computer readable medium. The computer-readable recording medium may be, for example, a removable recording medium (CD, DVD, Blu-ray disc, USB storage device, removable hard disk) or a fixed recording medium (ROM, RAM, computer equipped hard disk). Can be. The computer program recorded on the computer-readable recording medium may be transmitted to another computing device and installed in the other computing device through a network such as the Internet, thereby being used in the other computing device.

In the above description, it is described that all the elements constituting the embodiments of the present disclosure are combined or operated as one, but the technical spirit of the present disclosure is not necessarily limited to these embodiments. That is, within the scope of the present disclosure, all of the components may be selectively operated in one or more combinations.

Although the operations are shown in a specific order in the drawings, it should not be understood that the operations must be performed in the specific order or sequential order shown or that all the illustrated operations must be executed to achieve the desired results. In certain circumstances, multitasking and parallel processing may be advantageous. Moreover, the separation of the various configurations in the embodiments described above should not be understood as requiring that separation, and the described program components and systems may generally be integrated together into a single software product or packaged into multiple software products. Should be understood.

While the embodiments of the present disclosure have been described with reference to the accompanying drawings, a person of ordinary skill in the art may implement the present disclosure in other specific forms without changing the technical spirit or essential features. I can understand that there is. Therefore, it should be understood that the embodiments described above are exemplary in all respects and are not intended to be limiting. The protection scope of the present disclosure should be interpreted by the following claims, and all technical ideas within the scope equivalent thereto shall be interpreted as being included in the scope of the technical idea defined by the present disclosure.

Claims (16)

A preprocessor for preprocessing the input text; And
A speech synthesizer for inputting the preprocessed text and time signature information into a neural network-based speech synthesis model to synthesize a target voice reflecting the time signature information with respect to the input text;
The speech synthesis model,
An embedding module for converting the preprocessed text into a character embedding vector;
An aggregator module for generating an input vector constituting an input sequence by aggregating the beat information and the character embedding vector;
An encoder neural network that encodes the input sequence and outputs an encoded vector; And
A decoder neural network for decoding the encoded vector and outputting an output sequence associated with the target speech;
The speech synthesis unit,
The training text and correct time signature information preprocessed by the preprocessor are input to the speech synthesis model, and the predicted spectrogram data obtained as a result is compared with the correct answer spectrogram data to calculate an error value. Further include an attention error value to back-propagation to train the speech synthesis model,
The speech synthesis model,
An attention module located between the encoder neural network and the decoder neural network and providing attention information indicating a portion of the decoder neural network to concentrate in the encoded vector;
The attention error value is an error value between the attention information and the correct time signature information,
Speech synthesis device. delete According to claim 1,
The encoder neural network and the decoder neural network may be implemented based on a recurrent neural network (RNN) or a self-attention technique.
Speech synthesis device. According to claim 1,
The aggregator module,
Characterized in that the input vector is generated by concatenating the character embedding vector and the time signature.
Speech synthesis device. According to claim 1,
The aggregator module,
Characterized in that the input vector is generated by aggregating the character embedding vector and the time signature information using a predetermined model.
Speech synthesis device. According to claim 1,
The time signature information may be duration information set for each phoneme or syllable for the input text.
Speech synthesis device. According to claim 1,
The output sequence is composed of data in the form of a spectrogram,
The speech synthesis unit,
Further comprising a vocoder for converting the output sequence into the target voice.
Speech synthesis device. delete According to claim 1,
The weight of the encoder neural network and the weight of the decoder neural network is updated together through the backpropagation,
Speech synthesis device. According to claim 1,
The preprocessing unit,
A speech preprocessor for converting correct answer speech data in an audio format into the correct answer spectrogram data; And
Characteristic information analysis unit for extracting the correct time signature information by analyzing the correct answer voice data, characterized in that
Speech synthesis device. delete According to claim 1,
The aggregator module,
Generating the input vector by further aggregating rhyme information on the input text,
The encoder neural network,
And encoding the rhyme information included in the input vector together to further reflect the rhyme information in the target voice to generate the encoded vector.
Speech synthesis device. A preprocessor for preprocessing the input text;
A speech analyzing unit configured to extract rhyme information on the correct speech data through speech analysis on the correct speech data; And
A speech synthesizer for inputting the preprocessed text and rhyme information into a neural network-based speech synthesis model to synthesize a target voice reflecting the rhyme information with respect to the input text;
The speech synthesis model,
An embedding module for converting the preprocessed text into a character embedding vector;
An aggregator module for generating an input vector constituting an input sequence using the character embedding vector;
An encoder neural network that encodes the input sequence and outputs an encoded vector; And
A decoder neural network for decoding the encoded vector and outputting an output sequence associated with the target speech,
The decoder neural network receives rhyme information directly from the speech analyzer and decodes the encoded vector together with the encoded vector to output the output sequence.
Speech synthesis device. The method of claim 13,
The rhyme information includes pitch information set for each phoneme or syllable for the input text.
Speech synthesis device. delete The method of claim 13,
The speech synthesis model,
And an aggregator module for generating an input vector constituting the input sequence by aggregating the character embedding vector and the time signature information to further reflect time signature information.
The encoder neural network,
And encoding the beat information included in the input vector together so that the beat information is further reflected in the target voice to generate the encoded vector.
Speech synthesis device.

Images