KR20210027016A - Voice synthesis metohd and apparatus thereof - Google Patents

Abstract

Provided are a voice synthesis method and a device thereof. The method for an electronic device to synthesize a voice from a text includes the following operations of: obtaining a text inputted into the electronic device; obtaining a text representation by encoding the text; obtaining an audio representation of a first audio frame set; obtaining an audio representation of a second audio frame set based on the text representation and the audio representation of the first audio frame set; obtaining an audio feature of the second audio frame set through decoding on the audio representation of the second audio frame set; creating feedback information based on the audio feature of the second audio frame set; and synthesizing a voice based on at least one of an audio feature of the first audio frame set and the audio feature of the second audio frame set. Therefore, the present invention is capable of synthesizing a voice corresponding to an inputted text by obtaining a current audio frame based on feedback information.

Description

Voice synthesis method and apparatus {VOICE SYNTHESIS METOHD AND APPARATUS THEREOF}

The present invention relates to a method and apparatus for synthesizing speech.

Recently, various electronic devices are equipped with a text-to-speech (TTS) function to synthesize and output text into speech. In order to provide the TTS function, the electronic device may use a predetermined TTS model including a phoneme of text and voice data corresponding to the phoneme.

Recently, a speech synthesis method based on an artificial neural network (for example, a deep neural network) using an end-to-end learning method has been actively studied. It contains much more natural voice features than the method. Accordingly, it is required to design a new artificial neural network structure to reduce the amount of computation required for speech synthesis.

An object of the present disclosure is to provide a speech synthesis method and apparatus capable of synthesizing a speech corresponding to an input text by acquiring a current audio frame using feedback information including information on the energy of a previous audio frame. .

The objects of the present invention are not limited to the above-mentioned objects, and other objects and advantages of the present invention that are not mentioned can be understood by the following description, and will be more clearly understood by examples of the present invention. In addition, it will be easily understood that the objects and advantages of the present invention can be realized by the means shown in the claims and combinations thereof.

As a technical means for achieving the above-described technical problem, a first aspect of the present disclosure provides an operation of obtaining text input to an electronic device, and encoding the text using a text encoder of the electronic device, thereby providing text representation. Acquiring an audio representation of the first audio frame set from an audio encoder of the electronic device, acquiring an audio representation of the second audio frame set based on the text representation and the audio representation of the first audio frame set Operation, an operation of obtaining an audio characteristic of a second audio frame set by decoding an audio representation of the second audio frame set, an operation of generating feedback information based on the audio characteristic of the second audio frame set, and a first audio An electronic device may provide a method for synthesizing speech from text, including an operation of synthesizing speech based on at least one of an audio characteristic of a frame set or an audio characteristic of a second audio frame set.

In addition, as a technical means for achieving the above-described technical problem, a second aspect of the present disclosure is an electronic device for synthesizing speech from text, wherein the electronic device includes at least one processor, and the at least one processor is an electronic device. Obtaining text input to the device, obtaining a text representation by encoding the text, obtaining an audio representation of the first audio frame set, and obtaining a second audio frame set based on the text representation and the audio representation of the first audio frame set Obtain an audio representation of the second audio frame set, obtain an audio feature of the second audio frame set through decoding of the audio representation of the second audio frame set, generate feedback information based on the audio feature of the second audio frame set, and An electronic device for synthesizing speech from text, which synthesizes speech based on at least one of an audio characteristic of one audio frame set or an audio characteristic of a second audio frame set may be provided.

In addition, a third aspect of the present disclosure includes an operation of acquiring text input to an electronic device, an operation of encoding text to obtain a text representation, an operation of acquiring an audio representation of the first audio frame set, and text Acquiring an audio representation of the second audio frame set based on the representation and the audio representation of the first audio frame set, obtaining audio characteristics of the second audio frame set through decoding of the audio representation of the second audio frame set An operation, generating feedback information based on audio characteristics of the second audio frame set, and synthesizing speech based on at least one of the audio characteristics of the first audio frame set or the audio characteristics of the second audio frame set It is possible to provide a computer-readable recording medium in which a program for executing a method of synthesizing speech from a text to be performed on a computer is recorded.

According to the present disclosure, it is possible to provide a speech synthesis method and apparatus capable of synthesizing a speech corresponding to an input text by acquiring a current audio frame using feedback information including information on the energy of a previous audio frame. It works.

1A is a diagram conceptually illustrating a method of synthesizing speech from text by an electronic device according to some embodiments.
1B is a diagram conceptually illustrating a method of outputting an audio frame from a text in a time domain and generating feedback information from the output audio frame, according to an exemplary embodiment.
2 is a flowchart sequentially illustrating a method of synthesizing speech from text by using a speech synthesis model by an electronic device according to some embodiments.
3 is a diagram illustrating a process in which an electronic device synthesizes speech from text using a speech learning model according to some embodiments.
4 is a diagram illustrating a method of learning a speech synthesis model by an electronic device according to some embodiments.
5 is a diagram illustrating a method of generating feedback information by an electronic device according to some embodiments.
6 is a diagram illustrating a method of generating feedback information by an electronic device according to some embodiments.
7 is a diagram illustrating a method of synthesizing speech by an electronic device using a speech synthesis model including a convolutional neural network, according to some embodiments.
8 is a diagram illustrating a method of synthesizing speech by an electronic device using a speech synthesis model including an RNN neural network, according to some embodiments.
9 is a block diagram illustrating a configuration of an electronic device according to some embodiments.
10 is a block diagram showing the configuration of a server according to some embodiments.

Hereinafter, embodiments of the present disclosure will be described in detail with reference to the accompanying drawings so that those of ordinary skill in the art may easily implement the present disclosure. However, the present disclosure may be implemented in various different forms and is not limited to the embodiments described herein. In the drawings, parts irrelevant to the description are omitted in order to clearly describe the present disclosure, and similar reference numerals are attached to similar parts throughout the specification.

Throughout the specification, when a part is said to be "connected" with another part, this includes not only "directly connected" but also "electrically connected" with another element interposed therebetween. . In addition, when a part "includes" a certain component, it means that other components may be further included rather than excluding other components unless specifically stated to the contrary.

Hereinafter, the present disclosure will be described in detail with reference to the accompanying drawings.

1A is a diagram conceptually illustrating a method of synthesizing speech from text by an electronic device according to some embodiments.

1B is a diagram conceptually illustrating a method of outputting an audio frame from a text in a time domain and generating feedback information from the output audio frame, according to an exemplary embodiment.

Referring to FIG. 1A, an electronic device according to some embodiments uses a speech synthesis model including a text encoder 111, an audio encoder 113, an audio decoder 115, and a vocoder 117 from text 101. Voice 103 can be synthesized.

The text encoder 111 may obtain a text representation by encoding the input text 101.

The text representation is encoded information obtained through encoding of the input text, and may include information on a unique vector sequence corresponding to each character in the text.

The text encoder 111 obtains, for example, an embedding column for each character in the text 101 and encodes the obtained embedding column, so that a unique vector column corresponding to each character in the text 101 is It is possible to obtain a textual representation that includes information about it.

The text encoder 111 may be, for example, a module including at least one of a Convolution Neural Network (CNN), a Recurrent Neural Network (RNN), or a Long-Short Term Memory (LSTM). , Is not limited thereto.

The audio encoder 113 may obtain an audio representation of the first audio frame set.

The audio representation is encoded information obtained based on the text representation in order to synthesize the text into speech. The audio representation may be converted into an audio feature by performing decoding using the audio decoder 115.

The audio characteristic is information including a plurality of components having different spectrum distributions on a frequency, and may be information directly used for speech synthesis of the vocoder 117 to be described later. The audio feature may include, for example, information on at least one of spectrum, mel-spectrum, cepstrum, and mfccs, but is not limited thereto.

Meanwhile, in the first audio frame set (FS_1, see FIG. 1B), among audio frames generated from text representation, that is, all audio frames used for speech synthesis, audio characteristics are obtained in advance through an audio decoder 115 to be described later. The resulting audio frames (f_1, f_2, f_3, f_4, see FIG. 1B) may be included.

The audio encoder 113 may be, for example, a module including at least one of a convolutional neural network, a recurrent neural network, and an LSTM, but is not limited thereto.

The audio decoder 115 may obtain the audio representation of the second audio frame set FS_2 based on the text representation and the audio representation of the first audio frame set FS_1.

The electronic device according to some embodiments may acquire audio characteristics for each of the plurality of audio frames f_1 to f_8 in order to synthesize speech from text using the speech synthesis model.

Referring to FIG. 1B together, the electronic device may acquire audio characteristics of a plurality of audio frames f_1 to f_8 output from the audio decoder 115 in the time domain. In an embodiment, the electronic device does not acquire audio characteristics for each of all audio frames f_1 to f_8, and frame sets FS_1 and FS_2 including a preset number of audio frames among all frames f_1 to f_8 Is formed, and audio characteristics for the frame sets FS_1 and FS_2 may be obtained. The preset number may be, for example, four.

In the embodiment shown in FIG. 1B, the first audio frame set FS_1 includes a first audio frame f_1 to a fourth audio frame f_4, and the second audio frame set FS_2 is a fifth audio frame. It may include (f_5) to eighth audio frames (f_8). The second audio frame set FS_2 may include frames f_5 to f_8 succeeding the first audio frame set FS_1 in the time domain.

The electronic device extracts feature information on one of the first to fourth audio frames f_1 to f_4 included in the first audio frame set FS_1, and compresses it from at least one audio frame. Information can be extracted. In the embodiment shown in FIG. 1B, the electronic device extracts the audio characteristic information F0 of the first audio frame f_1, and compresses information from each of the first audio frame f_1 to the fourth audio frame f_4. E0, E1, E2, E3) can be extracted. Compression information (E0, E1, E2, E3) is, for example, the size of the amplitude value of the audio signal corresponding to the audio frame, the size of the root mean square (RMS) of the amplitude value of the audio signal, or audio It may include information on at least one of the magnitudes of the peak values of the signal.

In an embodiment, the electronic device includes audio characteristic information F0 and the first audio frame f_1 of the first audio frame f_1 among a plurality of audio frames f_1 to f_4 included in the first audio frame set FS_1. ) To the fourth audio frame f_4 by combining the compression information (E0, E1, E2, E3) for each of the second audio frame set (FS_2) to obtain feedback information (Feedback Information) for acquiring the audio characteristic information of the second audio frame set (FS_2). Can be generated. In the embodiment illustrated in FIG. 1B, it has been described that the electronic device acquires the audio characteristic information F0 from the first audio frame f_1 of the first audio frame set FS_1, but is not limited thereto. In another embodiment, the electronic device includes audio from any one of the second audio frame f_2 to the fourth audio frame f_4, not the first audio frame f_1 of the first audio frame set FS_1. Feature information can also be obtained.

In an embodiment of the present disclosure, the period of generating the feedback information of the electronic device may correspond to the number of audio frames acquired by the electronic device from text. In an embodiment, the feedback information generation period may be a length of a voice signal output through a preset number of audio frames. For example, in the case of outputting a 10 ms long voice signal through one audio frame, a voice signal corresponding to 40 ms may be output through four audio frames, and one feedback information is provided per 40 ms long output voice signal. Can be created. That is, the feedback information generation period may be the length of the output audio signal corresponding to the four audio frames. However, it is not limited thereto.

In an embodiment, the feedback information generation period may be determined based on speaker characteristics. For example, if a cycle for acquiring audio characteristics for four audio frames for a user having an average speech rate is determined as a feedback information generation cycle, the electronic device may generate 6 audio frames for a user with a relatively slow speech speed. The period for acquiring the audio characteristic for may be determined as the period for generating the feedback information. In the opposite example, the electronic device may determine a period for acquiring audio characteristics for two audio frames for a user having a relatively high speech speed as the feedback information generation period. In this case, the determination of the speed of speech may be made based on, for example, the measured number of phonemes per unit time. The rate of speech for each user is previously stored in a database, and the electronic device may refer to the database to determine a feedback information generation period according to the speech rate, and learn by using the determined feedback information generation period.

The electronic device may change the feedback information generation cycle based on the type of text. In an embodiment, the electronic device may identify the type of text using the preprocessor 310 (refer to FIG. 3 ). The preprocessor 310 may include, for example, a module including a grapheme to phoneme (G2P), a morpheme analyzer, etc., and by performing preprocessing using at least one of a G2P module and a morpheme analyzer, the phoneme sequence Or, you can print grapheme columns. For example, when the text is "Hello", the electronic device can separate consonants and vowels, such as "ㅇㅏㄴㄴㅕㅇㅎㅏㅅㅔㅇㅛ" through the preprocessor 310, and check the order and frequency of consonants and vowels. have. When the text is a vowel or silence, the electronic device may slowly change the cycle of generating the feedback information. For example, if the feedback information generation period is the length of the audio signal output through 4 audio frames, and the text is vowel or silence, the electronic device sets the feedback information generation period to the output audio signal corresponding to the 6 audio frames. Can be changed to length. As another example, when the type of text is a consonant sound or an unvoiced sound, the electronic device may shorten the feedback generation period. For example, when the text is a consonant sound or an unvoiced sound, the electronic device may change the feedback information generation period to the length of the output speech signal corresponding to the two audio frames.

For another example, the electronic device outputs a phoneme from the text through the preprocessor 310, converts each phoneme of the text into a phonetic symbol using a pre-stored phonetic dictionary, and pronounces the text according to the phonetic symbol. The information may be estimated, and a generation period of the feedback information may be changed based on the estimated pronunciation information.

In general, in the case of consonants or unvoiced sounds, the number of audio frames corresponding to the speech signal is small because the length of the speech signal is short, and in the case of vowels or silence, the number of audio frames corresponding to the speech signal is large due to the length of the speech signal. The electronic device of the present disclosure may improve the accuracy of obtaining attention information and thus improve speech synthesis performance by flexibly changing the feedback information generation period in response to the type of text such as consonant, vowel, silent, and unvoiced sound. In addition, when the number of audio frames for outputting a speech signal, such as consonant or unvoiced sound, is required to be relatively small, it is possible to reduce the amount of computation according to acquisition of audio feature information and feedback information from the audio frame.

The audio decoder 115 may obtain the audio representation of the second audio frames based on the audio representation of the first audio frames obtained in advance.

As described above, the electronic device according to some embodiments may reduce the amount of computation required to acquire the audio feature by acquiring the audio feature for synthesizing speech in units of a plurality of audio frames.

The audio decoder 115 may obtain audio characteristics of the second audio frames by decoding the audio representation of the second audio frames.

The audio decoder 115 may be, for example, a module including at least one of a convolutional neural network, a recurrent neural network, and an LSTM, but is not limited thereto.

The vocoder 117 may synthesize the voice 103 based on the audio characteristics obtained by the audio decoder 115.

The vocoder 117, for example, based on at least one of the audio characteristics of the first audio frames or the audio characteristics of the second audio frames acquired by the audio decoder 115, the voice 103 corresponding to the text 101 ) Can be synthesized.

The vocoder 117 may synthesize the voice 103 from the audio feature based on at least one of, for example, WaveNet, Parallel WaveNet, WaveRNN, or LPCNet. It is not limited thereto.

When the audio characteristic of the second audio frame set FS_2 is obtained, the audio encoder 113 receives, for example, the audio characteristic of the second audio frame set FS_2 from the audio decoder 115, and the text encoder ( An audio representation of the third audio frame set following the second audio frame set FS_2 may be obtained based on the text representation received from 111) and the audio characteristic of the second audio frame set FS_2.

Through the feedback loop method of the voice learning model, audio features of the first audio frames or audio features of the last audio frame set among the audio frames constituting the voice to be synthesized may be sequentially obtained.

Meanwhile, in a process of obtaining an audio representation of the second audio frame set FS_2 in order to synthesize a speech having a natural rhythm, the electronic device according to some embodiments ), instead of using the audio feature as it is, it can be converted into predetermined feedback information and used.

That is, the speech synthesis model obtains an audio characteristic for synthesizing a speech corresponding to the text by converting a text expression into an audio expression through the text encoder 111 and the audio decoder 115, and converts the obtained audio characteristic. Voice can be synthesized.

Hereinafter, a speech synthesis method of the present disclosure will be described in detail based on some embodiments in which a speech synthesis model used by an electronic device acquires and uses feedback information.

2 is a flowchart sequentially illustrating a method of synthesizing speech from text by using a speech synthesis model by an electronic device according to some embodiments.

A specific method of synthesizing speech by text using the speech synthesis model and a specific method of learning the speech synthesis model by the electronic device will be described later with reference to the embodiments of FIGS. 3 and 4.

Referring to FIG. 2, the electronic device according to some embodiments may obtain an input text in step S201.

In step S202, the electronic device may obtain a text representation by encoding the input text. In one embodiment, the electronic device encodes an embedding column for each character included in the input text using a text encoder 111 (refer to FIG. 1A), thereby providing information on a unique vector column corresponding to each character in the text. It is possible to obtain a text representation including.

In step S203, the electronic device may obtain an audio representation of the first audio frame set.

In step S204, the electronic device may obtain an audio representation of the second audio frame set based on the text representation and the audio representation of the first audio frame set.

In step S205, the electronic device acquires the audio characteristic of the second audio frame set from the audio representation information of the second audio frame set. In an embodiment, the electronic device decodes the audio representation of the second audio frame set using the audio decoder 115 (refer to FIG. 1A ), and obtains an audio characteristic of the second audio frame set through decoding. The audio characteristic is information including a plurality of components having different spectrum distributions on a frequency. The audio feature may include, for example, information on at least one of spectrum, mel-spectrum, cepstrum, and mfccs, but is not limited thereto.

In step S206, the electronic device may generate feedback information based on audio characteristics of the second audio frame set. Here, the feedback information may be information obtained from the audio characteristic of the second audio frame set in order to be used to acquire the audio characteristic of the third audio frame set following the second audio frame set.

The feedback information may include, for example, compression information on at least one audio frame included in the second audio frame set, in addition to information on audio characteristics of at least one audio frame among the second audio frame set.

The compression information on the audio frame may include information on the energy of the audio frame.

The compression information for the audio frame may include, for example, information on the total energy of the audio frame and the energy for each frequency of the audio frame. The energy of the audio frame may be a value related to the intensity of sound corresponding to the audio characteristic of the corresponding audio frame.

In an embodiment, when the audio characteristic of a specific audio frame is the 80th mel-spectrum, the corresponding audio frame M may be expressed in the form of the following equation.

In this case, the energy of the audio frame M may be obtained, for example, based on the following Mean of mel-spectrum equation.

Meanwhile, as another example, the energy of the audio frame M may be obtained based on the following Root Mean Square (RMS) of mel-spectrum equation.

Meanwhile, in another embodiment, when the audio characteristic of a specific audio frame is the 22nd order cepstrum, the corresponding audio frame C may be expressed in the form of the following equation.

In this case, in cepstrum, since the correlation between the first element and the actual sound intensity is relatively high, the energy of the audio frame C may be, for example, b ₁ , which is the first element of cepstrum.

The compression information for the audio frame is, for example, the amplitude of the audio signal corresponding to the audio frame, the root mean square (RMS) of the amplitude of the audio signal, or the peak of the audio signal ( peak) may include information on at least one of the magnitudes of the value.

The electronic device provides feedback by combining, for example, information on audio characteristics of at least one audio frame included in the second audio frame set and compression information for each of at least one audio frame included in the second audio frame set. Can generate information.

Steps S203 to S206 may be repeatedly performed for n audio frame sets. In an embodiment, the electronic device obtains the audio representation of the k-th audio frame set in step S203, and the audio representation of the k+1-th audio frame set based on the text representation and the audio representation of the k-th audio frame set in step S204. And, in step S205, audio feature information of the k+1 th audio frame set is obtained through decoding of the audio representation of the k+1 th audio frame set, and in step S206, the audio feature information of the k+1 th audio frame set Feedback information may be generated based on. Here, k is an ordinal number for a set of consecutive audio frames, and k may be 1, 2, 3, ..., n. When the value of k+1 is less than or equal to n, which is the number of all audio frame sets, the electronic device encodes feedback information of the k+1 th audio frame set using an audio encoder 314 (refer to FIG. 3), An audio representation of the k+2 th audio frame set following the k+1 th audio frame set may be obtained. That is, when the value of k+1 is less than or equal to n, which is the number of all audio frame sets, the electronic device may repeatedly perform steps S203 to S206.

A detailed method of generating the feedback information by the electronic device will be described later through the embodiments of FIGS. 5 and 6.

In step S207, the electronic device may synthesize speech based on at least one of an audio characteristic of the first audio frame set or an audio characteristic of the second audio frame set. In step S207, when the audio feature of the kth audio frame set or the audio feature of the k+1th audio frame set is acquired, speech may be synthesized, but is not limited thereto. In one embodiment, the electronic device repeats steps S203 to S206 until the audio feature of the n-th audio frame set is acquired, and then, the audio feature of the k+1-th audio frame set to the audio of the n-th audio frame set. Speech may be synthesized based on at least one of the features.

In FIG. 2, step S207 is shown to be sequentially performed after step S206, but is not limited thereto.

3 is a diagram illustrating a process in which an electronic device synthesizes speech from text using a speech learning model according to some embodiments.

Referring to FIG. 3, an electronic device according to some embodiments includes a preprocessor 310, a text encoder 311, a feedback information generator 312, an attention module 313, an audio encoder 314, and an audio decoder. Speech 303 can be synthesized from text 301 using a speech synthesis model including 315 and vocoder 317.

The preprocessing unit 310 is configured to obtain information on at least one of the utterances or meanings of the text that helps the text encoder 311 to be described later learn a pattern included in the input text 301, the text 301 ) Can be performed.

The text in natural language form may include a character string that harms the essential meaning of the text, such as typos and special characters. The preprocessor 310 may perform preprocessing on the text 301 in order to obtain information on at least one of speech or meaning of the text from the text 301 and learn a pattern included in the text.

The preprocessor 310 may include, for example, a module including a grapheme to phoneme (G2P), a morpheme analyzer, and the like, and such a module may perform preprocessing based on a preset rule or a learned model. The output of the preprocessor 310 may be, for example, a phoneme column or a grapheme column, but is not limited thereto.

The text encoder 311 may obtain a text representation by encoding the preprocessed text received from the preprocessor 310.

The audio encoder 314 may receive pre-generated feedback information from the feedback information generator 312 and encode the received feedback information to obtain an audio representation of the first audio frame set.

The attention module 313 identifies a portion of the text expression that requires attention, based on at least a part of the text expression received from the text encoder 311 and the audio expression of the first audio frame set transmitted from the audio encoder 314. It is possible to obtain attention information for doing.

As the input sequence of the speech synthesis model, a textual expression containing information about a vector sequence of a fixed size is generally used, so an attention-focusing technique for learning the mapping relationship between the input sequence and the output sequence of the speech synthesis model (attention mechanism) can be used.

In the speech synthesis model using the attention-focusing technique, the entire text input to the text encoder, that is, the text representation, may be re-referenced at each time-step at which audio features required for speech synthesis are acquired. In this case, the speech synthesis model does not refer to all parts of the text expression at the same ratio, but focuses on the parts related to the audio characteristics to be predicted at each time point, thereby increasing the efficiency and accuracy of speech synthesis.

Attention module 313, for example, based on at least a part of the text representation received from the text encoder 311 and the audio representation of the first audio frame set received from the audio encoder 314, the concentration of the text representation You can identify the parts you need. The attention module 113 may generate attention-concentration information including information on a portion of text expression that requires attention.

A detailed method of obtaining the audio representation of the second audio frame set by the electronic device based on the text representation and attention information will be described later with reference to the embodiments of FIGS. 7 and 8.

The audio decoder 315 is based on the attention information transmitted from the attention module 313 and the audio representation of the first audio frame set transmitted from the audio encoder 314, and the second audio following the first audio frame set. You can create an audio representation of a frame set.

The audio decoder 315 may obtain an audio characteristic of the second audio frame set by decoding an audio representation of the generated second audio frame set.

The vocoder 317 converts at least one of the audio characteristics of the first audio frame set or the audio characteristics of the second audio frame set received from the audio decoder 315 to synthesize the speech 303 corresponding to the text 301. I can.

Meanwhile, when the audio decoder 315 acquires the audio feature of the second audio frame set, the feedback information generator 312 may receive the audio feature of the second audio frame set from the audio decoder 315.

The feedback information generator 312 is used to obtain the audio characteristic of the third audio frame set following the second audio frame set based on the audio characteristic of the second audio frame set received from the audio decoder 315. Information can be obtained.

That is, the feedback information generation unit 312 is configured to obtain an audio characteristic of an audio frame set following the previously obtained audio frame set, based on the audio characteristic of the previously acquired audio frame set received from the audio decoder 315. Feedback information can be obtained.

Through the feedback loop method of the voice learning model, audio features of the first audio frame set to the audio features of the last audio frame set among audio frames constituting the voice to be synthesized may be sequentially obtained.

4 is a diagram illustrating a method of learning a speech synthesis model by an electronic device according to some embodiments.

In the speech synthesis model used by the electronic device according to some embodiments, for example, a predetermined text and an audio characteristic obtained from an audio signal corresponding to the predetermined text are input as learning data, and a speech corresponding to the input text is received. It can be learned through the process of synthesizing.

Referring to FIG. 4, a speech synthesis model learned by an electronic device according to some embodiments includes a preprocessor 310, a text encoder 311, a feedback information generator 312, an attention module 313, and an audio encoder. In addition to the 314, the audio decoder 315 and the vocoder 317, an audio feature extraction unit 411 for obtaining an audio feature from the target audio signal may be further included.

The audio feature extraction unit 411 may extract audio features of all audio frames constituting the input audio signal 400.

The feedback information generator 312 obtains audio features for all audio frames constituting the voice 403 from audio features for all audio frames of the audio signal 400 received from the audio feature extractor 411. It is possible to obtain necessary feedback information.

The audio encoder 314 may obtain an audio representation of all audio frames of the audio signal 400 by encoding feedback information received from the feedback information generator 312.

Meanwhile, the preprocessor 310 may perform preprocessing on the input text 401.

The text encoder 311 may obtain a text representation by encoding the preprocessed text received from the preprocessor 310.

The attention module 313 is based on the text expression received from the text encoder 311 and the audio expression for the entire audio frame of the audio signal 400 received from the audio encoder 314, and the portion of the text expression that needs to be focused It is possible to obtain attention information to identify the.

The audio decoder 315 generates the speech 403 based on the attention information received from the attention module 313 and the audio representation for all audio frames of the audio signal 400 received from the audio encoder 314. It is possible to obtain an audio representation for all of the constituting audio frames.

The audio decoder 315 may decode an audio representation of all audio frames constituting the voice 403 to obtain audio characteristics for all audio frames constituting the voice 403.

The vocoder 317 may synthesize the voice 403 corresponding to the text 401 based on audio characteristics of all audio frames constituting the voice 403 transmitted from the audio decoder 315.

The electronic device compares the audio characteristics of the audio frame constituting the synthesized speech 403 and the audio characteristics of the entire audio frame of the audio signal 400 with each other, and a weighting parameter that minimizes loss between both audio characteristics. The speech synthesis model can be trained by acquiring variables.

5 is a diagram illustrating a method of generating feedback information by an electronic device according to some embodiments.

The speech synthesis model used by the electronic device according to some embodiments may include a feedback information generator for obtaining feedback information from an audio feature.

The feedback information generator may generate feedback information used to acquire audio characteristics of the second audio frame set following the first audio frame set based on the audio characteristics of the first audio frame set obtained from the audio decoder.

The feedback information generator may, for example, obtain information on audio characteristics of at least one audio frame among the first audio frame set and at the same time obtain compression information on at least one audio frame among the first audio frame set. I can.

The feedback information generator may generate feedback information by combining, for example, information on audio characteristics of at least one audio frame obtained and compression information on at least one audio frame.

Referring to FIG. 5, the feedback information generator of the speech synthesis model according to some embodiments may extract information necessary for generating feedback information from the audio feature 511 (513 ).

The feedback information generation unit, for example, includes information on audio characteristics corresponding to each of the first to fourth audio frames 521, 522, 523, and 524 included in the first audio frame set 520 (F ₀ , F ₁ , F ₂ , F ₃ ), information necessary for generating feedback information can be extracted.

For example, the feedback information generator includes information on audio characteristics corresponding to each of the first to fourth audio frames 521, 522, 523, and 524 included in the first audio frame set 520 (F ₀ , F ₁ , F ₂ , F ₃ ) compression information for each audio frame (E ₀ , E ₁ , E ₂ , E ₃ ) can be obtained.

The compression information is, for example, the magnitude of the amplitude value of the audio signal corresponding to each of the first to fourth audio frames 521, 522, 523, and 524, the magnitude of the root mean square of the amplitude value of the audio signal, or It may include information on at least one of the magnitudes of the peak values of the audio signal.

The feedback information generation unit extracts at least one of _{information (F 0} , F ₁ , F ₂ , F ₃ ) on audio characteristics corresponding to each of the first to fourth audio frames 521, 522, 523, and 524 ( Feedback information 517 may be generated by combining the information 513).

The feedback information generator includes, for example, information F ₀ on audio characteristics corresponding to the first audio frame 521 and the first audio frame 521, the second audio frame 522, and the third audio frame ( Feedback information may be generated by combining _{523) and compression information E 0} , E ₁ , E ₂ , and E ₃ for each of the fourth audio frame 524. However, the present invention is not limited thereto, and in another embodiment, the feedback information generation unit provides audio characteristic information from any one of the second audio frame 522, the third audio frame 523, and the fourth audio frame 524. You can also get (F _{0 ).}

6 is a diagram illustrating a method of generating feedback information by an electronic device according to some embodiments.

Referring to FIG. 6, a feedback information generator of a speech synthesis model used by an electronic device according to some embodiments may extract information necessary for generating feedback information from an audio feature 611 (613 ).

For example, the feedback information generator includes information on audio characteristics corresponding to each of the first to fourth audio frames 621, 622, 623, and 624 included in the first audio frame set 620 (F ₀ , F ₁ , F ₂ , F ₃ ), information necessary for generating feedback information can be extracted.

The feedback information generation unit compresses each audio frame from _{information (F 1} , F ₂ , F ₃ ) on audio characteristics corresponding to each of the second audio frame 622 to the fourth audio frame 624. Information (E ₁ , E ₂ , E ₃ ) can be obtained.

The compression information is, for example, the magnitude of the amplitude value of the audio signal corresponding to each of the second audio frame 622 to the fourth audio frame 624, the magnitude of the root mean of the amplitude value of the audio signal, or of the audio signal. It may include information on at least one of the magnitudes of the peak values.

_{The feedback information generator combines at least one of the information (F 0} , F ₁ , F ₂ , F ₃ ) on the audio characteristic corresponding to the first audio frame set 620 and the extracted information 613 to provide feedback information 617. ) Can be created (615).

The feedback information generator may include, for example, information F ₀ on audio characteristics corresponding to the first audio frame 621 and compression information E on the second audio frame 622 to the fourth audio frame 624. ₁ , E ₂ , E ₃ ) can be combined to generate feedback information. However, the present invention is not limited thereto, and in another embodiment, the feedback information generation unit provides audio characteristic information from any one of the second audio frame 622, the third audio frame 623, and the fourth audio frame 624. You can also get (F _{0 ).}

When the feedback information obtained in the embodiments of FIGS. 6 and 5 are compared with each other, the feedback information obtained in the embodiment of FIG. 6 does not include _{compression information E 0 for the first audio frame 521.} Able to know.

That is, the speech synthesis model used by the electronic device of the present disclosure may generate feedback information by extracting and combining compressed information in a free manner from information on audio characteristics of the first audio frame sets 520 and 620.

7 is a diagram illustrating a method of synthesizing speech by an electronic device using a speech synthesis model including a convolutional neural network, according to some embodiments.

Referring to FIG. 7, an electronic device according to some embodiments includes a text encoder 711, a feedback information generator 712, an attention module 713, an audio encoder 714, an audio decoder 715, and a vocoder 717. A speech synthesis model including) can be used to synthesize speech from text.

The text encoder 711 may obtain a text representation K and a text representation V by encoding the input text L.

The textual expression K may be a textual expression used for generating attention information A used to determine which part of the textual expression is related to the audio expression Q, which will be described later.

Meanwhile, the text expression V may be a text expression used to obtain the audio expression R by identifying a portion requiring concentration of the text expression V based on the attention information A.

The text encoder 711 may include, for example, an embedding module and a 1D non-causal convolution layer for obtaining an embedding column for each character included in the text L.

Since the text encoder 711 can acquire information on the context of both the preceding and following characters for a predetermined character included in the text, a one-dimensional non-causal convolution layer may be used.

When an embedding column for each character included in the text L is obtained through the embedding module of the text encoder 711, and the obtained embedding column is input to the one-dimensional noncausal convolution layer, the same convolution operation result for the embedding column is obtained. Text representation K and text representation V can be output.

The feedback information generation unit 712 includes, for example, from the audio characteristics of the four first audio frame sets 720 obtained in advance through the audio decoder 715. Feedback information F1 used to acquire audio characteristics of the second audio frame set including four audio frames following the audio frames 721, 722, 723, and 724 may be generated.

For example, when the feedback information F1 to be generated corresponds to the initial feedback information for the start of the feedback loop, the feedback information generation unit 712 may include four audio frames 721, 722, 723, having a value of 0. Feedback information F1 used to acquire the audio characteristics of the four second audio frame sets following the four audio frames 721, 722, 723, and 724 may be generated from the audio characteristics of 724).

Feedback information F1 according to some embodiments, as described through the embodiment of FIG. 5, information F ₀ on the audio characteristic corresponding to the first audio frame 721 and the first audio frame 721 to the first audio frame 721 to the first audio frame 721 It may be generated by combining the _{compression information (E 0} , E ₁ , E ₂ , E ₃ ) for each of the 4 audio frames 724.

The audio encoder 714 may obtain the audio expression Q1 corresponding to the four audio frames 721, 722, 723 and 724 based on the feedback information F1 transmitted from the feedback information generator 712.

The audio encoder 714 may include, for example, a 1D causal convolution layer. Since the output of the audio decoder 715 can be fed back to the input of the audio encoder 714 during the speech synthesis process, the audio decoder 714 is A causal convolution layer can be used.

That is, the audio encoder 714, for example, with respect to an audio frame set that temporally precedes the feedback information F1 and four audio frames 721, 722, 723, 724 received from the feedback information generator 712. As a result of a convolution operation based on the generated feedback information (eg, F0), an audio expression Q1 corresponding to the four audio frames 721, 722, 723, and 724 may be obtained.

The attention module 713 focuses on the text expression V based on the text expression K transmitted from the text encoder 711 and the audio expression Q1 corresponding to the first audio frame set 720 transmitted from the audio encoder 714. Attention information A1 for identifying this necessary part can be obtained.

Attention module 713, for example, a matrix product between the text representation K received from the text encoder 711 and the audio representation Q1 corresponding to the first audio frame set 720 received from the audio encoder 714. Attention information A1 can be obtained by calculating.

Attention module 713, for example, in the process of obtaining the attention information A1, attention information generated for an audio frame set that temporally precedes the four audio frames 721, 722, 723, 724 You can refer to A0.

The attention module 713 may obtain the audio expression R1 by identifying a portion requiring attention from the text expression V based on the obtained attention information A1.

Attention module 713, for example, by obtaining a weight from the attention information A1, and calculating a weighted sum (weighted sum) between the attention information A1 and the text expression V based on the obtained weight, the audio expression R1 Can be obtained.

The attention module 713 may obtain the audio expression R1' by concatenating the audio expression R1 and the audio expression Q1 corresponding to the first audio frame set 720 with each other.

The audio decoder 715 may decode the audio expression R1' received from the attention module 713 to obtain an audio characteristic of the second audio frame set.

The audio decoder 715 may include, for example, a one-dimensional causal convolution layer. Since the output of the audio decoder 715 can be fed back to the input of the audio encoder 714 during the speech synthesis process, the audio decoder 715 is a one-dimensional information for not using information on a subsequent audio frame, that is, future information. A causal convolution layer can be used.

That is, the audio decoder 715 is, for example, the audio representation R1 and the audio representation Q1, the audio representation that was generated for the set of audio frames that temporally precede the four audio frames 721, 722, 723, 724 ( For example, as a result of a convolution operation based on the audio expression R0 and the audio expression Q0), the audio characteristics of the second audio frame set following the four audio frames 721, 722, 723, and 724 may be obtained.

The vocoder 717 may synthesize speech based on at least one of an audio characteristic of the first audio frame set 720 or an audio characteristic of the second audio frame set.

Meanwhile, when the audio characteristic of the second audio frame set is obtained, the audio decoder 715 may transmit the obtained audio characteristic of the second audio frame set to the feedback information generator 712.

The feedback information generator 712 may generate feedback information F2 used to acquire audio characteristics of a third audio frame following the second audio frame set based on the audio characteristics of the second audio frame set. The feedback information generation unit 712 is, for example, feedback information used to obtain an audio characteristic of a third audio frame following the second audio frame set based on the same method as the method of generating the above-described feedback information F1. Can generate F2.

The feedback information generator 712 may transmit the generated feedback information F2 to the audio encoder 714.

The audio encoder 714 may obtain the audio expression Q2 corresponding to the four second audio frames based on the feedback information F2 received from the feedback information generator 712.

The audio encoder 714, for example, feedback information F2 received from the feedback information generator 712 and feedback information generated for an audio frame set temporally preceding the four audio frames (eg, F0, As a result of the convolution operation based on at least one of F1), an audio expression Q2 corresponding to the four second audio frame sets may be obtained.

The attention module 713 is based on the text expression K transmitted from the text encoder 711 and the audio expression Q2 corresponding to the second audio frame set transmitted from the audio encoder 714, the portion of the text expression V that needs attention. Attention information A2 for identifying

Attention module 713, for example, calculates a matrix product between the text expression K received from the text encoder 711 and the audio expression Q2 corresponding to the second audio frame set received from the audio encoder 714 to pay attention. Concentration information A1 can be obtained.

Attention module 713, for example, in the process of obtaining the attention information A2, attention information generated for the audio frame set that temporally precedes the four second audio frame sets (e.g., attention You can refer to concentrated information A1).

The attention module 713 may obtain the audio expression R2 by identifying a portion requiring attention from the text expression V based on the obtained attention information A2.

Attention module 713, for example, can obtain the audio expression R2 by obtaining a weight from the attention information A2, and calculating a weighted sum between the attention information A2 and the text expression V based on the obtained weight. have.

The attention module 713 may obtain the audio expression R2' by combining the audio expression R2 and the audio expression Q2 corresponding to the second audio frame set with each other.

The audio decoder 715 may decode the audio expression R2' received from the attention module 713 to obtain an audio characteristic of the second audio frame set.

The audio decoder 715 is, for example, an audio representation (e.g., at least one of audio representations R0 and R1) that was generated for the audio representation R2 and the audio representation Q2 and an audio frame set that temporally precedes the four audio frames. , As a result of a convolution operation based on at least one of audio expressions Q0 and Q1), an audio characteristic of a third audio frame set following the second audio frame set may be obtained.

The vocoder 717 may synthesize speech based on at least one of an audio characteristic of the first audio frame set 720 to an audio characteristic of the third audio frame set.

The electronic device according to some embodiments may use the feedback loop used to acquire the audio characteristics of the first audio frame set 720 to the audio characteristics of the third audio frame set described above, and the characteristics of the audio frame set corresponding to the text L. You can do it over and over again until all are acquired.

The electronic device, for example, if it is determined that the attention information A generated by the attention module 713 corresponds to the last column among the embedding columns for each character included in the text L, It is determined that all the features of the audio frame set have been acquired, and repetition of the feedback loop may be terminated.

8 is a diagram illustrating a method of synthesizing speech by an electronic device using a speech synthesis model including an RNN neural network, according to some embodiments.

Referring to FIG. 8, an electronic device according to some embodiments uses a speech synthesis model including a text encoder 811, an attention module 813, an audio decoder 815, and a vocoder 817. It can be synthesized.

The text encoder 811 may obtain a text representation by encoding the input text.

The text encoder 811 may include, for example, an embedding module for obtaining an embedding column for each character included in text, a pre-net module for converting the embedding column into a text representation, and a CBHG module.

When an embedding row for each character included in the text is obtained through the embedding module of the text encoder 811, the obtained embedding row may be converted into a text representation in the pre-net module and the CBHG module.

The attention module 813 identifies a portion of the text expression that needs attention based on the text expression received from the text encoder 811 and the audio expression corresponding to the first audio frame set received from the audio decoder 815. Attention information can be obtained for.

For example, when the feedback information to be generated corresponds to the first feedback information for the start of the feedback loop, the feedback information generation unit 812 generates a start audio frame having a value of 0 as a first audio frame. As a result, feedback information used to acquire the audio feature of the second audio feature frame set may be generated.

When the feedback information is generated by the feedback information generator 812, the audio decoder 815 uses the Pre-net module and the Attention RNN module to encode the audio characteristics of the first audio frame to express the audio representation of the first audio frame. Can be obtained.

When the audio expression of the first audio frame is obtained, the attention module 813 may generate attention information based on the text expression to which the previous attention information is applied and the audio expression of the first audio frame. The attention module 813 may acquire the audio expression of the second audio frame set 820 by using the text expression and the generated attention information.

The audio decoder 815 may acquire audio characteristics of the second audio frame set 820 from the audio representation of the first audio frame and the audio representation of the second audio frame set 820 by using the Decoder RNN module. The vocoder 817 may synthesize speech based on at least one of an audio characteristic of the first audio frame set or an audio characteristic of the second audio frame set 820.

Meanwhile, when the audio characteristic of the second audio frame set 820 is obtained, the audio decoder 815 may transmit the audio characteristic of the obtained second audio frame set 820 to the feedback information generator 812.

The second audio frame set 820 may include a first audio frame 821 to a third audio frame 823. _{Feedback information according to some embodiments includes information F 0} on audio characteristics corresponding to the first audio frame 821 _{and compression information E 1 on} the second audio frame 822 and the third audio frame 823. , E ₂ ) can be generated by combining. In FIG. 8, the second audio frame set 820 is shown to include a total of three audio frames 821, 822, and 823, but this is only an example for convenience of description, and the number of audio frames is as shown. It is not limited. For example, the second audio frame 820 may include one, two, or four or more audio frames.

The feedback information generator 812 may deliver the generated feedback information to the audio decoder 815.

The audio decoder 815 receiving the feedback information encodes the audio characteristic of the second audio frame set 820 using the Pre-net module and the Attention RNN module based on the received feedback information and the previous feedback information. An audio representation of the audio frame set 820 may be obtained.

When the audio expression of the second audio frame set 820 is obtained, the attention module 813 generates attention information based on the text expression to which the previous attention information is applied and the audio expression of the second audio frame set 820 can do. The attention module 813 may acquire the audio expression of the third audio frame set 820 by using the text expression and the generated attention information.

The audio decoder 815 may acquire audio characteristics of the third audio frame from the audio representation of the second audio frame set 820 and the audio representation of the third audio frame set by using the Decoder RNN module.

The vocoder 817 may synthesize speech based on at least one of an audio characteristic of the first audio frame set to an audio characteristic of the third audio frame set.

The electronic device according to some embodiments repeats the feedback loop used to acquire the features of the first audio frame set to the third audio frame set described above until all the features of the audio frame set corresponding to the text are acquired. You can do it.

The electronic device, for example, if it is determined that the attention information generated by the attention module 813 corresponds to the last column among the embedding columns for each character included in the text, the audio frame set corresponding to the input text It is determined that all of the features of are acquired, and the iteration of the feedback loop may be terminated.

However, the present invention is not limited thereto, and the electronic device may terminate the repetition of the feedback loop by using a separate neural network model previously learned about the repetition time of the feedback loop. In an embodiment, the electronic device may terminate the repetition of the feedback loop using a separate neural network model trained to perform stop token prediction.

9 is a block diagram illustrating a configuration of an electronic device according to some embodiments.

Referring to FIG. 9, an electronic device 1000 according to some embodiments includes a processor 1001, a user input unit 1002, a communication unit 1003, a memory 1004, a microphone 1005, a speaker 1006, and a display. 1007).

The user input unit 1002 may receive text used for speech synthesis.

The user input unit 1002 includes, for example, a key pad, a dome switch, and a touch pad (contact type capacitive type, pressure type resistive film type, infrared detection method, etc.) in the user input unit 1002. A surface ultrasonic conduction method, an integral tension measurement method, a piezo effect method, etc.), a jog wheel, a jog switch, etc. may be provided, but are not limited thereto.

The communication unit 1003 may include one or more communication modules for communication with the server 2000. For example, the communication unit 1003 may include at least one of a short-range communication unit and a mobile communication unit.

The short-range wireless communication unit includes a Bluetooth communication unit, a Bluetooth Low Energy (BLE) communication unit, a Near Field Communication unit, a WLAN (Wi-Fi) communication unit, a Zigbee communication unit, and an infrared (IrDA) communication unit. Data Association) communication unit, WFD (Wi-Fi Direct) communication unit, UWB (ultra wideband) communication unit, Ant+ communication unit, etc. may be included, but is not limited thereto.

The mobile communication unit transmits and receives a radio signal with at least one of a base station, an external terminal, and a server on a mobile communication network. Here, the wireless signal may include a voice call signal, a video call signal, or various types of data according to transmission and reception of text/multimedia messages.

The memory 1004 may store a speech synthesis model used to synthesize text into speech.

The speech synthesis model stored in the memory 1004 may include a plurality of modules classified according to functions. The speech synthesis model stored in the memory 1004 is, for example, at least one module of a preprocessor, a text encoder, an attention module, an audio encoder, an audio decoder, a feedback information generator, an audio decoder, a vocoder, or an audio feature extractor. It may include.

The memory 1004 may store, for example, a program for controlling the operation of the electronic device 1000. The memory 1004 may include at least one instruction for controlling the operation of the electronic device 1000.

The memory 1004 may store, for example, information about an input text and a synthesized voice.

The memory 1004 is, for example, a flash memory type, a hard disk type, a multimedia card micro type, a card type memory (eg, SD or XD Memory, etc.), RAM (RAM, Random Access Memory), SRAM (Static Random Access Memory), ROM (ROM, Read-Only Memory), EEPROM (Electrically Erasable Programmable Read-Only Memory), PROM (Programmable Read-Only Memory), The storage medium may include at least one type of magnetic memory, magnetic disk, and optical disk, but is not limited thereto.

The microphone 1005 may receive a user's voice. When the user speaks, the voice input through the microphone 1005 may be converted into, for example, an audio signal used for learning a voice synthesis model stored in the memory 1004.

The speaker 1006 may output voice synthesized from text as sound.

The speaker 1006 may output a signal related to a function (eg, a call signal reception sound, a message reception sound, and a notification sound) performed by the electronic device 1000 as sound.

The display 1007 may display and output information processed by the electronic device 1000.

The display 1007 may display an interface for displaying text and speech synthesis results used for speech synthesis, for example.

The display 1007 may display, for example, an interface for controlling the electronic device 1000, an interface for displaying the status of the electronic device 1000, and the like.

The processor 1001 may generally control the overall operation of the electronic device 1000. For example, the processor 1001 executes programs stored in the memory 1004, so that the user input unit 1002, the communication unit 1003, the memory 1004, the microphone 1005, the speaker 1006, and the display 1007 ) Overall control.

As text is input, the processor 1001 may activate a speech synthesis model stored in the memory 1004 to start a speech synthesis process.

The processor 1001 may obtain a text representation by encoding text through a text encoder of a speech synthesis model.

The processor 1001 receives feedback information used to obtain an audio characteristic of a second audio frame set from an audio characteristic of the first audio frame set among audio frames generated from a text representation through a feedback information generator of the speech synthesis model. Can be generated. The second audio frame set may be, for example, an audio frame set including frames following the first audio frame set.

The feedback information may include, for example, information on audio characteristics of at least one audio frame included in the first audio frame set and compression information on at least one audio frame included in the first audio frame set. .

The processor 1001 includes, for example, information on audio characteristics of at least one audio frame included in the first audio frame set and at least one included in the first audio frame set through a feedback information generation unit of the speech synthesis model. Compressed information on the audio frame of is obtained, and feedback information may be generated by combining information on audio characteristics of the obtained at least one audio frame and compression information on the at least one audio frame.

The processor 1001 may generate an audio representation of the second audio frame set based on the text representation and feedback information.

The processor 1001, for example, through the attention module of the speech synthesis model, based on the text expression and the audio expression of the first audio frame set, provides attention information for identifying a portion of the text expression that requires attention. Can be obtained.

Processor 1001, for example, through the attention module of the speech synthesis model, based on the attention information to identify and extract a portion of the text expression that needs to be focused, and extract the result and the audio of the first audio frame set By combining the expressions, an audio expression of the second audio frame set may be obtained.

The processor 1001 may obtain audio characteristics of the second audio frame set by decoding the audio representation of the second audio frame set, for example, through an audio decoder of the speech synthesis model.

The processor 1001 may synthesize speech based on at least one of an audio characteristic of a first audio frame set or an audio characteristic of a second audio frame set, for example, through a vocoder of the speech synthesis model.

The processor 1001 according to some embodiments may, for example, perform an artificial intelligence operation. The processor 1001 is, for example, any one of a central processing unit (CPU), a graphics processing unit (GPU), a neural processing unit (NPU), a field programmable gate array (FPGA), and an application specific integrated circuit (ASIC). However, it is not limited thereto.

10 is a block diagram showing the configuration of a server according to some embodiments.

The speech synthesis method according to some embodiments may be performed by the electronic device 1000 and/or the server 2000 connected to the electronic device 1000 through wired or wireless communication.

Referring to FIG. 10, the server 2000 according to some embodiments may include a processor 2001, a communication unit 2002, and a memory 2003.

The communication unit 2002 may include one or more communication modules for communication with the electronic device 1000. For example, the communication unit 2002 may include at least one of a short-range communication unit and a mobile communication unit.

The short-range wireless communication unit includes a Bluetooth communication unit, a Bluetooth Low Energy (BLE) communication unit, a Near Field Communication unit, a WLAN (Wi-Fi) communication unit, a Zigbee communication unit, and an infrared (IrDA) communication unit. Data Association) communication unit, WFD (Wi-Fi Direct) communication unit, UWB (ultra wideband) communication unit, Ant+ communication unit, etc. may be included, but is not limited thereto.

The mobile communication unit transmits and receives a radio signal with at least one of a base station, an external terminal, and a server on a mobile communication network. Here, the wireless signal may include a voice call signal, a video call signal, or various types of data according to transmission and reception of text/multimedia messages.

The memory 2003 may store a speech synthesis model used to synthesize text into speech.

The speech synthesis model stored in the memory 2003 may include a plurality of modules classified according to functions. The speech synthesis model stored in the memory 2003 is, for example, at least one of a preprocessor, a text encoder, an attention module, an audio encoder, an audio decoder, a feedback information generation unit, an audio decoder, a vocoder, or an audio feature extraction unit. It may include.

The memory 2003 may store a program for controlling the operation of the server 2000. The memory 2003 may include at least one instruction for controlling the operation of the server 2000.

The memory 2003 may store, for example, information about an input text and a synthesized voice.

The memory 2003 is, for example, a flash memory type, a hard disk type, a multimedia card micro type, and a card type memory (for example, SD or XD Memory, etc.), RAM (RAM, Random Access Memory), SRAM (Static Random Access Memory), ROM (ROM, Read-Only Memory), EEPROM (Electrically Erasable Programmable Read-Only Memory), PROM (Programmable Read-Only Memory), The storage medium may include at least one type of magnetic memory, magnetic disk, and optical disk, but is not limited thereto.

The processor 2001 may generally control the overall operation of the server 2000. For example, the processor 2001 may overall control the communication unit 2002 and the memory 2003 by executing programs stored in the memory 2003.

The processor 2001 may receive text for speech synthesis from the electronic device 1000 through the communication unit 2002.

As the text is received, the processor 2001 may start the speech synthesis process by activating the speech synthesis model stored in the memory 2003.

The processor 2001 may obtain a text representation by encoding text through a text encoder of a speech synthesis model.

The processor 2001 receives feedback information used to obtain the audio characteristic of the second audio frame set from the audio characteristic of the first audio frame set among the audio frames generated from the text expression through the feedback information generator of the speech synthesis model. Can be generated. The second audio frame set may be, for example, an audio frame set including frames following the first audio frame set.

The feedback information may include, for example, information on audio characteristics of at least one audio frame included in the first audio frame set and compression information on at least one audio frame included in the first audio frame set. .

The processor 2001 includes, for example, information on audio characteristics of at least one audio frame included in the first audio frame set and at least one included in the first audio frame set through a feedback information generator of the speech synthesis model. Compressed information on the audio frame of is obtained, and feedback information may be generated by combining information on audio characteristics of the obtained at least one audio frame and compression information on the at least one audio frame.

The processor 2001 may generate an audio representation of the second audio frame set based on the text representation and feedback information.

The processor 2001, for example, through the attention module of the speech synthesis model, based on the text expression and the audio expression of the first audio frame set, provides attention information for identifying a portion of the text expression that requires attention. Can be obtained.

Processor 2001, for example, through the attention module of the speech synthesis model, based on the attention information to identify and extract a portion of the text expression that needs to be focused, and extract the result and the audio of the first audio frame set By combining the expressions, an audio expression of the second audio frame set may be obtained.

The processor 2001 may obtain audio characteristics of the second audio frame set by decoding the audio representation of the second audio frame set, for example, through an audio decoder of the speech synthesis model.

The processor 2001 may synthesize speech based on at least one of an audio characteristic of a first audio frame set or an audio characteristic of a second audio frame set, for example, through a vocoder of the speech synthesis model.

The processor 2001 according to some embodiments may, for example, perform an artificial intelligence operation. The processor 2001 is, for example, any one of a central processing unit (CPU), a graphics processing unit (GPU), a neural processing unit (NPU), a field programmable gate array (FPGA), and an application specific integrated circuit (ASIC). However, it is not limited thereto.

Some embodiments may also be implemented in the form of a recording medium including instructions executable by a computer, such as a program module executed by a computer. Computer-readable media may be any available media that can be accessed by a computer, and includes both volatile and nonvolatile media, removable and non-removable media. Further, the computer-readable medium may include a computer storage medium. Computer storage media includes both volatile and nonvolatile, removable and non-removable media implemented in any method or technology for storage of information such as computer readable instructions, data structures, program modules or other data.

In addition, in this specification, the "unit" may be a hardware component such as a processor or a circuit, and/or a software component executed by a hardware configuration such as a processor.

The above description of the present disclosure is for illustrative purposes only, and those of ordinary skill in the technical field to which the present disclosure pertains will be able to understand that it is possible to easily transform it into other specific forms without changing the technical spirit or essential features of the present disclosure. will be. Therefore, it should be understood that the embodiments described above are illustrative and non-limiting in all respects. For example, each component described as a single type may be implemented in a distributed manner, and similarly, components described as being distributed may also be implemented in a combined form.

The scope of the present disclosure is indicated by the claims to be described later rather than the detailed description, and all changes or modified forms derived from the meaning and scope of the claims and their equivalent concepts should be interpreted as being included in the scope of the present disclosure. do.

Claims (13)

In the method for the electronic device to synthesize speech from text,
Acquiring text input to the electronic device;
Encoding the text using a text encoder of the electronic device to obtain a text representation;
Obtaining an audio representation of a first audio frame set from an audio encoder of the electronic device;
Obtaining an audio representation of a second audio frame set based on the text representation and the audio representation of the first audio frame set;
Acquiring an audio characteristic of a second audio frame set by decoding an audio representation of the second audio frame set;
Generating feedback information based on audio characteristics of the second audio frame set; And
Synthesizing speech based on at least one of an audio characteristic of the first audio frame set or an audio characteristic of the second audio frame set;
Including, the method.
The method of claim 1,
The second audio frame set is
And at least one frame succeeding the first set of audio frames.
The method of claim 1,
The operation of generating the feedback information
Obtaining information on audio characteristics of at least one audio frame among the second audio frame set;
Obtaining compression information on at least one audio frame among the second audio frame set; And
Generating the feedback information by combining information on audio characteristics of the at least one audio frame and compression information on the at least one audio frame;
Including, the method.
The method of claim 3,
The above feedback information is
Used to obtain an audio characteristic of a third audio frame set that follows the second audio frame set.
The method of claim 3,
The compression information is
At least one of a magnitude of an amplitude value of an audio signal corresponding to each of the at least one audio frame, a magnitude of a root mean square (RMS) of an amplitude value of an audio signal, or a magnitude of a pitch value of an audio signal Containing information about, how.
The method of claim 1,
The operation of obtaining the audio representation is
Obtaining attention information for identifying a portion of the text expression that requires attention, based on at least a portion of the text expression and the audio expression of the first audio frame set; And
Obtaining an audio representation of the second audio frame set based on at least a portion of the textual representation and the attention information;
Including, the method.
In an electronic device for synthesizing speech from text,
The electronic device includes at least one processor,
The at least one processor
Obtaining text input to the electronic device,
Encoding the text to obtain a text representation,
Obtaining an audio representation of the first audio frame set,
Obtaining an audio representation of a second audio frame set based on the text representation and the audio representation of the first audio frame set,
Acquiring an audio characteristic of a second audio frame set through decoding of an audio representation of the second audio frame set,
Generating feedback information based on audio characteristics of the second audio frame set,
An electronic device for synthesizing speech based on at least one of an audio characteristic of the first audio frame set or an audio characteristic of the second audio frame set.
The method of claim 7,
The second audio frame set is
The electronic device comprising at least one frame following the first audio frame set.
The method of claim 7,
The at least one processor
Obtaining information on audio characteristics of at least one audio frame of the second audio frame set, obtaining compression information of at least one audio frame of the second audio frame set, and obtaining audio of the at least one audio frame The electronic device that generates the feedback information by combining feature information and compression information on the at least one audio frame.
The method of claim 9,
The above feedback information is
The electronic device, which is used to obtain an audio characteristic of a third audio frame set following the second audio frame set.
The method of claim 9,
The compression information is
At least one of a magnitude of an amplitude value of an audio signal corresponding to each of the at least one audio frame, a magnitude of a root mean square (RMS) of an amplitude value of an audio signal, or a magnitude of a pitch value of an audio signal Containing information about the electronic device.
The method of claim 8,
The at least one processor
On the basis of at least a part of the textual expression and the audio expression of the first audio frame set, obtaining attention information for identifying a part of the text expression requiring attention, and in at least a part of the text expression and the attention information Obtaining an audio representation of the second audio frame set based on the electronic device.
Obtaining text input to the electronic device;
Encoding the text to obtain a text representation;
Obtaining an audio representation of the first audio frame set;
Obtaining an audio representation of a second audio frame set based on the text representation and the audio representation of the first audio frame set;
Acquiring an audio characteristic of a second audio frame set by decoding an audio representation of the second audio frame set;
Generating feedback information based on audio characteristics of the second audio frame set; And
Synthesizing speech based on at least one of an audio characteristic of the first audio frame set or an audio characteristic of the second audio frame set;
A computer-readable recording medium storing a program for executing a method for synthesizing speech from text, including a computer.

Images