WO2020027619A1

WO2020027619A1 - Method, device, and computer readable storage medium for text-to-speech synthesis using machine learning on basis of sequential prosody feature

Info

Publication number: WO2020027619A1
Application number: PCT/KR2019/009659
Authority: WO
Inventors: 김태수; 이영근
Original assignee: 네오사피엔스 주식회사
Priority date: 2018-08-02
Filing date: 2019-08-02
Publication date: 2020-02-06
Also published as: KR20220000391A; KR20230043084A

Abstract

The present disclosure relates to a method for text-to-speech synthesis using machine learning on the basis of a sequential prosody feature. The text-to-speech synthesis using machine learning on the basis of a sequential prosody feature comprises the steps of: receiving input text; receiving a sequential prosody feature; and inputting the input text and the received sequential prosody feature to an artificial neural network text-to-speech synthesis model, so as to generate output speech data corresponding to the input text in which the received sequential prosody feature is reflected.

Description

Text-Speech Synthesis Method, Apparatus and Computer-readable Storage Media Using Machine Learning Based on Sequential Rhymes Features

The present disclosure relates to a method and system for text-to-speech synthesis using machine learning based on sequential rhyme features. More particularly, the present invention relates to a method and system for generating an output speech text for an input text in which sequential rhyme features are input to an artificial neural network text-voice synthesis model.

Speech synthesis, commonly referred to as Text-To-Speech (TTS), is a technology that requires not to pre-record the actual voice in applications that require human speech, such as announcements, navigation, and AI assistants. It is a technique used to play voice. A typical method of speech synthesis is to pre-cut and store speech in very short units such as phonemes, and to express the characteristics of speech as parameters by a concatenative TTS that synthesizes speech by combining the phonemes constituting the sentence to be synthesized. There is a parametric TTS for synthesizing parameters representing speech features constituting a sentence to be synthesized into a speech corresponding to a sentence using a vocoder.

On the other hand, in recent years, the speech synthesis method based on an artificial neural network has been actively studied, and the speech synthesized according to the speech synthesis method includes natural voice features as compared to the conventional methods. However, in the conventional speech synthesis method, regardless of the length of the input text or the length of the reference speech, only a fixed length rhyme feature is applied, so that the rhyme at a specific point in time of the synthesized speech cannot be controlled. The reason is that the probability of loss of information in time is quite high when a fixed length feature is forcibly applied to the reference voice. Accordingly, the conventional speech synthesis method cannot provide fine control of the rhythm for the synthesized speech in order to accurately represent the intentions or emotions of the people.

In addition, when the difference between the pitch range of the source speaker and the pitch of the target speaker is large, it may be difficult to reflect the rhyming characteristics of the source speaker as the target speaker. For example, if the source speaker is female and the target speaker is male, synthesizing the source speaker's rhymes with the target speaker's voice could result in the synthesized voice of the target speaker having a higher pitch than the normal pitch. . Considering this situation, it may be required to preprocess the rhyme feature before applying the rhyme feature to the artificial neural network model in order to improve the quality of the synthesized voice reflecting the rhyme feature.

The method and apparatus according to the present disclosure may generate output voice data for input text reflecting a sequential rhyme feature having a rhyme feature over time to solve the above problem.

In addition, in the method and apparatus according to the present disclosure, the sequential rhyme feature may be input to at least one of an encoder and a decoder of an artificial neural network text-to-speech synthesis model, and the sequential rhyme feature of a variable length is inputted to the length and / or synthesis of the input text. Attention modules may be used to tailor the length of speech.

In addition, the method and apparatus according to the present disclosure may normalize a plurality of embedding vectors corresponding to sequential rhyme features, and apply the normalized plurality of embedding vectors to an artificial neural network text-voice synthesis model.

The present disclosure can be implemented in a variety of ways, including a computer readable storage medium storing a method, system, apparatus, or instructions.

A text-to-speech synthesis method using machine learning based on sequential rhyme features according to an embodiment of the present disclosure includes receiving an input text and receiving a sequential prosody feature. And inputting the input text and the received sequential rhyme features into an artificial neural network text-voice synthesis model to generate output speech data for the input text reflecting the received sequential rhyme features.

The artificial neural network text-to-speech synthesis model of the text-to-speech synthesis method using machine learning based on sequential rhyme characteristics according to an embodiment of the present disclosure may include data representing a plurality of learning texts and a learning voice corresponding to the plurality of learning texts. Based on the machine learning based on the generated data, the data representing the learning voice may include a sequential rhyme feature of the learning voice.

The sequential rhyme feature of the text-to-speech synthesis method using machine learning based on the sequential rhyme feature according to an embodiment of the present disclosure may correspond to at least one unit of a frame, a character, a phoneme, a syllable, or a word. Information in chronological order, and the rhyme information includes at least one of information about a loudness of sound, information about a height of a sound, information about a length of a sound, information about a sound pause period, or information about a style of a sound. It may include.

Receiving a sequential rhyme feature of the text-voice synthesis method using machine learning based on the sequential rhyme feature according to an embodiment of the present disclosure includes receiving a plurality of embedding vectors representing the sequential rhyme feature. Each of the plurality of embedding vectors may correspond to rhyme information included in chronological order.

An artificial neural network text-voice synthesis model of a text-voice synthesis method using machine learning based on sequential rhyme features according to an embodiment of the present disclosure includes an encoder and a decoder, and text using machine learning based on sequential rhyme features. The voice synthesis method further comprises inputting the received plurality of embedding vectors into an attention module to generate a plurality of transform embedding vectors corresponding to respective portions of the input text provided to the encoder, wherein the plurality of transform embedding vectors The length is variable according to the length of the input text, and the step of generating output speech data for the input text includes inputting the generated plurality of transform embedding vectors into an encoder of the artificial neural network text-to-speech synthesis model and a plurality of transform embeddings. Generating output speech data for the input text in which the vector is reflected. The.

An artificial neural network text-to-speech synthesis model of a text-to-speech synthesis method using machine learning based on sequential rhyme characteristics according to an embodiment of the present disclosure includes an encoder and a decoder, and generates output speech data for input text. The method may include inputting the received plurality of embedding vectors into the decoder of the neural network text-voice synthesis model and generating output speech data for the input text in which the plurality of embedding vectors are reflected.

Text-to-speech synthesis method using machine learning based on sequential rhyme feature according to an embodiment of the present disclosure further comprises the step of receiving the speaker's utterance feature, the step of generating output speech data for the input text is a speaker And generating output speech data for the input text in which a plurality of embedding vectors reflecting sequential rhyme characteristics are simulated.

Receiving a speaker's utterance feature of a text-to-speech synthesis method using machine learning based on the sequential rhyme feature according to an embodiment of the present disclosure includes receiving a sequential rhyme feature of the speaker, and includes a plurality of embedding vectors The extracting step may include normalizing the extracted plurality of embedding vectors based on the sequential rhythm characteristics of the speaker, and generating the output speech data for the input text by simulating the speaker's speech and normalizing the plurality of embeddings. And generating output speech data for the input text in which the vector is reflected.

Normalizing the extracted plurality of embedding vectors of the text-to-speech synthesis method using machine learning based on the sequential rhyme characteristics according to an embodiment of the present disclosure may include: The method may include calculating an average value and subtracting the extracted plurality of embedding vectors by an average value of the embedding vectors calculated at each time step.

Receiving a sequential rhyme feature of a text-to-speech synthesis method using machine learning based on the sequential rhyme feature according to an embodiment of the present disclosure comprises: receiving rhyme information for at least a portion of the input text through a user interface And generating output speech data for the input text in which the received sequential rhyme features are reflected, and generating output speech data for the input text in which rhyme information for at least a portion of the input text is reflected. have.

According to one embodiment of the present disclosure, rhyme information for at least a portion of the input text may be input through a tag provided in a speech synthesis markup language.

According to an embodiment of the present disclosure, a text-voice synthesis method using machine learning based on sequential rhyme features may include receiving rhyme information on at least a portion of input text through a user interface and at least a portion of the received input text. The method may further include changing the received sequential rhyme feature based on the rhyme information, and generating the output voice data for the input text in which the received sequential rhyme feature is reflected, the input text in which the changed sequential rhyme feature is reflected. And generating output speech data for the.

According to one embodiment of the present disclosure, rhyme information for at least a portion of the input text, which is used to change the received sequential rhyme feature, may be input through a tag provided in a speech synthesis markup language.

Also, a program for implementing a text-to-speech synthesis method using machine learning based on the sequential rhyme characteristics as described above may be recorded in a computer-readable recording medium.

In addition, apparatus and technical means associated with the method of text-to-speech synthesis using machine learning based on the sequential rhyme characteristics as described above may also be disclosed.

According to some embodiments of the present disclosure, fine rhyme control for synthesized speech is provided because text-to-speech synthesis using machine learning is provided on the basis of sequential rhyme features having variable lengths over time. It is possible to more accurately convey the intention or emotion of a person through speech synthesis.

According to some embodiments of the present disclosure, the use of an attention in applying sequential rhyme features of varying lengths to at least one of an encoder and a decoder of an artificial neural network text-voice synthesis model results in input text and / or synthesis. Because it can be adjusted to correspond to the length of the voice, a variable length sequential rhyme feature can be effectively applied to the input text and / or synthesized voice regardless of its length.

According to some embodiments of the present disclosure, prior to applying the sequential rhyme feature to an artificial neural network text-to-speech synthesis model, preprocessing is performed to normalize a plurality of embedding vectors corresponding to the sequential rhyme feature. In the case of applying to the synthesized voice of another person, it is possible to further improve the quality of the synthesized voice reflecting the rhyme characteristics.

Embodiments of the present disclosure will be described with reference to the accompanying drawings, which will be described below, wherein like reference numerals denote similar elements, but are not limited thereto.

1 is an exemplary diagram illustrating a process of receiving input text and sequential rhyme features by a voice synthesizer according to an embodiment of the present disclosure and outputting a synthesized voice.

FIG. 2 is an exemplary diagram illustrating a process of outputting a synthesized speech using sequential rhyme features and input text extracted from a sequential rhyme feature extractor by a speech synthesizer according to an embodiment of the present disclosure.

3 is an exemplary diagram illustrating a process of outputting a synthesized voice by applying a sequential rhyme feature and a speaker's utterance feature to input text by a voice synthesizer according to an embodiment of the present disclosure.

4 is a block diagram of a text-to-speech synthesis system according to an embodiment of the present disclosure.

5 is a flowchart illustrating a text-voice synthesis method using machine learning based on sequential rhyme characteristics according to an embodiment of the present disclosure.

6 is an exemplary diagram illustrating a configuration of an artificial neural network based text-to-speech synthesis system according to an embodiment of the present disclosure.

FIG. 7 is an exemplary diagram illustrating a process of generating a synthesized speech by inputting sequential rhyme features to a decoder of a text-to-speech synthesis system in an artificial neural network based text-to-speech synthesis system according to an embodiment of the present disclosure.

8 is an exemplary diagram illustrating a process of generating a synthesized speech by inputting a sequential rhyme feature to an encoder of the text-voice synthesis system in an artificial neural network based text-to-speech synthesis system according to an embodiment of the present disclosure.

9 is an exemplary diagram illustrating a network of a sequential rhyme feature extraction unit configured to extract a plurality of embedding vectors representing sequential rhyme features from a voice signal or a sample according to an embodiment of the present disclosure.

10 is a schematic diagram of a text-to-speech synthesis system for outputting synthesized speech by applying a tag provided in a markup language to an input text according to an embodiment of the present disclosure.

11 is a block diagram of a text-to-speech synthesis system according to an embodiment of the present disclosure.

Advantages and features of the disclosed embodiments and methods of achieving them will be apparent with reference to the embodiments described below in conjunction with the accompanying drawings. However, the present disclosure is not limited to the embodiments disclosed below, but may be implemented in various forms, and the present embodiments are merely provided to make the present disclosure complete, and those of ordinary skill in the art to which the present disclosure belongs. It is provided only to fully inform the scope of the invention.

Terms used herein will be briefly described, and the disclosed embodiments will be described in detail.

The terminology used herein is to select general terms that are currently widely used as possible in consideration of the functions in the present disclosure, but may vary according to the intention or precedent of the person skilled in the relevant field, the emergence of new technologies and the like. In addition, in certain cases, there is a term arbitrarily selected by the applicant, and in this case, the meaning will be described in detail in the description of the invention. Therefore, the terms used in the present disclosure should be defined based on the meanings of the terms and the contents throughout the present disclosure, rather than simply the names of the terms.

A singular expression in this specification includes a plural expression unless the context clearly indicates that it is singular. Also, the plural expressions include the singular expressions unless the context clearly indicates the plural.

When a part of the specification 'includes' a certain component, this means that unless otherwise stated, it may include other components other than to exclude other components.

In addition, the term 'part' or 'module' as used herein refers to a software or hardware component, and the 'part' or 'module' plays certain roles. However, "part" or "module" is not meant to be limited to software or hardware. The 'unit' or 'module' may be configured to be in an addressable storage medium or may be configured to play one or more processors. Thus, as an example, a "part" or "module" may be used to refer to components such as software components, object-oriented software components, class components, and task components, processes, functions, properties, Procedures, subroutines, segments of program code, drivers, firmware, microcode, circuits, data, databases, data structures, tables, arrays, and variables. Functions provided within components and 'parts' or 'modules' may be combined into smaller numbers of components and 'parts' or 'modules' or into additional components and 'parts' or 'modules'. Can be further separated.

According to an embodiment of the present disclosure, the “unit” or “module” may be implemented as a processor and a memory. The term processor is to be broadly interpreted to include general purpose processors, central processing units (CPUs), microprocessors, digital signal processors (DSPs), controllers, microcontrollers, state machines, and the like. In some circumstances, a 'processor' may refer to an application specific semiconductor (ASIC), a programmable logic device (PLD), a field programmable gate array (FPGA), or the like. The term 'processor' refers to a combination of processing devices such as, for example, a combination of a DSP and a microprocessor, a combination of a plurality of microprocessors, a combination of one or more microprocessors in conjunction with a DSP core, or a combination of any other such configuration. May be referred to.

The term 'memory' should be interpreted broadly to include any electronic component capable of storing electronic information. The term memory refers to random access memory (RAM), read-only memory (ROM), non-volatile random access memory (NVRAM), programmable read-only memory (PROM), erase-programmable read-only memory (EPROM), electrical May also refer to various types of processor-readable media, such as erasable PROM (EEPROM), flash memory, magnetic or optical data storage, registers, and the like. If the processor can read information from and / or write information to the memory, the memory is said to be in electronic communication with the processor. The memory integrated in the processor is in electronic communication with the processor.

In the present disclosure, the 'sequential rhyme feature' may include rhyme information corresponding to at least one unit of a frame, phoneme, letter, syllable, or word in chronological order. Here, the rhyme information may include at least one of information on the size of the sound, information on the height of the sound, information on the length of the sound, information on the pause period of the sound, or information on the style of the sound. In addition, the style of sound may include any form, manner, or nuance that the sound or voice represents, and may include, for example, tone, intonation, emotion, and the like inherent in the sound or voice. In addition, the sequential rhyme feature may be represented by a plurality of embedding vectors, and each of the plurality of embedding vectors may correspond to rhyme information included in chronological order.

DETAILED DESCRIPTION Hereinafter, exemplary embodiments will be described in detail with reference to the accompanying drawings so that those skilled in the art may easily implement the present disclosure. In the drawings, parts irrelevant to the description are omitted for clarity.

1 is an exemplary view illustrating a process of outputting a synthesized voice 140 by receiving an input text 120 and a sequential rhyme feature 130 by the voice synthesizer 110 according to an embodiment of the present disclosure. The speech synthesizer 110 may be configured to output a synthesized speech corresponding to the input text using the artificial neural network text-to-speech synthesis model. Here, the neural network text-voice synthesis model may be a single neural network text-voice synthesis model. In one embodiment, the speech synthesizer 110 may correspond to the data recognizer 455 of FIG. 4 and / or the data recognizer 1020 of FIG. 10. In addition, the speech synthesizer 110 may be included or provided in a user terminal or a text-to-speech synthesis system.

According to an embodiment, the text input to the speech synthesizer 110 may include text received through an arbitrary interface (not shown). According to another embodiment, the voice recognizer (not shown) may receive a specific voice, convert it into a character corresponding to the input voice, and provide the converted character to the voice synthesizer 110 as input text. Accordingly, as shown in FIG. 1, the speech synthesizer 110 may receive a text 'HELLO' as a text input through an interface or a speech recognizer.

According to one embodiment, speech synthesizer 110 may be configured to receive sequential rhyme features. Here, the sequential rhyme feature may include rhyme information of each time unit according to a predetermined time unit. As illustrated in FIG. 1, the sequential rhyme feature may include information about a sound height, and may include, for example, information indicating a pitch over time of '11113'. According to one embodiment, this sequential rhyme feature may be extracted or determined from any extractor capable of extracting the rhyme feature for the sound, for example from a pitch tracker. According to another embodiment, the speech synthesizer 110 may receive arbitrary information indicating sequential rhyme information about the sound, for example, may receive information indicated by the score. According to another embodiment, speech synthesizer 110 receives sequential rhyme features corresponding to attribute values expressed in speech synthesis markup language over time for text input from any device. can do. This attribute value is described in detail below with reference to FIG. 9.

The speech synthesizer 110 may be configured to generate output data for the input text reflecting the received sequential rhyme characteristics. To this end, the speech synthesizer 110 may apply rhyme information according to the time sequence represented by the sequential rhyme characteristics to the input text. For example, as illustrated in FIG. 1, the speech synthesizer 110 may generate output speech data by reflecting '11113', which is information indicating a pitch according to time, received in the input text 'HELLO'. That is, the speech synthesizer 110 may generate an output voice corresponding to the questionable text 'HELLO?' Having a pitch of 'o' which is the last character of the input text higher than other characters. The voice thus generated may be output through an output device such as a speaker, or may be transmitted to another device having an I / O device.

FIG. 2 outputs the synthesized speech 240 using the sequential rhyme feature 210 and the input text 120 extracted from the sequential rhyme feature extractor 230 by the speech synthesizer 110 according to an embodiment of the present disclosure. It is an exemplary view showing the process of doing. In one embodiment, the sequential rhyme feature extractor 230 may correspond to the sequential rhyme feature extractor 410 of FIG. 4. Since the input text 120 and the speech synthesizer 110 have been described with reference to FIG. 1, redundant description thereof will be omitted.

According to an embodiment, the sequential rhyme feature extractor 230 may receive a voice signal or a voice sample 220 and extract the sequential rhyme feature 210 from the received voice signal or sample. Here, the received voice signal or sample may include voice spectral data representing information related to the sequential rhyme feature 210, and may include, for example, a melody, a voice of a specific speaker, and the like.

According to one embodiment, in extracting the sequential rhyme feature 210, any known suitable feature extraction method capable of extracting the sequential rhyme feature 210 from the speech signal or sample 220 may be used. According to an embodiment, an artificial neural network or a machine learning model may be used to extract sequential rhyme features. For example, the neural network or machine learning model used in the sequential rhyme feature extractor 230 may include a recurrent neural network (RNN), a long short-term memory model (LSTM), a deep neural network (DNN), and a convolution neural (CNN). It can be composed of any one or a combination of various artificial neural network models, including the network).

The sequential rhyme feature extractor 230 may extract a plurality of feature vectors (embedding vectors) representing the sequential rhyme features 210 by inputting the received voice signal or voice sample into the artificial neural network rhyme feature model. Here, each of the plurality of embedding vectors may correspond to a predetermined time unit (eg, frame, phoneme, letter, syllable or word). For example, the vector may include one of various speech feature vectors such as mel frequency cepstral coefficient (MFCC), linear predictive coefficients (LPC), perceptual linear prediction (PLP), and the like, but is not limited thereto. In addition, since the plurality of embedding vectors thus extracted include a rhyme feature or information in chronological order, the length of the vector may be variable or different depending on the length of the input speech sample.

The speech synthesizer 110 may generate voice output data reflecting the sequential rhyme features 210 extracted from the sequential rhyme feature extractor 230 in the received text 120. For example, the speech synthesizer 110 inputs embedding information corresponding to the input 'HELLO' text and a plurality of embedding vectors extracted by the sequential rhyme feature extractor 230 into the artificial neural network text-to-speech synthesis model and sequentially. It is possible to generate 'HELLO' voice data reflecting rhyme characteristics. The generated voice may be output through an output device such as a speaker, or transmitted to another device having an I / O device. FIG. 3 illustrates a sequential rhyme feature 210 and a speaker's utterance feature according to an embodiment of the present disclosure. It is a schematic diagram of the speech synthesizer 110 for outputting synthesized speech by applying 330 to the input text 120. As shown in FIG. 3, the speech synthesizer 110 may receive an input text 120, a sequential rhyme feature 210, and a speaker's utterance feature 330. Here, the sequential rhyme feature 210 may be extracted from the sequential rhyme feature extractor 230 based on the voice signal or the voice sample 220, and the speaker's utterance feature 330 may be used to extract the voice signal or the voice sample 320. Based on the speaker's utterance feature extractor 310 may be extracted. According to an embodiment, the voice signal or voice sample 220 input to the sequential rhyme feature extractor 230 may be different from the voice signal or voice sample 320 input to the voice feature extractor 310. In other embodiments, the two voice signals or

voice samples

220, 320 may be identical to each other. Since the speech synthesizer 110, the input text 120, the sequential rhyme feature 210, the speech signal or the voice sample 220, and the sequential rhyme feature extractor 230 have been described with reference to FIGS. The description is omitted.

The speech feature extractor 310 may be configured to extract the speaker's speech feature from the speech data. The speaker's utterance feature may include at least one of various elements such as style, rhyme, emotion, tone, pitch, etc., which may not only simulate the speaker's voice but also constitute the utterance. According to one embodiment, the speaker's utterance feature may include a one-hot speaker ID-vector representing the speaker. According to another embodiment, the speaker's utterance feature may include an embedding vector representing the speaker's utterance feature. In one embodiment, the speech feature extractor 310 may correspond to the speech feature extractor 415 of FIG. 4.

The speech synthesizer 110 may generate the output speech 340 by inputting the input text 120, the sequential rhyme feature 210, and the speaker's speech feature 330 into the artificial neural network text-to-speech synthesis model. The output voice 340 may include output voice data for the input text 120 reflecting the sequential rhyme feature 210 and the speaker's utterance feature 330. That is, the output voice 340 simulates the speaker's voice based on the speaker's utterance feature and reflects the sequential rhyme feature 210, thereby converting the input text 120 into the sequential rhyme feature 210 to which the speaker is input. It may be data synthesized by talking voice. For example, when the sequential rhyme feature 210 and the speaker's utterance feature 330 are extracted from the first speaker and the second speaker's voice different from the first speaker, respectively, the second speaker's voice is the time of the second speaker. The voice saying 'HELLO' may be output based on the rhyme information according to FIG. The voice thus generated may be output through an output device such as a speaker, or may be transmitted to another device having an I / O device.

2 and 3 illustrate that the speech synthesizer 110 receives a plurality of embedding vectors over time representing the sequential rhyme features 210 extracted from the sequential rhyme feature extractor 230, but is not limited thereto. The synthesizer 110 may receive an input value for a plurality of embedding vectors over time indicating the sequential rhyme feature 210 through an I / O device (not shown). Alternatively, a plurality of embedding vectors according to time representing the sequential rhyme feature 210 may be stored in advance in a storage medium (not shown), and the voice synthesizer 110 may access the storage medium to receive the plurality of embedding vectors. Can be. In addition, the modified information on the plurality of embedded vectors thus extracted or stored may be received through the I / O device, the plurality of embedded vectors may be modified according to the received modified information, and the modified plurality of embedded vectors may be speech synthesizers. May be received at 110.

3, the speaker's utterance feature 330 is illustrated as being extracted from the utterance feature extractor 310 and provided to the voice synthesizer 110, but the present disclosure is not limited thereto. An input value for the embedded embedding vector may be received through an I / O device (not shown). Alternatively, an embedding vector representing the utterance feature may be stored in advance in a storage medium (not shown), and the speech synthesizer 110 may access the storage medium to receive the embedding vector representing the utterance feature. In addition, the extracted information about the extracted or stored utterance feature may be received through the I / O device, an embedding vector indicating the utterance feature may be modified according to the received information, and an embedding vector indicating the modified utterance feature may be Voice synthesizer 110. FIG. 4 is a block diagram of a text-to-speech synthesis system 400 according to one embodiment of the present disclosure. The text-to-speech synthesis system 400 includes a communication unit 405, a sequential rhyme feature extractor 410, a speech feature extractor 415, a normalizer 420, a voice database 425, an attention module 430, an encoder. 435, a decoder 440, a post processor 445, a data learner 450, and a data recognizer 455. The communication unit 405 may be configured such that the text-to-speech synthesis system 400 transmits and receives signals or data with an external device. The external device may include a user terminal for providing a text-to-speech service. Alternatively, the external device may include other text-to-speech synthesis systems. Or, the external device can be any device including a voice database.

According to an embodiment, the communication unit 405 may be configured to receive text from an external device. Here, the text may include training text to be used for training the neural network text-voice synthesis model. Alternatively, the text may include input text that will be used to generate synthesized speech via an artificial neural network text-to-speech synthesis model. Such text may be provided to at least one of the voice database 425, the encoder 435, the decoder 440, the data learner 450, and the data recognizer 455.

The communication unit 405 may be configured to receive a voice signal or a voice sample through an external device. According to an embodiment, such a voice signal or a sample may be transmitted to the sequential rhyme feature extractor 410 to extract the sequential rhyme feature from the voice signal or the sample. According to another exemplary embodiment, the voice signal or sample may be transmitted to the voice feature extractor 415 so that the speaker's voice feature may be extracted from the voice signal or sample. The extracted sequential rhyme feature and / or speaker's utterance feature may be transmitted to the encoder 435 and / or the decoder 440 through the data learner 450 and used to train the neural network text-voice synthesis model. . Alternatively, the extracted sequential rhyme features and / or speaker's speech features are transmitted to the encoder 435 and / or the decoder 440 through the data recognizer 455 to synthesize synthesized speech from the artificial neural network text-voice synthesis model. Can be used to generate

In one embodiment, the communication unit 405 may receive a sequential rhyme feature from an external device. For example, the text-to-speech synthesis system 400 may receive the sequential rhyme feature extracted through the sequential rhyme feature extractor 230 of FIG. 2 through the communication unit 405. In another embodiment, the communication unit 405 may receive the speaker's speech feature from an external device. For example, the communicator 405 may transmit / receive the speaker's voice feature 330 from the speaker's voice feature extractor 310 of FIG. 3. The received sequential rhyme feature and / or the speaker's utterance feature are the normalizer 420, the voice database 425, the attention module 430, the encoder 435, the decoder 440, the data learner 450, or the data. It may be provided to at least one of the recognition unit 455.

In one embodiment, the communication unit 405 may receive rhyme information on the input text from the external device as a sequential rhyme feature. Here, the rhyme information may include an attribute value input through a tag provided in a speech synthesis markup language for each part (eg, phoneme, letters, syllables, words, etc.) of the input text.

According to an embodiment, the communication unit 405 may transmit information related to the generated output voice, that is, output voice data to an external device. In addition, the generated neural network text-to-speech synthesis model may be transmitted to the user terminal or another text-to-speech synthesis system through the communication unit 405.

In FIG. 4, the text-to-speech synthesis system 400 receives a text, a voice signal or a sample, a sequential rhyme feature, and a speaker's voice feature through the communication unit 405, or output voice data and an artificial neural network text-to-speech synthesis model. Although illustrated as being output through the communication unit 405, the text-to-speech synthesis system 400 may further include an input / output device (I / O device; not shown). Accordingly, the text-to-speech synthesis system 400 may directly receive an input from the user and output at least one of text, voice, and video to the user.

The sequential rhyme feature extractor 410 may be configured to receive a voice signal or a sample through the communication unit 405 or an input / output device, and extract the sequential rhyme feature from the received voice signal or sample. In one embodiment, the sequential rhyme feature extractor 410 may correspond to the sequential rhyme feature extractor 230 of FIGS. 2 and 3. For example, the sequential rhyme feature extractor 410 may extract the sequential rhyme features from the received voice signal or sample using a speech processing method such as Mel frequency sestol (MFC). Alternatively, sequential rhyme features may be extracted by inputting a trained rhyme feature extraction model (eg, an artificial neural network) using a voice sample. For example, the sequential rhyme feature may be represented by a plurality of embedding vectors corresponding to a predetermined unit over time. Here, the predetermined unit may correspond to at least one unit such as a frame, a phoneme, a letter, a syllable, a word, a word, and the like. According to another embodiment, the sequential rhyme feature extractor 410 may receive at least one of information about video, music or sheet music, and may be configured to extract sequential rhyme features from the received video, music and / or sheet music. Can be. According to an embodiment, correction information for a plurality of embedding vectors indicating sequential rhyme characteristics may be received through an I / O device (not shown), and the plurality of embedding vectors may be modified through the received information.

The extracted or modified sequential rhyme feature may be provided to the data learner 450 and / or the data recognizer 455 and provided to at least one of the encoder 414 and / or the decoder 440. According to an embodiment, the sequential rhyme feature may be provided to the normalizer 420 and / or the attention module 430 before being provided to the data learner 450 and / or the data recognizer 455. According to an embodiment, the sequential rhyme features extracted from the sequential rhyme feature extractor 410 may be stored in a storage medium (eg, the voice database 425) or an external storage device. Accordingly, when synthesizing the input text, one or more of a plurality of sequential rhyme features previously stored in the storage medium may be selected or specified, and the selected or specified sequential rhyme features may be used for speech synthesis.

The speech feature extractor 415 may be configured to receive the speaker's speech signal (eg, a voice sample) and extract the speaker's speech feature from the received speech signal. Here, the extracted utterance feature may simulate the speaker, and may include any feature included in the speaker's voice, for example, may be represented by a plurality of embedding vectors. In extracting the speaker's speech characteristics, any known suitable feature extraction method that can extract the speech characteristics from the speaker's speech signal may be used. For example, the voice feature extractor 415 may extract the speaker's voice feature from the voice sample using an artificial neural network or a machine learning model. In one embodiment, the speaker's utterance feature extractor 415 may correspond to the speaker's utterance feature extractor 310 of FIG. 3. The extracted speaker's voice feature may be provided to at least one of the data learner 450, the data recognizer 455, the encoder 435, or the decoder 440.

According to an embodiment, the speaker's voice feature extracted from the voice feature extractor 415 may be stored in the voice database 425 or an external storage device. Accordingly, when speech synthesis is performed on the input text, one or more of the voice features of the plurality of speakers previously stored in the storage medium may be selected or designated, and the voice features of the selected or designated speakers may be used for voice synthesis.

According to an embodiment, the speaker's utterance feature may include a speaker's sequential rhyme feature. To this end, for example, the speaker's voice feature extractor 415 may be configured to extract the speaker's sequential rhyme feature from the voice sample. As another example, the speaker's voice feature extractor 415 may provide a voice sample to the sequential rhyme feature extractor 410 to receive the speaker's sequential rhyme features extracted from the voice sample. The extracted sequential rhyme features of the speaker may be provided to the normalizer 420. In FIG. 4, the sequential rhyme feature extraction unit 410 and the speaker's utterance feature extraction unit 415 are illustrated as being configured as separate units, but are not limited thereto and may be configured as one unit.

The normalizer 420 may receive the speaker's sequential rhyme features (multiple embedding vectors) from the sequential rhyme features and the vocalization feature extractor 415 as the speaker's utterance features. . For clarity, sequential rhyme features received from the sequential rhyme feature extractor 410 will be referred to as first sequential rhyme features, and the sequential rhyme features of the speaker from the vocalization feature extractor 415 will be referred to as second sequential rhyme features. Can be.

The normalizer 420 may be configured to normalize the first sequential rhyme feature (eg, the plurality of embedding vectors) based on the second sequential rhyme feature (eg, the plurality of embedding vectors). Here, the first sequential rhyme feature may be a feature extracted from a speaker different from the speaker associated with the second sequential rhyme feature. According to one embodiment, the normalizer 420 may be configured to calculate an average value of a plurality of embedding vectors corresponding to the second sequential rhyme feature at each time step. In addition, the normalizer 420 may normalize the plurality of embedding vectors representing the first sequential rhyme features by subtracting the plurality of embedding vectors representing the first sequential rhyme features to an average value of the embedding vectors calculated at each time step. . The plurality of normalized embedding vectors may be provided to at least one of the voice database 425, the attention module 430, the encoder 435, the decoder 440, the data learner 450, or the data recognizer 455. Can be. Since a plurality of embedding vectors corresponding to the first sequential rhyme feature is normalized using an average value of the plurality of embedding vectors corresponding to the second sequential rhyme feature, an artificial neural network text-voice synthesis model is used to associate the second sequential rhyme feature. When a synthesized voice corresponding to an arbitrary text is generated to simulate the speaker and reflect the first sequential rhyme feature extracted from another speaker, the first sequential rhyme feature may be more naturally applied to the voices of different speakers.

The voice database 425 may store a learning text and a voice corresponding to the plurality of learning texts, and may be accessed by the learning text and the corresponding voice sound data learning unit 450. The learning text may be written in at least one language, and may include at least one of words, phrases, and sentences that can be understood by a person. In addition, the voice stored in the voice database 425 may include voice data from which a plurality of speakers read a training text. The training text and the voice data may be stored in advance in the voice database 425 or may be received from the communication unit 405. Based on the training text and the voice stored in the voice database 425, the data learning unit 450 may generate or learn an artificial neural network text-voice synthesis model. For example, the neural network text-synthesis model may include an encoder 435 and a decoder 440. As another example, the neural network text-synthesis model may include an encoder 435, a decoder 440, and a post processing processor 445.

According to one embodiment, the voice database 425 may be configured to store one or more sequential rhyme features. For example, the one or more sequential rhyme features may include sequential rhyme features normalized from the normalizer 420. In another embodiment, the voice feature extraction unit 415 may be configured to store voice features of one or more speakers. The stored sequential rhyme feature may be provided to at least one of the encoder 435 or the decoder 440 during speech synthesis by the data learner 450 and / or the data recognizer 455. In addition, the stored speaker's utterance feature may be provided to at least one of the encoder 435 or the decoder 440 during speech synthesis by the data learner 450 and / or the data recognizer 455.

The attention module 430 may receive the sequential rhyme feature or the normalized sequential rhyme feature from the sequential rhyme feature extractor 410 or the normalizer 420. According to an embodiment, the attention module 430 may be configured to receive a plurality of embedding vectors representing sequential rhyme characteristics and generate a plurality of transform embedding vectors corresponding to respective portions of the input text provided to the encoder 435. Can be. For example, the attention module 430 may be configured to determine which portion of the plurality of embedding vectors over time corresponds to which portion of the input text at the current time-step. The plurality of transform embedding vectors generated by the attention module 430 may be provided to the encoder 435 for speech synthesis.

The encoder 435 may receive the input text and may be configured to convert the input text into character embedding to generate it. For example, the encoder 435 may be configured as part of the neural network text-to-speech synthesis model. Such character embedding is input to a first artificial neural network text-voice synthesis model (eg, pre-net, CBHG module, DNN, CNN + DNN, etc.) corresponding to encoder 435 to hide hidden states of encoder 435. Can be generated. The first neural network text-voice synthesis model may be included in the artificial neural network text-voice synthesis model. According to an embodiment, the encoder 435 may further receive the sequential rhyme feature from the sequential rhyme feature extractor 410, the normalizer 420, or the attention module 430. Character embedding and sequential rhyme features may be input to the first artificial neural network text-to-speech synthesis model to generate hidden states of the encoder 435. In another embodiment, the encoder 435 may further receive the talker's talk feature from the talk feature extractor 415. The speaker's utterance feature may be input into the first artificial neural network text-to-speech synthesis model along with the character embedding and the sequential rhyme feature to generate hidden states of the encoder 435. The hidden states of the encoder 435 thus generated may be provided to the decoder 440.

Decoder 440 may be configured as part of an artificial neural network text-to-speech synthesis model. In one embodiment, decoder 440 may be configured to receive sequential rhyme features. The decoder 440 may receive the sequential rhyme feature from at least one of the sequential rhyme feature extractor 410 or the normalizer 420. The decoder 440 may receive hidden states corresponding to the input text from the encoder 435. Decoder 440 may also include an attention module configured to determine which portion of the input text is to be generated at the current time-step. Accordingly, the sequential rhyme features and / or hidden states corresponding to the input text may include a second artificial neural network text-voice synthesis model corresponding to the decoder 440 (eg, attention module, decoder RNN, attention RNN, pre-net). , DNN, etc.) to generate output voice data corresponding to the input text. The second neural network text-to-speech synthesis model may be included in the artificial neural network text-to-speech synthesis model.

In another embodiment, the decoder 440 may be configured to further receive the speaker's voice feature from the voice feature extractor 415. The sequential rhyme feature, the hidden states corresponding to the input text, and / or the speaker's utterance feature are input to a second artificial neural network text-to-speech synthesis model corresponding to the decoder 440 to generate output speech data corresponding to the input text. Can be. Such output speech data may include output speech data for input text that simulates the speaker's speech and reflects sequential rhyme characteristics.

The output voice data generated in this way may be represented by a mel spectrogram. However, the present invention is not limited thereto, and the output voice data may be represented by a linear spectrogram. The output voice data may be output to at least one of a speaker, a post processor 445, and a communication unit 405.

According to an embodiment, the post-processing processor 445 may be configured to convert the output voice data generated by the decoder 440 into voices output from the speaker. For example, the changed outputable speech can be represented by a waveform. Post-processing processor 445 may be configured to operate only if the output speech data generated at decoder 440 is inappropriate to be output from a speaker. That is, when the output voice data generated by the decoder 440 is suitable to be output from the speaker, the output voice data may be directly output to the speaker without passing through the post-processing processor 445. Accordingly, although the post-processing processor 445 is shown in FIG. 4 to be included in the text-to-speech synthesis system 400, the post-processing processor 445 may be configured not to be included in the text-to-speech synthesis system 400. have.

According to one embodiment, the post-processing processor 445 may be configured to convert the output speech data represented by the mel spectrogram generated by the decoder 440 into a waveform in the time domain. In addition, the post-processing processor 445 may amplify the size of the output voice data when the size of the signal of the output voice data does not reach a predetermined reference size. The post-processing processor 445 may output the converted output voice data to at least one of the speaker or the communication unit 405.

The data learner 450 may correspond to the data learner 1010 of FIG. 10. The data learner 450 may receive data representing a plurality of learning texts and corresponding learning voices through the voice database 425 or the communication unit 405. The data representing the learning text may include information on at least one letter. For example, the data representing the learning text may include a phoneme sequence corresponding to the learning text using a Grapheme-to-phoneme (G2P) algorithm. The data representing the learning speech may be data obtained by recording the speech read by a human text, a sound feature extracted from such recording data, a spectrogram, or the like. In one embodiment, the data representing the learning voice may include sequential rhyme features of the learning voice. In another embodiment, the data representing the learning voice may further include a utterance characteristic of the speaker who spoke the learning voice. The data learner 450 may generate an artificial neural network text-to-speech synthesis model by performing machine learning based on a pair of learning data corresponding to a plurality of learning texts and corresponding learning voices. In such learning, the learning text may be provided to a first artificial neural network text-to-speech synthesis model corresponding to the encoder of the artificial neural network text-to-speech synthesis model, wherein the sequential rhyme feature is the first artificial neural network text-to-speech synthesis model and / or It may be input to a second artificial neural network text-to-speech synthesis model corresponding to the decoder.

According to an embodiment, the data recognizer 455 may be configured to receive an input text and to receive a sequential rhyme feature. The input text and the sequential rhyme features may be input to the artificial neural network text-voice synthesis model to generate output speech data for the input text reflecting the received rhyme features. Here, the input text may be provided to the first artificial neural network text-voice synthesis model, and the sequential rhyme feature may be input to the first artificial neural network text-voice synthesis model and / or the second artificial neural network text-voice synthesis model. . As a result, output speech data corresponding to the input text in which the sequential rhyme features are reflected can be generated from the artificial neural network text-voice synthesis model. According to another embodiment, the data recognizer 455 may be configured to further receive the speaker's speech characteristics. The received speaker's utterance feature may be provided to the second artificial neural network text-to-speech synthesis model as well as the sequential rhyme feature. Under this operation, output speech data corresponding to the input text that simulates the speaker's speech and reflects the sequential rhyme characteristics can be generated from the artificial neural network text-to-speech synthesis model.

5 is a flowchart illustrating a text-voice synthesis method using machine learning based on sequential rhyme characteristics according to an embodiment of the present disclosure. First, in S510, the text speech-synthesis system 400 generates an artificial neural network text-voice synthesis model generated by performing machine learning based on a plurality of learning texts and voice data corresponding to the plurality of learning texts. Can be done. Here, the neural network text-voice synthesis model may be a single neural network text-voice synthesis model.

The text-to-speech synthesis system 400 may perform a step of receiving an input text at S520. In step S530, the text-to-speech synthesis system 400 Receiving a sequential rhyme feature may be performed. The text-to-speech synthesis system 400 may input the input text and the sequential rhyme feature into a pre-learned text-to-speech synthesis model to generate output speech data for the input text in which the sequential rhyme feature is reflected in S540. have.

6 is an exemplary diagram illustrating a configuration of an artificial neural network based text-to-speech synthesis system according to an embodiment of the present disclosure. In one embodiment, each of encoder 610 and decoder 620 and post-processor 630 may correspond to each of encoder 435, decoder 440 and post-processor 445 of FIG. 4.

According to an embodiment, the encoder 610 may receive character embedding for the input text, as shown in FIG. 6. According to another embodiment, the input text may include at least one of a word, phrase, or sentence used in one or more languages. For example, text such as a sentence such as 'HELLO' may be input. When the input text is received, the encoder 610 may divide the received input text into a letter unit, a letter unit, and a phoneme unit. According to another embodiment, the encoder 610 may receive input text divided into a Jamo unit, a character unit, and a phoneme unit. The encoder 610 may then generate the input text by converting it into character embedding.

The encoder 610 may be configured to generate text as pronunciation information. In one embodiment, encoder 610 may pass the generated character embedding to a pre-net including a fully-connected layer. In addition, the encoder 610 may provide an output from the pre-net to the CBHG module to output encoder hidden states e _i of the encoder, as shown in FIG. 6. For example, the CBHG module may include a 1D convolution bank, max pooling, highway network, and bidirectional gated recurrent unit (GRU).

In another embodiment, when encoder 610 receives input text or separated input text, encoder 610 may be configured to generate at least one embedding layer. According to an embodiment of the present disclosure, at least one embedding layer of the encoder 610 may generate character embedding based on input text divided into a Jamo unit, a character unit, and a phoneme unit. For example, the encoder 610 may use a machine learning model (eg, a probabilistic model or an artificial neural network) that has already been trained to obtain character embedding based on the separated input text. Further, the encoder 610 may update the machine learning model while performing machine learning. When the machine learning model is updated, the character embedding for the separated input text may also change. The encoder 610 may pass the character embedding to a deep neural network (DNN) module configured with a fully-connected layer. The DNN may include a general feedforward layer or a linear layer. The encoder 610 may provide an output of the DNN to a module including at least one of a convolutional neural network (CNN) or a recurrent neural network (RNN), and may generate hidden states of the encoder 610. CNNs can capture local characteristics according to convolution kernel size, while RNNs can capture long term dependencies. The hidden states of the encoder 610, that is, pronunciation information about the input text, are provided to the decoder 620 including an attention module, and the decoder 620 may be configured to generate such pronunciation information as a voice.

The decoder 620 may receive the hidden states e _i of the encoder from the encoder 610. In one embodiment, as shown in FIG. 6, the decoder 620 includes an attention module, a freenet consisting of a fully connected layer, and a gated recurrnt unit (GRU) and an attention recurrent neural network (RNN), a residual. The decoder may include a decoder RNN including a residual GRU. Here, the attention RNN may output information to be used in the attention module. In addition, the decoder RNN may receive position information of the input text from the attention module. That is, the location information may include information regarding which location of the input text the decoder 620 is converting to speech. The decoder RNN may receive information from the attention RNN. The information received from the attention RNN may include information about which voice the decoder 620 produced up to a previous time-step. The decoder RNN may generate the next output voice following the voice thus far generated. For example, the output voice may have a mel spectrogram form, and the output voice may include r frames.

In another embodiment, the freenet included in the decoder 620 may be replaced with a DNN configured with a fully-connected layer. Here, the DNN may include at least one of a general feedforward layer or a linear layer.

In one embodiment, decoder 620 may be configured to receive sequential rhyme features. For example, as shown in FIG. 6, the sequential rhyme feature extractor 410 may include a plurality of embedding vectors p1, p2,... Pn representing sequential rhyme features from a speech signal or a sample, where n is a speech sample. Proportional to the length of. Each of the plurality of embedding vectors may include a rhyme feature or information for each unit time. The manner in which the sequential rhythm feature extractor 410 inputs a plurality of embedding vectors p1, p2, ... pn from the voice signal or the sample to the decoder will be described in detail with reference to FIG.

In one embodiment, the decoder 620 may be configured to further receive the speaker's speech characteristics. For example, as shown in FIG. 6, the speaker's utterance feature may include a speaker ID input to the utterance feature extraction unit 415 to generate a speaker embedding vector e corresponding to the speaker's utterance feature as the speaker's utterance feature. have. As another example, the speaker's utterance feature may be generated by extracting the speaker's embedding vector from a voice signal or sample other than the speaker ID.

Also, the attention module of the decoder 620 may receive information from the attention RNN. The information received from the attention RNN may include information about which voice the decoder 620 produced up to a previous time-step. Also, the attention module of the decoder 620 may output the context vector based on the information received from the attention RNN and the information of the encoder. The information of the encoder 610 may include information about input text to generate speech. The context vector may include information for determining which portion of the input text to generate a speech at the current time-step. For example, the attention module of the decoder 620 generates a voice based on the front part of the input text at the beginning of the voice generation, and gradually generates a voice based on the back part of the input text as the voice is generated. Information can be output.

The decoder 620 inputs each of the embedding vectors p1, p2, ... pn and the speaker embedding vector e for each of the attentional RNN and decoder RNN in the sequential rhyme features. The structure of the neural network can be configured to decode differently for each speaker, and for each part of the input text. In FIG. 6, although a plurality of embedding vectors p1, p2,... Pn are illustrated as being extracted from the sequential rhyme feature extraction unit 410, the present invention is not limited thereto, and the decoder 620 may be normalized from the normalizer 420. A plurality of embedding vectors corresponding to the sequential rhyme features may be received, and the plurality of embedding vectors corresponding to the normalized sequential rhyme features are input for each time step of the attention RNN and the decoder RNN together with the embedding vector e of the speaker. Can be.

The dummy frames are frames that are input to the decoder 620 when there is no previous time-step. RNNs can do machine learning autoregressive. That is, the r frame output in the previous time-step 622 may be an input of the current time-step 623. Since there cannot be a previous time-step in the initial time-step 621, the decoder 620 may input a dummy frame into the initial time-step machine learning network.

The operations of the DNN, the attention RNN, and the decoder RNN may be performed repeatedly for text-to-speech synthesis. For example, the r frames obtained in the first time-step 621 may be input to the next time-step 622. In addition, the r frames output in the time-step 622 may be input to the next time-step 623.

Through the above-described process, not only the input text can be synthesized for each speaker, but also the rhyme characteristic for each part can be reflected in the input text in chronological order. That is, it is possible to control the rhyme characteristic at a specific time point of the synthesized speech corresponding to the input text. Accordingly, the text-to-speech synthesis system can control fine rhymes for the synthesized voice in order to more accurately convey people's intentions or emotions.

According to an embodiment, the decoder 620 may concatenate the mel spectrograms generated for each time-step in chronological order to obtain a voice of the mel spectrogram for the entire text. The voice of the mel spectrogram for the entire text may be output to the post processor 630. For example, the post processing processor 630 may correspond to the post processing processor 445 of FIG. 4.

According to an embodiment, the CBHG of the post-processor 630 may be configured to convert the mel scale spectrogram of the decoder 620 into a linear-scale spectrogram, as shown in FIG. 6. For example, the output signal of the CBHG of the post processing processor 630 may include a magnitude spectrogram. The phase of the output signal of the CBHG of the post-processor 630 may be recovered through a Griffin-Lim algorithm and may be inverse short-time fourier transform. The post processing processor 630 may output a voice signal in a time domain.

According to another embodiment, if encoder 610 is configured to include a CNN or RNN, post-processing processor 630 is configured to include a CNN or RNN, which CNN or RNN is associated with a CNN or RNN of encoder 610. Similar operations can be performed. That is, the CNN or RNN of the post processor 630 may capture local characteristics and long term dependencies. For example, the post processing processor 630 may be a vocoder. Accordingly, the CNN or RNN of the post processor 630 may output a linear-scale spectrogram. For example, the linear-scale spectrogram may include a magnitude spectrogram. The post-processing processor 630 may predict the phase of the spectrogram through the Griffin-Lim algorithm. The post processing processor 630 may output a voice signal of a time domain using an inverse short-time fourier transform.

According to another embodiment, the post processing processor 630 may generate a speech signal from the mel spectrogram based on the machine learning model. The machine learning model may include a model of machine learning the correlation between the mel spectrogram and the speech signal. For example, an artificial neural network model such as WaveNet or WaveGlow may be used.

The neural network-based text-to-speech synthesis system can be trained using a large database that exists as a pair of training texts and voice signals. According to an embodiment, the speech synthesizing apparatus may receive a text and define a loss function by comparing the output speech signal with a correct answer speech signal. The speech synthesis apparatus learns a loss function through an error back propagation algorithm, and finally obtains an artificial neural network that generates a desired speech output when an arbitrary text is input.

In such a neural network-based speech synthesis apparatus, text, speech characteristics, sequential rhyme characteristics, etc. may be input to an artificial neural network text-voice synthesis model to output a speech signal. The text-to-speech synthesis system compares the output speech signal with the correct speech signal and learns it. When receiving the text, the speaker's speech characteristics, and the sequential rhyme characteristics, the output speech data reads the text reflecting the sequential rhyme characteristics of the speaker. Can be generated.

FIG. 7 is an exemplary diagram illustrating a process of generating a synthesized speech by inputting sequential rhyme features to the decoder 720 of the text-to-speech synthesis system in an artificial neural network based text-to-speech synthesis system according to an embodiment of the present disclosure. . Here, each of the encoder 710, the decoder 720, the sequential rhyme feature extractor 730, and the attention module 740 is an encoder 435, a decoder 440, and a sequential rhyme feature extractor 410 of FIG. 4. And the attention module 430. Also, the encoder 710 and the decoder 720 may correspond to each of the encoder 610 and the decoder 620 of FIG. 6. In FIG. 7, the length N of the voice is assumed to be 4, and the length T of the text is 3, but the present invention is not limited thereto. The length N of the voice and the length T of the text may be any positive numbers different from each other.

According to an embodiment, as shown in FIG. 7, the sequential rhyme feature extractor 730 receives the spectrograms y ₁ , y ₂ , y ₃ , y _n , and includes a plurality of embeddings that represent sequential rhyme features. It can be configured to extract the vectors P ₁ , P ₂ , P ₃ , P _n . The plurality of embedding vectors P ₁ , P ₂ , P ₃ , and P _n extracted in this way may be provided to the decoder 720. For example, the extracted plurality of embedding vectors P ₁ , P ₂ , P ₃ , P _n may be provided to the N decoders RNNs and the attention RNNs of the decoder 720. In addition, hidden states e ₁ , e ₂ , e _T provided from the encoder 710 may be provided to the attention module 740, and the attention module 740 may be hidden states e ₁ , e ₂ , e. _T ) may generate transform hidden states e ' ₁ , e' ₂ , e ' _3, e' _N to correspond to the lengths of the spectrograms P ₁ , P ₂ , P ₃ , P _n . The generated transform hidden states e ' ₁ , e' ₂ , e ' _3, e' _{N are} connected together with the extracted plurality of embedding vectors P ₁ , P ₂ , P ₃ , P _n to form N decoders. It can be entered and processed in each of the RNN and the attention RNN. Since the processing in the decoder 720 is a process overlapping with the processing described with reference to FIG. 6, a detailed description thereof will be omitted. Through this process, the neural network text-voice synthesis model included in the encoder 710 and the decoder 720 may be trained so that the sequential rhyme characteristics may be more naturally reflected.

In FIG. 7, spectrograms y ₁ , y ₂ , y ₃ , y _n representing a specific voice are provided to the sequential rhythm feature extractor 730, and the same spectrograms y ₁ and y ₂ are provided through the decoder 620. , y ₃ , y _n ) is described. However, the present invention is not limited thereto, and a voice having a different length from that output through the decoder 720 may be input to the sequential rhythm feature extractor 730. In this case, an additional attention module (not shown) receives the plurality of embedding vectors extracted from the sequential rhyme feature extractor and converts the lengths of the received embedding vectors to correspond to the lengths of the voices output through the decoder 720. You can. Then, the transformed plurality of embedding vectors may be provided to the decoder 720.

FIG. 8 is an exemplary diagram illustrating a process of generating a synthesized speech by inputting sequential rhyme features to an encoder 820 of a text-to-speech synthesis system in an artificial neural network based text-to-speech synthesis system according to an embodiment of the present disclosure. . Here, each of the attention module 810, the encoder 820, the decoder 830, and the sequential rhyme feature extractor 840 is the attention module 430, the encoder 435, the decoder 440, and the sequential rhyme of FIG. 4. Each feature extractor 410 may correspond to each other. Also, the encoder 810 and the decoder 820 may correspond to each of the encoder 610 and the decoder 620 of FIG. 6. Although FIG. 8 illustrates that the voice length N is 4 and the text length T is 3, the present invention is not limited thereto, and the voice length N and the text length T may be any positive numbers different from each other.

According to an embodiment, as shown in FIG. 8, the sequential rhyme feature extractor 840 receives the spectrograms y ₁ , y ₂ , y ₃ , y _n , and includes a plurality of embeddings that represent sequential rhyme features. It can be configured to extract the vectors P ₁ , P ₂ , P ₃ , P _n . The plurality of embedding vectors P ₁ , P ₂ , P ₃ , and P _n extracted in this way may be provided to the attention module 810. The attention module 810 converts the plurality of input embedding vectors P ₁ , P ₂ , P ₃ , and P _n to correspond to the length T of the phoneme sequence corresponding to the encoder 820. ₁ ′, P ₂ ′, P _T ′). According to another embodiment, the sequential rhyme feature extractor 840 may generate a plurality of transform embedding vectors to correspond to words rather than phonemes. For example, when the i-th phone and the j-th phone belong to the same word (v), it may have a value of P _i '= P _j ', and the attention module 810 may include a plurality of transform embedding vectors corresponding to the word (

,

, ...,

Can be configured to generate Word length

May be any positive number different from N, T.

As an example of obtaining A, an average of transform embedding vectors of phonemes belonging to the same word may be used, but is not limited thereto, and an additional attention module may be used.

As illustrated in FIG. 8, each of the plurality of transform embedding vectors P ₁ ′, P ₂ ′, and P _T ′ generated in this manner may include hidden states e ₁ , e ₂ , and s corresponding to a phoneme sequence of the input text. e _T ) may be connected to correspond to each other. The connected hidden states e ₁ , e ₂ , e _T and the plurality of transform embedding vectors P ₁ ′, P ₂ ′, and P _T ′ may be provided to the decoder 830. In contrast, the decoder 830 stores the received hidden states e ₁ , e ₂ , e _T and the transform embedding vectors P ₁ ′, P ₂ ′, P _T ′ in the attention module of the decoder 830. A phoneme sequence y ₁ , y ₂ , y ₃ , y _n can be generated using the pre-net, N decoders RNN, and the attention RNN. In contrast, a plurality of transform embedding vectors corresponding to a word (

,

, ...,

) Is generated, a plurality of transform embedding vectors (

,

, ...,

Each of) may be connected to correspond to each of the hidden states corresponding to the word sequence of the input text. Since the processing in the decoder is a process overlapping with the processing described with reference to FIG. 6, a detailed description thereof will be omitted. Through this process, the neural network text-voice synthesis model included in the encoder 820 and the decoder 830 may be trained to more naturally reflect the sequential rhyme characteristics.

9 is an exemplary diagram illustrating a network of a sequential rhyme feature extraction unit 920 configured to extract a plurality of embedding vectors 930 representing sequential rhyme features from a voice signal or a sample 910 according to an embodiment of the present disclosure. . In one embodiment, the network of the sequential rhyme feature extraction unit 920 may include a convolutional neural network (CNN), batch-normalization (BN), a rectifier linear unit (ReLU), and a gated recurrent unit (GRU). When the CNN, BN, and ReLU receive a voice signal or a sample and input the output value to a gated recurrent unit (GRU), the CNN, BN, and ReLU may output a plurality of embedding vectors indicating sequential rhyme characteristics. For example, the voice signal or sample may be received in the form of log-Mel-spectrogram.

In one embodiment, when the data recognizer 455 infers speech synthesis, the speech signal or sample need not be speech data corresponding to the input text, and a randomly selected speech signal may be used. In contrast, when the data learner 450 learns speech synthesis, the speech signal or sample may include speech data corresponding to the input text.

In such a network, any spectrogram can be inserted into this network because there is no restriction in using the spectrogram. In addition, through this, it is possible to generate an embedded vector 930 representing sequential rhyme characteristics through immediate adaptation of the network. The spectrogram input as a voice signal or a sample may have a variable length, and lengths of a plurality of embedding vectors may vary according to the length. Although FIG. 9 illustrates a network including CNN, BN, ReLU, and GRU, a network including various layers may be constructed to extract sequential rhyme characteristics.

10 is a schematic diagram of a text-to-speech synthesis system 1000 for outputting synthesized speech by applying an attribute value input to a tag provided in a markup language according to an embodiment of the present disclosure to an input text. In one embodiment, the text-to-speech synthesis system 1000 may correspond to the text-to-speech synthesis system 400 of FIG. 4 and / or the text-to-speech synthesis system 1100 of FIG. 11.

In order to generate, adjust, or change the sequential rhyme information, the text-to-speech synthesis system 1000 may receive rhyme information for at least a portion of the text through the interface device. Here, the interface device may include any interface device directly connected to the text-voice synthesis system 1000 or connected through wired and / or wireless communication, and may include, for example, an interface of a user terminal. In addition, rhyme information for at least a portion of the text may be received through any text editor or voice editor capable of entering and editing any text.

According to an embodiment, the speech synthesis system 1000 may receive, as rhyme information, an attribute value corresponding to each part of the input text using a tag of any speech synthesis markup language provided in an arbitrary text editor. . For example, a tag provided in a speech synthesis markup language may include any tag for indicating an attribute included in a sequential rhyme feature. Rhyme information corresponding to the text portion between the start tag and the end tag may be input. For example, as shown in FIG. 10, '1. <speed = 1.5> I'm a boy. </ speed> 'may include rhyme information indicating a speed in a part of I'm a boy between the start tag and the end tag. As another example, as shown in Figure 10, '2. This is what <style = emphasis> I </ style> have. 'May include rhyme information that highlights the letter I between the start and end tags.

The text-to-speech synthesis system 1000 generates sequential rhyme information based on rhyme information on at least a portion of the received input text, or changes rhyme information corresponding to the input text among sequential rhyme information corresponding to the input text. The synthesized speech corresponding to the input text reflecting the generated or changed sequential rhyme information may be generated. According to an embodiment, the text-to-speech synthesis system 1000 may apply rhyme information (eg, attribute values) corresponding to each portion of the input text input to the reference embedding vector corresponding to the reference sequential rhyme information. have. Here, the reference embedding vector may include a plurality of embedding vectors representing predetermined sequential rhyme feature information. For example, the reference embedding vector includes a rhyme feature vector over time, and each rhyme feature information includes a plurality of sub-embedding vectors (eg, height, size, length, rest period, style vector, etc.) orthogonal to each other. It can be expressed as the weighted sum of. The text-to-speech synthesis system 1000 can separate the intrinsic elements of the reference embedding vector. For example, the text-to-speech synthesis system 1000 may obtain a plurality of unit embedding vectors orthogonal to each other based on the reference embedding vector. According to an embodiment, a method of separating elements embedded in an embedding vector may include independent component analysis (ICA), independent vector analysis (IVA), sparse coding, independent factor analysis (IAF), independent subspace analysis (ISA), and NMF. There may be various methods such as nonnegative matrix factorization. And so that the elements inherent in the embedding vector can be separated, the text-voice synthesis system 1000 can perform regularization when learning the text-voice synthesis system when learning the embedding vector for the sequential rhyme feature. have. This normalization may be performed through the normalizer 420 of FIG. 4. When the text-to-speech synthesis system 1000 performs machine learning by performing normalization during learning, the reference embedding vector may be learned as a sparse vector. Accordingly, the text-to-speech synthesis system 900 can accurately separate intrinsic elements using PCA (principle component analysis) in embedding vectors learned from sparse vectors. Under this configuration, the text-to-speech synthesis system 1000 may modify the reference embedding vector based on the attribute values in the tags provided by the speech synthesis markup language. For example, the text-to-speech synthesis system 1000 may change the weights for the plurality of unit embedding vectors based on the attribute values in the received tags.

In one embodiment, the text-to-speech synthesis system 1000 may be configured to modify the reference embedding vector based on attribute values in a tag provided by the received speech synthesis markup language. For example, the text-to-speech synthesis system 1000 may resynthesize the embedding vector corresponding to the sequential rhyme feature by multiplying and adding the weights changed according to the received attribute value to the plurality of unit embedding vectors. The text-to-speech synthesis system 1000 may output an embedding vector for the changed sequential rhyme feature information. The text-to-speech synthesis system 1000 inputs the modified embedding vector into the neural network text-to-speech synthesis model, and outputs the output speech data to the input text reflecting the information contained in the attribute value in the tag provided by the speech synthesis markup language. Can be converted into voice data.

11 is a block diagram of a text-to-speech synthesis system 1100 according to one embodiment of the disclosure.

Referring to FIG. 11, the text-to-speech synthesis system 1100 may include a data learner 1110 and a data recognizer 1120. Each of the data learner 1110 and the data recognizer 1120 of the text-to-speech synthesis system of FIG. 11 is a data learner 450 and the data recognizer used by the text-to-speech synthesis system 400 of FIG. 4. 455 may correspond to each other.

The data learner 1110 may obtain data to obtain a machine learning model. In addition, the data recognizer 1120 may generate the output voice by applying the data to the machine learning model. The text-to-speech synthesis system 1100 as described above may include a processor and a memory.

The data learner 1110 may perform voice learning on text. The data learner 1110 may learn a criterion about which voice to output according to the text. In addition, the data learner 1110 may learn a criterion about which voice feature to output a voice. The characteristic of the voice may include at least one of a phoneme pronunciation, a user's tone, intonation, or accentuation. The data learner 1110 acquires data to be used for learning and applies the acquired data to a data learning model to be described later, thereby learning a voice according to text.

The data recognizer 1120 may output a voice for the text based on the text. The data recognizer 1120 may output a voice from a predetermined text by using the learned data learning model. The data recognizer 1120 may obtain a predetermined text (data) according to a preset criterion by learning. In addition, the data recognizing unit 1120 may output a voice based on predetermined data by using the data learning model using the acquired data as an input value. Also, the result value output by the data learning model using the acquired data as an input value may be used to update the data learning model.

At least one of the data learner 1110 or the data recognizer 1120 may be manufactured in the form of at least one hardware chip and mounted on the electronic device. For example, at least one of the data learner 1110 or the data recognizer 1120 may be manufactured in the form of a dedicated hardware chip for artificial intelligence (AI), or an existing general purpose processor (eg, a CPU). Alternatively, the electronic device may be manufactured as a part of an application processor or a graphics dedicated processor (eg, a GPU) and mounted on the electronic devices described above.

In addition, the data learner 1110 and the data recognizer 1120 may be mounted on separate electronic devices, respectively. For example, one of the data learner 1110 and the data recognizer 1120 may be included in the electronic device, and the other may be included in the server. In addition, the data learner 1110 and the data recognizer 1120 may provide model information constructed by the data learner 1110 to the data recognizer 1120 via a wired or wireless connection. The data input to 1120 may be provided to the data learner 1110 as additional learning data.

Meanwhile, at least one of the data learner 1110 or the data recognizer 1120 may be implemented as a software module. When at least one of the data learning unit 1110 and the data recognizing unit 1120 is implemented as a software module (or a program module including an instruction), the software module may be a memory or computer readable non-readable. It may be stored in a non-transitory computer readable media. In this case, at least one software module may be provided by an operating system (OS) or by a predetermined application. Alternatively, some of the at least one software module may be provided by an operating system (OS) and others may be provided by a given application.

The data learner 1110 according to an embodiment of the present disclosure may include a data acquirer 1111, a preprocessor 1112, a training data selector 1113, a model learner 1114, and a model evaluator 1115. It may include.

The data acquirer 1111 may acquire data necessary for machine learning. Since a large amount of data is required for learning, the data acquirer 1111 may receive a plurality of texts and corresponding voices.

The preprocessor 1112 may preprocess the acquired data so that the acquired data may be used for machine learning to determine the mental state of the user. The preprocessor 1112 may process the acquired data into a preset format for use by the model learner 1114, which will be described later. For example, the preprocessor 1112 may acquire morpheme embedding by morphologically analyzing text and voice.

The training data selector 1113 may select data necessary for learning from the preprocessed data. The selected data may be provided to the model learner 1114. The training data selector 1113 may select data required for learning from preprocessed data according to a preset criterion. In addition, the training data selector 1113 may select data according to preset criteria by learning by the model learner 1114 to be described later.

The model learner 1114 may learn a criterion about which voice to output according to the text based on the training data. In addition, the model learner 1114 may learn by using a learning model that outputs a voice according to text as learning data. In this case, the data learning model may include a pre-built model. For example, the data learning model may include a model built in advance by receiving basic training data (eg, a sample image).

The data learning model may be constructed in consideration of the application field of the learning model, the purpose of learning, or the computer performance of the device. The data learning model may include, for example, a model based on a neural network. For example, models such as Deep Neural Network (DNN), Recurrent Neural Network (RNN), Long Short-Term Memory models (LSTM), Bidirectional Recurrent Deep Neural Network (BRDNN), and Convolutional Neural Networks (CNN) can be used as data learning models. But it is not limited thereto.

According to various embodiments of the present disclosure, when there are a plurality of pre-built data learning models, the model learning unit 1114 may determine a data learning model having a large correlation between the input learning data and the basic learning data as a data learning model to be trained. have. In this case, the basic training data may be previously classified by the type of data, and the data learning model may be pre-built for each type of data. For example, the basic training data is classified based on various criteria such as the region where the training data is generated, the time at which the training data is generated, the size of the training data, the genre of the training data, the creator of the training data, and the types of objects in the training data. It may be.

In addition, the model learner 1114 may train the data learning model using, for example, a learning algorithm including an error back-propagation method or a gradient descent method.

In addition, the model learner 1114 may learn the data learning model through, for example, supervised learning using the learning data as an input value. In addition, the model learner 1114 learns data through unsupervised learning that finds a criterion for situation determination by, for example, self-learning a type of data necessary for situation determination without any guidance. You can train the model. In addition, the model learner 1114 may learn the data learning model through, for example, reinforcement learning using feedback on whether the result of the situation determination according to the learning is correct.

In addition, when the data learning model is trained, the model learner 1114 may store the learned data learning model. In this case, the model learner 1114 may store the learned data learning model in a memory of the electronic device including the data recognizer 1120. Alternatively, the model learner 1114 may store the learned data learning model in a memory of a server connected to the electronic device through a wired or wireless network.

In this case, the memory in which the learned data learning model is stored may store, for example, commands or data related to at least one other element of the electronic device. The memory may also store software and / or programs. The program may include, for example, a kernel, middleware, an application programming interface (API) and / or an application program (or 'application'), or the like.

The model evaluator 1115 may input the evaluation data into the data learning model, and may cause the model learner 1114 to learn again when the result output from the evaluation data does not satisfy a predetermined criterion. In this case, the evaluation data may include preset data for evaluating the data learning model.

For example, the model evaluator 1115 may not satisfy a predetermined criterion when the number or ratio of the evaluation data whose recognition result is not accurate among the results of the learned data learning model for the evaluation data exceeds a preset threshold. It can be evaluated as. For example, when a predetermined criterion is defined at a ratio of 2%, when the trained data learning model outputs an incorrect recognition result for more than 20 evaluation data out of a total of 1000 evaluation data, the model evaluator 1115 learns. The data learning model can be evaluated as not suitable.

On the other hand, when there are a plurality of trained data learning models, the model evaluator 1115 evaluates whether each of the learned video learning models satisfies a predetermined criterion and uses the model satisfying the predetermined criterion as the final data learning model. You can decide. In this case, when there are a plurality of models satisfying a predetermined criterion, the model evaluator 1115 may determine any one or a predetermined number of models which are preset in the order of the highest evaluation score as the final data learning model.

At least one of the data acquirer 1111, the preprocessor 1112, the training data selector 1113, the model learner 1114, or the model evaluator 1115 in the data learner 1110 may be at least one. May be manufactured in the form of a hardware chip and mounted on an electronic device. For example, at least one of the data acquirer 1111, the preprocessor 1112, the training data selector 1113, the model learner 1114, or the model evaluator 1115 may be artificial intelligence (AI). It may be manufactured in the form of a dedicated hardware chip, or may be manufactured as part of an existing general purpose processor (eg, a CPU or an application processor) or a graphics dedicated processor (eg, a GPU) and mounted on the above-mentioned various electronic devices.

In addition, the data acquirer 1111, the preprocessor 1112, the training data selector 1113, the model learner 1114, and the model evaluator 1115 may be mounted in one electronic device or may be separate. Each of the electronic devices may be mounted. For example, some of the data acquirer 1111, the preprocessor 1112, the training data selector 1113, the model learner 1114, and the model evaluator 1115 are included in the electronic device, and the other part thereof is included in the electronic device. Can be included on the server.

In addition, at least one of the data acquirer 1111, the preprocessor 1112, the training data selector 1113, the model learner 1114, or the model evaluator 1115 may be implemented as a software module. A program in which at least one of the data obtaining unit 1111, the preprocessor 1112, the training data selecting unit 1113, the model learning unit 1114, or the model evaluating unit 1115 includes a software module (or instruction). Module may be stored on a computer readable non-transitory computer readable media. In this case, at least one software module may be provided by an operating system (OS) or by a predetermined application. Alternatively, some of the at least one software module may be provided by an operating system (OS) and others may be provided by a given application.

The data recognizer 1120 according to an exemplary embodiment of the present disclosure may include a data acquirer 1121, a preprocessor 1122, a recognition data selector 1123, a recognition result provider 1124, and a model updater 1125. It may include.

The data acquirer 1121 may acquire text required to output voice. In contrast, the data acquirer 1121 may acquire a voice required for outputting text. The preprocessor 1122 may preprocess the acquired data so that the obtained data may be used to output voice or text. The preprocessor 1122 may process the acquired data into a preset format so that the recognition result providing unit 1124, which will be described later, may use the acquired data for outputting voice or text.

The recognition data selector 1123 may select data necessary for outputting voice or text from the preprocessed data. The selected data may be provided to the recognition result provider 1124. The recognition data selector 1123 may select some or all of the preprocessed data according to preset criteria for outputting voice or text. In addition, the recognition data selector 1123 may select data according to a predetermined criterion by learning by the model learner 1114.

The recognition result providing unit 1124 may output the voice or the text by applying the selected data to the data learning model. The recognition result provider 1124 may apply the selected data to the data learning model by using the data selected by the recognition data selector 1123 as an input value. In addition, the recognition result may be determined by the data learning model.

The model updater 1125 may cause the data learning model to be updated based on the evaluation of the recognition result provided by the recognition result provider 1124. For example, the model updater 1125 may allow the model learner 1114 to update the data learning model by providing the model learner 1114 with the recognition result provided by the recognition result provider 1124. have.

Meanwhile, at least one of the data acquisition unit 1121, the preprocessing unit 1122, the recognition data selection unit 1123, the recognition result providing unit 1124, or the model updating unit 1125 in the data recognizing unit 1120 is at least one. It may be manufactured in the form of one hardware chip and mounted on an electronic device. For example, at least one of the data acquirer 1121, the preprocessor 1122, the recognition data selector 1123, the recognition result provider 1124, or the model updater 1125 may be artificial intelligence (AI). ) May be manufactured in the form of a dedicated hardware chip, or may be manufactured as a part of an existing general purpose processor (eg, a CPU or an application processor) or a graphics dedicated processor (eg, a GPU) and mounted on the aforementioned various electronic devices.

In addition, the data obtaining unit 1121, the preprocessor 1122, the recognition data selecting unit 1123, the recognition result providing unit 1124, and the model updating unit 1125 may be mounted in one electronic device or may be separate. May be mounted on the electronic devices. For example, some of the data obtaining unit 1121, the preprocessing unit 1122, the recognition data selecting unit 1123, the recognition result providing unit 1124, and the model updating unit 1125 are included in the electronic device, and some of the remaining units are included in the electronic device. May be included in the server.

In addition, at least one of the data acquirer 1121, the preprocessor 1122, the recognition data selector 1123, the recognition result provider 1124, or the model updater 1125 may be implemented as a software module. At least one of the data acquisition unit 1121, the preprocessor 1122, the recognition data selection unit 1123, the recognition result providing unit 1124, and the model updating unit 1125 includes a software module (or instruction). If implemented as a program module, the software module may be stored in a computer readable non-transitory computer readable media. In this case, at least one software module may be provided by an operating system (OS) or by a predetermined application. Alternatively, some of the at least one software module may be provided by an operating system (OS) and others may be provided by a given application.

In general, a user terminal providing the text-to-speech synthesis system and text-to-speech service described herein includes a wireless telephone, a cellular telephone, a laptop computer, a wireless multimedia device, a wireless communications personal computer (PC) card, a PDA, It may represent various types of devices, such as an external modem, an internal modem, a device communicating over a wireless channel, and the like. The device may be an access terminal (AT), an access unit, a subscriber unit, a mobile station, a mobile device, a mobile unit, a mobile telephone, a mobile, a remote station, a remote terminal, a remote unit, a user device, user equipment, It may have various names, such as a handheld device. Any device described herein may have memory for storing instructions and data, as well as hardware, software, firmware, or combinations thereof.

The techniques described herein may be implemented by various means. For example, these techniques may be implemented in hardware, firmware, software, or a combination thereof. Those skilled in the art will further appreciate that the various illustrative logical blocks, modules, circuits, and algorithm steps described in connection with the disclosure herein may be implemented in electronic hardware, computer software, or a combination of both. To clearly illustrate this interchangeability of hardware and software, various illustrative components, blocks, modules, circuits, and steps have been described above generally in terms of their functionality. Whether such functionality is implemented as hardware or software depends on the particular application and design constraints imposed on the overall system. Skilled artisans may implement the described functionality in varying ways for each particular application, but such implementation decisions should not be interpreted as causing a departure from the scope of the present disclosure.

In a hardware implementation, the processing units used to perform the techniques may include one or more ASICs, DSPs, digital signal processing devices (DSPDs), programmable logic devices (PLDs) ), Field programmable gate arrays (FPGAs), processors, controllers, microcontrollers, microprocessors, electronic devices, other electronic units designed to perform the functions described herein. May be implemented within a computer, or a combination thereof.

Accordingly, various illustrative logic blocks, modules, and circuits described in connection with the disclosure herein may be used in general purpose processors, DSPs, ASICs, FPGAs or other programmable logic devices, discrete gate or transistor logic, discrete hardware components, Or may be implemented or performed in any combination of those designed to perform the functions described herein. A general purpose processor may be a microprocessor, but in the alternative, the processor may be any conventional processor, controller, microcontroller, or state machine. A processor may also be implemented as a combination of computing devices, eg, a combination of a DSP and a microprocessor, a plurality of microprocessors, one or more microprocessors in conjunction with a DSP core, or any other such configuration.

In firmware and / or software implementations, the techniques may include random access memory (RAM), read-only memory (ROM), non-volatile random access memory (NVRAM), PROM ( On computer readable media such as programmable read-only memory (EPROM), erasable programmable read-only memory (EPROM), electrically erasable PROM (EEPROM), flash memory, compact disc (CD), magnetic or optical data storage devices, and the like. It may also be implemented as stored instructions. The instructions may be executable by one or more processors, and may cause the processor (s) to perform certain aspects of the functionality described herein.

If implemented in software, the functions may be stored on or transmitted over as one or more instructions or code on a computer-readable medium. Computer-readable media includes both computer storage media and communication media including any medium that facilitates transfer of a computer program from one place to another. Storage media may be any available media that can be accessed by a computer. By way of non-limiting example, such computer-readable media may be in the form of RAM, ROM, EEPROM, CD-ROM or other optical disk storage, magnetic disk storage or other magnetic storage devices, or desired program code in the form of instructions or data structures. Or any other medium that can be used for transfer or storage to a computer and that can be accessed by a computer. Also, any connection is properly termed a computer readable medium.

For example, if the software is transmitted from a website, server, or other remote source using coaxial cable, fiber optic cable, twisted pair, digital subscriber line (DSL), or wireless technologies such as infrared, wireless, and microwave, the coaxial cable , Fiber optic cable, twisted pair, digital subscriber line, or wireless technologies such as infrared, wireless, and microwave are included within the definition of the medium. As used herein, disks and disks include CDs, laser disks, optical disks, digital versatile discs, floppy disks, and Blu-ray disks, where the disks are usually Magnetically reproduce the data, while discs discs optically reproduce the data using a laser. Combinations of the above should also be included within the scope of computer-readable media.

The software module may reside in RAM memory, flash memory, ROM memory, EPROM memory, EEPROM memory, registers, hard disk, removable disk, CD-ROM, or any other type of storage medium known in the art. An exemplary storage medium may be coupled to the processor such that the processor can read information from or write information to the storage medium. In the alternative, the storage medium may be integral to the processor. The processor and the storage medium may reside in an ASIC. The ASIC may be present in the user terminal. In the alternative, the processor and the storage medium may reside as discrete components in a user terminal.

The previous description of the disclosure is provided to enable a person skilled in the art to make or use the disclosure. Various modifications of the disclosure will be readily apparent to those skilled in the art, and the generic principles defined herein may be applied to various modifications without departing from the spirit or scope of the disclosure. Thus, the present disclosure is not intended to be limited to the examples described herein but is to be accorded the widest scope consistent with the principles and novel features disclosed herein.

Although the example implementations may refer to utilizing aspects of the presently disclosed subject matter in the context of one or more standalone computer systems, the subject matter is not so limited, but rather in connection with any computing environment, such as a network or a distributed computing environment. It may be implemented. Moreover, aspects of the presently disclosed subject matter may be implemented in or across a plurality of processing chips or devices, and storage may be similarly affected across a plurality of devices. Such devices may include PCs, network servers, and handheld devices.

Although the subject matter has been described in language specific to structural features and / or methodological acts, it will be understood that the subject matter defined in the appended claims is not necessarily limited to the specific features or acts described above. Rather, the specific features and acts described above are described as example forms of implementing the claims.

Although the method referred to in this specification has been described with reference to specific embodiments, it is possible to implement it as computer readable code on a computer readable recording medium. Computer-readable recording media include all kinds of recording devices that store data that can be read by a computer system. Examples of computer-readable recording media include ROM, RAM, CD-ROM, magnetic tape, floppy disks, optical data storage devices, and the like. The computer readable recording medium can also be distributed over network coupled computer systems so that the computer readable code is stored and executed in a distributed fashion. And, the functional program, code and code segments for implementing the embodiments can be easily inferred by programmers in the art to which the present invention belongs.

While the present disclosure has been described in connection with some embodiments, it is to be understood that various modifications and changes can be made without departing from the scope of the present disclosure to those skilled in the art. something to do. Also, such modifications and variations are intended to fall within the scope of the claims appended hereto.

Claims

A text-to-speech synthesis method using machine learning based on sequential rhyme characteristics,

Receiving input text;

Receiving a sequential prosody feature; And

Inputting the input text and the received sequential rhyme features into an artificial neural network text-voice synthesis model to generate output speech data for the input text reflecting the received sequential rhyme features;

Text-voice synthesis method comprising a.
The method of claim 1,

The artificial neural network text-voice synthesis model is generated by performing machine learning based on a plurality of learning texts and data representing learning speech corresponding to the plurality of learning texts,

And the data indicative of the learning speech comprises a sequential rhyme feature of the learning speech.
The method of claim 1,

The sequential rhyme feature includes rhyme information corresponding to at least one unit of a frame, letter, phoneme, syllable, or word in chronological order,

The rhyme information includes at least one of information on a loudness of the sound, information on the height of the sound, information on the length of the sound, information on the pause period of the sound, or information on the style of the sound. , Text-to-speech synthesis method.
The method of claim 3, wherein receiving the sequential rhyme feature comprises receiving a plurality of embedding vectors representing the sequential rhyme feature.

Each of the plurality of embedding vectors corresponds to rhyme information included in the chronological order.
The method of claim 4, wherein

The neural network text-voice synthesis model comprises an encoder and a decoder,

Inputting the received plurality of embedding vectors into an attention module to generate a plurality of transform embedding vectors corresponding to respective portions of the input text provided to the encoder, wherein the length of the plurality of transform embedding vectors is equal to the length of the input text. Variable depending on length;

Generating output voice data for the input text may include:

Inputting the generated plurality of transform embedding vectors into an encoder of the neural network text-to-speech synthesis model; And

Generating output speech data for the input text in which the plurality of transform embedding vectors are reflected.

Comprising a text-to-speech synthesis method.
The method of claim 4, wherein

The neural network text-voice synthesis model comprises an encoder and a decoder,

Generating output voice data for the input text may include:

Inputting the received plurality of embedding vectors into a decoder of the neural network text-to-speech synthesis model; And

Generating output speech data for the input text reflecting the plurality of embedding vectors

Comprising a text-to-speech synthesis method.
The method of claim 4, wherein

Receiving a speaker's utterance feature,

Generating output speech data for the input text includes generating output speech data for the input text that mimics the speaker's speech and reflects a plurality of embedding vectors representing the sequential rhyme characteristics. Speech synthesis method.
The method of claim 7, wherein

Receiving the speaker's utterance feature comprises receiving a sequential rhyme feature of the speaker,

The extracting of the plurality of embedding vectors may include normalizing the extracted plurality of embedding vectors based on the sequential rhyme characteristic of the speaker.

Generating output speech data for the input text comprises simulating speech of the speaker and generating output speech data for the input text reflecting the normalized plurality of embedding vectors. .
The method of claim 8,

Normalizing the extracted plurality of embedding vectors may include:

Calculating an average value of an embedding vector representing the sequential rhyme characteristic of the speaker at each time step; And

Subtracting the extracted plurality of embedding vectors by an average value of the embedding vectors calculated in each time step.

Comprising a text-to-speech synthesis method.
The method of claim 1,

Receiving the sequential rhyme feature, receiving rhyme information for at least a portion of the input text via a user interface,

Generating output speech data for the input text in which the received sequential rhyme feature is reflected includes generating output speech data for the input text in which rhyme information for at least a portion of the input text is reflected. -Speech synthesis method.
The method of claim 10,

Rhyme information for at least a portion of the input text is input through a tag provided in a speech synthesis markup language,

Text-to-speech synthesis method.
The method of claim 1,

Receiving rhyme information on at least a portion of the input text through a user interface; And

Changing the received sequential rhyme feature based on rhyme information for at least a portion of the received input text,

Generating output speech data for the input text that reflects the received sequential rhyme feature includes generating output speech data for the input text that reflects the changed sequential rhyme feature. .
The method of claim 12,

Rhyme information for at least a portion of the input text, used to change the received sequential rhyme feature, is input via a tag provided in a speech synthesis markup language.
A computer-readable storage medium having recorded thereon a program comprising instructions for performing each step according to the method of text-to-speech synthesis using machine learning using the sequential rhyme feature of claim 1.