KR20230043084A - Method and computer readable storage medium for performing text-to-speech synthesis using machine learning based on sequential prosody feature - Google Patents

Abstract

The present disclosure relates to a method for synthesizing text-to-speech using machine learning based on a sequential prosody feature. The text-to-speech synthesis using machine learning based on the sequential prosody feature comprises: a step of receiving input text; a step of receiving a sequential prosody feature; and a step of generating output speech data for an input text reflecting the received sequential prosody feature by inputting the input text and the received sequential prosody feature into an artificial neural network text-to-speech synthesis model. Therefore, the present invention is capable of transmitting more accurately an intention or emotion of a person.

Description

Text-to-speech synthesis method, device and computer readable storage medium using machine learning based on sequential prosody features

The present disclosure relates to a method and system for text-to-speech synthesis using machine learning based on sequential prosodic features. More specifically, it relates to a method and system for generating output speech text for input text reflecting the sequential prosody features by inputting sequential prosody features to an artificial neural network text-speech synthesis model.

Voice synthesis technology, commonly called text-to-speech (TTS), is a technology that requires human voices, such as announcements, navigation, and artificial intelligence assistants, without pre-recording actual voices. It is a technology used to reproduce voice. A typical method of speech synthesis is a concatenative TTS method in which speech is cut in advance into very short units such as phonemes and stored, and speech is synthesized by combining phonemes constituting a sentence to be synthesized, and speech characteristics are expressed as parameters. There is a parameter synthesis method (parametric TTS) in which parameters representing voice features constituting a sentence to be synthesized are synthesized into a voice corresponding to the sentence using a vocoder.

Meanwhile, recently, an artificial neural network-based voice synthesis method has been actively researched, and a voice synthesized according to such a voice synthesis method includes natural voice characteristics compared to conventional methods. However, in the conventional voice synthesis method, only a prosody feature of a fixed length is applied regardless of the length of the input text or the length of the reference voice, so that the prosody of the synthesized voice at a specific time point cannot be controlled. The reason is that the probability of loss of information in time is very high when a feature of a fixed length is forcibly applied to a reference voice. Accordingly, conventional voice synthesis methods cannot provide fine prosody control for synthesized voices in order to accurately represent people's intentions or emotions.

In addition, when the difference between the pitch range of the source speaker and the pitch of the target speaker is large, it may be difficult to reflect the prosody characteristics of the source speaker to the target speaker. For example, when the source speaker is a woman and the target speaker is a man, if the source speaker's rhyme is synthesized with the target speaker's voice, the synthesized voice of the target speaker can have a higher pitch than the normal pitch. . Considering these circumstances, it may be required to pre-process the prosody features before applying the prosody features to the artificial neural network model in order to improve the quality of synthesized speech reflecting the prosody features.

The method and apparatus according to the present disclosure may generate output voice data for input text in which sequential prosody characteristics having prosody characteristics according to time are reflected in order to solve the above problems.

In addition, in the method and apparatus according to the present disclosure, sequential prosody features may be input to at least one of an encoder and a decoder of an artificial neural network text-speech synthesis model, and sequential prosody features of variable length may be input to the length and/or synthesis of input text. An attention module may be used to match the length of the voice.

In addition, the method and apparatus according to the present disclosure may normalize a plurality of embedding vectors corresponding to sequential prosody features and apply the normalized plurality of embedding vectors to an artificial neural network text-speech synthesis model.

The present disclosure may be implemented in a variety of ways, including a method, system, apparatus or computer readable storage medium storing instructions.

A text-to-speech synthesis method using machine learning based on sequential prosody features according to an embodiment of the present disclosure includes receiving input text and receiving sequential prosody features. and inputting the input text and the received sequential prosody features into an artificial neural network text-speech synthesis model to generate output voice data for the input text reflecting the received sequential prosody features.

An artificial neural network text-speech synthesis model of a text-speech synthesis method using machine learning based on sequential prosody features according to an embodiment of the present disclosure includes data representing a plurality of learning texts and learning voices corresponding to the plurality of learning texts. Data that is generated by performing machine learning based on and representing the learning voice may include sequential prosody features of the learning voice.

The sequential prosody feature of the text-speech synthesis method using machine learning based on the sequential prosody feature according to an embodiment of the present disclosure is a prosody corresponding to at least one unit of a frame, a character, a phoneme, a syllable, or a word. information in chronological order, and the prosody information includes at least one of information about loudness, information about pitch, information about length of sound, information about pause period of sound, or information about style of sound can include

The step of receiving the sequential prosody features of the text-speech synthesis method using machine learning based on the sequential prosody features according to an embodiment of the present disclosure includes receiving a plurality of embedding vectors representing the sequential prosody features, Each of the plurality of embedding vectors may correspond to prosody information included in chronological order.

An artificial neural network text-speech synthesis model of a text-speech synthesis method using machine learning based on sequential prosody features according to an embodiment of the present disclosure includes an encoder and a decoder, and includes an encoder and a decoder, and includes text using machine learning based on sequential prosody features. -The speech synthesis method further comprises generating a plurality of transform embedding vectors corresponding to respective parts of the input text provided to the encoder by inputting the received plurality of embedding vectors to an attention module, and generating a plurality of transform embedding vectors of the plurality of transform embedding vectors The length is variable according to the length of the input text, and the step of generating output voice data for the input text includes inputting the generated plurality of transform embedding vectors to an encoder of an artificial neural network text-speech synthesis model and a plurality of transform embeddings. It may include generating output voice data for the input text to which the vector is reflected.

An artificial neural network text-speech synthesis model of a text-speech synthesis method using machine learning based on sequential prosody features according to an embodiment of the present disclosure includes an encoder and a decoder, and generates output voice data for input text. may include inputting a plurality of received embedding vectors to a decoder of an artificial neural network text-speech synthesis model and generating output speech data for input text in which the plurality of embedding vectors are reflected.

A text-to-speech synthesis method using machine learning based on sequential prosody features according to an embodiment of the present disclosure further includes receiving vocal characteristics of a speaker, and generating output voice data for input text. and generating output voice data for the input text in which a plurality of embedding vectors representing sequential prosody characteristics are reflected.

The step of receiving vocalization characteristics of a speaker of a text-to-speech synthesis method using machine learning based on sequential prosody characteristics according to an embodiment of the present disclosure includes receiving sequential prosody characteristics of the speaker, and includes a plurality of embedding vectors. The step of extracting includes normalizing the plurality of embedding vectors extracted based on the speaker's sequential prosody features, and the step of generating output speech data for the input text simulates the speaker's voice and normalizes the plurality of embedding vectors. It may include generating output voice data for the input text to which the vector is reflected.

The step of normalizing the extracted plurality of embedding vectors of the text-speech synthesis method using machine learning based on the sequential prosody features according to an embodiment of the present disclosure includes: Calculating an average value and subtracting the plurality of extracted embedding vectors by the average value of the embedding vectors calculated at each time step may be included.

Receiving sequential prosody features of a text-to-speech synthesis method using machine learning based on sequential prosody features according to an embodiment of the present disclosure includes receiving prosodic information on at least a part of input text through a user interface. And, generating output voice data for the input text in which the received sequential prosody characteristics are reflected may include generating output voice data for the input text in which prosody information for at least a part of the input text is reflected. there is.

According to an embodiment of the present disclosure, prosody information for at least a part of the input text may be input through a tag provided in a speech synthesis markup language.

A text-to-speech synthesis method using machine learning based on sequential prosody features according to an embodiment of the present disclosure includes receiving prosody information for at least a portion of input text through a user interface, and performing the steps of receiving prosody information on at least a portion of the received input text. The method may further include changing the received sequential prosody characteristics based on prosody information for the received sequential prosody, and generating output voice data for the input text to which the received sequential prosody characteristics are reflected includes the input text to which the changed sequential prosody characteristics are reflected. It may include generating output voice data for.

According to an embodiment of the present disclosure, prosody information for at least a part of the input text, which is used to change the received sequential prosody characteristic, may be input through a tag provided in a speech synthesis markup language.

In addition, a program for implementing a text-speech synthesis method using machine learning based on sequential prosody characteristics as described above may be recorded on a computer-readable recording medium.

In addition, devices and technical means related to the text-to-speech synthesis method using machine learning based on sequential prosody characteristics as described above may also be disclosed.

According to some embodiments of the present disclosure, since text-to-speech synthesis using machine learning is provided based on a sequential prosody feature that includes prosody information over time and has a variable length, fine prosodic control over synthesized speech is provided. is possible, so that a person's intention or emotion can be more accurately conveyed through voice synthesis.

According to some embodiments of the present disclosure, in applying sequential prosody features of variable length to at least one of an encoder and a decoder of an artificial neural network text-speech synthesis model, attention is used to synthesize sequential prosody features into input text and/or Since it can be adjusted to correspond to the length of voice, sequential prosody features of variable length can be effectively applied to input text and/or synthesized voice regardless of its length.

According to some embodiments of the present disclosure, prior to applying the sequential prosody features to the artificial neural network text-speech synthesis model, since preprocessing of normalizing a plurality of embedding vectors corresponding to the sequential prosody features is performed, the prosody features of one person When applied to another person's synthesized voice, the quality of the synthesized voice in which prosody characteristics are reflected can be further improved.

Embodiments of the present disclosure will be described with reference to the accompanying drawings described below, wherein like reference numbers indicate like elements, but are not limited thereto.
1 is an exemplary diagram illustrating a process of outputting a synthesized voice by receiving input text and sequential prosody characteristics by a voice synthesizer according to an embodiment of the present disclosure.
2 is an exemplary diagram illustrating a process of outputting a synthesized voice using sequential prosody features extracted from a sequential prosody feature extractor and input text by a voice synthesizer according to an embodiment of the present disclosure.
3 is an exemplary diagram illustrating a process of outputting a synthesized voice by applying sequential prosody characteristics and vocalization characteristics of a speaker to input text by a voice synthesizer according to an embodiment of the present disclosure.
4 is a block diagram of a text-to-speech synthesis system according to an embodiment of the present disclosure.
5 is a flowchart illustrating a text-to-speech synthesis method using machine learning based on sequential prosody characteristics according to an embodiment of the present disclosure.
6 is an exemplary diagram illustrating the configuration of a text-speech synthesis system based on an artificial neural network according to an embodiment of the present disclosure.
7 is an exemplary diagram illustrating a process of generating synthesized speech by inputting sequential prosody features to a decoder of the text-speech synthesis system based on an artificial neural network according to an embodiment of the present disclosure.
8 is an exemplary diagram illustrating a process of generating a synthesized voice by inputting sequential prosody characteristics to an encoder of the text-to-speech synthesis system based on an artificial neural network according to an embodiment of the present disclosure.
9 is an exemplary diagram illustrating a network of sequential prosody feature extractors configured to extract a plurality of embedding vectors representing sequential prosody features from a speech signal or sample according to an embodiment of the present disclosure.
10 is a schematic diagram of a text-to-speech synthesis system for outputting a synthesized voice by applying a tag provided in a markup language to input text according to an embodiment of the present disclosure.
11 is a block diagram of a text-to-speech synthesis system according to an embodiment of the present disclosure.

Advantages and features of the disclosed embodiments, and methods of achieving them, will become apparent with reference to the following embodiments in conjunction with the accompanying drawings. However, the present disclosure is not limited to the embodiments disclosed below and may be implemented in various different forms, only the present embodiments make the present disclosure complete, and those of ordinary skill in the art to which the present disclosure belongs It is provided only to fully inform the person of the scope of the invention.

Terms used in this specification will be briefly described, and the disclosed embodiments will be described in detail.

The terms used in this specification have been selected from general terms that are currently widely used as much as possible while considering the functions in the present disclosure, but they may vary according to the intention of a person skilled in the related field, a precedent, or the emergence of new technologies. In addition, in a specific case, there is also a term arbitrarily selected by the applicant, and in this case, the meaning will be described in detail in the description of the invention. Therefore, terms used in the present disclosure should be defined based on the meaning of the term and the general content of the present disclosure, not simply the name of the term.

Expressions in the singular number in this specification include plural expressions unless the context clearly dictates that they are singular. Also, plural expressions include singular expressions unless the context clearly specifies that they are plural.

When it is said that a certain part 'includes' a certain element in the entire specification, it means that other elements may be further included without excluding other elements unless otherwise stated.

Also, the term 'unit' or 'module' used in the specification means a software or hardware component, and the 'unit' or 'module' performs certain roles. However, 'unit' or 'module' is not meant to be limited to software or hardware. A 'unit' or 'module' may be configured to reside in an addressable storage medium and may be configured to reproduce one or more processors. Thus, as an example, 'unit' or 'module' includes components such as software components, object-oriented software components, class components and task components, processes, functions, properties, procedures, subroutines, segments of program code, drivers, firmware, microcode, circuitry, data, databases, data structures, tables, arrays and variables. Functions provided within components and 'units' or 'modules' may be combined into smaller numbers of components and 'units' or 'modules', or may be combined into additional components and 'units' or 'modules'. can be further separated.

According to an embodiment of the present disclosure, a 'unit' or 'module' may be implemented as a processor and a memory. The term 'processor' should be interpreted broadly to include general-purpose processors, central processing units (CPUs), microprocessors, digital signal processors (DSPs), controllers, microcontrollers, state machines, and the like. In some circumstances, 'processor' may refer to an application specific integrated circuit (ASIC), programmable logic device (PLD), field programmable gate array (FPGA), or the like. The term 'processor' refers to a combination of processing devices, such as, for example, a combination of a DSP and a microprocessor, a combination of a plurality of microprocessors, a combination of one or more microprocessors in conjunction with a DSP core, or any other such configuration. may also refer to

The term 'memory' should be interpreted broadly to include any electronic component capable of storing electronic information. The term memory includes random access memory (RAM), read-only memory (ROM), non-volatile random access memory (NVRAM), programmable read-only memory (PROM), erasable-programmable read-only memory (EPROM), electrical may refer to various types of processor-readable media, such as erasable PROM (EEPROM), flash memory, magnetic or optical data storage, registers, and the like. A memory is said to be in electronic communication with the processor if the processor can read information from and/or write information to the memory. Memory integrated with the processor is in electronic communication with the processor.

In the present disclosure, 'sequential prosody features' may include prosody information corresponding to at least one unit of a frame, a phoneme, a letter, a syllable, or a word in chronological order. Here, the prosody information may include at least one of information about loudness, information about pitch, information about length of sound, information about a pause period of sound, or information about style of sound. In addition, the sound style may include any form, manner, or nuance expressed by the sound or voice, and may include, for example, tone, intonation, emotion, etc. inherent in the sound or voice. In addition, sequential prosody features may be represented by a plurality of embedding vectors, and each of the plurality of embedding vectors may correspond to prosody information included in chronological order.

Hereinafter, with reference to the accompanying drawings, embodiments will be described in detail so that those skilled in the art can easily carry out the embodiments. And in order to clearly describe the present disclosure in the drawings, parts irrelevant to the description are omitted.

1 is an exemplary diagram illustrating a process of outputting a synthesized voice 140 by receiving input text 120 and sequential prosody features 130 by a voice synthesizer 110 according to an embodiment of the present disclosure. The voice synthesizer 110 may be configured to output a synthesized voice corresponding to input text using an artificial neural network text-speech synthesis model. Here, the artificial neural network text-speech synthesis model may be a single artificial neural network text-speech synthesis model. In one embodiment, the voice synthesizer 110 may correspond to the data recognizer 455 of FIG. 4 and/or the data recognizer 1020 of FIG. 10 . Also, the voice synthesizer 110 may be included or provided in a user terminal or a text-to-speech synthesis system.

According to one embodiment, text input to the voice synthesizer 110 may include text received through an arbitrary interface (not shown). According to another embodiment, the voice recognizer (not shown) may receive a specific voice, convert it into text corresponding to the input voice, and provide the converted text to the voice synthesizer 110 as input text. Accordingly, as shown in FIG. 1 , the voice synthesizer 110 may receive the text 'HELLO' as a text input through an interface or a voice recognizer.

According to one embodiment, voice synthesizer 110 may be configured to receive sequential prosodic features. Here, the sequential prosody feature may include prosody information of each time unit according to a predetermined time unit. As shown in FIG. 1 , the sequential prosody feature may include information about the pitch of a sound, and may include, for example, information indicating a pitch according to time, '11113'. According to one embodiment, these sequential prosodic features may be extracted or determined from any extractor capable of extracting prosodic features for sounds, for example extracted from a pitch tracker. According to another embodiment, the voice synthesizer 110 may receive arbitrary information representing sequential prosody information for sounds, for example, information representing musical scores. According to another embodiment, the speech synthesizer 110 receives sequential prosodic features corresponding to attribute values expressed in a speech synthesis markup language according to time for text input from an arbitrary device. can do. These attribute values will be described in detail with reference to FIG. 9 below.

Speech synthesizer 110 may be configured to generate output data for input text in which the received sequential prosody characteristics are reflected. To this end, the voice synthesizer 110 may apply prosody information in chronological order represented by sequential prosody features to the input text. For example, as shown in FIG. 1 , the voice synthesizer 110 may generate output voice data by reflecting '11113', which is information indicating a pitch according to time, to the input text 'HELLO'. That is, the voice synthesizer 110 may generate an output voice corresponding to 'HELLO?', an interrogative text in which the pitch of 'o', the last character of the input text, is higher than that of other characters. The generated voice may be output through an output device such as a speaker or transmitted to another device having an I/O device.

FIG. 2 outputs a synthesized voice 240 using the sequential prosody features 210 and the input text 120 extracted from the sequential prosody feature extractor 230 by the speech synthesizer 110 according to an embodiment of the present disclosure. It is an example diagram showing the process. In one embodiment, the sequential prosody feature extractor 230 may correspond to the sequential prosody feature extractor 410 of FIG. 4 . Since the input text 120 and the voice synthesizer 110 have been described with reference to FIG. 1, duplicate descriptions are omitted.

According to an embodiment, the sequential prosody feature extractor 230 may receive a voice signal or voice sample 220 and extract a sequential prosody feature 210 from the received voice signal or sample. Here, the received voice signal or sample may include voice spectrum data representing information related to the sequential prosody characteristics 210, and may include, for example, a melody or a specific speaker's voice.

According to one embodiment, in extracting sequential prosody features 210, any known suitable feature extraction method capable of extracting sequential prosody features 210 from a speech signal or sample 220 may be used. According to one embodiment, an artificial neural network or machine learning model may be used to extract sequential prosodic features. For example, the artificial neural network or machine learning model used in the sequential prosody feature extractor 230 includes a recurrent neural network (RNN), a long short-term memory model (LSTM), a deep neural network (DNN), and a convolution neural network (CNN). network), etc., and may be composed of any one or a combination of various artificial neural network models.

The sequential prosody feature extractor 230 may extract a plurality of feature vectors (embedding vectors) representing the sequential prosody features 210 by inputting the received voice signal or voice sample to an artificial neural network prosody feature model. Here, each of the plurality of embedding vectors may correspond to a predetermined time unit (eg, frame, phoneme, letter, syllable, or word). For example, this vector may include one of various speech feature vectors such as mel frequency cepstral coefficient (MFCC), linear predictive coefficients (LPC), perceptual linear prediction (PLP), etc., but is not limited thereto. In addition, since the plurality of embedding vectors extracted in this way include prosody features or information in chronological order, the lengths of these vectors may be variable or different depending on the length of the input voice sample.

The voice synthesizer 110 may generate voice output data reflecting the sequential prosody features 210 extracted from the sequential prosody feature extractor 230 in the received text 120 . For example, the voice synthesizer 110 inputs the embedding information corresponding to the input 'HELLO' text and the plurality of embedding vectors extracted by the sequential prosody feature extractor 230 to an artificial neural network text-speech synthesis model to sequentially 'HELLO' voice data reflecting prosody characteristics may be generated. The voice thus generated may be output through an output device such as a speaker or transmitted to another device having an I/O device. FIG. 3 shows sequential prosody characteristics 210 and vocalization characteristics of a speaker according to an embodiment of the present disclosure. 330 is a schematic diagram of a speech synthesizer 110 that applies input text 120 to output synthesized speech. As shown in FIG. 3 , voice synthesizer 110 may receive input text 120 , sequential prosody features 210 and vocalization features 330 of a speaker. Here, the sequential prosody feature 210 can be extracted from the sequential prosody feature extractor 230 based on the voice signal or voice sample 220, and the speaker's vocalization feature 330 is based on the voice signal or voice sample 320. Based on this, it can be extracted from the speaker's vocalization feature extractor 310. According to an embodiment, the voice signal or voice sample 220 input to the sequential prosody feature extractor 230 may be different from the voice signal or voice sample 320 input to the vocalization feature extractor 310 . In another embodiment, the two voice signals or voice samples 220 and 320 may be identical to each other. Since the voice synthesizer 110, the input text 120, the sequential prosody feature 210, the voice signal or voice sample 220, and the sequential prosody feature extractor 230 have been described with reference to FIGS. 1 and 2, redundancy description is omitted.

The vocalization feature extractor 310 may be configured to extract a speaker's vocalization characteristics from voice data. The vocalization characteristics of the speaker may include at least one of various factors such as style, prosody, emotion, timbre, and pitch that may constitute the vocalization as well as copying the speaker's voice. According to one embodiment, the speaker's vocalization characteristics may include a one-hot speaker ID-vector representing the speaker. According to another embodiment, the speech characteristics of the speaker may include an embedding vector representing the speech characteristics of the speaker. In one embodiment, the vocalization feature extractor 310 may correspond to the vocalization feature extractor 415 of FIG. 4 .

The voice synthesizer 110 may generate an output voice 340 by inputting the input text 120, the sequential prosody characteristics 210, and the speaker's speech characteristics 330 to an artificial neural network text-to-speech synthesis model. The output voice 340 may include output voice data for the input text 120 in which the sequential prosody characteristics 210 and the vocalization characteristics 330 of the speaker are reflected. That is, the output voice 340 mimics the speaker's voice based on the speaker's speech characteristics and reflects the sequential prosody feature 210, so that the input text 120 is converted into the sequential prosody feature 210 input by the corresponding speaker. It may be data synthesized into a speaking voice. For example, when the sequential prosody features 210 and the speaker's phonation features 330 are extracted from the voices of a first speaker and a second speaker different from the first speaker, respectively, the second speaker is the voice of the second speaker at the time of the second speaker. A voice saying 'HELLO' may be output based on the prosody information according to . The generated voice may be output through an output device such as a speaker or transmitted to another device having an I/O device.

2 and 3, the voice synthesizer 110 is shown to receive a plurality of embedding vectors over time representing the sequential prosody features 210 extracted from the sequential prosody feature extractor 230, but is not limited thereto, and the voice synthesizer 110 is not limited thereto. The synthesizer 110 may receive input values for a plurality of embedding vectors according to time representing the sequential prosodic features 210 through an I/O device (not shown). Alternatively, a plurality of embedding vectors according to time representing the sequential prosody features 210 may be previously stored in a storage medium (not shown), and the voice synthesizer 110 accesses the storage medium and receives the plurality of embedding vectors. can In addition, correction information for the plurality of embedding vectors extracted or stored may be received through an I/O device, the plurality of embedding vectors may be corrected according to the received correction information, and the corrected plurality of embedding vectors may be transmitted to the voice synthesizer. It can be received at (110).

In addition, although it is shown in FIG. 3 that the vocalization feature 330 of the speaker is extracted from the vocalization feature extractor 310 and provided to the voice synthesizer 110, it is not limited thereto, and the voice synthesizer 110 extracts the voice characteristics An input value for the indicated embedding vector may be received through an I/O device (not shown). Alternatively, embedding vectors representing vocalization characteristics may be previously stored in a storage medium (not shown), and the voice synthesizer 110 may access the storage medium and receive the embedding vectors representing vocalization characteristics. In addition, correction information on the voice characteristics extracted or stored in this way may be received through an I/O device, an embedding vector representing the voice characteristics may be modified according to the received information, and the embedding vector representing the modified voice characteristics may be modified. It may be received by the voice synthesizer 110. 4 is a block diagram of a text-to-speech synthesis system 400 according to an embodiment of the present disclosure. The text-to-speech synthesis system 400 includes a communication unit 405, a sequential prosody feature extraction unit 410, a vocalization feature extraction unit 415, a normalizer 420, a speech database 425, an attention module 430, an encoder. 435, a decoder 440, a post-processor 445, a data learning unit 450, and a data recognizing unit 455. The communication unit 405 may be configured to allow the text-to-speech synthesis system 400 to transmit/receive signals or data with an external device. The external device may include a user terminal providing a text-to-speech synthesis service. Alternatively, the external device may include other text-to-speech synthesis systems. Alternatively, the external device may be any device including a voice database.

According to one embodiment, the communication unit 405 may be configured to receive text from an external device. Here, the text may include training text to be used for learning the artificial neural network text-speech synthesis model. Alternatively, text may include input text to be used to generate synthesized speech through an artificial neural network text-speech synthesis model. Such text may be provided to at least one of the voice database 425 , the encoder 435 , the decoder 440 , the data learner 450 , and the data recognizer 455 .

The communication unit 405 may be configured to receive a voice signal or a voice sample through an external device. According to an embodiment, the voice signal or sample may be transmitted to the sequential prosody feature extraction unit 410, and sequential prosody features may be extracted from the voice signal or sample. According to another embodiment, the speech signal or sample may be transmitted to the speech feature extraction unit 415, and the speech feature of the speaker may be extracted from the speech signal or sample. The extracted sequential prosody features and/or vocalization features of the speaker are transmitted to the encoder 435 and/or decoder 440 through the data learning unit 450 and used to learn an artificial neural network text-speech synthesis model. . In contrast, the sequential prosody features and/or vocalization features of the speaker extracted in this way are transmitted to the encoder 435 and/or decoder 440 through the data recognition unit 455 to generate a synthesized voice from the artificial neural network text-to-speech synthesis model. can be used to create

In one embodiment, the communication unit 405 may receive sequential prosody characteristics from an external device. For example, the text-to-speech synthesis system 400 may receive sequential prosody features extracted through the sequential prosody feature extractor 230 of FIG. 2 through the communication unit 405 . In another embodiment, the communication unit 405 may receive vocal characteristics of a speaker from an external device. For example, the communication unit 405 may transmit/receive the speaker's vocalization feature 330 from the speaker's vocalization feature extractor 310 of FIG. 3 . The sequential prosody characteristics and/or vocalization characteristics of the speaker received in this way are obtained by the normalizer 420, the speech database 425, the attention module 430, the encoder 435, the decoder 440, the data learning unit 450, or the data learning unit 450. It may be provided to at least one of the recognition units 455 .

In one embodiment, the communication unit 405 may receive prosody information on the input text as a sequential prosody feature from an external device. Here, the prosody information may include an attribute value input through a tag provided in a speech synthesis markup language for each part (eg, phonemes, letters, syllables, words, etc.) of the input text.

According to an embodiment, the communication unit 405 may transmit information related to the generated output voice, that is, output voice data to an external device. In addition, the generated artificial neural network text-speech synthesis model may be transmitted to a user terminal or other text-speech synthesis system through the communication unit 405 .

In FIG. 4, the text-to-speech synthesis system 400 receives text, voice signals or samples, sequential prosody characteristics, and speech characteristics of speakers through the communication unit 405, or the output voice data and the artificial neural network text-to-speech synthesis model are Although shown as being output through the communication unit 405, the text-to-speech synthesis system 400 may further include an input/output device (I/O device; not shown). Accordingly, the text-to-speech synthesis system 400 may directly receive an input from a user and may output at least one of text, voice, and video to the user.

The sequential prosody feature extraction unit 410 may be configured to receive a voice signal or sample through the communication unit 405 or an input/output device, and extract a sequential prosody feature from the received voice signal or sample. In one embodiment, the sequential prosody feature extractor 410 may correspond to the sequential prosody feature extractor 230 of FIGS. 2 and 3 . For example, the sequential prosody feature extractor 410 may extract a sequential prosody feature from a received voice signal or sample using a voice processing method such as Mel Frequency Cepstral (MFC). Alternatively, sequential prosody features may be extracted by inputting them to a prosody feature extraction model (eg, an artificial neural network) learned using voice samples. For example, sequential prosody features may be represented by a plurality of embedding vectors corresponding to a certain unit according to time. Here, the predetermined unit may correspond to at least one unit such as a frame, a phoneme, a letter, a syllable, a word, or a word. According to another embodiment, the sequential prosody feature extractor 410 may receive at least one of video, music, or sheet music information, and may be configured to extract sequential prosody features from the received video, music, and/or sheet music. can According to an embodiment, modification information for a plurality of embedding vectors representing sequential prosody characteristics may be received through an I/O device (not shown), and the plurality of embedding vectors may be modified through the received information.

The extracted or corrected sequential prosody features may be provided to the data learning unit 450 and/or the data recognizing unit 455 and provided to at least one of the encoder 414 and/or the decoder 440 . According to an embodiment, the sequential prosody features may be provided to the normalizer 420 and/or the attention module 430 before being provided to the data learning unit 450 and/or the data recognizing unit 455 . According to an embodiment, the sequential prosody features extracted from the sequential prosody feature extractor 410 may be stored in a storage medium (eg, voice database 425) or an external storage device. Accordingly, at the time of voice synthesis for the input text, one or more of a plurality of sequential prosody features previously stored in the storage medium may be selected or designated, and the selected or designated sequential prosody features may be used for voice synthesis.

The speech feature extraction unit 415 may be configured to receive a speaker's voice signal (eg, a voice sample) and extract the speaker's voice feature from the received voice signal. Here, the extracted vocalization feature simulates the speaker, may include any features included in the speaker's voice, and may be represented, for example, by a plurality of embedding vectors. In extracting the speech characteristics of the speaker, any known suitable feature extraction method capable of extracting speech characteristics from the speech signal of the speaker may be used. For example, the vocalization feature extraction unit 415 may extract a speaker's vocalization characteristics from a voice sample using an artificial neural network or a machine learning model. In one embodiment, the speaker's vocalization feature extraction unit 415 may correspond to the speaker's vocalization feature extractor 310 of FIG. 3 . The voice characteristics of the speaker extracted in this way may be provided to at least one of the data learning unit 450 , the data recognizing unit 455 , the encoder 435 , and the decoder 440 .

According to an embodiment, the vocalization characteristics of the speaker extracted from the vocalization feature extraction unit 415 may be stored in the voice database 425 or an external storage device. Accordingly, when synthesizing voices for input text, one or more of the speech characteristics of a plurality of speakers pre-stored in the storage medium may be selected or designated, and the speech characteristics of the selected or designated speakers may be used for speech synthesis.

According to one embodiment, the speaker's phonation characteristics may include the speaker's sequential prosodic characteristics. To this end, for example, the speaker's vocalization feature extraction unit 415 may be configured to extract sequential prosody features of the speaker from the voice sample. As another example, the speaker's vocalization feature extractor 415 may provide voice samples to the sequential prosody feature extractor 410 to receive the speaker's sequential prosody features extracted from the voice samples. The sequential prosody features of the speaker extracted in this way may be provided to the normalizer 420 . In FIG. 4 , the sequential prosody feature extraction unit 410 and the speaker's vocalization feature extraction unit 415 are shown as being configured as separate units, but are not limited thereto and may be configured as one unit.

The normalizer 420 may receive the sequential prosody features received from the sequential prosody feature extractor 410 and the sequential prosody features (a plurality of embedding vectors) of the speaker from the vocalization feature extractor 415 as the speaker's vocalization characteristics. . For explanation, the sequential prosody features received from the sequential prosody feature extractor 410 will be referred to as first sequential prosody features, and the sequential prosody features of the speaker from the vocalization feature extractor 415 will be referred to as second sequential prosody features. can

Normalizer 420 may be configured to normalize the first sequential prosody feature (eg, the plurality of embedding vectors) based on the second sequential prosody feature (eg, the plurality of embedding vectors). Here, the first sequential prosody feature may be a feature extracted from a speaker different from a speaker associated with the second sequential prosody feature. According to one embodiment, normalizer 420 may be configured to average a plurality of embedding vectors corresponding to second sequential prosody features at each time step. In addition, the normalizer 420 may normalize the plurality of embedding vectors representing the first sequential prosody feature by subtracting the plurality of embedding vectors representing the first sequential prosody feature by the average value of the embedding vectors calculated at each time step. . The plurality of normalized embedding vectors may be provided to at least one of the voice database 425, the attention module 430, the encoder 435, the decoder 440, the data learner 450, and the data recognizer 455. can Since the plurality of embedding vectors corresponding to the first sequential prosody feature are normalized using the average value of the plurality of embedding vectors corresponding to the second sequential prosody feature, the artificial neural network text-speech synthesis model is used to associate the second sequential prosody feature. When a synthesized voice corresponding to an arbitrary text is generated to mimic a speaker and reflect the first sequential prosody feature extracted from another speaker, the first sequential prosody feature can be more naturally applied to the voices of different speakers.

The voice database 425 may store learning texts and voices corresponding to a plurality of learning texts, and may be accessed by the learning unit 450 for learning texts and voice data corresponding thereto. The training text may be written in at least one language, and may include at least one of words, phrases, and sentences that can be understood by humans. Also, the voice stored in the voice database 425 may include voice data of a plurality of speakers reading the learning text. The learning text and voice data may be previously stored in the voice database 425 or may be received from the communication unit 405 . Based on the learning text and voice stored in the voice database 425, the data learning unit 450 may generate or learn an artificial neural network text-speech synthesis model. For example, an artificial neural network text-synthesis model may include an encoder 435 and a decoder 440 . As another example, the artificial neural network text-synthesis model may include an encoder 435 , a decoder 440 and a post-processor 445 .

According to one embodiment, voice database 425 may be configured to store one or more sequential prosodic features. For example, the one or more sequential prosody features may include normalized sequential prosody features from normalizer 420 . In another embodiment, it may be configured to store one or more speaker's vocalization characteristics extracted from the vocalization feature extraction unit 415 . The stored sequential prosody features may be provided to at least one of the encoder 435 and the decoder 440 during speech synthesis by the data learning unit 450 and/or the data recognizing unit 455 . In addition, the stored speech characteristics of the speaker may be provided to at least one of the encoder 435 and the decoder 440 during speech synthesis by the data learning unit 450 and/or the data recognizing unit 455 .

The attention module 430 may receive sequential prosody features or normalized sequential prosody features from the sequential prosody feature extractor 410 or the normalizer 420 . According to an embodiment, the attention module 430 is configured to receive a plurality of embedding vectors representing sequential prosody features and generate a plurality of transform embedding vectors corresponding to respective parts of the input text provided to the encoder 435. can For example, the attention module 430 may be configured to determine which part of the plurality of embedding vectors over time corresponds to which part of the input text at the current time-step. The plurality of transform embedding vectors generated by the attention module 430 may be provided to the encoder 435 for speech synthesis.

Encoder 435 may receive input text and may be configured to convert and generate input text into character embeddings. For example, encoder 435 may be configured as part of a neural text-to-speech synthesis model. These character embeddings are input to the first artificial neural network text-to-speech synthesis model (eg, pre-net, CBHG module, DNN, CNN+DNN, etc.) corresponding to the encoder 435 to obtain hidden states of the encoder 435. can create The first artificial neural network text-speech synthesis model may be included in the artificial neural network text-speech synthesis model. According to an embodiment, the encoder 435 may further receive sequential prosody features from the sequential prosody feature extractor 410 , the normalizer 420 , or the attention module 430 . Character embedding and sequential prosody features may be input to the first artificial neural network text-speech synthesis model to generate hidden states of the encoder 435 . In another embodiment, the encoder 435 may further receive speech features of the speaker from the speech feature extraction unit 415 . The speech characteristics of the speaker may be input to the first artificial neural network text-to-speech synthesis model along with character embedding and sequential prosody characteristics to generate hidden states of the encoder 435 . The hidden states of the encoder 435 generated in this way may be provided to the decoder 440 .

The decoder 440 may be configured as part of an artificial neural network text-speech synthesis model. In one embodiment, decoder 440 may be configured to receive sequential prosodic features. The decoder 440 may receive sequential prosody features from at least one of the sequential prosody feature extractor 410 and the normalizer 420 . The decoder 440 may receive hidden states corresponding to input text from the encoder 435 . In addition, the decoder 440 may include an attention module configured to determine from which part of the input text to generate speech at the current time-step. Accordingly, the hidden states corresponding to the sequential prosody features and/or the input text are the second artificial neural network text-speech synthesis model corresponding to the decoder 440 (eg, attention module, decoder RNN, attention RNN, Pre-net , DNN, etc.) and output voice data corresponding to the input text may be generated. The second artificial neural network text-speech synthesis model may be included in the artificial neural network text-speech synthesis model.

In another embodiment, the decoder 440 may be configured to further receive speech features of the speaker from the speech feature extraction unit 415 . Sequential prosody features, hidden states corresponding to the input text, and/or speech characteristics of the speaker are input to the second artificial neural network text-to-speech synthesis model corresponding to the decoder 440 to generate output voice data corresponding to the input text. can Such output voice data may include output voice data for input text in which a speaker's voice is simulated and sequential prosody characteristics are reflected.

The output voice data generated in this way may be expressed as a mel spectrogram. However, it is not limited thereto, and the output voice data may be expressed as a linear spectrogram. Such output voice data may be output to at least one of a speaker, the post-processing processor 445, and the communication unit 405.

According to one embodiment, the post-processing processor 445 may be configured to convert the output voice data generated by the decoder 440 into a voice capable of being output by a speaker. For example, the changed outputable voice may be expressed as a waveform. The post-processor 445 may be configured to operate only when the output voice data generated by the decoder 440 is inappropriate to be output from a speaker. That is, when the output voice data generated by the decoder 440 is suitable for being output from the speaker, the output voice data may be directly output to the speaker without going through the post-processing processor 445 . Accordingly, although the post-processor 445 is shown to be included in the text-to-speech synthesis system 400 in FIG. 4 , the post-processor 445 may not be included in the text-to-speech synthesis system 400. there is.

According to one embodiment, the post-processing processor 445 may be configured to convert the output voice data represented by the MEL spectrogram generated by the decoder 440 into a waveform in the time domain. Also, the post-processing processor 445 may amplify the size of the output voice data when the signal size of the output voice data does not reach a predetermined reference level. The post-processing processor 445 may output the converted output voice data to at least one of the speaker or the communication unit 405 .

The data learning unit 450 may correspond to the data learning unit 1010 of FIG. 10 . The data learning unit 450 may receive data representing a plurality of learning texts and corresponding learning voices through the voice database 425 or the communication unit 405 . Data representing the learning text may include information on at least one letter. For example, data representing the training text may include a phoneme sequence corresponding to the training text using a grapheme-to-phoneme (G2P) algorithm. The data representing the training voice may be data recorded by a human reading the training text, a sound feature extracted from such recorded data, or a spectrogram. In one embodiment, the data representing the learning voice may include sequential prosodic features of the learning voice. In another embodiment, the data representing the learning voice may further include vocal characteristics of a speaker who uttered the learning voice. The data learning unit 450 may generate an artificial neural network text-speech synthesis model by performing machine learning based on pairs of training data corresponding to a plurality of training texts and training voices corresponding thereto. During this learning, the training text may be provided to the first artificial neural network text-speech synthesis model corresponding to the encoder of the artificial neural network text-speech synthesis model, and the sequential prosody features may be applied to the first artificial neural network text-speech synthesis model and/or It may be input to the second artificial neural network text-speech synthesis model corresponding to the decoder.

According to one embodiment, data recognition unit 455 may be configured to receive input text and receive sequential prosodic features. Input text and sequential prosody features may be input to an artificial neural network text-speech synthesis model to generate output voice data for the input text in which the received prosody features are reflected. Here, the input text may be provided to the first artificial neural network text-speech synthesis model, and the sequential prosody features may be input to the first artificial neural network text-speech synthesis model and/or the second artificial neural network text-speech synthesis model. . As a result, output voice data corresponding to input text in which sequential prosody characteristics are reflected can be generated from an artificial neural network text-speech synthesis model. According to another embodiment, the data recognizing unit 455 may be configured to further receive speech characteristics of the speaker. The received vocalization characteristics of the speaker may be provided to the second artificial neural network text-speech synthesis model, similarly to the sequential prosody characteristics. Under this operation, output voice data corresponding to the input text that mimics the speaker's voice and reflects sequential prosody characteristics can be generated from the artificial neural network text-speech synthesis model.

5 is a flowchart illustrating a text-to-speech synthesis method using machine learning based on sequential prosody characteristics according to an embodiment of the present disclosure. First, in S510, the text-to-speech synthesis system 400 generates an artificial neural network text-to-speech synthesis model generated by performing machine learning based on a plurality of training texts and voice data corresponding to the plurality of training texts. can be done Here, the artificial neural network text-speech synthesis model may be a single artificial neural network text-speech synthesis model.

The text-to-speech synthesis system 400 may perform a step of receiving input text in S520. In step S530, the text-to-speech synthesis system 400 A step of receiving sequential prosody features may be performed. The text-to-speech synthesis system 400 inputs the input text and sequential prosody features to the pre-learned text-to-speech synthesis model, and generates output voice data for the input text in which the sequential prosody features are reflected, in S540. there is.

6 is an exemplary diagram illustrating the configuration of a text-speech synthesis system based on an artificial neural network according to an embodiment of the present disclosure. In one embodiment, each of encoder 610 and decoder 620 and post processor 630 may correspond to each of encoder 435, decoder 440 and post processor 445 in FIG. 4 .

According to an embodiment, the encoder 610 may receive character embedding for input text, as shown in FIG. 6 . According to another embodiment, the input text may include at least one of words, phrases, or sentences used in one or more languages. For example, text such as a sentence such as 'HELLO' may be input. When input text is received, the encoder 610 may divide the received input text into consonant units, character units, and phoneme units. According to another embodiment, the encoder 610 may receive input text divided into consonant units, character units, and phoneme units. Then, the encoder 610 may generate by converting the input text into character embedding.

Encoder 610 may be configured to generate text into pronunciation information. In one embodiment, the encoder 610 may pass the generated character embedding through a pre-net including a fully-connected layer. In addition, the encoder 610 may provide an output from the pre-net to the CBHG module to output hidden states e _i of the encoder as shown in FIG. 6 . For example, the CBHG module may include a 1D convolution bank, max pooling, a highway network, and a bidirectional gated recurrent unit (GRU).

In another embodiment, when the encoder 610 receives input text or separated input text, the encoder 610 may be configured to generate at least one embedding layer. According to an embodiment, at least one embedding layer of the encoder 610 may generate character embeddings based on input text divided into consonant units, character units, and phoneme units. For example, the encoder 610 may use a previously learned machine learning model (eg, a probabilistic model or an artificial neural network) to obtain character embeddings based on the separated input text. Furthermore, the encoder 610 may update the machine learning model while performing machine learning. When the machine learning model is updated, the character embedding of the separated input text may also be changed. The encoder 610 may pass the character embedding through a deep neural network (DNN) module composed of fully-connected layers. A DNN may include a general feedforward layer or a linear layer. The encoder 610 may provide an output of the DNN to a module including at least one of a convolutional neural network (CNN) or a recurrent neural network (RNN), and may generate hidden states of the encoder 610 . CNNs can capture local features depending on the convolution kernel size, whereas RNNs can capture long term dependencies. The hidden states of the encoder 610, that is, pronunciation information for the input text, are provided to the decoder 620 including the attention module, and the decoder 620 may be configured to generate the pronunciation information as voice.

The decoder 620 may receive the hidden states e _i of the encoder from the encoder 610 . In one embodiment, as shown in FIG. 6, the decoder 620 includes an attention module, a freenet composed of a fully connected layer, and a gated recurrnt unit (GRU), and includes an attention recurrent neural network (RNN), residual It may include a decoder RNN (decoder RNN) including a residual GRU (GRU). Here, the attention RNN may output information to be used in the attention module. Also, the decoder RNN may receive positional information of input text from the attention module. That is, the location information may include information about which location of the input text is being converted into voice by the decoder 620 . The decoder RNN may receive information from the attention RNN. The information received from the attention RNN may include information about what voice the decoder 620 has generated up to the previous time-step. The decoder RNN can generate the next output speech following the speech generated so far. For example, the output voice may have a form of a mel spectrogram, and the output voice may include r frames.

In another embodiment, FreeNet included in the decoder 620 may be replaced with a DNN composed of fully-connected layers. Here, the DNN may include at least one of a general feedforward layer and a linear layer.

In one embodiment, decoder 620 may be configured to receive sequential prosodic features. For example, as shown in FIG. 6, the sequential prosody feature extraction unit 410 includes a plurality of embedding vectors p1, p2, ... pn representing sequential prosody features from a voice signal or sample (where n is a voice sample). Proportional to the length of) can be extracted. Each of the plurality of embedding vectors may include prosody characteristics or information per unit time. A method in which the sequential prosody feature extractor 410 inputs a plurality of embedding vectors p1, p2, ... pn from a voice signal or sample to a decoder will be described in detail with reference to FIG. 7 below.

In one embodiment, decoder 620 may be configured to further receive vocal characteristics of the speaker. For example, as shown in FIG. 6 , a speaker ID may be input to the speech feature extraction unit 415 to generate a speaker embedding vector e corresponding to the speaker's speech feature as the speaker's speech feature. there is. As another example, the speaker's speech characteristics may be generated by extracting the speaker's embedding vector from a speech signal or sample rather than a speaker ID.

Also, the attention module of the decoder 620 may receive information from the attention RNN. The information received from the attention RNN may include information about what voice the decoder 620 has generated up to the previous time-step. Also, the attention module of the decoder 620 may output a context vector based on information received from the attention RNN and information of the encoder. Information of the encoder 610 may include information on input text to generate voice. The context vector may include information for determining from which part of the input text to generate speech in the current time-step. For example, the attention module of the decoder 620 generates voice based on the front part of the input text at the beginning of voice generation, and gradually generates voice based on the later part of the input text as the voice is generated. information can be printed.

The decoder 620 inputs each of the embedding vectors p1, p2, ... pn and the speaker embedding vector e according to the time unit among the sequential prosody features for each time step of the attention RNN and decoder RNN , the structure of the artificial neural network can be configured to decode differently for each speaker and for each part of the input text. In FIG. 6, a plurality of embedding vectors p1, p2, ... pn are illustrated as being extracted from the sequential prosody feature extraction unit 410, but are not limited thereto, and the decoder 620 normalizes them from the normalizer 420. A plurality of embedding vectors corresponding to the sequential prosody features may be received, and the plurality of embedding vectors corresponding to the normalized sequential prosody features are input for each time step of the attention RNN and the decoder RNN together with the embedding vector e of the speaker. It can be.

Dummy frames are frames input to the decoder 620 when no previous time-step exists. RNN can do machine learning with auto-regressive. That is, the r frame output at the previous time-step 622 can be an input to the current time-step 623. In the first time-step 621, there cannot be a previous time-step, so the decoder 620 can input a dummy frame to the machine learning network of the first time-step.

For text-to-speech synthesis, the operations of the DNN, attention RNN, and decoder RNN may be repeatedly performed. For example, r frames obtained in the first time-step 621 may be input to the next time-step 622 . Also, the r frames output at time-step 622 may be input to the next time-step 623 .

Through the above-described process, it is possible not only to synthesize the input text for each speaker, but also to reflect the prosody characteristics of each part in the input text in chronological order. That is, it is possible to control the prosody characteristics of the synthesized voice corresponding to the input text at a specific point in time. Accordingly, the text-to-speech synthesis system can control fine prosody of the synthesized voice in order to more accurately convey people's intentions or emotions.

According to an embodiment, the decoder 620 may acquire the voice of the mel spectrogram for the entire text by concatenating the mel spectrograms generated for each time-step in chronological order. The speech of the Mel spectrogram for the entire text may be output to the post-processing processor 630. For example, the post-processing processor 630 may correspond to the post-processing processor 445 of FIG. 4 .

According to one embodiment, the CBHG of the post-processor 630 may be configured to convert the mel-scale spectrogram of the decoder 620 to a linear-scale spectrogram, as shown in FIG. 6 . For example, the output signal of the CBHG of the post-processor 630 may include a magnitude spectrogram. The phase of the output signal of the CBHG of the post-processor 630 may be restored through a Griffin-Lim algorithm and subjected to an inverse short-time Fourier transform. The post-processing processor 630 may output the audio signal in the time domain.

According to another embodiment, if the encoder 610 is configured to include CNNs or RNNs, then the post-processor 630 is configured to include CNNs or RNNs, which CNNs or RNNs of the encoder 610 and the CNNs or RNNs. Similar actions can be performed. That is, the CNN or RNN of the post-processor 630 can capture local characteristics and long-term dependencies. For example, the post processor 630 may be a vocoder. Accordingly, the CNN or RNN of the post-processor 630 may output a linear-scale spectrogram. For example, a linear-scale spectrogram can include a magnitude spectrogram. The post-processing processor 630 may predict the phase of the spectrogram through the Griffin-Lim algorithm. The post-processing processor 630 may output a time domain audio signal using an inverse short-time Fourier transform.

According to another embodiment, the post-processing processor 630 may generate a voice signal from the MEL spectrogram based on a machine learning model. The machine learning model may include a machine learning model of a correlation between a Mel spectrogram and a voice signal. For example, an artificial neural network model such as WaveNet or WaveGlow may be used.

Such an artificial neural network-based text-speech synthesis system can be learned using a large-capacity database that exists as a pair of training text and voice signals. According to an embodiment, the voice synthesizer may receive text and compare the output voice signal with a correct voice signal to define a loss function. The speech synthesis apparatus may learn the loss function through an error back propagation algorithm, and finally obtain an artificial neural network that produces a desired speech output when arbitrary text is input.

In such an artificial neural network-based voice synthesizer, a voice signal may be output by inputting text, speaker's vocalization characteristics, sequential prosody characteristics, etc. to an artificial neural network text-speech synthesis model. The text-to-speech synthesis system learns by comparing the output voice signal with the correct answer voice signal, and when receiving the text, speaker's vocalization characteristics, and sequential prosody characteristics, output voice data that reads the text in which the sequential prosody characteristics are reflected in the corresponding speaker's voice. can create

7 is an exemplary diagram illustrating a process of generating synthesized speech by inputting sequential prosody characteristics to a decoder 720 of the text-speech synthesis system based on an artificial neural network according to an embodiment of the present disclosure. . Here, each of the encoder 710, decoder 720, sequential prosody feature extractor 730, and attention module 740 is the encoder 435, decoder 440, sequential prosody feature extractor 410 of FIG. and each of the attention module 430. Also, the encoder 710 and the decoder 720 may correspond to each of the encoder 610 and the decoder 620 of FIG. 6 . In FIG. 7, it is assumed that the length N of the voice is 4 and the length T of the text is 3, but is not limited thereto, and the length N of the voice and the length T of the text may be any positive number different from each other.

According to an embodiment, as shown in FIG. 7 , the sequential prosody feature extractor 730 receives spectrograms (y ₁ , y ₂ , y ₃ , y _n ), and inserts a plurality of embeddings representing the sequential prosody features. It can be configured to extract vectors (P ₁ , P ₂ , P ₃ , P _n ). The plurality of embedding vectors extracted in this way (P ₁ , P ₂ , P ₃ , P _n ) may be provided to the decoder 720 . For example, the extracted embedding vectors P ₁ , P ₂ , P ₃ , and P _n may be provided to N decoder RNNs and attention RNNs of the decoder 720 . In addition, the hidden states (e ₁ , e ₂ , e _T ) provided from the encoder 710 may be provided to the attention module 740 , and the attention module 740 may provide the hidden states (e ₁ , e ₂ , e T ). _T ) may generate transformation hidden states (e' ₁ , e' ₂ , e' _{3 , e' N ) to correspond to the lengths of the spectrograms (P 1 , P 2 , P 3 , P n ) .} The generated transformation hidden states (e' ₁ , e' ₂ , e' _{3 ,} e' _N ) are concatenated with a plurality of extracted embedding vectors (P ₁ , P ₂ , P ₃ , P _n ) to generate N decoders. It can be input and processed in each of the RNN and attention RNN. Since the process in the decoder 720 overlaps the process described in FIG. 6, detailed descriptions thereof are omitted. Through this process, the artificial neural network text-speech synthesis model included in the encoder 710 and the decoder 720 is trained so that sequential prosody characteristics can be reflected more naturally.

In FIG. 7 , spectrograms (y ₁ , y ₂ , y ₃ , y _n ) representing a specific voice are provided to the sequential prosody feature extractor 730, and the same spectrograms (y ₁ , y ₂ , y ₃ , y _n ) has been described, but is not limited thereto, and a voice having a different length from that output through the decoder 720 may be sequentially input to the prosody feature extraction unit 730 . In this case, an additional attention module (not shown) receives a plurality of embedding vectors extracted from the sequential prosody feature extraction unit and converts the lengths of the received plurality of embedding vectors to correspond to the length of voice output through the decoder 720. can make it Then, the converted plurality of embedding vectors may be provided to the decoder 720 .

8 is an exemplary view illustrating a process of generating a synthesized voice by inputting sequential prosody characteristics to an encoder 820 of the text-speech synthesis system based on an artificial neural network according to an embodiment of the present disclosure. . Here, each of the attention module 810, encoder 820, decoder 830, and sequential prosody feature extractor 840 is the attention module 430, encoder 435, decoder 440, and sequential prosody of FIG. It may correspond to each of the feature extraction units 410 . Also, the encoder 810 and the decoder 820 may correspond to each of the encoder 610 and the decoder 620 of FIG. 6 . In FIG. 8, it is assumed that the length N of the voice is 4 and the length T of the text is 3, but is not limited thereto, and the length N of the voice and the length T of the text may be any positive number different from each other.

,

, ...,

)를 생성하도록 구성될 수 있다. 단어의 길이

은 N, T와 상이한 임의의 양수일 수 있다.

를 구하는 한가지 예시로, 같은 단어에 속하는 음소들의 변환 임베딩 벡터들의 평균을 사용할 수 있으나, 이에 한정되지 않으며, 추가적인 어텐션 모듈 등이 이용될 수 있다. According to an embodiment, as shown in FIG. 8 , the sequential prosody feature extractor 840 receives spectrograms (y ₁ , y ₂ , y ₃ , y _n ), and inserts a plurality of embeddings representing the sequential prosody features. It can be configured to extract vectors (P ₁ , P ₂ , P ₃ , P _n ). The plurality of embedding vectors extracted in this way (P ₁ , P ₂ , P ₃ , P _n ) may be provided to the attention module 810 . The attention module 810 converts the plurality of input embedding vectors P ₁ , P ₂ , P ₃ , and P _n into a plurality of transform embedding vectors P to correspond to the length T of the phoneme sequence corresponding to the encoder 820. ₁ ', P ₂ ', P _T '). According to another embodiment, the sequential prosody feature extractor 840 may generate a plurality of transform embedding vectors to correspond to words rather than phonemes. For example, if the i-th phoneme and the j-th phoneme belong to the same word (v), they may have a value of P _i '=P _j ', and the attention module 810 includes a plurality of transformation embedding vectors corresponding to the word (

,

, ...,

) can be configured to generate. word length

can be any positive number different from N and T.

As an example of obtaining , an average of transformation embedding vectors of phonemes belonging to the same word may be used, but is not limited thereto, and an additional attention module may be used.

,

, ,,,,

)가 생성된 경우, 복수의 변환 임베딩 벡터(

,

, ...,

)의 각각은 입력 텍스트의 단어 시퀀스에 대응하는 숨겨진 상태들의 각각에 대응되도록 연결될 수 있다. 이러한 디코더 내의 처리 과정은 도 6에서 설명한 처리 과정과 중복되는 과정이므로 자세한 설명은 생략된다. 이러한 과정을 통하여 인코더(820) 및 디코더(830)에 포함된 인공신경망 텍스트-음성 합성 모델을 학습시켜서 순차적 운율 특징이 더욱 자연스럽게 반영되도록 할 수 있다. As shown in FIG. 8, each of the plurality of transform embedding vectors P ₁ ', P ₂ ', and P _T 'generated in this way corresponds to the phoneme sequence of the input text, and the hidden states (e ₁ , e ₂ , e _T ) may be connected so as to correspond to each of them. The hidden states (e ₁ , e ₂ , e _T ) connected in this way and the plurality of transform embedding vectors (P ₁ ′, P ₂ ′, P _T ′) may be provided to the decoder 830 . Unlike this, the decoder 830 converts the received hidden states e ₁ , e ₂ , e _T ) and transform embedding vectors P ₁ ', P ₂ ', P _T ' to the attention module of the decoder 830, Phoneme sequences y ₁ , y ₂ , y ₃ , and y _n may be generated using Pre-net, N decoder RNNs, and attention RNNs. Alternatively, a plurality of transformation embedding vectors corresponding to words (

,

, ,,,,

) is generated, a plurality of transform embedding vectors (

,

, ...,

) may be connected to correspond to each of the hidden states corresponding to the word sequence of the input text. Since the processing process in the decoder is the same as the process described in FIG. 6, a detailed description thereof will be omitted. Through this process, the artificial neural network text-speech synthesis model included in the encoder 820 and the decoder 830 is trained so that sequential prosody characteristics can be reflected more naturally.

9 is an exemplary diagram illustrating a network of sequential prosody feature extractors 920 configured to extract a plurality of embedding vectors 930 representing sequential prosody features from a voice signal or sample 910 according to an embodiment of the present disclosure. . In one embodiment, the network of the sequential prosody feature extractor 920 may include a convolutional neural network (CNN), batch-normalization (BN), rectifier linear unit (ReLU), and gated recurrent unit (GRU). CNN, BN, and ReLU can output a plurality of embedding vectors representing sequential prosody characteristics when a voice signal or sample is input and the output value is input to a gated recurrent unit (GRU). For example, a voice signal or sample may be received in the form of a log-Mel-spectrogram.

In one embodiment, when the data recognizing unit 455 infers speech synthesis, the speech signal or sample need not be speech data corresponding to the input text, and an arbitrarily selected speech signal may be used. In contrast, when the data learning unit 450 learns voice synthesis, the voice signal or sample may include voice data corresponding to the input text.

In this network, any spectrogram can be inserted into this network since there is no restriction on the use of the spectrogram. In addition, through this, it is possible to generate an embedic vector 930 representing sequential prosody characteristics through immediate adaptation of the network. A spectrogram input as a voice signal or sample may have a variable length, and the length of a plurality of embedding vectors may vary according to the length. Although FIG. 9 shows a network including CNN, BN, ReLU, and GRU, a network including various layers can be constructed to extract sequential prosody features.

10 is a schematic diagram of a text-to-speech synthesis system 1000 that outputs a synthesized voice by applying an attribute value input to a tag provided in a markup language to input text according to an embodiment of the present disclosure. In one embodiment, the text-to-speech synthesis system 1000 may correspond to the text-to-speech synthesis system 400 of FIG. 4 and/or the text-to-speech synthesis system 1100 of FIG. 11 .

In order to generate, adjust, or change sequential prosody information, the text-to-speech synthesis system 1000 may receive prosody information for at least a portion of text through an interface device. Here, the interface device may include any interface device directly connected to the text-to-speech synthesis system 1000 or connected through wired and/or wireless communication, and may include, for example, an interface of a user terminal. In addition, prosody information for at least a portion of text may be received through an arbitrary text editor or voice editor capable of inputting and editing arbitrary text.

According to an embodiment, the speech synthesis system 1000 may receive an attribute value corresponding to each part of input text as prosodic information using a tag of an arbitrary speech synthesis markup language provided by an arbitrary text editor. . For example, a tag provided by the voice synthesis markup language may include an arbitrary tag for representing a property included in a sequential prosody feature. Prosody information corresponding to a text portion between a start tag and an end tag may be input. For example, as shown in FIG. 10, '1. <speed=1.5>I'm a boy.</speed>' may include prosody information indicating speed in a part called I'm a boy between a start tag and an end tag. As another example, as shown in FIG. 10, '2. This is what <style=emphasis>I</style>have.' may include prosodic information indicating emphasis on the character I between the start tag and the end tag.

The text-to-speech synthesis system 1000 generates sequential prosody information based on the received prosody information for at least a portion of the input text, or changes prosody information corresponding to the input text among sequential prosody information corresponding to the input text, , it is possible to generate a synthesized voice corresponding to the input text in which generated or changed sequential prosody information is reflected. According to an embodiment, the text-to-speech synthesis system 1000 may apply prosody information (eg, attribute value) corresponding to each part of input text to a reference embedding vector corresponding to reference sequential prosody information. there is. Here, the reference embedding vector may include a plurality of embedding vectors representing predetermined sequential prosody feature information. For example, the reference embedding vector includes a prosody feature vector according to time, and each prosody feature information includes a plurality of orthogonal sub-embedding vectors (eg, height, size, length, rest period, style vector, etc.) can be expressed as a weighted sum of The text-to-speech synthesis system 1000 may separate intrinsic elements of the reference embedding vector. For example, the text-to-speech synthesis system 1000 may obtain a plurality of unit embedding vectors orthogonal to each other based on the reference embedding vector. According to one embodiment, methods for separating elements inherent in an embedding vector include independent component analysis (ICA), independent vector analysis (IVA), sparse coding, independent factor analysis (IFA), independent subspace analysis (ISA), and NMF. (nonnegative matrix factorization). In addition, the text-speech synthesis system 1000 may perform regularization during learning of the text-speech synthesis system when learning embedding vectors for sequential prosody features so that elements inherent in the embedding vector can be separated. there is. Such normalization may be performed through the normalizer 420 of FIG. 4 . When the text-to-speech synthesis system 1000 performs machine learning by performing normalization during learning, the reference embedding vector may be learned as a sparse vector. Accordingly, the text-to-speech synthesis system 900 may accurately separate an embedded element from an embedding vector learned as a sparse vector by using principle component analysis (PCA). Under this configuration, the text-to-speech synthesis system 1000 may modify the reference embedding vector based on attribute values in tags provided by the speech synthesis markup language. For example, the text-to-speech synthesis system 1000 may change weights for a plurality of unit embedding vectors based on attribute values in a received tag.

In one embodiment, the text-to-speech synthesis system 1000 may be configured to modify the reference embedding vector based on an attribute value in a tag provided by the received speech synthesis markup language. For example, the text-to-speech synthesis system 1000 may regenerate embedding vectors corresponding to sequential prosody features by multiplying and adding a plurality of unit embedding vectors by a weight changed according to the received attribute value. The text-to-speech synthesis system 1000 may output an embedding vector for the changed sequential prosody characteristic information. The text-to-speech synthesis system 1000 inputs the modified embedding vector to an artificial neural network text-to-speech synthesis model, and outputs speech data to input text reflecting information included in attribute values in tags provided by a speech synthesis markup language. can be converted into voice data.

11 is a block diagram of a text-to-speech synthesis system 1100 according to an embodiment of the present disclosure.

Referring to FIG. 11 , a text-to-speech synthesis system 1100 according to an embodiment may include a data learning unit 1110 and a data recognizing unit 1120. Each of the data learning unit 1110 and data recognizing unit 1120 of the text-speech synthesis system of FIG. 11 is the data learning unit 450 and data recognizing unit used by the text-speech synthesis system 400 of FIG. 455) may correspond to each of them.

The data learning unit 1110 may obtain a machine learning model by inputting data. Also, the data recognizing unit 1120 may generate an output voice by applying data to a machine learning model. The text-to-speech synthesis system 1100 as described above may include a processor and a memory.

The data learning unit 1110 may perform voice learning on text. The data learning unit 1110 may learn a criterion for which voice to output according to the text. Also, the data learner 1110 may learn a criterion for outputting a voice using which voice characteristics. Voice characteristics may include at least one of phoneme pronunciation, user's tone, intonation, or stress. The data learning unit 1110 acquires data to be used for learning and applies the acquired data to a data learning model to be described later, thereby learning speech according to text.

The data recognizing unit 1120 may output voice for the text based on the text. The data recognizing unit 1120 may output voice from predetermined text using the learned data learning model. The data recognizing unit 1120 may obtain predetermined text (data) according to a preset criterion through learning. In addition, the data recognizer 1120 may output a voice based on predetermined data by using the data learning model by using the obtained data as an input value. In addition, result values output by the data learning model using the acquired data as input values may be used to update the data learning model.

At least one of the data learning unit 1110 and the data recognizing unit 1120 may be manufactured in the form of at least one hardware chip and mounted in an electronic device. For example, at least one of the data learning unit 1110 and the data recognizing unit 1120 may be manufactured in the form of a dedicated hardware chip for artificial intelligence (AI), or an existing general-purpose processor (eg, CPU). Alternatively, it may be manufactured as a part of an application processor) or a graphics-only processor (eg GPU) and mounted in various electronic devices described above.

Also, the data learner 1110 and the data recognizer 1120 may be mounted on separate electronic devices. For example, one of the data learner 1110 and the data recognizer 1120 may be included in the electronic device, and the other may be included in the server. In addition, the data learning unit 1110 and the data recognizing unit 1120 may provide model information constructed by the data learning unit 1110 to the data recognizing unit 1120 through a wired or wireless connection, and the data recognizing unit ( Data input to 1120) may be provided to the data learning unit 1110 as additional learning data.

Meanwhile, at least one of the data learning unit 1110 and the data recognizing unit 1120 may be implemented as a software module. When at least one of the data learning unit 1110 and the data recognizing unit 1120 is implemented as a software module (or a program module including instructions), the software module is a memory or a computer-readable non-transitory module. It may be stored in a temporary readable recording medium (non-transitory computer readable media). Also, in this case, at least one software module may be provided by an Operating System (OS) or a predetermined application. Alternatively, some of the at least one software module may be provided by an Operating System (OS), and the other part may be provided by a predetermined application.

The data learning unit 1110 according to an embodiment of the present disclosure includes a data acquisition unit 1111, a pre-processing unit 1112, a learning data selection unit 1113, a model learning unit 1114, and a model evaluation unit 1115. can include

The data acquisition unit 1111 may acquire data necessary for machine learning. Since a lot of data is required for learning, the data acquisition unit 1111 may receive a plurality of texts and voices corresponding thereto.

The pre-processing unit 1112 may pre-process the acquired data so that the acquired data can be used for machine learning in order to determine the user's mental state. The pre-processing unit 1112 may process the acquired data into a preset format so that the model learning unit 1114 to be described later can use it. For example, the preprocessor 1112 may morpheme-analyze text and voice to obtain morpheme embeddings.

The learning data selection unit 1113 may select data necessary for learning from preprocessed data. The selected data may be provided to the model learning unit 1114 . The learning data selection unit 1113 may select data necessary for learning from preprocessed data according to a predetermined criterion. In addition, the learning data selection unit 1113 may select data according to a predetermined criterion by learning by the model learning unit 1114 to be described later.

The model learning unit 1114 may learn a criterion for what kind of voice to output according to the text based on the training data. In addition, the model learning unit 1114 may perform learning by using a learning model that outputs voice according to text as learning data. In this case, the data learning model may include a pre-built model. For example, the data learning model may include a model built in advance by receiving basic training data (eg, a sample image, etc.).

The data learning model may be constructed in consideration of the application field of the learning model, the purpose of learning, or the computer performance of the device. The data learning model may include, for example, a model based on a neural network. For example, models such as Deep Neural Network (DNN), Recurrent Neural Network (RNN), Long Short-Term Memory models (LSTM), Bidirectional Recurrent Deep Neural Network (BRDNN), Convolutional Neural Networks (CNN), etc. can be used as data learning models. may, but is not limited thereto.

According to various embodiments, the model learning unit 1114 may determine a data learning model having a high correlation between input learning data and basic learning data as a data learning model to be learned when a plurality of pre-built data learning models exist. there is. In this case, the basic learning data may be pre-classified for each type of data, and the data learning model may be pre-built for each type of data. For example, the basic training data is pre-classified according to various criteria such as the region where the training data was created, the time the training data was created, the size of the training data, the genre of the training data, the creator of the training data, and the type of object in the training data. may have been

In addition, the model learning unit 1114 may train the data learning model using a learning algorithm including, for example, error back-propagation or gradient descent.

In addition, the model learning unit 1114 may learn a data learning model through, for example, supervised learning using learning data as an input value. In addition, the model learning unit 1114, for example, data learning through unsupervised learning that discovers the criteria for determining the situation by self-learning the type of data necessary for determining the situation without any guidance. model can be trained. In addition, the model learning unit 1114 may learn a data learning model, for example, through reinforcement learning using feedback about whether a result of situation judgment according to learning is correct.

Also, when the data learning model is learned, the model learning unit 1114 may store the learned data learning model. In this case, the model learning unit 1114 may store the learned data learning model in the memory of the electronic device including the data recognizing unit 1120 . Alternatively, the model learning unit 1114 may store the learned data learning model in the memory of a server connected to the electronic device through a wired or wireless network.

In this case, the memory in which the learned data learning model is stored may also store, for example, commands or data related to at least one other component of the electronic device. Also, the memory may store software and/or programs. The program may include, for example, a kernel, middleware, an application programming interface (API), and/or an application program (or 'application').

The model evaluation unit 1115 may input evaluation data to the data learning model, and may cause the model learning unit 1114 to learn again when a result output from the evaluation data does not satisfy a predetermined criterion. In this case, the evaluation data may include preset data for evaluating the data learning model.

For example, the model evaluator 1115 determines that a predetermined criterion is not satisfied when the number or ratio of evaluation data for which the recognition result is not accurate among the results of the learned data learning model for the evaluation data exceeds a preset threshold. can be evaluated as For example, when a predetermined criterion is defined as a ratio of 2%, and a learned data learning model outputs an erroneous recognition result for evaluation data exceeding 20 out of a total of 1000 evaluation data, the model evaluation unit 1115 performs learning It can be evaluated that the applied data learning model is not suitable.

On the other hand, when there are a plurality of learned data learning models, the model evaluation unit 1115 evaluates whether each learned video learning model satisfies a predetermined criterion, and uses a model that satisfies the predetermined criterion as a final data learning model. can decide In this case, when there are a plurality of models that satisfy a predetermined criterion, the model evaluation unit 1115 may determine one or a predetermined number of models preset in order of high evaluation scores as the final data learning model.

On the other hand, at least one of the data acquisition unit 1111, the preprocessing unit 1112, the training data selection unit 1113, the model learning unit 1114, and the model evaluation unit 1115 in the data learning unit 1110 is at least one It can be manufactured in the form of a hardware chip and mounted in an electronic device. For example, at least one of the data acquisition unit 1111, the preprocessor 1112, the training data selection unit 1113, the model learning unit 1114, and the model evaluation unit 1115 is artificial intelligence (AI). It may be manufactured in the form of a dedicated hardware chip for the computer, or may be manufactured as a part of an existing general-purpose processor (eg, CPU or application processor) or graphics-only processor (eg, GPU) and mounted in various electronic devices described above.

In addition, the data acquisition unit 1111, the pre-processing unit 1112, the training data selection unit 1113, the model learning unit 1114, and the model evaluation unit 1115 may be installed in one electronic device, or may be installed in separate electronic devices. It may also be mounted on each electronic device. For example, some of the data acquisition unit 1111, the pre-processing unit 1112, the training data selection unit 1113, the model learning unit 1114, and the model evaluation unit 1115 are included in the electronic device, and the others are included in the electronic device. can be included in the server.

In addition, at least one of the data acquisition unit 1111, the preprocessing unit 1112, the training data selection unit 1113, the model learning unit 1114, and the model evaluation unit 1115 may be implemented as a software module. At least one of the data acquisition unit 1111, the pre-processing unit 1112, the training data selection unit 1113, the model learning unit 1114, and the model evaluation unit 1115 is a software module (or a program including instructions) module), the software module may be stored in a computer-readable, non-transitory computer readable recording medium (non-transitory computer readable media). Also, in this case, at least one software module may be provided by an Operating System (OS) or a predetermined application. Alternatively, some of the at least one software module may be provided by an Operating System (OS), and the other part may be provided by a predetermined application.

The data recognition unit 1120 according to an embodiment of the present disclosure includes a data acquisition unit 1121, a preprocessing unit 1122, a recognition data selection unit 1123, a recognition result providing unit 1124, and a model update unit 1125. can include

The data acquisition unit 1121 may acquire text necessary for outputting voice. Conversely, the data acquisition unit 1121 may obtain voice necessary for outputting text. The pre-processor 1122 may pre-process the acquired data so that the acquired data can be used to output voice or text. The pre-processing unit 1122 may process the acquired data into a preset format so that the recognition result providing unit 1124 to be described later may use the acquired data to output voice or text.

The recognition data selection unit 1123 may select data necessary for outputting voice or text from among preprocessed data. The selected data may be provided to the recognition result provider 1124 . The recognition data selector 1123 may select some or all of the preprocessed data according to a predetermined criterion for outputting voice or text. In addition, the recognition data selection unit 1123 may select data according to a standard set by learning by the model learning unit 1114 .

The recognition result providing unit 1124 may output voice or text by applying the selected data to the data learning model. The recognition result provider 1124 may apply the selected data to the data learning model by using the data selected by the recognition data selector 1123 as an input value. Also, the recognition result may be determined by a data learning model.

The model updating unit 1125 may update the data learning model based on the evaluation of the recognition result provided by the recognition result providing unit 1124 . For example, the model updater 1125 may provide the recognition result provided by the recognition result provider 1124 to the model learner 1114 so that the model learner 1114 updates the data learning model. there is.

Meanwhile, at least one of the data acquisition unit 1121, the preprocessing unit 1122, the recognition data selection unit 1123, the recognition result providing unit 1124, or the model updating unit 1125 in the data recognizing unit 1120, at least It is manufactured in the form of a single hardware chip and can be mounted in an electronic device. For example, at least one of the data acquisition unit 1121, the preprocessing unit 1122, the recognition data selection unit 1123, the recognition result providing unit 1124, or the model update unit 1125 is artificial intelligence (AI). ), or may be manufactured as a part of an existing general-purpose processor (eg, CPU or application processor) or graphics-only processor (eg, GPU) and mounted in various electronic devices described above.

In addition, the data acquisition unit 1121, the pre-processing unit 1122, the recognition data selection unit 1123, the recognition result providing unit 1124, and the model update unit 1125 may be mounted in one electronic device, or may be separately installed. may be mounted on each of the electronic devices. For example, some of the data acquisition unit 1121, the pre-processing unit 1122, the recognition data selection unit 1123, the recognition result providing unit 1124, and the model update unit 1125 are included in the electronic device, and the remaining part is included in the electronic device. may be included in the server.

In addition, at least one of the data acquisition unit 1121, the pre-processing unit 1122, the recognition data selection unit 1123, the recognition result providing unit 1124, and the model update unit 1125 may be implemented as a software module. At least one of the data acquisition unit 1121, the preprocessor 1122, the recognition data selection unit 1123, the recognition result providing unit 1124, or the model update unit 1125 includes a software module (or an instruction) When implemented as a program module), the software module may be stored in a computer-readable, non-transitory computer readable recording medium (non-transitory computer readable media). Also, in this case, at least one software module may be provided by an Operating System (OS) or a predetermined application. Alternatively, some of the at least one software module may be provided by an Operating System (OS), and the other part may be provided by a predetermined application.

In general, the text-to-speech synthesis system described in this specification and a user terminal providing the text-to-speech synthesis service include a wireless telephone, a cellular telephone, a laptop computer, a wireless multimedia device, a wireless communication PC (personal computer) card, a PDA, It may represent various types of devices, such as an external modem, an internal modem, or a device that communicates through a wireless channel. A device includes an access terminal (AT), an access unit, a subscriber unit, a mobile station, a mobile device, a mobile unit, a mobile telephone, a mobile, a remote station, a remote terminal, a remote unit, a user device, user equipment, It may have various names such as handheld device and the like. Any device described herein may have memory for storing instructions and data, as well as hardware, software, firmware, or combinations thereof.

The techniques described herein may be implemented by various means. For example, these techniques may be implemented in hardware, firmware, software, or combinations thereof. Those of skill would further appreciate that the various illustrative logical blocks, modules, circuits, and algorithm steps described in connection with the disclosure herein may be implemented as electronic hardware, computer software, or combinations of both. To clearly illustrate this interchangeability of hardware and software, various illustrative components, blocks, modules, circuits, and steps have been described above generally in terms of their functionality. Whether such functionality is implemented as hardware or software depends on the particular application and the design constraints imposed on the overall system. Skilled artisans may implement the described functionality in varying ways for each particular application, but such implementation decisions should not be interpreted as causing a departure from the scope of the present disclosure.

In a hardware implementation, the processing units used to perform the techniques may include one or more ASICs, DSPs, digital signal processing devices (DSPDs), programmable logic devices (PLDs) ), field programmable gate arrays (FPGAs), processors, controllers, microcontrollers, microprocessors, electronic devices, and other electronic units designed to perform the functions described herein. , a computer, or a combination thereof.

Accordingly, the various illustrative logical blocks, modules, and circuits described in connection with the disclosure herein may be implemented in a general-purpose processor, DSP, ASIC, FPGA or other programmable logic device, discrete gate or transistor logic, discrete hardware components, or may be implemented or performed in any combination of those designed to perform the functions described herein. A general purpose processor may be a microprocessor, but in the alternative, the processor may be any conventional processor, controller, microcontroller, or state machine. A processor may also be implemented as a combination of computing devices, eg, a combination of a DSP and a microprocessor, a plurality of microprocessors, one or more microprocessors in conjunction with a DSP core, or any other such configuration.

In firmware and/or software implementation, the techniques include random access memory (RAM), read-only memory (ROM), non-volatile random access memory (NVRAM), PROM ( on a computer-readable medium, such as programmable read-only memory (EPROM), erasable programmable read-only memory (EPROM), electrically erasable PROM (EEPROM), flash memory, compact disc (CD), magnetic or optical data storage device, or the like. It can also be implemented as stored instructions. Instructions may be executable by one or more processors and may cause the processor(s) to perform certain aspects of the functionality described herein.

If implemented in software, the functions may be stored on or transmitted over a computer readable medium as one or more instructions or code. Computer readable media includes both computer storage media and communication media including any medium that facilitates transfer of a computer program from one place to another. A storage media may be any available media that can be accessed by a computer. By way of non-limiting example, such computer readable media may include RAM, ROM, EEPROM, CD-ROM or other optical disk storage, magnetic disk storage or other magnetic storage devices, or desired program code in the form of instructions or data structures. It can be used for transport or storage to and can include any other medium that can be accessed by a computer. Also, any connection is properly termed a computer-readable medium.

For example, if the software is transmitted from a website, server, or other remote source using coaxial cable, fiber optic cable, twisted pair, digital subscriber line (DSL), or wireless technologies such as infrared, radio, and microwave, the coaxial cable , fiber optic cable, twisted pair, digital subscriber line, or wireless technologies such as infrared, radio, and microwave are included within the definition of medium. Disk and disc, as used herein, include CDs, laser discs, optical discs, digital versatile discs (DVDs), floppy disks, and blu-ray discs, where disks are usually It reproduces data magnetically, whereas discs reproduce data optically using a laser. Combinations of the above should also be included within the scope of computer readable media.

A software module may reside in RAM memory, flash memory, ROM memory, EPROM memory, EEPROM memory, registers, hard disk, a removable disk, a CD-ROM, or any other form of storage medium known in the art. An exemplary storage medium can be coupled to the processor such that the processor can read information from or write information to the storage medium. Alternatively, the storage medium may be integral to the processor. The processor and storage medium may reside within an ASIC. An ASIC may exist within a user terminal. Alternatively, the processor and storage medium may exist as separate components in a user terminal.

The previous description of the present disclosure is provided to enable any person skilled in the art to make or use the present disclosure. Various modifications of this disclosure will be readily apparent to those skilled in the art, and the general principles defined herein may be applied in various modifications without departing from the spirit or scope of the disclosure. Thus, this disclosure is not intended to be limited to the examples set forth herein but is to be accorded the widest scope consistent with the principles and novel features disclosed herein.

Although example implementations may refer to utilizing aspects of the presently disclosed subject matter in the context of one or more stand-alone computer systems, the subject matter is not so limited, but rather in conjunction with any computing environment, such as a network or distributed computing environment. may be implemented. Further, aspects of the presently-disclosed subject matter may be implemented in or across a plurality of processing chips or devices, and storage may be similarly affected across a plurality of devices. Such devices may include PCs, network servers, and handheld devices.

Although subject matter has been described in language specific to structural features and/or methodological acts, it will be understood that subject matter defined in the appended claims is not necessarily limited to the specific features or acts described above. Rather, the specific features and acts described above are described as example forms of implementing the claims.

Although the methods mentioned in this specification have been described through specific embodiments, it is possible to implement them as computer readable codes on a computer readable recording medium. A computer-readable recording medium includes all types of recording devices in which data readable by a computer system is stored. Examples of computer-readable recording media include ROM, RAM, CD-ROM, magnetic tape, floppy disk, and optical data storage devices. In addition, the computer-readable recording medium is distributed in computer systems connected through a network, so that computer-readable codes can be stored and executed in a distributed manner. In addition, functional programs, codes, and code segments for implementing the embodiments can be easily inferred by programmers in the technical field to which the present invention belongs.

Although the present disclosure has been described in relation to some embodiments in this specification, it should be noted that various modifications and changes can be made without departing from the scope of the present disclosure that can be understood by those skilled in the art. something to do. Moreover, such modifications and variations are intended to fall within the scope of the claims appended hereto.

110: voice synthesizer 120: input text
130: Sequential prosody feature 140: Synthetic voice
210 Sequential prosody feature 220 Voice signal or voice sample
230: sequential prosody feature extractor 240: synthesized voice
310: vocalization feature extractor 320: voice signal or voice sample
330 speaker's vocalization characteristics 340 output voice

Claims (13)

A text-to-speech synthesis method using machine learning based on sequential prosody characteristics, performed by a text-to-speech synthesis system, comprising:
receiving input text;
Receiving a sequential prosody feature including prosody information corresponding to at least one unit of a frame, a letter, a phoneme, a syllable, or a word included in the received input text in chronological order; and
generating output voice data for the input text in which the received sequential prosody features are reflected by inputting the input text and the received sequential prosody features to an artificial neural network text-speech synthesis model;
including,
The sequential prosody feature includes a plurality of embedding vectors representing prosody information included in the chronological order,
Each of the plurality of embedding vectors representing the prosody information included in the chronological order is sequentially applied to each of a plurality of time-steps of the artificial neural network text-speech synthesis model.
According to claim 1,
The artificial neural network text-speech synthesis model is generated by performing machine learning based on data representing a plurality of learning texts and a learning voice corresponding to the plurality of learning texts,
The text-to-speech synthesis method of claim 1 , wherein the data representing the learning voice includes sequential prosody characteristics of the learning voice.
According to claim 1,
The prosody information includes at least one of information about the loudness of the sound, information about the height of the sound, information about the length of the sound, information about the rest period of the sound, or information about the style of the sound , text-to-speech synthesis method.
According to claim 1,
The artificial neural network text-speech synthesis model includes an encoder and a decoder,
generating a plurality of transform embedding vectors corresponding to respective parts of the input text provided to the encoder by inputting the plurality of embedding vectors to an attention module, wherein the lengths of the plurality of transform embedding vectors correspond to the length of the input text Depending on the variable - further including,
Generating output voice data for the input text,
inputting the generated plurality of transformation embedding vectors to an encoder of the artificial neural network text-speech synthesis model; and
generating output speech data for the input text in which the plurality of transform embedding vectors are reflected;
Including, text-to-speech synthesis method.
According to claim 1,
The artificial neural network text-speech synthesis model includes an encoder and a decoder,
Generating output voice data for the input text,
inputting the plurality of embedding vectors to a decoder of the artificial neural network text-speech synthesis model; and
generating output speech data for the input text in which the plurality of embedding vectors are reflected;
Including, text-to-speech synthesis method.
According to claim 1,
further comprising the step of receiving vocalization characteristics of a speaker;
Wherein the generating of output voice data for the input text comprises generating output voice data for the input text in which the voice of the speaker is copied and the plurality of embedding vectors are reflected.
According to claim 6,
The step of receiving vocalization characteristics of the speaker includes receiving sequential prosodic characteristics of the speaker;
The method,
Normalizing the plurality of embedding vectors based on sequential prosodic characteristics of the speaker;
Wherein the generating of output speech data for the input text includes generating output speech data for the input text in which the voice of the speaker is simulated and the plurality of normalized embedding vectors are reflected. .
According to claim 7,
Normalizing the plurality of embedding vectors,
calculating an average value of embedding vectors representing sequential prosody characteristics of the speaker at each time step; and
Subtracting the plurality of embedding vectors by the average value of the embedding vectors calculated at each time step
Including, text-to-speech synthesis method.
According to claim 1,
Receiving the sequential prosody features includes receiving prosody information for at least a portion of the input text through a user interface;
The step of generating output voice data for the input text in which the received sequential prosody characteristic is reflected includes generating output voice data for the input text in which prosody information for at least a part of the input text is reflected. - Speech synthesis method.
According to claim 9,
Wherein the prosody information for at least part of the input text is input through a tag provided in a speech synthesis markup language.
According to claim 1,
receiving prosody information for at least a portion of the input text through a user interface; and
further comprising changing the received sequential prosody characteristic based on prosody information for at least a portion of the received input text;
The step of generating output voice data for the input text in which the received sequential prosody feature is reflected includes generating output voice data for the input text in which the changed sequential prosody feature is reflected. .
According to claim 11,
The text-to-speech synthesis method of claim 1 , wherein prosody information for at least a portion of the input text, which is used to change the received sequential prosody characteristic, is input through a tag provided in a speech synthesis markup language.
A computer-readable storage medium in which a program including instructions for performing each step according to the text-speech synthesis method using machine learning using sequential prosody features according to any one of claims 1 to 12 is recorded.

Images