KR20220096129A - Speech synthesis system automatically adjusting emotional tone - Google Patents

Abstract

Disclosed is a speech synthesis method based on an emotion. The speech synthesis method according to one embodiment of the present specification fine-tunes a BERT language model to train an emotion classification model, enables the emotion classification model to apply an input sentence, and performs speech synthesis by extracting a plurality of emotion information having different emotion weight values. Accordingly, the emotion information of the synthesis speech can be adjusted, and can perform speech synthesis that reflects a complex and subtle emotion through a complex emotion weight.

Description

Speech synthesis system that automatically adjusts emotional tones {SPEECH SYNTHESIS SYSTEM AUTOMATICALLY ADJUSTING EMOTIONAL TONE}

The present specification relates to a speech synthesis system for automatically adjusting emotional tones.

Conventional text-to-speech (TTS) processing outputs text as a pre-stored voice. The primary purpose of TTS processing is to deliver the semantic contents of the text, but recently TTS processing has made it possible to transmit not only the semantic content of the text but also the interactive meaning of the text to the other party. , the need is raised to allow the user's intention or emotion to be reflected in the voice output to actually experience an interactive conversation with the text sender.

Emotion-based speech synthesis extracts emotional elements from input sentences to determine pre-classified specific emotions, and reflects the determined emotional elements in the speech synthesis process. Accordingly, as for the emotional elements reflected in the speech synthesis process, the emotional elements classified in advance are reflected in the speech synthesis process. Here, pre-classified emotions may include major emotional states such as sadness, anger, surprise, and joy. However, it may not be possible to express an emotional state revealed through an actor's performance in a movie or a drama using only the emotional state classified in advance.

Accordingly, an object of the present specification is to provide a speech synthesis method and apparatus capable of adjusting emotion information of a synthesized speech in an emotion-based speech synthesis process.

Another object of the present specification is to provide a speech synthesis method and apparatus capable of reflecting a more subdivided emotional state in speech synthesis.

The technical problems to be achieved by the present specification are not limited to the technical problems mentioned above, and other technical problems not mentioned are clear to those of ordinary skill in the art to which the present invention belongs from the detailed description of the invention below. can be understood clearly.

A speech synthesis method in a speech synthesis apparatus according to an embodiment of the present specification includes: preparing training data to be used for speech synthesis; learning the emotion classification model by fine-tuning the pre-trained language model with respect to the learning data; extracting an emotion element corresponding to the input sentence by applying the input sentence to the emotion classification model; and wherein the extracted emotion factor includes a plurality of emotion items having different emotion weight values, and performing speech synthesis based on the input sentence, embedded speaker information, and the extracted emotion factor; includes

The pre-trained language model is a BERT language model, and the fine adjustment is to perform transfer learning on the pre-learned language model by using a target emotion corresponding to the input sentence as a tag. can mean that

The emotion classification model includes a plurality of divided large classification emotion items, each large classification emotion item includes a plurality of similar sub-category emotion items, and the large classification emotion item and the plurality of sub-classification emotion items are each emotion vector value may be embedded in the speech synthesis device.

The speech synthesis method may include: matching the emotion vector value on a coordinate plane, and outputting a matching result through an output unit of the speech synthesis device; determining an emotion vector value of the selected point by referring to a previously matched emotion vector value on the coordinate plane when a point corresponding to the emotion vector value not embedded in the speech synthesis apparatus is selected on the coordinate plane; and performing the speech synthesis based on the determined emotion vector value.

The emotion classification model may be configured to output a large classification emotion item corresponding to an input sentence and a weight value of the large classification emotion item.

The emotion classification model may be configured to output a sub-classification emotion item of a large-class emotion item corresponding to an input sentence and a weight value of the sub-classification emotion item.

The emotion classification model extracts an emotion element corresponding to the input sentence by a combination of at least two or more of a large category emotion item corresponding to the input sentence or a plurality of sub-category emotion items included in the large category emotion item, wherein a combination of the two or more Each emotion item may have a different weight value.

The weight value of each of the two or more combined emotion items is adaptively adjustable.

In the embedded speaker information, a vector value corresponding to a plurality of different speaker information, a voicestone of each of the plurality of speakers, and a voice tone corresponding to an emotion classified in the emotion classification model may be embedded in the speech synthesis device. can

In the speech synthesis method, the embedded speaker information is matched on a coordinate plane through the vector value, and outputting a matching result through a display of the speech synthesis device; determining a vector value of the selected point by referring to a vector value previously matched on the coordinate plane when a point corresponding to speaker information not embedded in the speech synthesis apparatus is selected on the coordinate plane; and performing the speech synthesis based on the voicestone corresponding to the determined vector value.

A speech synthesis apparatus according to another embodiment of the present specification includes an input unit; a storage unit for storing a pre-trained language model with respect to learning data to be used for speech synthesis; a speech synthesis unit for performing speech synthesis on the input sentence input through the input unit; A processor for learning an emotion classification model by fine-tuning the pre-trained language model with respect to the learning data; includes, wherein the processor applies the input sentence to the fine-tuned emotion classification model, An emotion element corresponding to the input sentence is extracted, and the extracted emotion element includes a plurality of emotion items having different emotion weight values, the input sentence, speaker information embedded in the speech synthesis device, and the extracted The voice synthesizer may be controlled to perform voice synthesis based on the emotional element.

The processor further trains the emotion classification model by using the target decrease corresponding to the input sentence as a tag.

The emotion classification model includes a plurality of divided large classification emotion items, each large classification emotion item includes a plurality of similar sub-category emotion items, and the large classification emotion item and the plurality of sub-classification emotion items are each emotion vector value is embedded in the speech synthesis device.

The emotion vector value is matched on a coordinate plane, and the processor outputs a matching result through an output unit of the speech synthesizing device, and a point on the coordinate plane corresponding to the emotion vector value not embedded in the speech synthesizing device. is selected, the emotion vector value of the selected point may be determined by referring to the emotion vector value previously matched on the coordinate plane, and the voice synthesis may be performed based on the determined emotion vector value.

The emotion classification model extracts an emotion element corresponding to the input sentence by a combination of at least two or more of a large category emotion item corresponding to the input sentence or a plurality of sub-category emotion items included in the large category emotion item, wherein a combination of the two or more Each emotion item may have a different weight value.

The weight value of each of the two or more combined emotion items is adaptively adjustable.

Preferably, in the embedded speaker information, a vector value corresponding to a plurality of different speaker information, a voicestone of each of the plurality of speakers, and a voice tone corresponding to an emotion classified in the emotion classification model is provided to the speech synthesis apparatus. is embedded.

The processor matches the embedded speaker information on a coordinate plane through the vector value to output an output unit of the speech synthesis apparatus, and a point on the coordinate plane corresponding to speaker information not embedded in the speech synthesis apparatus is selected, a vector value of the selected point may be determined by referring to a vector value matched previously on the coordinate plane, and the voice synthesis may be performed based on a voicestone corresponding to the determined vector value.

An object of the speech synthesis method and apparatus according to an embodiment of the present specification is to provide a speech synthesis method and apparatus capable of adjusting emotion information of a synthesized speech in an emotion-based speech synthesis process.

In addition, according to an embodiment of the present specification, a more subdivided emotional state may be reflected in voice synthesis.

The effects obtainable in the present invention are not limited to the above-mentioned effects, and other effects not mentioned will be clearly understood by those of ordinary skill in the art to which the present invention belongs from the following description. .

BRIEF DESCRIPTION OF THE DRAWINGS The accompanying drawings, which are included as a part of the detailed description to help the understanding of the present specification, provide embodiments of the present specification, and together with the detailed description, explain the technical features of the present specification.
1 is a block diagram of an apparatus for synthesizing speech based on emotion information according to an embodiment of the present specification.
FIG. 2 is a diagram for explaining a schematic flow of speech synthesis in which emotions are reflected in an input sentence according to an embodiment of the present specification; FIG.
3 is a flowchart of a speech synthesis method according to an embodiment of the present specification.
4A is a diagram for explaining an example of learning an emotion classification model using a BERT language model according to an embodiment of the present specification.
4B is a diagram for explaining an example of an output result of an emotion classification model according to an embodiment of the present specification.
5 is a diagram for explaining a classification system for emotion information that can be classified through an emotion classification model according to an embodiment of the present specification.
6 is a diagram for explaining an example of a result of applying an emotion classification model to an input sentence according to an embodiment of the present specification.
7 is a flowchart of a speech synthesis method for explaining another example of applying a composite emotion weight according to an embodiment of the present specification.
8 is a diagram for explaining an example in which an embedded emotion vector value is displayed on a coordinate plane space.
9 illustrates a process of performing voice synthesis based on emotion information according to an embodiment of the present specification in more detail through functional blocks.
10 is a flowchart of a speech synthesis method for applying multi-speaker information according to an embodiment of the present specification.
BRIEF DESCRIPTION OF THE DRAWINGS The accompanying drawings, which are included as a part of the detailed description to facilitate the understanding of the present invention, provide embodiments of the present invention, and together with the detailed description, explain the technical features of the present invention.

Hereinafter, the embodiments disclosed in the present specification will be described in detail with reference to the accompanying drawings, but the same or similar components are assigned the same reference numbers regardless of reference numerals, and redundant description thereof will be omitted. The suffixes "module" and "part" for components used in the following description are given or mixed in consideration of only the ease of writing the specification, and do not have distinct meanings or roles by themselves. In addition, in describing the embodiments disclosed in the present specification, if it is determined that detailed descriptions of related known technologies may obscure the gist of the embodiments disclosed in the present specification, the detailed description thereof will be omitted. In addition, the accompanying drawings are only for easy understanding of the embodiments disclosed in the present specification, and the technical idea disclosed herein is not limited by the accompanying drawings, and all changes included in the spirit and scope of the present invention , should be understood to include equivalents or substitutes.

Terms including an ordinal number such as 1st, 2nd, etc. may be used to describe various elements, but the elements are not limited by the terms. The above terms are used only for the purpose of distinguishing one component from another.

When an element is referred to as being “connected” or “connected” to another element, it is understood that it may be directly connected or connected to the other element, but other elements may exist in between. it should be On the other hand, when it is said that a certain element is "directly connected" or "directly connected" to another element, it should be understood that the other element does not exist in the middle.

The singular expression includes the plural expression unless the context clearly dictates otherwise.

In the present application, terms such as “comprises” or “have” are intended to designate that a feature, number, step, operation, component, part, or combination thereof described in the specification exists, but one or more other features It should be understood that this does not preclude the existence or addition of numbers, steps, operations, components, parts, or combinations thereof.

The above-described specification, it is possible to be implemented as computer-readable code on a medium in which the program is recorded. The computer-readable medium includes all types of recording devices in which data readable by a computer system is stored. Examples of computer-readable media include Hard Disk Drive (HDD), Solid State Disk (SSD), Silicon Disk Drive (SDD), ROM, RAM, CD-ROM, magnetic tape, floppy disk, optical data storage device, etc. There is also a carrier wave (eg, transmission over the Internet) that is implemented in the form of. Accordingly, the above detailed description should not be construed as restrictive in all respects but as exemplary. The scope of the present specification should be determined by a reasonable interpretation of the appended claims, and all modifications within the equivalent scope of the present invention are included in the scope of this specification.

1 is a block diagram of an apparatus for synthesizing speech based on emotion information according to an embodiment of the present specification.

The voice synthesis device (TTS Device) 100 shown in FIG. 1 may include an audio output device 110 for outputting voice processed by the TTS device 100 or another device.

1 discloses a speech synthesis device (TTS Device) 100 for performing speech synthesis. An embodiment of the present invention may include computer readable and computer executable instructions that may be included in the TTS device 100 . Although FIG. 1 discloses a plurality of components included in the TTS apparatus 100 , it goes without saying that components that are not disclosed may be included in the TTS apparatus 100 .

Meanwhile, some components disclosed in the TTS device 100 may appear multiple times in one device as a single component. For example, the TTS device 100 may include a plurality of input devices 120 , an output device 130 , or a plurality of controllers/processors 140 .

A plurality of TTS apparatuses may be applied to one speech synthesis apparatus. In such a multi-device system, the TTS device may include different components for performing various aspects of speech synthesis processing. The TTS device 100 shown in FIG. 1 is exemplary, and may be an independent device, or may be implemented as a larger device or one component of a system.

An embodiment of the present invention may be applied to a plurality of different devices and computer systems, for example, a general-purpose computing system, a server-client computing system, a telephone computing system, a laptop computer, a portable terminal, a PDA, a tablet computer, and the like. have. The TTS device 100 includes automatic teller machines (ATMs), kiosks, global positioning systems (GPS), home appliances (eg, refrigerators, ovens, washing machines, etc.), vehicles, and e-book readers. Readers) may be applied as a component of another device or system that provides a voice recognition function.

Referring to FIG. 1 , the TTS device 100 may include a voice output device 110 for outputting a voice processed by the TTS device 100 or another device. The audio output device 110 may include a speaker, a headphone, or other suitable component for propagating audio. The audio output device 110 may be integrated into the TTS device 100 or implemented separately from the TTS device 100 .

The TTS device 100 may include an address/data bus 190 for transferring data between components of the TTS device 100 . Each component in the TTS device 100 may be directly connected to other components through the bus 190 . Meanwhile, each component in the TTS device 100 may be directly connected to the TTS module 170 .

The TTS device 100 may include a processor 140 . The processor 208 may correspond to a CPU for processing data, computer readable instructions for processing data, and a memory for storing data and instructions. The memory 150 may include volatile RAM, non-volatile ROM, or other types of memory.

The TTS device 100 may include a storage 160 for storing data and commands. The storage 160 may include magnetic storage, optical storage, a solid-state storage type, and the like.

The TTS device 100 may be connected to a removable or external memory (eg, a removable memory card, a memory key drive, network storage, etc.) through the input device 120 or the output device 130 .

Computer instructions to be processed by the processor 140 for operating the TTS apparatus 100 and various components may be executed by the processor 140 , and include the memory 150 , the storage 160 , and an external device. Alternatively, it may be stored in a memory or storage included in the TTS module 170 to be described later. Alternatively, all or a portion of the executable instructions may be embodied in hardware or firmware in addition to software. One embodiment of the present invention may be implemented in various combinations of, for example, software, firmware and/or hardware.

The TTS device 100 includes an input device 120 and an output device 130 . For example, the input device 120 may include an audio output device 110 such as a microphone, a touch input device, a keyboard, a mouse, a stylus, or other input device. The output device 130 may include a display (visual display or tactile display), an audio speaker, headphones, a printer, or other output device. Input device 120 and/or output device 130 may also include interfaces for connecting external peripherals, such as Universal Serial Bus (USB), FireWire, Thunderbolt, or other connection protocols. Input device 120 and/or output device 130 may also include a network connection, such as an Ethernet port, modem, or the like. Wireless communication devices such as radio frequency (RF), infrared, Bluetooth, wireless local area network (WLAN) (such as WiFi) or 5G networks, Long Term Evolution (LTE) networks, WiMAN networks, 3G networks, such as It may include a wireless network wireless device. TTS device 100 may include the Internet or distributed computing environment via input device 120 and/or output device 130 .

The TTS device 100 may include a TTS module 170 for processing text data into an audio waveform including voice.

TTS module 170 may be connected to bus 190 , input device 120 , output device 130 , audio output device 110 , processor 140 and/or other components of TTS device 100 . have.

The source of textual data may be generated by an internal component of the TTS device 100 . In addition, the source of the text data may be received from an input device such as a keyboard or transmitted to the TTS device 100 through a network connection. The text may be in the form of a sentence including text, numbers and/or punctuation for conversion into speech by the TTS module 170 . The input text may also include a special annotation for processing by the TTS module 170, and may indicate how the specific text should be pronounced through the special annotation. Text data can be processed in real time or stored and processed later.

The TTS module 170 may include a front end 171 , a speech synthesis engine 172 , and a TTS storage unit 180 . The preprocessor 171 may convert the input test data into a symbolic linguistic representation for processing by the speech synthesis engine 172 . The speech synthesis engine 172 may convert the input text into speech by comparing the annotated phonetic units models with information stored in the TTS storage unit 180 . The preprocessor 171 and the speech synthesis engine 172 may include an embedded internal processor or memory, or may use the processor 1400 and the memory 150 included in the TTS device 100 . Commands for operating the preprocessor 171 and the speech synthesis engine 172 may be included in the TTS module 170 , the memory 150 and storage 160 of the TTS device 100 , or an external device.

The text input to the TTS module 170 may be transmitted to the preprocessor 171 for processing. The preprocessor 1710 may include a module for performing text normalization, linguistic analysis, and linguistic prosody generation.

The preprocessor 171 processes text input and generates standard text while performing text normalization operation, and converts numbers, abbreviations, and symbols to be the same as written ones. .

The preprocessor 171 may analyze the language of the normalized text while performing the language analysis operation to generate a series of phonetic units corresponding to the input text. Such a process may be referred to as phonetic transcription. The phonetic units are finally combined and include a symbolic representation of the sound units output by the TTS device 100 as speech. Various sound units may be used to segment text for speech synthesis. The TTS module 170 includes phonemes (individual sounds), half-phonemes, di-phones (last half of one phoneme combined with the first half of adjacent phonemes), and bi-phones (bi-phones). It can process speech based on phones (two consecutive speeds of sound), syllables, words, phrases, sentences, or other units. Each word may be mapped to one or more phonetic units. Such mapping may be performed using a language dictionary stored in the TTS device 100 .

The language analysis performed by the preprocessor 171 also includes the process of identifying different grammatical elements such as prefixes, suffixes, phrases, punctuation, and syntactic boundaries. may include This grammatical component may be used by the TTS module 1700 to create a natural audio waveform output. The language dictionary may also include letter-to-sound rules and other tools that may be used to pronounce previously unidentified words or letter combinations that may be generated by the TTS module 170 . . In general, the more information included in the language dictionary, the higher the quality of audio output can be guaranteed.

Based on the linguistic analysis, the preprocessor 171 may generate linguistic prosody annotated in phonetic units with prosodic characteristics indicating how the final sound unit should be pronounced in the final output voice. have.

The prosody characteristics may also be referred to as acoustic features. While performing the operation of this step, the preprocessor 171 may consider arbitrary prosodic annotations accompanying text input and integrate them into the TTS module 170 . Such acoustic features may include pitch, energy, duration, and the like. The application of the acoustic characteristics may be based on prosodic models available to the TTS module 170 . This prosody model represents how phonetic units should be pronounced in a particular situation. For example, a prosody model can calculate a phoneme's position in a syllable, a syllable's position in a word, or a word's position in a sentence or phrase. phrase), and neighboring phonetic units. As in the language dictionary, as the amount of information in the prosodic model increases, high-quality speech output can be guaranteed.

The output of the preprocessor 171 may include a series of phonetic units annotated with prosodic characteristics. The output of the preprocessor 171 may be referred to as a symbolic linguistic representation. The symbolic language representation may be transmitted to the speech synthesis engine 172 . The speech synthesis engine 172 performs a process of converting speech into an audio waveform to output to the user through the audio output device 110 . The speech synthesis engine 172 may be configured to convert the input text into high quality natural speech in an efficient manner. Such high-quality speech can be configured to be pronounced as similar to a human speaker as possible.

The speech synthesis engine 172 may perform speech synthesis using at least one or more other methods.

The unit selection engine 173 compares the recorded speech database with the symbolic linguistic representation generated by the preprocessor 171 . A unit selection engine 173 matches the symbol language representation with the spoken audio units in a speech database. Matching units are selected to form a speech output, and the selected matching units can be connected together. Each unit contains only an audio waveform corresponding to a phonetic unit, such as a short ,wav file of a specific sound, with a description of the various acoustic properties associated with the .wav file (pitch, energy, etc.). Rather, it may contain other information such as a word, sentence or phrase, the location where the voice unit is displayed in a neighboring voice unit.

The unit selection engine 173 may match the input text using all information in the unit database to generate a natural waveform. The unit database may contain examples of multiple voice units that provide different options to the TTS device 100 for linking the units to speech. One of the advantages of unit selection is that a natural natural voice output can be generated according to the size of the database. In addition, as the unit database becomes larger, the TTS device 100 can compose a natural voice.

On the other hand, for speech synthesis, there is a parameter synthesis method in addition to the aforementioned unit selection synthesis. In parametric synthesis, synthesis parameters such as frequency, volume, and noise may be transformed by the parametric synthesis engine 175, a digital signal processor, or other audio generating device to generate an artificial speech waveform.

Parametric synthesis can use acoustic models and various statistical techniques to match a symbolic linguistic representation to desired output speech parameters. For parameter synthesis, not only can speech be processed without a large database related to unit selection, but also accurate processing is possible at a high processing speed. The unit selection synthesis method and the parameter synthesis method may be performed individually or may be performed in combination to generate an audio audio output.

Parametric speech synthesis may be performed as follows. The TTS module 170 may include an acoustic model capable of converting a symbolic linguistic representation into a synthetic acoustic waveform of a text input based on audio signal manipulation. The acoustic model may include rules that may be used by the parameter synthesis engine 175 to assign specific audio waveform parameters to input speech units and/or prosodic annotations. can The rule may be used to calculate a score indicating the likelihood that a particular audio output parameter (frequency, volume, etc.) will correspond to a portion of the input symbolic language representation from the preprocessor 171 .

The parametric synthesis engine 175 may apply a plurality of techniques to match the speech to be synthesized with the input speech units and/or prosody annotations. One common technique uses the Hidden Markov Model (HMM), which can be used to determine the probability that the audio output should match the text input. The HMM can be used to convert parameters of speech and acoustic space into parameters to be used by a vocoder (digital voice encoder) to artificially synthesize a desired voice.

The TTS device 100 may include a voice unit database for use in unit selection.

The voice unit database may be stored in TTS storage 180 , storage 160 , or other storage configuration. The voice unit database may contain recorded speech utterances. The speech utterance may be text corresponding to the content of the utterance. In addition, the voice unit database may contain recorded voices (in the form of audio waveforms, feature vectors, or other formats) that occupy significant storage space in the TTS device 100 . Unit samples of the speech unit database may be classified in various ways including speech units (phonemes, diphones, words, etc.), linguistic prosody labels, acoustic feature sequences, speaker identities, and the like. Sample utterances can be used to create a mathematical model corresponding to a desired audio output for a particular speech unit.

When the speech synthesis engine 172 matches a symbolic linguistic representation, it may select a unit in the speech unit database that most closely matches the input text (including both phonetic units and prosody notes). In general, the larger the voice unit database, the greater the number of selectable unit samples, so that accurate speech output is possible.

Audio waveforms including audio output from the TTS module 213 may be transmitted to the audio output device 110 for output to a user. Audio waveforms, including speech, may be stored in a plurality of different formats, such as a series of feature vectors, uncompressed audio data, or compressed audio data. For example, the audio output may be encoded and/or compressed by an encoder/decoder prior to said transmission. The encoder/decoder may encode and decode audio data such as digitized audio data, feature vectors, and the like. In addition, the function of the encoder/decoder may be located in a separate component, or may be performed by the processor 140 and the TTS module 170, of course.

Meanwhile, the TTS storage 180 may store other information for speech recognition.

The contents of the TTS storage 180 may be prepared for general TTS use, and may be customized to include sounds and words likely to be used in a specific application. For example, for TTS processing by a GPS device, TTS storage 180 may contain customized voices specific to location and navigation.

Also for example, TTS storage 180 may be customized to a user based on a personalized desired voice output. For example, the user may prefer the output voice to a specific gender, a specific accent, a specific speed, and a specific emotion (eg, a happy voice). The speech synthesis engine 172 may include a special database or model to describe such user preferences.

The TTS device 100 may also be configured to perform TTS processing in multiple languages. For each language, the TTS module 170 may include data, commands and/or components specifically configured to synthesize speech in the desired language.

In order to improve performance, the TTS module 213 may modify or update the contents of the TTS storage 180 based on the feedback on the TTS processing result, so that the TTS module 170 is capable of being provided by a training corpus. As described above, speech recognition can be improved.

As the processing capability of the TTS device 100 is improved, voice output is possible by reflecting the emotional property of the input text. Alternatively, the TTS device 100 may output the voice by reflecting the intention (emotion information) of the user who wrote the input text even if the input text is not included in the emotion attribute.

When a model to be integrated into a TTS module that actually performs TTS processing is built, the TTS system may integrate the various components mentioned above and other components. For example, the TTS device 100 may insert an emotional element into the voice.

In order to output the voice to which the emotion information is added, the TTS device 100 may include an emotion insertion module 177 . The emotion insertion module 177 may be integrated into the TTS module 170 , or may be integrated as a part of the preprocessor 171 or the speech synthesis engine 172 . The emotion insertion module 177 may implement speech synthesis based on emotion information by using metadata corresponding to the emotion attribute.

FIG. 2 is a diagram for explaining a schematic flow of speech synthesis in which emotions are reflected in an input sentence according to an embodiment of the present specification; FIG.

Referring to FIG. 2 , the speech synthesis system according to an embodiment of the present specification may include an emotion recognition system and an emotion speech synthesis system. The emotion recognition system may receive an input sentence and extract an emotion corresponding to the input sentence. The emotion recognition system may extract a plurality of emotion elements from the input sentence. The plurality of extracted emotional elements may be three emotions of sadness, hurt, and anxiety, and the three emotions are similar emotional states, but mixed emotional states may be extracted at ratios of 61%, 32%, and 7%, respectively. The emotion speech synthesis system receives the extracted emotion information and the input sentence together as input data, and generates and outputs an audio waveform.

3 is a flowchart of a speech synthesis method according to an embodiment of the present specification. The speech synthesis method of FIG. 3 may be implemented in a speech synthesis apparatus. More specifically, the speech synthesis method may be implemented by a processor and/or a controller of the speech synthesis apparatus.

Referring to FIG. 3 , the processor may prepare training data ( S300 ).

Here, the learning data may include the contents of a conversation between the system and a person. Alternatively, as the learning data, a total of 60 emotion set data may be provided, including 6 major category emotion items and 10 small category emotion items belonging to each of the major category emotion items.

The processor may learn the emotion classification model through fine-tuning with respect to the pre-learning model (S310).

The pre-trained language model may be a BERT language model. The BERT language model may be a language model that can further improve the performance of a specific task through pre-training embedding before performing a specific task. When using the BERT language model, classification was performed using machine learning models such as LSTM and CNN for data classification. After performing fine tuning by transferring it, the NLP task can be performed. A desired task (emotion classification) can be performed by applying the BERT language model as a corpus for mass speech synthesis and applying additional models (machine learning models such as RNN and CNN) to the output of the BERT language model. In this case, even if only a simple DNN model is applied without applying a complicated CNN, LSTM, or Attention to the additional model, there is little difference in performance compared to when a complex model such as CNN is used.

4A is a diagram for explaining an example of learning an emotion classification model using a BERT language model according to an embodiment of the present specification. 4B is a diagram for explaining an example of an output result of an emotion classification model according to an embodiment of the present specification. 5 is a diagram for explaining a classification system for emotion information that can be classified through an emotion classification model according to an embodiment of the present specification. 6 is a diagram for explaining an example of a result of applying an emotion classification model to an input sentence according to an embodiment of the present specification.

Referring to FIG. 4A , the fine adjustment may mean performing transfer learning on the pre-training language model by using the target emotion corresponding to the input sentence as a tag. have. In other words, it is possible to perform the actual NLP task by transferring the pre-trained language model. An embodiment of the present specification trains a pre-trained language model to perform an NLP task capable of classifying emotions on an input sentence.

Referring to FIG. 4B , the processor applies an input sentence to the emotion classification model, and reflects the output result of the emotion classification model to speech synthesis. According to an embodiment, the emotion classification model may include a plurality of divided large-class emotion items, and each large-class emotion item may include a plurality of similar sub-category (sub-classification) emotion items. The emotion classification model may be configured to output a large classification emotion item corresponding to an input sentence and a weight value of the large classification emotion item. In addition, the emotion classification model may be configured to output a sub-category of the large classification emotion item and a weight value of the sub-classification emotion item corresponding to the input sentence. In addition, the emotion classification model extracts an emotion element corresponding to the input sentence by a combination of at least two or more of a large category emotion item corresponding to the input sentence or a plurality of sub-category emotion items included in the large category emotion item, The above combined emotion items may be configured to have different weight values, respectively.

That is, according to an embodiment of the present specification, as a result of applying the input sentence to the emotion classification model, one major category 1 (eg, sadness) may be output.

Also, for example, a first weight value may be assigned to the major category 1 (eg, sadness), and a second weight may be assigned to the sub-class 2 (eg, depressed) of the large category 1 . In this case, a weight may be applied to a plurality of emotion items such that the sum of the first weight value and the second weight value is 1. In this case, according to an embodiment of the present specification, the emotional weight of speech synthesis may be controlled so that the input sentence reflects not only a sad emotional state but also a depressed and sad emotional state. A third weight is added to sub-class 3 (eg, pessimistic) instead of sub-category 2 (eg, sadness) of the large classification 1, so that the emotion classification result is displayed along with the large classification 1 to which the first weight is added. When output, the "sad + pessimistic" emotional state will output a speech synthesis result different from the "sad + depressed" emotional state. In this case, the result of speech synthesis can more accurately reflect the emotional state inherent in the input sentence, compared to the case where only the major category 1 (eg, sadness) is output in the output value of the emotion classification model.

Also, for example, as a result of applying the input sentence to the emotion classification model, only a plurality of sub-classifications (sub-classifications) included in the large classification, not the large classification indicating the representative emotional state of a broad concept, may be output as a result. For example, a first weight is added to subcategory 1 (eg, stressed) of a major category 1 (eg, anxiety), and a second weight is added to subcategory 2 (eg, confused). and a third weight may be added to subclass 3 (eg, anxious) to output a final result. In this case, the processor may control the three emotional states of “stressed + confused + anxious” to be reflected in the final speech synthesis by applying the first weight, the second weight, and the third weight, respectively, to the input sentence. . In this case, too, there is an advantage in that the emotional state can be reflected in the voice synthesis more precisely compared to the result of simply reflecting the 'anxiety' emotional state in the voice synthesis.

5 is a diagram for explaining a classification system for emotion information that can be classified through an emotion classification model according to an embodiment of the present specification.

Referring to FIG. 5 , according to an embodiment of the present specification, emotional states are divided into six major categories (anger, sadness, anxiety, hurt, embarrassment, joy), and all emotional states included in each major category are 10 It is composed of dogs and can be divided into a total of 60 emotion items. Each emotion item can be managed in the form of a table by assigning a code. The emotion classification data in the form of a table shown in FIG. 5 is exemplary, and the number of large classifications and the number of small classifications may be further increased, and when the emotion classification data in the table form is updated, based on the updated emotion classification data, By performing the fine-tuning process again on the pre-trained language model, the emotion classification model can be updated.

6 is a diagram for explaining an example of a result of applying an emotion classification model to an input sentence according to an embodiment of the present specification.

Referring to FIG. 6 , when the input sentence is “happy” according to an embodiment of the present specification, a first weight (0.221) is added to the sixth major category (joy), and a fourth sub-classification (0.221) of the sixth major category ( A result may be output by adding a second weight (0.716) to ).

The processor may adaptively adjust the first weight and the second weight. For example, the example shown in FIG. 6 is a result of the emotion classification model applying the first weight and the second weight as a default, and the first weight and the second weight may be changed through a user input. Also, for example, the processor may automatically change the values of the first weights and the second weights by analyzing the sentences input later in the state in which the first weights and the second weights are output. Of course, the criteria for adaptively adjusting the first weight and the second weight are not limited to the above-described examples, and various criteria may be applied. Accordingly, according to one embodiment of the present specification, complex and subtle human emotional states can be more precisely classified and used for speech synthesis, and in particular, when applied to the emotion classification model for sentences described in the script of a movie or drama, As described above, the weight may be automatically controlled in consideration of the emotional state of the previous sentence and the next sentence.

According to an embodiment of the present specification, emotion information may be embedded in the speech synthesis apparatus in the form of a fixed emotion vector value. As the number of types and branches of the emotion information increases, the capacity of the emotion vector value increases. Therefore, the type of emotion information is limitedly embedded in consideration of system resources and the like. Hereinafter, another example of synthesizing voices including various emotions through complex emotion weights in consideration of the system resources will be described with reference to FIG. 7 .

7 is a flowchart of a speech synthesis method for explaining another example of applying a complex emotion weight according to an embodiment of the present specification, and FIG. 8 is for explaining an example in which an embedded emotion vector value is displayed on a coordinate plane space. It is a drawing.

Referring to FIG. 7 , the processor may match the embedded emotion vector value on the coordinate plane and display it through the output unit of the speech synthesis apparatus ( S700 ).

Referring to FIG. 8 , the embedded emotion vector values may be grouped by type of emotion item and disposed on a planar space. For example, if it is assumed that emotions that can be classified through the emotion classification model are divided into 10 major categories, each major category may include a plurality of sub-class emotion items including similar emotion items, and a plurality of sub-class emotions Items may be arranged forming one cluster (801,802,803, ?,809,810) on the coordinate space. In FIG. 8 , the blank area in which the emotion vector cluster does not exist may mean that matching emotion information does not exist.

The processor may select a specific point on the coordinate plane (S710).

When the specific point is a point where the emotion vector cluster exists, since there is an embedded emotion vector, the processor may reflect the corresponding emotion vector to speech synthesis (S740). However, when the specific point is a point corresponding to an emotion vector value that is not embedded in the speech synthesis apparatus (eg, area 830 in FIG. 8 ), the processor determines a previously matched emotion vector value on the coordinate plane ( The emotion vector value of the selected point may be determined with reference to (emotion vector values corresponding to regions 801,802,803, ?,809,810) (S730).

The processor may perform voice synthesis by generating a voice signal to be synthesized based on the determined emotion vector value.

9 illustrates a process of performing voice synthesis based on emotion information according to an embodiment of the present specification in more detail through functional blocks.

Referring to FIG. 9 , the speech synthesis apparatus according to an embodiment of the present specification may embed a plurality of different speaker information and a plurality of different emotional information. In the embedded speaker information, a vector value of a plurality of different speakers, a voicestone of each of the plurality of speakers, and a voicestone corresponding to an emotion classified in the emotion classification model may be embedded in the speech synthesis apparatus.

In the process of using the emotion vector value for speech synthesis in the above-described embodiment, a speaker embedding vector value corresponding to the emotion vector value may be considered together. For example, the processor may select speaker information that can best reflect the emotion information determined according to the emotion classification model, and use the selected speaker's voice for speech synthesis. In addition, a voice that can best reflect the classified emotion among the selected speaker's voices may be selected and used for voice synthesis. To this end, in the information embedded in the speech synthesis apparatus according to an embodiment of the present specification, speaker information may be embedded in addition to the emotion vector, and the speaker information may also be embedded in the form of a vector value. In addition, the embedded emotion vector may be embedded for each emotion type.

When an input sentence is input, the processor may determine emotion information corresponding to the input sentence through the emotion classification model, and extract an emotion vector value corresponding to the determined emotion information. Also, the processor may select a speaker matching the extracted emotion vector value. The processor may determine a speaker vector value matching the extracted emotion vector value from among the plurality of embedded speaker information.

The processor may perform a speech synthesis process based on an encoding result of the input sentence, embedded emotion information, and embedded speaker information.

On the other hand, similar to the method described in the process of selecting emotion information with reference to FIGS. 7 to 8 , in the present specification, voice synthesis may be performed based on a voicestone other than an embedded speaker.

10 is a flowchart of a speech synthesis method for applying multi-speaker information according to an embodiment of the present specification.

Referring to FIG. 10 , the processor may match the speaker information (speaker vector value) embedded in the speech synthesis apparatus on a coordinate plane and output it through the output unit of the speech synthesis apparatus ( S1000 ).

Similar to FIG. 8 , the embedded speaker vector values may be grouped for each speaker and disposed on a planar space. For example, if the voices of 10 speakers are stored in the speech synthesis device, the processor arranges the speaker vectors of each of the 10 speakers in a flat space, and the voices of the 10 speakers are stored separately as different voicestones for each emotion. can For example, if one speaker records a speaker voice with voices corresponding to 6 different emotions, 6 different emotional voicestones for one speaker may be arranged in one cluster area on the coordinate space, respectively. have.

When a specific point is selected on the coordinate plane space in which the speaker information is batched (S1100), the processor may determine whether a speaker vector value corresponding to the selected point is embedded in the speech synthesis apparatus (S1200).

When the specific point is a point where a speaker vector exists, the processor may reflect the voicestone of the corresponding speaker to the speech synthesis because the embedded speaker vector exists ( S1400 ). However, when the specific point is a point corresponding to a speaker vector value not embedded in the speech synthesis apparatus, the processor may determine the speaker vector value of the selected point by referring to the speaker vector value previously matched on the coordinate plane. There is (S1300). Then, the processor may generate a speech signal to be synthesized based on the determined speaker vector value.

When the specific point is a point where the emotion vector cluster exists, since there is an embedded emotion vector, the processor may reflect the corresponding emotion vector to speech synthesis (S740). However, when the specific point is a point corresponding to an emotion vector value that is not embedded in the speech synthesis apparatus (eg, area 830 in FIG. 8 ), the processor determines a previously matched emotion vector value on the coordinate plane ( The emotion vector value of the selected point may be determined with reference to (emotion vector values corresponding to regions 801,802,803, ?,809,810) (S730).

The processor may perform voice synthesis by generating a voice signal to be synthesized based on the determined emotion vector value.

The present invention described above can be implemented as computer-readable codes on a medium in which a program is recorded. The computer-readable medium includes all types of recording devices in which data readable by a computer system is stored. Examples of computer-readable media include Hard Disk Drive (HDD), Solid State Disk (SSD), Silicon Disk Drive (SDD), ROM, RAM, CD-ROM, magnetic tape, floppy disk, optical data storage device, etc. There is also a carrier wave (eg, transmission over the Internet) that is implemented in the form of. Accordingly, the above detailed description should not be construed as restrictive in all respects but as exemplary. The scope of the present invention should be determined by a reasonable interpretation of the appended claims, and all modifications within the equivalent scope of the present invention are included in the scope of the present invention.

Claims (11)

preparing training data to be used for speech synthesis;
learning the emotion classification model by fine-tuning the pre-trained language model with respect to the learning data;
extracting an emotion element corresponding to the input sentence by applying the input sentence to the emotion classification model; and
The extracted emotion element includes a plurality of emotion items having different emotion weight values,
performing speech synthesis based on the input sentence, embedded speaker information, and the extracted emotional element;
A speech synthesis method comprising a.
The method of claim 1,
The pre-trained language model is a BERT language model,
The fine adjustment is
A speech synthesis method, characterized in that transfer learning is performed on the pre-learned language model by using a target emotion corresponding to the input sentence as a tag.
The method of claim 1,
The emotion classification model is
Including a plurality of divided large-category analysis items, each large-category analysis item includes a plurality of similar sub-category analysis items,
The speech synthesizing method according to claim 1, wherein the major category emotion item and the plurality of sub-category emotion items are embedded in the speech synthesis device as emotion vector values, respectively.
4. The method of claim 3,
matching the emotion vector value on a coordinate plane, and outputting a matching result through a display of the voice synthesizer;
determining an emotion vector value of the selected point by referring to a previously matched Gapjeong vector value on the coordinate plane when a point corresponding to the emotion vector value not embedded in the speech synthesis apparatus is selected on the coordinate plane; and
performing the speech synthesis based on the determined emotion vector value;
Speech synthesis method further comprising a.
4. The method of claim 3,
The emotion classification model is
Speech synthesis method, characterized in that it is configured to output a large classification emotion item corresponding to an input sentence and a weight value of the large classification emotion item.
4. The method of claim 3,
The emotion classification model is
Speech synthesis method, characterized in that it is configured to output a sub-class emotion item of a large-class emotion item that matches an input sentence and a weight value of the sub-class emotion item.
4. The method of claim 3,
The emotion classification model is
Extracting an emotion element corresponding to the input sentence by a combination of at least two or more of a large category emotion item corresponding to the input sentence or a plurality of sub-category emotion items included in the large category emotion item,
The speech synthesis method, characterized in that the two or more combined emotion items each have different weight values.
8. The method of claim 7,
The speech synthesis method, characterized in that the weight value of each of the two or more combined emotion items is adaptively adjustable.
The method of claim 1,
The embedded speaker information is
The speech synthesis method according to claim 1, wherein a vector value of a plurality of different speaker information, a voicestone of each of the plurality of speakers, and a voice tone corresponding to an emotion classified in the emotion classification model are embedded in the speech synthesis apparatus.
10. The method of claim 9,
The embedded speaker information is
matching on a coordinate plane through the vector value, and outputting a matching result through a display of the voice synthesizer;
determining a vector value of the selected point by referring to a vector value previously matched on the coordinate plane when a point corresponding to speaker information not embedded in the speech synthesis apparatus is selected on the coordinate plane; and
performing the speech synthesis based on the voicestone corresponding to the determined vector value;
Speech synthesis method further comprising a.
input unit;
a storage unit for storing a pre-trained language model with respect to learning data to be used for speech synthesis;
a speech synthesis unit for performing speech synthesis on the input sentence input through the input unit;
A processor for learning the emotion classification model by fine-tuning the pre-trained language model with respect to the learning data;
The processor is
applying the input sentence to the fine-tuned emotion classification model to extract an emotion element corresponding to the input sentence, wherein the extracted emotion element includes a plurality of emotion items having different emotion weight values; and controlling the speech synthesis unit to perform speech synthesis based on a sentence, speaker information embedded in the speech synthesis device, and the extracted emotional element.

Images