KR20210012265A - Providing method of voice, learning method for providing voice and apparatus thereof - Google Patents

Abstract

According to one embodiment, a voice providing method and device can be provided, wherein the voice providing method receives a text sequence, extracts a first feature vector from the text sequence, generates a second feature vector by applying an embedding vector previously learned for each condition to the first feature vector, determines context information of the text from the second feature vector, synthesizes a voice corresponding to the text sequence based on the context information, and provides the synthesized voice. Therefore, the voice providing method and device may provide a personalized text-to-speech (TTS) service for which various emotions and tones of a speaker are reflected.

Description

Voice providing method, learning method for providing voice, and devices thereof {PROVIDING METHOD OF VOICE, LEARNING METHOD FOR PROVIDING VOICE AND APPARATUS THEREOF}

The embodiments relate to a method of providing a voice, a method of learning for providing a voice, and devices thereof.

With the development of artificial intelligence technology, various voice services (e.g., sound source provision, control through voice recognition, autonomous driving, augmented reality, etc.) are being provided through smart devices such as AI (Artificial Intelligence) secretaries or AI speakers. . In addition, with the advent of the 5G era, which is the era of hyper-connectivity, incoming traffic from various smart devices has increased, and it is necessary to provide individual voice services that have been learned in the optimal domain for each device. However, at present, it is common to simply synthesize and provide a voice corresponding to an input text, and it is difficult to provide a speaker's various emotions, tones, and expressions in various languages.

According to an embodiment, a personalized speech synthesis (Text-To-Speech; TTS) service in which various emotions and tones of a speaker are reflected may be provided.

According to an embodiment, text may be converted into various languages and provided.

According to an embodiment, a voice reflecting various variables such as gender, age, and ambient noise may be generated through a personalized voice synthesis (TTS) service.

According to an embodiment, training data suitable for a new service and domain can be immediately generated and learned, thereby reducing the cost of generating training data and improving performance of a speech recognition engine.

According to an embodiment, a method for synthesizing a speech includes receiving a text sequence; Extracting a first feature vector from the text sequence; Generating a second feature vector by applying a pre-learned embedding vector for each condition to the first feature vector; Determining context information of text from the second feature vector; Synthesizing a speech corresponding to the text sequence based on the context information; And providing the synthesized voice.

The condition may include at least one of emotion, tone, and language conversion.

The generating of the second feature vector may include generating the second feature vector by applying a pre-learned embedding vector for each of a plurality of emotions; Generating the second feature vector by applying a pre-learned embedding vector for each of a plurality of tones; And generating the second feature vector by applying pre-learned embedding vectors for each of a plurality of languages for language conversion.

The plurality of sensibilities may include at least one of happiness, joy, sadness, depression, anger, joy, youthfulness, lightness, and irritation.

The plurality of tones may include at least one of a voice tone and a speed change.

The voice providing method further includes receiving a selection for a personalized condition from a user, and extracting the first feature vector comprises determining a personalized TTS (Text to Speech) engine corresponding to the personalized condition. ; And outputting the first feature vector by applying the text sequence to the personalized TTS engine.

The personalization condition may include at least one of the user's age, the user's gender, and the user's preferred voice frequency band.

The synthesizing of the speech may include predicting a distribution of intonation, intensity, tone, and frequency corresponding to the text sequence based on the context information; And synthesizing the speech using the predicted intonation, intensity, tone and the predicted frequency distribution.

The speech synthesis method includes mixing the synthesized speech with noise related to a speaker; And outputting the sound mixed with the noise.

According to an embodiment, a learning method includes receiving reference audios corresponding to a plurality of emotions; Training a network for classifying each of the reference speech signals according to the plurality of emotions; Collecting prosody features corresponding to sensibilities of the reference speech signals using the trained network; And determining an embedding vector for each of the plurality of emotions based on the prosody features.

The plurality of sensibilities may include at least one of happiness, joy, sadness, depression, anger, joy, youthfulness, lightness, and irritation.

The collecting of the prosody features may include extracting prosody features common to a specific emotion of the reference voice signals irrespective of the voice content of the reference voice signals.

The learning method may further include training a network for classifying each of the reference speech signals into a plurality of tones and a plurality of languages.

The learning method further includes receiving a noise signal corresponding to an environment in which the reference speech signals are collected, and training a network for classifying each of the plurality of emotions includes the reference speech signals and the reference speech signal Mixing a noise signal corresponding to the environment in which they are collected; And training a network for classifying the mixed signal according to the plurality of emotions and the environment.

According to an embodiment, a speech synthesis apparatus includes: a communication interface for receiving a text sequence; Extracting a first feature vector from the text sequence, applying a pre-learned embedding vector for each condition to the first feature vector to generate a second feature vector, and determining context information of the text from the second feature vector, A processor for synthesizing a speech corresponding to the text sequence based on the context information; And a speaker outputting the synthesized voice.

According to an embodiment, the learning apparatus includes a communication interface for receiving reference voice signals corresponding to a plurality of emotions; And training a network for classifying each of the reference speech signals according to the plurality of emotions, and collecting prosody features corresponding to the sensations of the reference speech signals using the trained network, based on the prosody characteristics. And a processor for determining an embedding vector for each of the plurality of emotions, and the communication interface outputs the embedding vector.

According to one side, it is possible to provide a personalized speech synthesis (Text-To-Speech; TTS) service in which various emotions and tones of the speaker are reflected.

According to one side, text may be converted into various languages and provided.

According to one side, a voice reflecting various variables such as gender, age, and ambient noise may be generated through a personalized voice synthesis (TTS) service.

According to one side, training data suitable for a new service and domain can be immediately generated and learned, thereby reducing the cost of generating training data and improving the performance of the speech recognition engine.

1 is a flowchart illustrating a method of providing a voice according to an embodiment.
2 is a diagram illustrating a process of generating a synthesized speech according to an embodiment.
3 is a diagram for explaining a process of learning a speech recognition engine according to an embodiment.
4 is a diagram for explaining the configuration and operation of a speech synthesis (TTS) manager according to an embodiment.
5 is a flow chart showing a learning method according to an embodiment
6 is a diagram for describing a process of training a speech synthesis (TTS) model according to an embodiment.
Fig. 7 is a block diagram of an apparatus for providing a voice according to an embodiment.
Fig. 8 is a block diagram of a learning device according to an embodiment.

Hereinafter, exemplary embodiments will be described in detail with reference to the accompanying drawings. The same reference numerals in each drawing indicate the same members.

Various changes may be made to the embodiments described below. The embodiments described below are not intended to be limited to the embodiments, and should be understood to include all changes, equivalents, and substitutes thereto.

The terms used in the examples are used only to describe specific embodiments, and are not intended to limit the embodiments. Singular expressions include plural expressions unless the context clearly indicates otherwise. In the present specification, terms such as "comprise" or "have" are intended to designate the presence of features, numbers, steps, actions, components, parts, or combinations thereof described in the specification, but one or more other features. It is to be understood that the presence or addition of elements or numbers, steps, actions, components, parts, or combinations thereof, does not preclude in advance.

Unless otherwise defined, all terms used herein, including technical or scientific terms, have the same meaning as commonly understood by one of ordinary skill in the art to which the embodiment belongs. Terms as defined in a commonly used dictionary should be interpreted as having a meaning consistent with the meaning in the context of the related technology, and should not be interpreted as an ideal or excessively formal meaning unless explicitly defined in this application. Does not.

In addition, in the description with reference to the accompanying drawings, the same reference numerals are assigned to the same components regardless of the reference numerals, and redundant descriptions thereof will be omitted. In describing the embodiments, when it is determined that a detailed description of related known technologies may unnecessarily obscure the subject matter of the embodiments, the detailed description thereof will be omitted.

1 is a flow chart showing a method for providing voice according to an embodiment. Referring to FIG. 1, a speech providing apparatus according to an embodiment receives a text sequence (110).

The speech providing apparatus extracts a first feature vector from the text sequence (120). The speech providing apparatus may extract the first feature vector by applying the text sequence received in step 110 to a personalized text to speech (TTS) engine. Depending on the embodiment, the voice providing device may receive a selection of a personalization condition from a user. In this case, the speech providing apparatus may output the first feature vector by determining a personalized TTS engine corresponding to the personalization condition and applying the text sequence to the personalized TTS engine. Personalization conditions may include, for example, a user's age, a user's gender, a user's preferred voice frequency band, and the like.

The speech providing apparatus generates a second feature vector by applying a pre-learned embedding vector for each condition to the first feature vector (130). Here, the condition may include, for example, emotion, tone, and language conversion. The speech providing apparatus may generate a second feature vector by applying an embedding vector learned in advance for each of a plurality of emotions. In this case, the plurality of sensibilities may include, for example, happiness, joy, sadness, depression, anger, joy, youthfulness, lightness, irritation, and the like. In addition, the plurality of tones may include, for example, a voice tone and a speed change. The speech providing apparatus may generate a second feature vector by applying an embedding vector learned in advance for each of a plurality of tones. Alternatively, the apparatus for providing speech may generate the second feature vector by applying pre-learned embedding vectors for each of a plurality of languages for language conversion.

The speech providing device determines context information of the text from the second feature vector (140). The speech providing apparatus may determine text context information from the second feature vector using, for example, the attention model. The attention model can predict the next spectrogram from the context information of the text. The spectrogram is, for example, a sound spectrogram and may represent the distribution of intonation, intensity, tone, and frequency.

The speech providing apparatus synthesizes speech corresponding to the text sequence based on the context information (150). The speech providing apparatus may predict a distribution of intonation, intensity, tone and frequency corresponding to the text sequence, for example, based on context information. The speech providing apparatus may synthesize speech using the predicted intonation, intensity, tone, and predicted frequency distribution.

The voice providing device outputs the synthesized voice (160). Depending on the embodiment, the apparatus for providing a voice may mix noise related to a speaker with the synthesized voice, and may output a voice mixed with the noise. Here, the noise related to the speaker corresponds to noises generated during everyday life at school, for example, when the speaker is a student, and when the speaker is an office worker, it corresponds to noises generated during daily life at the company. I can. As described above,'noise related to the speaker' may be used as a meaning of encompassing all of the various noises that may occur in relation to the speaker's age range, the main living space, and the main living environment.

The voice providing apparatus may be, for example, a built-in apparatus with the voice synthesis apparatus 200 described with reference to FIG. 2 below, or may include a separate voice synthesis apparatus. The voice providing device may be, for example, a device including an AI assistant function such as a smart phone, an AI speaker, an Internet of Things (IoT) device, and a set-top box, or other smart home appliances, smart vehicles It may be, and may be devices that perform similar functions.

2 is a diagram illustrating a process of generating a synthesized speech according to an embodiment. Referring to FIG. 2, a speech synthesis (TSS) apparatus 200 according to an exemplary embodiment transmits text to speech through an end-to-end based deep neural network (DNN). It can be synthesized. The deep neural network includes, for example, a text encoder 210, a style embedding module 220, a conditioning module 230, an attention model 240, and a decoder. ) 250 may be included.

It is assumed that a text sequence is input to the speech synthesis (TSS) device 200. In this case, the text encoder 210 may convert the input text sequence into a number or vector representing characteristics of the text sequence. The text encoder 210 may determine a feature vector (eg, a first feature vector) representing a feature of the text sequence from the input text sequence. The text encoder 210 may be configured as a CBHG network including, for example, a primary convolution bank, a highway network, and a bidirectional gated recurrent unit (GRU). For example, when the speech synthesis (TSS) apparatus 200 is learned, text and a correct answer sound source may be input together.

The style embedding module 220 may provide pre-learned embedding vectors for each condition. At this time, the condition is the user's emotions (e.g., happiness, joy, sadness, depression, anger, pleasure, cheerfulness, cheerfulness, irritability, etc.), the user's tone (e.g., voice tone, speed change, etc.), and language Conversion (eg, English->Korean, Chinese->Korean, German->English, etc.) may be included.

The conditioning module 230 may generate a second feature vector by applying a pre-learned embedding vector for each condition to the first feature vector determined by the text encoder 210.

The attention model 240 may determine context information of the text from the second feature vector determined through the conditioning module 230. The attention model 240 may predict a next spectrogram from context information of text. The spectrogram is, for example, a sound spectrogram and may represent intonation, intensity, tone and frequency distribution.

The decoder 250 may generate a next spectrogram based on the spectrogram after prediction by the attention model 240. The attention model 240 and the decoder 250 may be composed of, for example, an attention-recurrent neural network (RNN). The decoder 250 may output the synthesized voice by converting the next spectrogram into a sound source or voice.

The decoder 250 may include a vocoder (not shown). The vocoder can convert parameter values for each phoneme into a sound source or voice. The vocoder can be performed by converting the spectrogram generated by the decoder 250 into a sound source or voice. The vocoder may include, for example, a Griffin-Lim vocoder algorithm and a Neural vocoder.

3 is a diagram illustrating a process of learning a speech recognition engine according to an exemplary embodiment. Referring to FIG. 3, an operation when a request for generating training data for a speech recognition engine is received by a learning device according to an embodiment is illustrated.

When a request to generate training data is received (310), the learning device may generate training data through, for example, a voice synthesis manager (TTS manager) (320). The speech synthesis manager generates optimal learning data using, for example, a speech synthesis module of a general unit selection system (USS), or generates optimal learning data by using a boilerplate recorded frequently used sentences. Alternatively, various speech characteristics, noise, and frequency formats may be set through a deep learning-based personalized speech synthesis module to generate optimal learning data required for speech recognition. The voice synthesis module of the phoneme splicing method may generate a sound source by recording, for example, a large number of sound sources, converting a database (DB) into a phoneme unit, and splicing a requested character into an optimal phoneme unit. In addition, the deep learning speech synthesis module may generate a sound source by predicting sound information by modeling based on deep learning by extracting characteristic parameters of speech, for example.

For example, let's say that the learning data that the learning device wants to generate is the learning data for the kids service. The learning device may generate desired learning data by mixing voices of children under 9 years of age and noise generated in school. A method of generating the learning data by the speech synthesis manager will be described in detail with reference to FIG. 4 below.

The learning device may store the training data generated in step 320 in various storages such as, for example, NAS storage or AWS S3 (330).

The learning device may train optimal data of a Speech To Text (STT) module for each service or domain through the learning data (340).

For example, in the case of providing a navigation service, the learning device may include gender, age, vehicle noise (e.g., noise when starting/off, 30 km/h per hour, 50 km/h per hour, 100 km/h per hour, Optimal learning data can be generated by setting various parameters or conditions for noise when driving 120 km/h per hour and noise due to window on/off). In this way, the learning device can train the STT module through optimal learning data for various services.

4 is a diagram illustrating a configuration and operation of a speech synthesis (TTS) manager according to an embodiment. 4, the speech synthesis manager 320 according to an embodiment is based on a deep learning method, and includes a deep learning-based TTS module 410, a speech characteristic module 420, and a mixing module 430. I can.

The deep learning-based TTS module 410 selects a personalized TTS engine corresponding to the personalization condition when a personalization condition such as, for example, the user's age, the user's gender, and the user's preferred voice frequency band, is input from the user. You can decide. At this time, the age of the user may vary from, for example, a teenager or a 20's to an elderly person of 60's or older.

For example, suppose that reference audios corresponding to a plurality of emotions such as happiness, joy, sadness, depression, anger, calm, and cheerfulness are input. The voice characteristic module 420 may train a network that classifies each of the reference voice signals according to a plurality of styles (or emotions). In this case, the voice characteristic module 420 may include a classifier for classifying a plurality of emotions.

The speech characteristic module 420 may determine an embedding vector for each of a plurality of emotions by extracting prosody features common to a specific emotion regardless of the content of the reference speech signals using the trained network. At this time, the determined embedding vector is reflected when a condition is applied to the text encoder (for example, see the text encoder 210 in FIG. 2) of the personalized TTS engine during speech synthesis and can be used to synthesize speech. have.

According to an embodiment, the voice characteristic module 420 may additionally use a tone conversion module or a language conversion module in the case of tone or language conversion. For example, the synthesized voice may be applied to the tone conversion module or the text sequence may be applied to the language conversion module.

The mixing module 430 may add noise according to the surrounding environment to the voice generated through the voice characteristic module 420 or adjust the frequency of the generated voice to a frequency band desired by the speaker.

Depending on the embodiment, the speech synthesis manager 320 generates a speech through a boilerplate using the USS method or a general corpus instead of the deep learning-based TTS module 410, and/or the generated speech Noise can also be added.

5 is a flowchart showing a learning method according to an embodiment. Referring to FIG. 5, the learning apparatus according to an embodiment receives reference audios corresponding to a plurality of emotions (510 ). In this case, the plurality of sensibilities may include, for example, happiness, joy, sadness, depression, anger, joy, youthfulness, lightness, irritation, and the like. The standard voice signals are, for example, "Hello" with a joyful sensibility, "Hello" with a sad sensibility, "Hello" with an angry sensibility, "Thank you" with a happy sensibility, and "Hello" with a gloomy sensibility Voice signals corresponding to various sensibilities, such as "thank you" and "thank you," uttered with a cheerful sensibility, may be included.

The learning apparatus trains a network for classifying each of the reference speech signals according to a plurality of emotions (520). Alternatively, the learning apparatus may train a network that classifies each of the reference speech signals into a plurality of tones and a plurality of languages. According to an embodiment, the learning apparatus may receive a noise signal corresponding to an environment in which the reference speech signals are collected. For example, when the reference voice signals are collected at a place where young students gather (such as a school or academy), the noise signal is generated by various types of noises (school bell, playground, desk, etc.) The sound of pulling the stool, the noise of the chatter during breaks, etc.) can be included.

The learning apparatus may train a network that mixes the reference speech signals and a noise signal corresponding to an environment in which the reference speech signals are collected, and classifies the mixed signal according to a plurality of emotions and environments.

The learning apparatus collects prosody features corresponding to sensibilities of the reference speech signals using the trained network (530). The learning device may collect prosody features corresponding to emotions such as joy, sadness, and anger inherent in the reference voice signals. The learning apparatus may extract prosody features common to specific sensibilities of the reference speech signals regardless of the speech content of the reference speech signals.

The learning device determines an embedding vector for each of a plurality of emotions based on prosody features (540).

6 is a diagram illustrating a process of training a speech synthesis (TTS) model according to an embodiment. Referring to FIG. 6, a process in which the learning apparatus according to an embodiment determines an embedding vector for each of a plurality of emotions from reference speech signals is illustrated.

The learning device may include a prosody encoder 610, a prosody embedding module 620, an attention model 630, and a classifier 640.

When an SSML (Speech Synthesis Markup Language) tag including a plurality of emotions and a reference speech signal(s) 605 are received, the prosody encoder 610 performs a reference speech signal (regardless of the speech content of the reference speech signal 605). In 605), prosody features common to specific emotions can be extracted.

The prosody embedding module 620 may include pre-learned embedding vectors for each condition. The embedding vector may correspond to a feature vector expressing a common sense of rhythm, pitch, pitch, etc., which appears when expressing various emotions such as happiness, joy, sadness, depression, anger, joy, youthfulness, cheerfulness, and irritation, for example. have.

The attention model 630 classifies the embedding vector included in the rhyme embedding module 620 for each sensibility such as happy, sad, angry, cheerful, cheerful, etc. through the classifier 640, and is then added to the reference voice signal(s) 605. The embedding vector 650 for each optimized style (by emotion) indicating the degree of emotional expression, intonation, intensity, tone and frequency distribution that expresses the corresponding emotion may be output.

7 is a block diagram of an apparatus for providing a voice according to an embodiment. Referring to FIG. 7, the apparatus 700 for providing a voice according to an embodiment includes a communication interface 710 and a processor 730. The voice providing apparatus 700 may further include a memory 750 and a speaker 770. The communication interface 710, the processor 730, the memory 750, and the speaker 770 may communicate with each other through a communication bus 705.

Communication interface 710 receives a text sequence.

The processor 730 extracts a first feature vector from the text sequence. The processor 730 generates a second feature vector by applying a pre-learned embedding vector for each condition to the first feature vector. The processor 730 determines context information of the text from the second feature vector. The processor 730 synthesizes a speech corresponding to the text sequence based on the context information.

The memory 750 may store a text sequence and/or a speech synthesized corresponding to the text sequence.

The speaker 770 outputs the voice synthesized by the processor 730.

In addition, the processor 730 may perform at least one method or an algorithm corresponding to at least one method described above with reference to FIGS. 1 to 2. The processor 730 may be a data processing device implemented in hardware having a circuit having a physical structure for executing desired operations. For example, desired operations may include code or instructions included in a program. For example, a data processing device implemented in hardware is a microprocessor, a central processing unit, a processor core, a multi-core processor, and a multiprocessor. , Application-Specific Integrated Circuit (ASIC), and Field Programmable Gate Array (FPGA).

The processor 730 may execute a program and control the audio providing device 700. The program code executed by the processor 730 may be stored in the memory 750.

In addition, the memory 750 may store various types of information generated during processing in the processor 730 described above. In addition, the memory 750 may store various types of data and programs. The memory 750 may include a volatile memory or a nonvolatile memory. The memory 750 may include a mass storage medium such as a hard disk to store various types of data.

8 is a block diagram of a learning device according to an embodiment. Referring to FIG. 8, the learning apparatus 800 according to an embodiment includes a communication interface 810, a processor 830, and a memory 850. The communication interface 810, the processor 830, and the memory 850 may communicate with each other through a communication bus 805.

The communication interface 810 receives reference voice signals corresponding to a plurality of emotions. The communication interface 810 may output an embedding vector determined by the processor 830 for each of a plurality of emotions.

The processor 830 trains a network that classifies each of the reference speech signals according to a plurality of emotions. The processor 830 collects prosody features corresponding to the sentiments of the reference speech signals using the trained network. The processor 830 determines an embedding vector for each of a plurality of emotions based on prosody features.

In addition, the processor 830 may perform at least one method or an algorithm corresponding to at least one method described above with reference to FIGS. 3 to 6. The processor 830 may be a data processing device implemented in hardware having a circuit having a physical structure for executing desired operations. For example, desired operations may include code or instructions included in a program. For example, a data processing device implemented in hardware is a microprocessor, a central processing unit, a processor core, a multi-core processor, and a multiprocessor. , Application-Specific Integrated Circuit (ASIC), and Field Programmable Gate Array (FPGA).

The processor 830 may execute a program and control the learning device 800. The program code executed by the processor 830 may be stored in the memory 850.

The memory 850 may store various types of information received through the communication interface 810. In addition, the memory 850 may store various types of information generated during processing in the processor 830 described above. In addition, the memory 850 may store various types of data and programs. The memory 850 may include a volatile memory or a nonvolatile memory. The memory 850 may include a mass storage medium such as a hard disk to store various types of data.

The method according to an embodiment may be implemented in the form of program instructions that can be executed through various computer means and recorded in a computer-readable medium. The computer-readable medium may include program instructions, data files, data structures, etc. alone or in combination. The program instructions recorded in the medium may be specially designed and configured for the present invention, or may be known and usable to those skilled in computer software. Examples of computer-readable recording media include magnetic media such as hard disks, floppy disks, and magnetic tapes, optical media such as CD-ROMs and DVDs, and magnetic media such as floptical disks. -A hardware device specially configured to store and execute program instructions such as magneto-optical media, and ROM, RAM, flash memory, and the like. Examples of program instructions include not only machine language codes such as those produced by a compiler but also high-level language codes that can be executed by a computer using an interpreter or the like. The above-described hardware device may be configured to operate as one or more software modules to perform the operation of the present invention, and vice versa.

As described above, although the present invention has been described by the limited embodiments and drawings, the present invention is not limited to the above embodiments, and various modifications and variations from these descriptions are those of ordinary skill in the field to which the present invention belongs. This is possible.

Therefore, the scope of the present invention is limited to the described embodiments and should not be defined, and should be defined by the claims and equivalents as well as the claims to be described later.

700: voice providing device
705: communication bus
710: communication interface
730: processor
750: memory
770: speaker

Claims (17)

Receiving a text sequence;
Extracting a first feature vector from the text sequence;
Generating a second feature vector by applying a pre-learned embedding vector for each condition to the first feature vector;
Determining context information of text from the second feature vector; And
Synthesizing a speech corresponding to the text sequence based on the context information; And
Providing the synthesized voice
Containing, speech synthesis method. The method of claim 1,
The above conditions are
A speech synthesis method comprising at least one of emotion, tone, and language conversion. The method of claim 2,
Generating the second feature vector comprises:
Generating the second feature vector by applying a pre-learned embedding vector for each of a plurality of emotions;
Generating the second feature vector by applying a pre-learned embedding vector for each of a plurality of tones; And
Generating the second feature vector by applying pre-learned embedding vectors for each of a plurality of languages for language conversion
The speech synthesis method comprising at least one of. The method of claim 3,
The plurality of emotions
A speech synthesis method that includes at least one of happiness, joy, sadness, melancholy, angry, cheerful, cheerful, cheerful, and irritable. The method of claim 3,
The plurality of tones
Speech synthesis method comprising at least one of a voice tone and a speed change. The method of claim 2,
Step of receiving a selection of personalization conditions from the user
Including more,
Extracting the first feature vector
Determining a personalized TTS (Text to Speech) engine corresponding to the personalization condition; And
Outputting the first feature vector by applying the text sequence to the personalized TTS engine
Containing, speech synthesis method. The method of claim 6,
The above personalization conditions are
The voice synthesis method comprising at least one of the user's age, the user's gender, and a voice frequency band preferred by the user. The method of claim 2,
The step of synthesizing the voice
Predicting a distribution of intonation, intensity, tone, and frequency corresponding to the text sequence based on the context information; And
Synthesizing the speech using the predicted intonation, intensity, tone and the predicted frequency distribution
Containing, speech synthesis method. The method of claim 2,
Mixing a speaker-related noise with the synthesized speech; And
Outputting a voice mixed with the noise
Further comprising a, speech synthesis method. Receiving reference audios corresponding to a plurality of emotions;
Training a network for classifying each of the reference speech signals according to the plurality of emotions;
Collecting prosody features corresponding to sensibilities of the reference speech signals using the trained network; And
Determining an embedding vector for each of the plurality of emotions based on the prosody features
Containing, learning method. The method of claim 10,
The plurality of emotions
A learning method that includes at least one of happiness, joy, sadness, melancholy, angry, cheerful, cheerful, cheerful, and irritable. The method of claim 10,
Collecting the prosody features
Extracting prosody features common to specific sensibilities of the reference speech signals irrespective of the speech content of the reference speech signals
Containing, learning method. The method of claim 10,
Training a network for classifying each of the reference speech signals according to a plurality of tones and a plurality of languages
Further comprising, learning method. The method of claim 10,
Receiving a noise signal corresponding to the environment in which the reference voice signals are collected
Including more,
Training a network for classifying each of the plurality of emotions
Mixing the reference speech signals and a noise signal corresponding to an environment in which the reference speech signals are collected; And
Training a network for classifying the mixed signal according to the plurality of emotions and the environment
Containing, learning method. A computer program stored in a computer-readable recording medium in combination with hardware to execute the method of any one of claims 1 to 14. A communication interface for receiving a text sequence;
Extracting a first feature vector from the text sequence, applying a pre-learned embedding vector for each condition to the first feature vector to generate a second feature vector, and determining context information of the text from the second feature vector, A processor for synthesizing a speech corresponding to the text sequence based on the context information; And
A speaker that outputs the synthesized voice
Containing, speech synthesis device. A communication interface for receiving reference voice signals corresponding to a plurality of emotions; And
Train a network for classifying each of the reference speech signals according to the plurality of emotions, collect prosody characteristics corresponding to the sensations of the reference speech signals using the trained network, and based on the prosody characteristics Processor for determining an embedding vector for each of the plurality of emotions
Including,
The communication interface is
A learning device that outputs the embedding vector.

Images