WO2019139428A1 - Procédé de synthèse vocale à partir de texte multilingue - Google Patents

Procédé de synthèse vocale à partir de texte multilingue Download PDF

Info

Publication number
WO2019139428A1
WO2019139428A1 PCT/KR2019/000509 KR2019000509W WO2019139428A1 WO 2019139428 A1 WO2019139428 A1 WO 2019139428A1 KR 2019000509 W KR2019000509 W KR 2019000509W WO 2019139428 A1 WO2019139428 A1 WO 2019139428A1
Authority
WO
WIPO (PCT)
Prior art keywords
language
speech
text
data
learning
Prior art date
Application number
PCT/KR2019/000509
Other languages
English (en)
Korean (ko)
Inventor
김태수
이영근
Original Assignee
네오사피엔스 주식회사
Priority date (The priority date is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the date listed.)
Filing date
Publication date
Application filed by 네오사피엔스 주식회사 filed Critical 네오사피엔스 주식회사
Priority to JP2020538690A priority Critical patent/JP7142333B2/ja
Priority to CN201980007944.2A priority patent/CN111566655B/zh
Priority to EP19738599.0A priority patent/EP3739476A4/fr
Priority claimed from KR1020190003979A external-priority patent/KR102199067B1/ko
Publication of WO2019139428A1 publication Critical patent/WO2019139428A1/fr
Priority to US16/682,390 priority patent/US11217224B2/en
Priority to US17/533,459 priority patent/US11769483B2/en
Priority to JP2022121111A priority patent/JP7500020B2/ja

Links

Images

Classifications

    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06F—ELECTRIC DIGITAL DATA PROCESSING
    • G06F40/00—Handling natural language data
    • G06F40/40—Processing or translation of natural language
    • G—PHYSICS
    • G10—MUSICAL INSTRUMENTS; ACOUSTICS
    • G10L—SPEECH ANALYSIS TECHNIQUES OR SPEECH SYNTHESIS; SPEECH RECOGNITION; SPEECH OR VOICE PROCESSING TECHNIQUES; SPEECH OR AUDIO CODING OR DECODING
    • G10L13/00—Speech synthesis; Text to speech systems
    • G10L13/02—Methods for producing synthetic speech; Speech synthesisers
    • G10L13/033—Voice editing, e.g. manipulating the voice of the synthesiser
    • G—PHYSICS
    • G10—MUSICAL INSTRUMENTS; ACOUSTICS
    • G10L—SPEECH ANALYSIS TECHNIQUES OR SPEECH SYNTHESIS; SPEECH RECOGNITION; SPEECH OR VOICE PROCESSING TECHNIQUES; SPEECH OR AUDIO CODING OR DECODING
    • G10L13/00—Speech synthesis; Text to speech systems
    • G10L13/08—Text analysis or generation of parameters for speech synthesis out of text, e.g. grapheme to phoneme translation, prosody generation or stress or intonation determination
    • G—PHYSICS
    • G10—MUSICAL INSTRUMENTS; ACOUSTICS
    • G10L—SPEECH ANALYSIS TECHNIQUES OR SPEECH SYNTHESIS; SPEECH RECOGNITION; SPEECH OR VOICE PROCESSING TECHNIQUES; SPEECH OR AUDIO CODING OR DECODING
    • G10L15/00—Speech recognition
    • G10L15/02—Feature extraction for speech recognition; Selection of recognition unit
    • G—PHYSICS
    • G10—MUSICAL INSTRUMENTS; ACOUSTICS
    • G10L—SPEECH ANALYSIS TECHNIQUES OR SPEECH SYNTHESIS; SPEECH RECOGNITION; SPEECH OR VOICE PROCESSING TECHNIQUES; SPEECH OR AUDIO CODING OR DECODING
    • G10L15/00—Speech recognition
    • G10L15/26—Speech to text systems

Definitions

  • This disclosure relates to multilingual text-to-speech synthesis methods and systems.
  • the present invention also relates to a method and apparatus for synthesizing a text of a second language into a voice of a speaker based on a voice characteristic of the speaker using the first language.
  • Speech synthesis technology is a technique that is used in applications that require human voice, such as announcement, navigation, It is a technique used for reproducing voice.
  • a typical method of speech synthesis is concatenative TTS in which speech is synthesized by pre-cutting and storing speech in a very short unit such as a phoneme, combining phonemes constituting a sentence to be synthesized, And a parameter synthesis method (parametric TTS) for synthesizing parameters representing speech features constituting a sentence to be synthesized into a speech corresponding to a sentence using a vocoder.
  • TTS text-to-speech
  • the method and apparatus according to the present disclosure is capable of generating a multilingual TTS machine learning model end-to-end with only text input and output audio for multiple languages .
  • the method and apparatus according to the present disclosure may synthesize speech from text, reflecting speech characteristics, emotional characteristics, and rhyme characteristics of the speaker.
  • a multilingual text-to-speech synthesis method is a method for synthesizing a learning text of a first language and a learning speech of a first language corresponding to a learning text of a first language
  • Receiving second learning data including learning speech data of a second language corresponding to learning text of a second language and learning text of a second language
  • a multi-lingual text-to-speech synthesis method includes receiving a speech characteristic of a speaker for a first language, receiving input text of a second language, inputting text of a second language, And generating output speech data for an input text of a second language that simulates the speech of the speaker by inputting the speech characteristics of the speaker for one language into a single artificial neural network text-speech synthesis model.
  • a speaker's utterance characteristic for a first language of a multilingual text-to-speech synthesis method is generated by extracting a feature vector from speech data uttered by a speaker in a first language.
  • a multi-lingual text-to-speech synthesis method in accordance with an embodiment of the present disclosure includes receiving an emotion feature and inputting text in a second language, a speech feature and an emotion feature of a speaker for a first language, Generating an output speech data for an input text of a second language that is input to an artificial neural network text-speech synthesis model to simulate a speech of a speaker.
  • a method for multi-lingual text-to-speech synthesis in accordance with an embodiment of the present disclosure includes receiving a prosody feature and inputting a second language's input text, a speaker's vocal and rhyme characteristics for a first language, Generating an output speech data for an input text of a second language that is input to an artificial neural network text-speech synthesis model to simulate a speech of a speaker.
  • the prosodic feature of the multilingual text-to-speech synthesis method includes at least one of information on the speech speed, information on the pronunciation strength, information on the pitch height, and information on the idle duration.
  • a multi-lingual text-to-speech synthesis method includes receiving an input speech of a first language, extracting a feature vector from the input speech of the first language, Converting input speech of a first language into input text of a first language; converting input text of a first language into input text of a second language; And generating output speech data of a second language for an input text of a second language that simulates the speech of the speaker by inputting a speaker's utterance characteristic for the speech into a single artificial neural network text-speech synthesis model.
  • the multilingual text-to-speech synthesis method uses a Grapheme-to-phoneme (G2P) algorithm to convert a learning text of a first language and a learning text of a second language into a phoneme sequence Conversion.
  • G2P Grapheme-to-phoneme
  • the single artificial neural network text-to-speech synthesis model of the multilingual text-to-speech synthesis method is characterized in that a text-to-speech synthesis model of a single artificial neural network includes a phoneme of a first language and an input of similarity information .
  • the program for implementing the multi-language text-to-speech synthesis method as described above can be recorded on a computer-readable recording medium.
  • FIG. 1 is a diagram showing that a speech synthesizer synthesizes English speech using a single artificial neural network text-speech synthesis model learned for a plurality of languages.
  • FIG. 2 is a diagram showing that a speech synthesizer synthesizes a Korean speech using a single artificial neural network text-speech synthesis model learned for a plurality of languages.
  • FIG. 3 is a flow diagram illustrating a method for generating a single artificial neural network text-speech synthesis model in accordance with an embodiment of the present disclosure.
  • FIG. 4 is a diagram showing a machine learning unit according to an embodiment of the present disclosure.
  • FIG. 5 is a diagram showing how speech synthesizer according to one embodiment of the present disclosure synthesizes output speech data based on speech characteristics of a speaker for a first language and input text of a second language.
  • FIG. 6 is a diagram showing that a speech synthesizer according to an embodiment of the present disclosure generates output speech data based on speech characteristics of a speaker for a first language, input text of a second language, and emotion characteristics.
  • FIG. 7 is a diagram illustrating that a speech synthesizer according to an embodiment of the present disclosure generates output speech data based on a speaker's speech feature for a first language, an input text of a second language, and a prosody feature .
  • FIG. 8 is a diagram showing a configuration of a speech translation system according to an embodiment of the present disclosure.
  • FIG. 9 is a diagram showing a configuration of a rhyme translator according to an embodiment of the present disclosure.
  • FIG. 10 is a diagram illustrating a configuration of a multi-language text-to-speech synthesizer according to an embodiment of the present disclosure.
  • Fig. 11 shows correspondence relationships between IPA (International Phonetic Alphabet) and KoG2P phonemes and phonemes having common pronunciation in English and Korean.
  • IPA International Phonetic Alphabet
  • 13 is a spectrogram showing the similarity between a voice generated in English phonemes and a voice generated in Korean phonemes.
  • CER 14 is a chart showing a character error rate (CER) according to time change of English data used for learning a TTS machine learning model.
  • 15 is a block diagram of a text-to-speech synthesis system in accordance with one embodiment of the present disclosure.
  • part used in the specification means software or hardware component, and "part " However, “part” is not meant to be limited to software or hardware. “Part” may be configured to reside on an addressable storage medium and may be configured to play back one or more processors.
  • part (s) refers to components such as software components, object oriented software components, class components and task components, and processes, Subroutines, segments of program code, drivers, firmware, microcode, circuitry, data, databases, data structures, tables, arrays and variables.
  • the functions provided in the components and “parts " may be combined into a smaller number of components and” parts " or further separated into additional components and “parts ".
  • processor may be embodied in a processor and memory.
  • the term “processor” should be broadly interpreted to include a general purpose processor, a central processing unit (CPU), a microprocessor, a digital signal processor (DSP), a controller, a microcontroller, In some circumstances, a “processor” may refer to an application specific integrated circuit (ASIC), a programmable logic device (PLD), a field programmable gate array (FPGA)
  • ASIC application specific integrated circuit
  • PLD programmable logic device
  • FPGA field programmable gate array
  • processor refers to a combination of processing devices, such as, for example, a combination of a DSP and a microprocessor, a combination of a plurality of microprocessors, a combination of one or more microprocessors in conjunction with a DSP core, It can also be called.
  • memory should be broadly interpreted to include any electronic component capable of storing electronic information.
  • the terminology memory may be any suitable memory such as random access memory (RAM), read-only memory (ROM), non-volatile random access memory (NVRAM), programmable read-only memory (PROM), erase- May refer to various types of processor-readable media such as erasable programmable read-only memory (PROM), flash memory, magnetic or optical data storage devices, registers, and the like.
  • RAM random access memory
  • ROM read-only memory
  • NVRAM non-volatile random access memory
  • PROM programmable read-only memory
  • erase- May to various types of processor-readable media such as erasable programmable read-only memory (PROM), flash memory, magnetic or optical data storage devices, registers, and the like.
  • the memory is said to be in electronic communication with the processor if the processor is able to read information from and / or write information to the memory.
  • the memory integrated in the processor is in electronic communication with the processor.
  • &quot first language " may refer to one of various languages used by various countries or people such as Korean, Japanese, Chinese, and English, and " It can refer to one of the languages used.
  • FIG. 1 is a diagram showing that speech synthesizer 110 synthesizes English speech using a single artificial neural network text-speech synthesis model learned for a plurality of languages.
  • a single artificial neural network text-speech synthesis model may be a combination of Korean and English data.
  • the speech synthesizer 110 can receive the English text and the utterance characteristic of the Korean speaker.
  • the English text may be "Hello? &Quot;
  • the utterance characteristic of the Korean speaker may be a feature vector extracted from the voice data uttered by the Korean speaker in Korean.
  • the speech synthesizer 110 inputs the received English text and the utterance characteristic of the Korean speaker into a single artificial neural network text-speech synthesis model to synthesize the speech of the Korean speaker and synthesizes the voice saying "Hello? &Quot; can do. That is, the voice output by the speech synthesizer 110 may be a voice in which the Korean speaker pronounces "Hello? &Quot; in English.
  • FIG. 2 is a diagram showing that the speech synthesizer 210 synthesizes a Korean speech using a single artificial neural network text-speech synthesis model learned for a plurality of languages.
  • a single artificial neural network text-speech synthesis model may be a combination of Korean and English data.
  • the speech synthesizer 210 may receive Korean text and utterance characteristics of an American speaker.
  • the Korean text may be "Hello? &Quot;
  • the utterance characteristic of the American speaker may be a feature vector extracted from the voice data uttered by the American speaker in English.
  • the speech synthesizer 210 inputs the received Korean text and utterance characteristics of the American speaker into a single artificial neural network text-speech synthesis model to synthesize the voice of the American speaker and synthesize the voice saying "Hello? &Quot; can do. That is, the voice output by the speech synthesizer 210 may be a voice that the American speaker pronounces "Hello? &Quot; in Korean.
  • a multilingual text-to-speech synthesis system includes first learning data including learning speech data of a first language and learning speech data of a first language corresponding to learning text of a first language, (Step 310).
  • the multi-lingual text-to-speech synthesis system may perform (320) receiving second learning data comprising learning text in a second language and learning speech data in a second language corresponding to learning text in a second language .
  • the multilingual text-to-speech synthesis system learns similarity information between the phonemes of the first language and the phonemes of the second language based on the first learning data and the second learning data, and generates a single artificial neural network text-to- -peech synthesis) model (step 330).
  • a method of generating a single artificial neural network text-speech synthesis model will be described in more detail with reference to FIG.
  • the machine learning unit 420 may correspond to the data learning unit 1510 in Fig.
  • the machine learning unit 420 can receive a pair of learning data 411 of a plurality of first languages.
  • the pair of learning data 411 of the first language may include learning text data of the first language and learning speech data of the first language corresponding to the learning text of the first language.
  • the learning text of the first language may include at least one letter and the machine learning unit 420 may convert the phoneme sequence into a phoneme sequence using a Grapheme-to-phoneme algorithm.
  • the learning speech data of the first language may be data on which human-read speech is recorded in the learning text of the first language, a sound feature or a spectrogram extracted from the recording data, and the like.
  • the first learning data may not include a language identifier or language information for the first language.
  • the machine learning unit 420 can receive a pair of learning data 412 of a plurality of second languages.
  • the pair of learning data 412 of the second language may include learning text data of the second language and learning speech data of the second language corresponding to the learning text of the second language.
  • the first language and the second language may be different languages.
  • the learning text of the second language may include at least one letter and the machine learning unit 420 may convert the phoneme sequence into a phoneme sequence using a Grapheme-to-phoneme algorithm.
  • the learning speech data of the second language may be data on which human-read speech is recorded in the learning text of the second language, a sound feature or a spectrogram extracted from the recording data, and the like.
  • the second learning data may not include the language identifier or the language information for the second language.
  • the machine learning unit 420 performs a machine learning based on the received pairs of learning data 411 of the first language and the pairs of learning data 412 of the plurality of second languages to generate a single artificial neural network text-
  • the speech synthesis model 430 can be generated.
  • the machine learning unit 420 learns similarity information between the phonemes of the first language and the phonemes of the second language, without prior knowledge of the first language and the second language, and generates a single artificial neural network text- Model 430 may be generated.
  • the machine learning unit 420 may include a language identifier for the first language, a language identifier for the second language, a phoneme in the first language, and similarity information for pronunciation between the phonemes in the second language, Based on a pair of learning data 411 of a plurality of first languages and a pair of learning data 412 of a plurality of second languages without receiving the similarity information on the representation between the phonemes of the second language and phonemes By learning similarity information between phonemes in one language and phonemes in a second language, a single artificial neural network text-speech synthesis model can be generated.
  • the language identifier may be an identifier indicating one of various languages used by various countries or people such as Korean, Japanese, Chinese, and English.
  • the similarity information on the pronunciation may be information in which phonemes having similar pronunciation are pronounced between the languages, and the similarity information on the notation may be information in which phonemes having similar notations between languages are associated. Similarity information is described in more detail with reference to FIGS. 11 and 12. FIG.
  • FIG. 4 shows the generation of a single artificial neural network text-speech synthesis model by receiving learning data for two languages.
  • the present invention is not limited to this, and it is also possible to receive learning data in three or more languages, A single artificial neural network text-speech synthesis model may be generated.
  • the text may be synthesized and output in voice using a single artificial neural network text-to-speech synthesis model 430 generated by the machine learning unit 420.
  • a method of synthesizing and outputting text by voice using a single artificial neural network text-speech synthesis model 430 will be described in more detail with reference to FIGS. 5 to 7.
  • FIG. 5 to 7 A method of synthesizing and outputting text by voice using a single artificial neural network text-speech synthesis model 430 will be described in more detail with reference to FIGS. 5 to 7.
  • FIG. 5 illustrates an exemplary embodiment of a speech synthesizer 520 according to an embodiment of the present disclosure that synthesizes output speech data 530 based on a speaker's speech feature 511 for a first language and input text 512 in a second language.
  • the speech synthesizer 520 may correspond to the data recognition unit 1520 of FIG.
  • the speech synthesizer 520 may be used to receive the single artificial neural network text-speech synthesis model generated by the machine learning unit 420 of FIG. 4 and to synthesize the output speech data. As shown, the speech synthesizer 520 may receive the speech feature 511 of the speaker for the first language and the input text 512 of the second language.
  • the speaker's utterance characteristic 511 for the first language can be generated by extracting a feature vector from speech data uttered by the speaker in the first language.
  • a speaker's utterance characteristic may include the tone or height of the speaker.
  • the input text 512 of the second language may include at least one letter in a second language.
  • the speech synthesizer 520 can generate the output speech data 530 by inputting the speaker's utterance characteristic 511 for the first language and the input text 512 of the second language into a single artificial neural network text- have.
  • the output speech data 530 may be speech data obtained by synthesizing the input text 512 of the second language by speech, and may reflect the speech characteristics 511 of the speaker for the first language.
  • the output speech data 530 is obtained by synthesizing the speech of the speaker on the basis of the speaker's utterance characteristic 511 with respect to the first language, so that the speaker is synthesized with the speech of the input text 512 of the second language Lt; / RTI >
  • the output speech data 530 may be output to a speaker or the like.
  • Figure 6 illustrates a speech synthesizer 620 according to one embodiment of the present disclosure that generates an output speech 611 based on a speaker's speech feature 611, a second language's input text 612, and an emotion feature 613 for a first language.
  • the speech synthesizer 620 may correspond to the data recognition unit 1520 of FIG.
  • the speech synthesizer 620 may receive the single artificial neural network text-speech synthesis model generated by the machine learning unit 420 of FIG. 4 and use it to synthesize the output speech data 630.
  • the speech synthesizer 620 may receive the speech features 611 of the speaker for the first language, the input text 612 of the second language, and the emotion feature 613.
  • the speaker's utterance characteristic for the first language and the input text of the second language have been described with reference to FIG. 5, and a duplicate description will be omitted.
  • emotion feature 613 may represent at least one of joy, sadness, anger, fear, trust, disgust, surprise, expectation.
  • emotion feature 613 may be generated by extracting feature vectors from speech data.
  • the speech synthesizer 620 inputs the speech characteristic 611 of the speaker for the first language, the input text 612 of the second language and the emotion characteristic 613 to the single artificial neural network text-speech synthesis model, 630 < / RTI >
  • the output speech data 630 may be speech data obtained by synthesizing the input text 612 of the second language by voice and may include the speech characteristic 611 and the emotion characteristic 613 of the speaker for the first language have. That is, the output speech data 630 simulates the voice of the speaker based on the speaker's utterance characteristic 611 for the first language, and reflects the emotion characteristic 613 to determine the emotion characteristic 613 ) To the input text 612 of the second language. For example, if the emotion feature 613 represents anger, the speech synthesizer 620 may generate output speech data 630 that speaks as if the speaker is raging the input text 612 of the second language. In one embodiment, the output speech data 630 may be output to a speaker or the like.
  • FIG. 7 illustrates a speech synthesizer 720 according to one embodiment of the present disclosure that is based on a speaker's speech feature 711 for a first language, an input text 712 for a second language, and a prosody feature 713 And generates output audio data 730.
  • FIG. The speech synthesizer 720 may correspond to the data recognition unit 1520 of FIG.
  • the speech synthesizer 720 can be used to receive the single artificial neural network text-speech synthesis model generated by the machine learning unit 420 of FIG. 4 and synthesize the output speech data 730.
  • the speech synthesizer 720 may receive the speech features 711 of the speaker, the input text 712 of the second language, and the rhyme feature 713 for the first language.
  • the speaker's utterance characteristic for the first language and the input text of the second language have been described with reference to FIG. 5, and a duplicate description will be omitted.
  • the prosodic feature 713 may include at least one of information on the speech rate, information on the pronunciation strength, information on the pitch height, and information on the dormant period (e.g., information on break-reading).
  • the rhyme feature 713 may be generated by extracting feature vectors from speech data.
  • the speech synthesizer 720 inputs the speech characteristics 711 of the first language, the input text 712 of the second language and the prosodic feature 713 for the first language into a single artificial neural network text-speech synthesis model, 730).
  • the output speech data 730 may be speech data obtained by synthesizing the input text 712 of the second language by voice, and the speech characteristic 711 and the prosodic characteristic 713 may be reflected. That is, the output speech data 730 simulates the speech of the speaker on the basis of the speech characteristic 711 of the speaker for the first language and reflects the prosodic characteristic 713 so that the prosodic characteristic 713 And the second language input text 712 with the second language.
  • the speech synthesizer 720 may determine that the speaker has entered the input text 712 of the second language in terms of the speech rate, the pronunciation strength, the pitch height, the pause interval
  • the output audio data 730 can be generated.
  • a speech synthesizer may be configured by inputting at least one of a speaker's utterance characteristic, emotion characteristic, and prosodic characteristic of the first language together with the input text of the second language.
  • the speech translation system 800 includes a speech recognizer 810, a machine translator 820, a speech synthesizer 830, a vocal feature extractor 840, an emotion feature extractor 850, a prosody feature extractor 860, translation 870).
  • the speech synthesizer 830 may correspond to the data recognition unit 1520 of FIG.
  • the voice translation system 800 may receive the input voice of the first language.
  • the input speech of the first language may be transmitted to the speech recognizer 810, the vocal feature extractor 840, the emotion feature extractor 850, and the prosodic feature extractor 860.
  • the speech recognizer 810 may receive the input speech of the first language and convert it into input text of the first language.
  • the machine translator 820 included in the speech translation system 800 may convert the input text of the first language into input text of the second language and translate the input text to the speech synthesizer 830.
  • the utterance feature extractor 840 may extract a feature vector from the input speech of the first language and generate a utterance characteristic of the speaker that uttered the input speech of the first language.
  • the speech synthesizer 830 inputs the input text of the second language and the speech characteristics of the speaker for the first language into a single artificial neural network text-speech synthesis model to generate a speech corresponding to the input text of the second language
  • the output speech data of the second language can be generated.
  • the output speech of the second language may be a voice synthesized by reflecting the utterance characteristic of the speaker who uttered the input speech of the first language.
  • the emotion feature extractor 850 may extract the emotion feature from the input speech of the first language and deliver it to the speech synthesizer 830.
  • the speech synthesizer 830 inputs the input text of the second language, the speech characteristics and emotion characteristics of the speaker for the first language into a single artificial neural network text-speech synthesis model to simulate the speech of the speaker,
  • the output speech data of the second language corresponding to the input text of the second language in which the emotion characteristic of the voice is reflected can be generated.
  • the output speech of the second language may be a voice synthesized by reflecting the utterance characteristic and the emotion characteristic of the speaker uttering the input speech of the first language.
  • Prosodic feature extractor 860 may extract the prosodic feature from the input speech of the first language.
  • the prosodic feature extractor 860 may transfer the extracted prosodic features to the prosodic translator 870 to translate the prosodic features for the first language into the prosodic features for the second language. That is, the rhyme translator 870 can generate information to reflect the rhyme characteristics extracted from the input speech of the first language to the output speech of the second language.
  • the speech synthesizer 830 inputs the input text of the second language, the speech characteristics of the speaker for the first language and the translated rhyme characteristics into a single artificial neural network text-speech synthesis model to simulate the speech of the speaker,
  • the output speech data of the second language corresponding to the input text of the second language in which the prosodic characteristic of the input speech of the second language is reflected.
  • the output speech of the second language may be a voice synthesized by reflecting the utterance characteristic and the prosodic characteristic of the speaker uttering the input speech of the first language.
  • features such as the speaking speed of the input speech of the first language, intermittent reading, and emphasis may be applied to the output speech of the second language.
  • the rhyme translator 870 may generate information for emphasizing the word of the second language corresponding to the highlighted word of the first language .
  • the speech synthesizer 830 can generate speech by emphasizing words of a second language corresponding to words emphasized in the first language, based on the information received from the rhyme translator 870.
  • the speech synthesizer 830 inputs the input text of the second language, the speech characteristics of the speaker for the first language, the emotion characteristics and the translated rhyme characteristics into a single artificial neural network text-speech synthesis model,
  • the output speech data of the second language corresponding to the input text of the second language in which the emotion characteristic and the rhythm characteristic of the input speech of the first language are reflected can be generated.
  • the output speech of the second language may be a voice synthesized by reflecting the utterance characteristic, the emotion characteristic, and the prosodic characteristic of the speaker uttering the input speech of the first language.
  • the voice of the speaker can be simulated and output voice of the second language can be reproduced in a similar voice by extracting the characteristics of the speaker from the input voice of the first language and synthesizing the translated voice. Can be generated. Further, when the emotion characteristic of the speaker is extracted from the input speech of the first language, the output speech of the second language can be generated more naturally by simulating the emotion for the utterance of the speaker. In addition, when the prosody characteristics of the speaker are extracted from the input speech of the first language, a more natural output speech of the second language can be generated by simulating the prosody of the speaker.
  • FIG. 8 shows that speech is synthesized by extracting all the vocal characteristics, emotional characteristics, and prosodic features from the input speech of the first language, but the present invention is not limited thereto.
  • at least one of a vocal feature, an emotion feature, and a rhyme feature may be extracted from the input speech of the other speaker.
  • the emotion feature and the rhyme feature may be extracted from the input speech of the first language, while the speech feature may be extracted from other input speech (e.g., the celebrity speech) to synthesize the speech.
  • the synthesized voice reflects the emotion and the rhyme of the speaker who uttered the input voice of the first language, but the voice of the speaker (e.g., the famous person) who uttered another input voice may be reflected.
  • the rhyme translator 870 may include a prosody encoder 910, attention 920, and a prosody decoder 930.
  • the prosodic encoder 910 may receive the prosodic feature of the first language extracted from the input speech of the first language (source language) by the prosodic feature extractor.
  • the received first rhyme feature is converted to a rhyme feature of the second language (the language to be translated) via the rhyme encoder 910, the attention 920, and the rhyme decoder 930.
  • the rhyme translator 870 can transform the rhyme characteristics of the original language into the rhyme features of the language to be translated by learning using a sequence-to-sequence model (seq2seq) . That is, the sequence-to-sequence learning model is applied to an encoder-decoder architecture based on a recurrent neural network (RNN) (see “Sequence to Sequence Learning with Neural Networks," Ilya Sutskever, et al.
  • RNN recurrent neural network
  • FIG. 10 is a diagram illustrating a configuration of a multi-lingual text-to-speech synthesizer 1000 according to an embodiment of the present disclosure.
  • the multilingual text-to-speech synthesizer 1000 may include an encoder 1010, a decoder 1020, and a vocoder 1030.
  • the encoder 1010 may receive the input text.
  • the input text may be in a plurality of languages, and may not include information on language identifiers or languages.
  • the input text may be "Hello” or "How are you?" And the like.
  • the encoder 1010 can separate the received input text into alphabet, letter, and phoneme units. Or the encoder 1010 may receive input text separated in alphabet, character, and phoneme units.
  • the encoder 1010 may include at least one embedded layer (e.g., EL language 1, EL language 2, ..., EL language N). At least one embedded layer of the encoder 1010 may convert each of the input texts separated by alphabet, letter, and phoneme into a text embedding vector. The encoder 1010 may use a previously learned machine learning model to transform the discrete input text into a text embedding vector. The encoder can update the machine learning model while performing machine learning. When the machine learning model is updated, the text embedding vector for the discrete input text can also be changed.
  • EL language 1 e.g., EL language 1, EL language 2, ..., EL language N
  • At least one embedded layer of the encoder 1010 may convert each of the input texts separated by alphabet, letter, and phoneme into a text embedding vector.
  • the encoder 1010 may use a previously learned machine learning model to transform the discrete input text into a text embedding vector.
  • the encoder can update the machine learning model while performing machine learning. When the
  • the encoder 1010 may input the text embedding vector to a Deep Neural Network (DNN) module configured as a fully-connected layer.
  • DNN Deep Neural Network
  • the DNN may be a general feedforward layer or a linear layer.
  • the encoder 1010 may input the output of the DNN to a module including at least one of a convolutional neural network (CNN) and a recurrent neural network (RNN).
  • a module including at least one of CNN and RNN can receive the output (s) of the embedding layer of the decoder 1020 along with the DNN output.
  • CNN can capture local characteristics according to the size of the convolution kernel, and the RNN can capture long term dependency.
  • a module containing at least one of CNN and RNN may output the hidden states (h) of the encoder 1010 as an output.
  • the embedding layer of the decoder 1020 may perform an operation similar to the embedding layer of the encoder 1010.
  • the embedding layer may receive the speaker ID.
  • the speaker ID may be a one-hot vector.
  • the speaker ID of "Trump” may be designated as "1,” the speaker ID of "MoonJin” may be designated as "2,” and the speaker ID of "Obama” may be designated as "3".
  • the embedding layer of decoder 1020 may convert the speaker ID into a speaker embedding vector s. Decoder 1020 may use the already learned machine learning model to transform the speaker ID into the speaker embedding vector s. Decoder 1020 may update the machine learning model while performing machine learning. If the machine learning model is updated, the speaker embedding vector s for the speaker ID can also be changed.
  • the Attention of the decoder 1020 can receive the hidden states h of the encoder from the encoder 1010. Also, the attentions of the decoder 1020 may receive information from the Attention RNN. The information received from the Attention RNN may be information on which speech the decoder 1020 has generated until the previous time-step. The decoder 1020 can also output the context vector C t based on the information received from the Attention RNN and the hidden states h of the encoder. The hidden states (h) of the encoder may be information about the input text to which the speech should be generated.
  • the context vector Ct may be information for determining from which part of the input text the speech will be generated at the current time-step.
  • the attentions of the decoder 1020 may generate information based on the beginning of the text input at the beginning of speech generation, and generate information based on the later part of the text input as the speech is generated Can be output.
  • the decoder 1020 inputs the speaker embedding vector s to a module including at least one of the Attention RNN, the Decoder RNN, and the CNN and RNN of the Encoder 1010, Can be constructed.
  • the RNN of the decoder 1020 may be configured in an autoregressive manner. That is, the output of the r frames output at the previous time-step can be used as an input to this time step. Since there is no previous time step in the initial time step 1022, dummy frames may be input to the DNN.
  • the decoder 1022 may include a DNN configured as a fully-connected layer.
  • the DNN may be a general feedforward layer or a linear layer.
  • the decoder 1022 may include an Attention RNN configured as a GRU.
  • Attention RNN is a layer that outputs information to be used in Attention. Attention is described above, so a detailed description is omitted.
  • Decoder 1020 may include a decoder RNN configured with a residual GRU.
  • the decoder RNN may receive location information of the input text from the Attention. That is, the location information may be information on which position of the input text the decoder 1020 is converting to speech.
  • the decoder RNN may receive information from the Attention RNN.
  • the information received from the Attention RNN may be information on which voice the decoder has generated up to the previous time-step and information about the voice to be generated in this time-step.
  • the decoder RNN can generate the next output speech that will follow the speech generated so far.
  • the output speech may have a mel-spectrogram shape and may consist of r frames.
  • the operation of the DNN, the Attention RNN and the Decoder RNN may be repeatedly performed for text-to-speech synthesis.
  • the r frames obtained in the initial time step 1022 may be the inputs of the next time step 1024.
  • the r frames output in the time-step 1024 may be the inputs of the next time-step 1026.
  • the text-to-speech synthesis system can concatenate mel-spectrograms at each time step in chronological order to obtain a mel-spectrogram for the entire text.
  • the mel-spectrogram for the entire text generated at the decoder 1020 may be output to the first vocoder 1030 or the second vocoder 1040.
  • the first vocoder 1030 may include a module including at least one of CNN and RNN and a Griffin-Lim reconstruction module.
  • a module including at least one of CNN and RNN of the first vocoder 1030 may perform an operation similar to a module including at least one of CNN and RNN of the encoder 1010. [ That is, the module including at least one of CNN and RNN of the first vocoder 1030 can capture the regional characteristics and long-term dependency, and can output a linear-scale spectrogram.
  • the first vocoder 1030 may apply a Griffin-Lim algorithm to the linear-scale spectrogram to output a speech signal corresponding to the input text, with a voice corresponding to the speaker ID.
  • the second vocoder 1040 may obtain a speech signal from the mel spectrogram based on a machine learning model.
  • the machine learning model may have learned a network that predicts speech signals from mel-spectrograms.
  • a machine learning model can be a model such as WaveNet or WaveGlow.
  • the second vocoder 1040 may be used in place of the first vocoder 1030.
  • the artificial neural network-based multi-language text-to-speech synthesizer 1000 is learned by using a large-capacity database existing as a pair of learning texts of a multi-language language and corresponding learning speech signals.
  • the multi-lingual text-to-speech synthesizer 1000 can receive the training text and compare the output speech signal with the training speech signal to define a loss function.
  • the speech synthesizer learns the loss function through the error back propagation algorithm and finally obtains the artificial neural network with the desired speech output when arbitrary text is input.
  • the multi-lingual text-to-speech synthesizer 1000 can synthesize a voice simulating a voice of a specific speaker using a single artificial neural network text-speech synthesis model generated by the above method.
  • the multi-language text-to-speech synthesizer 1000 can synthesize voices of a speaker in a language different from the native language of a specific speaker by synthesizing voices of the speaker. That is, the multilingual text-to-speech synthesizer 1000 can synthesize a speech in which a speaker who speaks a first language speaks a second language. For example, a voice can be synthesized as if a trump is spoken in Korean in the input Korean text.
  • Fig. 11 shows correspondence relationships between IPA (International Phonetic Alphabet) and KoG2P phonemes and phonemes having common pronunciation in English and Korean. Pronunciation of different languages can be described by the International Phonetic Alphabet (IPA), an alphabetic system. IPA for pronunciation of different languages can be used as similarity information.
  • the conversion tables of IPA-CMUdict and IPA-KoG2P are shown in Table 1110.
  • IPA International Phonetic Alphabet
  • Table 1110 shows a one-to-one correspondence between the first language phoneme and the second language phoneme, but a subset including phonemes having a common pronunciation of the first language and the second language can be selected. For example, a subset of phonemes with a common pronunciation of English and Korean is shown in Table 1120.
  • the first language and the second language may have different character systems, and may have different pronunciation systems.
  • IPA which is the same alphabetic system
  • the speech synthesis model can be obtained through standardized processing for each language. IPA, however, does not completely represent the similarity of pronunciation or notation of different languages, although each language is represented by the same alphabetic system.
  • the IPA alphabet used in the first language may not be used at all in the second language. Since the speech synthesis model can not know which IPA alphabet in the second language will correspond to the IPA alphabet used in the first language, only the speech synthesis model specific to each language can be obtained when IPA is used.
  • the speech synthesis model for the first language can only process data associated with the first language and can not process data associated with the second language.
  • the speech synthesis model for the second language can only process data associated with the second language and can not process data associated with the first language.
  • the text-to-speech synthesis system can calculate the cosine distance between phonemes for anchor phonemes of languages based on a machine learning model.
  • the phoneme embedding vectors obtained based on the machine learning model can be used to calculate the cosine distance.
  • the cosine distance between phonemes can indicate the similarity between phonemes.
  • the phonemic embedding of the five closest English words for Korean phonemes based on the calculated cosine distance between phonemes is shown in Table 1210.
  • the numbers 0, 1 and 2 after the English phoneme embedding represent "no stress", "primary stress” and "secondary stress", respectively. While CMUdict distinguishes emphasized pronunciations, IPA may not distinguish emphasized pronunciations.
  • the symbols in parentheses are the IPA symbols.
  • the five closest phonemic embedding for an anchor phoneme based on the machine learning model according to one embodiment of the present disclosure is similar to table 1120 of FIG. That is, the machine learning model according to an embodiment of the present disclosure may include similarity information on pronunciation between phonemes of a first language and phonemes of a first language, similarity information on notation, language identifier / language information for a first language, Even if the language identifier / language information for the second language is not input at the time of learning, it can be confirmed that similar pronunciation or notation of the language is automatically learned.
  • the text-to-speech synthesis system can perform text-to-speech synthesis (TTS) on a plurality of languages learned based on a single artificial neural network text-speech synthesis model.
  • TTS text-to-speech synthesis
  • the spectrogram 13 is a spectrogram showing the similarity between a voice generated in English phonemes and a voice generated in Korean phonemes.
  • the spectrogram 1310 contains the sentences "He has many good friends" in English phoneme sequences HH, IY1, HH, AE1, Z, M, EH1, N, IY0, G, UH1, D, R, EH1, N, D, Z).
  • the spectrogram 1320 generates the Korean phoneme sequences h0, wi, h0, ya, s0, mf, ye, nf, ii, and hn, which are generated by replacing each phoneme in the English phoneme sequence of the same sentence with the closest Korean phoneme. kk, yo, tt, ph, ks, ye, nf, tt, s0).
  • the comparison between the spectrogram 1310 and the spectrogram 1320 shows that the result of synthesizing the voice with the English phoneme sequence is similar to the result of synthesizing the voice with the Korean phoneme sequence.
  • high-quality speech synthesis results can be obtained even if speech of the second language is synthesized by using phonemes of the first language. That is, even if the text of the second language is synthesized by voice using the utterance characteristic of the speaker uttered in the first language, the result that the corresponding speaker of the first language is the utterance in the second language can be obtained.
  • the table 1410 shows the character error rate (CER) according to the time change of the English data used for learning the TTS machine learning model.
  • CER character error rate
  • the table 1410 shows the error rate in which the person who listened to the voice output synthesized from the text to characterize the voice synthesis quality records the characters and compares the result with the original text.
  • the CER does not greatly differ even if the time of the English learning data used increases .
  • the amount of data in Korean used in machine learning is larger than the amount of data in English, so that the CER can be already reduced to a critical level. It can be confirmed that the CER can be sufficiently reduced when the text-to-speech synthesis system performs the machine learning using data exceeding a critical amount.
  • the TTS machine learning model is learned using a large amount of Korean learning data and a small amount of English learning data, it can be confirmed that the result of synthesizing the English text with speech is produced with relatively high quality.
  • the present disclosure it is possible to create a multilingual TTS machine learning model end-to-end with only text input and output audio for multiple languages.
  • different languages need a notation that can be commonly used in various languages such as IPA in order to express a linguistic feature set, or a dictionary information about the similarity between languages is needed did.
  • linguistic features are not required, so that each language may use a different alphabet and does not require prior knowledge of similarity between languages.
  • this disclosure teaches the model end-to-end so that it is not necessary to predict the features needed in the existing TTS, such as phoneme duration, using a separate model, (TTS) task with a neural network model.
  • TTS separate model,
  • the present disclosure in the process of extracting the text encoding from the text encoder, it is possible to control the Korean / English speech according to whether the speaker ID is used. For example, if the pronunciation of the second language is strong when generating the voice of the first language, a penalty may be given to the learning. According to the machine learning model to which the penalty is applied, speech can be generated more closely to the pronunciation of the first language.
  • the text-to-speech synthesis system 1500 may include a data learning unit 1510 and a data recognition unit 1520.
  • the data learning unit 1510 can input data and acquire a machine learning model.
  • the data recognition unit 1520 can also apply the data to the machine learning model to generate output speech.
  • the text-to-speech synthesis system 1500 as described above may include a processor and a memory.
  • the data learning unit 1510 can learn the voice of the text.
  • the data learning unit 1510 can learn a criterion as to which voice to output according to the text.
  • the data learning unit 1510 can learn a criterion as to which voice feature should be used to output the voice.
  • the feature of the speech may include at least one of pronunciation of the phoneme, tone of the user, accentuation, or accentuation.
  • the data learning unit 1510 acquires data to be used for learning, and applies the obtained data to a data learning model, which will be described later, so as to learn speech based on the text.
  • the data recognition unit 1520 can output a voice for the text based on the text.
  • the data recognition unit 1520 can output speech from a predetermined text using the learned data learning model.
  • the data recognition unit 1520 can acquire predetermined text (data) according to a preset reference by learning.
  • the data recognition unit 1520 can output a voice based on predetermined data by using the acquired data as an input value and using the data learning model. Further, the resultant value output by the data learning model with the obtained data as an input value can be used to update the data learning model.
  • At least one of the data learning unit 1510 or the data recognition unit 1520 may be manufactured in at least one hardware chip form and mounted on the electronic device.
  • at least one of the data learning unit 1510 and the data recognition unit 1520 may be manufactured in the form of a dedicated hardware chip for artificial intelligence (AI), or may be a conventional general-purpose processor Or an application processor) or a graphics processor (e.g., a GPU), and may be mounted on various electronic devices already described.
  • AI artificial intelligence
  • a graphics processor e.g., a GPU
  • the data learning unit 1510 and the data recognition unit 1520 may be mounted on separate electronic devices, respectively.
  • one of the data learning unit 1510 and the data recognizing unit 1520 may be included in the electronic device, and the other one may be included in the server.
  • the data learning unit 1510 and the data recognizing unit 1520 may provide the model information constructed by the data learning unit 1510 to the data recognizing unit 1520 through the wired or wireless communication, 1520 may be provided to the data learning unit 1510 as additional learning data.
  • At least one of the data learning unit 1510 and the data recognition unit 1520 may be implemented as a software module.
  • the software module may be a memory or a computer readable non- And may be stored in non-transitory computer readable media.
  • the at least one software module may be provided by an operating system (OS) or by a predetermined application.
  • OS operating system
  • OS operating system
  • OS operating system
  • the data learning unit 1510 includes a data acquisition unit 1511, a preprocessing unit 1512, a learning data selection unit 1513, a model learning unit 1514, and a model evaluation unit 1515 .
  • the data acquisition unit 1511 can acquire data necessary for machine learning. Since a lot of data is required for learning, the data acquisition unit 1511 can receive a plurality of texts and corresponding sounds.
  • the preprocessing unit 1512 can pre-process the acquired data so that the acquired data can be used for machine learning to determine the psychological state of the user.
  • the preprocessing unit 1512 can process the acquired data into a predetermined format so that it can be used by the model learning unit 1514 to be described later.
  • the preprocessor 1512 may morpheme text and speech to obtain morpheme embedding.
  • the learning data selection unit 1513 can select data necessary for learning from the preprocessed data.
  • the selected data may be provided to the model learning unit 1514.
  • the learning data selection unit 1513 can select data required for learning from among the preprocessed data according to a preset reference.
  • the learning data selection unit 1513 can also select data according to a predetermined reference by learning by the model learning unit 1514, which will be described later.
  • the model learning unit 1514 can learn a criterion as to which speech to output according to the text based on the learning data.
  • the model learning unit 1514 can use a learning model for outputting speech according to text as learning data.
  • the data learning model may include a pre-built model.
  • the data learning model may include a pre-built model that receives basic learning data (e.g., a sample image, etc.).
  • the data learning model can be constructed considering the application field of the learning model, the purpose of learning, or the computer performance of the device.
  • the data learning model may include, for example, a model based on a neural network.
  • models such as Deep Neural Network (DNN), Recurrent Neural Network (RNN), Long Short-Term Memory models (LSTM), Bidirectional Recurrent Deep Neural Network (BRDNN), and Convolutional Neural Networks But is not limited thereto.
  • the model learning unit 1514 can determine a data learning model to learn a data learning model having a great relation between the input learning data and the basic learning data, if there are a plurality of data learning models that are built in advance have.
  • the basic learning data may be pre-classified according to the type of data, and the data learning model may be pre-built for each data type.
  • the basic learning data may be pre-classified by various criteria such as an area where the learning data is generated, a time at which the learning data is generated, a size of the learning data, a genre of the learning data, a creator of the learning data, .
  • model learning unit 1514 can learn a data learning model using, for example, a learning algorithm including an error back-propagation method or a gradient descent method.
  • the model learning unit 1514 can learn a data learning model through supervised learning using, for example, learning data as an input value.
  • the model learning unit 1514 learns, for example, the type of data necessary for the situation determination without any further guidance, thereby to perform data learning (e.g., learning) through unsupervised learning
  • the model can be learned.
  • the model learning unit 1514 can learn the data learning model through reinforcement learning using, for example, feedback as to whether the result of the situation judgment based on learning is correct.
  • the model learning unit 1514 can store the learned data learning model.
  • the model learning unit 1514 can store the learned data learning model in the memory of the electronic device including the data recognition unit 1520.
  • the model learning unit 1514 may store the learned data learning model in the memory of the server connected to the electronic device and the wired or wireless network.
  • the memory in which the learned data learning model is stored may also store instructions or data associated with, for example, at least one other component of the electronic device.
  • the memory may also store software and / or programs.
  • the program may include, for example, a kernel, a middleware, an application programming interface (API), and / or an application program (or "application").
  • the model evaluation unit 1515 inputs the evaluation data to the data learning model, and if the result output from the evaluation data does not satisfy the predetermined criterion, the model evaluation unit 1515 can cause the model learning unit 1514 to learn again.
  • the evaluation data may include predetermined data for evaluating the data learning model.
  • the model evaluation unit 1515 when the number or ratio of evaluation data whose recognition result is not correct is greater than a predetermined threshold value among the results of the learned data learning model for evaluation data, the model evaluation unit 1515 .
  • a predetermined criterion is defined as a ratio of 2%, and the learned data learning model outputs an incorrect recognition result for evaluation data exceeding 20 out of a total of 1,000 evaluation data, Can be assessed as inappropriate.
  • the model evaluating unit 1515 evaluates whether each of the learned moving learning models satisfies a predetermined criterion, and uses a model satisfying a predetermined criterion as a final data learning model You can decide. In this case, when there are a plurality of models satisfying the predetermined criterion, the model evaluation unit 1515 can determine any one or a predetermined number of models previously set in descending order of the evaluation score, using the final data learning model.
  • At least one of the data acquiring unit 1511, the preprocessing unit 1512, the learning data selecting unit 1513, the model learning unit 1514, or the model evaluating unit 1515 in the data learning unit 1510 includes at least one And can be mounted on an electronic device.
  • at least one of the data acquisition unit 1511, the preprocessor 1512, the learning data selection unit 1513, the model learning unit 1514, or the model evaluation unit 1515 may be an artificial intelligence (AI) Or may be implemented as part of a conventional general-purpose processor (e.g., a CPU or an application processor) or a graphics-only processor (e.g., a GPU) and mounted on the various electronic devices described above.
  • AI artificial intelligence
  • a conventional general-purpose processor e.g., a CPU or an application processor
  • a graphics-only processor e.g., a GPU
  • the data acquisition unit 1511, the preprocessing unit 1512, the learning data selection unit 1513, the model learning unit 1514, and the model evaluation unit 1515 may be mounted on one electronic device, Electronic devices, respectively.
  • some of the data acquisition unit 1511, the preprocessing unit 1512, the learning data selection unit 1513, the model learning unit 1514, and the model evaluation unit 1515 are included in the electronic device, May be included in the server.
  • At least one of the data acquisition unit 1511, the preprocessing unit 1512, the learning data selection unit 1513, the model learning unit 1514, and the model evaluation unit 1515 may be implemented as a software module.
  • At least one of the data acquisition unit 1511, the preprocessing unit 1512, the learning data selection unit 1513, the model learning unit 1514 or the model evaluation unit 1515 is a software module (or a program including an instruction) Module), the software module may be stored in a computer-readable, readable non-transitory computer readable media.
  • the at least one software module may be provided by an operating system (OS) or by a predetermined application.
  • OS operating system
  • some of the at least one software module may be provided by an operating system (OS)
  • some of the software modules may be provided by a predetermined application.
  • the data recognizing unit 1520 includes a data obtaining unit 1521, a preprocessing unit 1522, a recognition data selecting unit 1523, a recognition result providing unit 1524, and a model updating unit 1525, . ≪ / RTI >
  • the data acquisition unit 1521 can acquire the text necessary for outputting the voice. Conversely, the data acquisition unit 1521 can acquire the voice necessary for outputting the text.
  • the preprocessing unit 1522 may preprocess the acquired data so that the acquired data may be used to output voice or text.
  • the preprocessing unit 1522 can process the obtained data into a predetermined format so that the recognition result providing unit 1524, which will be described later, can use the data obtained for outputting voice or text.
  • the recognition data selection unit 1523 can select data necessary for outputting voice or text among the preprocessed data.
  • the selected data may be provided to the recognition result provider 1524.
  • the recognition data selection unit 1523 can select some or all of the preprocessed data according to a predetermined criterion for outputting voice or text.
  • the recognition data selecting unit 1523 can also select data according to a predetermined reference by learning by the model learning unit 1514. [
  • the recognition result provider 1524 may apply the selected data to the data learning model to output voice or text.
  • the recognition result providing unit 1524 can apply the selected data to the data learning model by using the data selected by the recognition data selecting unit 1523 as an input value.
  • the recognition result can be determined by the data learning model.
  • the model updating unit 1525 can cause the data learning model to be updated based on the evaluation of the recognition result provided by the recognition result providing unit 1524.
  • the model updating unit 1525 may provide the model learning unit 1514 with the recognition result provided by the recognition result providing unit 1524 so that the model learning unit 1514 can update the data learning model have.
  • At least one of the data acquiring unit 1521, the preprocessing unit 1522, the recognition data selecting unit 1523, the recognition result providing unit 1524, or the model updating unit 1525 in the data recognizing unit 1520 may be, It can be manufactured in the form of one hardware chip and mounted on the electronic device. At least one of the data acquisition unit 1521, the preprocessing unit 1522, the recognition data selection unit 1523, the recognition result providing unit 1524 or the model updating unit 1525 may be an artificial intelligence Or may be mounted on a variety of electronic devices as described above and manufactured as part of a conventional general purpose processor (e.g., a CPU or an application processor) or a graphics dedicated processor (e.g., a GPU).
  • a conventional general purpose processor e.g., a CPU or an application processor
  • a graphics dedicated processor e.g., a GPU
  • some of the data acquisition unit 1521, the preprocessing unit 1522, the recognition data selection unit 1523, the recognition result providing unit 1524, and the model updating unit 1525 are included in the electronic device, May be included in the server.
  • At least one of the data acquisition unit 1521, the preprocessing unit 1522, the recognition data selection unit 1523, the recognition result providing unit 1524, and the model updating unit 1525 may be implemented as a software module.
  • At least one of the data acquisition unit 1521, the preprocessing unit 1522, the recognition data selection unit 1523, the recognition result providing unit 1524 or the model updating unit 1525 is a software module Program modules), the software modules may be stored in a computer-readable, readable non-transitory computer readable media.
  • the at least one software module may be provided by an operating system (OS) or by a predetermined application.
  • OS operating system
  • OS operating system
  • some of the at least one software module may be provided by an operating system (OS)
  • some of the software modules may be provided by a predetermined application.
  • the above-described embodiments of the present invention can be embodied in a general-purpose digital computer that can be embodied as a program that can be executed by a computer and operates the program using a computer-readable recording medium.
  • the computer-readable recording medium includes a storage medium such as a magnetic storage medium (e.g., ROM, floppy disk, hard disk, etc.), optical reading medium (e.g., CD ROM,

Landscapes

  • Engineering & Computer Science (AREA)
  • Physics & Mathematics (AREA)
  • Audiology, Speech & Language Pathology (AREA)
  • Computational Linguistics (AREA)
  • Health & Medical Sciences (AREA)
  • Multimedia (AREA)
  • Acoustics & Sound (AREA)
  • Human Computer Interaction (AREA)
  • Theoretical Computer Science (AREA)
  • General Health & Medical Sciences (AREA)
  • General Physics & Mathematics (AREA)
  • General Engineering & Computer Science (AREA)
  • Artificial Intelligence (AREA)
  • Computer Vision & Pattern Recognition (AREA)
  • Machine Translation (AREA)

Abstract

La présente invention concerne un procédé et un système de synthèse vocale à partir de texte multilingue. Un procédé de synthèse vocale à partir de texte multilingue comprend les étapes suivantes : recevoir des premières données d'entraînement contenant un texte d'entraînement d'une première langue et des données de parole d'entraînement de la première langue correspondant au texte d'entraînement de la première langue ; recevoir des deuxièmes données d'entraînement contenant un texte d'apprentissage d'une deuxième langue et des données de parole d'entraînement de la deuxième langue correspondant au texte d'entraînement de la deuxième langue ; et produire un modèle de synthèse vocale à partir de texte à base de réseau neuronal artificiel unique par apprentissage d'informations de similarité entre un phonème de la première langue et un phonème de la deuxième langue en fonction des premières données d'apprentissage et des deuxièmes données d'apprentissage.
PCT/KR2019/000509 2018-01-11 2019-01-11 Procédé de synthèse vocale à partir de texte multilingue WO2019139428A1 (fr)

Priority Applications (6)

Application Number Priority Date Filing Date Title
JP2020538690A JP7142333B2 (ja) 2018-01-11 2019-01-11 多言語テキスト音声合成方法
CN201980007944.2A CN111566655B (zh) 2018-01-11 2019-01-11 多种语言文本语音合成方法
EP19738599.0A EP3739476A4 (fr) 2018-01-11 2019-01-11 Procédé de synthèse vocale à partir de texte multilingue
US16/682,390 US11217224B2 (en) 2018-01-11 2019-11-13 Multilingual text-to-speech synthesis
US17/533,459 US11769483B2 (en) 2018-01-11 2021-11-23 Multilingual text-to-speech synthesis
JP2022121111A JP7500020B2 (ja) 2018-01-11 2022-07-29 多言語テキスト音声合成方法

Applications Claiming Priority (6)

Application Number Priority Date Filing Date Title
KR10-2018-0004047 2018-01-11
KR20180004047 2018-01-11
KR20180036377 2018-03-29
KR10-2018-0036377 2018-03-29
KR1020190003979A KR102199067B1 (ko) 2018-01-11 2019-01-11 다중 언어 텍스트-음성 합성 방법
KR10-2019-0003979 2019-01-11

Related Child Applications (1)

Application Number Title Priority Date Filing Date
US16/682,390 Continuation US11217224B2 (en) 2018-01-11 2019-11-13 Multilingual text-to-speech synthesis

Publications (1)

Publication Number Publication Date
WO2019139428A1 true WO2019139428A1 (fr) 2019-07-18

Family

ID=67219814

Family Applications (1)

Application Number Title Priority Date Filing Date
PCT/KR2019/000509 WO2019139428A1 (fr) 2018-01-11 2019-01-11 Procédé de synthèse vocale à partir de texte multilingue

Country Status (1)

Country Link
WO (1) WO2019139428A1 (fr)

Cited By (9)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN111247581A (zh) * 2019-12-23 2020-06-05 深圳市优必选科技股份有限公司 一种多语言文本合成语音方法、装置、设备及存储介质
CN111312228A (zh) * 2019-12-09 2020-06-19 中国南方电网有限责任公司 一种基于端到端的应用于电力企业客服的语音导航方法
CN111858961A (zh) * 2020-07-27 2020-10-30 西交利物浦大学 用于知识图谱中节点和链接的多语言知识匹配方法及装置
CN112365882A (zh) * 2020-11-30 2021-02-12 北京百度网讯科技有限公司 语音合成方法及模型训练方法、装置、设备及存储介质
CN112652291A (zh) * 2020-12-15 2021-04-13 携程旅游网络技术（上海）有限公司 基于神经网络的语音合成方法、系统、设备及存储介质
GB2591245A (en) * 2020-01-21 2021-07-28 Samsung Electronics Co Ltd An expressive text-to-speech system
CN113409761A (zh) * 2021-07-12 2021-09-17 上海喜马拉雅科技有限公司 语音合成方法、装置、电子设备以及计算机可读存储介质
JP2021177228A (ja) * 2020-05-08 2021-11-11 コリア アドバンスド インスティチュート オブ サイエンス アンド テクノロジィ 多言語多話者個性表現音声合成のための電子装置およびこの処理方法
US11830473B2 (en) 2020-01-21 2023-11-28 Samsung Electronics Co., Ltd. Expressive text-to-speech system and method

Citations (3)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
US20090055162A1 (en) * 2007-08-20 2009-02-26 Microsoft Corporation Hmm-based bilingual (mandarin-english) tts techniques
JP2010237323A (ja) * 2009-03-30 2010-10-21 Toshiba Corp 音声モデル生成装置、音声合成装置、音声モデル生成プログラム、音声合成プログラム、音声モデル生成方法および音声合成方法
US20110218804A1 (en) * 2010-03-02 2011-09-08 Kabushiki Kaisha Toshiba Speech processor, a speech processing method and a method of training a speech processor

Patent Citations (3)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
US20090055162A1 (en) * 2007-08-20 2009-02-26 Microsoft Corporation Hmm-based bilingual (mandarin-english) tts techniques
JP2010237323A (ja) * 2009-03-30 2010-10-21 Toshiba Corp 音声モデル生成装置、音声合成装置、音声モデル生成プログラム、音声合成プログラム、音声モデル生成方法および音声合成方法
US20110218804A1 (en) * 2010-03-02 2011-09-08 Kabushiki Kaisha Toshiba Speech processor, a speech processing method and a method of training a speech processor

Non-Patent Citations (5)

* Cited by examiner, † Cited by third party
Title
ILYA SUTSKEVER ET AL., SEQUENCE TO SEQUENCE LEARNING WITH NEURAL NETWORKS, 2014
JAVIER LATORRE: "New approach to the polyglot speech generation by means of an HMM-based speaker adaptable synthesizer", SPEECH COMMUNICATION, vol. 48, no. 10, October 2006 (2006-10-01), pages 1227 - 1242, XP027926287, Retrieved from the Internet <URL:https://www.sciencedirect.com/science/article/pii/S0167639306000483> *
See also references of EP3739476A4 *
TATYANA POLYAKOVA: "Main issues in grapheme-to-phoneme conversion for TTS", PROCESAMIENTO DEL LENGUAJE NATURAL 2005, January 2005 (2005-01-01), pages 29 - 34, XP055625392, Retrieved from the Internet <URL:https://www.researchgate.net/publication/28166791_Main_issues_in_grapheme-to-phoneme_conversion_for_TTS> *
YOUNGGUN LEE: "Learning pronunciation from a foreign language in speech synthesis networks", 22 October 2018 (2018-10-22), XP081053625, Retrieved from the Internet <URL:https://openreview.net/revisions?id=HkeYCNgooQ> *

Cited By (14)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN111312228A (zh) * 2019-12-09 2020-06-19 中国南方电网有限责任公司 一种基于端到端的应用于电力企业客服的语音导航方法
CN111247581A (zh) * 2019-12-23 2020-06-05 深圳市优必选科技股份有限公司 一种多语言文本合成语音方法、装置、设备及存储介质
CN111247581B (zh) * 2019-12-23 2023-10-10 深圳市优必选科技股份有限公司 一种多语言文本合成语音方法、装置、设备及存储介质
US11830473B2 (en) 2020-01-21 2023-11-28 Samsung Electronics Co., Ltd. Expressive text-to-speech system and method
GB2591245A (en) * 2020-01-21 2021-07-28 Samsung Electronics Co Ltd An expressive text-to-speech system
GB2591245B (en) * 2020-01-21 2022-06-15 Samsung Electronics Co Ltd An expressive text-to-speech system
JP2021177228A (ja) * 2020-05-08 2021-11-11 コリア アドバンスド インスティチュート オブ サイエンス アンド テクノロジィ 多言語多話者個性表現音声合成のための電子装置およびこの処理方法
CN111858961A (zh) * 2020-07-27 2020-10-30 西交利物浦大学 用于知识图谱中节点和链接的多语言知识匹配方法及装置
CN111858961B (zh) * 2020-07-27 2024-02-02 西交利物浦大学 用于知识图谱中节点和链接的多语言知识匹配方法及装置
CN112365882B (zh) * 2020-11-30 2023-09-22 北京百度网讯科技有限公司 语音合成方法及模型训练方法、装置、设备及存储介质
CN112365882A (zh) * 2020-11-30 2021-02-12 北京百度网讯科技有限公司 语音合成方法及模型训练方法、装置、设备及存储介质
CN112652291A (zh) * 2020-12-15 2021-04-13 携程旅游网络技术（上海）有限公司 基于神经网络的语音合成方法、系统、设备及存储介质
CN112652291B (zh) * 2020-12-15 2024-04-05 携程旅游网络技术（上海）有限公司 基于神经网络的语音合成方法、系统、设备及存储介质
CN113409761A (zh) * 2021-07-12 2021-09-17 上海喜马拉雅科技有限公司 语音合成方法、装置、电子设备以及计算机可读存储介质

Similar Documents

Publication Publication Date Title
WO2019139428A1 (fr) Procédé de synthèse vocale à partir de texte multilingue
JP7500020B2 (ja) 多言語テキスト音声合成方法
WO2019139430A1 (fr) Procédé et appareil de synthèse texte-parole utilisant un apprentissage machine, et support de stockage lisible par ordinateur
WO2019139431A1 (fr) Procédé et système de traduction de parole à l&#39;aide d&#39;un modèle de synthèse texte-parole multilingue
WO2020190054A1 (fr) Appareil de synthèse de la parole et procédé associé
WO2020027619A1 (fr) Procédé, dispositif et support d&#39;informations lisible par ordinateur pour la synthèse vocale à l&#39;aide d&#39;un apprentissage automatique sur la base d&#39;une caractéristique de prosodie séquentielle
WO2020190050A1 (fr) Appareil de synthèse vocale et procédé associé
US5806033A (en) Syllable duration and pitch variation to determine accents and stresses for speech recognition
KR102306844B1 (ko) 비디오 번역 및 립싱크 방법 및 시스템
JP2002520664A (ja) 言語に依存しない音声認識
WO2022045651A1 (fr) Procédé et système pour appliquer une parole synthétique à une image de haut-parleur
Nursetyo LatAksLate: Javanese script translator based on Indonesian speech recognition using sphinx-4 and google API
Zhang et al. Chinese speech synthesis system based on end to end
Lopez-Gonzalo et al. Automatic prosodic modeling for speaker and task adaptation in text-to-speech
Samsudin et al. Constructing a Reusable Linguistic Resource for a Polyglot Speech Synthesis
Jiang et al. Overview of the IBM Mandarin text-to-speech system
CANDEIAS CHAPTER TWELVE THE DIALOGUE BETWEEN MAN AND MACHINE: THE ROLE OF LANGUAGE THEORY
Kayte et al. Overview of Hidden Markov Model for Test-To-Speech Synthesis Methods
Sajini et al. Speaker Independent Text to Speech for Malayalam
JPS58223195A (ja) 擬似連続発声に於ける音声認識方式

Legal Events

Date Code Title Description
121 Ep: the epo has been informed by wipo that ep was designated in this application

Ref document number: 19738599

Country of ref document: EP

Kind code of ref document: A1

ENP Entry into the national phase

Ref document number: 2020538690

Country of ref document: JP

Kind code of ref document: A

NENP Non-entry into the national phase

Ref country code: DE

ENP Entry into the national phase

Ref document number: 2019738599

Country of ref document: EP

Effective date: 20200811