TW202016921A - Method for speech synthesis and system thereof - Google Patents

Abstract

A method for speech synthesis is to first perform text analysis on a Chinese-foreign language mixed sentence to generate linguistic information, and the linguistic information includes Chinese part-of-speech and the foreign language part-of-speech decided by the Chinese syntactic structure in the sentence, and perform tone prediction process to generate tone information about the foreign language, and then perform breakpoint prediction via the linguistic information to generate breakpoint information of the Chinese-foreign language mixed sentence. Finally, the linguistic information, the tone information and the breakpoint information are integrated to generate a synthesized speech of the Chinese and foreign mixed sentence. Further, a system for speech synthesis is disclosed.

Description

Speech synthesis method and system

The invention relates to a speech output, especially a speech synthesis method and system of mixed Chinese and foreign language sentences.

For a Chinese society whose mother tongue is Chinese, the Mandarin-English speech synthesis system (Text-To-Speech, TTS for short) is used relatively frequently. The so-called Sino-British mixed TTS refers to Chinese as the main language, supplemented by English, that is, English vocabulary or phrases are embedded in Chinese sentences, and the English vocabulary embedded in the Chinese words and sentences can be the translation of proper nouns, abbreviations and Chinese words Or idioms, so use English vocabulary to replace the corresponding Chinese words, for example, "Facebook is a well-known online community", "Internet of Things IoT is a popular search keyword on the Internet in recent years", "implementation of this application "The difficulty is quite high", "good professor teaches good student", "our ideas do not match, so I do not follow his way" and other daily expressions.

Like the traditional single-language TTS, the construction of the mixed-language TTS requires the recording of a specific speaker's speech database, and the establishment of text analysis, prosody generation and synthesis processing and other related procedures, but it is used in mixed-language TTS applications. Among the requirements, each of the above procedures requires special consideration.

Regarding the recording of the speech database, since the Chinese-English mixed speech data is mainly Chinese and supplemented by English, the ratio of Chinese and English contained in it is extremely disparate, so it is necessary to achieve sufficient speech synthesis units that can contain two languages It is not easy to change with rich rhythm, so the present invention adopts the method and system of recording the required Chinese and English voice data separately by the same speaker.

On the other hand, since the audio data recorded in a foreign language (second language or English) by a native Chinese speaker is easily affected by the native language (first language or Chinese), this impact includes differences in acoustics, prosody and grammar. feature.

Therefore, the system construction of the present invention is based on the following features:

First, in terms of acoustics, English pronunciation is easily replaced by similar Chinese pronunciation.

Second, in the change of rhythm, tone information in Chinese will replace stress information in English, that is, tone borrowing, and the generation of breakpoints in rhythm will mostly be based on the customary habits of the mother tongue.

Thirdly, in the grammatical information of Part Of Speech as an example, the part of speech of the embedded English vocabulary or phrase should be determined by the Chinese grammatical structure of the entire sentence, not just by each To judge the original part of speech of English words or phrases.

Therefore, according to the above features, the present invention provides a method of speech synthesis, which includes: performing a text analysis on a mixed sentence of Chinese and foreign languages to generate a Chinese grammatical structure that includes Chinese parts of speech and is determined by the entire sentence The foreign language part of speech of the foreign language and other language information, and the tone information of the foreign language is generated by the tone prediction process; the language information is generated by the breakpoint prediction process to generate the breakpoint information of the mixed sentence of the Chinese and the foreign language; And integrate the language information, tone information and breakpoint information through a speech synthesis process to generate a synthesized speech of the mixed sentence of the Chinese language and the foreign language.

In the aforementioned speech synthesis method, the character analysis includes traditional character analysis and foreign part-of-speech tagging. The traditional character analysis uses conventional methods to indicate the part-of-speech of Chinese. As for the model required for foreign part-of-speech tagging, it uses both Chinese and foreign words. The mixed text data generates input information through traditional text analysis, and then the input information is used to construct a foreign language part-of-speech tagging model through the process of foreign part-of-speech tagging training.

In the aforementioned speech synthesis method, the model required for the process of tone prediction processing is to generate input information through the text analysis using mixed text data of Chinese and foreign languages, and the data in the mixed text data of Chinese and foreign languages In the foreign language, the tone generated by the annotator adopts the silent method as the target information, and a tone prediction model is built by combining the input information and the target information to perform a tone prediction training procedure.

In the aforementioned speech synthesis method, the model required for the breakpoint prediction process uses Chinese text data to generate the language information, and the breakpoint mark of the Chinese speech database corresponding to the Chinese text data is used as the target information. Combined with the language information and target information, the Chinese breakpoint prediction training program is built to build a breakpoint prediction model.

In the aforementioned speech synthesis method, among the models required for the speech synthesis process, Chinese speech data and foreign language speech data and Chinese and foreign languages are used. Corresponding language, tone, and breakpoint information are integrated and trained to build an HTS model. In addition, the tone information of the text-to-speech database must be generated through tone-marking processing by means of a tone-recognition model. The tone-recognition model is built by the process of tone-recognition training by the Chinese speech database and the corresponding Chinese text data.

In addition, the present invention also provides a speech synthesis system, which includes: a text analysis module, which performs text analysis on mixed sentences of Chinese and foreign languages to generate a Chinese grammatical structure that includes Chinese parts of speech and is determined by the entire sentence The foreign language part of speech of the foreign language and other language information; the tone prediction module, which generates tonal information for foreign languages of mixed sentences of Chinese and foreign languages; the breakpoint prediction module, uses the language information to generate mixed sentences of the Chinese and foreign languages Breakpoint information; and the HTS module, which integrates the language information, tone information, and breakpoint information to produce a synthesized speech of the mixed sentence of the Chinese and the foreign language.

It can be seen from the above that the speech synthesis method and system of the present invention mainly generate language information including Chinese part-of-speech and foreign part-of-speech through text analysis, and generate tonal information of the foreign language through a process of tone prediction; the language The information generates breakpoint information of the mixed sentence of Chinese and foreign languages through a breakpoint prediction process; and integrates the language information, tone information, and breakpoint information to generate a synthesized speech of the mixed sentence of Chinese and foreign languages.

101‧‧‧ Mixed Chinese and Foreign Languages

102‧‧‧ Chinese text data

103‧‧‧ Foreign language information

104‧‧‧Chinese voice database

105‧‧‧ Foreign language audio database

111‧‧‧Text analysis

112‧‧‧ Foreign Language Part of Speech Training

121‧‧‧ foreign language tone labeling (marker's silent way)

122‧‧‧Tone prediction training

131‧‧‧ character analysis and processing

132‧‧‧ Chinese breakpoint prediction training

141‧‧‧Tone Recognition Training

151‧‧‧ tone label processing

152‧‧‧Breakpoint mark processing

153‧‧‧ Integrated training

161‧‧‧ Foreign language part of speech model

162‧‧‧ tone prediction model

163‧‧‧Breakpoint prediction model

164‧‧‧ tone recognition model

165‧‧‧ HTS model

200‧‧‧ input sentence signal

201‧‧‧ character analysis

202‧‧‧ tone prediction

203‧‧‧Breakpoint prediction

204‧‧‧Speech Synthesis Mark Generator

205‧‧‧Speech synthesis

206‧‧‧ output synthesized speech

FIG. 1 is a schematic diagram of the training process of the speech synthesis method of the present invention.

Figure 2 is the synthesis processing operation of the speech synthesis method of the present invention Schematic diagram of the architecture.

Figure 3 is a diagram of the relationship between the prosody structure of the breakpoint prediction model and the breakpoint type of the speech synthesis method of the present invention.

The following describes the implementation of the present invention by specific specific examples. Those skilled in the art can easily understand other advantages and effects of the present invention from the contents disclosed in this specification.

It should be noted that the structure, ratio, size, etc. shown in the drawings of this specification are only used to match the content disclosed in the specification, for those who are familiar with this skill to understand and read, not to limit the implementation of the present invention The limited conditions do not have technical significance. Any modification of structure, change of proportional relationship or adjustment of size should still fall within the scope of the invention without affecting the efficacy and the purpose of the invention. The technical content disclosed by the invention can be covered. At the same time, the terms such as "front", "back" and "one" quoted in this specification are only for the convenience of description, and are not used to limit the scope of the invention, the change of the relative relationship or Adjustments, without substantial changes in the technical content, should be regarded as the scope of the invention.

In general mixed Chinese and foreign sentences, some interesting phenomena can be found. For example, Chinese and English mixed sentences are mainly based on the grammatical structure of Chinese, so when determining the part of speech of the embedded English vocabulary or phrase, it should not be limited to its original possible part of speech. This approach can also modify the general English vocabulary. Misuse problem. For example, in the example sentence of "the implementation of this application is quite difficult", "implement" should be marked as a noun or adjective, not a verb, according to the Chinese grammatical structure.

Furthermore, the interaction between the Chinese language that emphasizes tone and the English language that emphasizes stress in the intersection of Chinese and English words, since native Chinese speakers will be affected by the mother tongue in English pronunciation, Therefore, it is preferred to replace accent information of English vocabulary or phrases with tonal information. For example, in the example sentences of "good professor teaches good student" and "our ideas do not match, so I do not follow his way", both "good" and "no" will be followed by The effect of the pronunciation of English vocabulary has different tonal results. For the word "good", it is a two-tone sound and a three-tone sound in order, and for the word "no", it is a two-tone sound and a four-tone sound, which means that the English vocabulary can define the tone information, and then the neighboring Chinese words have a tonal effect.

The present invention includes features such as voice database requirements, part-of-speech information, tone information, and prosody breakpoint information.

Among the requirements of the speech database, the basic requirements of the speech database required for speech synthesis are sufficient synthesis units and rich rhythm changes. The use of mixed Chinese and English speech data can avoid the mismatch between training and testing. However, due to the disparity in the ratio of Chinese to English, the basic requirements of the above two corpora are not easy to meet, and because native speakers of Chinese will have inconsistencies in the variability of English pronunciation performance, this phenomenon is The mixed Chinese and English sentences are more obvious, so the present invention adopts the method of recording the voice data of Chinese and English by the same language.

In the part-of-speech information, due to user misuse or habitual simplification, the English vocabulary or phrase embedded in the Chinese-English mixed sentence is prone to error. The practice of the present invention is to embed by determining the Chinese grammatical structure of the entire sentence The part of speech of English vocabulary or phrases. Specifically, the method used by the present invention to replace the conventional TTS is to give foreign languages a special classification for the embedded English vocabulary or phrase; or, only use the possible parts of speech of the English vocabulary or phrase to be embedded in the English dictionary as Judgments based.

In the tonal information, since native speakers of Chinese are affected by the native language in the pronunciation of English, tone borrowing occurs. Therefore, the present invention replaces the stress of vocabulary or phrases in the English language with tonal information News. For Sino-British mixed TTS, the adjustment of the tone information is related to two processing procedures. The first is a text analysis procedure, which needs to generate corresponding tone information for the English vocabulary or phrases in the input sentence. The second The term is an English speech database used for prosody analysis and synthetic parameter model training. It needs to provide tone information, enabling the combination of Chinese speech database and the use of consistent features for Sino-British mixed TTS related processing.

In the prosody breakpoint information, the conventional TTS method only analyzes the text data, and then combines the information such as word breaks and part of speech to establish the grammatical structure of the basic phrase and determine the prosody boundary. However, because the grammatical boundary and the prosodic boundary are not necessarily consistent, the method of the present invention replaces the conventional grammatical boundary represented by the basic phrase by statistically distributing the prosody breakpoint distribution in the speech data, that is, using the training speech data to present Changes in prosody breakpoints build a breakpoint prediction model, and since the person who records the language is Chinese as the mother tongue, the recorded English speech data is easily affected by the mother tongue, so the prosody changes presented in the Chinese speech data are relatively stable, resulting in In contrast, the rhythm changes presented in English speech data are subject to large inconsistencies, so the present invention uses Chinese speech data to build the breakpoint prediction model.

Figures 1 and 2 are the Chinese-foreign language mixed speech synthesis method of the present invention, which includes training processing operations and synthesis processing operations. In this embodiment, the foreign language department uses English as an example, but is not limited to English.

In addition, the Chinese-foreign mixed speech synthesis method can also use hardware, software and/or firmware as a Chinese-foreign mixed speech synthesis system, including a foreign language part-of-speech tagging module that can implement foreign language parts-of-speech as shown in Figure 1 Mark model 161, the tone prediction module can implement the tone prediction model 162 as shown in Figure 1, the breakpoint prediction module can implement the breakpoint prediction model 163 as shown in Figure 1, and the tone recognition module can be implemented As the tone recognition model 164 shown in FIG. 1, the HTS module can implement the HTS model 165 shown in FIG. 1, and the text analysis module can implement the text analysis 111 shown in FIG. However, the invention is not limited to this.

As shown in FIG. 1, the relevant procedures of the training processing operation of the present invention include: building a foreign part-of-speech tag model 161 for English vocabulary or phrases in Chinese-English mixed sentences; and building English vocabulary for Chinese-English mixed sentences Or phrase phrase prediction model 162; build a breakpoint prediction model 163 of the Chinese speech database 104; build a tone recognition model 164 for providing tone information for a foreign language speech database 105 such as English; and integrate Chinese and English Build a speech synthesis system (HMM-based speech synthesis system (H-Triple-S for short) based on the Hidden Markov Model (HMM) based on the speech data and its language, tone, breakpoints and other information , That is, HTS) model 165, and the data required to build these models include mixed Chinese and foreign text data 101, Chinese speech database 104 and its corresponding Chinese text data 102, foreign language speech database 105 and its Corresponding to the foreign language text data 103, the construction of these models can be regarded as the core processing program of the training processing operation of the present invention.

The foreign language part-of-speech tag model 161 is built for English vocabulary or phrases in Chinese-English mixed sentences, which uses the language information generated by the Chinese-foreign mixed text data 101 through the text analysis 111 as the input signal, and then Techniques for training of foreign part-of-speech tags in conditional random fields (CRF) 112 to build the foreign part-of-speech tagging model 161 for embedded English vocabulary or phrases, where the text analysis 111 belongs to traditional word processing The program includes word breaking, basic part-of-speech tagging, regularization and transposition processing, and the foreign-language part-of-speech tagging model 161 is a model required by the invention for the part-of-speech adjustment process newly added to traditional text analysis for the second The processing procedure of the character analysis 201 in the synthesis processing operation shown in the figure, so the part-of-speech adjustment process can be regarded as a follow-up process of the traditional character analysis processing, and the English vocabulary or film in the mixed Chinese and English sentences can be improved by this process The part of speech of the language indicates the correct rate.

In this embodiment, for the English vocabulary or phrase in the mixed Chinese-English sentence, the CRF technology is used to build the foreign language part-of-speech tagging model 161, which uses the mixed Chinese and foreign language text data 101 as training data, and its input The signal includes the possible part of speech, prefix information, suffix information and the word information and part-of-speech information of the adjacent vocabulary of the English vocabulary, and the signal of the output tag is determined by the grammatical structure of the sentence of the Chinese vocabulary of the pending English vocabulary Part of speech information.

The tone prediction model 162 is built for English vocabulary or phrases in Chinese and English mixed sentences, and the required information is mixed Chinese and foreign words The language information (input information) of the data 101 processed by the text analysis 111, and the tonal information (target information) of the English vocabulary or phrase in the mixed Chinese and foreign text data 101, and then combining the two information and using the CRF technology The procedure of tone prediction training 122 is performed to build the tone prediction model 162. Specifically, the procedure for annotating 121 foreign-language tones in an embedded English vocabulary or phrase is to generate the tonal information by means of the meditation 121 of the annotator. The annotator is a native Chinese speaker who is fluent in English discourse and has completed the construction. The model will be applied to the program of tone prediction 202 in the synthesis processing operation of FIG. 2, in particular, the object of the processing of tone prediction 202 is a mixture of Chinese and English text data, not all English text data.

In this embodiment, for the English vocabulary or phrase in the mixed Chinese-English sentence, the tone prediction model 162 is built using CRF technology, which uses the mixed Chinese-foreign text data 101 as training data, and its input information includes : Word information; syllable information; syllable position information in the word; part-of-speech information; punctuation information following the current syllable; original tone information; information on the original tone combination of the previous syllable and the current syllable; the previous syllable, current syllable and The syllable combination information of the next syllable; and the combination information of the previous syllable, the position of the current syllable and the next syllable in the word, etc.

Furthermore, in the information of the original tone, for the Chinese vocabulary, the information of the original tone is the tone of each syllable, and for the English vocabulary, the information of the original tone is the original accent information of the English vocabulary .

In addition, the output mark required for the training data of the tone prediction training 122 is the tone information of the syllables of the English vocabulary to be processed through manual marking, and then the CRF technology is used to complete the establishment of the tone prediction model 162.

The establishment of the breakpoint prediction model 163 mentioned above is due to the fact that the speaker who provided the training speech data is Chinese as the mother tongue, and the recorded foreign language (or English) speech data is easily affected by the mother tongue, resulting in the presentation of the Chinese speech data The rhythm changes are relatively stable. On the other hand, because the Chinese-English mixed sentence system of the present invention is mainly Chinese and English is supplementary, the present invention uses independent Chinese speech data to build the breakpoint prediction model 163.

As shown in FIG. 1, the input information of the breakpoint prediction model 163 is the language information of the Chinese text data 102 after the text analysis process 131, and the corresponding Chinese voice database 104 is processed by the statistical breakpoint marking process 152 The generated breakpoint mark is used as the target information, and then combined with the language information and the target information, the Deep Neural Network (DNN) technology is used to perform the Chinese breakpoint prediction training 132 procedure. Specifically, the function of the breakpoint marking process 152 is to perform statistics on the pitch, pitch, volume, and pause of the Chinese voice database 104 to generate appropriate thresholds, and then use these thresholds to perform voice data The labeling of the prosody breakpoint is to complete the breakpoint prediction model 163 and supply the program for the breakpoint prediction 203 of the synthesis processing operation shown in FIG. 2.

In this embodiment, the training data used in the Chinese breakpoint prediction training 132 of the breakpoint prediction model 163 is the Chinese speech database 104 and the corresponding Chinese text data 102, wherein the breakpoint mark is used The method of processing 152 the breakpoint annotation for the Chinese speech database 104 is as follows:

The first point, as shown in Figure 3, defines the four-level prosody structure and the related seven types of breakpoints B0, B1, B2-1, B2-2, B2-3, B3, B4, where the first order The hierarchy is syllable (labeled as SYL), which has corresponding breakpoints B0, B1, and the second hierarchy is prosodic word (labeled as PW), which has corresponding breakpoints B2-1, B2- 2, B2-3, the third hierarchical system is prosodic phrase (marked as PPh), which has a corresponding breakpoint B3, and the fourth hierarchical system is breath group (marked as BG) or prosodic sentence A prosodic group (labeled PG) has a corresponding breakpoint B4.

The second point is to perform statistics on various rhythm parameters for the Chinese speech database 104 to generate appropriate thresholds, and then use these thresholds to mark the seven kinds of rhythm breakpoints B0, B1, B2-1 for the speech data. B2-2, B2-3, B3, B4. For example, the prosody parameter system includes syllable pause length, vowel energy saving low point, normalized fundamental frequency jump value, normalized syllable extension factor, and fundamental frequency pause length between syllables.

The third point, the correlation between the stated prosody parameters and the breakpoint markers is as follows: the syllable pause length affects the most important prosody parameters of the breakpoint markers, and most word boundaries where punctuation marks appear will have a longer Pauses, such as breakpoints B3, B4, and breakpoint B4 has a longer pause length than breakpoint B3, so syllable pause length can be used to distinguish breakpoint B3 from breakpoint B4. The syllable positions of other non-word boundaries usually have short pauses or non-pauses, such as breakpoint B0 and breakpoint B1. These two types of breakpoints B0 and B1 can further use the fundamental frequency pause between syllables and the low point of energy saving Two characteristics are distinguished. The boundary of words that are not punctuation marks includes three characteristics of syllable pause length above medium level, fundamental frequency jump and syllable extension, etc., which are classified as breakpoints B2-2, B2-1, B2-3 respectively.

Furthermore, the marks of the breakpoints B0, B1, B2-1, B2-2, B2-3, B3, B4 will be used as the output target values required to build the breakpoint prediction model 163, as for the breakpoint prediction model The input features of 163 include: phoneme category; whether the current breakpoint is a word boundary; punctuation symbol category; the current sentence length in syllables; the length of the previous sentence in syllables; the next sentence in syllables The length of the word; the distance to the previous punctuation mark in syllables; the distance to the next punctuation mark in syllables; the part-of-speech of the current word and its previous p words; the current word and its preceding p words The number of syllables; the part of speech of q words after the current word; the number of syllables in q words after the current word.

In addition, p and q can be adjusted according to the requirements of system complexity, and then the breakpoint prediction model 163 is built using the DNN architecture.

The tone identification model 164 described above provides tone information for English speech data. In order to use the consistent features for the prosody analysis of Chinese and foreign speech data and the integration training 153 of the HTS model 165, the extraction of prosody features in foreign speech data is to replace accent information with tone information.

As shown in FIG. 1, the tone recognition model 164 is built by the Chinese speech database 104 and its corresponding Chinese text data 102 in conjunction with DNN technology for tone recognition training 141, which is used for subsequent tone labeling processing 151, to Provides tone information for English voice data. In particular, the object of the tone labeling process 151 is all English data, so the model required by the tone labeling process 151 is not the same as the tone prediction model 162 whose setting object is Chinese and English mixed data.

In this embodiment, the training data used to build the tone recognition model 164 is the Chinese speech database 104 and its corresponding Chinese text data 102, and the input parameters include the positive values obtained for the logarithmic fundamental frequency trajectory of the syllable Intersection coefficient, syllable length, syllable energy saving, and pause length between syllables. Among them, the coverage of the input parameters of these characteristics can be adjusted according to the requirements of the system complexity. Usually, the current syllable and the three syllables before and after it are selected. In the interval, the tone recognition model 164 is built using the DNN framework.

The construction of the HTS model 165 mentioned above is obtained by integrating and training 153 Chinese and English speech data and its language, tone, breakpoints and other information. The information includes an independent Chinese speech database 104 and foreign language speech data Library 105, and the HTS label data required for training is generated by integrating the language information of the two speech databases, the information of the breakpoint labeling process 152, and the tone information, etc., for use in the synthesis processing operation of FIG. 2 In the speech synthesis 205 procedure. Therefore, during the training process of the HTS model 165, the required annotation files and question sets have the following characteristics: one of the characteristics is that the audio data in two different languages can use the same annotation format, while the other The feature is that the problem set used for model training can share prosody information related to tones and breakpoints.

In this embodiment, the HTS model 165 is built using the HTS-2.3 toolkit, and the training voice data includes individual Chinese voice data and English voice data, together with language information and breakpoint prediction processing after integrated text analysis processing HTS annotations generated after the breakpoint information and tone information about embedded English words or phrases. The HTS annotations include: phoneme categories (Consider the current and the two before and after a total of five phonemes); Tone category (consider the current and before and after each a total of three syllables); The vowel category of the current syllable; The current break point before and after the breakpoint category; The current syllable phoneme position; Current Syllable positions of PW, PPh and BG/PG; current PW positions of PPh and BG/PG; current PPh positions of BG/PG.

The present invention also discloses a synthesis processing operation, as shown in FIG. 2, which includes: inputting sentence sentence signal 200 mixed with Chinese and English through text analysis 201 to generate corresponding language information, the text analysis including analysis processing of Chinese And the part-of-speech tagging process for the English vocabulary or phrase in the sentence signal; the English vocabulary or phrase in the sentence signal is generated by the tone prediction 202 procedure to generate the corresponding tone information; the language The information passes through the process of break prediction 203 to generate corresponding breakpoint information; integrate the language information, tone information and breakpoint information, and pass the HMM-based speech synthesis label generator (HTS label generator) 204 Generate corresponding HTS annotations; and use the HTS annotations to generate and output synthesized speech 206 through HMM-based speech synthesis 205 processing.

In this embodiment, the synthesis processing operation is associated with the training processing operation, that is, the four processing procedures such as text analysis 201, tone prediction 202, breakpoint prediction 203, and speech synthesis 205 all need to pass the training The model built by the processing operation is carried out.

In the procedure of the text analysis 201, the part-of-speech tagging process needs to use the foreign language part-of-speech tagging model 161 built by the CRF technology of the training processing operation, and uses the same input sample, according to the Chinese grammatical structure of the sentence Decide the part of speech of the embedded English vocabulary or phrase.

In the procedure of the tone prediction 202, it needs to use the tone prediction model 162 built by the CRF technology of the training processing operation, and use the same input template to generate the corresponding tone information.

In the process of the breakpoint prediction 203, it needs to use the breakpoint prediction model 163 built by the DNN technology of the training processing operation, and uses the same input features to generate the corresponding breakpoint information.

In the speech synthesis mark generator 204, the language information, tone information and breakpoint information are integrated, and corresponding HTS annotations are generated, wherein the format of the HTS annotations is based on the HTS model 165 of the training processing operation.

In the procedure of the speech synthesis 205, according to the HTS annotation, the synthesized speech is generated by the procedure of the speech synthesis 205 through the HTS model 165 of the training process.

In summary, in the speech synthesis method and system of the present invention, the language information including Chinese part-of-speech and foreign part-of-speech is generated by text analysis, and the tonal information of the foreign language is generated by a tone prediction process; Language information generates breakpoint information for mixed sentences of Chinese and foreign languages through a breakpoint prediction process; and integrates the language information, tone information, and breakpoint information to generate synthesized speech of the mixed sentences of Chinese and foreign languages.

The above embodiments are used to exemplify the principles and effects of the present invention, rather than to limit the present invention. Anyone who is familiar with this skill can modify the above embodiments without departing from the spirit and scope of the present invention. Therefore, the scope of protection of the rights of the present invention should be as listed in the scope of patent application mentioned later.

200‧‧‧ input sentence signal

201‧‧‧ character analysis

202‧‧‧ tone prediction

203‧‧‧Breakpoint prediction

204‧‧‧Speech Synthesis Mark Generator

205‧‧‧Speech synthesis

206‧‧‧ output synthesized speech

Claims (8)

A speech synthesis method includes: performing text analysis on mixed sentences of Chinese and foreign languages to generate language information including Chinese parts of speech and foreign parts of speech, and generating tone information of the foreign language through a process of tone prediction; passing the language information through The breakpoint prediction process generates the breakpoint information of the mixed sentence of the Chinese and the foreign language; and integrates the language information, tone information, and breakpoint information through the speech synthesis process to generate the synthesized speech of the mixed sentence of the Chinese and the foreign language . The speech synthesis method as described in item 1 of the patent application scope, wherein the character analysis includes traditional character analysis and foreign language part-of-speech mark processing. The speech synthesis method as described in item 2 of the patent application scope, wherein the model required for the foreign language part-of-speech tagging process uses the mixed text data of the Chinese and the foreign language to generate input information through traditional text analysis, and then the input information Built with foreign part-of-speech tag training. The speech synthesis method as described in item 1 of the patent application scope, wherein the model required for the tone prediction process is to generate input information through the text analysis using the mixed text data of the Chinese and the foreign languages, and the Chinese and The foreign language in the mixed-language text data of the foreign language is established as the target information by using the tone generated by the annotator in the silent way, and is combined with the input information and the target information to carry out the tone prediction training process. The speech synthesis method as described in item 1 of the patent application scope, wherein the model required for the breakpoint prediction procedure is generated using Chinese text data The language information, and the breakpoint mark of the Chinese speech database corresponding to the Chinese text data is used as the target information, and then combined with the language information and the target information to build a Chinese breakpoint prediction training program. The speech synthesis method as described in item 1 of the patent application scope, in which the model required for the speech synthesis process is information on Chinese speech data and foreign language speech data and the language, tones, and breakpoints corresponding to the Chinese language and the foreign language Built for integration training. The speech synthesis method as described in item 6 of the patent application scope, wherein the tone information of the foreign language speech database must be generated through the tone labeling process by means of a tone recognition model, and the tone recognition model is built by the Chinese speech database It is based on the procedure of tone recognition training with the corresponding Chinese text data. A speech synthesis system includes: a text analysis module, which performs text analysis on mixed sentences of Chinese and foreign languages to generate language information including Chinese parts of speech and foreign parts of speech; a tone prediction module, which targets the Chinese and foreign languages Mixed-language sentences generate tonal information in foreign languages; breakpoint prediction module uses the language information to generate breakpoint information for mixed-language sentences in Chinese and foreign languages; and HTS module integrates the language information, tonal information and breakpoint information To generate a synthesized speech of the mixed sentence of the Chinese and the foreign language.

Images