TWI467566B - Polyglot speech synthesis method - Google Patents

Description

Multilingual speech synthesis method

The present invention relates to a speech synthesis method, and more particularly to a multi-language speech synthesis method.

Since speech is the most direct medium for human communication, it is very important to use speech as a medium for human-computer interaction. Many products based on speech synthesis technology have emerged. For example: mobile phone voice-activated dialing, text-to-speech synthesis (TTS system), instant voice navigation system and many other practical applications.

In addition, in today's society, speakers often use different languages to communicate during conversations. This phenomenon most often occurs in a particular word, or the phrase expresses its meaning in a language that is fluent. For example: help me cancel tomorrow's meeting → help me cancel the meeting tomorrow; this class is not down, I will pass all. All pass is in English. The dialogue is often used in this sentence for a brief expression. It is mixed in Chinese sentences. The sentence changes English from Chinese. This is called code-switching problem.

However, traditional TTS can only be synthesized into a single language, which is no longer sufficient for users. For multilingual synthesis systems, multilingual linguistic corpus is generally used, but this is not as easy to collect as a single language. On the other hand, if you combine the language corpus of many different speakers, you will face the inconsistency of the speaker's characteristics. In order to deal with the inconsistency, you must do the sound conversion.

In general, in speech synthesis technology, HMM-based model adaptation and voice conversion are utilized. However, in model adaptation, you will encounter problems where the cross-language phonemes are not exactly the same. In the sound conversion, there is a problem that cross-language phonemes cannot collect parallel corpora.

Therefore, how to provide a multi-language speech synthesis method can solve the above problems and improve the speech synthesis performance, which is one of the current important topics.

In view of the above problems, an object of the present invention is to provide a multi-lingual speech synthesis method capable of solving conventional problems and improving speech synthesis performance.

In order to achieve the above object, a multi-lingual speech synthesis method according to the present invention comprises: selecting a first language as a main language, and collecting a first corpus of the first language; selecting a second language as a secondary language, and Collecting a second corpus of the second language; classifying, by the first corpus and the second corpus, the plural first pronunciation unit of the first language and the plural second pronunciation unit of the second language, the second pronunciation The unit includes a plurality of special pronunciation units; and a first pronunciation unit corresponding to the special pronunciation units.

In an embodiment, the first corpus is more than the second corpus.

In an embodiment, the multi-lingual speech synthesis method further comprises: training the first speech model of the first language with the first corpus; and training the second speech model of the second language with the second corpus.

In an embodiment, the multi-lingual speech synthesis method further comprises: classifying the first pronunciation units and the second pronunciation units by an International Phonetic Alphabet (IPA).

In an embodiment, before determining the first pronunciation unit corresponding to the special pronunciation units, the multi-lingual speech synthesis method further comprises: subdividing the first corpus into a plurality of first speech boxes; and placing the second language The material is subdivided into a plurality of second sound boxes, such that the special sounding unit pairs and the first sounding unit are performed by the sound box correspondence in the time series.

In an embodiment, one of the first languages is divided into a plurality of the first sound boxes. One of the second language phonemes is divided into a plurality of the second sound boxes.

In one embodiment, the step of determining the first pronunciation unit corresponding to the particular pronunciation unit is performed by an Articulatory attribute or an Auditory feature.

In an embodiment, after the special pronunciation unit corresponds to the first pronunciation unit, the multi-lingual speech synthesis method further comprises: training the second language by the first pronunciation units corresponding to the special pronunciation unit. The speech model of these special pronunciation units.

In an embodiment, the multi-lingual speech synthesis method further comprises: adding a speech model of the special pronunciation units of the second language to the speech model of the first language.

As described above, the multi-lingual speech synthesis method of the present invention can solve the problem that the trans-language phonemes are not identical, and can solve the problem that the parallel corpus cannot be collected, thereby generating multi-language speech synthesis of any speaker.

DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS In the following, a preferred embodiment of the invention will be described with reference to the accompanying drawings.

1 is a flow chart of steps of a multi-language speech synthesis method according to a preferred embodiment of the present invention, including step S01: selecting a first language as a primary language, and collecting a first corpus of the first language; step S02: Selecting a second language as a secondary language, and collecting a second corpus of the second language; step S03: using the first corpus and the second corpus to use the first vocal unit of the first language and the second language The plurality of second pronunciation units are classified, the second sounding units include a plurality of language-specific phone units; and step S04: determining the first sounding unit corresponding to the special sounding units. The multi-language speech synthesis method of the present embodiment is described in detail below, wherein the first language is in Chinese and the second language is in English, but this is not intended to limit the present invention. In addition, it should be noted that the above steps S01 to S04 do not represent the absolute order. For example, step S02 may be performed first and then step S01 may be performed.

First, select a first language as the primary language and collect the first corpus of one of the first languages. Here, the first corpus is taken as an example from the same language, for example, the president of Taiwan.

Furthermore, a second language is selected as the secondary language and a second corpus of the second language is collected. Here, the second corpus is taken as an example from the same language, for example, the US President. The purpose of this embodiment is to produce a voice that speaks English in the accent of the President of Taiwan. Further, in the present embodiment, the first corpus as the main language is more than the second corpus as the secondary language.

In addition, after steps S01 and S02, the multi-lingual speech synthesis method may further include: training the first speech model of the first language by using the first corpus; and training the second speech model of the second language by using the second corpus; . In this aspect, the corpus design and construction of different linguistics are carried out in Chinese and English, and the training corpus is used to train all the acoustic model of the phoneme, that is, the first speech model (Chinese) and the second speech model (English) ) training. Model training includes parameter analysis of speech signal parameterization, spectrum acquisition and pitch. In the above training, this embodiment uses the STRAIGHT analysis and synthesis algorithm, which can obtain accurate fundamental frequency parameters and Cepstral feature.

Next, step S03 is performed.

There are differences in the basic pronunciation units between Chinese and English in linguistics. The former is in the Sino-Tibetan language family and the latter is in the Indo-European language family. Therefore, as shown in Figure 2, there are only 24 common tones between Chinese and English (not considering Chinese tones). In addition, there are 16 Chinese and no phonemes in English. There are also 16 Chinese phonemes in English. Because the language is different and the corresponding phoneme cannot be found, it is generally called unseen phone. The unknown phonology constitutes the basic phoneme of a language-specific phone.

In order to deal with the problem of unknown sound, the present embodiment classifies the phonemes of Chinese and English by the pronunciation attribute of the phoneme (the first sounding unit and the second sounding unit can each be composed of at least one phoneme). The Articulatory Attribute can include the Place Of Articulation and the Manner Of Articulation. The characteristic is that the position and pronunciation of the phone are not changed by the language, nor by the speaker. The effect is a robust characteristic parameter. Linguists have developed the International Phonetic Alphabet (IPA) based on the pronunciation properties of each phoneme. In this embodiment, a Chinese-English pronunciation attribute vowel table and a consonant table are established according to rules set by a conventional linguist, as shown in FIGS. 3 and 4. The Chinese corresponding to the English in Fig. 4 are illustrated here: Bilabial, Labio-dental, Dental, Alveolar, Post-Alveolar, Palatal (hard) Arpeggio), Velar, Softlot, Plosive, Implosive, Fricative, Nasal, Trill, Lateral, Approximant (near tone). The classification method adopts the classification method of decision trees. The problem set used by the decision tree also refers to the definition of international phonetic symbols. The phonemes with the same attributes are classified into the same kind, some of which are for Chinese and English, such as babble vowels. , put English /r/ and Chinese /ㄦ/ together, other media-like sounds /w/, /y/, and Chinese phonemes with intermediate sounds such as /ㄓㄨ+*/(* It is any kind of phoneme, and + means the latter one. In this example, it means that any sound after "ㄓㄨ" matches the classification condition of this kind), /ㄐㄧ+*/ is divided together. The following are classification decision tree issues and design principles:

Vowel related problems: including single vowel-related problems such as /a/, /e/, /i/, /o/, /u/; vowel position problems such as front, middle, and back; double vowels Related issues; children's vowels and other issues.

Consonant related problems: there are pronunciation position problems, such as Velar, Coronal, etc.; pronunciation mode problems: like Plosive, Nasal, etc.

When the decision tree is split, in addition to considering the minimum description length (MDL), it must be noted that at least one of the Chinese and English phonemes after splitting is within the decision tree node, as shown in FIG. 5. This shows the Chinese corresponding to English in Figure 5: L_vowel (whether the phoneme on the left is a vowel), C_Labial (whether the current phoneme is a lip tone), L_fricative (whether the phoneme on the left is a wipe), L_O_vowel (on the left) Whether the phoneme is /O/like vowel).

In order to solve the problem of the correspondence between the unknown sound and the special sounding unit, step S04 is performed to determine the first sounding unit corresponding to the special sounding units.

One of the features of this embodiment is that the corresponding system is performed using a smaller "frame". Therefore, the multi-lingual speech synthesis method may further include: subdividing the first corpus into a plurality of first sound boxes; and subdividing the second corpus into a plurality of second sound boxes. It is even possible to divide one of the first language into a plurality of first sound boxes and divide one of the second language into a plurality of second sound boxes. A sound box can be, for example, 5 milliseconds (ms). Furthermore, the step of determining the first pronunciation unit corresponding to the particular pronunciation unit is performed by a pronunciation attribute or an Auditory feature. The pronunciation attribute and the auditory parameter are explained below.

The pronunciation attribute is a very important feature of multi-language speech processing. It can provide the necessary subtle information and handle the pronunciation variation better than other features. It is a strong speech parameter and is not easy to occur because the speaker is different or A situation in which different language parameters change too much. In this embodiment, all the corpora define a total of 22 different pronunciation event detectors according to the classification manner of the pronunciation attribute of the IPA, as shown in FIG. 6, which illustrates the Chinese corresponding to the English in FIG. 6: P (Vowel |x(t)) (the input signal x(t) is the likelihood of the vowel), P(Fricative |x(t)) (the input signal x(t) is the likelihood of the squeak) , P(Nasal | x(t)) (the input signal x(t) is the likelihood of the nasal sound). The detectors used will use a mixture of different acoustical speech characteristics to create a model of the pronunciation event detector, such as the Mel Cepstral Coefficient (MFCC), the zero crossing rate, and the pitch. (pitch), energy, etc. These event detectors can train a classifier by corpus, such as using a neural network or a support vector machine (SVM) to train the classifier. The final output is a set of 22 The vector of dimensions, each dimension representing the probability value of this sound for each Articulatory attribute vector (AAMA vector).

The auditory parameters are designed for human hearing and have the following characteristics: nonlinear perceptual measures such as the Barker metric and the Meyer metric, which are due to the perception of the human ear in the frequency domain, not the full frequency domain. The same sensitivity.

Spectral amplitude compression is logarithmic compression because the human ear's perception of intensity is not linear in the sense of hearing, but rather closer to the logarithmic curve.

Equal loudness curve, loudness is the unit of sound size measurement, based on the single-frequency sound of 1kHz, the corresponding SPL (Sound Pressure Level) is connected to a curve when the loudness of the auditory at different frequencies is the same as the loudness at 1kHz. It is the equal loudness curve.

The shadowing effect occurs when a certain frequency has a specific sound intensity, and a sound of a different frequency needs to strengthen the sound intensity to be received by the human ear. There are two main types, one is frequency masking, the low frequency sound tends to mask high frequency sound, and the other is Temporal Masking.

Referring to FIG. 7, the embodiment adopts the parameters of the Lyon's auditory model, and the steps of the acquisition are as follows: first, the signal is pre-emphasis, and then the 86 filters F ₁ to F _{86 are} combined. Cascade filter, followed by a series of half wave rectification (HWR), which simulates the unidirectional motion of the inner hair cell, and the rectifier has a negative input waveform The half-week elimination function halved the energy. The output of each half-wave rectifier is then cascaded through four automatic gain control (AGC). The AGC changes the value over time and the output of the adjacent rectifier. The final output shows the sound waves entering the human ear. Thereafter, the neural firing rate is transmitted through different sites, and can be expressed by an auditory physiological inner ear auditory model (Cochleagram).

The object of the frame alignment method proposed in this embodiment is to use the first corpus of the first language to select a sequence of sound frames having the closest utterance attribute and auditory characteristic to the second corpus of the second language ( Frame sequence), the steps are as follows:

1. In the established decision tree, find the cluster to which each phoneme tag is classified in the second corpus (refer to Figure 4), and find the corresponding phoneme in the first language from the group.

2. Calculate the substitution cost of the first and second frames, and pick out the first n candidates (candidate).

3. Calculate the concatenation cost between each candidate.

4. Using dynamic programming to find the part of the first corpus corresponding to the best path with the concatenation cost and the minimum substitution cost in the candidate group, and regard it as parallel corpus.

5. Parallel corpus is used to perform pitch Gaussian mixture model (GMM) transformation.

The following are the parameter definitions of the respective frames, the cepstral parameters and the articulation attributes (AA) of the respective frames of the first corpus and the second corpus, and the cepstral parameters include the Mel generalized coefficients (Mel generalized Coefficient, MGC) and auditory features.

The first sound box characteristic parameters are as follows:

The second box characteristic parameters are as follows:

The characteristic parameters are each defined as follows:

SFp _n : The cepstrum parameter of the nth frame in the first corpus.

SFs _n : The cepstrum parameter of the nth sound box in the second corpus.

AFp _n : the pronunciation parameter of the nth sound box in the first corpus (articulatory feature)

AFs _n : the articulatory feature of the nth frame in the second corpus.

AFp _n ^- : The articulatory feature of the n-1th frame in the first corpus.

AFs _n ^- : The articulatory feature of the n-1th frame in the second corpus.

AFp _n ⁺ : the articulatory feature of the n+1th frame in the first corpus.

AFp _n ⁺ : the articulatory feature of the n+1th frame in the second corpus.

The following description of the Substitution cost Substitution cost calculation method uses the Euclidean distance, as in Equation (1):

Where i and j represent the respective sound box index values of the characteristic parameters of the first sound box and the second sound box, and dim is the dimension of the voice parameter. In the actual implementation, the first n nearest speakers will be retained. This is because each candidate with the closest distance to the second corpus has different sound characteristics of the contextual information in the original corpus. . Therefore, in order to consider the characteristics of the front and back sounds into the concatenation cost, the first n most recent sound boxes are retained. In addition, because the sound box of the training corpus has a large number of sound boxes, a vowel sound contains tens of thousands of sound boxes, so the nearest neighbor (Nearest Neighbor) algorithm can be used to find out the first n candidates of each frame in advance to reduce the instant. The computational time spent on computation.

The following is a description of the Concatenation cost Concatenation cost calculation formula (2)

In this embodiment, the context of the context between the candidate and the two layers is considered. The first is the connection relationship, that is, the articulatory feature (AFp _i ) of the candidate itself at the time point i and the articulatory feature (AFp _i-1 ⁺ ) of the frame following the original sound file at the previous time point i-1. . The other layer relationship is the front-end relationship. The candidate at time t is preceded by the original audio file articulatory feature (AFp _i ^- ) and the i1 time point of the candidate's articulatory feature (AFp _i-1 ), for example : If the candidate of the first corpus of time point i and i-1 is originally connected, the value of the shortest Euclidean distance obtained after subtraction is 0.

A schematic diagram of the calculation of the sound box corresponding to this embodiment is shown in FIG. 8. Its algorithm mathematical formula is as shown in equation (3):

among them

Sub(Us _i , Up _i ) is the aforementioned substitution cost, and Con (Up _i-1 , Up _i ) is the concatenation cost. Us _i is the basic synthesis unit of the second corpus at time point i, and Up _i is the basic synthesis unit of the first corpus time point i. Finally, the Viterbi algorithm can be used for dynamic programming to obtain the sequence of the lowest cost frame and treat it as the corpus of the first corpus parallel to the second language.

The following describes the GMM conversion of Pitch.

Due to the sequence of the calibrated sound box, the phonology is not the actual pitch curve of the second language. After calibration, each frame f _{0 is} not changed, and the fundamental frequency curve of the sequence of the sound box obtained by the proposed method (f ₀ contour) There is still a gap between the actual second language and a conversion. Since the filtering of f ₀ has been performed when the candidate was previously selected, the GMM conversion of the joint density can be directly performed. X f ₀ is defined as the speaker of the first language, y is f ₀ of the speaker of the second ^{language, z = [x T y T} ] T GMM for estimating the parameters (α, μ, Σ), the conversion Function is

Q is the total number of GMMs

FIG. 9 is a schematic diagram showing the result obtained by the above-mentioned sound box corresponding method in the embodiment, which displays a plurality of second sound boxes of the second corpus, and the first sound frame of the corresponding first corpus is dispersed in the first In a number of paragraphs of a corpus, and represented by hatching.

After selecting the first pronunciation unit (such as a sound box) corresponding to the special pronunciation units (such as a sound box), the multi-lingual speech synthesis method further includes: first, by the corresponding special pronunciation unit The pronunciation unit trains the speech models of the special pronunciation units of the second language, and adds the speech models of the special pronunciation units of the second language to the speech model of the first language.

It should be noted that in addition to the special pronunciation unit using the above-mentioned sound box corresponding method, generally the non-special pronunciation unit can also use the same sound box corresponding method to find the corresponding sound box, and can be added to the training of the voice model.

In summary, the preferred embodiment of the present invention is based on "pronunciation attributes" and/or "auditory parameters", and proposes a framework for integrating corpora of various languages to implement general speech synthesis. The method is to use a large collection of specific language corpora as the primary language speaker (Primary Language Speaker), and the corpus of other languages uses the cross-language pronunciation attribute to classify and correspond with the main language speakers. First, collect a large number of corpora of a major language and a secondary language, and use IPA to define all the pronunciation units in the language (which can include at least one phone). Then, the speech segments of the special language in the secondary language are classified and corresponding to the speech segments of the main language, wherein the classification and the corresponding basis are the context information of the Articulatory attribute ( The contextual information is combined with the Cepstral feature and the Auditory feature as the basis for selecting the corresponding sound frame, and the sound box is matched for the speech segments of the two languages. Finally, the speech segment corresponding to the good main language can be used to train the special pronunciation unit speech model of the secondary language, so that the speech model of the secondary language special pronunciation unit is added to the main language, and the multi-language speech synthesizer can be realized. Implementation.

In summary, the multi-lingual speech synthesis method of the present invention can solve the problem that the trans-language phonemes are not identical, and can solve the problem that the parallel corpus cannot be collected, thereby generating multi-language speech synthesis of any speaker.

The above is intended to be illustrative only and not limiting. Any equivalent modifications or alterations to the spirit and scope of the invention are intended to be included in the scope of the appended claims.

Steps for S01~S04‧‧‧Multilingual Speech Synthesis Method

1 is a flow chart showing the steps of a multi-lingual speech synthesis method according to a preferred embodiment of the present invention;

Figure 2 is a schematic diagram of the basic pronunciation units of linguistics in Chinese and English;

Figure 3 is a view showing the position of the vowel sound in Chinese and English;

Figure 4 is a table showing the position and mode of the pronunciation of Chinese and English consonants;

Figure 5 is a schematic diagram of decision tree classification;

Figure 6 is a schematic diagram of a pronunciation attribute detector;

Figure 7 is a schematic flow chart of Lyon's auditory parameters;

FIG. 8 is a schematic diagram of calculation of a sound box corresponding to an embodiment of the present invention;

FIG. 9 is a schematic diagram of the result obtained by the sound box corresponding method in the embodiment.

S01～S04. . . Steps in a multilingual speech synthesis method

Claims (10)

A multi-language speech synthesis method includes: selecting a first language as a main language, and collecting a first corpus of the first language; selecting a second language as a secondary language, and collecting one of the second languages a second corpus; using the first corpus and the second corpus to classify the plural first pronunciation unit of the first language and the plural second pronunciation unit of the second language, the second pronunciation unit including a plurality of special a pronunciation unit, the special pronunciation unit including an unknown sound between the first language and the second language; and determining the first pronunciation units corresponding to the special pronunciation units. The multi-lingual speech synthesis method of claim 1, wherein the first corpus is more than the second corpus. The multi-lingual speech synthesis method according to claim 1, further comprising: training the first speech model of the first language by using the first corpus; and training the second language by using the second corpus; A second speech model. The multi-lingual speech synthesis method according to claim 1, further comprising: classifying the first pronunciation unit and the second pronunciation unit by an international phonetic symbol. The multi-lingual speech synthesis method according to claim 1, wherein before determining the first pronunciation unit corresponding to the special pronunciation units, the method further comprises: subdividing the first corpus into a plurality of first speech frames. And subdividing the second corpus into a plurality of second syllables such that the first vocalization unit corresponds to the first vocalization unit by the corresponding phonological frame correspondence. The multi-lingual speech synthesis method according to claim 5, wherein the phoneme of the first language is divided into a plurality of the first sound frames. The multi-lingual speech synthesis method according to claim 5, wherein the phoneme of the second language is divided into a plurality of the second sub-frames. The multi-lingual speech synthesis method according to claim 1, wherein the step of determining the first pronunciation units corresponding to the special pronunciation units is performed by a pronunciation attribute or an auditory parameter. The multi-lingual speech synthesis method according to claim 1, wherein the special pronunciation unit corresponds to the first pronunciation unit, and further comprises: the first pronunciation unit corresponding to the special pronunciation unit by corresponding To train the speech model of the special pronunciation units of the second language. The multi-lingual speech synthesis method of claim 9, further comprising: adding the speech model of the special pronunciation unit of the second language to the speech model of the first language.