TWI385644B - Singing voice synthesis method - Google Patents

Description

Singing sound synthesis method

The present invention relates to a vocal synthesis method.

With the rapid development of computer hardware technology, the relatively capable processing can be more and more complicated. Nowadays, speech synthesis has been widely used in various places, such as station name broadcast system on the bus, telephone voice system or cash machine. on.

However, the current common speech systems are mostly used to simulate speech, and for the synthesis of speech samples, most of them are simple combinations, so the synthesized speech sounds like a robot. Although in recent years, the problem of discontinuity of adjacent speech has been smoothed, so that the synthesized speech sounds smoother, but for the singing voice with pitch change, the aforementioned discontinuity problem is more obvious, and the aforementioned smoothing Processing is not enough to simulate the synthesized speech of approximate singing voice, especially when the length of a certain synthesized speech is different from the length of the speech sample, simply stretching or shortening the speech sample will make the synthesized speech sound less natural.

One aspect of the present invention is to provide a vocal synthesis method for synthesizing a plurality of speech samples into a vocal synthesized sound.

The singing voice synthesis method comprises the following steps: (a) extracting a song spectrum from a song document; (b) extracting the plurality of voice samples from a voice sample database according to the song score; (c) according to the song score, Determining a pitch curve related to the score; (d) calculating a harmonic parameter regarding the synthesized sound of the vocal sound according to the pitch curve and harmonic parameters of the plurality of voice samples; and (e) according to the The harmonic parameters of the vocal synthesis sound are calculated to calculate the vocal synthesis sound.

Step (b) is to add a plurality of lyrics before the song score, and then extract the plurality of voice samples corresponding to the plurality of lyrics from a voice sample database according to the plurality of lyrics. Step (c) further includes adjusting the pitch curve according to the curve change of the pitch curve, so that the adjusted pitch curve is more in line with the actual singer, and the vocal sound generated by the pitch change of the score High adjustment.

Wherein step (d) first performs Fourier transform on the voiced portion of the voice sample to obtain harmonic parameters of the voice sample, and then adjusts harmonic parameters of the plurality of voice samples according to the pitch curve to obtain the singing Harmonic parameters of the sonic synthesis sound. If the length of time of the speech sample is different from the length of time of the pitch curve corresponding to the speech sample, then the length of the delay segment of the voiced portion of the speech sample is adjusted. In addition, the frequency domain wave of the speech sample and the frequency domain wave system of the vocal synthesis sound corresponding to the speech sample are substantially the same, so that the vocal synthesis sound can still retain the timbre of the speech sample.

Step (e) firstly copying the corresponding silent part of the voice sample according to the harmonic parameter of the vocal synthesized sound; and then determining the harmonic parameter of the vocal synthesized sound of the corresponding voiced part of the voice sample The initial phase; then, according to the harmonic parameter of the vocal synthesized sound of the corresponding voice part of the voice sample, the time domain amplitude of the vocal synthesized sound corresponding to the voice sample is calculated in a phase superposition manner; Finally, the time domain amplitude is adjusted according to the corresponding volume, and thus the vocal synthesized sound can be formed.

Therefore, the vocal synthesis method of the present invention preserves the timbre of the original speech sample, and can adjust the pitch curve according to the change of the pitch curve, so that the synthesized vocal synth sound is in definition, naturalness and fluency. Both are significantly improved over the prior art.

The advantages and spirit of the present invention will be further understood from the following detailed description of the invention.

Referring to FIG. 1, FIG. 1 is a flow chart showing a method for synthesizing singing voice according to an embodiment of the present invention. The vocal synthesis method according to this embodiment mainly includes two major parts: one is the synthesis of human voice; the other is the synthesis of musical instrument sound. After the two are made, the mixing process is performed to make a WAV file (or other music file format) for subsequent playback.

As shown in Figure 1, in the part of the singing voice synthesis, the first song from a song A piece of music is captured, as shown in step S102. In general, the music score file may be a Musical Instrument Digital Interface (MIDI) file, and thus the music score extracted in this case is free of lyrics, but the present invention is not limited thereto. In order to make the subsequent work more convenient, the extracted scores can be expressed in the form of a table, as shown in Table 1 below. In order to facilitate the retrieval of subsequent speech samples, you can first fill in the lyrics in Table 1, such as "one" and "flash" in Table 1. Of course, if the lyrics already contain lyrics, the scores you have taken can contain The required lyrics are not required to be filled in separately.

In addition, the pitch in the score is expressed as an integer value, and the true frequency value can be calculated according to the twelve-average law shown in the following equation: F=F _c4 *2 ^(pitch)/12

Where F is the calculated actual frequency value, F _c4 is the frequency value of the center C, which is about 261.626 Hz, and pitch is the pitch value read from the score (table).

Then, according to the score, the pitch curve of the score is determined, as shown in step S104. According to the pitch and length of the score (that is, the duration of the pitch) The pitch curve directly derived from time) can be adjusted based on the fact that the singer changes the pitch of the actual singing voice due to the pitch change (ie, the curve of the pitch curve). Usually, when the current one is different from the pitch of the current sound, the pitch curve at the beginning of the current sound will have a rise and fall, or a fall and then rise. Therefore, the present invention proposes an adjustment rule as will be described later.

In the pitch curve, if a target pitch value is not equal to the next pitch value, the target pitch value is adjusted according to the rate of change of the next pitch value relative to the target pitch value. In this case, the pitch value of the target pitch value can be further divided into three adjustment rules. If the previous pitch value is lower relative to the pitch value of the target, it is adjusted according to the first adjustment rule shown in the following equation: Where 0≦x≦1, y is the adjustment multiple, p _i is the current pitch value of the target, and p _i+1 is the next pitch value. If the previous pitch value is the same as the pitch value of the target, it is adjusted according to the second adjustment rule shown in the following formula: If the previous pitch value is higher relative to the pitch value of the target, it is adjusted according to the third adjustment rule shown in the following equation:

Further, in the pitch curve, if the target pitch value is equal to the next pitch value, the target pitch value is adjusted by a predetermined multiple value. Similarly, in this case, the three pitch adjustment values can be further distinguished according to the previous pitch value of the target pitch value. If the previous pitch value is lower relative to the pitch value of the target, it is adjusted according to the first adjustment rule shown in the following equation: If the previous pitch value is the same as the pitch value of the target, it is adjusted according to the second adjustment rule shown in the following formula: If the previous pitch value is higher than the pitch value of the target, it is adjusted according to the third adjustment rule shown in the following formula:

The aforementioned adjustments are made based on the interaction of adjacent pitches, but usually the time difference between a certain tone and its next tone is greater than a certain At the degree, this one sound will not be affected by its next sound. Therefore, before the above rules are applied, it is necessary to judge whether the pitch value of the target and the next pitch value are different, and the corresponding time difference is greater than For a predetermined length of time, the predetermined length of time may be set to 0.9 seconds, but the present invention is not limited thereto. If the result of the determination is yes, the next pitch value is regarded as the same as the pitch value of the target, and the pitch curve is adjusted; that is, the latter rule is adjusted according to the latter three rules.

After the above adjustment of the pitch curve is completed, the following step S106 may be followed by or in parallel. As shown in step S106, the plurality of speech samples are retrieved from a speech sample database according to the plurality of lyrics filled in the score, and spectral analysis is performed to obtain harmonic parameters for each speech sample. Based on the characteristics of the human voice, a sound can observe the silent part and the sound part in the waveform of the time domain. The silent part is a non-periodic waveform, and it cannot be analyzed for harmonic parameters, so only the Fourier transform is performed on the voiced part. Since the voice sample is recorded by human voice, it also includes a silent part and a voiced part. The vocal part will be Fourier transformed to obtain the harmonic parameters of the speech sample; the unvoiced part is directly copied to the vocal synthesis.

Regarding the Fourier transform of a speech sample, the voiced part of a speech sample is first cut into a plurality of sound boxes of the same length according to its time domain waveform, each sound box has 256 sample points, and the voice sampling frequency is 22050 Hz, so each The length of the sound box is about 12ms. Two sound boxes (512 sample points) first Multiply a Hamming Window and perform a fast Fourier transform. After conversion, multiple sets of harmonic parameters are obtained. Each set of harmonic parameters includes amplitude (Amplitude), frequency (Frequency) and phase (Phase). The amplitude, frequency and phase of the fundamental frequency can be obtained by interpolation using Cubic Hermite Spline, and the recording is based on the same pitch, length and volume as the recorder is recording. Therefore, when recalculating the fundamental frequency, the search range can be set between 200 and 400 Hz, and the larger amplitude is the initial value. After the fundamental frequency related parameters are determined, the other harmonic search initial values can be determined according to the multiple relationship, and the three Hermitian splines are still used for the recursive calculation to obtain other harmonic parameters.

After obtaining the harmonic parameters of the speech sample and the pitch curve generated based on the music score, the harmonic parameters of the synthesized sound of the singing voice can be calculated according to the aforementioned harmonic parameters and the pitch curve, as shown in step S108. Since the length of time of each voice sample is substantially the same, the length of time required for the voice sample corresponding to the pitch curve is not the same, so the length of the voice sample needs to be adjusted.

In general, the voiced portion of the speech sample conforms to the ADSR model (as shown in Figure 2), which includes an Attack (A) segment, a Decay (D) segment, a Delay (Sustain, S) segment, and The section of the ending (R) section; when the person stretches the sound, it mainly lengthens the length of the delay section. Therefore, if the length of time of the speech sample is different from the length of time of the pitch curve corresponding to the speech sample, then the voiced portion of the speech sample is adjusted. The length of time for the delay segment. If the amplitude of the shortening adjustment is too large, the rising section, the falling section and the ending section also need to be shortened according to the ratio of the length of the synthesized sound to the voice sample. To facilitate the implementation of the above processing, when processing a speech sample, the length of time for each segment of the speech sample can be expressed in a tabular manner, as shown in Table 2 below. Wherein the number of sample points is the unit of the calculation time length, when the length of each speech sample is defined to be the same, the R segment (end segment) can be subtracted from the entire length by the silent portion, the AD segment (rising, falling segment) And the S section (delay section) is calculated and is not listed separately; however, the invention is not limited thereto.

However, it should be noted that the tone of the speech sample is mainly represented by its frequency domain envelope. Therefore, when adjusting the speech samples, the waveform shape should be kept the same after adjustment, even if the frequency domain of the speech sample is The frequency domain collinearity corresponding to the vocal synthesis sound corresponding to the speech sample is substantially the same, so that the adjusted synthesized sound still retains the timbre of the speech sample. In order to be able to accurately control the spectral waveform to remain the same, the Cubic Hermite Spline used in the previous spectrum analysis should be used as well. The main purpose of the Cubic Hermite Spline is to calculate the amplitude of the harmonic frequency of the synthesized sound. Find the frequency points at the left and right ends of the interval where the harmonic frequency of the synthesized sound is located to obtain the amplitude value of the frequency within the interval, that is, the amplitude value of the synthesized harmonic frequency.

In addition, if there is a transition sound in the score, that is, a speech sample corresponds to a plurality of pitch values, the corresponding speech sample needs to be segmented. That is, if the pitch curve corresponding to the speech sample includes a plurality of pitch values, the first pitch value of the plurality of pitch values corresponds to the rising segment of the voiced portion of the speech sample, and falls. The segment and part of the delay segment, the last pitch value corresponds to the end segment of the voiced portion of the speech sample and a portion of the delay segment. In the case of a turning sound (that is, there are two pitch values corresponding to the same lyric), the first pitch value corresponds to the rising section, the falling section, and the partial delay section, and the second pitch value That is, the delay section and the end section of the corresponding part. In the case of two transitions (ie, three pitch values corresponding to the same lyric), the first pitch value corresponds to the raised segment, the falling segment, and a portion of the delay segment, the second pitch The value is the delay segment for the portion, and the third pitch value is the delay segment and the end segment of the corresponding portion. In the case of more transitional sounds, it is analogous to the above, and will not be described again.

In addition, regarding the simulation of the vibrato effect, based on the actual singing, the vibrato is often used in the case where the same vocal is longer, so a predetermined length of time can be preset as a judgment as to whether or not the vibrating effect needs to be added. basis. That is, if the length of time corresponding to a pitch value is greater than the predetermined length of time in the pitch curve, a sine wave function is added to the harmonic parameter of the vocal synthesized sound corresponding to the pitch value. This predetermined length of time can be set to 1 second, but the present invention is not limited thereto.

Following step S108, the vocal synthesis sound is calculated based on the harmonic parameters of the vocal synthesized sound as shown in step S110. Since the silent portion of the speech sample is not subjected to Fourier transform, when the vocal synth sound is generated, the unvoiced portion is added in a copy manner. Then, determining an initial phase of the harmonic parameter of the vocal synthesized sound of the corresponding voice part of the voice sample; and then, according to the harmonic parameter of the vocal synthesized sound of the corresponding voice part of the voice sample, The harmonic equation and the phase superposition method are used to calculate the amplitude values of each sample point and superimpose them to obtain the time domain amplitude of the vocal synthesized sound corresponding to the speech sample. Finally, adjust the time domain amplitude based on the volume recorded in the score. Therefore, the vocal synthesis of a syllable is completed, and the vocal synthesis of other syllables is completed, and the complete vocal synthesis can be synthesized.

Since the actual singer is singing, the lyrics with lower pitch are always whispered relatively, that is, the volume will be lower, so in the aforementioned adjustment volume process (ie, adjusting the time domain amplitude), if corresponding When the pitch value of the domain amplitude is less than a predetermined pitch value, the time domain amplitude is adjusted by a predetermined multiple value. For example, the predetermined pitch value is the bass C, and the predetermined multiple value is 0.6; however, the present invention is not limited thereto. And if the pitch value corresponding to the time domain amplitude is equal to or greater than the predetermined pitch value, the time domain amplitude is linearly adjusted according to the pitch value. For example, adjust it by the following formula: Where vol is the multiple of the time domain amplitude value to be adjusted, and note is the pitch value read from the score (expressed as an integer value, for example, the center C is 0). According to the above description, the previously calculated time domain amplitude is adjusted, and the resulting vocal synthesized sound will be more close to the real singer, and the volume of the actual singing will change.

At this point, the complete vocal synthesis sound has been completed, but the background music needs to be synthesized separately (as shown in steps S112, S114 and S116), and then mixed with the vocal synthesis sound (as shown in step S118). After mixing, it can be directly stored in the WAV format (but the invention is not limited thereto), and then played by other players or playing software (as shown in step S120). The sound synthesis and mixing and playback programs of the instruments are well-known and will not be described again. In addition, many of the coefficients in the above formulas are examples, which can be adjusted according to the actual measurement situation.

In summary, the vocal synthesis method of the present invention utilizes maintaining the frequency domain wave of the original speech sample so that after adjusting the pitch, the tone of the original speech sample can still be maintained, and the tone is adjusted according to the change of the pitch curve itself. The high curve can further simulate the pitch changes made by real people when they sing due to the difference in tunes. In addition, for the transition sound, the vibrato, and the like, the vocal synthesis method of the present invention also proposes a simulation method, so that the synthesized singer synthesized sound has obvious improvement in sharpness, naturalness and fluency compared with the prior art. . If more speech samples are created on the speech samples, such as speech samples of different pitches and different timbres (adults, children, boys, girls, etc.), the speech samples can be more flexible and can be combined to be more consistent. Real person The singing sings are synthesized.

The features and spirit of the present invention will be more apparent from the detailed description of the preferred embodiments. On the contrary, the intention is to cover various modifications and equivalents within the scope of the invention as claimed.

S102~S120‧‧‧Steps

1 is a flow chart showing a method for synthesizing singing voice according to an embodiment of the present invention.

Figure 2 is a schematic diagram showing each section of the ADSR model.

S102~S120‧‧‧Steps

Claims (17)

A vocal synthesis method for synthesizing a plurality of speech samples into a vocal synthesis sound, the vocal synthesis method comprising the steps of: (a) extracting a score from a slang file; (b) according to the score Determining a pitch curve related to the score; (c) extracting the plurality of voice samples from a phonetic sample database according to the score; (d) according to the pitch curve and the harmonic of the plurality of voice samples a wave parameter to calculate a harmonic parameter related to the vocal synthesized sound; and (e) calculating the vocal synthesized sound based on a harmonic parameter regarding the vocal synthesized sound. The singing voice synthesis method according to claim 1, wherein the step (c) is performed by: adding a plurality of lyrics to the music score; and extracting from a voice sample database according to the plurality of lyrics The plural speech samples of the lyrics should be plural. The vocal synthesis method according to claim 1, wherein the step (b) further comprises the step of: (b1) adjusting the pitch curve according to a curve change of the pitch curve. The singing voice synthesizing method according to claim 3, wherein the step (b1) is implemented by the following steps: in the pitch curve, if a target pitch value is not equal to the next pitch a value, the pitch value of the target is adjusted according to a rate of change of the next pitch value relative to the pitch value of the target, and if the target pitch value is equal to the next pitch value, the predetermined pitch value is a predetermined multiple The value adjusts the pitch value of the target. The vocal synthesis method according to claim 4, wherein in the step (b1), if the time difference between the target pitch value and the next pitch value is greater than a predetermined length of time, The predetermined multiplier value adjusts the pitch value of the target. The singing voice synthesizing method according to claim 5, wherein the predetermined length of time is 0.9 seconds. The singing voice synthesizing method according to claim 1, wherein each of the voice samples includes a silent portion and a voiced portion, and the step (d) is performed by the following steps: (d1) for the voiced portion of the voice sample Performing Fourier transform to obtain harmonic parameters of the speech sample; and (d2) adjusting harmonic parameters of the plurality of speech samples according to the pitch curve to obtain harmonic parameters regarding the vocal synthesized sound. The singing voice synthesis method according to claim 7, wherein the step (d1) comprises calculating a harmonic parameter of the voice sample by using a cubic Hermite spline. The singing voice synthesizing method according to claim 7, wherein the voiced portion of the voice sample comprises a rising section, a falling section, a delay section and an ending section, in step (d2) And if the length of time of the speech sample is different from the length of time of the pitch curve corresponding to the speech sample, adjusting the length of the delay segment of the voiced portion of the speech sample. The singing voice synthesis method according to claim 9, wherein in the step (d2), if the pitch curve corresponding to the voice sample contains a plurality of sounds When the value is high, the first pitch value of the plurality of pitch values corresponds to the rising section, the falling section, and the partial delay section of the voiced portion of the voice sample, and the last pitch value is The end segment of the voiced portion of the speech sample and the delayed segment of the portion. The singing voice synthesizing method according to claim 7, wherein in the step (d2), the frequency domain wave of the voice sample is substantially the same as the frequency domain wave of the voice sample corresponding to the voice sample. The singing voice synthesizing method according to claim 7, wherein the step (d2) comprises calculating a harmonic parameter of the singing synthesized sound by using a cubic Hermite spline. The vocal synthesis method of claim 7, wherein the step (d2) further comprises: if the length of the corresponding pitch value is greater than a predetermined length of time in the pitch curve, then A sine wave function is added to the harmonic parameter of the vocal synthesis sound of the pitch value. The singing voice synthesizing method according to claim 7, wherein the song spectrum includes a plurality of sounds corresponding to the pitch curve, the step (d) includes determining the silent portion of the voice sample, and the step (e) The following steps are performed: copying the corresponding silent part of the voice sample according to the harmonic parameter of the vocal synthesized sound; determining the initial phase of the harmonic parameter of the vocal synthesized sound of the corresponding voiced part of the voice sample And calculating a time domain amplitude of the vocal synthesized sound corresponding to the voice sample according to a harmonic parameter of the vocal synthesized sound of the corresponding voice part of the voice sample; and according to the corresponding volume To adjust the time domain amplitude. Such as the singing voice synthesis method described in claim 14 of the patent scope, wherein the step The step (e) further includes the step of: adjusting the time domain amplitude by a predetermined multiple value if the pitch value corresponding to the time domain amplitude is less than a predetermined pitch value; and if the time domain amplitude sound is corresponding When the high value is equal to or greater than the predetermined pitch value, the time domain amplitude is linearly adjusted according to the pitch value. The singing voice synthesizing method according to claim 15, wherein the predetermined pitch value is a low frequency C, and the predetermined multiple value is 0.6. The vocal synthesis method according to claim 1, wherein the harmonic parameter of the speech sample or the harmonic parameter of the vocal synthesis sound comprises a frequency, an amplitude and a phase of the complex array.