EP1505570B1 - Méthode de synthèse de voix chantée - Google Patents

Méthode de synthèse de voix chantée Download PDF

Info

Publication number
EP1505570B1
EP1505570B1 EP03017548.3A EP03017548A EP1505570B1 EP 1505570 B1 EP1505570 B1 EP 1505570B1 EP 03017548 A EP03017548 A EP 03017548A EP 1505570 B1 EP1505570 B1 EP 1505570B1
Authority
EP
European Patent Office
Prior art keywords
spectrum
data
voice
amplitude
frequency
Prior art date
Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
Expired - Fee Related
Application number
EP03017548.3A
Other languages
German (de)
English (en)
Other versions
EP1505570A1 (fr
Inventor
Hideki Kenmochi
Jordi Bonada
Alex Loscos
Current Assignee (The listed assignees may be inaccurate. Google has not performed a legal analysis and makes no representation or warranty as to the accuracy of the list.)
Yamaha Corp
Original Assignee
Yamaha Corp
Priority date (The priority date is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the date listed.)
Filing date
Publication date
Application filed by Yamaha Corp filed Critical Yamaha Corp
Priority to EP03017548.3A priority Critical patent/EP1505570B1/fr
Publication of EP1505570A1 publication Critical patent/EP1505570A1/fr
Application granted granted Critical
Publication of EP1505570B1 publication Critical patent/EP1505570B1/fr
Anticipated expiration legal-status Critical
Expired - Fee Related legal-status Critical Current

Links

Images

Classifications

    • G—PHYSICS
    • G10—MUSICAL INSTRUMENTS; ACOUSTICS
    • G10L—SPEECH ANALYSIS TECHNIQUES OR SPEECH SYNTHESIS; SPEECH RECOGNITION; SPEECH OR VOICE PROCESSING TECHNIQUES; SPEECH OR AUDIO CODING OR DECODING
    • G10L13/00—Speech synthesis; Text to speech systems
    • G10L13/02—Methods for producing synthetic speech; Speech synthesisers
    • G10L13/033—Voice editing, e.g. manipulating the voice of the synthesiser
    • G—PHYSICS
    • G10—MUSICAL INSTRUMENTS; ACOUSTICS
    • G10L—SPEECH ANALYSIS TECHNIQUES OR SPEECH SYNTHESIS; SPEECH RECOGNITION; SPEECH OR VOICE PROCESSING TECHNIQUES; SPEECH OR AUDIO CODING OR DECODING
    • G10L21/00—Speech or voice signal processing techniques to produce another audible or non-audible signal, e.g. visual or tactile, in order to modify its quality or its intelligibility
    • G10L21/02—Speech enhancement, e.g. noise reduction or echo cancellation
    • G10L21/0208—Noise filtering
    • G10L21/0216—Noise filtering characterised by the method used for estimating noise
    • G10L21/0232—Processing in the frequency domain

Definitions

  • This invention relates to a singing voice synthesizing method, a singing voice synthesizing apparatus and a storage medium by using a phase vocoder technique.
  • FIG. 21 shows a singing voice synthesizing apparatus adopting the technique explained in Japanese Patent No. 2906970 .
  • a singing voice signal is input, and at Step S2, a SMS analyzing process and a section logging process is executed to the input singing voice signal.
  • the input voice signal is divided into a series of time frames, and one set of a magnitude spectrum data is generated in each frame by Fast Fourier Transform (FFT) and the like, and a linear spectrum corresponding to plurality of peaks from one set of magnitude spectrum data by each frame.
  • a data representing an amplitude value and frequency of these linear spectrums are called a Deterministic Component.
  • a spectrum of the deterministic component is subtracted from a spectrum of an input voice waveform to obtain a remaining difference spectrum. This remaining difference spectrum is called Stochastic Component.
  • the voice synthesis unit is a structural element of lyrics.
  • a voice synthesis unit is consisted of a single phoneme such as [a] or [i] or, a phonemic chain (a chain of a plurality of phonemes) such as [a_i] or [a_p].
  • a deterministic component data and stochastic component data are stored for every voice synthesis unit.
  • lyrics data and melody data are input.
  • Step S4 a phonemic series/voice synthesis unit conversion process is executed on the phonemic series that the lyrics data represents to divide the phonemic series into a voice synthesis unit.
  • the deterministic component data and the stochastic component data are read from the database DB as a voice synthesis unit data for every voice synthesis unit.
  • a voice synthesis unit connecting process is executed on the voice synthesis unit data (the deterministic component data and the stochastic component data) read from the database DB to connect voice synthesis unit data in an order of pronunciations.
  • new deterministic component data adapting to the musical note pitch is generated based on the musical note pitch that the deterministic component data and the melody data indicate for every voice synthesis unit.
  • a spectrum intensity is adjusted to be a form of a spectrum envelope that the deterministic component data processed at Step S5 is taken over, a musical tone of the voice signal input at Step S1 can be reproduced with the new deterministic component data.
  • Step S7 the deterministic component data generated at Step S6 is added to the stochastic component data executed the process at Step S5 in every voice synthesis unit. Then, at Step S8, the data to which the adding process is executed at Step S7 is converted to a synthesized voice signal of time region by a reverse FFT and the like in each voice synthesis unit.
  • voice syntheses units corresponding to voice synthesis units [#s], [s_a], [a], [a_i], [l], [i_t], [a], and [a#] (# represents a silence) are read from the database DB, and they are connected each other at Step S5. Then, at Step S6, a deterministic component data that has a pitch corresponding to the input musical note pitch is generated in each voice synthesis unit. After the adding process at Step S7 and the converting process at Step S8, a singing voice signal of [saita] can be obtained.
  • the singing voice is caught as an artificial voice because the voice signal pitch input at Step S1 is converted corresponding to the input musical note pitch at Step S6 and the stochastic component data is added to the deterministic component data with the converted pitch at Step S7.
  • the stochastic component data is sounded being split in a section of a long sound such as [i] in singing [saita].
  • the inventors of the present invention suggested that an amplitude spectrum distribution in a lower region that the stochastic component data represents is adjusted corresponding to the input musical note pitch before (refer to Japanese Patent Application No. 2000-401041 ). However, if the stochastic component data is adjusted as above, it is not easy to control splitting and resounding of the stochastic component completely.
  • the SMS technique is on the assumption that a voice signal is consisted of a deterministic component and a stochastic component, and it is a fundamental problem that the voice signal cannot be split into the deterministic component and the stochastic component as the SMS technique.
  • phase vocoder technique is explained in a specification of the U.S. Patent No. 3360610 .
  • a signal was represented by a filter bank before and recently has been represented by a frequency region as a result of the FFT of input signal.
  • the phase vocoder technique is widely used for a time-stretch (stretching or shortening of a time axis without changing the original pitch), a pitch-shift (changing a pitch without changing the time length) and the like.
  • this kind of pitch changing technique the result of FFT of the input signal is not used as it is.
  • the pitch-sift is executed by moving the spectrum distribution on a frequency axis in each spectrum distribution region after dividing the FFT spectrum into a plurality of spectrum distribution regions centered at a local peak.
  • J. Laroche and M. Dolson "New Phase-Vocoder Techniques for Real-Time Pitch Shifting, Chorusing, Harmonizing, and Other Exotic Audio Modifications" J. Audio Eng. Soc., Vol. 47, No.
  • the prior art fails to disclose adjustment of spectrum intensity of a spectrum distribution region including the local peak. Further, the prior art requires a high computational load for obtaining an adjustment.
  • a voice waveform corresponding to a voice synthesis unit (a phoneme or a phonemic chain) is executed a frequency analysis, and a frequency spectrum is detected. Then an amplitude spectrum data and a phase spectrum data are generated based on the frequency spectrum.
  • a desired pitch is designated
  • the amplitude spectrum data and the phase spectrum data are adjusted corresponding to the designated pitch, and a synthesized voice signal in a time region is generated based on the adjusted amplitude spectrum data and the adjusted phase spectrum data.
  • voice synthesizing is executed without splitting the result of the frequency analysis of the voice waveform into a deterministic component and a stochastic component, the stochastic component may not split and resound. Therefore, a natural synthesized sound can be obtained. Also, a natural synthesized sound can be obtained in a case of a voiced fricative or plosive sound.
  • the second singing voice synthesizing method corresponds to the case that the amplitude spectrum data and the phase spectrum data are stored in a database in each voice synthesis unit after executing the processes up to the step of generating the phase spectrum data, or the case that the process up to the step of generating the phase spectrum data is executed with other apparatus. That is, in the second singing voice synthesizing method, in a obtaining step, the amplitude spectrum data and the phase spectrum data corresponding to the voice synthesis unit of the voice to be synthesized are obtained from other apparatus or the database, and a process after the step to designate pitch is executed in the same method as the first singing voice synthesizing method. Therefore, according to the second singing voice synthesizing method, a natural synthesized sound can be obtained as the first singing voice synthesizing method.
  • the first and second singing voice synthesizing apparatuses are to execute the before-described first and second singing voice synthesizing methods by using the voice synthesis unit database, and a natural singing voice synthesized voice can be obtained.
  • the third and the fourth singing voice synthesizing apparatuses are to execute the before described first or second singing voice synthesizing methods by using the voice synthesis unit database, and can obtain a natural singing voice synthesized sound. Moreover, spectral intensities and phases at a connecting part of the sequential voice synthesis units are adjusted to be the same or approximately the same to each other at the time of connecting the amplitude spectral data and the phase spectral data to be modified for connecting the voice synthesis units in the order of the pronunciation; therefore, it is prevented to generate noise at the time of generating the synthesized voice.
  • amplitude spectrum data and phase spectrum data are generated based on a result of a frequency analyzing of a voice waveform corresponding to a voice synthesis unit, and the amplitude spectrum data and the phase spectrum data are adjusted corresponding to a designated pitch. Then, since a synthesized voice signal in a time region is generated based on the adjusted amplitude spectrum data and the adjusted phase spectrum data, a situation that the stochastic component splits and resounds as the conventional example that the result of the frequency analysis is split into the deterministic component and the stochastic component will not occur principally, and an effect that enables a natural or high-quality singing voice synthesizing can be obtained.
  • FIG. 1 is a block diagram showing a circuit structure of a singing voice synthesizing apparatus according to an embodiment of the present invention.
  • This singing voice synthesizing apparatus has a structure wherein a small computer 10 controls operations.
  • a central processing unit (CPU) 12, a read only memory (ROM) 14, a random access memory (RAM) 16, a singing voice input unit 17, a lyrics/melody input unit 18, a control parameter input unit 20, an external storage unit 22, a displaying unit 24, a timer 26, a digital/analogue (D/A) conversion unit 28, a musical instrument digital interface (MIDI) interface 30, a communication interface 32 and the like are connected to a bus 11.
  • the CPU executes various kinds of processes related to the singing voice synthesizing and the like according to a program stored in the ROM 14.
  • the processes related to the singing voice synthesizing are explained later referring FIGs. 2 to 7 and the like.
  • the RAM 16 includes various kinds of storing regions such as a working region at a time of various processes in the CPU 12.
  • storing regions for example, inputting data storing regions are respectively corresponding to the input units 17, 18 and 20. The details will be explained later.
  • the singing voice input unit 17 has a microphone, a voice inputting terminal and the like for inputting a singing voice signal, and equips an analogue/digital (A/D) conversion device that converts the input singing voice signal to a digital waveform data.
  • the digital waveform data to be input is stored in a predetermined region in the RAM 16.
  • the lyrics/melody input unit 18 equips a keyboard that can input letters and numbers, and a reading device that can read scores. It can input a melody data that represents a series of musical notes (including rest) that consists a lyrics data and melody that represents a phonemic series that consists the lyrics of a desired singing voice.
  • the lyrics data and the melody data to be input are stored in a predetermined region in the RAM 16.
  • the control parameter input unit 20 equips parameter setting devices such as a switch, a volume and the like, and can set a control parameter for controlling a musical expression of the singing voice synthesized voice.
  • a musical tone, a pitch classification (high, middle, low, etc.), a pitch throb (a pitch bend, vibrato, etc.), dynamics classification (high, middle, low, etc. of a volume level), a tempo classification (fast, middle, slow, etc. tempo) and the like can be set as the control parameter.
  • the control parameter data that represents a control parameter to be set is stored in a predetermined region in the RAM 16.
  • the external storage unit 22 includes one or plural kinds of removable storing mediums such as a flexible disk (FD), a compact disk (CD), a digital versatile disk (DVD), a magneto optical disk (MO) and the like. Data can be transmitted from the storing medium to the RAM 16 in a state that a desired storing medium is loaded into the external storage unit 22. If a loaded medium is writable such as HD and FD, data in the RAM 16 can transmit to the storing medium.
  • a flexible disk FD
  • CD compact disk
  • DVD digital versatile disk
  • MO magneto optical disk
  • a storing medium of the external storage unit can be used instead of the ROM 14.
  • the program stored in the storing medium is transmitted from the external storage unit 22 to the RAM 16.
  • the CPU is executed operations according to the program stored in the RAM 16.
  • the displaying unit 24 includes a displaying device such as a liquid crystal displaying device, and can display various kids of information such as the above-described results of the frequency analysis and the like.
  • the timer 26 generates a tempo clock signal TCL in a cycle corresponding to a tempo that a tempo data TM designates, and the tempo clock signal TCL is provided to the CPU 12.
  • the CPU 12 executes a signal outputting process to the D/A conversion unit 28 based on the tempo clock signal TCL.
  • a tempo that the tempo data TM designates can be set flexibly by a tempo setting device in an input unit 20.
  • the D/A conversion unit 28 converts a synthesized digital voice signal to an analogue voice signal.
  • the analogue voice signal transmitted from the D/A conversion unit 28 is converted to audio sound by a sound system 34 including an amplifier, a speaker, etc..
  • the MIDI interface 30 is provided for executing a MIDI communication to a MIDI device 36 that is separate from this singing voice synthesizing apparatus, and is used for receiving data for singing voice synthesizing from the MIDI device 36 in the present invention.
  • a data for singing voice synthesizing a lyrics data and a melody data corresponding to a desired singing voice, a control parameter data for controlling a musical expression and the like can be received.
  • These data for singing voice synthesizing are formed according to what is called a MIDI format, and the MIDI format may preferably be adapted for the lyrics data and the melody data input from the input unit 18 and the control parameter data input from the input unit 20.
  • a MIDI system exclusive data which a manufacturer can define on its own form will be preferable for enabling the data to be read before other data.
  • a singer (or a musical tone) designating data may be used in a case that the voice synthesis unit data is stored in a later-described database by each singer (or each musical tone).
  • program change data of MIDI can be used.
  • the communication interface 32 is provided for data communication to other computer 38 via the communication network (for example, local area network (LAN), the Internet, and a telephone line) 37.
  • the programs and various kinds of data necessary for executing the present invention may be loaded from the computer 38 to the RAM 16 or the external storage unit 22 via the communication network 37 and the communication interface 32 according to a downloading demand.
  • the singing voice signal is input from the input unit 17 via the microphone or the voice inputting terminal to execute the A/D conversion, and the digital waveform data that represents the voice waveform of the input signal is stored in the RAM 16.
  • FIG. 8A shows an example of the input voice waveform. Moreover, in FIG. 8A and other figures, "t" represents time.
  • a section waveform is logged at each section corresponding to each voice synthesis unit (phoneme or phonemic chain) for the digital waveform data to be stored (the digital waveform data is divided).
  • the voice synthesis unit there are a vowel phoneme, a phonemic chain of vowel and consonant or consonant and vowel, phonemic chain of consonant and consonant, phonemic chain of vowel and vowel, a phonemic chain of silence and consonant or vowel, a phonemic chain of vowel or consonant and silence and the like.
  • a vowel phoneme there is a long sound phoneme that is sung by lengthening a vowel.
  • a section waveform is logged corresponding to each of [#s], [s_a], [a], [a_i], [I], [i_t], [t_a], [a] and [a#].
  • one or plurality of time frame(s) is fixed by each section waveform, a frequency spectrum (an amplitude spectrum and a phase spectrum) are detected by executing the frequency analysis for each frame by the FFT and the like. Then, the data that represents the frequency spectrum is stored in a predetermined region in the RAM 16.
  • a frame length may be a certain length or a flexible length. To make the frame length a flexible length, after the frequency analysis of one frame as a fixed length, a pitch is detected from the result of the frequency analysis, and a frame length corresponding to the detected pitch is set, and the frequency analysis can be executed on the frame again.
  • a pitch is detected from the result of the frequency analysis, a next frame length is set corresponding to the detected pitch, and the frequency analysis of the next frame can be executed.
  • the number of frames will be one frame or plurality of frames for a single phoneme consisted of only vowel, and will be plurality of frames for the phonemic chain.
  • FIG. 8B the frequency spectrum obtained by the frequency analysis of the voice waveform in FIG. 8A by the FFT is shown.
  • "f" represents frequency.
  • a pitch is detected based on the amplitude spectrum by voice synthesis unit, and a pitch data that represents a detected pitch is generated to store in a predetermined region in the RAM 16.
  • the pitch detection can be executed by an averaging method of all frames of the pitches obtained by each frame.
  • Step 48 plurality of local peaks of a spectrum intensity (amplitude) on the amplitude spectrum are detected by each frame.
  • a method wherein a peak whose amplitude value is the maximum is detected from the next plurality (for example, 4) peaks can be used.
  • the detected plurality of local peaks P1, P2, P3... are indicated.
  • a spectrum distribution region corresponding to each local peak by each frame on the amplitude spectrum is designated, and an amplitude spectrum data represents the amplitude spectrum distribution in the region depending on the frequency axis to store in a predetermined region in the RAM 16.
  • a method for designating the spectrum distribution region there are a method wherein each half of the frequency axes cut between two adjacent local peaks are assigned to a spectral distribution region including the local peak closer to the half and a method wherein the bottom where the amplitude is the lowest is found between the adjacent two local peaks, and the frequency of the bottom is used as a boundary of the adjacent spectrum distribution regions.
  • FIG. 8B shows an example of the former method wherein the spectrum distribution regions R 1 , R 2 , R 3 ... are respectively assigned to the local peaks P 1 , P 2 , P 3 ...
  • a phase spectrum data that represents the phase spectrum distribution in each spectrum distribution depending on the frequency axis by each frames based on the phase spectrum is generated, and it is stored in a predetermined region in the RAM 16.
  • FIG. 10A the amplitude spectrum distribution and the phase spectrum distribution in one frame in one spectrum distribution region are respectively shown with curves AM 1 and PH 1 .
  • a pitch data, an amplitude spectrum data and a phase spectrum data are stored in a voice synthesis unit database by each voice synthesis unit.
  • the RAM 16 or the external storage device 22 can be used as a voice synthesis unit database.
  • FIG. 3 shows an example of a state of a storing in a voice synthesis unit database DBS.
  • Voice synthesis unit data each corresponding to a single phoneme such as [a], [i], etc.
  • voice synthesis unit data each corresponding to a phonemic chain such as [a_i], [s_a], etc. are stored in the database DBS.
  • the pitch data, the amplitude spectrum data and the phase spectrum data are stored as voice synthesis unit data.
  • a natural (or high quality) singing voice can be synthesized.
  • voice synthesis unit data M1, M2 and M3 respectively corresponding to the tempo classifications "slow”, “middle”, and “fast” while the pitch classification is “low” and the dynamics classification is “small” are recorded by having a singer A sing in all the combination of the tempo classifications "slow”, “middle”, “fast”, the pitch classifications "high”, “middle”, “low” and the dynamics classifications "large”, “middle”, “small”.
  • the voice synthesis unit data corresponding to the other combinations are recorded in the same way.
  • the pitch data generated at Step 46 is used when it is judged to which of "low", “middle” and “high” of the pitch classification the voice synthesis unit data is belonging.
  • a multiplicity of the voice synthesis units are recorded in the database DBS with different pitch classifications, dynamics classifications and pitch classifications by having the singer B sing similar to the above described singer A. Also, voice synthesis units other than [a] are recorded in the same manner as described above.
  • the singing voice signal can be input via the interface 30 or 32, and the voice synthesis unit data can be generated in accordance with the input voice signal.
  • the database DBS can be stored not only in the RAM 16 or the external storage unit 22 but also in the ROM14, a storage unit of the MIDI device 36, a storage unit of the computer 38, etc.
  • FIG. 4 shows an example of a singing voice synthesizing process.
  • lyrics data and melody data for a desired song are input from the input unit 18 and are-stored into the RAM 16.
  • the lyrics data and the melody data can be also input via the interface 30 or 32.
  • a phoneme series corresponding to the input lyrics data is converted into individual voice synthesis units.
  • voice synthesis unit data pitch data, amplitude spectrum data and phase data
  • a tone color, a pitch classification, a dynamics classification, a tempo classification, etc. may be input from the input unit 20 as control parameters, and voice synthesis unit data corresponding to the control parameters directed by the data.
  • duration of the pronunciation of the voice synthesis unit is corresponding to the number of the voice synthesis unit data. That is, when the voice synthesizing is executed by using the voice synthesis unit data to be stored without changing, the duration of the pronunciation corresponding to the number of the voice synthesis unit data can be obtained.
  • the duration of the pronunciation may be inappropriate depending on a duration of the musical note (an input musical note length), a set tempo and the like, and changing the duration of pronunciation will be necessary.
  • the number of read frames of the voice synthesis unit data may be controlled corresponding to the input note length, the set tempo and the like.
  • the voice synthesis unit data is read skipping a part of frames.
  • voice synthesis unit data is red repeatedly.
  • the duration of the pronunciation tends to be changed. Synthesizing the long sound is explained later with reference to FIGs. 14 to 16 .
  • the amplitude spectrum data of each frame is adjusted corresponding to a pitch of the input musical note corresponding to each voice synthesis unit. That is, the amplitude spectrum distribution that the amplitude spectrum data represents by each spectrum distribution region is moved on the frequency axis to be a pitch corresponding to the input musical note pitch.
  • FIGs. 10A and 10B show an example of moving the spectrum distribution region AM 1 to the region AM 2 for rising the pitch of the spectrum distribution region with a local peak frequency of fi and the lower and the upper limit frequencies are f l and f u .
  • the lower limit frequency F i and the upper limit Fu are decided corresponding to each frequency difference "f i -f i " and "f u -f i ".
  • FIG. 9A shows the spectrum distribution regions R 1 , R 2 , R 3 (same as shown in FIG. 8B ) respectively having the local peaks P 1 , P 2 , P 3
  • FIG. 9B shows an example of moving the spectrum distribution regions toward the higher note in a direction of the frequency axis.
  • the frequency, the lower limit frequency f 11 and the upper limit frequency f 12 of the local peak P 1 are decided as same as the same method with reference to FIG. ) 10 described in the above. It also can be applied to other spectrum distribution region.
  • the spectrum distribution is moved toward the higher pitch side on the frequency axis to rise the pitch, it can be moved toward the lower pitch side on the frequency axis to lower the pitch.
  • two spectrum distribution regions Ra and Rb are partly overlapped as shown in FIG. 11 .
  • the local peak Pb and the spectrum distribution region Pb that has a lower limit frequency f b1 (f b1 ⁇ f a2 ) and ) the upper limit frequency f b2 (f b2 >f a2 ) to the spectrum distribution region Ra are overlapped in frequency regions f a1 to f a2 .
  • the frequency regions f b1 to f a2 are divided into two by a central frequency, the upper frequency f a2 in the region Ra is converted to a predetermined frequency that is lower than the f c , and the lower frequency f b1 in the region Rb is converted to a predetermined frequency that is higher than the f c .
  • a spectrum distribution AMa can be used in a frequency region that is lower than the f c
  • a spectrum distribution AMa can be used in a frequency region that is higher than the f c .
  • the spectrum envelope stretches and shortens only by changing the frequency setting, and a problem that the musical tone is different from the input voice waveform arises.
  • the spectrum intensity of local peaks of one or plurality of the spectrum distribution region is adjusted along with the spectrum envelope corresponding to a linked line with the local peaks of a series of spectrum distribution region by each frame.
  • FIG. 12 shows an example of the spectrum intensity adjustment
  • FIG. 12A shows a spectrum envelope EV corresponding to local peaks P 11 to P 18 before the pitch-shift.
  • the spectrum intensity is increased or decreased to be along with the spectrum envelope to the spectrum envelope EV at a time that the local peaks P 11 to P 18 are moved on the frequency axis as shown in P 21 to P 28 in FIG. 12B .
  • a musical tone that is same as the input voice waveform can be obtained.
  • Rf is a frequency region lacked with the spectrum envelope.
  • the pitch is raised, transferring the local peaks such as P 27 , P 28 and the like to the frequency region Rf as shown in FIG. 12B may be necessary.
  • the spectrum envelope of the frequency region Rf is obtained by an interpolation method as shown in FIG. 12B , and the spectrum intensity of the local peak is adjusted according to the obtained spectrum envelope EV.
  • the spectrum envelope is preferably expressed with a curve or a straight line.
  • FIG. 13 shows two kinds of spectrum envelope curves EV 1 and EV 2 .
  • the curve EV 1 simply expresses the spectrum envelope with a line graph by linking each of local peaks by a straight line.
  • the curve EV 2 expresses the spectrum envelope by a cubic spline function. When the curve EV 2 is used, the interpolation can accurately be executed..
  • the phase spectrum data is adjusted by each voice synthesis unit corresponding to the adjustment of the amplitude spectrum data of each frame. That is, in a spectrum distribution region that includes ith local peak in a frame as shown FIG. 10A , a phase spectrum distribution PH 1 is corresponding to an amplitude distribution AM 1 .
  • the amplitude spectrum distribution AM 1 is moved as AM 2 at Step 66, it is necessary that the phase spectrum distribution PH 1 is adjusted corresponding to the amplitude spectrum distribution AM 2 . This is for making the phase spectrum distribution PH 1 a sine wave at a frequency at a local peak of a target place of the moving.
  • a phase interpolation amount ⁇ 1 related to the spectrum distribution region that contains ith local peak is provided with a following equation A1.
  • ⁇ i 2 ⁇ f i T ⁇ 1 ⁇ t
  • the interpolation amount ⁇ i that is obtained by the equation A1 is added to a phase of each phase spectrum in the regions F i to F u as shown in FIG. 10B , and the phase at a frequency F i of the local peak is ⁇ i + ⁇ i .
  • phase interpolation as described in the above is executed for each spectrum distribution region.
  • the fundamental frequency of the input voice that is, the pitch that the pitch data in the voice synthesis unit data represents
  • ⁇ i 2 ⁇ f o k T ⁇ 1 ⁇ t
  • a reproduction starting time is decided corresponding to the set tempo and the like by each voice synthesis unit.
  • the reproduction starting time depends on the set tempo and the input musical note length and can be represented with a clock count of the tempo clock signal TCL.
  • the reproduction starting time of the voice synthesis unit [s_a] is set in order to start [a] other than [s] at a note-on time that is decided by the input musical note length and the set tempo.
  • the lyrics data and the melody are input on real time base.
  • the lyrics data and the melody data are input before the note-on time in order to be possible to set the reproduction starting time described in the above.
  • a spectrum intensity level can be adjusted between the voice synthesis units.
  • This level adjustment process is executed for both of the amplitude spectrum data and the phase spectrum data, and it is executed for preventing a noise generated at a time of a synthesizing voice generation with a data connection at a next Step 74.
  • the amplitude spectrum data are connected to each another, and the phase spectrum data are connected to each another. Then, at Step 76, the amplitude spectrum data and the phase spectrum data are converted to a synthesized voice signal (a digital waveform data) of the time region by each voice synthesis unit.
  • FIG. 5 shows an example of a conversion process at Step 76.
  • a reverse FFT process is executed on the frame data (the amplitude spectrum data and the phase spectrum data) of the frequency region to obtain the synthesized voice signal of the time region.
  • a windowing process is executed on the synthesized voice signal of the time region. In this process, a time windowing function is multiplied on the synthesized voice signal of the time region.
  • an overlapping process is executed on the synthesized voice signal of the time region. In this process, the synthesized voice signal of the time region is connected by overlapping the waveform of the voice synthesis unit in an order.
  • the synthesized voice signal is output to the D/A converting unit 28 referring to the reproduction starting time decided at Step 78.
  • the singing voice is generated to be synthesized from the sound system 34.
  • FIG. 6 shows another example of the singing voice analyzing process.
  • the singing voice signal is input as same way as that is described before with reference to Step 40, and the digital waveform data that represents the voice waveform of the input signal is stored in the RAM 16.
  • the singing voice signal may be input via the interface 30 or 32.
  • Step 82 a section waveform is logged by each section corresponding to the voice synthesis unit for the digital waveform data to be stored as same way as that is described before with reference to Step 42.
  • a section waveform data (the voice synthesis unit data) that represents the section waveform by each voice synthesis unit is stored in the voice synthesis unit database.
  • the RAM 16 and the external storage unit 22 can be used as the voice synthesis unit database, and
  • the ROM 14, a storing device in the MIDI device 36 and the storing device in the computer 38 may be used depending on a request.
  • section waveform data m1, m2, m3... which are different in the singer (the musical tone), the pitch classification, the dynamics classification and the tempo classification by each voice synthesis unit can be stored in the voice synthesis unit database DBS as same way as that is described before with reference to FIG. 3 .
  • Step 90 the lyrics data and the melody data corresponding to the desired singing voice are input as same way as that is described before with reference to Step 60.
  • the phonemic series that the lyrics data represents is converted to individual voice synthesis unit as same way as that is described before with reference to Step 62.
  • the section waveform data (the voice synthesis unit data) corresponding to each voice synthesis unit is read from the database that is executed the storing process at Step 84.
  • data such as the musical tone, the pitch classification, the dynamics classification and the tempo classification are input as a control parameter from the input unit 20, and the section waveform data corresponding to the control parameter that the data instructs may be read.
  • the duration of pronunciation of the voice synthesis unit may be changed corresponding to the input musical note length and the set tempo as same way as that is described before with reference to Step 64. For doing this, when the voice waveform is read, reading the voice waveform may be continued only for a desired duration of pronunciation by omitting a part of the voice waveform or repeating a part or whole of the voice waveform.
  • Step 96 one or plurality of time frames are decided for the section waveform by each section waveform data to be read, and the frequency analysis is executed by each frame by the FFT and the like to detect the frequency spectrum (the amplitude spectrum and the phase spectrum). Then data that represents the frequency spectrum is stored in a predetermined region in the RAM 16.
  • Step 98 the same processes as Steps 46 to 52 in FIG. 2 are executed to generate the pitch data, the amplitude spectrum data and the phase spectrum data by each voice synthesis unit. Then at Step 100, the same processes as Steps 66 to 78 in FIG. 4 are executed to synthesize the singing voice and reproduce it.
  • the singing voice synthesizing process in FIG. 7 is compared to the singing voice synthesizing process in FIG. 4 .
  • the pitch data, the amplitude spectrum data and the phase spectrum data by each voice synthesis unit are obtained from the database to execute the singing voice synthesizing.
  • the section waveform data by each voice synthesis unit is obtained from the database to execute the singing voice synthesizing.
  • FIG. 14 shows the pitch-shift process and a musical tone adjustment process (corresponding to Step 66 in FIG. 4 ) related to a long sound of a single phoneme such as [a].
  • a data set (a section waveform data) of the pitch data, the amplitude spectrum data and the phase spectrum data shown in FIG. 3 is provided in the database.
  • the voice synthesis unit data that is different in the singer (the musical tone), the pitch classification, the dynamics classification and the tempo classification is stored in the database.
  • the control parameter such as a desired singer (a desired musical tone)
  • pitch classification, dynamics classification and tempo classification is designated in the input unit 20
  • the voice synthesis unit data corresponding to the control parameter to be designated is read.
  • the pitch changing process that is the same as the process at Step 66 is executed on an amplitude spectrum data FSP that is resulted from a long sound voice synthesis unit data SD. That is, the spectrum distribution is moved where a pitch corresponds to the input musical note pitch that the input musical note pitch data PT shows on the frequency axis by each spectrum distribution region of each frame related to amplitude spectrum data FSP.
  • the process returns to the start to read again.
  • a method to repeat the reading in a time sequential order can be adapted depending on a necessity.
  • the voice synthesis unit data SD is read from the end to the start after it is read to the end, and a method to repeat the reading in a time sequential order and the reading in a time reverse order depending on the necessity may be adapted.
  • a reading starting point at a time of the reading in a time reverse order may be set randomly.
  • a pitch throb data that represents a time sequential pitch change is stored corresponding to each of a long voice synthesis unit data M1 (or m1), M2 (or m2) and M3 (or m3), etc. such as [a].
  • the pitch throb data VP to be read is added on the input musical note pitch, and the pitch changing at Step 110 is controlled corresponding to the pitch controlling data as addition result.
  • the pitch throb for example, the pitch bend, vibrato and the like
  • the pitch throb can be added on the synthesized voice to obtain a natural synthesized voice.
  • the pitch throb data may be used by modifying one or plurality of pitch throb data corresponding to the voice synthesis unit by interpolation corresponding to the control parameter such as the musical tone and the like.
  • a musical tone adjustment process is executed on an amplitude spectrum data FSP' that is executed the pitch changing process at Step 110. This process is to set the musical tone of the synthesized voice adjusting the spectrum intensity according to the spectrum envelope by each frame as described before with reference to FIG. 12 .
  • FIG. 15 shows an example of the musical tone adjustment process at Step 114.
  • the spectrum envelope data that represents one typical spectrum envelope corresponding to the voice synthesis unit of the long sound [a] is stored in the database shown in FIG. 3 .
  • the spectrum envelope data corresponding to the voice synthesis unit of the long sound is read from the database DBS.
  • a spectrum envelope setting process is executed based one the spectrum envelope data to be read. That is, the spectrum envelope is set by adjusting the spectrum intensity in order to be along with the spectrum envelope indicated by the spectrum envelope data for each amplitude spectrum data of each frame of plurality of n frames amplitude spectrum data FR i to FR n in a frame group FR of long sounds. As a result, an appropriate musical tone can be added on the long sound.
  • a spectrum envelope throb data that represents a time sequential spectrum envelope change is stored corresponding to each of a long voice synthesis unit data such as [a] M (or m1), M2 (or m2) and M3 (or m3) in the database DBS shown in FIG. 3 , and the spectrum envelope throb data corresponding to the control parameter to be designated responding to designating the control parameter such as the musical tone, the pitch classification, the dynamics classification and the tempo classification in the input unit 20 may be read.
  • the spectrum envelope throb data VE to be read is added on the spectrum envelope throb data to be read at Step 116, and the spectrum envelope setting at Step 118 is controlled corresponding to the spectrum envelope controlling data as addition result.
  • the musical tone throb for example, tone bend and the like
  • the pitch throb data may be used by modifying one or plurality of pitch throb data corresponding to the voice synthesis unit by interpolation corresponding to the control parameter such as the musical tone and the like.
  • FIG. 16 shows another example of the musical tone adjustment process at Step 114.
  • a singing voice synthesizing of a phoneme series (e.g., [s_a]) - a single phoneme (e.g., [a]) - a phoneme series (e.g., [a_i]) such as the above described example of singing [saita] is a typical example
  • FIG. 16 shows the example of the typical singing voice synthesizing.
  • a phoneme series e.g., [s_a]
  • a single phoneme e.g., [a]
  • a phoneme series e.g., [a_i]
  • a former note in amplitude spectrum data PFR of the last frame of the former note is corresponding to, for example, the phoneme series [s_a]
  • a long sound of n frames amplitude spectrum data FR i to FR n of long sound is corresponding to, for example, the single phoneme [a]
  • a latter note in amplitude spectrum data PFR of the first frame of the latter note is corresponding to, for example, the phoneme series [a_i].
  • the spectrum envelope is extracted from am amplitude spectrum data PFR of a last frame of a former note, and the spectrum envelope is extracted from am amplitude spectrum data NFR of a first frame of the latter note. Then two spectrum envelopes to be extracted are executed a time interpolation, and a spectrum envelope data that represents a spectrum envelope for a long sound is formed.
  • the spectrum envelope is set by adjusting the spectrum intensity in order to be along with the spectrum envelope that the spectrum envelope data to be formed at Step 120 indicates for each amplitude spectrum data of each frame of plurality of n frames amplitude spectrum data FR i to FR n .
  • an appropriate musical tone can be added on the long sound between the phonemic chains.
  • the spectrum envelope setting can be controlled by reading the spectrum envelope throb data VE from the database DBS corresponding to the control parameter such as musical tone and the like as same as the before-described process with reference to Step 118. By doing this, a natural synthesized voice can be obtained.
  • a spectrum envelope of each frame of a voice synthesis unit is analyzed into a slope component represented by a straight line (or an index function) and one or plurality of harmonic components represented by an index function as shown in FIG. 17 . That is, an intensity of the harmonic component is calculated based on the slope component, and the spectrum envelope is represented by adding the slope component and the harmonic component. Also, a value extended the slope component to 0Hz is called a gain of the slope component.
  • two voice synthesis units [a_i] and [i_a] as shown in FIG. 18 are connected each other. Since these voice synthesis units are originally extracted from different recordings, there is a miss matching in musical tones and levels of connecting part [i]. Then, a step of a waveform is formed at the connecting part as shown in FIG. 18 , and it is heard as a noise.
  • a step at the connecting point is eliminated and generation of noise can be prevented.
  • the parameters for harmonic components of both voice synthesis unit data is multiplied by a function (cross fade parameter) that makes parameters to be 0.5 at the connecting point and the products of the multiplication are added together.
  • cross fade parameter a function that makes parameters to be 0.5 at the connecting point and the products of the multiplication are added together.
  • FIG. 19 an example wherein the cross-fading is executed by adding waveforms, each representing time sequential change of intensity of the first harmonic component (based on the slope components) for a voice synthesis unit [a_i] or [i_a] and each waveform is multiplied by the cross fade parameter.
  • the cross fading can be executed also on parameters such as other harmonic components and slope components as same as the above.
  • FIG. 20 is an example of the level adjustment process (corresponding to Step 72).
  • the level adjustment process in the case that [a_i] and [i_a] are connected to synthesize is explained.
  • the level adjustment is executed in order to be almost same amplitudes before and after the connecting point of voice synthesis units instead of cross fading.
  • the level adjustment can be executed by multiplying a certain or a transitional coefficient to the amplitude of the voice synthesis unit.
  • the above described smoothing process or level adjustment process is applied not only to the amplitude spectrum data but also to the phase spectrum data for adjustment of phase. As a result, production of noise can be prevented, and high quality singing voice synthesizing can be achieved. Further, in the smoothing process or level adjustment process, although the spectrum intensities are completely agreed at the connecting point, the spectrum intensities can be approximately agreed.

Landscapes

  • Engineering & Computer Science (AREA)
  • Computational Linguistics (AREA)
  • Health & Medical Sciences (AREA)
  • Audiology, Speech & Language Pathology (AREA)
  • Human Computer Interaction (AREA)
  • Physics & Mathematics (AREA)
  • Acoustics & Sound (AREA)
  • Multimedia (AREA)
  • Electrophonic Musical Instruments (AREA)
  • Reverberation, Karaoke And Other Acoustics (AREA)

Claims (14)

  1. Procédé de synthèse de voix de chant, comprenant les étapes suivantes :
    (a) détecter un spectre de fréquence en analysant une fréquence d'une forme d'onde vocale correspondant à une unité de synthèse vocale d'une voix à synthétiser ;
    (b) détecter une pluralité de crêtes locales d'une intensité de spectre sur lespectre de fréquence ;
    (c) désigner, pour chacune de la pluralité de crêtes locales, une région de distribution de spectre comprenant la crête locale et des spectres avant et après sur le spectre de fréquence, et générer des données de spectre d'amplitude représentant une distribution de spectre d'amplitude selon un axe des fréquences pour chaquerégion de distribution de spectre ;
    (d) générer des données de spectre de phase représentant une distribution de spectre de phase selon l'axe des fréquences pour chaquerégion de distribution de spectre ;
    (e) désigner une hauteur de note pour la voix à synthétiser ;
    (f) ajuster les données de spectre d'amplitude en déplaçant, pour chaque région de distribution de spectre, la distribution de spectre d'amplitude représentée par les données de spectre d'amplitude le long de l'axe des fréquencesconformément à la hauteur de note, tout en ajustant l'intensité de spectre de la crête locale conformément à une enveloppe de spectre EV du spectre de fréquence ;
    (g) ajuster, pour chaque région de distribution de spectre, les données de spectre de phase conformément aux données de spectre d'amplitude ajustées ; et
    (h) convertir les données de spectre d'amplitude ajustées et les données de spectre de phase ajustées en un signal vocal synthétisé d'une région temporelle.
  2. Procédé de synthèse de voix de chant selon la revendication 1, dans lequel l'étape (e) de désignation de hauteur de note désigne la hauteur de note conformément à des données de pulsation de hauteur de note représentant une variation de la hauteur de note dans une séquence temporelle.
  3. Procédé de synthèse de voix de chant selon la revendication 1, dans lequel l'étape (f) d'ajustement de données de spectre d'amplitude règle l'enveloppe de spectre qui varie dans une séquence temporelle conformément à des données de pulsation d'enveloppe de spectre représentant une variation de l'enveloppe de spectre EV du spectre de fréquence pour une séquence temporelle pour des trames séquentielles dans le temps.
  4. Procédé de synthèse de voix de chant selon la revendication 3, dans lequel les données de pulsation d'enveloppe de spectre correspondent à un paramètre de commande pour contrôler une expression musicale de la voix à synthétiser.
  5. Procédé de synthèse de voix de chant, comprenant les étapes suivantes :
    (a) obtenir des données de spectre d'amplitude et des données de spectre de phase correspondant à une unité de synthèse vocale d'une voix à synthétiser, les données de spectre d'amplitude étant des données représentant une distribution de spectre d'amplitude selon un axe des fréquences pour chaque région de distribution de spectre pour chacune d'une pluralité de crêtes locales d'une intensité de spectre comprenant la crête locale et des spectres avant et après dans un spectre de fréquence obtenu par une analyse de fréquence d'une forme d'onde vocale de l'unité de synthèse vocale, et les données de spectre de phase sont des données représentant une distribution de spectre de phase selon l'axe des fréquences pour chaque région de distribution de spectre ;
    (b) désigner une hauteur de note pour la voix à synthétiser ;
    (c) ajuster les données de spectre d'amplitude en déplaçant, pour chaque région de distribution de spectre, la distribution de spectre d'amplitude représentée par les données de spectre d'amplitude le long de l'axe des fréquences conformément à la hauteur de note, tout en ajustant l'intensité de spectre de la crête locale conformément à une enveloppe de spectre EV des données de spectre d'amplitude obtenues ;
    (d) ajuster, pour chaque région de distribution de spectre, les données de spectre de phase conformément aux données de spectre d'amplitude ajustées ; et
    (e) convertir les données de spectre d'amplitude ajustées et les données de spectre de phase ajustées en un signal vocal synthétisé d'une région temporelle.
  6. Procédé de synthèse de voix de chant selon la revendication 3, dans lequel les données de pulsation de hauteur de note correspondent à un paramètre de commande pour contrôler une expression musicale de la voix à synthétiser.
  7. Dispositif de synthèse de voix de chant, comprenant :
    un dispositif de désignation qui est adapté à désigner une unité de synthèse vocale et une hauteur de note pour une voix à synthétiser ;
    un dispositif de lecture qui est adapté à lire des données de forme d'onde vocale représentant une forme d'onde correspondant à l'unité de synthèse vocale comme données d'unité de synthèse vocale, à partir d'une base de données d'unités de synthèse vocale ;
    un premier dispositif de détection qui est adapté à détecter un spectre de fréquence en analysant une fréquence de la forme d'onde vocale représentée par les données de forme d'onde vocale ;
    un deuxième dispositif de détection qui est adapté à détecter une pluralité de crêtes locales d'une intensité de spectre sur le spectre de fréquence ;
    un premier dispositif de génération qui est adapté à désigner, pour chacune de la pluralité de crêtes locales, une région de distribution de spectre comprenant la crête locale et des spectres avant et après sur le spectre de fréquence et à générer des données de spectre d'amplitude représentant une distribution de spectre d'amplitude selon un axe des fréquences pour chaque région de distribution de spectre ;
    un deuxième dispositif de génération qui est adapté à générer des données de spectre de phase représentant une distribution de spectre de phase selon l'axe des fréquences pour chaque région de distribution de spectre ;
    un premier dispositif d'ajustement qui est adapté à ajuster les données de spectre d'amplitude en déplaçant, pour chaque région de distribution de spectre, la distribution de spectre d'amplitude représentée par les données de spectre d'amplitude le long de l'axe des fréquences conformément à la hauteur de note, tout en ajustant l'intensité de spectre de la crête locale conformément à une enveloppe de spectre EV du spectre de fréquence ;
    un deuxième dispositif d'ajustement qui est adapté à ajuster, pour chaque région de distribution de spectre, les données de spectre de phase conformément aux données de spectre d'amplitude ajustées ; et
    un dispositif de conversion qui est adapté à convertir les données de spectre d'amplitude ajustées et les données de spectre de phase ajustées en un signal vocal synthétisé d'une région temporelle.
  8. Dispositif de synthèse de voix de chant selon la revendication 7, dans lequel
    le dispositif de désignation est adapté à désigner un paramètre de commande pour contrôler une expression musicale de la voix à synthétiser, et
    le dispositif de lecture est adapté à lire des données d'unité de synthèse vocale correspondant à l'unité de synthèse vocale et au paramètre de commande.
  9. Dispositif de synthèse de voix de chant selon la revendication 7, dans lequel
    le dispositif de désignation est adapté à désigner une longueur de note et/ou un tempo pour la voix à synthétiser, et
    le dispositif de lecture est adapté à continuer à lire les données d'unité de synthèse vocale pendant un temps correspondant à la longueur de note et/au tempo en omettant une partie ou en répétant une partie ou la totalité des données d'unité de synthèse vocale.
  10. Dispositif de synthèse de voix de chant, comprenant :
    un dispositif de désignation qui est adapté à désigner une unité de synthèse vocale et une hauteur de note pour une voix à synthétiser ;
    un dispositif de lecture qui est adapté à lire des données de spectre d'amplitude et des données de spectre de phase correspondant à l'unité de synthèse vocale comme données d'unité de synthèse vocale à partir d'une base de données d'unités de synthèse vocale, dans lequel les données de spectre d'amplitude sont des données représentant une distribution de spectre d'amplitude selon un axe des fréquences pour chaque région de distribution de spectre pour chacune d'une pluralité de crêtes locales d'une intensité de spectre comprenant la crête locale et des spectres avant et après dans un spectre de fréquence obtenu par une analyse de fréquences d'une forme d'onde vocale de l'unité de synthèse vocale, et les données de spectre de phase sont des données représentant une distribution de spectre de phase selon l'axe des fréquences pour chaque région de distribution de spectre ;
    un premier dispositif d'ajustement qui est adapté à ajuster les données de spectre d'amplitude en déplaçant, pour chaque région de distribution de spectre, la distribution de spectre d'amplitude représentée par les données de spectre d'amplitude le long de l'axe des fréquences conformément à la hauteur de note, tout en ajustant l'intensité de spectre de la crête locale conformément à une enveloppe de spectre EV des données de spectre d'amplitude lues ;
    un deuxième dispositif d'ajustement qui est adapté à ajuster, pour chaque région de distribution de spectre, les données de spectre de phase conformément aux données de spectre d'amplitude ajustées ; et
    un dispositif de conversion qui est adapté à convertir les données de spectre d'amplitude ajustées et les données de spectre de phase ajustées en un signal de voix synthétisée d'une région temporelle.
  11. Dispositif de synthèse de voix de chant, comprenant :
    désigner un dispositif qui est adapté à désigner une unité de synthèse vocale et une hauteur de note pour chacune deplusieurs voix à synthétiser séquentiellement ;
    un dispositif de lecture qui est adapté à lire des données de forme d'onde vocale correspondant à chaque unité de synthèse de voix désignée par le dispositif de désignation à partir d'une base de données de synthèse vocale ;
    un premier dispositif de détection qui est adapté à détecter un spectre de fréquence en analysant une fréquence de la forme d'onde vocale correspondant à chaque forme d'onde vocale ;
    un deuxième dispositif de détection qui est adapté à détecter une pluralité de crêtes locales d'une intensité de spectre sur le spectre de fréquence correspondant à chaque forme d'onde vocale ;
    un premier dispositif de génération qui est adapté à désigner, pour chacune de la pluralité des crêtes locales pour chaque unité de synthèse vocale, une région de distribution de spectre comprenant la crête locale et des spectres avant et après sur le spectre de fréquence et à générer des données de spectre d'amplitude représentant une distribution de spectre d'amplitude selon un axe des fréquences pour chaque région de distribution de spectre ;
    un deuxième dispositif de génération qui est adapté à générer des données de spectre de phase représentant une distribution de spectre de phase selon l'axe des fréquences pour chaque région de distribution de spectre de chaque unité de synthèse vocale ;
    un premier dispositif d'ajustement qui est adapté à ajuster les données de spectre d'amplitude en déplaçant, pour chaque région de distribution de spectre de chaque unité de synthèse vocale, la distribution de spectre d'amplitude représentée par les données de spectre d'amplitude le long de l'axe des fréquencesconformément à la hauteur de note, tout en ajustant l'intensité de spectre de la crête locale conformément à une enveloppe de spectre EV du spectre de fréquence ;
    un deuxième dispositif d'ajustement qui est adapté à ajuster, pour la région de distribution de spectre de chaque unité de synthèse vocale, les données de spectre de phase conformément aux données de spectre d'amplitude ajustées ;
    un premier dispositif de connexion qui est adapté à connecter les données de spectre d'amplitude ajustées pour connecter des unités de synthèse vocale séquentielles correspondant respectivement aux voix à synthétiser séquentiellement dans un certain ordre de prononciation, dans lequel les intensités de spectre sont ajustées de façon à concorder ou concorder approximativement entre elles au niveau de points de connexion des unités de synthèse vocale séquentielles ;
    un deuxième dispositif de connexion qui est adapté à connecter les données de spectre de phase ajustées pour connecter des unités de synthèse vocale séquentielles correspondant respectivement aux voix à synthétiser séquentiellement dans un certain ordre de prononciation, dans lequel les phases sont ajustées de façon à concorder ou concorder approximativement entre elles au niveau de points de connexion des unités de synthèse vocale séquentielles ;
    un dispositif de conversion qui est adapté à convertir les données de spectre d'amplitude connectées et les données de spectre de phase connectées en un signal vocal synthétisé d'une région temporelle.
  12. Dispositif de synthèse de voix de chant, comprenant :
    un dispositif de désignation qui est adapté à désigner une unité de synthèse vocale et une hauteur de note pour chacune deplusieurs voixà synthétiser séquentiellement ;
    un dispositif de lecture qui est adapté à lire des données de forme d'onde vocale correspondant à chaque unité de synthèse vocale désignée par le dispositif de désignation à partir d'une base de données d'unités de synthèse vocale, dans lequel les données de spectre d'amplitude sont des données représentant une distribution de spectre d'amplitude selon un axe des fréquences pour chaque région de distribution de spectre pour chacune d'une pluralité de crêtes locales d'une intensité de spectre comprenant la crête locale et des spectres avant et après dans un spectre de fréquence obtenu par une analyse de fréquences d'une forme d'onde vocale de l'unité de synthèse vocale, et les données de spectre de phase sont des données représentant une distribution de spectre de phase selon l'axe des fréquences pour chaque région de distribution de spectre ;
    un premier dispositif d'ajustement qui est adapté à ajuster les données de spectre d'amplitude en déplaçant, pour chaque région de distribution de spectre de chaque unité de synthèse vocale, la distribution de spectre d'amplitude représentée par les données de spectre d'amplitude le long de l'axe des fréquences conformément à la hauteur de note, tout en ajustant l'intensité de spectre de la crête locale conformément à une enveloppe de spectre EV des données de spectre d'amplitude lues ;
    un deuxième dispositif d'ajustement qui est adapté à ajuster, pour chaque région de distribution de spectre de chaque unité de synthèse vocale, les données de spectre de phase conformément aux données de spectre d'amplitude ajustées ;
    un premier dispositif de connexion qui est adapté à connecter les données de spectre d'amplitude ajustées pour connecter des unités de synthèse vocale séquentielles correspondant respectivement aux voix à synthétiser séquentiellement dans un certain ordre de prononciation, dans lequel les intensités de spectre sont ajustées de façon à concorder ou concorder approximativement entre elles au niveau de points de connexion des unités de synthèse vocale séquentielles ;
    un deuxième dispositif de connexion qui est adapté à connecter les données de spectre de phase ajustées pour connecter des unités de synthèse vocale séquentielles correspondant respectivement aux voix à synthétiser séquentiellement dans un certain ordre de prononciation, dans lequel les phases sont ajustées pour concorder ou concorder approximativement entre elles au niveau de points de connexion des unités de synthèse vocale séquentielles ; et
    un dispositif de conversion qui est adapté à convertir les données de spectre d'amplitude connectées et les données de spectre de phase connectées en un signal vocal synthétisé d'une région temporelle.
  13. Support de stockage mémorisant un programme pour un procédé de synthèse de voix de chant, le programme comprenant les instructions pour :
    (a) détecter un spectre de fréquence en analysant une fréquence d'une forme d'onde vocale correspondant à une unité de synthèse vocale d'une voix à synthétiser ;
    (b) détecter une pluralité de crêtes locales d'une intensité de spectre sur le spectre de fréquence ;
    (c) désigner, pour chacune de la pluralité de crêtes locales, une région de distribution de spectre comprenant la crête locale et des spectres avant et après sur le spectre de fréquence, et générer des données de spectre d'amplitude représentant une distribution de spectre d'amplitude selon un axe des fréquences pour chaque région de distribution de spectre ;
    (d) générer des données de spectre de phase représentant une distribution de spectre de phase selon l'axe des fréquences pour chaque région de distribution de spectre ;
    (e) désigner une hauteur de note pour la voix à synthétiser ;
    (f) ajuster les données de spectre d'amplitude en déplaçant, pour chaque région de distribution de spectre, la distribution de spectre d'amplitude représentée par les données de spectre d'amplitude le long de l'axe des fréquences conformément à la hauteur de note, tout en ajustant l'intensité de spectre de la crête locale conformément à une enveloppe de spectre EV du spectre de fréquence ;
    (g) ajuster, pour chaque région de distribution de spectre, les données de spectre de phase conformément aux données de spectre d'amplitude ajustées ; et
    (h) convertir les données de spectre d'amplitude ajustées et les données de spectre de phase ajustées en un signal vocal synthétisé d'une région temporelle.
  14. Support de stockage mémorisant un programme pour un procédé de synthèse de voix de chant, le programme comprenant les instructions pour :
    (a) obtenir des données de spectre d'amplitude et des données de spectre de phase correspondant à une unité de synthèse vocale d'une voix à synthétiser, les données de spectre d'amplitude étant des données représentant une distribution de spectre d'amplitude selon un axe des fréquences pour chaque région de distribution de spectre pour chacune d'une pluralité de crêtes locales d'une intensité de spectre comprenant la crête locale et des spectres avant et après dans un spectre de fréquence obtenu par une analyse de fréquences d'une forme d'onde vocale de l'unité de synthèse vocale, et les données de spectre de phase sont des données représentant une distribution de spectre de phase selon l'axe des fréquences pour chaque région de distribution de spectre ;
    (b) désigner une hauteur de note pour la voix à synthétiser ;
    (c) ajuster les données de spectre d'amplitude en déplaçant, pour chaque région de distribution de spectre, la distribution de spectre d'amplitude représentée par les données de spectre d'amplitude le long de l'axe des fréquences conformément à la hauteur de note, tout en ajustant l'intensité de spectre de la crête locale conformément à une enveloppe de spectre EV des données de spectre d'amplitude obtenues ;
    (d) ajuster, pour chaque région de distribution de spectre, les données de spectre de phase conformément aux données de spectre d'amplitude ajustées ; et
    (e) convertir les données de spectre d'amplitude ajustées et les données de spectre de phase ajustées en un signal vocal synthétisé d'une région temporelle.
EP03017548.3A 2003-08-06 2003-08-06 Méthode de synthèse de voix chantée Expired - Fee Related EP1505570B1 (fr)

Priority Applications (1)

Application Number Priority Date Filing Date Title
EP03017548.3A EP1505570B1 (fr) 2003-08-06 2003-08-06 Méthode de synthèse de voix chantée

Applications Claiming Priority (1)

Application Number Priority Date Filing Date Title
EP03017548.3A EP1505570B1 (fr) 2003-08-06 2003-08-06 Méthode de synthèse de voix chantée

Publications (2)

Publication Number Publication Date
EP1505570A1 EP1505570A1 (fr) 2005-02-09
EP1505570B1 true EP1505570B1 (fr) 2017-10-11

Family

ID=33547617

Family Applications (1)

Application Number Title Priority Date Filing Date
EP03017548.3A Expired - Fee Related EP1505570B1 (fr) 2003-08-06 2003-08-06 Méthode de synthèse de voix chantée

Country Status (1)

Country Link
EP (1) EP1505570B1 (fr)

Families Citing this family (4)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN103489451B (zh) * 2012-06-13 2016-11-23 百度在线网络技术（北京）有限公司 移动终端的语音处理方法及移动终端
UA74516U (ru) * 2012-06-18 2012-10-25 Сергей Александрович Лапковский Способ настройки параметров музыкальной композиции по с.а.лапковскому
CN113506560B (zh) * 2021-07-21 2023-06-09 四川启睿克科技有限公司 一种保持音高的歌声合成方法及装置
CN113889073B (zh) * 2021-09-27 2022-10-18 北京百度网讯科技有限公司 语音处理方法、装置、电子设备和存储介质

Non-Patent Citations (1)

* Cited by examiner, † Cited by third party
Title
DOLSON M: "THE PHASE VOCODER: A TUTORIAL", COMPUTER MUSIC JOURNAL, CAMBRIDGE, MA, US, vol. 10, no. 4, 21 December 1986 (1986-12-21), pages 14 - 27, XP009029676, ISSN: 0148-9267 *

Also Published As

Publication number Publication date
EP1505570A1 (fr) 2005-02-09

Similar Documents

Publication Publication Date Title
US6992245B2 (en) Singing voice synthesizing method
US7016841B2 (en) Singing voice synthesizing apparatus, singing voice synthesizing method, and program for realizing singing voice synthesizing method
US6804649B2 (en) Expressivity of voice synthesis by emphasizing source signal features
EP1701336B1 (fr) Appareil et procédé de traitement du son, et programme correspondant
US7135636B2 (en) Singing voice synthesizing apparatus, singing voice synthesizing method and program for singing voice synthesizing
Schnell et al. Synthesizing a choir in real-time using Pitch Synchronous Overlap Add (PSOLA).
US6944589B2 (en) Voice analyzing and synthesizing apparatus and method, and program
JP3966074B2 (ja) ピッチ変換装置、ピッチ変換方法及びプログラム
EP1505570B1 (fr) Méthode de synthèse de voix chantée
CN100524456C (zh) 歌唱声音合成方法和装置
JP4349316B2 (ja) 音声分析及び合成装置、方法、プログラム
JP2000010597A (ja) 音声変換装置及び音声変換方法
JP4565846B2 (ja) ピッチ変換装置
Bonada et al. Sample-based singing voice synthesizer using spectral models and source-filter decomposition
JPH0895588A (ja) 音声合成装置
JP3540609B2 (ja) 音声変換装置及び音声変換方法
JPH1031496A (ja) 楽音発生装置
JP3979213B2 (ja) 歌唱合成装置、歌唱合成方法並びに歌唱合成用プログラム
JP2001312300A (ja) 音声合成装置
EP1160766B1 (fr) Codage de l'expression dans une voix de synthèse
Bonada et al. Special Session on Singing Voice-Sample-Based Singing Voice Synthesizer Using Spectral Models and Source-Filter Decomposition
JPH0962297A (ja) フォルマント音源のパラメータ生成装置
JP2000020100A (ja) 音声変換装置及び音声変換方法
JP2005004105A (ja) 信号生成装置及び信号生成方法
JPH03296100A (ja) 音声合成装置

Legal Events

Date Code Title Description
PUAI Public reference made under article 153(3) epc to a published international application that has entered the european phase

Free format text: ORIGINAL CODE: 0009012

AK Designated contracting states

Kind code of ref document: A1

Designated state(s): AT BE BG CH CY CZ DE DK EE ES FI FR GB GR HU IE IT LI LU MC NL PT RO SE SI SK TR

AX Request for extension of the european patent

Extension state: AL LT LV MK

17P Request for examination filed

Effective date: 20050808

AKX Designation fees paid

Designated state(s): DE GB IT

RAP1 Party data changed (applicant data changed or rights of an application transferred)

Owner name: YAMAHA CORPORATION

17Q First examination report despatched

Effective date: 20090109

R17C First examination report despatched (corrected)

Effective date: 20090109

REG Reference to a national code

Ref country code: DE

Ref legal event code: R079

Ref document number: 60350676

Country of ref document: DE

Free format text: PREVIOUS MAIN CLASS: G10L0013020000

Ipc: G10L0013033000

GRAP Despatch of communication of intention to grant a patent

Free format text: ORIGINAL CODE: EPIDOSNIGR1

RIC1 Information provided on ipc code assigned before grant

Ipc: G10L 13/033 20130101AFI20170307BHEP

Ipc: G10L 21/0232 20130101ALN20170307BHEP

INTG Intention to grant announced

Effective date: 20170324

GRAS Grant fee paid

Free format text: ORIGINAL CODE: EPIDOSNIGR3

GRAA (expected) grant

Free format text: ORIGINAL CODE: 0009210

AK Designated contracting states

Kind code of ref document: B1

Designated state(s): DE GB IT

REG Reference to a national code

Ref country code: GB

Ref legal event code: FG4D

REG Reference to a national code

Ref country code: DE

Ref legal event code: R096

Ref document number: 60350676

Country of ref document: DE

REG Reference to a national code

Ref country code: DE

Ref legal event code: R097

Ref document number: 60350676

Country of ref document: DE

PLBE No opposition filed within time limit

Free format text: ORIGINAL CODE: 0009261

STAA Information on the status of an ep patent application or granted ep patent

Free format text: STATUS: NO OPPOSITION FILED WITHIN TIME LIMIT

PG25 Lapsed in a contracting state [announced via postgrant information from national office to epo]

Ref country code: IT

Free format text: LAPSE BECAUSE OF FAILURE TO SUBMIT A TRANSLATION OF THE DESCRIPTION OR TO PAY THE FEE WITHIN THE PRESCRIBED TIME-LIMIT

Effective date: 20171011

26N No opposition filed

Effective date: 20180712

PGFP Annual fee paid to national office [announced via postgrant information from national office to epo]

Ref country code: DE

Payment date: 20180823

Year of fee payment: 16

PGFP Annual fee paid to national office [announced via postgrant information from national office to epo]

Ref country code: GB

Payment date: 20180822

Year of fee payment: 16

REG Reference to a national code

Ref country code: DE

Ref legal event code: R119

Ref document number: 60350676

Country of ref document: DE

GBPC Gb: european patent ceased through non-payment of renewal fee

Effective date: 20190806

PG25 Lapsed in a contracting state [announced via postgrant information from national office to epo]

Ref country code: DE

Free format text: LAPSE BECAUSE OF NON-PAYMENT OF DUE FEES

Effective date: 20200303

PG25 Lapsed in a contracting state [announced via postgrant information from national office to epo]

Ref country code: GB

Free format text: LAPSE BECAUSE OF NON-PAYMENT OF DUE FEES

Effective date: 20190806