EP0396141A2 - Vorrichtung und Verfahren zur Synthetisierung von Gesang in Echtzeit - Google Patents

Vorrichtung und Verfahren zur Synthetisierung von Gesang in Echtzeit Download PDF

Info

Publication number
EP0396141A2
EP0396141A2 EP19900108393 EP90108393A EP0396141A2 EP 0396141 A2 EP0396141 A2 EP 0396141A2 EP 19900108393 EP19900108393 EP 19900108393 EP 90108393 A EP90108393 A EP 90108393A EP 0396141 A2 EP0396141 A2 EP 0396141A2
Authority
EP
European Patent Office
Prior art keywords
real time
electrical signals
midi
generating
varying
Prior art date
Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
Withdrawn
Application number
EP19900108393
Other languages
English (en)
French (fr)
Inventor
Florian Schneider
Gert Joachim Ott
Gert Jalass
Current Assignee (The listed assignees may be inaccurate. Google has not performed a legal analysis and makes no representation or warranty as to the accuracy of the list.)
Individual
Original Assignee
Individual
Priority date (The priority date is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the date listed.)
Filing date
Publication date
Application filed by Individual filed Critical Individual
Publication of EP0396141A2 publication Critical patent/EP0396141A2/de
Withdrawn legal-status Critical Current

Links

Images

Classifications

    • G—PHYSICS
    • G10—MUSICAL INSTRUMENTS; ACOUSTICS
    • G10H—ELECTROPHONIC MUSICAL INSTRUMENTS; INSTRUMENTS IN WHICH THE TONES ARE GENERATED BY ELECTROMECHANICAL MEANS OR ELECTRONIC GENERATORS, OR IN WHICH THE TONES ARE SYNTHESISED FROM A DATA STORE
    • G10H5/00—Instruments in which the tones are generated by means of electronic generators
    • G10H5/005—Voice controlled instruments
    • G—PHYSICS
    • G10—MUSICAL INSTRUMENTS; ACOUSTICS
    • G10L—SPEECH ANALYSIS OR SYNTHESIS; SPEECH RECOGNITION; SPEECH OR VOICE PROCESSING; SPEECH OR AUDIO CODING OR DECODING
    • G10L13/00—Speech synthesis; Text to speech systems
    • G10L13/02—Methods for producing synthetic speech; Speech synthesisers
    • G—PHYSICS
    • G10—MUSICAL INSTRUMENTS; ACOUSTICS
    • G10H—ELECTROPHONIC MUSICAL INSTRUMENTS; INSTRUMENTS IN WHICH THE TONES ARE GENERATED BY ELECTROMECHANICAL MEANS OR ELECTRONIC GENERATORS, OR IN WHICH THE TONES ARE SYNTHESISED FROM A DATA STORE
    • G10H2250/00—Aspects of algorithms or signal processing methods without intrinsic musical character, yet specifically adapted for or used in electrophonic musical processing
    • G10H2250/315—Sound category-dependent sound synthesis processes [Gensound] for musical use; Sound category-specific synthesis-controlling parameters or control means therefor
    • G10H2250/455—Gensound singing voices, i.e. generation of human voices for musical applications, vocal singing sounds or intelligible words at a desired pitch or with desired vocal effects, e.g. by phoneme synthesis

Definitions

  • the present invention relates to voice synthesizing and in particular to a system for the synthesizing of singing in real time.
  • Speech synthesizing systems are known wherein the singing of a song is simulated.
  • two systems of this type are shown in US-A-4,731,847 and US-A-4,527,274.
  • the main object of the present invention is to provide a system for and method of synthesizing the singing of song in real time.
  • MIDI musical instrument digital interface hardware
  • speech processor integrated circuit By making use of musical instrument digital interface hardware (MIDI) and a speech processor integrated circuit, it is possible to create a singing voice directly from a keyboard input in real time, that is, whith the duration and frequency at which the keyboard keys are actuated controlling the time, loudness and pitch of the singing voice.
  • MIDI musical instrument digital interface hardware
  • speech processor integrated circuit By making use of musical instrument digital interface hardware (MIDI) and a speech processor integrated circuit, it is possible to create a singing voice directly from a keyboard input in real time, that is, whith the duration and frequency at which the keyboard keys are actuated controlling the time, loudness and pitch of the singing voice.
  • any sppec synthesizing device can be used in the system according to the present invention.
  • the sounds can be played in real time.
  • the duration of the sounds is dependent upon how long a key on the keyboard is depressed and is not determined by preprogrammed duration as described in US-A-4,527,274.
  • the present invention enabels one to do so.
  • the quality of sounds can be manipulated in real time by means of changing the vocal tract length, the dynamic values can be altered in real time, the interpolation timing or transfer function between one sound and the following sound can be altered in real time, the pitch of the sounds can be changed in real time independently from the above-mentioned parameters and the envelope of the sounds which includes the attack and decay are programmable.
  • the sound and pitch range of the synthesizer can be further enhanced and improved by inserting an external sound source into the chip and replacing the internal excitation function.
  • the external sound can be easily taken from any sound source allowing one to feed different wave forms or noise to simulate whispering. Also chords or any composed sound could be fed into the speech synthesizer filters.
  • chords or any composed sound could be fed into the speech synthesizer filters.
  • both the sound characteristics of the speech synthesizer and the external sound sources can be drastically changed.
  • the expressive subtleties of singing, such as vibrato and portamento, are also obtainable and which are not possible in the prior art devices.
  • the present invention also enables one to store the MIDI generated data in any MIDI memory device and which can be loaded and transferred onto storage media because of the international MIDI standard. Thus no specific computer software or hardware is needed besides the ability to understand MIDI codes.
  • the speech synthesizer and MIDI combination can be used to generate a large amount of electronic sounds and organ like timbres.
  • the combination can also be used to generate spectres with vocal qualities resembling a voice choir and additionally one can program sequences in such a specific way that speech like sounds or even intelligible speech can be generated.
  • the attributes of the MIDI commands allow synchronization of high resolution with the MIDI time code and this means that one can perfectly synchronize one or more MIDI speech synthesizers with one another and with existing or prerecorded music.
  • a capella music or a synthetic vocal orchestra can be realized.
  • synthesized lyrics can be programmed and edited to that a synthetic voice sings along with the music, synchronized by the MIDI time code. Because the synchronization proportionally relates to the speed of the music, the strings of speech sounds can be speeded up or slowed down in relation to the rhythm and the result is a 100% accurate timing which can hardly be achieved by a human singer.
  • a phoneme editor compiler can be utilized to easily and accurately synchronize the system.
  • the system comprises the combination of the MIDI interface and speech synthesis in a musical context.
  • the sounds are configured on a MIDI keyboard according to the sounding properties of the speed sounds from dark to bright vowels, followed by the voiced consonants and voiceless consonants and finally plosives so that a great variety of musically useful timbers like formants, noise bands, stationary sounds and percussive sounds can be generated to create the electronic sounds covering a broad spectrum.
  • the speech synthesizer vocal tract length or filter frequency and interpolation speed can be controlled by the MIDI pitch bend and modulation wheel or any other MIDI controller.
  • envelope parameters for all sounds except the plosives, in this case attack time and decay time can be determined by specific MIDI control change sequences.
  • the MIDI implementation is such that keys 36-93 control speech sounds on channels 1, 3, 5 and 7 whereas the pitch control is controlled on channels 2, 4, 6 and 8.
  • the pitch bend wheel is used to control vocal tract length, the modulation wheel controls speed of filter interpolation and the velocity controls dynamic loudness control.
  • a phoneme speech synthesizer 17 now produced by Artic, formerly by the Votrax Division of the Federal Screw Works, Troy, Mich, USA, specifically, the Votrax SC-02 speech synthesizer and the data sheet for that synthesizer is incorporated herein by reference. Also incorporated by reference is the MIDI specification 1.0 by the International MIDI Association, Sun Valley, CA, USA, which is incorporated in the MIDI interface 15.
  • the system also includes a processing unit is shown that consists of CPU 10, which may be a 6502 CPU, a ROM 12 for data storage purposes, which may be a 6116, and Address Decoding Logic 13. These parts are connected via a common computer bus 14 containing address, data and command lines. Timing is controlled by a clock generator 18.
  • MIDI interface 15 Also connected to the compuer bus are MIDI interface 15, mainly an ACIA 6850, and a buffer register 16, consisting of two 75LS245.
  • One or more speech processors chips 17 are connected to the buffer 16. These chips are provided with additional audio circuitry, the audio output 19 which buffers the audio output of the SC-02 and makes it suitable for audio amplifiers, and the audio input 20 at pins 3 and 5 which allows an external audio signal, e.g. from a synthesizer, to be fed into the SC-02 thus replacing the internal tone generator.
  • the tone source can be selected using an appropriate switch 22.
  • the main taks of the processing unit is to receive MIDI data from interface 15, to translate it in several ways into SC-02 specific data and to output these data via the buffer 16 to the SC-02 17.
  • the SC-02 provides 8 registers, of which 5 are used in the singing process Phoneme, Inflection, Articulation, Amplitude and Filter Frequency.
  • the kind of interpreting and translating of the received MIDI data into data for these registers can depend on the switch position of the mode selection 21 or can be controlled by specific MIDI data, e.g. Program Change events.
  • the MIDI Note On events receivd on a MIDI channel N from keyborad 1 shown in Fig. 2 turn on a specific phoneme by writing the number of the phoneme into the Phoneme register of the SC-02.
  • the phoneme number is generated by a translation table which translates the Note Number into a phoneme number.
  • the Velocity of the Note on envent is used to affect the Amplitude register of the SC-02.
  • Note Off events received on channel N are used to turn off the phoneme by writing the code for the phoneme 'Pause' into the phoneme register.
  • Note On events received on channel N + 1 are used to select the appropriate singing frequency.
  • the MIDI Note On events received on a MIDI channel N are used to select the appropriate singing frequency.
  • the Note Numbers of these events are translated into values for the Inflection register like above.
  • the velocity of the Note On event is used to affect the Amplitude register of the SC-02.
  • Note Off events received on channel N are used to turn off the phoneme by writing the value 0 into the Amplitude register.
  • Program Change events turn on a specific phoneme by writing the number of the phoneme into the Phoneme register of the SC-02.
  • the phoneme number is generated by a translation table which translates the Program Number into a phoneme number.
  • Pitch Wheel Change events are translated into values for the Filter Frequency and Continuous Controller events for Modulation Wheel are translated into values for the Articulation register. This way of interpretation allows the user to play the SC-02 using MIDI compatible keyboard 1 in a way similar to a common expander device with special voice-like sounds.
  • Two preferred implementations of the invention include using a single SC-02 with 2 modes of interpreting the MIDI events, and using four SC-02 chips with 6 modes of interpreting the MIDI events.
  • a MIDI-compatible sequencer-program is designed for the system using a phoneme editor-compiler synchronizing syllables of speech to MIDI-timecode.
  • Any text-editor can be used to write a "phonetic score" which determines all parameters for the artificial singer like pitch, dynamic, timing, etc.
  • the digital controlled analogue chip is using the concept of phoneme synthesis, which makes it relatively easy to program for real time application.
  • the sounding elements of speech or singing are defined as vowels, voiced and unvoiced consonants and plosives, a variety of timbres like formants, noise bands and percussive sounds are generated.
  • the 54 phonemes of the chip are configures to match with 54 MIDI-notes on any MIDI-keyboard from 36 to 93 in groups of vowels, consonants etc. as shown in Fig. 2.
  • the speech synthesizer functions like a MIDI-expander, to be "played” with a keyboard or driven by the "speech-sequencer” program.
  • the chip is suitable as a multi bandpass-filter to process external sound sources.
  • Any text to be uttered or sung is a potential "phonetic score”.
  • First a text is analyzed by its phonemic structure, through a text to speech conversion either automatically by rules, or by manual input.
  • syllables, or phonemes strings have to be divided or chopped in more or less equal frames based on the 24 pulses of the MIDI-clock.
  • the program calculates a timebase of 24 divided by the number of phonemes, so that one syllable consisting of 7 and another consisting of 2 phonemes have the same duration. If the automatic conversion does not sound satisfying, subtle variations are possible through editing.
  • the next parameter is the dynamic value :L: loudness
  • Values range from 0 to 9, configured either to a linear or lograithmic equivalent of the 1- 127 velocity scale.
  • the last parameter is pitch or tone "T"
  • filter frquency and interpolation rate are controlled in realtime by MIDI pitchbend- and modulation wheel.
EP19900108393 1989-05-04 1990-05-03 Vorrichtung und Verfahren zur Synthetisierung von Gesang in Echtzeit Withdrawn EP0396141A2 (de)

Applications Claiming Priority (2)

Application Number Priority Date Filing Date Title
US34736789A 1989-05-04 1989-05-04
US347367 1989-05-04

Publications (1)

Publication Number Publication Date
EP0396141A2 true EP0396141A2 (de) 1990-11-07

Family

ID=23363411

Family Applications (1)

Application Number Title Priority Date Filing Date
EP19900108393 Withdrawn EP0396141A2 (de) 1989-05-04 1990-05-03 Vorrichtung und Verfahren zur Synthetisierung von Gesang in Echtzeit

Country Status (1)

Country Link
EP (1) EP0396141A2 (de)

Cited By (3)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
EP0723256A2 (de) * 1995-01-17 1996-07-24 Yamaha Corporation Karaokevorrichtung, wobei die Sangstimme durch eine Modellstimme verändert wird
EP0729130A2 (de) * 1995-02-27 1996-08-28 Yamaha Corporation Karaokevorrichtung mit Erzegung von einer an eine Gesangstimme zugefügten synthetischen Stimme
EP2733696A1 (de) * 2012-11-14 2014-05-21 Yamaha Corporation Sprachsyntheseverfahren und Sprachsynthesevorrichtung

Cited By (8)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
EP0723256A2 (de) * 1995-01-17 1996-07-24 Yamaha Corporation Karaokevorrichtung, wobei die Sangstimme durch eine Modellstimme verändert wird
EP0723256A3 (de) * 1995-01-17 1996-11-13 Yamaha Corp Karaokevorrichtung, wobei die Sangstimme durch eine Modellstimme verändert wird
US5955693A (en) * 1995-01-17 1999-09-21 Yamaha Corporation Karaoke apparatus modifying live singing voice by model voice
EP0729130A2 (de) * 1995-02-27 1996-08-28 Yamaha Corporation Karaokevorrichtung mit Erzegung von einer an eine Gesangstimme zugefügten synthetischen Stimme
EP0729130A3 (de) * 1995-02-27 1997-01-08 Yamaha Corp Karaokevorrichtung mit Erzegung von einer an eine Gesangstimme zugefügten synthetischen Stimme
US5857171A (en) * 1995-02-27 1999-01-05 Yamaha Corporation Karaoke apparatus using frequency of actual singing voice to synthesize harmony voice from stored voice information
EP2733696A1 (de) * 2012-11-14 2014-05-21 Yamaha Corporation Sprachsyntheseverfahren und Sprachsynthesevorrichtung
US10002604B2 (en) 2012-11-14 2018-06-19 Yamaha Corporation Voice synthesizing method and voice synthesizing apparatus

Similar Documents

Publication Publication Date Title
JP3333022B2 (ja) 歌声合成装置
US6191349B1 (en) Musical instrument digital interface with speech capability
US7249022B2 (en) Singing voice-synthesizing method and apparatus and storage medium
US5857171A (en) Karaoke apparatus using frequency of actual singing voice to synthesize harmony voice from stored voice information
JP3144273B2 (ja) 自動歌唱装置
EP1512140B1 (de) Musikalisches notierungssystem
CN113160779A (zh) 电子乐器、方法及存储介质
Lindemann Music synthesis with reconstructive phrase modeling
US11417312B2 (en) Keyboard instrument and method performed by computer of keyboard instrument
JPH11184490A (ja) 規則音声合成による歌声合成方法
JP3518253B2 (ja) データ編集装置
EP0396141A2 (de) Vorrichtung und Verfahren zur Synthetisierung von Gesang in Echtzeit
JP3307283B2 (ja) 歌唱音合成装置
JPH11126083A (ja) カラオケ再生装置
JPH11282483A (ja) カラオケ装置
WO2020217801A1 (ja) オーディオ情報再生方法および装置、オーディオ情報生成方法および装置、並びにプログラム
JP3233036B2 (ja) 歌唱音合成装置
JP2002221978A (ja) ボーカルデータ生成装置、ボーカルデータ生成方法および歌唱音合成装置
JP3265995B2 (ja) 歌唱音声合成装置及び方法
JPH0895588A (ja) 音声合成装置
JP2904045B2 (ja) カラオケ装置
JP7276292B2 (ja) 電子楽器、電子楽器の制御方法、及びプログラム
JPH1031496A (ja) 楽音発生装置
Subramanian Synthesizing Carnatic music with a computer
JPS6183600A (ja) 歌声合成演奏装置

Legal Events

Date Code Title Description
PUAI Public reference made under article 153(3) epc to a published international application that has entered the european phase

Free format text: ORIGINAL CODE: 0009012

AK Designated contracting states

Kind code of ref document: A2

Designated state(s): DE FR GB

STAA Information on the status of an ep patent application or granted ep patent

Free format text: STATUS: THE APPLICATION IS DEEMED TO BE WITHDRAWN

18D Application deemed to be withdrawn

Effective date: 19921202