EP1970892A1 - Procédé pour établir un signal de contrôle d'harmonie contrôlé en temps réel par un signal d'entrée de guitare - Google Patents

Procédé pour établir un signal de contrôle d'harmonie contrôlé en temps réel par un signal d'entrée de guitare Download PDF

Info

Publication number
EP1970892A1
EP1970892A1 EP07005051A EP07005051A EP1970892A1 EP 1970892 A1 EP1970892 A1 EP 1970892A1 EP 07005051 A EP07005051 A EP 07005051A EP 07005051 A EP07005051 A EP 07005051A EP 1970892 A1 EP1970892 A1 EP 1970892A1
Authority
EP
European Patent Office
Prior art keywords
input
harmony
control signal
signal
time
Prior art date
Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
Withdrawn
Application number
EP07005051A
Other languages
German (de)
English (en)
Inventor
Guangji Shi
Current Assignee (The listed assignees may be inaccurate. Google has not performed a legal analysis and makes no representation or warranty as to the accuracy of the list.)
TC Group AS
Original Assignee
TC Group AS
Priority date (The priority date is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the date listed.)
Filing date
Publication date
Application filed by TC Group AS filed Critical TC Group AS
Priority to EP07005051A priority Critical patent/EP1970892A1/fr
Publication of EP1970892A1 publication Critical patent/EP1970892A1/fr
Withdrawn legal-status Critical Current

Links

Images

Classifications

    • G—PHYSICS
    • G10—MUSICAL INSTRUMENTS; ACOUSTICS
    • G10H—ELECTROPHONIC MUSICAL INSTRUMENTS; INSTRUMENTS IN WHICH THE TONES ARE GENERATED BY ELECTROMECHANICAL MEANS OR ELECTRONIC GENERATORS, OR IN WHICH THE TONES ARE SYNTHESISED FROM A DATA STORE
    • G10H3/00—Instruments in which the tones are generated by electromechanical means
    • G10H3/12—Instruments in which the tones are generated by electromechanical means using mechanical resonant generators, e.g. strings or percussive instruments, the tones of which are picked up by electromechanical transducers, the electrical signals being further manipulated or amplified and subsequently converted to sound by a loudspeaker or equivalent instrument
    • G10H3/125—Extracting or recognising the pitch or fundamental frequency of the picked up signal
    • G—PHYSICS
    • G10—MUSICAL INSTRUMENTS; ACOUSTICS
    • G10H—ELECTROPHONIC MUSICAL INSTRUMENTS; INSTRUMENTS IN WHICH THE TONES ARE GENERATED BY ELECTROMECHANICAL MEANS OR ELECTRONIC GENERATORS, OR IN WHICH THE TONES ARE SYNTHESISED FROM A DATA STORE
    • G10H1/00—Details of electrophonic musical instruments
    • G10H1/36—Accompaniment arrangements
    • G10H1/38—Chord
    • G—PHYSICS
    • G10—MUSICAL INSTRUMENTS; ACOUSTICS
    • G10H—ELECTROPHONIC MUSICAL INSTRUMENTS; INSTRUMENTS IN WHICH THE TONES ARE GENERATED BY ELECTROMECHANICAL MEANS OR ELECTRONIC GENERATORS, OR IN WHICH THE TONES ARE SYNTHESISED FROM A DATA STORE
    • G10H5/00—Instruments in which the tones are generated by means of electronic generators
    • G10H5/005—Voice controlled instruments
    • G—PHYSICS
    • G10—MUSICAL INSTRUMENTS; ACOUSTICS
    • G10H—ELECTROPHONIC MUSICAL INSTRUMENTS; INSTRUMENTS IN WHICH THE TONES ARE GENERATED BY ELECTROMECHANICAL MEANS OR ELECTRONIC GENERATORS, OR IN WHICH THE TONES ARE SYNTHESISED FROM A DATA STORE
    • G10H2210/00—Aspects or methods of musical processing having intrinsic musical character, i.e. involving musical theory or musical parameters or relying on musical knowledge, as applied in electrophonic musical tools or instruments
    • G10H2210/031—Musical analysis, i.e. isolation, extraction or identification of musical elements or musical parameters from a raw acoustic signal or from an encoded audio signal
    • G10H2210/066—Musical analysis, i.e. isolation, extraction or identification of musical elements or musical parameters from a raw acoustic signal or from an encoded audio signal for pitch analysis as part of wider processing for musical purposes, e.g. transcription, musical performance evaluation; Pitch recognition, e.g. in polyphonic sounds; Estimation or use of missing fundamental
    • G—PHYSICS
    • G10—MUSICAL INSTRUMENTS; ACOUSTICS
    • G10H—ELECTROPHONIC MUSICAL INSTRUMENTS; INSTRUMENTS IN WHICH THE TONES ARE GENERATED BY ELECTROMECHANICAL MEANS OR ELECTRONIC GENERATORS, OR IN WHICH THE TONES ARE SYNTHESISED FROM A DATA STORE
    • G10H2210/00—Aspects or methods of musical processing having intrinsic musical character, i.e. involving musical theory or musical parameters or relying on musical knowledge, as applied in electrophonic musical tools or instruments
    • G10H2210/155—Musical effects
    • G10H2210/245—Ensemble, i.e. adding one or more voices, also instrumental voices
    • G10H2210/251—Chorus, i.e. automatic generation of two or more extra voices added to the melody, e.g. by a chorus effect processor or multiple voice harmonizer, to produce a chorus or unison effect, wherein individual sounds from multiple sources with roughly the same timbre converge and are perceived as one

Definitions

  • the invention relates to a method of establishing a harmony control signal controlled in real-time by a guitar audio input signal according to claim 1 and an apparatus for implementing the method.
  • a problem in the prior art has been that state-of-the-art harmony processors are somewhat restricted in use due to the fact that such real-time processors are controlled by keyboards or other monophonic control signal establishing provisions.
  • a keyboard is well-suited for the purpose as keyboards by nature establishes such control signals typically as a so-called MIDI (Musical Instrument Digital Interface)signal, which may be transmitted by simple measures to other relevant devices such as other keyboards, modules, audio processors, sequencers, etc.
  • the control signals provided are typically polyphonic and regarded as well-suited for the purpose of controlling e.g. a harmony processor in real-time.
  • a real-time harmony processing by nature requires a voice input as an input signal simultaneous with the above-mentioned analysis in order to provide the voice material upon which a the harmony processing may be based.
  • the invention relates to method of establishing a harmony control signal controlled in real-time by a guitar audio input signal (GAS), comprising the steps of
  • the harmony control signal may within the scope of the invention comprise the complete signal required for establishment of a harmony.
  • the harmony control signal may advantageously be established by a harmony processor.
  • a pre-stage to this harmony control signal may be established in a separate unit coupled to the harmony processor by means of MIDI, and the pre-stage signal, - the input audio extraction representation (IAER), may then be established as a chord/harmony/scale -extraction information which may be transferred to the harmony processor as chord-forming notes readable on the MIDI input of the harmony processor together with further control signals adding, e.g. transmitted as MIDI exclusive, which may be applied by the harmony processor as a basis for establishing the finally rendering polyphonic harmony.
  • the input audio extraction representation may e.g. comprise so-called pitch class information. Definitions and explanation on pitch class information is explained in detailed description. It may further comprise information related to analysis of the input signals, i.e, at least the polyphonic guitar signal and optionally and preferably also on information obtained from the second input harmony control signal (SIH). Such information may e.g. comprise chord detection, scale detection, automatic key detection, etc. According to an advantageous embodiment of the invention, such information comprises a combination of chord and scale information. Chord information may be based on analysis of the first input harmony input control signal (FIH), e.g.
  • FIH first input harmony input control signal
  • an input from a guitar alone or it may be based on analysis of the first input harmony input control signal (FIH) in combination with the second input harmony control signal (SIH).
  • note information may be combined from the two mentioned inputs.
  • Scale information may e.g. be based on the first input harmony input control signal (FIH) or the second input harmony control signal (SIH) alone or in combination. As the second input harmony control signal is typically monophonic, this signal will be very suitable for scale extraction.
  • a harmony control signal basically relates to signals which may add one or further harmonies to the second input harmony control signal (SIH).
  • SIH the second input harmony control signal
  • a harmony control signal is established on the basis of information extracted from the first input harmony input control signal (FIH) and preferably also the second input harmony control signal (SIH).
  • a correlation between chord and scale may be established for the purpose of e.g. correcting established harmonies on the basis chord detections.
  • the input audio extraction representation (IAER) where the input audio extraction representation is based on both the first and the second input harmony control signal (SIH).
  • SIH input harmony control signal
  • a primitive example may e.g. comprise a note detection from the first input, e.g. a guitar and a one-note detection from the second input and where the detected three notes together may be extracted to be a chord.
  • a polyphonic voice signal (PVS) is provided on the basis of a voice harmony control signal (HCS).
  • HCS voice harmony control signal
  • the polyphonic voice signal is typically an audio signal represented analog or digitally.
  • the polyphonic voice signal may e.g. be established in a separate unit by a state of the art harmony processor under control of the voice harmony control signal (HCS) insofar the harmony processor are able to receive and interpret the control signals.
  • HCS voice harmony control signal
  • a polyphonic voice signal may also be referred to as a harmony voice signal comprising two or more different voice tones.
  • a harmony may e.g. comprise one voice signal basically corresponding to the second input harmony control signal and at least one further harmony tone based on a pitch modulation of the second input harmony control signal, i.e. the voice audio input signal (VAS).
  • VAS voice audio input signal
  • a real-time method is regarded as a method which may be applied for live performance.
  • relatively small delays may appear trough the system, as long as the delay will not result in a delay of the harmony to be produced in a degree that obscures the live performance.
  • the first input harmony input control signal may e.g. be a direct A/D converted version of the guitar audio input signal (GAS) or any derivative or modification thereof.
  • the second input harmony control signal may e.g. be a direct A/D converted version of the voice audio input signal (VAS) or any derivative or modification thereof.
  • the first input control signal representing a guitar audio input signal may typically be formed on the basis of an A/D conversion of an analog input from a guitar.
  • the signal is by nature polyphonic, although, a guitar player may evidently choose to play monophonic from time to time.
  • a guitar input is typically polyphonic and by nature polyphonic when playing chords.
  • the polyphonic voice signal may e.g. be provided by a prior art harmony processor such as the TC Helicon Voice Live by communicating relevant input audio extraction representation (IAER) to the unit by means of MIDI.
  • a prior art harmony processor such as the TC Helicon Voice Live by communicating relevant input audio extraction representation (IAER) to the unit by means of MIDI.
  • GAS guitar audio input signal
  • information, audio extraction representations may be extracted from the input audio guitar signal at very low sample rates, less than about 13 kHz, thereby enabling the complete process of evaluating a polyphonic guitar input signal, extracting the relevant information and establishing a polyphonic voice signal (PVS) on the basis of said input audio extraction representation (IAER) on the basis thereof.
  • PVS polyphonic voice signal
  • a surprising effect of the invention is thus that a real-time establishment of voice harmonies may be established if the sample rate of the controlling polyphonic guitar input signal is sufficiently low.
  • This is definitely a breakthrough in relation to establishment of polyphonic voice harmonies as reliable extraction based on a polyphonic guitar in a live-environment would be expected to rely on not only the fundamental tones but more likely on the harmonics thereof
  • a detection on harmonics would e.g. result in a faster output from e.g. an FFT algorithm as the wavelengths are very low.
  • a harmony control signal controlled in real-time by a guitar audio input signal (GAS) according to any of the claims 1-3, wherein said low rate is less than about 13 kHz and greater than about 1.3 kHz.
  • a sample rate lower than about 13 kHz is sufficient to establish input audio extraction which may be applied for the purpose of real-time establishment of voice harmonies on the basis of a guitar audio input signal.
  • the extraction performed on the input signal may be performed at sufficient high speed in order to obtain the relevant extracted guitar audio information and at the same time provide a voice harmony fast enough to facilitate a live - real-time - establishment of harmonies.
  • an audio extraction representation is obtained through detection of fundamental tones of a guitar input of less than 6.5 kHz, preferably less than through detection of fundamental tones of a guitar input of less than 4.5 kHz.
  • a harmony control signal controlled in real-time by a guitar audio input signal GAS
  • said audio extraction representation is obtained through detection of fundamental tones of a guitar input of less than 3.0 kHz.
  • a first input harmony input control signal (FIH) is provided on the basis of A/D conversion of said guitar audio input signal (GAS).
  • a first input harmony input control signal is provided on the basis of A/D conversion of said guitar audio input signal (GAS) and a subsequent down-sampling of said audio input signal (GAS).
  • the guitar input signal is A/D converted at a relatively high sample rate - such as 44.1 kHz and subsequently down-sampled to a sample rate less than about 13 kHz and greater than about 1.3 kHz.
  • the guitar input signal may be subject to e.g. room processing or other relevant audio processing as a normal audio processing requires a higher sample rate than the sample rate required for the audio extraction representation.
  • an input audio extraction representation is established on the basis of said first input harmony input control signal (FIH) and said second input harmony control signal (SIH).
  • information extraction applicable for the subsequent harmony generation may furthermore be obtained on the basis of analysis of both the first input harmony input control signal (FIH) and the second input harmony control signal (SIH), i.e. typically the a guitar input and a voice input.
  • the audio extraction may thus both rely on information provided and extracted from the guitar input and information provided and extracted from the voice input.
  • the first input harmony input control signal (FIH) and the second input harmony control signal (SIH) may be analyzed in one combined process or two separate processes.
  • the two processes may according to one embodiment of the invention be established as two processes performed in separate hardware.
  • the hardware may e.g. communicate via MIDI.
  • said input audio extraction representation is established on the basis of said first input harmony input control signal (FIH), said second input harmony control signal (SIH) and at least one further input signal (FTS).
  • the at least one further input signal may e.g. comprise an input from a further instrument, monophonic or polyphonic and the further input signal may both comprise an audio signal or e.g. a control signal in the form of e.g. a MIDI signal.
  • Relevant information from the one or a plurality of further input signals may be both polyphonic or monophonic as even monophonic information may be applied for the purpose of deriving very important scale information which, in addition to chord information may result in a strong and efficient control and establishment of the voice harmony control signal HCS and thereby the polyphonic voice signal (PVS).
  • HCS voice harmony control signal
  • PVS polyphonic voice signal
  • said first input harmony input control signal us analyzed in time windows of less than about 1500 ms, preferably less than about 1000 ms.
  • the maximum value strongly relates to the maximum acceptable delay through the system as an extraction which delays the output generation of a polyphonic voice signal too much would comprise the abilities of using the method in live applications.
  • said first input harmony input control signal is analyzed in time windows of more than about 80 ms, preferably more than about 100 ms.
  • the minimum time window relates to the input guitar signal and should be long enough to facilitate detection of the lowest relevant frequency component at the input.
  • such lowest frequency may e.g. have a frequency of about 70-85 Hz.
  • said first input harmony input control signal is analyzed in time windows of 80 ms to 1500 ms.
  • said first input harmony input control signal is analyzed in time windows of 100 ms to 1000 ms.
  • said first harmony input control signal is analyzed in overlapping time windows, preferably by FFT evaluation.
  • Any suitable algorithm for detection of notes may be applied within the scope of the invention when extracting information from each time window.
  • an FFT evaluation or any other suitable derivative thereof may be applied within the scope of the invention.
  • the overlapping time windows are repeated and analyzed in intervals less that the duration of the time window.
  • the overlapping time windows are repeated and analyzed in intervals less than 100ms, preferably less than 50 ms.
  • the overlapping time window may e.g. be repeated in shorter intervals, such as down to about 2 - 4ms. E.g. about 5- to about 40 ms.
  • the input audio extraction representation is established by note detection.
  • input audio extraction representation comprises notes extracted from the guitar audio input. Further information may be extracted.
  • the input audio extraction representation is established by note and chord detection.
  • the input audio extraction representation is established by note and chord and scale detection.
  • the establishing a voice harmony control signal (HCS) and/or the polyphonic voice signal (PVS) is provided on the basis of said input audio extraction representation (IAER) and said second input harmony control signal (SIH) and wherein said polyphonic voice signal (PVS) is established as an output signal time-synchronized with said second input harmony control signal (SIH).
  • the establishing a voice harmony control signal (HCS) and/or the polyphonic voice signal (PVS) is provided on the basis of said input audio extraction representation (IAER) and said second input harmony control signal (SIH) and wherein said polyphonic voice signal (PVS) is established as an output signal time-synchronized with said first input harmony input control signal (FIH).
  • the polyphonic harmony is established as a harmony of two or more different voices and wherein one of the voices is based on the second input harmony control signal (SIH) and wherein the at least another of the voices are based on a pitch shifted version of said second input harmony control signal (SIH) and wherein said two or more voices are time-syncronized with said second input harmony control signal (SIH).
  • the input audio extraction representation deduces chords on the basis said first input harmony control signal (FIH) and scale information on the basis of said second input harmony control signal (SIH).
  • a hardware structure comprising a signal processor for implementing the method of claims 1-23.
  • the required hardware structure may comprise any suitable prior art signal processor, any suitable associated memory, cache, bus, store and audio converters.
  • the hardware is divided into two separate units, the first unit comprising an input () for said first input harmony control signal (FIH) the second unit comprising an input () for said second input harmony control signal (SIH) wherein the two units communicates via a digital interface, e.g. a MIDI interface.
  • a digital interface e.g. a MIDI interface
  • a hardware structure comprising a signal processor for implementing the method of claims 1-23, wherein the hardware is implemented on a computer such as a PC or a Macintosh.
  • the hardware is implemented in a stand alone unit.
  • the standalone unit may preferable comprise a foot-controlled pedal.
  • the first input harmony input control signal (FIH) is established by a high resolution A/D converter.
  • the prior art does not consider using relevant information from several simultaneous audio sources and/or user inputs.
  • relevant additional or supporting information is available from using several sources which together form a much more robust basis for accurate determination of key scale and harmony determination relevant information.
  • additional sources may e.g. be several voice inputs, string instrument inputs, keyboard and piano audio, brass instrument inputs as well as audio and midi information from electronic instruments sources and accompanying electronically based playback devices.
  • additional relevant sources may be applied.
  • the emerging appearance of audio networks allows a further relevant facility to provide the above relevant additional information for accurate as well as enhanced harmony determination.
  • harmony generation is primarily a desire to quickly and accurately detect the relevant actual key and scale in which a piece of music is played, secondarily on the basis of sung notes to produce harmony generating outputs in accordance with default or desired harmony types.
  • the harmony generation may take such form as to produce less advanced harmonies in the case that the available extracted information is of a less substantial or robust character, and visa versa produce more advanced harmonies as the available information is of a more substantial or robust character.
  • Pitch octave profile pitch harmony relevant information may be kept to enhance or substantiate and increase the harmony decision logic process to increase robustness and accuracy.
  • Schemes for increasing robustness include statistical and repeated value type logic as well as neural networks type processing to allow such improvements.
  • the device may advantageously comprise a filtering and down-sampling module which down-samples the signal to lower sampling rate; a time domain partitioning module which partitions the time domain signal into a sequence of overlapping short segments (frames); a frame attributes check module which checks the properties of each frame; a windowing and FFT(Fast Fourier Transform) processing module which transforms each valid frame into frequency domain; a pitch class profile estimation module which estimates pitch class profile from the frequency spectrum; a harmony extraction module which extracts important MIDI notes for harmony.
  • a filtering and down-sampling module which down-samples the signal to lower sampling rate
  • a time domain partitioning module which partitions the time domain signal into a sequence of overlapping short segments (frames)
  • a frame attributes check module which checks the properties of each frame
  • a windowing and FFT(Fast Fourier Transform) processing module which transforms each valid frame into frequency domain
  • a pitch class profile estimation module which estimates pitch class profile from the frequency spectrum
  • the filtering and down-sampling module may down-sample the input audio signal to be no more than 12 kHz and no less than 1.3 kHz for efficient and reliable common harmony information extraction processing.
  • This module can optionally apply a low-cut filter with cut-off frequency no more than 85 Hz to reduce low frequency noise.
  • the time domain partitioning module may divide the down-sampled signal into a sequence of overlapping frames with step size no more than 100 milliseconds.
  • the duration of each frame should be no more than 1 second and no less than 100 milliseconds.
  • a frame attribute check module may check the attributes of each frame to determine whether the frame is suitable for harmony information processing using one or multiple checking methods such as voiced/unvoiced check and frame energy check.
  • a windowing and FFT processing module which may window each frame and may transform the windowed frame into frequency domain using FFT.
  • the windowing function may include Hanning, Hamming, etc.
  • a pitch class profile estimation module may compute the pitch class profile based on the frequency spectrum. It calculates the strength of a semitone using the peaks found within the frequency span of the semitone. It then calculates the pitch class strength profile by summing the strength of semitones that belong to the same pitch class.
  • a chord estimation module may extract important harmony notes based on the pitch class profile, the music key and scale, and optionally with lead vocal input and historical data.
  • a harmony information extraction system may optionally detect the best matching key and scale dynamically based on/or supplemented by a MIDI note history received from a MIDI generating device.
  • a harmony information extraction device can output standard MIDI output to drive an existing harmony product with MIDI interface.
  • a harmony information extraction device may feature an enable/disable interface to allow a user to engage or disengage the harmony information processing, e.g. by foot control.
  • a harmony information extraction device can optionally provide a guitar tuner which tunes a guitar with a reference frequency of 440 Hz.
  • a harmony information extraction device can optionally provide the interface to allow a user to specify a key and scale either through manual selection or by playing on a guitar.
  • a harmony information extraction device can optionally provide the interface to allow a user to specify the playing style.
  • a harmony information extraction device can be optionally changed to a non real-time mode to handle the corresponding functionalities.
  • One of several objects of the present invention is to overcome the drawbacks in the prior art methods and apparatuses, and to provide a real-time harmony information extraction device which is capable of extracting important harmony information quickly and accurately.
  • the system first filters and down-samples the guitar input signal. It then partitions the down-sampled signal into overlapping short-time segments (frames). For each frame, it checks its attributes to determine whether this frame is suitable for further analysis. Each valid frame is windowed and transformed into the frequency domain through FFT to obtain the frequency spectrum. The system then computes the pitch class profile using the peaks detected within a frequency span of a semitone. It then determines the important harmony notes using the pitch class profile, the music key and scale, and optionally with historical data and vocal input.
  • the system down-samples the input signal to be no more than 12 kHz and no less than 1.3 kHz for efficient and reliable guitar MIDI note extraction. It can optionally cut frequency no more than 85 Hz to remove low frequency noise.
  • the system partitions the filtered and down-sampled signal into overlapping frames with step size no more than 100 milliseconds.
  • the interval of each frame is no more than 1 second and no less than 100 milliseconds.
  • the small step size improves the time resolution of information extraction, and reduces the processing latency.
  • the system checks the attributes of each frame to determine whether the frame is suitable for harmony information extraction.
  • the goal is to skip frames that are not suitable for harmony information extraction.
  • each selected frame is windowed and transformed into frequency domain to compute the pitch class profile.
  • the system determines the strength of each semitone using peaks detected within the frequency span of the semitone, and computes the pitch class profile by adding the note strengths that belong to the same pitch class.
  • the system determines the important MIDI notes by considering the pitch class profile, the music key and scale, and optionally considers vocal input and historical data.
  • Down-sampling is the process of reducing the sampling rate of a signal. This is usually done to reduce the data rate or the size of the data.
  • the down-sampling factor (commonly denoted by M) is usually an integer or a rational fraction greater than unity. This factor multiplies the sampling time or, equivalently, divides the sampling rate.
  • MIDI Musical Instrument Digital Interface
  • MIDI does not transmit an audio signal or media but simply transmits digital data "event messages" such as the pitch and intensity of musical notes to play, control signals for parameters such as volume, vibrato and panning, cues and clock signals to set the tempo.
  • event messages such as the pitch and intensity of musical notes to play
  • control signals for parameters such as volume, vibrato and panning, cues and clock signals to set the tempo.
  • Fig. 1A shows the general hardware structure of a guitar extraction unit to be used in a harmony information extraction system according to one of several embodiments of the invention.
  • the system comprises a digital signal processor 2, a microprocessor 8, an A/D converter 1, a UART 5, and input/output ports GUITAR INPUT, MIDI OUT.
  • Both the digital signal processor 8 and the microprocessor can have ROMs 6,3 and RAMs 7,4 to store the required program and data.
  • the digital signal processor runs the input audio extraction processing algorithm while the microprocessor handles the user interface.
  • the A/D converter converts the analog guitar input into digital form white the UART transmits the MIDI information.
  • the system can be expanded to comprise multiple A/D converters and UARTs to handle additional inputs and output signals.
  • the system may moreover interact with user controls 9 and display(s) 10.
  • a polyphonic voice signal may then be generated by a harmony processor (not shown) connected to the hardware structure via a MIDI connection.
  • the voice harmony may e.g. be a TC Helicon VoiceWorks Harmony FX Voice Processor controlled by MIDI.
  • This harmony processor comprises a voice input and it may generate harmonies on the basis of the harmony processing algorithm of this unit and under real-time control by the output signal of the guitar extraction unit of fig. 1A .
  • the structure of fig. 1A may also e.g. be modified to include a harmony processor, thereby rendering the MIDI outbound connection superfluous.
  • Fig. 1B illustrates a method of establishing a harmony control signal controlled in real-time by a guitar audio input signal according to an embodiment of the invention.
  • the embodiment may e.g. be implemented in a system which the hardware structure of fig. 1A forms part of.
  • the methods involves the steps of a establishing a harmony control signal controlled in real-time on the basis of a guitar audio input signal GAS fed to a corresponding hardware structure via a guitar input.
  • a first input harmony control signal FIH is then generated on the basis of said guitar audio input signal GAS and this signal is subsequently analyzed for the purpose of generating an input audio extraction representation IAER.
  • This input audio extraction representation may e.g. comprise note or chord information derived from the polyphonic guitar audio signal GAS.
  • the first harmony input control signal FIH may e.g. comprise a straightforward A/D conversion of guitar audio signal GAS or any processed modification thereof.
  • the input audio extraction representation IAER may be based on the first input harmony control signal FIH alone or e.g. advantageously in combination with said second input harmony control signal SIH.
  • the method involves the steps of providing a second input harmony control signal SIH on the basis of a voice audio input signal VAS.
  • the voice audio input signal VAS is obtained from a voice input.
  • a harmony control signal HCS may be established.
  • This signal is understood to be a decision making for the purposes of establishing harmonies fitting to the input signals, e.g. the second and the first input signal.
  • Such harmony decision may e.g. primitively include that a note "E” must be established as a harmony if the second input harmony signal, representing a voice, turns out to be a "C” and that the a has been extracted to be "C-major".
  • Such decision-making algorithms may be more or less complicated.
  • extraction and harmony extraction may be performed by means of neural networks, i.e. artificial intelligence.
  • neural networks i.e. artificial intelligence.
  • the method establishes a polyphonic voice signal PVS on the basis of said input audio extraction representation IAER and said second input harmony control signal SIH.
  • the second input harmony control signal SIH is applied as a signal upon which a harmony generation is based, e.g. by adding further pitch modulated voices.
  • the second input harmony control signal may optionally and advantageously be subject to input audio extraction as well thereby adding further information to the input audio extraction representation IAER.
  • Such information may e.g. comprise scale information as the voice input signal is typically monophonic.
  • Information obtained from the first input harmony input control signal FIH may typically relate to chord or harmony relevant extractions.
  • the input audio extraction representation IAER is then applied for establishment of a polyphonic voice signal PVS on the basis of said input audio extraction representation IAER and said second input harmony control signal SIH, i.e. a voice input signal.
  • the voice input signal is typically obtained by a microphone of any suitable kind.
  • a control signal may be obtained from one or further instruments, polyphonic or monophonic.
  • the further instruments may also include a further monophonic voice input.
  • chord or scale information may be supplemented in an easy and effective way, thereby improving the quality or the generation speed of the input audio extraction representation IAER.
  • Fig. 2a shows an application of the harmony information extraction system.
  • the harmony information extraction device functions as an independent unit.
  • the MIDI outputs are sent to a harmony generation device to control harmony.
  • the harmony generation device may e.g. be a TC Helicon VoiceWorks Harmony FX Voice Processor controlled by MIDI.
  • Fig. 2b An alternative form of this application is shown in Fig. 2b .
  • the harmony information extraction unit functions inside a harmony generation device.
  • Fig. 3 shows the block diagram of a harmony information extraction algorithm according to an embodiment of the invention.
  • the guitar audio input is sampled with a suitable sampling rate such as 44.1 kHz.
  • the filtering and down-sampling module acts accordingly to down-sample the signal to a sampling rate that is no more than 12 kHz and no less than 1.3 kHz.
  • the time domain partitioning module partitions the down-sampled signal into a sequence of overlapping frames.
  • the duration of each frame is no more than 1 second and no less than 100 milliseconds.
  • the step size between two consecutive frames is no more than 100 milliseconds. It then checks the attributes of each frame to determine whether this frame is suitable for further analysis with one or a combination of multiple measures.
  • Each valid frame is windowed and transformed into the frequency domain through FFT to obtain the frequency spectrum.
  • the system then computes the pitch class profile using the peaks detected within a frequency span of a semitone. Finally, it determines the important harmony notes based on the pitch class profile, the music key and scale, and optionally with vocal input and historical data.
  • Fig. 4 shows the general flow diagram of the filtering and down-sampling module according to an embodiment of the invention.
  • a low-cut filter is applied to the input to reduce low-frequency noise. Then the signal goes through an anti-aliasing filter followed by the down-sampling operation.
  • the purpose of the anti-aliasing filter is to remove high frequency components that could cause aliasing during the down-sampling operation.
  • the lowest note on a regularly tuned guitar is E2, which is 82.4 Hz (assuming the tuning reference is 440Hz). Sometimes, a player may intentionally tune the lowest note to D2, which is 73.4 Hz.
  • a low-cut filter with cut-off frequency of no more than 85 Hz can be used to reduce low frequency noise.
  • the highest note played on a guitar varies with the number of frets available (or playable). On a 21-fret guitar, the highest note you can play is C#6, which is 1108.7 Hz. The highest note in a playable chord is typically lower than this value.
  • down-sampling is desirable for efficient DSP operations.
  • the system can also sample the input audio signal directly at a lower sampling rate.
  • the absolute minimum sampling rate should be no less than 1.3 kHz to be able to process commonly used guitar chords. With the power of DSP hardware continues to increase, it is possible to process the signal at higher sampling rates such as 11 kHz (or 12 kHz for a 48 kHz input).
  • the purpose of the setup is to convert a guitar audio input signal GAS into a sample rate appropriate and applicable with the invention.
  • An alternative to this process of converting a relatively high speed and subsequently down-sampling the signal may e.g. be an initial A/D conversion directly into the desired low sample rate.
  • the applied A/D converted must be a high-resolution converter such as a delta-sigma or a PWM A/D converter.
  • the harmony note extraction contains a frame attribute check module which checks the properties of each frame to determine whether this frame is suitable for harmony information extraction processing.
  • the guitar audio input can contain many segments that do not contain any useful harmony information. It is crucial to skip these segments.
  • the system can utilize one or a combination of multiple techniques to check the attributes of each frame. These techniques include but are not limited to voiced/unvoiced check, energy check, etc.
  • Fig. 5 shows the general flow diagram of one example of a pitch class profile estimation module according to an embodiment of the invention.
  • the module first estimates the strength of each semitone based on the frequency spectrum, which can be obtained through either FFT or constant Q transform. If FFT is used, the system estimates the strength of each semitone by finding the peaks within the frequency span of each semitone. The system can utilize the maximum peak found for each semitone, and use it to represent the strength of that semitone. The system then adds semitone strengths that belong to the same pitch class to obtain the pitch class profile. Alternatively, the system may use all the peaks presented in the frequency span of a semitone and averaging them before summing.
  • the pitch class profile estimation module can optionally apply either a fixed or a variable threshold (or a combination of both) to the semitone strength so that only the semitone strength that exceeds the threshold is used for pitch class profile estimation.
  • Fig. 6 shows an alternative approach for pitch class profile estimation.
  • the system estimates the strength of each semitone as discussed above. After that, the system first selects the unique pitch class candidates. Then it refines the unique pitch class among the pitch class candidates. Finally, the system calculates the strength of each unique pitch class by adding the strength of its harmonics or sub-harmonics.
  • Fig. 7 shows a flow diagram of a chord estimation module according to one embodiment of the invention.
  • the system first selects the best match chord candidate by comparing the pitch class profile with the default chord patterns. Then it checks to see if the chord candidate is the same as the previous chord displayed. If the chord candidate is the same as the previous chord, the system skips the remaining steps and returns. Otherwise, the system checks if the current chord candidate is different from the previous chord candidate. If the current chord candidate is different from the previous chord candidate, the system updates both the previous chord candidate and its counter. Otherwise, it simply increases the counter of the previous chord candidate. Then it checks to see if the previous chord candidate counter exceeds the pre-determined threshold. If yes, the system outputs this chord and update previous chord. It also reset previous chord candidate and its counter. If no, the system returns.
  • chord priorities can be assigned as shown in table 1.
  • Table 1 An example chord priority table when the key is C major. High Priority C, Dm, Em, F, G, Am, Gsus, C_M7, F_M7, D_m7, A_m7, G_m7, C_m7, F_m7 Medium Priority Fm, Bb, D, Csus, Gm, Fsus, F_m7, D_m7 Low Priority Bdim, B_dim7 None All other chords
  • chord priorities can be used in conjunction with chord likelihood to select the best chord candidates.
  • Fig. 8 shows the general steps involved in finding the best chord candidates.
  • the system first computes the likelihood of each chord type by matching the pitch class profile with the default chord patterns. For example, [1,0,0,0,1,0,0,1,0,0,0,0] can be used to represent C major.
  • the matching process can be carried out by taking the inner product of the pitch class profile and the default chord patterns.
  • the pitch class profile vector is shifted one element at a time to find the correct root of each chord.
  • one can also use a weighted default chord vector by utilizing either neural networks or machine learning techniques.
  • the system determines the top matching chord candidates by sorting the likelihood values. It then determines the best matching chord candidate either by select the chord with the highest likelihood value or considers chord likelihood values in combination with chord priorities and chord history.
  • harmony note extraction module within the scope of this invention is to utilize music key and scale information, pitch class profile information, and optionally with historical data and vocal input to extract important harmony notes.
  • the music key and scale information can be obtained through user manual input, historical data, or guitar input.
  • the algorithm can optionally detect/adapt to a player's style for better decision making.
  • Fig. 9 illustrates a method of establishing a harmony control signal controlled in real-time by a guitar audio input signal according to an embodiment of the invention.
  • the embodiment may e.g. be implemented in a system which the hardware structure of fig. 1A forms part of.
  • the method basically corresponds to the method described above with reference to fig. 1B , but now the method has been implemented in two separate hardware units 100 and 200.
  • the first hardware unit 100 is dedicated for the receipt of a first input harmony control signal FIH and the second hardware unit 200 is dedicated for the receipt of a second input harmony control signal SIH.
  • the first input harmony control signal FIH may typically comprise a polyphonic guitar input signal received through a dedicated input.
  • the second input harmony control signal SIH may typically comprise a voice signal received through a dedicated input.
  • the second hardware unit 200 may e.g. comprise a TC Helicon VoiceWorks Harmony FX Voice Processor controlled by MIDI received from the first hardware unit 100.
  • the methods involves the steps of a establishing a harmony control signal controlled in real-time on the basis of a guitar audio input signal GAS fed to a corresponding hardware structure via a guitar input.
  • a first input harmony control signal FIH is then generated on the basis of said guitar audio input signal GAS and this signal is subsequently analyzed for the purpose of generating an input audio extraction representation IAER.
  • This input audio extraction representation may e.g. comprise note or chord information derived from the polyphonic guitar audio signal GAS.
  • the first harmony input control signal FIH may e.g. comprise a straightforward A/D conversion of guitar audio signal GAS or any processed modification thereof.
  • the input audio extraction representation IAER may be based on the first input harmony control signal FIH alone or e.g. advantageously in combination with said second input harmony control signal SIH.
  • the method involves the steps of providing a second input harmony control signal SIH on the basis of a voice audio input signal VAS.
  • the voice audio input signal VAS is obtained from a voice input.
  • a harmony control signal HCS may be established.
  • This signal is understood to be a decision making for the purposes of establishing harmonies fitting to the input signals, e.g. the second and the first input signal.
  • Such harmony decision may e.g. primitively include that a note "E” must be established as a harmony if the second input harmony signal, representing a voice, turns out to be a "C” and that the a has been extracted to be "C-major".
  • Such decision-making algorithms may be more or less complicated.
  • extraction and harmony extraction may be performed by means of neural networks, i.e. artificial intelligence.
  • neural networks i.e. artificial intelligence.
  • the method establishes a polyphonic voice signal PVS on the basis of said input audio extraction representation IAER and said second input harmony control signal SIH.
  • the second input harmony control signal SIH is applied as a signal upon which a harmony generation is based, e.g. by adding further pitch modulated voices.
  • the second input harmony control signal may optionally and advantageously be subject to input audio extraction as well thereby adding further information to the input audio extraction representation IAER.
  • Such information may e.g. comprise scale information as the voice input signal is typically monophonic.
  • Information obtained from the first input harmony input control signal FIH may typically relate to chord or harmony relevant extractions.
  • the input audio extraction representation IAER is then applied for establishment of a polyphonic voice signal PVS on the basis of said input audio extraction representation IAER and said second input harmony control signal SIH, i.e. a voice input signal.
  • the voice input signal is typically obtained by a microphone of any suitable kind.
  • a control signal may be obtained from one or further instruments, polyphonic or monophonic.
  • the further instruments may also include a further monophonic voice input.
  • chord or scale information may be supplemented in an easy and effective way, thereby improving the quality or the generation speed of the input audio extraction representation IAER.
EP07005051A 2007-03-12 2007-03-12 Procédé pour établir un signal de contrôle d'harmonie contrôlé en temps réel par un signal d'entrée de guitare Withdrawn EP1970892A1 (fr)

Priority Applications (1)

Application Number Priority Date Filing Date Title
EP07005051A EP1970892A1 (fr) 2007-03-12 2007-03-12 Procédé pour établir un signal de contrôle d'harmonie contrôlé en temps réel par un signal d'entrée de guitare

Applications Claiming Priority (1)

Application Number Priority Date Filing Date Title
EP07005051A EP1970892A1 (fr) 2007-03-12 2007-03-12 Procédé pour établir un signal de contrôle d'harmonie contrôlé en temps réel par un signal d'entrée de guitare

Publications (1)

Publication Number Publication Date
EP1970892A1 true EP1970892A1 (fr) 2008-09-17

Family

ID=38283687

Family Applications (1)

Application Number Title Priority Date Filing Date
EP07005051A Withdrawn EP1970892A1 (fr) 2007-03-12 2007-03-12 Procédé pour établir un signal de contrôle d'harmonie contrôlé en temps réel par un signal d'entrée de guitare

Country Status (1)

Country Link
EP (1) EP1970892A1 (fr)

Cited By (1)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
JP2020038328A (ja) * 2018-09-05 2020-03-12 国立大学法人秋田大学 コード認識方法、コード認識プログラム、及びコード認識システム

Citations (3)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
EP0282458A2 (fr) * 1987-02-06 1988-09-14 KETRON S.r.l. Dispositif automatique pour la reproduction simultanée de notes ayant des intervalles de fréquence musicale préétablis fournis par des tables
EP0729130A2 (fr) * 1995-02-27 1996-08-28 Yamaha Corporation Dispositif de karaoke générant une voix d'accompagnement synthétique ajoutée à une voix chantée
US6124544A (en) * 1999-07-30 2000-09-26 Lyrrus Inc. Electronic music system for detecting pitch

Patent Citations (3)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
EP0282458A2 (fr) * 1987-02-06 1988-09-14 KETRON S.r.l. Dispositif automatique pour la reproduction simultanée de notes ayant des intervalles de fréquence musicale préétablis fournis par des tables
EP0729130A2 (fr) * 1995-02-27 1996-08-28 Yamaha Corporation Dispositif de karaoke générant une voix d'accompagnement synthétique ajoutée à une voix chantée
US6124544A (en) * 1999-07-30 2000-09-26 Lyrrus Inc. Electronic music system for detecting pitch

Non-Patent Citations (3)

* Cited by examiner, † Cited by third party
Title
DIGITECH: "Vocalist Live 2", WEBSITE, 20 February 2007 (2007-02-20), Internet, XP002445565, Retrieved from the Internet <URL:http://web.archive.org/web/20070220102230/http://www.digitech.com/products/Vocalist_Live2/> *
DIGITECH: "Vocalist Live 2, Vocal Harmony and Effects Processor", OWNER'S MANUAL, February 2007 (2007-02-01), INTERNET, XP002445566 *
VAIL M: "DIGITECH VHM 5 VOCALIST. ÖVOCAL HARMONY PROCESSOR", KEYBOARD, MUSIC PLAYER NETWORK, SAN FRANSISCO, CA, US, vol. 18, no. 11, 1 November 1992 (1992-11-01), pages 146 - 150, XP000307521, ISSN: 0730-0158 *

Cited By (1)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
JP2020038328A (ja) * 2018-09-05 2020-03-12 国立大学法人秋田大学 コード認識方法、コード認識プログラム、及びコード認識システム

Similar Documents

Publication Publication Date Title
US7667126B2 (en) Method of establishing a harmony control signal controlled in real-time by a guitar input signal
US10283099B2 (en) Vocal processing with accompaniment music input
Durrieu et al. A musically motivated mid-level representation for pitch estimation and musical audio source separation
JP6290858B2 (ja) 発話の入力オーディオエンコーディングを、対象歌曲にリズム的に調和する出力へと自動変換するための、コンピュータ処理方法、装置、及びコンピュータプログラム製品
Muller et al. Signal processing for music analysis
US7582824B2 (en) Tempo detection apparatus, chord-name detection apparatus, and programs therefor
CN110634501A (zh) 音频提取装置、机器训练装置、卡拉ok装置
US20100192753A1 (en) Karaoke apparatus
EP1701336B1 (fr) Appareil et procédé de traitement du son, et programme correspondant
CN112382257B (zh) 一种音频处理方法、装置、设备及介质
CN111418005A (zh) 声音合成方法、声音合成装置及程序
EP3610477A1 (fr) Accompagnement autogénéré provenant du chant d&#39;une mélodie
Lerch Software-based extraction of objective parameters from music performances
EP1970892A1 (fr) Procédé pour établir un signal de contrôle d&#39;harmonie contrôlé en temps réel par un signal d&#39;entrée de guitare
JP5728829B2 (ja) 電子音楽装置およびハーモニー音生成方法を実現するためのプログラム
Freire et al. Study of the tremolo technique on the acoustic guitar: Experimental setup and preliminary results on regularity
Verma et al. Real-time melodic accompaniment system for indian music using tms320c6713
JP5310677B2 (ja) 音源分離装置、及びプログラム
JP2000010597A (ja) 音声変換装置及び音声変換方法
CN100359559C (zh) 自动音源相似度比对方法及装置
JP2000003200A (ja) 音声信号処理装置及び音声信号処理方法
JPH11143460A (ja) 音楽演奏に含まれる旋律の分離方法、分離抽出方法および分離除去方法
JP2003216147A (ja) 音響信号の符号化方法
Saranya et al. Orchestrate-A GAN Architectural-Based Pipeline for Musical Instrument Chord Conversion
Molina et al. Dissonance reduction in polyphonic audio using harmonic reorganization

Legal Events

Date Code Title Description
PUAI Public reference made under article 153(3) epc to a published international application that has entered the european phase

Free format text: ORIGINAL CODE: 0009012

AK Designated contracting states

Kind code of ref document: A1

Designated state(s): AT BE BG CH CY CZ DE DK EE ES FI FR GB GR HU IE IS IT LI LT LU LV MC MT NL PL PT RO SE SI SK TR

AX Request for extension of the european patent

Extension state: AL BA HR MK RS

AKX Designation fees paid
REG Reference to a national code

Ref country code: DE

Ref legal event code: 8566

STAA Information on the status of an ep patent application or granted ep patent

Free format text: STATUS: THE APPLICATION IS DEEMED TO BE WITHDRAWN

18D Application deemed to be withdrawn

Effective date: 20090318