EP2099024B1 - Method for acoustic object-oriented analysis and note object-oriented processing of polyphonic sound recordings - Google Patents

Description

The present invention relates to a method for sound object-oriented analysis and for note object-oriented processing of polyphonic sound recordings according to claim 1.

It has long been known to subject sound recordings with musical content to a sonic reworking. In the studio technique of earlier years, this still took place with the aid of expensive hardware components, such as e.g. With complex filter banks, computer and special computer programs are used today, which are far less expensive and therefore have found a wider distribution. Another contributing factor to this development has been the advent of digital recording technology. The aim of such post-processing is usually to improve the sound of the recordings or incorporate sound effects. This kind of tonal post-processing has in common that it works purely effektorientiert and the musical content of the signal is not able to recognize, but understands the audio signal only as a time-varying signal amplitude.

In the prior art, for example, from the EP 0 750 776 B1 or the DE 696 14 938 T2 a method and apparatus for changing the sound and pitch of audio signals is known. It is considered disadvantageous that it is not possible to work with complex sound material there, as is the case with conventional music production.

It is desirable to process audio material at the level of the single notes from which the sound recording is made. It is known in the art to extract single notes from an audio recording in view of their note height, note length and time of occurrence. Such a note extraction is eg from the DE 10 2004 049 477 A1 known to determine a melody line from an audio signal. From the WO 02/084641 A1 For example, the transfer of an audio signal into a note-based description is known in order to be able to reference the audio signal in a database. Editing the extracted notes, eg by changing the frequency or shifting in time, does not take place. These documents cite further prior art.

A particularly critical issue with audio material processing is that the original sound impression is e.g. a vocal part should be preserved even after editing. This succeeds excellently in the state of the art software "Melodyne" of Celemony Software GmbH, which is based on a note-based approach. However, this software assumes that there is unanimous material. Chord instruments such as guitar, piano or choir singing can not be satisfactorily processed sound-based. So far, such chord recordings could only be cut in chords or edited by time-stretching in time or pitch, but without access to the individual notes of a chord. Thus, it has not been possible to change a single chord tone (e.g., the E of a C major chord) (e.g., to Es for C minor) without copying the other notes of the chord at the same time.

However, chord recognition and approaches for identifying individual notes are already known in the prior art, eg for printing notes (software WIDI) or for automatically recognizing titles ( DE 10 2004 049 477 A1 ).

M. Every and John E. Szymanski: Separation of Synchronous Pitched Notes by Spectral Filtering of Harmonics, IEEE Transactions on Audio, Speech and Language Processing, Vol. 5 September 2006 , shows the decomposition of a total sound into individual sounds that can be assigned to notes (Section II. C, page 1847, right column middle to end of the first paragraph). It is described that in a spectrum (spectral amplitude over the frequency, Fig. 1 ) Peaks ("peak picking") belonging to notes. With some mathematical effort, a limited number of notes is taken out of the abundance of peaks. After this note determination step, a filter is specified by means of which the signal signals assigned to the notes are extracted from the total signal. After the separation of the notes, a residual signal remains, in which different parts are contained. A determination of event objects and a link of such event objects with note objects does not take place here. In general, it is only an analysis and subordinate to a breakdown of the total signal into individual notes. There is no editing of individual note objects.

Chris Duxbury, Nicolas Chetry, Mark Sandler and Mike Davies: An Efficient Two-Stage Implementation of Harmonic Matching Pursuit, EUSIPCO 2004, XP-002533839 explains that in the case of note extraction from an overall sound, overtones must be taken into account. It is therefore calculated a so-called harmonic energy by including at a certain frequency and energies to multiples of this frequency. This harmonic energy is calculated for different fundamental frequencies and the fundamental frequency taken as the actual note frequency, which provides the highest harmonic energy value. The overall sound signal is subsequently reduced by a signal to be assigned to the note. There is no identification of percussive objects and no assignment to time-coincident musical objects. Editing note objects by a user does not occur. Only note objects for monophonic sound recordings are to be identified.

It is the object of the present invention to show a way in which a notation-oriented processing of polyphonic sound material can take place. This object is achieved by a method according to claim 1. Advantageous embodiments are specified in the subclaims.

In a first step, the method according to the invention automatically identifies musical objects in the sense of notes in a recorded audio material. These objects are then sonically extracted from the overall recording in a second step and thereby made manipulatable without noticeable loss of sound or noticeable distortion of the sound. This sonic extraction is not found in the prior art and means that a part of the overall sound is assigned to an identified note object. Thus, the original recording can even be changed musically in such a way that the notes can change their relative position to each other in pitch and time, but the original sound impression is retained. The method further provides that the identified note objects are made available to the user for editing. The user can then change individual or several musical objects, eg in the pitch. After such a processing by a user, the sound reproduction, the resynthesis by the modified object is played together with the non-changed objects or with the reduced by the changed object total signal. The starting material can already be present in digital form or as an analog signal. Before the analysis of an analog signal, digitization may then have to be performed.

The method can find various applications. It can be e.g. deliberately eliminate individual mutilers: a pianist accidentally struck a note too much, which is removed in post-production. Another application is the re-tuning, so the correction for a detuned guitar or a dirty string replacement. Temperate recordings can be transferred to pure mood. It can be re-harmonized recordings, such as the riff of a guitar from C major in F minor. So far, a chord could be moved only in pitch overall, but not changed the harmonic relationship of its individual tones. The possibilities even extend to a new composition, because there is access to the single tones.

The method according to the invention is aimed primarily at recorded individual tracks during music production. It is also applicable to ready mixed titles to make them appear in a new musical garb.

With the previous technique, the above was only possible for tonal monophonic material, ie for processing, for example, isolated recorded vocal or brass parts in which only one note with identifiable pitch sounds simultaneously. With the method according to the invention, the goal of single-note editing becomes feasible for polyphonic instruments, ie those in which several notes or whole chords are usually heard simultaneously, such as the piano, guitar, etc., whereby not only the chord as a whole becomes transposable (ie change the pitch while retaining the relative pitch references within a chord), but also and especially the notes within a chord sound can be changed relative to each other, such as changing a chord sound from major to minor.

The method primarily refers to already recorded musical material, not to analysis and processing in "real time", i. at the moment of musical performance. This has to do with the fact that the method according to the invention presupposes a meaningful recognition of "notes" in the sense of closed objects in a certain context. Therefore, while the analysis can be carried out in the background during an ongoing recording, it has to be based on an already recorded time relationship of the order of magnitude e.g. of a few seconds.

The method is designed to find and distinguish individual pitches or pitch curves in the overall signal, but not to separate individual sound sources. So it's not about, e. to separate the individual sources of sounds or voices from a recording of the sounds on one or more speakers in a room. As a rule, two notes played at the same pitch by two different instruments simultaneously are identified as a single object, as is the sound of many First Violins in the orchestra playing the same note as a single note. The notion of the note object predominantly used here emphasizes that "notes" in the sense of this invention need not mean notes in the actual musical sense, although an identified note object may or may not correspond to a note in the actual musical sense.

Notes to be found need not have a constant pitch over time, such as a piano tone, but can also have a pitch progression over time, such as a vibrato and / or portamento vocal. If such pitch gradients are consistent in themselves, the notes remain detectable as the signal tracks time. In this respect, two tones that intersect at different pitch in the pitch can be identified as two different notes.

The procedure essentially consists of the two main steps a) identification of the individual objects involved in the overall sound, ie the notes and, if necessary, the more eventful / percussive sound events, and b) sound decomposition of the overall sound into the found individual objects whose sum gives the overall sound, each of which, however, can be separately manipulated without affecting the sound of the other objects or the overall sound in the sense of unwanted audible artifacts. Especially in step b), the present invention differs from the prior art.

The result of the identification step a) can also be used alone without the sound decomposition step b), if it is only about the recognition and presentation of the musical content, without interfering with the sound itself. This can be the case if you want to create a musical notation from an existing recording or if you want to graphically illustrate the musical content in a different way. Or it may serve to recognize and name the sequence of musical harmonies, e.g. to enrich the music with other instruments.

The quality of the result of identification step a) also influences the quality of the sound decomposition in step b). If step a) only serves to determine the musical content as mentioned above, it usually suffices to determine that a sound object occurs at a certain time, for example the pitch of the note "cis", and how long this object sounds. If, on the other hand, the tonal decomposition in step b) is the goal, then it is advantageous to evaluate as much information as possible about the course and parameters of the note objects found, such as the exact course of the pitch curve over time, the amplitude of the object and theirs Change in the time, the manner of inserting a note, the consistency of the mixture of their partials in relation to notes of similar pitch in the same recording, etc .. Depending on the claim but you can also do without certain information.

As described, the main characteristic of the individual objects to be found in the audio material is that they have a consistent pitch or consistent pitch progression. Furthermore, they should have a consistent progression of their individual partials. That is, it is assumed that a sound object to be found consists of the course of a fundamental tone, as well as an essentially unlimited number of harmonics, which should be approximately integer multiples of the fundamental tone frequency. Furthermore, the course of the partials belonging to a sound object should not have any unmotivated jumps. These assumptions are derived from the properties of the sounds produced by natural instruments. Therefore, the method according to the invention may have its limits where music has been synthesized arbitrarily arbitrarily.

The objects to be identified in the audio material and described above may be referred to as "sonic" objects, in the claims they are named as note objects. Its main characteristic is to have a pitch or pitch curve over a perceptible duration, and the waveform of its time signal behaves essentially periodically or quasi-periodically. It distinguishes the non-sound objects, the noisy objects. Event-aware objects are a subset of the noisy objects.

The event objects are characterized by the fact that they have a sudden increase in amplitude in the time signal and therefore are not periodic at this point. In addition, you usually sound fast. These objects are also generated musically by a sudden event, such as striking or plucking a string, or hitting a percussion instrument such as a drum. In this context, they can also be further distinguished by the method according to the invention: if on a event-related object immediately follows the increase in amplitude of a sound object, it can be assumed that the event represents the attack of a note and can therefore be assigned to this note. In this case, the sonic and the eventual object can be summarized for presentation and handling into a single object, which is also to be designated as a note object. For sound generation in the resynthesis, ie in the sound reproduction after editing a note object or several note objects, but it may be useful to process the two mentioned sub-objects separately for reasons of sound.

If the eventual object can not be assigned to a subsequent sonic object, it can be assumed that it is a purely percussive-rhythmically understood note without pitch meaning, such as a drum beat. This can be treated differently in the further processing accordingly.

The noisy objects have in common with the eventual objects that they have no periodic sound component, but differ from them in that they do not start suddenly and can not end quickly, but can be extended in time. Such objects may e.g. Sibilants of the consonants in the vocals, breathing noises, background noise of the player like a finger slide on the guitar string, or even not belonging to the actual music noise. Such noisy objects could e.g. be made available to the user separately for further manipulation or simply flat rate treated as a "residual signal".

Fig. 1

das Audio-Signal F(A,t) als Amplitude A über die Zeit t;

Fig. 2

einen vergrößerten Zeitausschnitt aus der Fig. 1 über 0.5 Sekunden;

Fig. 3

nach Transformation des Audio-Signals der Fig. 1 in den Frequenzbereich die grautonkodierten Energien der einzelnen Bins in ihrer Augenblicksfrequenz, wobei dunkle Bereiche hohe Energie bedeuten, in einer Darstellung der Frequenz f in Cent über der Zeit t: F(f, t, E);

Fig. 4

eine Darstellung gemäß Fig. 3 mit einem Schnitt in x- und y-Richtung, der die errechneten Energien E bei dieser Frequenz f₁ und der Zeit t₁ zeigt;

Fig. 5

eine Energielandschaft F(f, t, E) ähnlich der in Fig. 4 gezeigten, bei der die Energiewerte über eine Fensterfunktion aufsummiert und in Zeitrichtung geglättet wurden, mit einem Schnitt in x- und y-Richtung bei der Frequenz f₁ und der Zeit t₁;

Fig. 6

eine Relevanzlandschaft zu der Energielandschaft der Fig. 5 mit einem Schnitt in x- und y-Richtung mit dem ersten gefundenen Maximum; und

Fig. 7

alle gefundenen Notenobjekte nach Abschluss einer iterativ ausgeführten Identifizierung.

Below, the method according to the invention will be explained with reference to an embodiment with reference to the figures, in which results of individual method steps are shown. The procedure was carried out on a 6-second excerpt from a piano recording of Invention No. 1, C major by JS Bach. Show it:

Fig. 1

the audio signal F (A, t) as amplitude A over time t;

Fig. 2

an enlarged time excerpt from the Fig. 1 over 0.5 seconds;

Fig. 3

after transformation of the audio signal of the Fig. 1 in the frequency domain, the gray-tone coded energies of the individual bins at their instantaneous frequency, where dark areas indicate high energy, in a representation of the frequency f in cents over time t: F (f, t, E);

Fig. 4

a representation according to Fig. 3 with a section in the x and y directions, showing the calculated energies E at this frequency f ₁ and the time t ₁ ;

Fig. 5

an energy landscape F (f, t, E) similar to the one in Fig. 4 shown, in which the energy values were summed over a window function and smoothed in time direction, with a section in the x and y direction at the frequency f ₁ and the time t ₁ ;

Fig. 6

a relevance landscape to the energy landscape of the Fig. 5 with a section in the x and y directions with the first maximum found; and

Fig. 7

all found note objects after completion of an iterative identification.

There are used to explain the invention terms that are initially defined in their meaning.

The terms "time domain" and "frequency domain" correspond to the usual use of time domain and frequency domain in English, and therefore concern investigations or calculations either in the original time signal F (A, t) itself (= time domain) or in its representation F (FIG. f, t) in the form transferred over a discrete Fourier transformation, in particular FFT, (= frequency range).

"Window functions" are used to show or hide a signal when it is being processed at a time or frequency location. So you can refer to the time or to the frequency spectrum. The window shape is not fixed and may in a simple case be a triangle window. Better results, e.g. a von Hann window. The choice of the window shape can be optimized for the respective purpose. For the embodiment of a method according to the invention described below, overlapping windows are used.

Bins are the frequency bands that result from the FFT. This also refers to the possibly changed frequencies of a band, if the method of the instantaneous frequency is applied.

The instantaneous frequency of a bin results from the consideration of the phase value of the respective bin. If the analysis windows overlap, the instantaneous frequency of the particular bin can be determined from the difference between the bin's expected phase value and the actual phase value of the bin. The more frequent the overlap, the more adjacent bins can represent a particular frequency that does not have to match the computational frequency of the bin itself.

For the method according to the invention for the bins, energetic quantities are calculated which receive the short notation E and subsequently referred to as "energy" and "Magnitude" should be called. After the FFT each energy is assigned to each bin, which is calculated from the real and imaginary parts of the Fourier series according to (Re * Re) + (Im * Im), so that a value related to the amplitude of the frequency is produced. The magnitude is the root of it. For the optimal scaling of the respective value in the evaluation, however, the magnitude value can possibly be raised to a suitable value between 1 and 2, so that the size then obtained represents something between energy and magnitude. The term energy is used here in a more general sense, not in the sense of sound energy or other energy terms.

The cent is a measure of frequency relations used in the musical context, ie interval sizes defined as Cent = log (f1 / f2) / log (2) * 1200. Therefore, in this unit, a musical interval is always the same regardless of its absolute pitch, namely half tone = 100 cents, octave = 1200 cents.

To identify note objects, the method described serves pictorially an energy landscape, which is understood as meaning a three-dimensional mathematical entity F (t, f, E) in which the x-axis is the time t, the y-axis the frequency f and the z Axis represents an energy E at the respective time / frequency location. In this case, the frequency axis is expediently scaled by cent so that sound intervals in each frequency range are always the same. The energy landscape is represented in the method to be described by discrete points, the time axis through the measuring points in time in about 0.01 s distance, the frequency axis by points in the interval distance of eg 5 cents each. In some process steps, the discrete points are converted via window functions into continuous spectrograms, which is to be regarded as optional.

The examined audio material is after recording and analog-digital conversion, e.g. as audio file in PCM format (pulse-code modulation, time- and time-discrete signal). Size data in the further text, such. for analysis windows, refer to a digital signal that is available with a sample rate of 44100 samples / s. For other sample rates, the sizes would have to be adjusted accordingly.

a) Identification of note objects and event objects

The method described by way of example works both in the analysis and in the sound extraction for specific subtasks directly in the time domain ( Fig. 1 and 2 ), for others in the frequency domain ( Fig. 3 to 7 ). The treatment in the time domain is better suited for the event-like objects, the treatment in the frequency range better for the sound objects. Since the search is for discrete note objects in time, the signal is not processed continuously, but a time period is always buffered both in the time domain and in the frequency domain, which is then examined.

Fig. 1 shows a waveform F (A, t) to an audio recording. FIG. 2 shows an enlarged section to it. 1 denotes an event object in these figures, which can be fixed to an amplitude jump. In contrast, the area designated 2 shows a significantly more periodic course.

For processing in the frequency domain, the signal is read out with uniformly overlapping and overlapping window functions and first transferred via an FFT into a complex array for the respective time slice. The size of the FFT can be eg 2048 samples, the overlap should be at least 4 times. This results in intervals between the time slices of, for example, 512 samples or around 0.01 sec.

An example of a further processed transformation result shows FIG. 3 , There is shown a function F (f, t, E), which results from the Fourier transform of the Figures 1 and 2 shown time signal F (A, t) results in that the instantaneous frequencies and energies are determined to the bins. Coded as shades of gray, the energies of the individual bins are plotted in their instantaneous frequency over time. Event objects 3 are distinguished in this illustration by the uniform distribution of the instantaneous frequencies, which indicates the noiseiness of this object. Note objects are characterized by the concentration of energy at a few instantaneous frequencies. This difference is due to the fact that a basically non-periodic waveform can only be adequately represented by a superposition of a large number of periodic functions, while periodic ranges can be well represented by a few periodic functions. Periodic areas in the time signal therefore lead to a strong energy superposition of adjacent bins 4 on a common instantaneous frequency.

To the signal F (f, t, E) in the frequency domain, the following values are determined: the magnitudes of all bins, the instantaneous frequencies of all bins, the tonality values of all bins. The tonality value is an arithmetic variable that stands for the degree of periodicity in the binary frequency. It is calculated for each bin by determining how close the instantaneous frequencies of the neighboring bins are to the instantaneous frequency of the bin in question. The number of neighbor bins included is equal to the number of window overlaps, as this determines how many bins can represent a frequency. The tonal value of a bin becomes higher the closer the instantaneous frequencies of the bins in its environment are. A high tonality value means that a note object tends to be present, while a low tonality value tends to speak for an event object. The tonality values are scaled to a value range between 0 and 1. In addition, each bin is assigned a noise value directly from the tonality value is derived and calculated as a 1-tonality value. A sudden increase in noise values speaks for an event object.

It is then generated an energy landscape, exemplified in Fig. 4 is shown as a section along the time t ₁ and the frequency f ₁ , and represents the distribution of the energy E in the time t and frequency f and the actual finding of the note objects serves as ridges in this landscape. A smoothing in time direction can be performed. Furthermore, for the further calculations, one favors by fading in the energy values via a window function from the energy landscape of the FIG. 4 obtained modified energy landscape used. It then results in a Fig. 5 shown smoothed function F (f, t, E). In both figures objects 5 emerge clearly, which receive high energy values.

Since the first thing to do is to find sound objects, the calculated energies of the bins are weighted with the ascertained tonality values: for each time slice, its energy is multiplied by its tonality value for each bin. This weighting changes the result only gradually, can therefore be omitted. Then, according to the instantaneous frequency of the bin, its cent position (= y position) is determined in the landscape and, starting from this point, the energy tonality product with a certain distribution width in cent direction is added to the landscape via a window function. The width of the distribution window in cents is expedient on the order of a semitone. The representation of the Fig. 4 such a weighting with the Tonalitätswerten was already based. If all time slices in the landscape are summed up, the landscape can be smoothed in time direction with a low-pass filter (see FIG. 5 ). This makes it easier to find connected note objects as ridges. The total energy of the landscape is summed up and is available as a value for the abort criterion of the subsequent iteration.

The identification of the note objects is done by an iteration in such a way that the most prominent object in the sense of the maximum is traced as a ridge and its energy is then subtracted from the landscape, whereupon the next most clearly projecting object is searched, etc. Das However, the most pronounced sound object is not identical with the highest ridge in the energy landscape F (f, t, E). This is due to the fact that a sound object is not defined by a single ridge in the energy landscape, but that it must be assumed that the energy in ridges on the integer multiples of a fundamental frequency also belong to just this sought sound object because they overtones the fundamental tone with the fundamental frequency. It may well be that the energy of the fundamental tone is weaker than that of the higher partials, but nevertheless the object is to be found and tracked on the frequency of the fundamental tone. The background of these considerations is that although the fundamental frequency determines the pitch of a sound, the sound of a sound is decisively influenced by the overtones.

To take this into account, a second landscape is pictorially created, the so-called relevance landscape, which is exemplified in Fig. 6 is shown, and in its x- and y-axes and their size with the energy landscape F (f, t, E) is identical, but their z-values E 'derived from this: this is for each xy coordinate point of the relevance landscape F '(f, t, E') is formed as the z-value E 'the sum of all z-values E which are in the energy landscape F (f, t, E) at this xy point and at all points which correspond to the integer frequency multiples of the starting point. It is expedient to add the existing there energy value with decreasing weight with increasing atomic number of frequency multiples. This results in a relevance landscape F '(f, t, E'), in which the possible fundamental tones are taken into account with their partials, and their highest point is the most relevant in terms of sound Point of the most relevant note is. In the FIG. 6 The relevant relevance landscape F '(f, t, E') shows the same energetically projecting objects 5. It has proved to be due to the consideration of the energies contained in the overtones FIG. 5 result in a shift in relative energy levels.

Finding the note objects in the relevance landscape generated as just described, which is basically just a special energy landscape, namely a consideration of the harmonic energy, is carried out by an iterative mathematical process. It will be explained below how finding the note objects in this relevance landscape occurs without the invention being restricted thereto. For the discovery of the note objects could in principle also take place in one of the other energy landscapes described above or further modified, but this would have the disadvantage that overtones would be identified as separate notes, and e.g. by post-processing with the basic tones would be to connect. The task of separating note objects also in terms of sound is then solved well if the combination of fundamental and overtones succeeds. Therefore, the maximum search in the relevance landscape is preferred because it leads to the best results.

First the highest point of the relevance landscape is searched. In Fig. 6 the energy maximum was found at t ₁ and f ₁ . The ridge of the ridge belonging to this maximum is tracked forward and backward in time direction. For this purpose, in the respectively adjacent time slice of the landscape, the maximum in the landscape closest to the last found point in the pitch direction is sought. If the distance to the nearest maximum is such that a continuation of the pitch line as the same object is not plausible, for example, a jump of more than 50 cents from one time slice to the next, the search will be aborted in the current direction. Similarly, the search is aborted when a certain height value of the found maximum is exceeded, such as 10% of the initial value. The Tracing the ridge is done expediently in the relevance landscape, because its course corresponds better due to the co-weighting of the partials the pitch of the searched object. If the search is canceled in both directions, a new note object is created and all points of the found ridge are added to it as its pitch curve. In the claims in this context is the speech that a value field belonging to the maximum is determined. The method of determining this value field can also be carried out differently than described above, for example by using other mathematical methods. For example, the value field could be queried in all directions from the maximum point by point until a threshold is exceeded in each direction. All points above the threshold would be assigned to the maximum as a value field.

Subsequently, the energy landscape E (f, t, E) is deprived of the arithmetic energy of the found note object, namely at the locations of the course of its fundamental tone and that of all partials, ie the integer multiples of the fundamental frequency. In principle, this could also take place in the relevance landscape E '(f, t, E'), but this would be a poorer embodiment, because the possible superimposition of overtones, which belong to different basic tones, is better offset by energy withdrawal from the energy landscape.

Advantageously, however, not all the energy present there is withdrawn, but only a fixed proportion, eg 50%. For example, this percentage can be set as a parameter by the user to other values because, depending on the audio material, other parts may produce better results. For example, if the harmonics are strongly superimposed, a reduction to 25% can lead to better results. The withdrawal of only a portion of the energy makes sense, since it is not known at first whether other note objects that are sounding at the same time do not have partials that are close to the partials of the first found note object. By the only partial deduction of energy further notes objects in the following iterations can still be found.

Reducing the energy at the given frequency locations in the energy landscape E (f, t, E) advantageously takes place again in the form of a window function which fades up and down in the frequency direction and whose width is of the order of one semitone. If a model of the overtone spectrum of the sound is known, e.g. because a reference spectrum exists or is known model-wise for the instrument producing the sound, the energy extraction in the harmonics can be carried out according to this reference spectrum or model. The note object "remembers" the amount of energy withdrawn by it for its later evaluation by attributing that portion of energy to it.

In the time range affected by the newly found note object, the relevance landscape is recalculated as described above, since in this time range the energy landscape has changed as the basis for the relevance landscape due to the withdrawal of energy.

For the newly found note object, it is determined whether it overlaps in time and in the course of its fundamental frequency with another previously found note object. If it intersects with one of them or if it directly adjoins the fact that it is plausible that it is the same note object, it will be added to it (if necessary, by extending its pitch progression). Otherwise, it will be included as a new note object in the set of found note objects. Since in each iteration step only e.g. 50% of the energy for the note object is removed, usually every note object is found several times in the course of the iteration.

The iteration continues by searching for the highest point in the changed relevance landscape. The iteration will take until you reach a Abort criterion continued. An advantageous termination criterion for the iteration is the reduction of energy in relation to the original energy in the energy landscape. The iteration can be interrupted, for example, if only 10% of the original energy is present in the energy landscape. This can also be made available to the user as a parameter to be changed.

Locating the event-like objects, which are characterized by a sudden increase of the noisy component in the signal, can be done either in the time domain signal by tracking the increase of above all high-pass filtered signal components, or in the frequency domain using the noise values of the bins that are weighted with the energies of the respective bins, are added up for each time slice. In both cases, one obtains a progression curve of the noisy component in the overall signal. At the points of the largest slopes of this curve, if necessary defined by a threshold value for the slope, the event-related objects are to be assumed.

The eventual objects found in the previous step may either be isolated in themselves in the signal, as will be the case with purely percussive events, or they may be the note noises of note objects previously found in the iteration as plucked or struck tonal instruments such as guitar, piano, etc. will be the case. In order to distinguish this, for each event-based object found at the time of its occurrence, it is examined whether a significant increase in its energy occurs at one or more of the note objects located there immediately after the occurrence of the event. If this is the case, the event-aware object is understood as a stop of the note object and assigned to this. If the energy increase is the case for multiple notes, the eventual object is assigned to all these notes. If the energy increase occurs in the middle of a note object, the note object is separated there and there understood as a new note object. If no corresponding note object is found at the time of the eventual object, the eventual object is understood as an isolated percussive event. FIG. 7 shows the note objects found in this example along with the event-like objects marked as vertical dashes that could be assigned to these note objects.

Finding the note-like objects should advantageously be followed by an evaluation step. When searching for note objects in the described iteration, more objects are usually found than are musically plausible. Therefore, the set of found note objects will be checked at the end according to different plausibility criteria and possibly not sufficiently plausible note objects removed. A plausibility criterion is e.g. the relative energy and the occlusion. In general, too many small note objects with too little energy are found in the described iterative procedure. Therefore, it examines how much energy the note has in proportion to the total energy in its time domain. If it has too little relative energy, it can be removed.

Sometimes objects are also identified as independent notes, which are actually overtones of another existing note. Here, e.g. It will be examined whether the higher note has an independent progression in pitch, amplitude and duration, or whether it essentially runs like a lower note in these parameters. If the latter is the case, the object can be removed or added to the lower note.

Further reviews can be made according to musical criteria. For example, if a note object is very isolated in its pitch range (very high or very low if there are no other notes), it is musically unlikely. For example, if a note deals with other notes in its It is musically very probable, even if it is otherwise rather weak, etc. All these criteria can be mapped mathematically and weighted, for example, in order to arrive at as plausible a set of musical notation objects as possible ,

This described identification step may also be followed by intervention by the user, to whom the found note objects are graphically displayed in a suitable manner, eg in the Fig. 7 and, for example, mouse-sensitive and / or menu-controlled as a note identified objects can still split or separate notes to an object can unite. Of course, the user can also delete individual objects or add other objects to be considered. For this purpose, he can be offered the objects for activation, which were previously rated as insufficiently relevant in the automatic evaluation.

The automatic identification can optionally be optimized by depositing the notes of the recorded piece of music, so that in the above-mentioned method based on the stored notes is specifically trying to find there basic tones that correspond to the frequencies of the deposited notes. This can e.g. done by evaluating a stored MIDI file containing the notes of the recorded composition. Alternatively, support tracks can also be recorded simultaneously with the recording of the actually used overall signal, as e.g. by single microphone of the participating instrumentalists or singers, or in a guitar by individual removal of the strings. In the then pronounced monophonic signal of the individual voices, the desired notes of the total signal can be identified more clearly and thus the overall sound is better separated for processing.

b) Tone assignment to note objects

After the identification of the individual objects involved in the overall sound has been completed, in a next step the sonic decomposition of the overall sound can take place. The most accurate determination possible of the existing note objects, their pitch and amplitude course and the nature of their insertion is an influencing factor for the quality of the result of the sound decomposition. In the sonic decomposition described below, the overall sound is only decomposed as far as possible into individual sound objects, as is necessary for the resynthesis of the new overall sound. So if, for example, If the user only touches a single note in a complex chord and modifies its pitch, then only the signal of that single note must be extracted and subtracted from the original signal. Thus, the more notes are generated the more signals are generated. Each is then a monophonic-periodic signal and can be changed, summed and played back using known methods for time- and pitch-independent playback.

In a first step of sound decomposition, the eventual objects are extracted from the original signal. When the original signal is split into individual signals belonging to the individual note objects, the event-sensitive locations in the time signal are smeared by the division of the frequency spectrum. Therefore, it makes sense to first separate the locations of the event-related objects from the time signal and to perform the decomposition into the individual signals of the note objects on the resulting residual signal. However, this optional method step can also be omitted.

To separate the event-like objects, a modified time signal is first generated, in which the sound components are subtracted as completely as possible. For this purpose, in the frequency domain, the magnitudes of all bins are multiplied by the noise values of the bins and, with these new magnitudes and the original phases, a time signal is again generated via the FFT. optional For example, factors for damping the lower signal components can be included in the factors for the magnitudes, since the higher proportions are often more relevant for the event-related objects. From this new noisy time signal, at the points where eventual objects have been found in the above-described identification step, the time signal of the eventual objects is cut out with a suitable window function, eg with a short rise time of about 0.005 s and a release time of about 0.05 s. These short time signals of the event-related objects are subtracted from the original time signal and stored separately, for example.

This is followed by a separation of the note objects from the original signal. The division of the original signal (from which the eventful components were deducted) into the individual sounds of the note objects occurs in the frequency domain. For this purpose, the original signal is first, after its modification, transformed into the frequency domain by the separation of the event objects (see above).

The division into the individual sounds of the note objects is based on the fact that each note object in each time slice announces a "claim" on a spectral portion of the total signal F (f, t, E). This claim is mathematically represented by spectral proportional factors, which are calculated for each note object from a spectral component function, which is obtained, for example, from a model of the sound of a single note. This model can be easily specified, or it can be modeled on the real sound of an instrument, if the instrument to which the note object is based is known. In the present example, the model is based on the following components: It is assumed that harmonics exist as spectral components in integer multiples of the fundamental frequency for the fundamental tone progression of the note object. It is further assumed that the amplitudes of the overtones follow a fundamental tone of an overtone model. In the simplest case this can be the decrease of the amplitude with the reciprocal of the harmonic number. The overtone model but it can also reproduce the harmonic amplitude progression derived from an empirical sound. Finally, it is assumed that the harmonic amplitudes are related to the course of the fundamental energy of the note object. In the simplest case, this can be assumed to be proportional to the fundamental tone energy, but another context can also be derived from an empirical sound.

Based on these assumptions, a spectral fraction function is provided, e.g. may be different for different instruments, and for each note object in each time slice, the spectral proportions are calculated, that is, the claims this object has on each bin.

If several share functions are stored, e.g. be provided that the user selects one of the share functions. There may also be an automatic selection, e.g. if the user inputs with which instrument the note object was played, or if it is automatically recognized that the note object was played on a particular instrument, e.g. can result from this that the overtone amplitude ratios of a note object correspond to a deposited share function.

The amount of the calculated proportional factors depends on the partial frequencies and the amplitudes of the partials, which follow eg from the underlying model of the sound of a single note. Furthermore, the amount of the proportional factors depends on the proximity or distance of the sub-tone frequency to the instantaneous frequency of the respective bin. The amount of the proportional factors as a function of the distance may, for example, enter via a weighting curve in the frequency range, which should be wide enough, for example, to permit even slight deviations from the given frequency. On the other hand, the weighting curve in the central area should be so narrow that the harmonic component factors of different concurrent notes with different fundamental pitch separated sufficiently well and the overtones of the correct note can be assigned. A suitable weighting curve for evaluating the removal of the frequencies may be, for example, a von Hann window raised to the fourth power, which in its total width corresponds, for example, to two semitones.

If in the respective current time slice all note objects found there as sounding have registered their claims on the share factors on all bins, the sum of the proportional factors of all notes for each one bin is normalized to 1. For each note object, a separate time signal with the duration of this note object is created. For each time slice, the magnitudes or other appropriate energetic value of all bins are divided among the note objects according to the normalized proportion factors. These parts of the individual note objects in the frequency domain are transformed back into the time domain with the original phases via the FFT and the time signals are accumulated on the individual time signals of the note objects.

Since the magnitude components or other energetic components have previously been changed, the ends of the signal in the time windows after retransformation into the time domain are no longer faded out to 0, which leads to undesirable artifacts. Therefore, the result of the inverse transformation should be windowed again. For this purpose, it is useful to take the root of the values of the actually provided window function and then apply this window before the FFT and after the inverse FFT.

Finally, the note objects are merged with the eventual shares. It was already described above that the event objects were assigned to the note objects, and that a time signal was generated for the event-related objects. Now the note objects to which eventful objects have been assigned, this time signal can be added to their beginning. If an eventful object has multiple note objects have been assigned, because it was assumed that this was struck simultaneously, the time signal of the event-like object is distributed in its amplitude to the associated note objects. This may conveniently be done in proportion to the energies of the note objects themselves or done on the basis of the assumed instrument model.

Event-related objects to which no note objects have been assigned can be available with their extracted time signal as independent percussive objects.

If all the note objects found and the event objects associated with them have their time signal generated, these time signals of all notes will be subtracted from the original signal. Because the sound model used in the sound split assumes that the note objects consist essentially of partials that are approximately integer multiples of a fundamental frequency, not all of the sound, and thus the entire time signal, will have been distributed among the individual objects. Therefore, after deducting the timing signals of all the individual objects from the original signal, a residual signal containing the rather noisy components remains. This residual signal can be easily reproduced in the resynthesis, or it can be made available to the user, as a whole or disassembled into further individual objects, for further processing.

The time sequence of the above-mentioned individual method steps can also be chosen differently. Thus, e.g. the assignment of event objects to note objects also be made only immediately before the resynthesis. This applies analogously to other process steps, e.g. the identification of event and note objects or the calculation of proportional factors.

Claims (31)

1. A method for sound-object-oriented analysis and note-object-oriented processing of a polyphonic digitised sound recording which is provided as a time signal F(A, t), comprising the following analysis and processing steps:

   - reading the time signal F(A, t) section by section, with use being made of a window function and overlapping windows,

   - Fourier transform of the read signal to the frequency domain, more particularly by executing a discrete Fourier transform,

   - calculating an energy value E for each bin (4) from the frequency amplitude resulting from the Fourier transform, more particularly by squaring the real and imaginary contents or forming an energy value derived therefrom,

   - generating a three-dimensional function F(t, f, E),

   - identifying event objects (1, 3) which are characterised in that they have an abrupt amplitude rise and are not periodic at this point and which, in most of the cases, are quickly fading,

   - identifying note objects the main property of which is that they have a tone pitch or a tone pitch pattern for a perceivable duration and that the curve shape of their time signal, in essence, has a periodic or quasi-periodic behaviour,

   - comparing the point in time of the occurrence of event objects (1, 3) and note objects and allocating event objects (1, 3) to note objects in case of an occurrence that is plausible in relation to time,

   - calculating spectral content factors for each note object,

   - allocating signal contents of the frequency signal F(f, t, E) to found note objects by means of the calculated content factors,

   - reverse transforming the frequency signal contents allocated to a note object into a time signal,

   - graphical representation of the note objects and/or event objects (1, 3) in a time/frequency graph on a monitor,

   - processing one or a plurality of note objects under user control or in an automated manner,

   - storing the time signals of processed note objects,

   - reproducing the stored time signals of processed note objects together with the time signal that is reduced by the time signal allocated to a note object.
2. The method according to Claim 1, characterised in that a function F'(t, f, E') is calculated from the function F(t, f, E) with energy values E' that are the sum total of all energy values E at a time t and at a fundamental frequency f and all multiples thereof.
3. The method according to Claim 2, characterised in that the energy values with the multiples of the fundamental frequency are added after weighting with a factor other than 1.
4. The method according to Claim 2 or 3, characterised in that the following method steps are carried out to identify the note objects:

   - determining an energy maximum in function F'(f, t, E'),

   - determining a coherent value field belonging to the maximum,

   - allocating the found value field to one note object each.
5. The method according to Claim 4, characterised in that the energy values E of the value field allocated to the note object are subtracted from function F(t, f, E).
6. The method according to Claim 5, characterised in that the energy values E are only subtracted at the level G*E, wherein 0 < G < 1 is applicable to factor G.
7. The method according to Claim 6, characterised in that factor G is a parameter that can be modified by the user.
8. The method according to Claim 5, 6 or 7, characterised in that the search for the maximum is continued at a function from which the energy values have been subtracted and at a function calculated therefrom.
9. The method according to any one of the preceding claims, characterised in that the search for the maximum is carried out iteratively until a termination criterion is reached.
10. The method according to Claim 9, characterised in that a total energy value Etot is calculated for function F(t, f, E) and the iteration is terminated as soon as a certain content H * E_tot of this total value has been allocated to the found note objects, in particular when this content is in excess of 90 per cent.
11. The method according to Claim 10, characterised in that factor H is a parameter that can be modified by the user.
12. The method according to any one of the preceding claims, characterised in that, in a subsequent automated rectification process, an identified note object is discarded when one or a plurality of the following criteria is fulfilled:

    - the energy of the found note object is very low as compared with the total energy,

    - the tone pitch and amplitude pattern of the note object essentially corresponds to that of another note object of lower frequency, in particular when the lower frequency is possible as the fundamental frequency,

    - the frequency spacing of a note object from the other note objects is very large.
13. The method according to any one of the preceding claims, characterised in that, in a post-processing step, a user separates, connects and/or deletes automatically identified note objects.
14. The method according to any one of the preceding claims, characterised in that an instantaneous frequency is calculated for each bin (4) from the phase differences of neighbouring bins (4), said instantaneous frequency being used as the frequency of the bin (4) in function F(t, f, E) or F'(t, f, E').
15. The method according to any one of the preceding claims, characterised in that a tonality value and/or a noise value is calculated for each bin (4) in order to find the event objects (1, 3).
16. The method according to Claim 15, characterised in that the energy values of each bin (4) are weighted with the tonality value.
17. The method according to any one of the preceding claims, characterised in that filed notes are accessed for identification of the note objects.
18. The method according to any one of the preceding claims, characterised in that the found event objects (1, 3) are extracted from the time signal F(A, t) and the sound decomposition is carried out at the remaining signal.
19. The method according to Claim 18, characterised in that the event objects (1, 3) are stored separately.
20. The method according to any one of the preceding claims, characterised in that the spectral content factors of a note object are calculated from a filed spectral content function.
21. The method according to Claim 20, characterised in that the filed spectral content function is the mathematical image of a sound model of the note object, the sound model formulating one or a plurality of the following postulates:

    - there are spectral components on integer multiples of the fundamental tone frequency, i.e., harmonics on harmonic frequencies, the spectral components being associated with a fundamental tone having a fundamental tone frequency,

    - the amplitude pattern of the harmonics associated with a fundamental tone follows principles which result from a harmonic model or a harmonic amplitude pattern that has been determined empirically,

    - the harmonic amplitudes have a fixed ratio to the pattern of the fundamental tone energy.
22. The method according to Claim 20, characterised in that the filed spectral content function is the mathematical image of the sound of a note played with a specific instrument.
23. The method according to Claim 20, characterised in that a plurality of spectral content functions is filed, more particularly different content functions for a plurality of instruments, in particular such content functions according to Claim 22.
24. The method according to Claim 23, characterised in that one out of the plurality of content functions is selected by the user.
25. The method according to Claim 23, characterised in that one out of the plurality of content functions is automatically allocated to a note object when the user enters the instrument with which the note was played or when the user automatically detects such an instrument.
26. The method according to any one of Claims 20 to 25, characterised in that, using a window function, the spectral content function carries out a weighting within the frequency range at a predefined frequency width.
27. The method according to any one of the preceding claims, characterised in that a residual signal is calculated by subtracting all of the time signals allocated to the note objects and the event objects (1, 3) from the original time signal.
28. The method according to Claim 27, characterised in that the residual signal is subjected to a further identification of note or event objects (1, 3).
29. The method according to any one of the preceding claims, characterised in that, for the purpose of sound reproduction after a note object has been processed, the sound content of the note object is subtracted from the total sound and the differential signal thus obtained is played together with the sound content of the note object.
30. The method according to Claim 27, characterised in that, during sound reproduction after a note object has been processed, the residual signal is also reproduced by a user.
31. A computer programme with a programme code for applying the method according to any one of the preceding claims if the computer programme is running on a computer.

Images