EP1743324B1 - Device and method for analysing an information signal - Google Patents

Abstract

In order to analyse an information signal, a significant short-time spectrum is extracted from the information signal. The extraction device (16) is embodied in such a way as to extract the short-time spectra which come closer to a specific characteristic than other short-time spectra of the information signal. The extracted short-time spectra are then decomposed (18) into component signals, by ICA analysis, a component signal spectrum representing a profile spectrum of a sound source which generates a sound corresponding to the required characteristic. An amplitude envelope is calculated (20) for each profile spectrum from a series of short-time spectra of the information signal and from the determined profile spectra, said envelope indicating how the profile spectrum of a sound source generally varies over time. The profile spectra and associated amplitude envelopes describe the information signal that can be further evaluated, e.g. for the purposes of a transcription in the case of a music signal.

Description

The present invention relates to the analysis of information signals, such as audio signals, and more particularly to the analysis of information signals consisting of a superposition of sub-signals, wherein a sub-signal may originate from a single source or a group of individual sources.

The progressive development of digital distribution media for multimedia content leads to a large variety of data offered. For the human user, the limit of the manageable has long been exceeded. Thus, the content description of the data by metadata becomes increasingly important. Basically, the goal is not only text files but also z. B. music files, video files or other information signal files to make searchable, with the same comfort as in common text databases is sought. One approach to this is the well-known MPEG 7 standard.

In particular, in the analysis of audio signals, ie signals that include music and / or speech, the extraction of fingerprints is of great importance.

The aim is also to "enrich" audio data with metadata to z. B. to recover a piece of music based on a fingerprint metadata. The "fingerprint" should on the one hand be meaningful, and on the other hand be as short and concise as possible. "Fingerprint" thus refers to a compressed generated from a music signal Information signal, which does not contain the metadata, but for referencing to the metadata eg by searching in a database is used, for example in a system for the identification of audio material ("AudioID").

Usually, music data consists of the superposition of sub-signals from single sources. While pop music typically has relatively few individual sources, namely the singer, the guitar, the bass guitar, the drums, and a keyboard, the number of sources for an orchestral piece can become very large. An orchestral piece and a pop music piece, for example, consist of a superposition of the tones emitted by the individual instruments. An orchestral piece or piece of music thus represents a superimposition of partial signals from individual sources, the partial signals being the sounds produced by the individual instruments of the orchestra or pop music ensemble, and the individual instruments being individual sources.

Alternatively, groups of original sources can also be considered as individual sources, so that at least two individual sources can be assigned to one signal.

An analysis of a general information signal is shown below by way of example only with reference to an orchestra signal. The analysis of an orchestra signal can be done in many ways. Thus, there may be a desire to recognize the individual instruments and to extract from the overall signal the individual signals of the instruments and possibly to convert them into a musical notation, wherein the notation would act as "metadata". Further possibilities of the analysis are to extract a dominant rhythm, whereby a rhythm extraction on the basis of the percussion instruments better rather than on the basis of the more sound-giving instruments, which are also referred to as harmonic-sustained or "harmonic sustained" instruments. While percussion instruments typically include timpani, drums, rattles or other percussion instruments, the harmonic sustained instruments include all other instruments such as violins, wind instruments, etc.

Furthermore, the percussion instruments include all those acoustic or synthetic tone generators that contribute to the rhythm section due to their sound characteristics (e.g., rhythm guitar).

For example, it would be desirable for the rhythm extraction of a piece of music to extract only percussive parts from the entire piece of music and then perform rhythm recognition on the basis of these percussive parts, without the rhythm recognition being "disturbed" by signals from the harmonically sustained instruments.

On the other hand, any analysis aimed at extraction of metadata that requires only information from the harmonic sustained instruments (e.g., a harmonic or melodic analysis) will benefit from upstream separation and further processing of the harmonically sustained portions.

More recently, the use of Blind Source Separation (BSS) and Independent Component Analysis (ICA) techniques for signal processing and signal analysis has been reported. Areas of application are especially in biomedical engineering, communication technology, artificial intelligence and image processing.

Generally, the term BSS includes techniques for separating signals from a mix of signals with a minimum of prior knowledge of the nature of the signals and the mixing process. The ICA is a method that makes use of the assumption that the sources underlying a mix are, at least to a degree, statistically independent of each other. Furthermore, the mixing process is assumed to be fixed in time and the number of mixed signals observed is not less than the number of source signals underlying the mixture.

• [1] M.A. Casey and A. Westner, "Separation of Mixed Audio Sources by Independent Subspace Analysis", in Proc. of the International Computer Music Conference, Berlin, 2000
• [2] I.F.O. Orife, "Riddim: A rhythm analysis and decomposition tool based on independent subspace analysis", Master thesis, Darthmouth College, Hanover, New Hampshire, 2001
• [3] C. Uhle, C. Dittmar and T. Sporer, "Extraction of Drum Tracks from polyphonic Music using Independent Subspace Analysis", in Proc. of the Fourth International Symposium on Independent Component Analysis, Nara, Japan, 2003
• [4] D. Fitzgerald, B. Lawlor and E. Coyle, "Prior Subspace Analysis for Drum Transcription", in Proc. Of the 114th AES Convention, Amsterdam, 2003
• [5] D. Fitzgerald, B. Lawlor and E. Coyle,"Drum Transcription in the presence of pitched instruments using Prior Subspace Analysis", in Proc. of the ISSC, Limerick, Ireland, 2003
• [6] M. Plumbley, "Algorithms for Non-Negative Independent Component Analysis", in IEEE Transactions on Neural Networks, 14 (3), pp 534- 543, May 2003
• [7] T. HEITTOLA ET AL.: "Locating Segments with Drums in Music Signals" 3RD INTERNATIONAL CONFERENCE ON MUSIC INFORMATION RETRIEVAL ISMIR 2002, Oktober 2002 (2002-10), Seiten 1-6, XP002341222

An extension of the ICA is the Independent Subspace Analysis (ISA). Here, the components are subdivided into independent subspaces or subspaces whose components need not be statistically independent. By transforming the music signal, a multi-dimensional representation of the mixed signal is determined and the last assumption for the ICA is met. Various methods for calculating the independent components have been developed in recent years. Relevant references, some of which also deal with the analysis of audio signals, are the following:

• [1] MA Casey and A. Westner, "Separation of Mixed Audio Sources by Independent Subspace Analysis," in Proc. of the International Computer Music Conference, Berlin, 2000
• [2] IFO Orife, "Riddim: A rhythm analysis and decomposition tool based on independent subspace analysis", Master thesis, Dartmouth College, Hanover, New Hampshire, 2001
• [3] C. Uhle, C. Dittmar and T. Sporer, "Extraction of Drum Tracks from Polyphonic Music using Independent Subspace Analysis", in Proc. of the Fourth International Symposium on Independent Component Analysis, Nara, Japan, 2003
• [4] D. Fitzgerald, B. Lawlor and E. Coyle, Prior Priority Analysis for Drum Transcription, in Proc. Of the 114th AES Convention, Amsterdam, 2003
• [5] D. Fitzgerald, B. Lawlor and E. Coyle, "Drum Transcription in the Presence of Pitched Instruments Using Prior Subspace Analysis", in Proc. of the ISSC, Limerick, Ireland, 2003
• [6] M. Plumbley, "Algorithms for Non-Negative Independent Component Analysis," IEEE Transactions on Neural Networks, 14 (3), pp 534-543, May 2003
• [7] T. HEITTOLA ET AL .: "Locating Segments with Drums in Music Signals" 3RD INTERNATIONAL CONFERENCE ON MUSIC INFORMATION RETRIEVAL ISMIR 2002, October 2002 (2002-10), pages 1-6, XP002341222

In [1] a method for the separation of single sources from mono audio signals is presented. In [2] an application for a separation into single tracks and then the rhythm analysis is given. In [3] a component analysis is performed to obtain a separation into percussive and non-percussive sounds of a polyphonic piece. In [4] Independent Component Analysis (ICA) is applied to amplitude bases, which are obtained by means of generally calculated frequency bases from a spectrogram representation of a drum track. This is done for the purpose of transcription. In [5] this method is extended to polyphonic music pieces.

The first above-mentioned publication by Casey is shown below by way of example for the prior art. This publication describes a technique for separating mixed audio sources by the art the independent subspace analysis. For this purpose, an audio signal is split into individual component signals using BSS techniques. To determine which of the individual component signals belong to a multicomponent subspace, a grouping is carried out in such a way that the similarity of the components to one another is represented by a so-called Ixegram. The Ixegram is referred to as the cross-entropy matrix of the independent components. It is calculated by examining each individual component signal in pairs in a correlation calculation to find a measure of how similar two components are. Therefore, an exhaustive pairwise similarity computation is performed over all the component signals, resulting in a similarity matrix in which all component signals are plotted along a y-axis, and in which all component signals are also plotted along the x-axis. This two-dimensional array provides a similarity measure for each component signal, each time with a different component signal. The Ixegram, ie the two-dimensional matrix, is now used to perform a clustering, for which a grouping is performed using a cluster algorithm based on diadic data. To perform optimal partitioning of the Ixegram into k classes, a cost function is defined that measures the compactness within a cluster and determines the homogeneity between clusters. The cost function is minimized, so that ultimately results in an assignment of individual components to individual subspaces. Applied to a signal representing a speaker in the context of a continuous waterfall noise, the subspan results in the speaker, the reconstructed information signal of the speaker subspace showing significant attenuation of the waterfall noise.

A disadvantage of the concepts described is the fact that the case occurs very likely that the signal components of a source come to lie on different component signals. This is why, as stated above, a complex and computation-intensive similarity calculation among all the component signals is performed to obtain the two-dimensional similarity matrix, on the basis of which, by means of a cost function to be minimized, then a division of component signals into subspaces is carried out.

Furthermore, it is disadvantageous that in the case in which there are several individual sources, ie where the output signal is not known a priori, a similarity distribution exists after a long calculation, but that the similarity distribution itself does not yet provide any actual insight into the actual audio scene. Thus, the viewer merely knows that certain component signals are similar to each other in terms of the minimized cost function. However, he does not know which information these ultimately obtained subspaces carry or which original individual source or which group of individual sources are represented by a subspace.

The Independent Subspace Analysis (ISA) can thus be used to decompose a time-frequency representation, such as a spectrogram, of an audio signal into independent component spectra. The previous methods described above rely either on a calculation-intensive determination of frequency and amplitude bases from the entire spectrogram or on a priori defined frequency bases. Such a priori defined frequency bases or profile spectra consist for example in that one says that a trumpet is most likely to be found in one piece, and then an example spectrum of a trumpet is used for signal analysis.

This procedure has the disadvantage that one must know from the outset all occurring instruments, which already runs counter to the automated processing in principle. Another disadvantage is that, if you want to work exactly, there are not only trumpets, for example, but many different types of trumpets, all of which differ in their timbre and thus in their spectrum. If you proceeded in such a way that you now use all types of sample spectra for component analysis, the process is again very complex and gets a very high degree of redundancy, since typically not all conceivable different trumpets occur in one piece, but only trumpets of a single species, ie with a single profile spectrum, or perhaps with a few different timbres, so few profile spectra. It is even more problematic with different notes of a trumpet, especially since each tone has a spread / compressed profile spectrum depending on the pitch. To account for this, the computational effort is also immense.

On the other hand, decomposition due to ISA concepts is extremely computationally expensive and susceptible to interference when the entire spectrogram is used. It should be noted that a spectrogram typically consists of a sequence of individual spectra, wherein a hopping period is defined between the individual spectra, and wherein a spectrum represents a specific number of samples, so that a spectrum has a specific time length, ie a block of samples of the signal. Typically, the duration represented by the block of samples from which a spectrum is computed is represented will be much larger than the hopping time to obtain a satisfactory spectrogram in view of the required frequency resolution and time resolution required. On the other hand, however, it can be seen that this spectrogram representation is extremely redundant. For example, considering the case that a hopping period is 10 ms, and that a spectrum is based on a block of samples having a time length of, for example, 100 ms, each sample occurs in 10 consecutive spectra. The redundancy generated thereby can drive the computational time requirements to astronomical heights, especially when a larger number of instruments are being searched for.

Furthermore, the approach of working on the basis of the entire spectrogram is disadvantageous for cases in which not all the sources contained in a signal are to be extracted, but only, for example, sources of a certain type, that is to say sources having a specific characteristic , Such a characteristic may involve percussive sources, ie percussion instruments, or so-called pitched instruments, also referred to as harmonic sustained instruments, which are typical melody instruments such as trumpet, violin, etc. A method that works on the basis of all these sources is then too time-consuming and ultimately also not robust enough, for example, if only a few sources, namely the sources that are to fulfill a specific characteristic, are to be extracted. In this case, individual spectra of the spectrogram in which such sources do not occur, or only very slightly, falsify or "wash out" the overall result, since these spectrogram spectra are of course also included in the final Component analysis calculation as the significant spectra.

In [7] another method is disclosed to determine from a music signal sections with sound signals of a percussion instrument.

The object of the present invention is to provide a robust and computationally efficient concept for analyzing an information signal.

This object is achieved by an apparatus for analyzing an information signal according to claim 1, a method for analyzing an information signal according to claim 24 or a computer program according to claim 25.

The present invention is based on the finding that a robust and efficient information signal analysis is achieved by first extracting significant short-term spectra or short-term spectra derived from significant short-term spectra, such as difference spectra etc. from the entire information signal or from the spectrogram of the information signal, respectively Short-term spectra are extracted, which come closer to a specific characteristic than other short-term spectra of the information signal.

Preferably, short-term spectra having percussive portions are extracted, and thus short-term spectra having harmonic components are not extracted. In this case, the specific characteristic is a percussive or percussion characteristic.

The extracted short-term spectra or short-term spectra derived from the extracted short-term spectra are then fed to means for decomposing the short-term spectra into component signal spectra, a component signal spectrum representing a profile spectrum of a sound source, which generates a tone which corresponds to the desired characteristic, and wherein another component signal spectrum represents a different profile spectrum of a sound source which generates a sound which also corresponds to the desired characteristic.

Finally, on the basis of the profile spectra of the sound sources, an amplitude envelope is calculated over time, whereby the profile spectra and the original short-term spectra are used for the calculation of the amplitude envelope over time, so that for each time point at which a short-term spectrum was taken off Amplitude value is obtained.

The information thus obtained, namely different profile spectra and amplitude envelopes for the profile spectra, provides a complete description of the music or information signal with respect to the specified characteristic which has been extracted so that this information may already be sufficient To make a transcription, so to first identify with the concepts of feature extraction and segmentation, which instrument "belongs" to the profile spectrum, and what rhythmic is present, so what climbs and drop events are present that indicate notes of this instrument played at certain times.

The present invention is advantageous in that for calculating the component analysis, that is to say for disassembling, not the entire spectrogram is used, but only extracted short-term spectra, ie that the calculation of the independent subspace analysis (ISA) takes place only on the basis of a subset of all spectra, so that the computational requirements be lowered. It also increases the robustness of finding certain sources, especially other short-term spectra that do not meet the specified characteristics are not present in the component analysis and thus do not represent any disturbance or "blurring" of the actual spectra.

In addition, the inventive concept is advantageous in that the profile spectra are determined directly from the signal, without resulting in the problem of prefabricated profile spectra, which in turn would lead to either inaccurate results or to increased computational effort.

Preferably, the concept according to the invention is used to detect and classify percussive, non-harmonic instruments in polyphonic audio signals in order to obtain profile spectra as well as amplitude envelopes for the individual profile spectra.

Fig. 1

ein Blockschaltbild der erfindungsgemäßen Vorrichtung zum Analysieren eines Informationssignals;

Fig. 2

ein Blockschaltbild einer bevorzugten Ausführungsform der erfindungsgemäßen Vorrichtung zum Analysieren eines Informationssignals;

Fig. 3a

ein Beispiel für eine Amplitudenhüllkurve für eine perkussive Quelle;

Fig. 3b

ein Beispiel für ein Profilspektrum für eine perkussive Quelle;

Fig. 4a

ein Beispiel für eine Amplitudenhüllkurve für ein harmonisch ausgehaltenes Instrument; und

Fig. 4b

ein Beispiel für ein Profilspektrum für ein harmonisch ausgehaltenes Instrument.

Preferred embodiments of the present invention will be explained below in detail with reference to the accompanying drawings. Show it:

Fig. 1

a block diagram of the inventive apparatus for analyzing an information signal;

Fig. 2

a block diagram of a preferred embodiment of the inventive apparatus for analyzing an information signal;

Fig. 3a

an example of an amplitude envelope for a percussive source;

Fig. 3b

an example of a profile spectrum for a percussive source;

Fig. 4a

an example of an amplitude envelope for a harmonically sustained instrument; and

Fig. 4b

an example of a profile spectrum for a harmoniously sustained instrument.

Fig. 1 shows a preferred embodiment of an inventive apparatus for analyzing an information signal supplied via an input line 10 to a device 12 for providing a sequence of short-term spectra representing the information signal. As shown by a detour line 14 in FIG. 1, which is shown in phantom, the information signal may also be supplied, for example, in time, to means 16 for extracting significant short-term spectra or short-term spectra derived from the short-term spectra from the information signal is designed for extracting to extract such short-term spectra, which come closer to a specific characteristic than other short-term spectra of the information signal.

The extracted spectra, ie the original short-term spectra or the short-term spectra derived from the original short-term spectra, for example by differentiating, differentiating and rectifying or by other operations, are fed to a device 18 for decomposing the extracted short-term spectra into component signal spectra, wherein a component signal spectrum represents a profile spectrum of a sound source, which generates a sound corresponding to the characteristic sought, and wherein another profile spectrum represents another sound source which generates a sound which also corresponds to the sought characteristic.

The profile spectra are finally fed to an amplitude envelope calculating means 20 for the one sound source, the amplitude envelope indicating how the profile spectra of a sound source change over time, and in particular how the intensity or weighting of a profile spectrum changes over time. The device 20 is designed to work on the basis of the sequence of short-term spectra on the one hand and on the basis of the profile spectra on the other hand, as can be seen from FIG. On the output side, the means 20 for calculating provides amplitude envelopes for the sources, while the means 18 provides profile spectra for the sound sources. The profile spectra and the associated amplitude envelopes provide a complete description of the portion of the information signal that corresponds to the specific characteristic. Preferably, this portion is the percussive portion of a piece of music. Alternatively, however, this share could also be the harmonic component. In this case, the means for extracting significant short-term spectra would be designed differently, as in the case where the specific characteristic is a percussive characteristic.

Hereinafter, referring to FIG. 2, a preferred embodiment of the present invention is shown. Preferably, with the profile spectra F and the amplitude envelopes E, a detection and classification of percussive, non-harmonic instruments is performed, as also shown by a block 22 in FIG. This will be discussed later.

As can be seen in Fig. 2, the means 12 for providing a train of short-term spectra is adapted to provide, by means of an appropriate time-frequency transformation to generate an amplitude spectrogram X. The time / frequency device 12 is preferably a device for performing a short-time Fourier transform with a certain hopping period, or comprises filter banks. Optionally, a phase spectrogram is also obtained as an additional information source, as shown in FIG. 2 by a phase arrow 13. Thereupon, by differentiating along the time extent of each individual spectrogram line, ie each individual frequency bin, a difference spectrogram Ẋ is obtained, as represented by the differentiator 16a. The negative components resulting from differentiation are set to zero or - alternatively - made positive. This results in a non-negative difference spectrogram Ẋ . This non-negative

The difference spectrogram is fed to a maximum searcher 16c, which is designed to search for the occurrence of local maxima in a detection function e, which is calculated before the maximum searcher 16c, after the times t, that is, for the indices of the corresponding spectrogram columns. As will be explained later, the detection function can be obtained, for example, by summing over all rows of X and then smoothing.

Optionally, it is preferred to use the phase information provided via phase line 13 from block 12 to block 16c as an indicator of the reliability of the found maxima. The spectra for which the maximum seeker detects a maximum in the detection function are used as X _t and represent the extracted ones

Short-term spectra.

In block 18a, a Principle Component Analysis (PCA) is performed. Here, a sought number of components d is first set. Then, the PCA is performed by a suitable method such as Singular Value Decomposition or Eigenvalue Decomposition over the columns of the matrix X _t . $$\tilde{\mathrm{X}}\mathrm{=}{\hat{\mathrm{X}}}_t\mathrm{\cdot }\mathrm{T}$$

The transformation matrix T effects a dimensional reduction on X, which results in a reduction in the number of columns of this matrix. Furthermore, a decorrelation and variance normalization is achieved. In block 18b, a non-negative independent component analysis is then performed. Here, the method of non-negative Independent Component Analysis shown in [6] is performed on X to calculate a separation matrix A. According to the following equation, X is decomposed into independent components. $$\mathrm{F}\mathrm{=}\mathrm{A}\mathrm{\cdot }\stackrel{\mathrm{~}}{\mathrm{X}}$$

Independent components F are interpreted as static spectral profiles or profile spectra of the occurring sound sources. In a block 20, the amplitude base or the amplitude envelope E is then extracted according to the following equation for the individual sound sources. $$\mathrm{e}\mathrm{=}\mathrm{F}\mathrm{\cdot }\mathrm{X}$$

The amplitude base is interpreted as a set of time-varying amplitude envelopes of the corresponding spectral profiles.

According to the invention, the spectral profile is obtained from the music signal itself. As a result, the computational complexity compared to the previous method is reduced, and it is achieved a higher robustness compared to stationary signal components, ie signal components due to Harmonic Sustained instruments.

In a block 22, a feature extraction and a classification operation are then performed. In particular, the components are divided into two subsets, namely first in a subset with the properties not percussive, so quasi-harmonic, and in another percussive subset .. In addition, the components with the property percussive / dissonant further classified in various instrument classes.

For the division into the two subsets the characteristics of percussivity or spectral dissonance are used.

• geglättete Version der spektralen Profile als Suchmuster in einer Trainingsdatenbank mit Profilen einzelner Instrumente, spektraler Zentroid, spektrale Ausbreitung, spektrale Schiefheit, Mittenfrequenzen, Intensitäten, Ausdehnung, Schiefheit der deutlichsten Partiallinien, ...

The following characteristics are used for instrument classification:

• smoothed version of the spectral profiles as a search pattern in a training database with profiles of individual instruments, spectral centroid, spectral propagation, spectral skew, center frequencies, intensities, extent, skewness of the clearest partial lines, ...

• Kick Drum, Snare Drum, Hi-Hat, Cymbal, Tom, Bongo, Conga, Woodblock, Cowbell, Timbales, Shaker, Tabla, Tambourine, Triangle, Daburka, Castagnets, Handclaps.

For example, the following instrument classes can be classified:

• Kick Drum, Snare Drum, Hi-Hat, Cymbal, Tom, Bongo, Conga, Woodblock, Cowbell, Timbale, Shaker, Tabla, Tambourine, Triangle, Daburka, Castagnets, Handclaps.

In a block 24, in order to further increase the robustness of the concept according to the invention, a decision can then be made for drum inserts or an acceptance or acceptance of percussive maxima. Thus, maxima with a transient rise in the amplitude envelope above a variable threshold are assumed to be a percussive event, while maxima having a transient rise below the variable threshold are discarded or recognized as an artifact and ignored. The variable threshold preferably varies with the total amplitude over a larger range around the maximum. The output is in an appropriate form that associates with the time of percussive events an instrument class, an intensity, and possibly other information, such as note or rhythm information in MIDI format.

At this point, it should be noted that the means 16 for extracting significant short-term spectra may be designed to perform this extraction on the basis of actual short-time spectra, as obtained, for example, in a short-time Fourier transformation. Particularly, in the example of application of the present invention in which the specific characteristic is the percussion characteristic, it is preferable to extract not actual short-term spectra but short-term spectra from a differentiated spectrogram, that is, differential spectra. The differentiation, as shown in block 16a in Figure 2, results in the sequence of short-term spectra into a sequence of derived or differentiated spectra, each (differentiated) short-term spectrum now containing the changes between an original spectrum and the next spectrum. This will be stationary shares in one Signal, so for example, signal components due to Harmonic Sustained instruments robust and reliably eliminated. This is because the differentiation emphasizes changes in the signal and suppresses equal proportions. Percussive instruments, however, are characterized by the fact that the sounds produced by these instruments are highly transient in terms of their time course.

Moreover, it is preferable to perform the PCA 18a and the non-negative ICA 18b, that is, more generally, the decomposition operation for decomposing the extracted short-term spectra in the block 18 of FIG. 1 not with the original short-term spectra but with the derived short-time spectra. In this case, the effect is exploited that for strongly transient signals the differentiated signal is very similar to the original signal before differentiation, which is the case in particular when there are very rapid changes in a signal. This applies to percussive instruments.

It should also be noted that disassembly means 18, which performs a PCA 18a with subsequent non-negative ICA (18b), anyway performs a weighted linear compensation of the extracted spectra provided by the device for determining a profile spectrum. This means that the extracted spectra as a whole are subjected to specific weighting factors calculated by the individual methods and are combined linearly, that is to say by subtraction or addition. Therefore, at least in part, the effect is observed that the means 18 for storing the extracted short-term spectra may have a differentiation-counteracting functionality, so that the profile spectra obtained for the sound sources are not differentiated profile spectra but the actual profile spectra are. In any case, it has been found that the use of differentiated spectra, ie difference spectra from a difference spectrogram in conjunction with a decomposition algorithm in the device 18, based on a weighted linear combination of the individual extracted spectra, to profile spectra for the individual High quality sound sources and high selectivity.

If, on the other hand, only stationary components were processed further, that is to say the specific characteristic is not a percussive but a harmonic characteristic, then it is preferable to achieve a preprocessing of the spectrogram by integration, ie by summation, in order to increase the stationary components compared to the transient components. In this case too, it is preferable to calculate the profile spectra for the individual - then harmonic - sound sources using the sum spectra, ie the integrated spectrogram.

Below, individual functionalities of the inventive concept are shown in more detail. However, in a preferred embodiment of the present invention, typical digital audio signals are initially preprocessed by preprocessing the device 8. Further, as a PCM audio signal inputted to the preprocessing means 8, it is preferable to supply 16-bit-per-second-wide mono-files at a sampling frequency of 44.1 Hz. These audio signals, that is, this stream of audio samples, which may also be a stream of video samples and generally a stream of information samples, are fed to pre-processor 8 for time-domain preprocessing using software-based emulation an acoustic effect device, often referred to as an "exciter". In this concept, the pre-processing stage 8 amplifies the high-frequency portion of the audio signal. This is accomplished by performing a non-linear distortion on a high-pass filtered version of the signal and adding the result of the distortion to the original signal. It turns out that this preprocessing is particularly favorable when hi-hats are to be judged, or similarly high-sounding low-intensity idiophones. Their energetic weight relative to the overall music signal is increased by this step, while most harmonically-endured and lower-pitched percussion instruments are not affected.

Another positive side effect is the fact that MP3 encoded and decoded files inherently low-pass filtered by this process again receive high frequency information.

A spectral representation of the preprocessed time signal is then obtained using the time / frequency means 12, which preferably performs a short-time Fourier transform (STFT).

To implement the time / frequency means, a relatively large block size of preferably 4096 values and high overlap are preferred. First, a good spectral resolution for the lower frequency range, ie for the lower spectral coefficients needed. Further, the temporal resolution is increased to a desired accuracy by obtaining a small hop size, that is, a small hop interval between adjacent blocks. In the preferred embodiment, as is 4096 samples per block subjected to a short-time Fourier transform, which corresponds to a time block length of 92 ms. The hop size is 10 ms. This means that each sample occurs over 9 consecutive times in a short-term spectrum.

The device 12 is designed to obtain an amplitude spectrum X. The phase information can also be calculated and, as will be explained later, used in the extreme value or maximum searcher 16c.

The magnitude spectrum X now has n frequency bins or frequency coefficients and m columns or frames, ie individual short-time spectra. The time-variant changes of each spectral coefficient are differentiated over all frames, by differentiator 16a, to minimize the impact of harmonic-sustaining sound sources, and to facilitate the subsequent detection of transients. The differentiation, which preferably has a difference between two short-term spectra of the sequence, may also have certain normalizations.

It should be noted that the differentiation may lead to negative values, so that half-wave rectification is performed in a block 16b to remove this effect. Alternatively, however, the negative signs could simply be reversed, which is not preferred in view of the later component decomposition.

Due to the rectifier 16b, a non-negative difference spectrogram is thus obtained, which is supplied to the maximum finder 16c.

The maximum searcher 16c performs event detection, which will be discussed below. The detection of multiple local extremes, and preferably local maxima associated with transient deployment events in the music signal, is performed by first defining a time tolerance that separates two consecutive percussion inserts. In the preferred embodiment, a time of 68 ms is used as a constant value derived from the time resolution and knowledge about the music signal. In particular, this value determines the number of individual spectra or differentiated individual spectra which must occur at least between two consecutive inserts. The use of this minimum distance is also supported by the observation that a sixteenth note takes 60 ms at an upper tempo limit of a very high tempo of 250 bpm.

In order to be able to carry out an automated maximum search, a detection function is derived from the differentiated and rectified spectrum, ie from the sequence of rectified (different) short-term spectra, on the basis of which the maximum search can be carried out. In order to get a value of this function for each time, a sum over all frequency coefficients or all spectral bins is simply determined. For smoothing this then resulting one-dimensional function over time, a convolution of the function obtained is performed with a suitable Hann window, so that a relatively smooth function e is obtained. In order to obtain the positions t of the maxima, a sliding window of the tolerance length is "pushed" over the entire path e in order to obtain the ability to obtain a maximum per step.

The reliability of the Maximas search is improved by preferentially retaining only the maxima that appear in a window for more than one time, since they are most likely to be the peaks of interest. Thus, it is preferred to use the maxima which represent a maximum over a predetermined threshold of times, for example three times, the threshold ultimately depending on the ratio of the block length to the hop size. From this it can be seen that a maximum, if it really is a significant maximum, must actually be a maximum a certain number of times, ie ultimately a certain number of overlapping spectra, if it is thought that with the numerical values previously shown, each sample in at least 9 consecutive short-term spectra "mixed in".

In the preferred embodiment of the present invention, as represented by the phase arrow, the unwrapped phase information of the original spectrogram is used as the reliability function. It has been found that in the phase information, a significant positive-going phase shift must occur in addition to an estimated insertion time t, thereby avoiding that small ripples are erroneously considered as "onsets".

According to the invention, a small section of the difference spectrogram, namely a short-term spectrum produced by differentiation, is then extracted and fed to the subsequent decomposition device.

The functionality of the device 18a for performing a principal component analysis will be discussed below. From the steps described in the previous section, the information about the time of occurrence t and the spectral compositions of the inserts, ie the extracted short-term spectra X _t , are derived. For real music signals, one typically finds a large number of transient events within the duration of the piece of music. Even with a simple example of a piece at a speed of 120 beats per minute (bpm), it turns out that in a four-minute segment, 480 events can be set, assuming that only quarter notes occur. With regard to the goal of finding only a few significant subspaces or profile spectra, principal component analysis (PCA) on X _t is thus applied to the extracted short-term spectra or to short-term spectra derived from the extracted short-term spectra.

Using this known technique, it is possible to reduce the entire set of collected short-term spectra to a limited number of decorrelated principal components, resulting in a good representation of the original data with little reconstruction error. For this purpose, an eigenvalue decomposition (EVD) of the covariance matrix of the data set is calculated. From the set of eigenvectors, the eigenvectors having the d largest eigenvalues are selected to provide the coefficients for the linear combination of the original vectors according to the following equation: $$\tilde{\mathrm{X}}\mathrm{=}{\hat{\mathrm{X}}}_t\mathrm{\cdot }\mathrm{T}$$

Therefore, T describes a transformation matrix that is actually a subset of the manifold of eigenvectors. In addition, the reciprocal values of the eigenvalues become used as scaling factors, which not only leads to a decorrelation, but also provides a variance normalization, which in turn leads to a whitening effect or a whitening effect. Alternatively, a singular value decomposition (SVD) of X _t can also be used. It has been found that the SVD is equivalent to the PCA with EVD. The whitened components X are subsequently fed to the ICA stage 18b, which will be discussed below.

Generally speaking, Independent Component Analysis (ICA) is a technique used to decompose a set of mixed linear signals into their original sources or component signals. A requirement for optimal behavior of the algorithm is the statistical independence of the sources. Preferably, a non-negative ICA is used that builds on the intuitive concept of optimizing a cost function that describes the non-negativity of the components. This cost function is related to a reconstruction error introduced by axis pair rotations of two or more variables in the positive quadrant of the common probability density function (PDF). The assumptions for this model imply that the original source signals are positive and have a nonzero PDF at zero, and that they are linearly independent to some extent. The first concept is always met, since the vectors subjected to the ICA result from the differentiated and half-wave equally weighted version X of the original spectrogram X , which thus never contains values less than zero, but certainly values equal to zero. The second constraint is taken into account when the spectra collected at operating times are considered to be the linear combinations of a small set of original source spectra containing the instruments considered characterize. This, of course, is a pretty rough approximation, but it turns out to be sufficiently good in a variety of cases.

Furthermore, it is preferably used that the spectra having inserts, and in particular the spectra of actual percussion instruments have no invariant structures, but are not subject to any changes in their spectral composition here. Nevertheless, it can be assumed, however, that there are characteristic features characteristic of spectral profiles of percussion sounds, thus allowing the whitened components X to be separated into their potential source or profile spectra F according to the following equation. $$\mathrm{F}\mathrm{=}\mathrm{A}\mathrm{\cdot }\stackrel{\mathrm{~}}{\mathrm{X}}$$

A denotes a dxd demixing matrix determined by the ICA process which actually separates the individual components X. The sources F are also called profile spectra in this document. Each profile spectrum has n frequency bins, just like a spectrum of the original spectrogram, but is identical for all times except the amplitude normalization, ie the amplitude envelope. This means that such a profile spectrum contains only the spectral information related to an onset spectrum of an instrument. In order to preferably circumvent any scaling of the components introduced by PCA and ICA, a transformation matrix R is used according to the following equation: $$\mathrm{R}\mathrm{=}\mathrm{T}\cdot {\mathrm{A}}^T$$

The normalization of R with its absolute maximum value results in weighting coefficients in a range of -1 to +1, so that spectral profiles extracted using the equation below $$\mathrm{F}\mathrm{=}{\hat{\mathrm{X}}}_t\mathrm{\cdot }\mathrm{R}$$

Have values in the range of the original spectrogram. Further normalization is achieved by dividing each spectral profile by its L2 norm.

As previously stated, the assumption of independence and the assumption of invariance for given short-term spectra are not always 100 percent satisfied. So it comes as no surprise that the spectral profiles obtained after demixing still have certain dependencies. However, this should not be considered a faulty behavior. Tests with spectral profiles of individual drum sounds have shown that the spectral profiles also have a strong dependence between the input spectra of different percussive instruments. One way to measure the degree of mutual overlap and similarity along the frequency axis is to perform crosstalk measurements. For illustrative purposes, the spectral profiles obtained from the ICA process may be considered as a transfer function of highly frequency selective parts in a filterbank, with overlapping passbands leading to crosstalk in the output of the filterbank channels. The crosstalk measure between two spectral profiles is calculated according to the following equation. $${\mathrm{C}}_{i.j}=\frac{{\mathrm{F}}_i\cdot {\mathrm{F}}_j^T}{{\mathrm{F}}_i\cdot {\mathrm{F}}_i^T}$$

In the above equation, i ranges from 1 to d, j ranges from 1 to d, and it holds that j is other than i. In fact, this value is related to the known cross-correlation coefficient, but uses a different normalization.

Based on the determined profile spectra, amplitude envelope determination is now performed in block 20 of FIG. For this purpose, the original spectrogram, that is to say the sequence of short-time spectra obtained, for example, by means 12 of FIG. 1 or in time / frequency / converter 12 of FIG. 2, is used. The following equation applies: $$\mathrm{e}\mathrm{=}\mathrm{F}\mathrm{\cdot }\mathrm{X}$$

As a second information source, the differentiated version of the amplitude envelopes can also be determined from the difference spectrogram according to the following equation: $$\hat{\mathrm{e}}\mathrm{=}\mathrm{F}\mathrm{\cdot }\hat{\mathrm{X}}$$

Essential to this concept is that no further ICA calculation is performed with the amplitude envelopes. Instead, the inventive concept gives highly specialized spectral profiles that are very close to the spectra of the instruments that actually appear in the signal. Nevertheless, the extracted amplitude envelopes are only in certain cases beautiful capture functions with sharp peaks, for example for dance-oriented music with very dominant percussive rhythm parts. Often, the amplitude envelopes contain smaller peaks and plateaus, which may be due to the above mentioned crosstalk effects.

Reference will now be made to a more detailed implementation of feature extraction and classification device 22. It is known that the actual number of components for real music signals is initially unknown. "Components" in this context mean both the spectral profiles and the corresponding amplitude envelopes. If the number d of extracted components is too low, artifacts of the disregarded components will most likely occur in other components. On the other hand, if too many components are extracted, the most prominent components are split into several components. Unfortunately, this split can occur even with the right number of components and sometimes make it difficult to detect the real components.

To overcome this problem, a maximum number d of components in the PCA or ICA process is specified. Subsequently, the extracted components are classified using a set of spectral-based and time-based features. The classification should provide two pieces of information. First, the components are to be eliminated from the further process, which are recognized with high certainty as non-percussive. Furthermore, the remaining components are to be assigned to predefined instrument classes.

A suitable measure of the discrimination of the amplitude envelopes is given by the percussivity, which is mentioned in the third technical publication. Here, a modified version is used, using the correlation coefficient between corresponding amplitude envelopes in Ê and E. The degree of correlation between the two vectors tends to be small when the characteristic plateaus are balanced Sounds are in the non-differentiated amplitude envelopes E emerge. These probably disappear in the differentiated version Ê . Both vectors are much more similar in the case of transient amplitude envelopes derived from percussive tones. For this purpose, reference is made to Fig. 3a and Fig. 4a. FIG. 3a shows a very fast and very high amplitude envelope for a percussive source, while FIG. 4a shows an amplitude envelope for a harmonically sustained instrument. Fig. 3a is an amplitude envelope for a kick drum, while Fig. 4a is an amplitude envelope for a trumpet. The amplitude envelope for the trumpet shows a relatively rapid rise, and then a relatively slow decay, as is typical of harmonic sustained instruments. On the other hand, the amplitude envelope for a percussive element, as shown in Fig. 3a, increases very rapidly and very sharply, but also drops again just as fast and steeply, as a drum sound typically does not linger very long due to the nature of the generation of this tone or decays.

The amplitude envelopes can thus be used as well for classification or feature extraction as the profiled spectra explained below, which in the case of a percussive source (Fig. 3b, Hi-Hat) and Fig. 4b in the case of a harmonically sustained instrument (guitar) clearly differ. For example, the harmonically sustained instrument shows a clear manifestation of the harmonics, whereas the percussive source has a rather noisy spectrum, which does not have pronounced harmonics, but overall has an area in which energy is concentrated Energy is concentrated, is very broadband.

Thus, preferably, a spectrally-based measure, that is, a measure derived from the profile spectra (eg, FIGS. 3b and 4b), is used to obtain spectra of harmonically sustained tones of spectra related to percussive sounds separate. Again, in the preferred embodiment, a modified version of the calculation of this measure is used, showing tolerance to spectral lag phenomena, all harmonics dissonance and proper normalization. A higher degree of computational efficiency is achieved by replacing an original dissonance function with a frequency pair weighting matrix.

The assignment of spectral profiles to a-priori-defined classes of percussive instruments is provided by a simple classifier for classifying the k nearest neighbors with spectral profiles of individual instruments as a training database. The distance function is calculated from at least one correlation coefficient between a query profile and a database profile. In order to verify the classification in cases of low reliability, ie at low correlation coefficients, or to verify a multiple occurrence of the same instruments, additional features that provide detailed information about the shape of the spectral profile are extracted. These include the previously mentioned individual features.

The functionality of the decider 24 in FIG. 2 will be discussed further below. Drum-type inserts are detected in the amplitude envelopes, such as in the amplitude envelope in Figure 3a, using conventional peak selection methods, also referred to as peak picking. Only peaks in a tolerance range in addition to the original times t, ie the times at which the maximum seeker 16c gave a result, are considered as candidates for missions. Remaining peaks extracted from the amplitude envelopes are first stored for further consideration. The value of the amount of the amplitude envelope is assigned to each insert candidate at its position. If this value does not exceed a predetermined dynamic threshold, then the mission is not accepted. The threshold varies over the amount of energy in a larger time range surrounding the inserts. Most of the crosstalk influence of harmonically sustained instruments as well as simultaneously playing percussive instruments can be reduced in this step. Further, it is preferred to distinguish whether concurrent inserts of different percussive instruments actually exist or only exist due to crosstalk effects. A solution to this problem is preferably to accept these further occurrence events whose value is relatively high compared to the value of the strongest instrument at the time of use.

According to the invention, an automatic detection and preferably also an automatic classification of non-pitched percussive instruments in real polyphonic music signals is thus achieved, the starting basis for this being the profile spectra on the one hand and the amplitude envelope on the other hand. From the percussive instruments, the rhythmic information of a piece of music can be well extracted, which in turn should lead to a favorable note-to-note transcription.

Depending on the circumstances, the inventive method for analyzing an information signal can be implemented in hardware or in software. The implementation may be on a digital storage medium, in particular a floppy disk or CD with electronically readable control signals, which can cooperate with a programmable computer system, that the method is performed. In general, the invention thus also consists in a computer program product with a program code stored on a machine-readable carrier for carrying out the method when the computer program product runs on a computer. In other words, the invention can thus be realized as a computer program with a program code for carrying out the method when the computer program runs on a computer.

Claims (25)

1. Device for analyzing an information signal, comprising:

   means (16) for extracting significant short-time spectra or significant short-time spectra, which are derived from short-time spectra of the information signal, from a time sequence of short-time spectra representing the information signal, the means (16) for extracting being configured to extract, from the time sequence, such short-time spectra which come closer to a specific characteristic than other short-time spectra of the information signal;

   means (18) for decomposing the extracted short-time spectra into component signal spectra, a component signal spectrum representing a profile spectrum of a tone source which generates a tone corresponding to the characteristic sought for, and another component signal spectrum representing a profile spectrum of another tone source which generates a tone corresponding to the characteristic sought for; and

   means (20) for calculating an amplitude envelope for the tone sources, an amplitude envelope for a tone source indicating how a profile spectrum of the tone source changes over time, using the profile spectra and a sequence of short-time spectra representing the information signal.
2. Device as claimed in claim 1, wherein the means (16) for extracting is configured to pre-process (8) the information signal such that signal portions present in the information signal at higher frequencies are accentuated, in the information signal, as compared to signal portions present in the information signal at lower frequencies.
3. Device as claimed in claim 2, wherein the means (16) for extracting is configured to
   subject the information signal to high-pass filtering,
   distort the high-pass filtered version of the information signal in a non-linear manner, and
   add the non-linearly distorted signal to the original information signal
   in pre-processing (8).
4. Device as claimed in one of the preceding claims, wherein the means (16) for extracting is configured to subject the information signal to a time-domain/frequency-domain conversion (12) to obtain a sequence of short-time spectra, two short-time spectra adjacent in time relating to portions of the information signal which overlap except for a hopping interval.
5. Device as claimed in claim 4, wherein each short-time spectrum comprises a sequence of spectral coefficients, and
   wherein the means (16) for extracting is configured to differentiate (16a) the sequence of short-time spectra in terms of time to obtain a sequence of differentiated short-time spectra, a differentiated short-time spectrum proving information about changes in a short-time spectrum as compared to a preceding or subsequent short-time spectrum.
6. Device as claimed in claim 5, wherein the means (16) for extracting is configured to obtain a differentiated short-time spectrum by forming, for each spectral coefficient, a difference of the spectral coefficient in a current short-time spectrum and a previous or subsequent short-time spectrum.
7. Device as claimed in claim 5 or 6, wherein the means (16) for extracting is configured to rectify (16b) the differentiated short-time spectra so that a rectified differentiated short-time spectrum does not exhibit any negative values.
8. Device as claimed in one of claims 5 to 7, wherein the means (16) for extracting is configured to determine significant short-time spectra on the basis of the differentiated short-time spectra.
9. Device as claimed in claim 8, wherein the means (16) for extracting is configured to sum up (16c), for each differentiated short-time spectrum, spectral coefficients, or values derived from spectral coefficients, from the differentiated short-time spectrum, so as to obtain a sum value for a short-time spectrum, so that a detection function over time results.
10. Device as claimed in claim 9, wherein the means (16) for extracting is configured to smooth the detection function over time.
11. Device as claimed in claim 9 or 10, wherein the means (16) for extracting is configured to find (16c) maxima in the detection function at a point in time, and to use a differentiated short-time spectrum or a short-time spectrum as a significant spectrum having a point in time associated therewith at which the detection function exhibits a maximum.
12. Device as claimed in one of claims 9 to 11, wherein the means (16) for extracting is configured to regard only such maxima of the detection function as significant that are spaced apart in time by more than a predefined time period.
13. Device as claimed in one of claims 4 to 12, wherein the means (16) for extracting is configured to determine amount spectra as a sequence of short-time spectra, and to use phase information of the short-time spectra when extracting the significant short-time spectra.
14. Device as claimed in one of the preceding claims, wherein the means (18) for decomposing is configured to add (18a) the extracted short-time spectra in a weighted manner so as to obtain a reduced number of extracted short-time spectra.
15. Device as claimed in one of claims 1 to 14, wherein the means (18) for decomposing is configured to perform (18a) a principal component analysis for dimension reduction so as to obtain processed short-time spectra.
16. Device as claimed in one of the preceding claims, wherein the means (18) for decomposing is configured to perform an independent component analysis (18b) to produce a plurality of component signals, a component signal being associated with an information source contributing to the information signal.
17. Device as claimed in one of the preceding claims, wherein the means (20) for calculating the amplitude envelope is configured to multiply a matrix including the profile spectra and a matrix including a sequence of short-time spectra of the information signal, so as to obtain the amplitude envelopes for the tone sources.
18. Device as claimed in one of the preceding claims, wherein the means for calculating the amplitude envelope is configured to further determine a differentiated amplitude envelope using the profile spectra for the tone sources and using the difference spectrogram.
19. Device as claimed in one of the preceding claims, further comprising means (22) for classifying the component signals into percussive component signals and non-percussive component signals.
20. Device as claimed in claim 19, wherein the means (22) for classifying is configured to perform a classification on the basis of the profile spectra and/or the amplitude envelopes.
21. Device as claimed in claim 19 or 20, wherein the means (20) for classifying is configured to extract a feature from the profile spectra or the amplitude envelopes, and to compare it with features of known sources in a database.
22. Device as claimed in one of the preceding claims, further comprising means (24) for examining the amplitude envelopes for a tone source to accept a maximum in the amplitude envelope as an onset of a signal from the tone source when the means (16) for extracting had extracted a significant short-time spectrum at a point in time similar within a threshold.
23. Device as claimed in one of the preceding claims, wherein the means (20) for calculating the amplitude envelope is configured to calculate the amplitude envelope for a tone source in such a manner that the amplitude envelope indicates how an intensity or weighting of a profile spectrum of the tone source changes over time.
24. Method for analyzing an information signal, comprising:

    extracting (16) significant short-time spectra or significant short-time spectra, which are derived from short-time spectra of the information signal, from a time sequence of short-time spectra representing the information signal, the short-time spectra extracted from the time sequence being such short-time spectra which come closer to a specific characteristic than other short-time spectra of the information signal;

    decomposing (18) the extracted short-time spectra into component signal spectra, a component signal spectrum representing a profile spectrum of a tone source which generates a tone corresponding to the characteristic sought for, and another component signal spectrum representing a profile spectrum of another tone source which generates a tone corresponding to the characteristic sought for; and

    calculating (20) an amplitude envelope for the tone sources, an amplitude envelope for a tone source indicating how a profile spectrum of the tone source changes over time, using the profile spectra and a sequence of short-time spectra representing the information signal.
25. Computer program having a program code adapted to perform the method for analyzing an information signal according to claim 24, when the computer program runs on a computer.

Images