EP1743324A1

EP1743324A1 - Device and method for analysing an information signal

Info

Publication number: EP1743324A1
Application number: EP05744658A
Authority: EP
Inventors: Christian Dittmar; Christian Uhle; Jürgen HERRE
Original assignee: Fraunhofer Gesellschaft zur Forderung der Angewandten Forschung eV
Current assignee: Fraunhofer Gesellschaft zur Forderung der Angewandten Forschung eV
Priority date: 2004-05-07
Filing date: 2005-04-29
Publication date: 2007-01-17
Anticipated expiration: 2025-04-29
Also published as: ATE377240T1; DE102004022660A1; WO2005114651A1; DE102004022660B4; DE502005001838D1; JP2007536587A; EP1743324B1

Abstract

In order to analyse an information signal, a significant short-time spectrum is extracted from the information signal. The extraction device (16) is embodied in such a way as to extract the short-time spectra which come closer to a specific characteristic than other short-time spectra of the information signal. The extracted short-time spectra are then decomposed (18) into component signals, by ICA analysis, a component signal spectrum representing a profile spectrum of a sound source which generates a sound corresponding to the required characteristic. An amplitude envelope is calculated (20) for each profile spectrum from a series of short-time spectra of the information signal and from the determined profile spectra, said envelope indicating how the profile spectrum of a sound source generally varies over time. The profile spectra and associated amplitude envelopes describe the information signal that can be further evaluated, e.g. for the purposes of a transcription in the case of a music signal.

Description

Device and method for analyzing an information signal

description

The present invention relates to the analysis of information signals, such as audio signals, and in particular to the analysis of information signals which consist of a superposition of partial signals, wherein a partial signal can originate from a single source or a group of individual sources.

The progressive development of digital distribution media for multimedia content leads to a great variety of offered data. The limit of manageability has long been exceeded for human users. The description of the content of data using metadata is becoming increasingly important. Basically, the goal is not only text files but also z. B. make music files, video files or other information signal files searchable, with the same convenience as in common text databases is sought. One approach to this is the well-known MPEG 7 standard.

The extraction of fingerprints is of great importance in particular when analyzing audio signals, that is to say signals which comprise music and / or speech.

The aim is also to "enrich" audio data with metadata, in order to retrieve metadata for a piece of music, for example, on the basis of a fingerprint. The "fingerprint" should be meaningful on the one hand, and be as short and concise as possible on the other. “Fingerprint” thus designates a com- pact generated from a music signal limited information signal, which does not contain the metadata, but is used for referencing the metadata, for example by searching a database, for example in a system for identifying audio material (“AudioID”).

Music data usually consists of superimposing partial signals from individual sources. While there is typically a relatively small number of individual sources in pop music, namely the singer, the guitar, the bass guitar, the drums and a keyboard, the number of sources for an orchestral piece can be very large. An orchestral piece and a pop music piece, for example, consist of an overlay of the tones emitted by the individual instruments. An orchestral piece or any piece of music thus represents a superposition of partial signals from individual sources, the partial signals being the tones generated by the individual instruments of the orchestra or pop music ensemble, and the individual instruments being individual sources.

Alternatively, groups of original sources can also be understood as individual sources, so that at least two individual sources can be assigned to a signal.

An analysis of a general information signal is shown below using an orchestral signal as an example. An orchestral signal can be analyzed in a number of ways. For example, there may be a desire to recognize the individual instruments and to extract the individual signals of the instruments from the overall signal and, if necessary, to convert them into a musical notation, the musical notation functioning as "metadata". Further possibilities of the analysis are a dominant rhythm to extract, with a rhythm extraction based on the percussion instruments This is done on the basis of the more tone-giving instruments, which are also referred to as harmonic sustained instruments. While percussion instruments typically include timpani, drums, racing or other percussion instruments, these are among the harmonic ones sustained instruments all other instruments, such as violins, wind instruments, etc.

The percussion instruments also include all those acoustic or synthetic sound generators that contribute to the rhythm section due to their sound characteristics (e.g. rhythm guitar).

For example, for rhythm extraction of a piece of music, it would be desirable to extract only percussive parts from the entire piece of music and then carry out a rhythm recognition on the basis of these percussive parts without the rhythm recognition being “disturbed” by signals from the harmonically sustained instruments.

On the other hand, any analysis with the aim of extracting metadata that only requires information from the harmonic instruments (e.g. a harmonic or melodic analysis) will benefit from an upstream separation and further processing of the harmonic components.

Recently, the use of blind source separation (BSS) and independent component analysis (ICA) technology for signal processing and signal analysis has been reported. Areas of application can be found in particular in biomedical engineering, communication technology, artificial intelligence and image processing.

In general, the term BSS encompasses techniques for separating signals from a mix of signals with a minimum of prior knowledge of the nature of the signals and the mixing process. The ICA is a process that makes use of the assumption that the sources on which a mix is based are at least to a certain extent statistically independent of one another. Furthermore, the mixing process is assumed to be unchangeable in time and the number of mixed signals observed is not less than the number of source signals on which the mixing is based.

An extension of the ICA is the Independent Subspace Analysis (ISA). Here the components are divided into independent subspaces or subspaces, the components of which do not have to be statistically independent. By transforming the music signal, a multi-dimensional representation of the mixed signal is determined and the last assumption for the ICA is met. Various methods for calculating the independent components have been developed in recent years. Relevant references, some of which also deal with the analysis of audio signals, are as follows:

[1] M.A. Casey and A. Westner, "Separation of Mixed Audio Sources by Independent Subspace Analysis," in Proc. of the International Computer Music Conference, Berlin, 2000

[2] IFO Orife, "Riddi: A rhythm analysis and decomposition tool based on independent subspace analysis", Master thesis, Darthmouth College, Hanover, New Hampshire, 2001 [3] C. Uhle, C. Dittmar and T. Sporer, "Extraction of Drum Tracks from polyphonic Music using Independent Subspace Analysis", in Proc. of the Fourth International Symposium on Independent Component Analysis, Nara, Japan, 2003

[4] D. Fitzgerald, B. Lawlor and E. Coyle, "Prior Subspace Analysis for Drum Transcription", in Proc. Of the 114th AES Convention, Amsterdam, 2003 [5] D. Fitzgerald, B. Lawlor and E. Coyle, "Drum Transcription in the presence of pitched Instruments using Prior Subspace Analysis", in Proc. of the ISSC, Limerick, Ireland, 2003

[6] M. Plubley, "Algorithms for Non-Negative Independent Component Analysis", in IEEE Transactions on Neural Networks, 14 (3), pp 534-543, May 2003

[1] shows a procedure for the separation of single sources from mono audio signals. In [2] an application for a separation into single tracks and then the rhythm analysis is given. In [3] a component analysis is carried out in order to achieve a separation into percussive and non-percussive sounds of a polyphonic piece. In [4], the Independent Component Analysis (ICA) is applied to amplitude bases that are obtained from a spectrogram representation of a drum track using generally calculated frequency bases. This is done for the purpose of transcription. In [5] this process is extended to polyphonic pieces of music.

Casey’s first publication mentioned above is shown below by way of example for the prior art. This publication describes a technique for separating mixed audio sources by technology the independent subspace analysis. For this purpose, an audio signal is split into individual component signals using BSS techniques. To determine which of the individual component signals belong to a multicomponent subspace, a grouping is carried out in such a way that the similarity of the components to one another is represented by a so-called Ixegram. The Ixegram is called the cross entropy matrix of the independent components. It is calculated by examining all individual component signals in pairs in a correlation calculation in order to find a measure of how similar two components are. An exhaustive pairwise similarity calculation is therefore carried out over all component signals, so that a similarity matrix results, in which all component signals are plotted along a y-axis and in which all component signals are also plotted along the x-axis. This two-dimensional array provides a measure of similarity for each component signal with a different component signal. The ixegram, ie the two-dimensional matrix, is now used to carry out clustering, for which purpose a grouping is carried out using a cluster algorithm based on diadic data. In order to optimally partition the Ixegram into k classes, a cost function is defined that eats the compactness within a cluster and determines the homogeneity between clusters. The cost function is minimized, so that ultimately there is an assignment of individual components to individual subspaces. Applied to a signal that represents a speaker in the context of a continuous waterfall noise, the speaker results as a subspace, the reconstructed information signal of the speaker subspace showing a significant attenuation of the waterfall noise. A disadvantage of the concepts described is the fact that the case that the signal components of a source come to lie on different component signals is very likely to occur. This is the reason why, as has been explained above, a complex and computation-intensive similarity calculation is carried out among all component signals in order to obtain the two-dimensional similarity matrix, on the basis of which a component function is then ultimately classified using a cost function to be minimized is carried out in subspaces.

A further disadvantage is that in the case where there are several individual sources, i.e. where the output signal is not known a priori, a similarity distribution does exist after a long calculation, but that the similarity distribution itself does not yet provide any actual insight into the actual audio scene. The viewer only knows that certain component signals are similar to one another with regard to the minimized cost function. However, he does not know which information these subspaces ultimately received or which original individual source or which group of individual sources are represented by a subspace.

The Independent Subspace Analysis (ISA) can thus be used to break down a time-frequency representation, eg a spectrogram, of an audio signal into independent component spectra. The previous methods described previously rely either on a calculation-intensive determination of frequency and amplitude bases from the entire spectrogram or on a priori defined frequency bases. Such a priori defined frequency bases or profile spectra consist, for example, in that one says that a trumpet is very likely to be in one piece and that a sample spectrum of a trumpet is then used for signal analysis.

The disadvantage of this procedure is that you know from the outset all the instruments that occur, which in principle runs counter to automated processing. Another disadvantage is that if you want to work precisely, there are not only trumpets, for example, but many different types of trumpets, all of which differ in their timbre and thus in their spectrum. If one were to proceed in such a way that all types of sample spectra are now used for component analysis, the process is again very complex and gets a very high level of redundancy, since typically not all conceivable different trumpets occur in one piece, but only one type of trumpet, that is with a single profile spectrum, or maybe with a few different timbres, ie a few profile spectra. It becomes even more problematic with different notes of a trumpet, especially since each tone has a spread / compressed profile spectrum depending on the pitch. To take this into account, the computing effort is also immense.

On the other hand, the disassembly due to ISA concepts is extremely computationally expensive and prone to failure if the entire spectrogram is used. It should be pointed out that a spectrogram typically consists of a sequence of individual spectra, a hopping time period being defined between the individual spectra, and a spectrum representing a certain number of samples, so that a spectrum has a certain length of time, ie a block of samples of the signal is associated. Typically, the duration of the block of samples from which a spectrum is calculated is repeated. is presented to be significantly greater than the hopping time in order to obtain a satisfactory spectrogram with regard to the required frequency resolution and with regard to the required time resolution. On the other hand, however, it can be seen that this spectrogram representation is extremely redundant. If, for example, the case is considered that a hopping time period is 10 ms and that a spectrum is based on a block of samples with a time length of 100 ms, for example, each sample occurs in 10 successive spectra. The redundancy generated in this way can drive the computing time requirements to astronomical heights, particularly when a larger number of instruments is sought.

Furthermore, the approach of working on the basis of the entire spectrogram is disadvantageous in those cases in which not all of the sources contained in a signal are to be extracted, but only, for example, sources of a certain type, that is to say sources that have a specific characteristic , Such a characteristic can relate to percussive sources, ie percussion instruments, or so-called pitched instruments, which are also referred to as harmonic-sustained instruments, which are typical melody instruments such as trumpet, violin, etc. A method that works on the basis of all of these sources is then too complex and ultimately not robust enough if, for example, only a few sources, namely the sources that are to fulfill a specific characteristic, are to be extracted. In this case, individual spectra of the spectrogram, in which such sources do not or only very slightly occur, falsify or "blur" the overall result, since these spectra of the spectrogram naturally also appear in the last Component analysis calculations like the significant spectra.

The object of the present invention is to create a robust and computationally time-efficient concept for analyzing an information signal.

This object is achieved by a device for analyzing an information signal according to claim 1, a method for analyzing an information signal according to claim 24 or a computer program according to claim 25.

The present invention is based on the finding that a robust and efficient information signal analysis is achieved by firstly extracting significant short-term spectra or short-term spectra derived from significant short-term spectra, such as difference spectra, etc., from the entire information signal or from the spectrogram of the information signal, with such Short-term spectra are extracted that come closer to a specific characteristic than other short-term spectra of the information signal.

Short-term spectra which have percussive components are preferably extracted, and thus short-term spectra which have harmonic components are not extracted. In this case, the specific characteristic is a percussive or drum characteristic.

The extracted short-term spectra or short-term spectra derived from the extracted short-term spectra are then fed to a device for decomposing the short-term spectra into component signal spectra, a component signal spectrum representing a profile spectrum of a sound source, which produces a sound that corresponds to the characteristic sought, and wherein another component signal spectrum represents a different profile spectrum of a sound source that generates a sound that also corresponds to the characteristic sought.

Finally, an amplitude envelope over time is calculated on the basis of the profile spectra of the sound sources, the determined profile spectra as well as the original short-term spectra being used for the calculation of the amplitude envelope over time, so that for each point in time at which a short-term spectrum was recorded, a Amplitude value is obtained.

The information obtained in this way, namely different profile spectra and amplitude envelopes for the profile spectra, provide a complete description of the music or information signal with regard to the specified characteristic, after which it has been extracted, so that this information may already be sufficient, in order to make a transcription, that is, to first use concepts of feature extraction and segmentation to determine which instrument "belongs" to the spectrum of profiles, and which rhythm is present, that is to say which rises and falls are present, which indicate notes of this instrument played at certain points in time ,

The present invention is advantageous in that the entire spectrogram is not used to compute the component analysis, but only extracted short-term spectra, that is to say the calculation of the independent subspace analysis (ISA) takes place only on the basis of a subset of all spectra, so that the computing requirements be lowered. Furthermore, the robustness with regard to the finding of certain sources is also increased, sometimes other short-term spectra that do not meet the specified characteristics, are not available in the component analysis and therefore do not represent a disturbance or "blurring" of the actual spectra.

In addition, the concept according to the invention is advantageous in that the profile spectra are determined directly from the signal without the problem of the prefabricated profile spectra resulting, which in turn would lead to either inaccurate results or to increased computational effort.

The concept according to the invention is preferably used for the detection and classification of percussive, non-harmonic instruments in polyphonic audio signals, in order to obtain both profile spectra and amplitude envelopes for the individual profile spectra.

Preferred embodiments of the present invention are explained in detail below with reference to the accompanying drawings. Show it:

1 shows a block diagram of the device according to the invention for analyzing an information signal;

2 shows a block diagram of a preferred embodiment of the device according to the invention for analyzing an information signal;

3a shows an example of an amplitude envelope for a percussive source;

3b shows an example of a profile spectrum for a percussive source; 4a shows an example of an amplitude envelope for a harmonic sustained instrument; and

4b shows an example of a profile spectrum for a harmoniously endured instrument.

1 shows a preferred exemplary embodiment of a device according to the invention for analyzing an information signal which is fed via an input line 10 to a device 12 for providing a sequence of short-term spectra which represent the information signal. As is shown by a detour line 14 in FIG. 1, which is drawn with a broken line, the information signal can also be supplied, for example in temporal form, to a device 16 for extracting significant short-term spectra or short-term spectra derived from the short-term spectra from the information signal, whereby the extracting device is designed to extract those short-term spectra that come closer to a specific characteristic than other short-term spectra of the information signal.

The extracted spectra, ie the original short-term spectra or the short-term spectra derived from the original short-term spectra, for example by differentiation, differentiation and rectification or by other operations, are fed to a device 18 for decomposing the extracted short-term spectra into component signal spectra, a component signal spectrum represents a profile spectrum of a sound source that produces a sound that corresponds to the characteristic sought, and another profile spectrum represents another sound source that generates a sound that also corresponds to the characteristic sought. The profile spectra are finally fed to a device 20 for calculating an amplitude envelope for the one sound source, the amplitude envelope indicating how the profile spectra of a sound source change over time, and in particular how the intensity or weighting of a profile spectrum changes over time. The device 20 is designed to work on the basis of the sequence of short-term spectra on the one hand and on the basis of the profile spectra on the other hand, as can be seen from FIG. 1. On the output side, the device 20 for calculating provides amplitude envelopes for the sources, while the device 18 supplies profile spectra for the sound sources. The profile spectra and the associated amplitude envelopes provide a complete description of the portion of the information signal that corresponds to the specific characteristic. This part is preferably the percussive part of a piece of music. Alternatively, this part could also be the harmonic part. In this case, the device for extracting significant short-term spectra would be designed differently than in the case in which the specific characteristic is a percussive characteristic.

A preferred exemplary embodiment of the present invention is illustrated below with reference to FIG. 2. Detection and classification of percussive, non-harmonic instruments is preferably carried out with the profile spectra F and the amplitude envelopes E, as is also represented by a block 22 in FIG. 2. However, this will be discussed later.

As can be seen from FIG. 2, the device 12 is designed to provide a sequence of short-term spectra in order to use a suitable time-frequency Transformation to generate an amplitude spectrogram X. The time / frequency device 12 is preferably a device for performing a short-term Fourier transformation with a specific hopping period, or comprises filter banks. Optionally, a phase spectrogram is also obtained as an additional information source, as shown by a phase arrow 13 in FIG. 2. A differentiation is then made by differentiation along the temporal extent of each individual spectrogram line, that is to say each individual frequency bin.

Renzspektrogramm X, as it is by the Differen¬

ornamental 16a is shown. The negative parts resulting from differentiation are set to zero or - alternatively - made positive. This results in a non-

negative difference spectrogram X. This non-negative

Difference spectrogram is fed to a maximum finder 16c, which is designed to search for the times t, that is to say for the indices of the corresponding spectrogram columns, for the occurrence of local maxima in a detection function e, which is calculated before the maximum finder 16c. As will be explained later, the detection function can, for example, be added up over all times

len of X and subsequent smoothing.

Optionally, it is preferred to use the phase information, which is supplied from block 12 to block 16c via phase line 13, as an indicator of the reliability of the maxima found. The spectra for which the maximum seeker detects a maximum in the detection function

are used as X _t and represent the extracted

Short-term spectra. Principle Component Analysis (PCA) is performed in block 18a. Here, a desired number of components d is first determined. Then the PCA is applied to the columns using a suitable procedure such as singular value decomposition or eigenvalue decomposition

the matrix X _t performed

X = X, T

The transformation matrix T causes a dimension reduction to X, which results in a reduction in the number of columns in this matrix. Decorrelation and normalization of variance are also achieved. A non-negative independent component analysis is then carried out in block 18b. Here, the method of non-negative independent component analysis on X shown in [6] for calculating a separation matrix A is carried out. According to the equation below, X is broken down into independent components.

F = A-X

Independent components F are interpreted as static spectral profiles or profile spectra of the sound sources that occur. The amplitude base or the amplitude envelope E is then extracted in a block 20 according to the following equation for the individual sound sources.

E = F-X

The amplitude base is interpreted as a set of time-varying amplitude envelopes of the corresponding spectral profiles. According to the invention, the spectral profile is obtained from the music signal itself. As a result, the computational complexity is reduced compared to the previous methods, and greater robustness is achieved with respect to stationary signal components, ie signal components due to harmony-sustained instruments.

A feature extraction and a classification operation are then carried out in a block 22. In particular, the components are differentiated into two subsets, namely first into a subset with the properties non-percussive, ie quasi-harmonic, and into another percussive subset. In addition, the components with the property percussive / dissonant are further classified in different instrument classes ,

The characteristics of percussiveness or spectral dissonance are used to divide the two subsets.

The following features are used for instrument classification:

smoothed version of the spectral profiles as a search pattern in a training database with profiles of individual instruments, spectral centroid, spectral spread, spectral skewness, center frequencies, intensities, extent, skewness of the clearest partial lines, ...

The following instrument classes can be classified, for example:

Kick Drum, Snare Drum, Hi-Hat, Cymbal, Tom, Bongo, Conga, Woodblock, Cowbell, Timbales, Shaker, Tabla, Tambourine, Triangle, Daburka, Castagnets, Handclaps. In block 24, a decision to use drums or to accept or accept percussive maxima can then be made to further increase the robustness of the inventive concept. Thus, maxima with a transient increase in the amplitude envelope above a variable threshold value are assumed to be a percussive event, while maxima with a transient increase below the variable threshold value are discarded or recognized as an artifact and ignored. The variable threshold value preferably varies with the overall amplitude in a larger area around the maximum. The output takes place in a suitable form, which assigns an instrument class, an intensity and possibly other information to the time of percussive events, such as note or rhythm information in MIDI format.

At this point, it should be pointed out that the device 16 can be designed to extract significant short-term spectra in order to carry out this extraction on the basis of actual short-term spectra, such as are obtained, for example, in the case of a short-term Fourier transformation. In particular in the application example of the present invention, in which the specific characteristic is the drum characteristic or the percussive characteristic, it is preferred not to extract actual short-term spectra, but rather short-term spectra from a differentiated spectrogram, that is to say from difference spectra. The differentiation, as shown in block 16a in FIG. 2, leads the sequence of short-term spectra to a sequence of derived or differentiated spectra, each

(Differentiated) short-term spectrum now contains the changes between an original spectrum and the next spectrum. This turns stationary parts into one Signal, for example signal components due to harmony sustained instruments, robustly and reliably eliminated. This is because the differentiation highlights changes in the signal and suppresses equal parts. However, percussive instruments are characterized by the fact that the tones generated by these instruments are highly transient with regard to their timing.

In addition, it is preferred to perform the PCA 18a and the non-negative ICA 18b, that is to say more generally, the decomposition operation for decomposing the extracted short-term spectra in block 18 of FIG. 1 not with the original short-term spectra but with the derived short-term spectra. The effect is exploited that, for strongly transient signals, the differentiated signal is very similar to the original signal before the differentiation, which is particularly the case when there are very rapid changes in a signal. This applies to percussive instruments.

Furthermore, it should be pointed out that the device 18 for disassembly, which a PCA 18a carries out with a subsequent non-negative ICA (18b), anyway carries out a weighted linear compensation of the extracted spectra, which are supplied by the device, in order to determine a profile spectrum. This means that the extracted spectra as a whole are subjected to certain weighting factors calculated according to the individual methods and are combined linearly, that is to say by subtraction or addition. Therefore, the effect is observed, at least in part, that the device 18 for storing the extracted short-term spectra can have a functionality that counteracts the differentiation, so that the profile spectra that are determined for the sound sources do not differentiated profile spectra but the actual profile spectra are. In any case, it has been found that the use of differentiated spectra, i.e. of difference spectra from a difference spectrograph in connection with a decomposition algorithm in the device 18, which is based on a weighted linear combination of the individual extracted spectra, to profile spectra for the individual sound sources of high quality and high selectivity.

If, on the other hand, only stationary parts were processed further, i.e. the specific characteristic is not a percussive but a harmonic characteristic, it is preferred to preprocess the spectrogram by integration, i.e. by summation, in order to reinforce the stationary parts compared to the transient parts. In this case too, it is preferred to calculate the profile spectra for the individual - then harmonic - sound sources using the sum spectra, ie the integrated spectrogram.

In the following, individual functionalities of the concept according to the invention are presented in more detail. In a preferred exemplary embodiment of the present invention, however, typical digital audio signals are first preprocessed by preprocessing the device 8. Furthermore, it is preferred to supply mono files with a width of 16 bits per sample at a sampling frequency of 44.1 Hz as the PCM audio signal which is input into the preprocessing device 8. These audio signals, that is to say this stream of audio samples, which can also be a stream of video samples and generally a stream of information samples, are fed to the preprocessing device 8 in order to carry out preprocessing in the time domain using a software-based emu In this concept, preprocessing stage 8 amplifies the high-frequency portion of the audio signal. This is accomplished by performing non-linear distortion with a high-pass filtered version of the signal, and by the result of the distortion is added to the original signal. It turns out that this preprocessing is particularly advantageous when it comes to judging hi-hats or similarly high-sounding idiophones with low intensity. Their energetic weight with regard to the overall music signal is thereby Step increased, while most harmonic-sustained instruments and percussion instruments with a lower tone are not affected.

Another positive side effect is the fact that MP3-encoded and decoded files that were inherently low-pass filtered by this process get radio frequency information again.

A spectral representation of the preprocessed time signal is then obtained using the time / frequency device 12, which preferably carries out a short-time Fourier transform (STFT; STFT = Short Time Fourier Transform).

A relatively large block size of preferably 4096 values and a high overlap are preferred for implementing the time / frequency device. First of all, a good spectral resolution is required for the lower frequency range, ie for the lower spectral coefficient. Furthermore, the temporal resolution is increased to a desired accuracy by maintaining a small hop size, that is to say a small hop interval between adjacent blocks. In the preferred embodiment, as is 4096 samples per block have been subjected to a short-time Fourier transformation, which corresponds to a temporal block length of 92 ms. A value of 10 ms is used as the hop size. This means that each sample value occurs more than 9 times in succession in a short-term spectrum.

The device 12 is designed to obtain an amplitude spectrum X. The phase information can also be calculated and, as will be explained later, used in the extreme value or maximum finder 16c.

The magnitude spectrum X now has n frequency bins or frequency coefficients and m columns or frames, ie individual short-term spectra. The time-variant changes of each spectral coefficient are differentiated across all frames or individual spectra, specifically by the differentiator 16a, in order to decimate the influence of harmonic sound sources and to simplify the subsequent detection of transients. The differentiation, which preferably has a difference between two short-term spectra of the sequence, can also have certain normalizations.

It should be noted that the differentiation can lead to negative values, so that half-wave rectification is carried out in a block 16b in order to remove this effect. As an alternative, however, the negative signs could also simply be reversed, but this is not preferred in view of the subsequent component decomposition.

Because of the rectifier 16b, a non-negative difference spectrogram is thus obtained, which is fed to the maximum searcher 16c. The maximum searcher 16c carries out an event detection, which will be discussed below. The acquisition of several local extreme values and preferably of local maxima, which are assigned to transient use events in the music signal, is carried out by first defining a time tolerance that separates two successive drum uses. In the preferred embodiment, a time of 68 ms is used as a constant value derived from the time resolution and knowledge of the music signal. In particular, this value determines the number of frames or individual spectra or differentiated individual spectra that must occur at least between two successive uses. The use of this minimum distance is also supported by the observation that a sixteenth note lasts 60 ms at an upper tempo limit of a very high tempo of 250 bpm.

In order to be able to carry out an automated maximum search, a detection function is derived from the differentiated and rectified spectrum, that is to say from the sequence of rectified (different) short-term spectra, on the basis of which the maximum search can be carried out. In order to get a value of this function for every point in time, a sum is simply determined over all frequency coefficients or all spectral bins. To smooth this resulting one-dimensional function over time, the function obtained is folded using a suitable Hann window, so that a relatively smooth function e is obtained. In order to obtain the positions t of the maxima, a sliding window of the tolerance length is "pushed" over the entire path e in order to achieve the ability to obtain a maximum per step. The reliability of the maximum search is improved by preferably only retaining the maxima that appear in a window for more than one point in time, since they are very likely the peaks of interest. Thus, it is preferred to use the maxima that represent a maximum over a predetermined threshold of times, for example three times, the threshold ultimately being dependent on the ratio of the block length to the hop size. It can be seen from this that a maximum, if it is really a significant maximum, a certain number of points in time, that is to say ultimately a certain number of overlapping spectra, must actually be a maximum if it is thought that with the previously shown numerical values, each sample value in "intermingled" at least 9 successive short-term spectra.

In the preferred embodiment of the present invention, as shown by the phase arrow, the unwrapped phase information of the original spectrogram is used as a reliability function. It has been found that a significant positively directed phase jump must occur in addition to an estimated application time t in the phase information, which prevents small ripples from being incorrectly regarded as inserts or “onsets”.

According to the invention, a small section of the difference spectrogram, namely a short-term spectrum created by differentiation, is now extracted and fed to the subsequent decomposition device.

The functionality of the device 18a for performing a principal component analysis is discussed below. addressed. The information about the time of occurrence t and the spectral compositions of the inserts, ie the extracted short-term spectra X _t , are thus derived from the steps described in the previous section. In the case of real music signals, a large number of transient events are typically found within the duration of the piece of music. Even a simple example of a piece at a speed of 120 beats per minute (bpm) shows that there can be 480 events in a four-minute section, provided that only quarter notes occur. The principal component analysis is used to find only a few significant subspaces or profile spectra

(PCA) on X _t on the extracted short-term spectra

or applied to short-term spectra derived from the extracted short-term spectra.

Using this known technique, it is possible to reduce the entire set of short-term spectra collected to a limited number of decorrelated principal components, resulting in a good representation of the original data with a small reconstruction error. For this purpose, an eigenvalue decomposition (EVD) of the covariance matrix of the data set is calculated. From the set of eigenvectors, the eigenvectors with the d largest eigenvalues are selected to provide the coefficients for the linear combination of the original vectors according to the following equation:

X = X, T

Therefore, T describes a transformation matrix that is actually a subset of the manifold of the eigenvectors. ^' In addition, the reciprocal values of the eigenvalues used as scaling factors, which not only leads to a decorrelation, but which also provides a standardization of variance, which in turn leads to a whitening or a whitening effect. Alternatively, a singular value decomposition (SVD) of X _t can also be used. It has been found that the SVD is equivalent to the PCA with EVD. The whitened components X are subsequently fed into the ICA stage 18b, which will be discussed below.

Generally speaking, Independent Component Analysis (ICA) is a technique that is used to convert a set of linear mixed signals into their original sources or

Disassemble component signals. A requirement for optimal behavior of the algorithm is the statistical independence of the sources. A non-negative ICA is preferably used, which is based on the intuitive concept of optimizing a cost function that describes the non-negativity of the components. This cost function is related to a reconstruction error introduced by axis pair rotations of two or more variables in the positive quadrant of the common probability density function (PDF). The assumptions for this model imply that the original source signals are positive and have a non-zero PDF at zero, and that they are linearly independent to some extent. The first concept is always fulfilled, since the vectors which are subjected to the ICA result from the differentiated and half-wave-balanced version X of the original spectrogram X, which thus never comprises values less than zero, but certainly values equal to zero. The second limitation is taken into account when the spectra collected at the time of use are considered to be the linear combinations of a small set of original source spectra that characterize instruments. Of course, this means a fairly rough approximation, but it turns out to be sufficiently good in the multitude of cases.

Furthermore, use is preferably made of the fact that the spectra that have inserts, in particular the spectra of actual drum instruments, have no invariant structures, but are not subjected to any changes with regard to their spectral composition here. Nevertheless, it can be assumed that there are characteristic properties which are characteristic of spectral profiles of drum sounds, which thus enable the whitened components X to be separated into their potential source or profile spectra F according to the following equation.

F = A-X

A denotes a d x d segregation matrix, which is determined by the ICA process, which actually separates the individual components X. Sources F are also referred to as profile spectra in this document. Like the spectrum of the original spectrogram, each profile spectrum has n frequency bins, but is identical for all times - apart from the amplitude normalization - that is, the amplitude envelope. This means that such a profile spectrum only contains the spectral information that relates to an onset spectrum of an instrument. In order to preferably bypass any scaling of the components introduced by PCA and ICA, a transformation matrix R is used in accordance with the following equation:

R = TA ⁷ The normalization of R with its absolute maximum value leads to weighting coefficients in a range from -1 to +1, so that spectral profiles are extracted using the following equation

F = X, R

Have values in the range of the original spectrogram. Further standardization is achieved by dividing each spectral profile through its L2 standard.

As has already been stated, the assumption of independence and the assumption of invariance for given short-term spectra is not always 100 percent fulfilled. So it is no surprise that the spectral profiles obtained after segregation can still have certain dependencies. However, this should not be seen as faulty behavior. Tests with spectral profiles of individual drum tones have shown that the spectral profiles also have a strong dependency between the application spectra of different percussive instruments. One way to measure the degree of mutual overlap and similarity along the frequency axis is by taking crosstalk measurements. For illustrative reasons, the spectral profiles obtained from the ICA process can be viewed as a transfer function of highly frequency-selective parts in a filter bank, with overlapping passbands leading to crosstalk in the output of the filter bank channels. The crosstalk measure between two spectral profiles is calculated according to the following equation.

In the above equation, i ranges from 1 to d, j ranges from 1 to d, and it holds that j is not equal to i. In fact, this value is related to the known cross-correlation coefficient, but it uses a different standardization.

Based on the determined profile spectra, an amplitude envelope determination is now carried out in block 20 of FIG. 2. The original spectrogram, i.e. the sequence of e.g. short-term spectra obtained by means 12 of FIG. 1 or in time / frequency / converter 12 of FIG. 2 are used. The following equation applies:

E = F X

The differentiated version of the amplitude envelopes from the difference spectrogram can also be determined as a second information source according to the following equation:

E = F X

It is essential to this concept that no further ICA calculation is carried out with the amplitude envelopes. Instead, the concept according to the invention provides highly specialized spectral profiles that are very close to the spectra of the instruments that actually appear in the signal. Nevertheless, the extracted amplitude envelopes are only in certain cases beautiful recording functions with sharp peaks, for example for dance-oriented music with very dominant percussive rhythm components. The amplitude envelopes often contain smaller peaks and plateaus, which can result from the above-mentioned cross talk effects. In the following, reference is made to a more detailed implementation of the device 22 for feature extraction and classification. It is known that the actual number of components for real music signals is initially unknown. In this context, "components" mean both the spectral profiles and the corresponding amplitude envelopes. If the number d of components extracted is too low, artifacts of the components not taken into account are very likely to occur in other components. On the other hand, if too many components are extracted, The most prominent components are divided into several components, which can disadvantageously occur even with the correct number of components and can sometimes make it difficult to record the real components.

In order to overcome this problem, a maximum number d of components is specified in the PCA or ICA process. The extracted components are then classified using a set of spectral-based and time-based features. The classification is intended to provide two pieces of information. First, the components from the further process that are recognized as non-percussive with high certainty are to be eliminated. Furthermore, the remaining components should be assigned to predefined instrument classes.

A suitable measure for the differentiation of the amplitude envelopes is given by the percussiveness, which is mentioned in the third specialist publication. A modified version is used here, in which the correlation coefficient between corresponding amplitude envelopes in

E and E is used. The degree of correlation between the two vectors tends to be small if the characteristic plateaus that are sustained on harmonic Tones related, appear in the undifferentiated amplitude envelopes E. These very likely disappear in the differentiated version E. Both vectors are much more similar in the case of transient amplitude envelopes that originate from percussive tones. For this purpose, reference is made to FIGS. 3a and 4a. In Fig. 3a a very fast and very high increasing amplitude envelope for a percussive source is shown, while in Fig. 4a an amplitude envelope for a harmonic sustained instrument is shown. Figure 3a is an amplitude envelope for a kick drum, while Figure 4a is an amplitude envelope for a trumpet. The amplitude envelope for the trumpet shows a relatively rapid rise, but then a relatively slow decay, as is typical for harmoniously sustained instruments. In contrast, the amplitude envelope for a percussive element, as shown in Fig. 3a, rises very quickly and very strongly and also falls again just as quickly and steeply, since a drum sound typically does not linger very long due to the nature of the generation of this sound or subsides.

The amplitude envelopes can thus be used for classification or feature extraction just as well as the profile spectra explained below, which are evident in the case of a percussive source (Fig. 3b; hi-hat) and Fig. 4b in the case of a harmoniously sustained instrument (guitar) differ. For example, the harmonic sustained instrument shows a clear expression of the harmonics, while the percussive source has a rather noise-like spectrum that does not have any clearly defined harmonics, but which overall has an area in which energy is concentrated, this area, where energy is concentrated, is very broadband. A spectral-based measure, i.e. a measure that is derived from the profile spectra (for example, FIGS. 3b and 4b), is therefore preferably used to obtain spectra of harmonic sustained tones from spectra related to percussive tones separate. Again, in the preferred embodiment, a modified version of the calculation of this measure is used, which shows a tolerance to spectral lag phenomena, a dissonance with all harmonics and a suitable standardization. A higher level of computational efficiency is achieved by replacing an original dissonance function with a weighting matrix for frequency pairs.

The assignment of spectral profiles to a-priori-defined classes of percussive instruments is created by a simple classifier for classifying the k nearest neighbors with spectral profiles of individual instruments as a training database. The distance function is calculated from at least one correlation coefficient between a query profile and a database profile. In order to verify the classification in cases of low reliability, that is to say with low correlation coefficients, or to verify a repeated occurrence of the same instruments, additional features which provide detailed information about the shape of the spectral profile are extracted. These include the individual features already mentioned.

The functionality of the decision maker 24 in FIG. 2 will be discussed further below. Drum-like inserts are captured in the amplitude envelopes, such as the amplitude envelope in Fig. 3a, using conventional tip selection techniques, also referred to as peak picking. Only peaks in a tolerance range Rich in addition to the original times t, that is to say the times in which the maximum seeker 16c delivered a result, are primarily regarded as candidates for missions. Remaining peaks extracted from the amplitude envelopes are initially saved for further considerations. The value of the magnitude of the amplitude envelope is assigned to each candidate candidate at his position. If this value does not exceed a predetermined dynamic threshold, then the bet will not be accepted. The threshold varies across the amount of energy in a larger temporal area surrounding the stakes. Most of the crosstalk influence of harmoniously sustained instruments as well as percussive instruments playing at the same time can be reduced in this step. Furthermore, it is preferred to differentiate whether simultaneous inserts from different percussive instruments actually exist or only exist due to crosstalk effects. A solution to this problem is preferably to accept these further occurrence events, the value of which is relatively high compared to the value of the strongest instrument at the time of use.

According to the invention, automatic detection and preferably also automatic classification of non-pitched percussive instruments in real polyphonic music signals is thus achieved, the starting point for this being the profile spectra on the one hand and the amplitude envelope curve on the other hand. The rhythmic information of a piece of music can also be extracted well from the percussive instruments, which in turn should lead to a favorable note-to-note transcription.

Depending on the circumstances, the invention

Methods for analyzing an information signal can be implemented in hardware or in software. The implement Menting can take place on a digital storage medium, in particular a floppy disk or CD with electronically readable control signals, which can cooperate with a programmable computer system in such a way that the method is carried out. In general, the invention thus also consists in a computer program product with a program code stored on a machine-readable carrier for carrying out the method when the computer program product runs on a computer. In other words, the invention can thus be implemented as a computer program with a program code for carrying out the method if the computer program runs on a computer.

Claims

claims

1. An apparatus for analyzing an information signal, having the following features: a device (16) for extracting significant short-term spectra or significant short-term spectra derived from short-term spectra of the information signal from the information signal, the device (16) being designed for extracting such short-term spectra extract that come closer to a specific characteristic than other short-term spectra of the information signal; a device (18) for decomposing the extracted short-term spectra into component signal spectra, a component signal spectrum representing a profile spectrum of a sound source that generates a sound that corresponds to the characteristic sought, and another component signal spectrum representing a profile spectrum of another sound source that is a sound generated that corresponds to the characteristic sought; and means (20) for calculating an amplitude envelope for the sound sources, an amplitude envelope for a sound source indicating how a profile spectrum of the sound source changes over time, using the profile spectra and a sequence of short-term spectra representing the information signal.

2. Device according to claim 1, in which the device (16) is designed for extracting in order to preprocess the information signal in such a way (8) that signal portions in the information signal at higher frequencies than signal portions in the information signal at lower frequencies in the information signal are emphasized.

3. Apparatus according to claim 2, wherein the means (16) for extracting is designed to subject the information signal to a high pass filter in the preprocessing (8), to distort the high pass filtered version of the information signal non-linearly, and to distort the non-linearly Add signal to the original information signal.

4. Device according to one of the preceding claims, wherein the device (16) is designed to extract the information signal to a time domain frequency domain conversion (12) to obtain a sequence of short-term spectra, wherein two temporally adjacent short-term spectra refer to sections of the information signal that overlap apart from a hopping interval.

5. Apparatus according to claim 4, in which each short-term spectrum each has a sequence of spectral coefficients, and in which the device (16) is designed to extract in order to differentiate the sequence of short-term spectra in time (16a) in order to obtain a sequence of differentiated ones To obtain short-term spectra, where in the case of a differentiated short-term spectrum, evidence of changes in a short-term spectrum to a short-term spectrum preceding or following in time.

6. Apparatus according to claim 5, in which the device (16) is designed for extracting in order to obtain a differentiated short-term spectrum by forming a difference of the spectral coefficient in a current short-term spectrum and a previous or subsequent short-term spectrum for each spectral coefficient.

7. Apparatus according to claim 5 or 6, in which the device (16) is designed for extraction in order to rectify the differentiated short-term spectra (16b), so that a rectified differentiated short-term spectrum has no negative values.

8. Device according to one of claims 5 to 7, wherein the device (16) is designed for extracting in order to determine significant short-term spectra based on the differentiated short-term spectra.

9. The device according to claim 8, wherein the means (16) is designed to extract, for each differentiated short-term spectrum, spectral coefficients or values derived from spectral coefficients from the differentiated short-term spectrum (16c), in order to obtain a sum value for a short-term spectrum, so that gives a detection function over time.

10. The apparatus of claim 9, wherein the means (16) for extracting is designed to smooth the detection function over time.

11. The apparatus of claim 9 or 10, wherein the means (16) for extracting is designed to find maxima in the detection function at a time (16c), and to use a differentiated short-term spectrum or a short-term spectrum as a significant spectrum, the a point in time is assigned at which the detection function has a maximum.

12. The device according to one of claims 9 to 11, wherein the device (16) is designed for extracting to consider only maxima of the detection function as significant, which are spaced apart from one another by more than a predefined period of time.

13. Device according to one of claims 4 to 12, wherein the device (16) is designed for extracting to determine magnitude spectra as a result of short-term spectra and to use phase information of the short-term spectra in the extraction of the significant short-term spectra.

14. Device according to one of the preceding claims, in which the device (18) is designed to decompose to add the extracted short-term spectra weighted (18a) in order to obtain a reduced number of extracted short-term spectra.

15. The device according to one of claims 1 to 14, wherein the device (18) is designed for disassembly in order to reduce the dimensions a principal component Perform analysis (18a) to obtain processed short-term spectra.

16. Device according to one of the preceding claims, in which the means (18) for decomposing is designed to carry out an independent component analysis (18b) in order to generate a plurality of component signals, a component signal being a source of information relating to the Contributes information signal is assigned.

17. Device according to one of the preceding claims, in which the means (20) for calculating the amplitude envelope is designed to multiply a matrix which comprises the profile spectra and a matrix which comprises a sequence of short-term spectra of the information signal for the sound sources to get the amplitude envelopes.

18. Device according to one of the preceding claims, wherein the means for calculating the amplitude envelope is designed to further determine a differentiated amplitude envelope using the profile spectra for the sound sources and using the difference spectrogram.

19. Device according to one of the preceding claims, further comprising means (22) for classifying the component signals into percussive component signals and non-percussive component signals.

20. The apparatus of claim 19, wherein the classifying device (22) is designed to classify on the basis of the profile spectra and / or the amplitude envelopes.

21. The apparatus of claim 19 or 20, wherein the device (20) is designed to classify in order to extract a feature from the profile spectra or the amplitude envelopes and to compare them with features of known sources in a database.

22. The apparatus of any preceding claim, further comprising means (24) for examining the amplitude envelopes for a sound source to accept a maximum in the amplitude envelope as an insert of a signal from the sound source when the means (16) for Extract had extracted a significant short-term spectrum at a point in time similar to a threshold.

23. Device according to one of the preceding claims, in which the means (20) for calculating the amplitude envelope is designed to calculate the amplitude envelope for a sound source such that the amplitude envelope indicates how an intensity or weighting of a profile spectrum of the sound source changes over time.

24. A method of analyzing an information signal, comprising the following steps:

Extracting (16) significant short-term spectra or significant short-term spectra derived from short-term spectra of the information signal, extracting those short-term spectra which come closer to a specific characteristic than other short-term spectra of the information signal; Decomposing (18) the extracted short-term spectra into component signal spectra, a component signal spectrum representing a profile spectrum of a sound source that generates a sound that corresponds to the characteristic sought, and another component signal spectrum representing a profile spectrum of another sound source that generates a sound, which corresponds to the characteristic sought; and calculating (20) an amplitude envelope for the sound sources, an amplitude envelope for a sound source indicating how a profile spectrum of the sound source changes over time, using the profile spectra and a sequence of short-term spectra representing the information signal.

25. Computer program with a program code for performing the method for analyzing an information signal according to claim 24, when the computer program runs on a computer.