EP1671315B1 - Process and device for characterising an audio signal - Google Patents

Abstract

In order to characterise an audio signal, a sequence of quantified application points in time for each of at least two audio sources is prepared on the basis of a quantification grid (1). A common period length at the basis of the at least two audio sources is determined using the sequences of application points (12). The sequence of application points is then subdivided into corresponding subsequences (14), the length of a subsequence being equal to the common period length. Finally, the subsequences for the first audio source are combined into a first combined subsequence and the subsequences for the second audio source are combined into a second combined subsequence (16), using a pattern histogram, for example, in order to characterise the audio signal, for example, its rhythm, speed or type, on the basis of the first combined subsequence and of the second combined subsequence.

Description

The present invention relates to the analysis of audio signals, and more particularly to the analysis of audio signals for purposes of classifying and identifying audio signals to characterize the audio signals.

The progressive development of digital distribution media for multimedia content leads to a large variety of data offered. For the human user, the limit of the manageable has long been exceeded. Thus, the content description of the data by metadata becomes increasingly important. Basically, the goal is not only text files but also z. B. music files, video files or other information signal files to make searchable, with the same comfort as in common text databases is sought. One approach to this is the well-known MPEG 7 standard.

In particular, in the analysis of audio signals, ie signals that include music and / or speech, the extraction of fingerprints is of great importance.

The aim is also to "enrich" audio data with metadata to z. B. to recover a piece of music based on a fingerprint metadata. The "fingerprint" should on the one hand be meaningful, and on the other hand be as short and concise as possible. "Fingerprint" thus refers to a compressed generated from a music signal Information signal, which does not contain the metadata, but for referencing to the metadata eg by searching in a database is used, for example in a system for the identification of audio material ("AudioID").

Usually, music data consists of the superposition of sub-signals from single sources. While pop music typically has relatively few individual sources, namely the singer, the guitar, the bass guitar, the drums, and a keyboard, the number of sources for an orchestral piece can become very large. An orchestral piece and a pop music piece, for example, consist of a superposition of the tones emitted by the individual instruments. An orchestral piece or piece of music thus represents a superimposition of partial signals from individual sources, the partial signals being the sounds produced by the individual instruments of the orchestra or pop music ensemble, and the individual instruments being individual sources.

Alternatively, groups of original sources can also be considered as individual sources, so that at least two individual sources can be assigned to one signal.

An analysis of a general information signal is shown below by way of example only with reference to an orchestra signal. The analysis of an orchestra signal can be done in many ways. Thus, there may be a desire to recognize the individual instruments and to extract from the overall signal the individual signals of the instruments and possibly to convert them into a musical notation, wherein the notation would act as "metadata". Other possibilities of analysis exist in extracting a dominant rhythm, whereby rhythm extraction on the basis of the percussion instruments is better than on the basis of the more sound-giving instruments, which are also referred to as harmonic-sustained or "harmonic sustained" instruments. While percussion instruments typically include timpani, drums, rattles or other percussion instruments, the harmonic sustained instruments include all other instruments such as violins, wind instruments, etc.

Furthermore, the percussion instruments include all those acoustic or synthetic tone generators that contribute to the rhythm section due to their sound characteristics (e.g., rhythm guitar).

For example, it would be desirable for the rhythm extraction of a piece of music to extract only percussive parts from the entire piece of music and then perform rhythm recognition on the basis of these percussive parts, without the rhythm recognition being "disturbed" by signals from the harmonically sustained instruments.

There are various possibilities in the art for automatically extracting different patterns from pieces of music or detecting the presence of patterns. In Coyle, EJ, Shmulevich, I., "A System for Machine Recognition of Music Patterns", IEEE Int. Conf. on Acoustic, Speech, and Signal Processing, 1998, http://www2.mdanderson.org/app/ilya/Publications/icassp98mp r.pdf , searches for melodic themes. For this purpose, a topic is given. Then it is searched where it occurs.

In Schroeter, T., Doraisamy, S., Rüger, S., "From Raw Polyphonic Audio to Locating Recurring Themes", ISMIR, 2000, http://ismir2000.ismir.net/poster/shroeter_ruger.pdf, is becoming melodic Searching for topics in a transcribed representation of the music signal. Again, the topic is given and searched where it occurs.

In contrast to the usual structure of occidental music, melodic fragments, unlike the rhythmic structure, usually do not appear periodically. For this reason, many methods of searching for melodic fragments are limited to the individual finding of their occurrence. In contrast to this, in the field of rhythmic analysis, the interest is preferentially in finding periodic structures.

In Meudic, B., "Musical Pattern Extraction: from Repetition to Musical Structure", in Proc. CMMR, 2003, http: //www.ircam.fr/equipes/repmus/RMPapers/ CMMR-meudic2003.pdf identify melodic patterns using a self-similarity matrix .

Meek, Colin, Birmingham, WP, "Thematic Extractor", ISMIR, 2001, http://ismir2001.ismir.net/pdf/meek.pdf, is looking for melodic themes. In particular, sequences are searched, wherein the length of a sequence of two notes can be up to a predetermined number.

Smith, L., Medina, R. "Discovering Themes by Exact Pattern Matching", 2001, http://citeseer.ist.psu. edu / 498226.html is looking for melodic themes with a self-similarity matrix.

In Lartillot, O., Perception-Based Musical Pattern Discovery, in Proc. IFMC, 2003, http://www.ircam.fr/equipes/ repmus / lartillot / cmmr / cmmr.pdf is also looking for melodic themes.

Brown, J.C., "Determination of the Meter of Musical Scores by Autocorrelation," J. of the Acoust, Soc. Of America, vol. 94, no. 4, 1993 is determined from a symbolic representation of the music signal, namely on the basis of a MIDI representation using a periodic function (autocorrelation), the time signature of the underlying piece of music.

Similarly, in Meudic, B. "Automatic Meter Extraction from MIDI files", Proc. JIM, 2002, http://www.ircam. fr / equipes / repmus / RMPapers / JIM-benoit2002.pdf , where on the estimation of periodicities a tempo and clock estimate of audio signals is made.

Goto, M, "Real-time beat tracking for drumless audio signals: Chord change detection for musical decisions", Speech Communication, Elsevier, No. 3-4, April 1999, pages 311-335, discloses a means for providing a sequence of application times of sounds for at least one sound source disclosed.

Methods for identifying melodic themes are only of limited suitability for identifying periodicities present in a sound signal, since, as has been said, musical themes are recurrent, but not so much a basic periodicity in a piece of music, but rather, when have at all higher-level periodicity information in itself. In any case, methods for the identification of melodic themes are very complex, since in the search for melodic themes the different variations of the topics must be considered. So it is known from the music world that topics are usually varied, namely, for example, by transposition, mirroring, etc.

The object of the present invention is to provide an efficient and reliable concept for characterizing a sound signal.

This object is achieved by a device for characterizing a sound signal according to claim 1, a method for characterizing a sound signal according to claim 20 or a computer program according to claim 21.

The present invention is based on the finding that a characteristic of a sound signal which can be calculated efficiently and is informative with respect to many information can be determined on the basis of a sequence of application times by period length determination, division into subsequences and summary into a combined subsequence as a characteristic.

Furthermore, preferably not only a single sequence of deployment times of a single instrument, ie a single sound source along the time is considered, but at least two episodes of deployment times of two different sound sources occurring in parallel in the music piece are considered. Since it can typically be assumed that all sound sources or at least one subset of sound sources, such as the percussive sound sources in a piece of music, are based on the same period length, a common period length is determined using the sequences of application times of the two sound sources based on at least two sound sources. According to the invention, each sequence of deployment times is subdivided into respective subsequences, wherein a length of a subsequence is equal to the common period length.

The characteristic extraction then takes place on the basis of a summary of the subsequences for the first sound source into a first combined subsequence and on the basis of a summary of the subsequences for the second sound source into a second combined subsequence, the combined subsequences being characteristic of the sound signal can be used and used for further processing, such as for extracting semantically meaningful information about the entire piece of music, such as genre, tempo, time signature, similarity to other pieces of music, etc.

The combined subsequence for the first sound source and the combined subsequence for the second sound source thus form a drum pattern of the sound signal if the two sound sources, which have been taken into account based on the sequence of application times, are percussive sound sources, such as drums, other drum Instruments or any other percussive instruments, which are characterized by the fact that their pitch does not decide their pitch, but that their characteristic spectrum or the rise and fall of an output sound and not the pitch of higher musical importance.

The procedure according to the invention thus serves for the automatic extraction of preferably drum patterns from a preferably transcribed, so z. B. note representation of a music signal. This representation may be in MIDI format or automatically determined from an audio signal using digital signal processing techniques. such as Independent Component Analysis (ICA) or certain variations thereof, such as non-negative Independent Component Analysis, or generally concepts known as Blind Source Separation (BSS).

In a preferred embodiment of the present invention, for the extraction of a drum pattern, recognition of the note inserts, ie start times, for each different instrument and pitch for tonal instruments is first performed. Alternatively, a reading out of a score can take place, wherein this reading can consist in a reading in of a MIDI file or can consist in a scanning and image processing of a musical notation or in the acceptance of manually typed notes.

Hereupon, in a preferred embodiment of the present invention, a raster is determined according to which the billet times are quantized, after which the billboard times are then quantized.

The length of the drum pattern is then determined as the length of a musical measure, as an integral multiple of the length of a musical measure, or as an integral multiple of the length of a musical count.

This is followed by a determination of a frequency of occurrence of a specific instrument per metric position with a pattern histogram.

Then, a selection of the relevant entries is made, finally, a shape of the drum pattern as the preferred one Characteristic for the sound signal. Alternatively, the pattern histogram can be processed as such. The pattern histogram is also a compressed representation of the musical events, ie the score, and contains information on the degree of variation and preferred beats, with flatness of the histogram indicative of a large variation, while a very "mountainous" histogram indicates a high indicates stationary signal in the sense of a self-similarity.

To improve the validity of the histogram, it is preferred to first perform a preprocessing to subdivide a signal into characteristic mutually similar regions of the signal and to extract a drum pattern only for mutually similar regions in the signal and another for other characteristic regions in the signal Determine drum pattern.

The present invention is advantageous in that a robust and efficient way of calculating a characteristic of a sound signal is obtained, in particular due to the subdivision carried out, which is very robust and equally feasible for all signals according to the period length which can also be determined by statistical methods. Furthermore, the concept according to the invention is scalable to the extent that the meaningfulness and accuracy of the concept can be increased at the price of a higher computing time without further ado that more and more episodes of occurrence of more and more different sound sources, ie instruments, in the determination of common Period length and are included in the determination of the drum pattern, so that the calculation of the summarized subsequences becomes more and more complex.

However, an alternative scalability is also to calculate a certain number of combined subsequences for a certain number of sound sources, and then, depending on the further processing interest to rework the resulting summarized subsequences and thus reduce their explanatory power as needed. Histogram entries below a certain threshold may e.g. B. be ignored. However, histogram entries can also be quantized per se or only be generally binarized depending on the threshold decision to the effect that a histogram merely contains the statement that a histogram entry is or is not in the summarized subsequence at a specific point in time.

The concept according to the invention is a robust method due to the fact that many subsequences are "merged" into a combined subsequence, but nevertheless can be executed efficiently since no numerically intensive processing steps are required.

In particular, percussive instruments without pitch, which are also called drums in the following, play an essential role, especially in popular music. Lots of information about rhythm and musical genre is in the drums played "notes", which z. B. could be used in an intelligent and intuitive search in music archives to perform classifications or at least Vorklassifikationen can.

The notes played by drums often form recurring patterns, also known as drum patterns. A drum pattern can serve as a compressed representation of the played notes by extracting a note of the length of a drum pattern from a longer note image. This can be extracted from drum pattern semantically meaningful information about the entire piece of music, such as genre, tempo, time signature, similarity to other pieces of music, etc.

Fig. 1

ein Blockschaltbild einer erfindungsgemäßen Vorrichtung zum Charakterisieren eines Tonsignals;

Fig. 2

eine schematische Darstellung zur Erläuterung der Ermittlung der Noteneinsatzpunkte;

Fig. 3

ein schematisches Diagramm zur Darstellung eines Quantisierungsrasters und einer Quantisierung der Noten anhand des Rasters;

Fig. 4

eine beispielhafte Darstellung von gemeinsamen Periodenlängen, die durch statistische Periodenlängenbestimmungen unter Verwendung sämtlicher Instrumente erhalten werden können;

Fig. 5

ein beispielhaftes Pattern-Histogramm als Beispiel für zusammengefasste Unterfolgen für die einzelnen Tonquellen (Instrumente); und

Fig. 6

ein nachverarbeitetes Pattern-Histogramm als Beispiel für ein alternatives Charakteristikum des Tonsignals.

Preferred embodiments of the present invention will be explained below in detail with reference to the accompanying drawings. Show it:

Fig. 1

a block diagram of a device according to the invention for characterizing a sound signal;

Fig. 2

a schematic representation for explaining the determination of the Noteneinsatzpunkte;

Fig. 3

a schematic diagram showing a quantization grid and a quantization of the notes on the grid;

Fig. 4

an exemplary representation of common period lengths that can be obtained by statistical period length determinations using all instruments;

Fig. 5

an exemplary pattern histogram as an example of summarized subsequences for the individual sound sources (instruments); and

Fig. 6

a postprocessed pattern histogram as an example of an alternative characteristic of the audio signal.

Fig. 1 shows an inventive device for characterizing a sound signal. First, FIG. 1 includes means 10 for providing a sequence of deployment times for each sound source from at least two sound sources over time. The deployment times are preferably already quantized deployment times, which are present in a quantization grid. While FIG. 2 shows a sequence of application times of notes from different sound sources, ie instruments 1, 2,..., N, which are designated by "x" in FIG. 2, FIG. 3 shows one in a raster in FIG FIG. 3 shows a quantized sequence of quantized use times for each sound source, ie for each instrument 1, 2,..., N.

FIG. 3 simultaneously represents a matrix or list of insertion times, wherein a column in FIG. 3 corresponds to a distance between two grid points or grid lines and thus represents a time interval in which a note insert is present or not depending on the sequence of use times. In the embodiment shown in Fig. 3 z. B. in the column, which is designated by the reference numeral 30, a note insert from instrument 1, and this also applies to the instrument 2, as indicated by the "x" in the two lines 1 and 2 associated with the instruments in FIG. 3 is indicated. On the other hand, the instrument n has no note insertion time in the time interval shown by the reference numeral 30.

The plurality of sequences of preferably quantized application times are supplied by the device 10 to a means 12 for determining a common period length. The means 12 for determining a common period length is designed so as not to determine its own period length itself for each succession of application times, but to find a common period length which is most likely to underlie the at least two sound sources. This is based on the fact that even if z. B. play several percussion instruments in one piece, all play more or less the same rhythm, so that a common period length must exist to which virtually all instruments that contribute to the audio signal, so all sound sources will hold.

The common tone period length is then supplied to a means 14 for dividing each sequence of use times to obtain on the output side a set of subsequences for each sound source.

For example, referring to FIG. 4, it can be seen that a common period length 40 has been found for all of the instruments 1, 2,..., N, where the means 14 is arranged to be divided into subsequences to all To divide sequences of deployment times into subsequences of the length of the common period length 40. The sequence of application instants for the instrument would then, as shown in FIG. 4, be divided into a first subsequence 41, a subsequent second subsequence 42 and a subsequent subsequence 43, in order thus to illustrate the example of FIG the sequence for the instrument 1 to obtain three subsequences. Similarly, the other consequences for the instruments 2, ..., n also become divided into corresponding contiguous subsequences, as illustrated by the sequence of deployment times for the instrument 1.

The sets of subsequences for the sound sources are then supplied to a means 16 for combining for each sound source to obtain a combined subsequence for the first sound source and a combined subsequence for the second sound source as a characteristic for the sound signal. The summary preferably takes place in the form of a pattern histogram. The subsequences for the first instrument are superimposed aligned with each other such that the first interval of each subsequence is effectively "above" the first interval of each other subsequence. Then, as shown with reference to Fig. 5, the entries in each slot of a combined suborder and in each histogram bin of the pattern histogram are counted. In the example shown in FIG. 5, the combined subsequence for the first sound source would thus be a first line 50 of the pattern histogram. For the second sound source, so z. As the instrument 2, the combined subsequence would be the second line 52 of the pattern histogram, etc. Overall, the pattern histogram in Fig. 5 thus represents the characteristic for the sound signal, which can then be used for various other purposes.

In the following, various exemplary embodiments for determining the common period length in step 12 will be discussed. The finding of the pattern length can be realized in various ways, namely, for example, from an a-priori criterion, which immediately gives an estimate of the periodicity / pattern length due to the existing Provides note information, or alternatively z. By a preferably iterative search algorithm which accepts a number of hypotheses for the pattern length and checks their plausibility based on the resulting results. For example, this may also be done again by evaluating a pattern histogram, as is also preferably implemented by the merge means 16, or using other self-similarity measures.

As has been stated, the pattern histogram, as shown in Fig. 5, may be generated by the merge means 16. Alternatively, the pattern histogram may also consider the intensities of the individual notes in order to weight the notes according to their relevance. Alternatively, as shown in Fig. 5, the histogram may only contain information as to whether there is a sound in a subsequence or in a bin or time slot of a subsequence, or not. In this case, a weighting of the individual notes would not be included in the histogram in terms of their relevance.

In a preferred embodiment of the present invention, the characteristic shown in Fig. 5, which is here preferably a pattern histogram, is further processed. In this case, note selection can be made on the basis of a criterion, for example by comparing the frequency or the combined intensity values with a threshold value. This threshold may also depend on the type of instrument or the flatness of the histogram, among other things. The Drum Pattern entries can be Boolean sizes, with a "1" for the fact would stand for a note, while a "0" would stand for the fact that no note occurs. Alternatively, an entry in the histogram can also be a measure of how high the intensity (loudness) or relevance of the note occurring in this time slot over the music signal is considered. Looking at Figure 6, it will be seen that the threshold was chosen to mark all time slots or bins in the pattern histogram for each instrument with an "x" where the number of entries is greater than or equal to 3 is. In contrast, all bins are deleted in which the number of entries is less than 3, namely, for example, 2 or 1 amounts.

According to the invention, therefore, a musical "result" or score is generated from percussive instruments that are not or not significantly characterized by a pitch. A musical event is defined as the occurrence of a sound of a musical instrument. Preferably, only percussive instruments are considered without a substantial pitch. Events are detected in the audio signal and classified into instrument classes, with the timing of events being quantized on a quantization grid, also referred to as a tatum grid. Furthermore, the musical measure or the length of a clock is calculated in milliseconds or else a number of quantization intervals, and furthermore preferably also clocks are identified. The identification of rhythmic structures based on the frequency of the occurrence of musical events at certain positions in the drum pattern allows a robust identification of the tempo and provides valuable information for the positioning the timing lines, when also musical background knowledge is used.

It should be noted that the musical score or characteristic preferably comprises the rhythmic information such as start time and duration. Although the estimation of this metric information, namely a time signature, is not necessarily necessary for the automatic synthesis of the transcribed music, it is nevertheless needed for the generation of a valid musical score and for reproduction by human reproducers. Therefore, an automatic transcription process can be divided into two tasks, namely the detection and classification of the musical events, ie notes, and the generation of a musical score from the detected notes, ie the drum pattern, as has already been explained above. For this purpose, the metric structure of the music is preferably estimated, wherein a quantization of the temporal positions of the detected notes as well as a detection of upbeats and a determination of the position of the clock lines can be made. In particular, the extraction of the musical score for percussive instruments without significant pitch information from polyphonic musical audio signals is described. The detection and classification of the events is preferably performed by the method of independent subspace analysis.

1. 1. J. Karhunen, "Neural approaches to independent component analysis and source separation", Proceedings of the European Symposium on Artificial Neural Networks, S. 249-266, Bruges, 1996.
2. 2. M.A. Casey and A. Westner, "Separation of Mixed Audio Sources by Independent Subspace Analysis", Proceedings of the International Computer Music Conference, Berlin, 2000.
3. 3. J.-F. Cardoso, "Multidimensional independent component analysis", Proceedings of ICASSP'98, Seattle, 1998.
4. 4. A. Hyvärinen, P.O. Hoyer and M. Inki, "Topographic Independent analysis", Neural Computation, 13(7), S. 1525-1558, 2001.
5. 5. S. Dubnov, "Extracting Sound Objects by Independent Subspace Analysis" Proceedings of AES 22^nd International Conference on Virtual, Synthetic and Entertainment Audio, Helsinki, 2002.
6. 6. J.-F. Cardoso and A. Souloumiac, "Blind beamforming for non Gaussian signals" IEE Proceedings, Bd. 140, Nr. 6, S. 362-370, 1993.
   Ein Ereignis wird als Auftreten einer Note eines musikalischen Instruments definiert. Der Auftrittszeitpunkt einer Note ist also der Zeitpunkt, zu dem die Note in dem musikalischen Stück auftritt. Das Audiosignal wird in Teile segmentiert, wobei ein Segment des Audiosignals ähnliche rhythmische Eigenschaften hat. Dies wird unter Verwendung eines Abstandsmaßes zwischen kurzen Rahmen des Audiosignals durchgeführt, das durch einen Vektor von Audiomerkmalen auf niedriger Ebene dargestellt wird. Das Tatum-Grid und höhere metrische Ebenen werden aus den segmentierten Teilen separat ermittelt. Es wird angenommen, dass sich die metrische Struktur innerhalb eines segmentierten Teils des Audiosignals nicht verändert. Die erfassten Ereignisse sind vorzugsweise mit dem abgeschätzten Tatum-Grid ausgerichtet. Dieser Prozess entspricht in etwa der bekannten Quantisierungsfunktion in üblichen MIDI-Sequenzer-Softwareprogrammen für die Musikproduktion. Die Taktlänge wird aus der quantisierten Ereignisliste abgeschätzt, und wiederkehrende rhythmische Strukturen werden identifiziert. Die Kenntnis über die rhythmischen Strukturen wird für die Korrektur des geschätzten Tempos verwendet und für die Identifikation der Position der Taktlinien unter Verwendung musikalischen Hintergrundwissens.

An extension of the ICA is the Independent Subspace Analysis (ISA). Here, the components are subdivided into independent subspaces or subspaces whose components need not be statistically independent. Through a transformation of the music signal becomes a multi-dimensional representation of the mixed signal and met the latest assumption for the ICA. Various methods for calculating the independent components have been developed in recent years. Relevant references, some of which also deal with the analysis of audio signals, are the following:

1. J. Karhunen, "Neural approaches to independent component analysis and source separation", Proceedings of the European Symposium on Artificial Neural Networks, pp. 249-266, Bruges, 1996.
2. Case MA and A. Westner, "Separation of Mixed Audio Sources by Independent Subspace Analysis," Proceedings of the International Computer Music Conference, Berlin, 2000.
3. 3rd J.-F. Cardoso, "Multidimensional Independent Component Analysis", Proceedings of ICASSP'98, Seattle, 1998.
4. 4. A. Hyvärinen, PO Hoyer and M. Inki, "Topographic Independent Analysis", Neural Computation, 13 (7), pp. 1525-1558, 2001.
5. 5. S. Dubnov, "Extracting Sound Objects by Independent Subspace Analysis" Proceedings of AES 22 ^nd International Conference on Virtual, Synthetic and Entertainment Audio, Helsinki., 2002
6. 6. J.-F. Cardoso and A. Souloumiac, "Blind beamforming for non Gaussian signals" IEE Proceedings, Vol. 140, No. 6, pp. 362-370, 1993.
   An event is defined as the occurrence of a note of a musical instrument. The appearance time of a note is thus the time at which the note occurs in the musical piece. The audio signal is segmented into parts, with a segment of the audio signal having similar rhythmic properties. This is done using a distance measure between short frames of the audio signal represented by a vector of low-level audio features. The tatum grid and higher metric levels are determined separately from the segmented parts. It is assumed that the metric structure does not change within a segmented portion of the audio signal. The detected events are preferably aligned with the estimated tatum grid. This process is similar to the well-known quantization function in common MIDI sequencer software programs for music production. The measure length is estimated from the quantized event list and recurring rhythmic structures are identified. The knowledge about the rhythmic structures is used for the correction of the estimated tempo and for the identification of the position of the timing lines using musical background knowledge.

Hereinafter, preferred embodiments of various inventive elements will be discussed. Preferably, means 10 for providing sequences of times of use for a plurality of sound sources performs quantization. The detected events are preferably quantized in the Tatum grid. The tatum grid is estimated using the note usage times of the recorded events along with note times that operate using conventional note-taking techniques. The Creating the Tatum Grid based on the detected percussive events works reliably and robustly. It should be noted that the distance between two halftone dots in a piece of music usually represents the fastest played note. Thus, if there are at most sixteenth notes in a piece of music and no ones faster than the sixteenth notes, the distance between two dots of the tatum grid is equal to the length of a sixteenth note of the tone signal.

In the general case, the distance between two halftone dots corresponds to the largest note value needed to represent all occurring note values or time periods by forming integer multiples of this note value. The grid spacing is thus the largest common divisor of all occurring note durations / period lengths etc.

In the following, two alternative approaches for the determination of the Tatum grid are presented. First, as a first approach, the tatum grid is represented using a 2-way mismatch procedure (TWM). A series of trial values for the tatum period, that is, the spacing of two halftone dots, is derived from a histogram for an inter-onset interval (IOI). The calculation of the IOI is not limited to consecutive onsets, but to virtually all pairs of onsets in a timeframe. Tatum candidates are calculated as integer fractions of the most common IOI. The candidate is selected that best predicts the harmonic structure of the IOI according to the 2-way mismatch error function. The estimated tatum period is subsequently calculated by calculating the error function between the comb grid and the tatum period is derived and calculates the onset times of the signal. Thus, the histogram of the IOI is generated and smoothed by means of an FIR low-pass filter. Thus, Tatum candidates are calculated by dividing the IOI according to the peaks in the IOI histogram by a set of values between e.g. B. 1 and 4 received. A raw estimate for the Tatum period is derived from the IOI histogram after applying the TWM. Thereafter, the phase of the tatum grid and an exact estimate of the tatum period are calculated by means of the TWM between the billets and several tatum grids with periods close to the previously estimated tatum period.

The second method refines and presents the tatum grid by computing the best match between the note insert vector and the tatum grid, using a correlation coefficient R _xy between the note insert vector x and the tatum y. $$R_{xy}=\frac{\underset{i=1}{\overset{n}{\Sigma }}\left(x_i-\tilde{x}\right)\cdot (y_i-\tilde{y})}{\sqrt{\underset{i=1}{\overset{n}{\Sigma }}{\left(x_i-\tilde{x}\right)}^2\underset{i=1}{\overset{n}{\Sigma }}{\left(y_i-\tilde{y}\right)}^2}}$$

To follow small tempo variations, the tatum grid for adjacent frames with z. B. a length of 2.5 seconds estimated. The transitions between the tatum grids of adjacent frames are smoothed by low-pass filtering the IOI vector of the tatum grid points, and the tatum grid is restored from the smoothed IOI vector. Then each event is assigned to its closest grid position. This is a kind of quantization.

The score can then be written as a matrix T _ik , i = 1,... N and j = 1,..., M, where n denotes the number of instruments acquired, and m is equal to the number of tatum grid Is equal to the number of columns in the matrix. The intensity of the detected events can either be removed or used, resulting in a Boolean matrix or resulting in a matrix of intensity values.

Hereinafter, specific embodiments of the device 12 for determining a common period length will be discussed. The quantized representation of the percussive events provides valuable information for the estimation of the musical measure or a periodicity that underlies the playing of the sound sources. The periodicity at the clock level, for example, is determined in two stages. First, a periodicity is calculated to then estimate the cycle length.

$$AMDF\left(\mathit{\tau }\right)={\displaystyle \mathit{\sum }_{j=1}^{\mathit{\tau }}}{\left(x_j-x_{j+\mathit{\tau }}\right)}^2$$

Preferably, the periodic functions used are the autocorrelation function (ACF) or the mean magnitude difference function (AMDF), as shown in the following equations. $$ACF\left(\mathit{\tau }\right)={\displaystyle \underset{i=1}{\overset{\mathit{\tau }}{\mathit{\Sigma }}}}{x}_i{x}_{i+\mathit{\tau }}$$

$$AMDF\left(\mathit{\tau }\right)={\displaystyle \underset{j=1}{\overset{\mathit{\tau }}{\mathit{\Sigma }}}}{\left(x_j-x_{j+\mathit{<figure-callout\; id="2"\; label="\tau "\; filenames="imgf0002.png,imgf0003.png"\; state="\{\{state\}\}">\tau </figure-callout>}}\right)}^2$$

The AMDF is also used to estimate the fundamental frequency for music and speech signals and to estimate the musical measure.

In the general case, a periodicity function measures the similarity or dissimilarity between the signal and its temporally different version. Different similarity measures are known. For example, there is the Hamming distance (HD), which calculates a dissimilarity between two Boolean vectors B ₁ and B ₂ according to the following equation. $$HD=\mathit{<figure-callout\; id="1"\; label="sum\; b"\; filenames="imgf0002.png,imgf0003.png"\; state="\{\{state\}\}">sum\; </figure-callout>}\left(b_{}\right)\left({}_1\underset{}{\vee }{\mathrm{<figure-callout\; id="2"\; label="b"\; filenames="imgf0002.png,imgf0003.png"\; state="\{\{state\}\}">b</figure-callout>}}_2\right)$$

A suitable extension for the comparison of the rhythmic structures results from the different weighting of similar hits and rest periods. The similarity B between two sections of a score T ₁ and T ₂ is then calculated by weighted summation of the Boolean operations, as shown below. $$B=a<figure-callout\; id="1"\; label="\cdot \; T"\; filenames="imgf0002.png,imgf0003.png"\; state="\{\{state\}\}">\cdot \; </figure-callout>T_{}{}_1\wedge {\mathrm{<figure-callout\; id="2"\; label="T"\; filenames="imgf0002.png,imgf0003.png"\; state="\{\{state\}\}">T</figure-callout>}}_2+b{\cdot }_{⌜}{\mathrm{<figure-callout\; id="1"\; label="T"\; filenames="imgf0002.png,imgf0003.png"\; state="\{\{state\}\}">T</figure-callout>}}_1\wedge {\cdot }_{⌜}{T}_2-\mathrm{<figure-callout\; id="1"\; label="c\; \cdot \; T"\; filenames="imgf0002.png,imgf0003.png"\; state="\{\{state\}\}">c\; </figure-callout>}\cdot T_{}{}_1\underset{}{\vee }{\mathrm{<figure-callout\; id="2"\; label="T"\; filenames="imgf0002.png,imgf0003.png"\; state="\{\{state\}\}">T</figure-callout>}}_2$$

In the above equation, the weights a, b, and c are initially set to a = 1, b = 0, 5, and c = 0. a weights the occurrence of common notes, b weights the occurrence of common breaks, and c weights the occurrence of a difference, ie, one note occurs in one score and no note occurs in the other score. The similarity measure M is obtained by summing the elements of B, as set forth below. $$M={\displaystyle \underset{i=1}{\overset{n}{\Sigma }}}{\displaystyle \underset{j=1}{\overset{m}{\Sigma }}}{B}_{ij}$$

This similarity measure is similar to the Hamming distance in that differences between matrix elements are similarly accounted for. Subsequently, a modified Hamming distance (MHD) is used as the distance measure. In addition, the influence of distinct instruments can be controlled by means of a weighting vector ν _i , i = 1,..., N, determined either by using musical foresight, e.g. By placing more importance on small drums (snare drums) or on deep instruments, or depending on the frequency and regularity of the appearance of the instruments: $$M_v={\displaystyle \underset{i=1}{\overset{n}{\Sigma }}}{v}_i\cdot {\displaystyle \underset{j=1}{\overset{m}{\Sigma }}}{B}_{ij}$$

In addition, the similarity measures for Boolean matrices can be extended by weighting B with the average of T ₁ and T ₂ to account for intensity values. Distances or dissimilarities are regarded as negative similarities. The periodicity function P = f (M, 1) is calculated by calculating the similarity measure M between the score T and a shifted version thereof, based on a displacement 1. The time signature is determined by comparing P with a number of metric models. The implemented metric models Q consist of a train of spikes with typical accent positions for different time signatures and micro times. A micro-time is the integer ratio between the duration of a musical beat, that is, the note value that determines the musical tempo (eg, quarter-note), and the duration of a tatum period.

The best match between P and Q is obtained when the correlation coefficient reaches its maximum. In the current state of the system, 13 metric models are implemented for seven different time signatures.

Recurring structures are recorded in order to get started. B. to capture, and to obtain a robust pace estimation. For the acquisition of drum patterns, a score T is obtained from the length of a stroke b by summation of the matrix elements T with a similar metric position according to the following equation: $$\mathit{T\; \text{'}}={\displaystyle \underset{k=1}{\overset{p}{\Sigma }}}{T}_{i.j}+\left(k-1\right)b$$

In the above equation, b denotes an estimated clock length and p denotes the number of clocks in T. Hereinafter, T 'is referred to as a score histogram or a pattern histogram. Drum patterns are obtained from the score histogram T 'by searching for score elements T' _{i, j} with large histogram values. Patterns longer than one clock are retrieved by repeating the procedure described above for integer values of the measured length. The pattern length with the most hits, relative to the pattern length itself, is selected to obtain a maximum representative pattern as a further or alternative characteristic for the sound signal.

Preferably, the identified rhythmic patterns are interpreted using a set of rules derived from musical knowledge. Preferably, equidistant occurrences occur identified by individual instruments and evaluated with reference to the instrument class. This leads to an identification of playing styles that often occur in popular music. An example is the very frequent use of the snare-drum or tambourines, or hand claps in the second and fourth beats in a four-quarter cycle. This concept, called backbeat, serves as an indicator of the position of the timing lines. If there is a backbeat pattern, a measure starts between two small drum attacks.

Another indication of the positioning of the timing lines is the occurrence of kick drum events, that is, events of a typically foot operated large drum.

It is assumed that the start of a musical measure is marked by the metric position where most kick drum notes occur.

A preferred application of the characteristic as obtained by means 16 for summarizing for each sound source, as shown and described in Fig. 1, as e.g. As shown in Fig. 5 or 6, is in the genre classification of popular music. From the drum patterns obtained, various high-level features can be derived to identify typical playing styles. A classification procedure evaluates these features in conjunction with information about the musical measure, ie the speed, in z. As beats per minute or beats per minute and using the percussive instruments used. The concept is based on the fact that all percussive instruments carry rhythm information and are often played repetitively. Drum patterns have genre-specific characteristics. Therefore, these drum patterns can be used to classify the music genre.

For this purpose, a classification of different playing styles (playing style) is performed, each of which is assigned to individual instruments. For example, one style of play is that events occur only on every quarter note. An associated instrument for this style of play is the kick-drum, so the big drum of the drums operated by the foot. This style of playing is abbreviated FS.

For example, an alternate style of playing is that events occur in every second and fourth quarter note of a four-fourths beat. This is mainly played by the small drum (snare drum) and tambourines, so the hand claps. This style of play is abbreviated as BS. Exemplary other play styles are that notes often appear on the first and third notes of a triplet. This is abbreviated as SP and is often observed in a hi-hat or cymbal.

So there are game styles specific to different musical instruments. For example, the first feature FS is a boolean value and true if kick-drum events occur only on each quarter note. Only for certain values are Boolean variables not calculated, but certain numbers are determined, such as the relation between the number of off-beat events and the number of on-beat events, such as those from a hi-hat, a shaker or a tambourine.

Typical combinations of drum instruments are classified into one of the various drum set types, such as rock, jazz, latin, disco, and techno, to provide another feature for genre classification. The classification of the drum set is not derived using the instrument sounds, but by generally examining the occurrence of drum instruments in various pieces belonging to each genre. For example, the drum set type Rock is characterized by a kick drum, a snare drum, a hi-hat and a pelvis. In contrast, comes in the type "Latin" a bongo, a conga, claves and shakers.

Another set of features is derived from the rhythmic features of the drum score or drum pattern. These features include musical tempo, time signature, micro-time, etc. In addition, a measure of the variation in the occurrence of kick drum notes is obtained by counting the number of different IOI that occur in the drum pattern.

The classification of the musical genre using the drum pattern is performed using a rule-based decision network. Potential genre candidates will be rewarded if they fulfill a hypothesis currently under investigation and will be "punished" if they do not fulfill aspects of a hypothesis that is currently under investigation. This process results in the selection of favorable feature combinations for each genre. The rules for a rational decision become more representative of observations Pieces and derived from musical knowledge in itself. Values for reward or punishment are set empirically considering the robustness of the extraction concept. The resulting decision for a particular musical genre is made for the genre candidate who has the maximum number of rewards. For example, the disco genre is recognized when a drum set type is disco, when the tempo is in the range of 115 to 132 bpm, when a time signature is 4/4 bit and the micro time is equal to 2. Further, another feature of the genre Disco is that a play style FS z. B. is present, and that z. B. yet another style of play is present, namely the events occur on each off-beat position. Similar criteria can be applied to other genres such as hip-hop, soul / funk, drum and bass, jazz / swing, rock / pop, heavy metal, Latin, waltz, polka / punk or techno.

Depending on the circumstances, the inventive method for characterizing a sound signal can be implemented in hardware or in software. The implementation may be on a digital storage medium, in particular a floppy disk or CD with electronically readable control signals, which may interact with a programmable computer system such that the method is executed. In general, the invention thus also consists in a computer program product with a program code stored on a machine-readable carrier for carrying out the method when the computer program product runs on a computer. In other words, the invention can thus be realized as a computer program with a program code for carrying out the method when the computer program runs on a computer.

Claims (21)

1. A device for characterizing a tone signal, comprising:

   means (10) for providing a sequence of entry times of tones for at least one tone source;

   means (12) for determining a common period length underlying the at least one tone source using the at least one sequence of entry times;

   means (14) for dividing the at least one sequence of entry times into respective sub-sequences, wherein a length of a sub-sequence is equal to the common period length or derived from the common period length; and

   means (16) for combining the sub-sequences for the at least one tone source into one combined sub-sequence, wherein the combined sub-sequence is a characteristic for the tone signal.
2. The device according to claim 1,
   wherein means (10) for providing is implemented in order to provide at least two sequences of entry times for at least two tone sources,
   wherein means (12) for determining is implemented in order to determine the common period length for the at least two tone sources,
   wherein means (14) for dividing is implemented in order to divide the at least two sequences of entry times according to the common period length, and
   wherein means (16) for combining is implemented in order to combine the sub-sequences for the second tone source into a second combined sub-sequence, wherein the first combined sub-sequence and the second combined sub-sequence represent the characteristic for the tone signal.
3. The device according to claim 1, wherein means for providing (10) is implemented in order to provide for each of the at least two tone sources one sequence of quantized entry times, wherein the entry times are quantized with regard to a quantization raster, wherein a raster point distance between two raster points is equal to a shortest distance between two tones in the tone signal or equal to the greatest common divisor of the duration of tones in the musical signal.
4. The device according to claims 1, 2 or 3, wherein means (10) for providing is implemented in order to provide the entry times of percussive instruments, but not the entry points of harmonic instruments.
5. The device according to one of the preceding claims, wherein means for determining (12) is implemented
   to determine for each of a plurality of hypothetical common period lengths a probability measure, and
   to select the hypothetical common period length from the plurality of hypothetical common period lengths as a common period length whose probability measure indicates that the hypothetical common period length is the common period length for the at least two tone sources.
6. The device according to claim 5, wherein means (12) for determining is implemented in order to determine the probability measure on the basis of a first probability measure for the first tone source and on the basis of a second probability measure for the second tone source.
7. The device according to claims 5 or 6, wherein means (12) for determining is implemented in order to calculate the probability measures by a comparison of the sequence of entry points to a shifted sequence of entry points.
8. The device according to one of the preceding claims, wherein means (14) for dividing is implemented to generate a list for each sub-sequence, wherein the list comprises an associated piece of information for each raster point and for each tone source, wherein the information relates to whether an entry point exists at a raster point or not.
9. The device according to one of the preceding claims, wherein means (10) for providing is implemented in order to generate a list for each tone source, wherein the list for each raster point of a raster comprises an associated piece of information whether there is an entry time of a tone at the raster point.
10. The device according to one of the preceding claims, wherein means (16) for combining is implemented in order to generate a histogram as a combined sub-sequence.
11. The device according to claim 10, wherein means (16) for combining is implemented to generate the histogram such that each raster point of a tone raster of the combined sub-sequence represents a histogram bin.
12. The device according to claims 10 or 11, wherein means (16) for combining is implemented to increment a count value for an associated bin in the histogram in each sub-sequence for a tone source when finding an input or by increasing the same by adding a measure determined by the input, wherein the input is a measure for the intensity of a tone that has an entry for the entry time.
13. The device according to one of the preceding claims, wherein means (16) for combining is implemented to output, in the first combined sub-sequence and the second combined sub-sequence, only values of the sub-sequences as a characteristic which are above a threshold.
14. The device according to one of the preceding claims, wherein means (16) for combining is implemented in order to normalize the sub-sequences with regard to the common length or to normalize the first combined sub-sequence or the second combined sub-sequence with regard to the common length.
15. The device according to one of the preceding claims, wherein means (10) for providing is implemented in order to generate segments with a unique rhythmical structure from an audio signal, and
    wherein means (16) for combining is implemented in order to generate the characteristic for a segment having a unique rhythmical structure.
16. The device according to one of the preceding claims, further comprising:

    means for extracting a feature from the characteristic for the tone signal; and

    means for determining a musical genre to which the tone signal belongs, using the feature.
17. The device according to claim 16, wherein means for determining is implemented in order to use a rule-based decision network, a pattern recognition means or a classifier.
18. The device according to one of the preceding claims, further comprising means for extracting a tempo from the characteristic.
19. The device according to claim 18, wherein means for extracting is implemented to determine the tempo on the basis of the common period length.
20. A method for characterizing a tone signal, comprising the following steps:

    providing (10) a sequence of entry times of tones for at least one tone source;

    determining (12) a common period length underlying the at least one tone source using the at least one sequence of entry times;

    dividing (14) the at least one sequence of entry times into respective sub-sequences, wherein a length of a sub-sequence is equal to the common period length or is derived from the common period length; and

    combining (16) the sub-sequences for the at least one tone source into one combined sub-sequence, wherein the combined sub-sequence represents a characteristic for the tone signal.
21. A computer program having a program code for performing the method according to claim 20 when the computer program runs on a computer.

Images