EP1758096A1 - Method and Apparatus for Pattern Recognition in Acoustic Recordings - Google Patents

Abstract

A signal is divided into frequency ranges which are transformed for spectral distribution into at least one coefficient file. Harmonic decomposition of the coefficient file is then performed followed by sample allocation. When transforming each of the frequency ranges, first transformation is performed based on frequency division and second transformation is performed based on time division. Independent claims are also included for the following: (1) a computer program product; (2) a pattern recognition device; and (3) the coefficient data.

Description

The invention relates to a method and a device for pattern recognition in acoustic recordings according to the preamble of claim 1 or 13, as well as a computer program product and a data structure product.

In many applications there is a need to recognize patterns in recordings of acoustic signals and to convert them for use. Examples of this are seismic measurements, vibration analysis in mechanical engineering, the selection of audio signals in the hearing aid sector, speech analysis or the conversion of music into playable or changeable formats. The basic problem in all these areas is always the same, in the following example is the pattern recognition in recordings of pieces of music, without justifying a restriction to this purpose. The method according to the invention or the device according to the invention can also be used to solve other problems, in particular from the areas explicitly described above.

For the processing of acoustic recordings or audio signals, these are usually digitized today. For example, a recording by suitable sensors, wherein the recorded signal is sampled and stored digitized. A widespread approach is the conversion and storage in WAVE format. In order to allow for the human ear lossless conversion and storage, usually a sampling 44.1 kHz and 16-bit resolution, so that maximum for the human ear perceptible frequencies the Nyquist theorem is satisfied.

Although all acoustically relevant components are thus recorded in this format, it is possible for the human ear to reproduce without recognizable loss. However, this format requires a large memory space, e.g. is disadvantageous in a transmission on the Internet, as long transmission times are the result. In addition, there is no storage of resolved patterns, i. a separation of e.g. different musical instruments are not, so that, for example, no simple change of the recording is possible, e.g. by swiping an instrument.

Another data format that embodies virtually the opposite information content is the MIDI format, where MIDI stands for Musical Instrument Digital Interface. This format, developed for the exchange of data between synthesizers, transmits control signals instead of audio data, which can be reproduced by a synthesizer or displayed graphically or visually. The widely used GM standard encodes and later plays 128 sound colors. Due to the comparatively small file size, this format is well suited for transmission on the Internet. However, this low bandwidth of timbres can not reproduce the natural sound. In addition, the MIDI format has a dependency on the playback of the hardware.

In the prior art, various approaches are pursued that allow pattern recognition in audio signals, often converting from wave to MIDI files.

For example disclosed US 6,140,568 a system and method for automatically detecting and identifying a plurality of frequencies simultaneously contained in an audio signal, such as the duration, amplitude and phase of those frequencies. From these frequencies harmonic components are filtered out to determine the fundamental frequencies. The system includes a computer readable medium having executable code for decomposing the signal into its sinusoidal components by computation and comparison between the input signal and sinusoidal waves having various combinations of phase and amplitude. The system also uses various optimization and error correction routines.

In Scripture US 6,355,869 B1 For example, a method and system for generating notes from a recording of music and the generation of an editable music format are described. The method is based on saving the music recording as a wave file from which a pseudo-wave file is generated for each relevant section in the recording. For each pseudo-wave file, a sequence file is generated, from which in turn a list of events is generated. This list is converted into a MIDI file or other note-readable file and imported to print the notes in a note program.

While pattern recognition for various types of patterns and identification of many musical instruments can be accomplished with the prior art approaches, some pattern types still present problems. Thus, with previous methods just the drum parts in audio signals can be resolved only poorly and displayed in notes. The problem is with the drums in that it provides a wide range of spectral contributions, which can not be clearly separated and analyzed with the previous methods.

Furthermore, on the data-storage side, the widespread MIDI format only allows for storage or playback, which brings heavy losses with regard to the original sound quality.

The object of the present invention is therefore to provide an improved method or an improved device, which also enables the resolution of components with a wide range of spectral contributions.

Another object of the invention is also to enable identification of percussion shares in recordings of music.

Another object of the invention is to enable improved interactive variability of acoustic recordings.

A further object of the invention is the provision of a data structure product which, with the storage of control signals, allows reproduction as true to the original as possible, so that, for example, the advantages of wave and MIDI format are combined without having to accept their disadvantages.

These objects are achieved according to the invention by the features of claims 1 and 13 or by the characterizing features the dependent claims solved or the solutions trained.

The inventive method or the device according to the invention for pattern recognition in acoustic recordings analyze acoustic signals, as are detected, for example, by microphones. These signals may represent musical pieces, speech, machine vibrations, seismic vibrations or other forms of mechanical vibration.

The signal is preferably digitized after or during recording to allow signal processing on computers, where data storage is e.g. in wave format. Alternatively or additionally, an implementation of the method is also possible in analog technology, e.g. through an appropriate circuit.

The detected and stored signal is subsequently divided into individual frequency ranges, e.g. Octaves, decomposed, for which known methods can be used. An example of this is the Pyramidendekomposition, in which the input signal is decomposed into different subband signals with different frequency ranges. Typically, the first subband includes only the highest frequencies. The subsequent subbands then contain the respective next lower signal components.

The frequency ranges are subsequently spectrally decomposed, from which follows a set of coefficients. According to the invention, this spectral decomposition takes place in two independent transformation processes.

Transformation algorithms suitable for this purpose are, for example, the Fourier transformation, fast Fourier transformation, wavelet transformation, sine transformation or cosine transformation, the discrete variants in particular being suitable.

One of the two independent transformation processes is optimized in terms of temporal resolution. For this purpose, the temporal window is chosen comparatively short, so that the time course is well resolved. However, the time limitation reduces the frequency resolution, so that the other transformation process analyzes the same frequency range with a comparatively large temporal window, so that a higher resolution of the frequencies takes place for this purpose. Both transforms each provide a set of coefficients for the contributing frequency components. The resulting TF output image (TF for frequency-time image) is now in turn subdivided into subbands over time and / or time and frequency, which in turn corresponds to a transformation with longer time constants. Various frequency-time (TF) images are used to detect signals or signal characteristics and to reconstruct original signals (input signals).

$$A_c\left(t,,f\right)=\int I\left(t\right)\cos \left(\mathit{\omega t}\right)dt$$

wobei

A_s (t, f)

den Sinusanteil des Ausgangssignals,

A_c (t,f)

den Cosinusanteil des Ausgangssignals,

ω

die Kreisfrequenz der zu untersuchenden Frequenzkomponente und

t

die Zeit bezeichnen.

These transformations are thus optimized for different tasks, such as the subdivision into percussive and harmonic signal components. As a possible transformation, the Fourier transformation is described purely by way of example: $$A_s\left(t,,f\right)=\int I\left(t\right)\sin \left(\mathit{.omega.t}\right)dt$$

$$A_c\left(t,,f\right)=\int I\left(t\right)\cos \left(\mathit{.omega.t}\right)dt$$

in which

A _s ( t, f )

the sine component of the output signal,

A _c ( t, f )

the cosine component of the output signal,

ω

the angular frequency of the frequency component to be examined and

t

to designate the time.

wobei ⊗ eine allgemeine Verknüpfung bezeichnet, die im einfachsten Falle einer Addition entspricht. Werden auch Beiträge nächsthöherer bzw. darüberliegender Schichten berücksichtigt so ergibt sich $$\mathit{TF}\left(n,,t,,f\right)=A_s\left(t,,f\right)\otimes A_c\left(t,,f\right)\otimes \mathit{TF}\left(n-1,t,f\right)$$

wobei TF(n-1,t,f) den Beitrag der nächsthöheren Schicht n -1 bezeichnet. Auch können A_s(t, f) und A_c(t,f) im üblichen Fall die Amplituden und Phasenwerte der Fouriertransformation darstellen $$\mathit{Amp}\left(t,,f\right)=\sqrt{A_s{\left(t,,f\right)}^2+A_c{\left(t,,f\right)}^2}$$

bzw. $$\phi =\mathrm{atan}\left(\frac{A_s\left(t,,f\right)}{A_c\left(t,,f\right)}\right)$$

The signal stored after the transformation in the layers of the output quantity is a mixture of the transformation output signals and a pyramid decomposition of the next higher level of the pyramid $$\mathit{TF}\left(n,,t,,f\right)=A_s\left(t,,f\right)\otimes A_c\left(t,,f\right)$$

where ⊗ denotes a general link, which in the simplest case corresponds to an addition. If contributions of higher or higher layers are also taken into account, this results $$\mathit{TF}\left(n,,t,,f\right)=A_s\left(t,,f\right)\otimes A_c\left(t,,f\right)\otimes \mathit{TF}\left(n-1.t.f\right)$$

where TF (n- 1 , t, f) denotes the contribution of the next higher layer n- 1. Also, A _s (t, f) and A _c (t, f) may represent the amplitudes and phase values of the Fourier transform in the usual case $$\mathit{Amp}\left(t,,f\right)=\sqrt{A_s{\left(t,,\mathrm{<figure-callout\; id="2"\; label="f"\; filenames="imgf0003.png,imgf0004.png"\; state="\{\{state\}\}">f</figure-callout>}\right)}^2+A_c{\left(t,,\mathrm{<figure-callout\; id="2"\; label="f"\; filenames="imgf0003.png,imgf0004.png"\; state="\{\{state\}\}">f</figure-callout>}\right)}^2}$$

respectively. $$\phi =\mathrm{atan}\left(\frac{A_s\left(t,,f\right)}{A_c\left(t,,f\right)}\right)$$

The individual layers of the pyramid can be generated from a combination of high and low pass filters and subsampling. This TF pyramid can also be generated multiple times to accommodate different purposes such as signal analysis and signal reconstruction.

Information from one or more of the layers is combined in a filter, for example a two-dimensional filter with an average core, into a one-dimensional vector from which note events can then be derived, for example, with detection of local maxima.

In addition to the arrangement with two transformations, which are optimized for harmonic and percussive signals, for example, it is also possible to use a scheme in which one or more transformations fill a multilayer output region. This means that for each octave of the subband input signal, a transformation is performed for one (1) to several (12 for one octave with semitones, 14 or 16 to filter in the frequency direction with filters), which is a frequency / time Image generated. This image can be created from the signal of one or more transformations. For example, parts of the frequency-optimized transformation can be mixed with parts of the percussive transformation in such a way that a clear demarcation between harmonic and percussive signals becomes possible.

After transforming the frequency ranges, the spectral decomposition is completed by generating at least one coefficient file. In this coefficient file, coefficients are taken from the coefficient sets of the two transformations, wherein the coefficients can be selected from one of the two sets or else can be generated as a mixture of coefficients. Thus, the two sets of coefficients of the different transforms are transformed into a coefficient file in an overall transformation under selection or blending, this file then containing portions of both transformations.

The generation of the coefficient file uses heuristics, given information, e.g. from previous analyzes, or statistical evaluations of the current signal. Basically, all frequency bands are routed through both transformation processes. However, also, e.g. due to given information, only one of the two transformation processes are used for individual frequency bands, so that only the result of this step is used further.

The selection or mixture of coefficients for generation can be done by various methods.

In one approach, a first Fourier transform with a long time window and a second Fourier transform with a short time window and subsequent low-pass filter are performed. For the results of both transformations, the real part is calculated and their ratio educated. Based on this ratio, it is decided from which transformation the coefficient is selected.

Another approach relies on analyzing the slope in a plot of phase vs. frequency, i. the frequency-dependent slope of the phase signal. By setting thresholds or calculating a weighting parameter, a determination is made as to which coefficient is used or whether and how a mixture of coefficients takes place.

The use of given information is done by comparing the sets of coefficients obtained by the transformations with a set of stored coefficients. This comparison serves as a selection criterion for the coefficients or their mixture.

The complete transformation process eventually creates a file containing the selected or mixed coefficients. In addition, statistical information regarding the signal may still be stored in this file.

On the basis of this coefficient file, the harmonic decomposition takes place, which finally leads to an assignment of spectral components to patterns, such as special musical instruments. The detected patterns or events can be displayed graphically after conversion, eg as notes, or reproduced by synthesizers. Patterns or events are to be understood here as the characteristic components in an acoustic signal whose identification represents the aim of the analysis. These can be, for example, individual musical instruments, words or seismic parameters.

Basis of the decomposition according to the invention not only the coefficients themselves, but also their aggregates, e.g. the temporal integral of an amplitude for a given frequency, or statistical information.

For the decomposition, in the simplest case, a comparison can be made with a database in which examples of patterns are stored. Such databases are available, for example, for musical instruments.

Another possibility is the construction of a model for the patterns to be identified, this model being e.g. can be built from the current signal with statistical methods. The model is iteratively compared to the signal and gradually optimized. If the remaining residual falls below a predefined threshold value, the method is aborted and the pattern recognition is considered to be sufficiently good.

For feature or note recognition, different approaches can be used alternatively or cumulatively.

Thus, for example, characteristic features of the individual musical instruments are determined by suitable one- or two-dimensional filters in the individual layers of the TF pyramid. These features can then be assigned directly to the individual musical instruments and their representation in notation format (eg Midi or internal format). Alternatively, the features are supplied to a neural network as input variables.

In this neural network, the regions of the TF pyramids determined by the features are more closely examined, for example, by pixel-to-pixel comparison in a delimited environment of the feature. The determined results of these comparisons can, coupled with the feature recognition, bring about an improvement of the feature recognition. For example, feature centers, feature thresholds and frequency-time extent of feature recognition are adjusted. With these methods, characteristics for percussive and / or harmonic sounds can be determined. In particular, this will produce individual sounds of an instrument, e.g. Guitar, bass, drums and cymbals of a drum kit, but also piano and guitar chords recognized. In basically the same way, seismic events or linguistic features, e.g. be blanked background noise in an acoustic communication link, to be analyzed.

Because features often repeat in the input signals, features and patterns that have been identified can be used to search the entire information content (TF) for such iterations.

The determined patterns are classified according to predetermined criteria or after the analysis by assignment, wherein this assignment can be performed by the computer program fully automatically, semi-automatically or interactively by the program user. To improve the classification of the patterns, the result set (TF) can be examined for comparable patterns. This method is time-saving, since the transformation can often be a comparatively long-lasting process.

All methods of the prior art for music recognition have so far lead to a static, not interactively correctable score image, which is faulty or incorrect in the sense of the desired representation. For improvement, according to the invention, methods are available which make the generated emergency presentation modifiable by interactive specification of parameters between the computer program and the user. For example, harmonics identified by temporal information (e.g., guitar and piano chords) can be improved or changed.

Thus, for example, the clock division can be added or changed manually as a temporal classification. Notation requires a classification in a temporal sense in such a way that the note values determined can be assigned note lengths. A function in the user program makes it possible to mark the beginning of the cycle and an automatic function of the program then determines the missing cycles between these markings. This process can be repeated until the clock division is satisfactory. However, it is also possible to use functions which automatically recognize the clock division.

An improvement in harmony recognition by temporal classification is possible due to a division of the information content into measures that can be used to improve the harmony recognition by making use of the fact that in real music played the harmonies often change at the clock change.

Insufficient setting of automatic or manual thresholds in prior art note recognition means that the time-consuming process of note recognition must be restarted. According to the invention, threshold values for the note recognition can also be subsequently changed, so that the recognized notes can be made available to the user in an optimal representation. For this purpose, criteria, for example features, are provided with a threshold value so that signals below the threshold value are not displayed as musical notes and also do not sound.

In this case, the user can interact with the system feedback also affect the result. For example, this may be derived from its - e.g. obtained by listening to the recorded piece of music - knowledge of the occupation of a music group preselection of the existing musical instruments manually specify. This predetermined information then facilitates and accelerates the harmonic decomposition or the pattern recognition. The basis of this modifiability thus represents the method according to the invention, which includes modeling with variable coefficients which is or can not be achieved in the prior art.

In order to ensure optimal use and interactive variability, an adapted presentation of the results with different elements takes place. For the selection and modification of events, an event image is generated, for example as an image with notation-like groups of lines, which correspond to pitches, arranged in the Y-direction. In the X direction, the time is plotted or a currently proportional size. Events are going through Noteheads or patterns or images generally available through symbols of a font or bitmap or other graphic formats. The Y position in the picture is assigned to the properties of the event by the assignment table or a mathematical function, eg the note height D6 (Midi 74) as the second line from above).

Once the bars are fixed, the events can also be displayed in standard music notation.

A representation may also be in the form of lead-sheets as a one-to-one-page summaries of a piece of music. Leadsheets in the traditional sense are created by hand. With the method according to the invention, an automatic generation of leadsheets can now also be carried out. For this purpose, marks are set in the piece of music which describe definable areas of the piece of music, e.g. Introduction, 1st verse, 1st chorus, intermediate part, etc. The method then generates from the determined notes, bars, and chords a summarized representation of all or part of the piece of music. This presentation can then be added to the lyrics, which then also in the score is also insertable.

A pitch threshold control allows note values to be activated, displayed and sounded. It can be determined whether events are hidden or whether the pitch should be shifted by a certain amount, for example an octave, whereby the notes are then played one octave lower and recorded. As a result, the result can be improved to such an extent that, if notes by their harmonic Shares are detected, they can be transposed to the fundamental frequency.

With suitable selection instruments, e.g. a mouse, keyboard or other tool, individual or groups of notes may be selected and optionally subsequently, e.g. via Midi, to be played. According to the invention, it is possible to reconstruct the original sounds, which led to the emergence of the event, and to play them back via the music system of the computer. These reconstructions can now also be stored separately in music files.

For further separation into different musical instruments, note events can be selected with the methods mentioned and copied or moved to other soundtracks.

To improve the drum result as a repetitive sequence with accentuation, methods are available which can detect a correlation of repetitive patterns, wherein the correlation length can be determined automatically by the algorithms of the program or by the user or by setting the clocks. Through this correlation, also different parts of a piece of music can be identified. The drum patterns determined in this way are also listed on the lead sheets.

wobei der korrespondierende dynamische Fall gemäss $$S\left(t_0\right)={\displaystyle \sum _{t_1-t_0}^{t_2-t_0}\sum _{f=0}^{f_{\max }}{\left(P\left(t-t_0,f\right)\otimes R\left(t,,f\right)\right)}^2}$$

formuliert werden kann. Hierbei bezeichnen P ein Signalmuster und R ein Referenzmuster. Als Verknüpfungen ⊗ können beispielsweise Subtraktion oder Multiplikation verwendet werden. Das Referenzmuster kann ein Muster an einer anderen Stelle der TF-Matrix sein oder ein vorabgespeichertes Muster oder aber ein Muster, welches aus einer Kombination bestehender Muster, beispielsweise durch Mittelwertbildung, entstanden ist. Im dynamischen Fall werden beide Muster gegeneinander zeitlich verschoben, so dass eine zeitabhängige Übereinstimmung ableitbar ist. Bei kleinen Werten von S besteht eine grosse Ähnlichkeit der zu vergleichenden Muster. In einer aus Vergleichen aller Muster miteinander erstellten Matrix AS sind die Elemente AS(i,j) = S(i,j). With the previously mentioned method of recognizing percussion notes, it is possible to mark areas in TF layers from whose surroundings patterns can be derived. A part or all of these patterns are compared with each other, for example, the method of the sum of the squares of the differences superimposed pixels can be used as a criterion, which can be formulated as follows for the static case $$S={\displaystyle \underset{t_1}{\overset{{\mathrm{<figure-callout\; id="2"\; label="t"\; filenames="imgf0003.png,imgf0004.png"\; state="\{\{state\}\}">t</figure-callout>}}_2}{\Sigma }}\underset{\mathrm{f}=0}{\overset{f_{\mathrm{Max}}}{\Sigma }}{\left(P\left(t,,f\right)\otimes R\left(t,,\mathrm{<figure-callout\; id="2"\; label="f"\; filenames="imgf0003.png,imgf0004.png"\; state="\{\{state\}\}">f</figure-callout>}\right)\right)}^2}$$

the corresponding dynamic case according to $$\mathrm{<figure-callout\; id="0"\; label="S\; t"\; filenames="imgf0003.png"\; state="\{\{state\}\}">S\; </figure-callout>}\left(t_{}\right)\left({}_0\right)={\displaystyle \underset{t_1-t_0}{\overset{t_2-{\mathrm{<figure-callout\; id="0"\; label="t"\; filenames="imgf0003.png"\; state="\{\{state\}\}">t</figure-callout>}}_0}{\Sigma }}\underset{f=0}{\overset{f_{\mathrm{Max}}}{\Sigma }}{\left(P\left(t-{\mathrm{<figure-callout\; id="0"\; label="t"\; filenames="imgf0003.png"\; state="\{\{state\}\}">t</figure-callout>}}_0.f\right)\otimes R\left(t,,\mathrm{<figure-callout\; id="2"\; label="f"\; filenames="imgf0003.png,imgf0004.png"\; state="\{\{state\}\}">f</figure-callout>}\right)\right)}^2}$$

can be formulated. Here, P denotes a signal pattern and R denotes a reference pattern. As links ⊗, for example, subtraction or multiplication can be used. The reference pattern may be a pattern at another location of the TF matrix, or a prestored pattern, or a pattern formed from a combination of existing patterns, such as averaging. In the dynamic case, both patterns are shifted in time relative to each other, so that a time-dependent match can be derived. For small values of S there is a great similarity of the patterns to be compared. In a matrix AS created from comparing all the patterns with one another, the elements AS (i, j) = S (i, j).

For classification, groups are formed and assigned to a graph. Here, there is a connection of each pattern to the pattern that is most similar. Due to preprogrammed features, the patterns are then classified and assigned note values.

The recognition of chords in pieces of music is done in the same way as described above for pattern recognition drum notes.

The detection of harmonic sounds, such as Guitar, bass, piano, melody or vocals, uses thresholds. A threshold value determines whether a frequency of a TF layer is active or not. In the simplest case, each active frequency is converted to a note, with position, note height, and length, i. the entry above the threshold to the exit at the transition from active to below the threshold, be determined. This method is used, for example, for the recognition of instruments which produce only a few overtones, e.g. a sine organ.

mit F ₀ als Grundfrequenz und H ₁ ,H ₂ ,H ₃ ,...H_n als Höherharmonischen, d.h. H ₁ = 2 · F ₀ , H ₂ = 3 · F ₀ etc., gebildet, wobei als Verknüpfung ⊗ beispielsweise eine Multiplikation gewählt werden kann. Danach werden die Bereiche aktiviert, die einen zuvor ermittelten oder festgelegten Schwellwert überschreiten, als Ereignisse ermittelt und in Noten umgewandelt.For harmonic signals with high harmonic components, ie the frequencies are at frequencies that will be a multiple of the fundamental frequency for one or more layers of the TF pyramid, the products $$F_{}$$$${}_0\to {\mathrm{<figure-callout\; id="0"\; label="F"\; filenames="imgf0003.png"\; state="\{\{state\}\}">F</figure-callout>}}_0<figure-callout\; id="1"\; label="\otimes \; H"\; filenames="imgf0003.png,imgf0005.png"\; state="\{\{state\}\}">\otimes \; </figure-callout>\left(H_{}\right)\left({}_1+{\mathrm{<figure-callout\; id="2"\; label="H"\; filenames="imgf0003.png,imgf0004.png"\; state="\{\{state\}\}">H</figure-callout>}}_2+{\mathrm{<figure-callout\; id="3"\; label="H"\; filenames="imgf0003.png,imgf0004.png"\; state="\{\{state\}\}">H</figure-callout>}}_3+...H_n\right)$$

with F ₀ as the fundamental frequency, and H _1, H _2, H _3, ... H _n as a higher harmonic, that is, H ₁ = 2 * F _0, H ₂ = 3 · F ₀ etc. formed, wherein ⊗ as a link, for example a Multiplication can be chosen. Thereafter, those areas are activated which exceed a previously determined or fixed threshold value, determined as events and converted into grades.

• o Position im Lied
• o Länge des Ereignisses
• o Text
• o Frequenz
• o Notenhöhe
• o Detektionsvolumen
• o Musikinstrument
• o Amplitude
• o Koeffizienten.

In addition, note objects can be collected. Each grade typically has the following characteristics:

• o position in the song
• o Length of the event
• o text
• o frequency
• o Grade
• o detection volume
• o musical instrument
• o amplitude
• o coefficients.

For this purpose, collections (collections) of notes can be created, which are typically divided into instruments in soundtracks. These collections can be stored in files on a computer system. Such files may also be transmitted via the Internet, by wire or by electromagnetic transmission. As examples of transmission protocols Http, Tcp, Https, SOAP, etc. are listed, but other formats are possible.

The detected events or notes are displayed in one or more ways. For example, one embodiment illustrates the events as a combination of symbols (note heads), where the vertical axis corresponds to a common note image and the horizontal axis of time. Since with a standard five-line note each line can stand for three notes (eg g, ges and gis), these states can be represented by different symbols, eg a regular notehead for g, a triangle with a top for ges and a triangle with top up for gis. In addition, the event length can be indicated by a rectangle. Another possible Presentation of the results is the usual notation.

In contrast to the method according to the invention, which allows an adaptation of the results, methods of the prior art have the disadvantage that threshold values have to be set before the time-consuming analysis. Inadequate settings require the entire analysis process to be repeated, which is cumbersome, unfriendly, prone to error, and time consuming. The method according to the invention has the advantage that threshold values for the note recognition can also be set after the analysis. This allows the results to be adjusted in real time to the wishes of the user. This method combines the possibilities of note recognition with the notation representation in a way that allows the results to be individually adapted by interaction of the program user with the analysis software.

With the special application method of semi-automatic setting of bar lines, positions in the event image can be marked, which musically mark the first beat of a bar. In this approach, at least one clock is set by two markers and thus given a temporal information. The program then automatically calculates the missing bars for the whole song, eg with the help of extrapolation. Due to the inaccuracy of the set bar and tempo variations in the song deviations from the ideal result often arise, ie the assumption that all bars are set correctly. Additional first beats of a bar can be set by the user, in which case the new bar layout is recalculated.

The threshold controller shown above can also be used as a pitch filter, i. as an instrument to set cutoff frequencies, in which case note events with pitches above (or below or centered around) a threshold are not displayed or just displayed and played. Optionally, notes that are outside the threshold can be returned to the range of displayed events by pitch transposition (octave shift). As an example, consider a low pass that does not display scores above 60 (middle C (C5) according to Midi Standard, 61 = cis5). In one case, a note of pitch 70 is no longer displayed and / or played, otherwise the note is transposed down one octave (70-12 semitone steps = 58), thus the note with pitch 58 is shown and played. This method is used to reduce erroneously recognized octave jumps in tunes in which the harmonic signals were recognized instead of the fundamental tones.

In the context of the transformation or the harmonic decomposition further methods can be used in addition. For example, the coefficients of adjacent frequencies can be obtained by interpolation or by statistical methods.

Likewise, coefficients may be supplemented or replaced by using synthetically generated coefficients as well as those from previous recordings, an earlier analysis of the same signal or mixtures thereof. For example, for a drum upper frequency components can be artificially supplemented from a database.

The generated coefficient files may be in their own format or, if appropriate, after conversion, in a common data format, e.g. MIDI or Wave format. Equally, such files can also be imported and their contents used or modified in the method according to the invention.

From the coefficients, finally, the original or original-sounding signals can be generated by an inverse transformation, for example in wave format, which can then be reproduced, for example, via the computer music system and loudspeaker. In the special case, sounds represented by music notes or images of any kind on the screen can be reconstructed and played back from the TF coefficients.

The method according to the invention or the logical or physical interconnection of the device will be explained in more detail below on the basis of the sequence and arrangement relationships of the individual components and the graphical representation on a screen by way of example and purely schematically.

Fig.1

eine schematische Darstellung der einzelnen Schritte des erfindungsgemässen Verfahrens;

Fig.2

eine schematische Darstellung von Bereitstellungsalternativen für ein Eingangssignal;

Fig.3

eine schematische Darstellung der Zerlegung des Eingangssignals in Frequenzbereiche;

Fig.4

eine schematische Darstellung eines Transformierens der Frequenzbereiche;

Fig.5

eine schematische Darstellung der Schritte zur Notenerkennung durch harmonischen Dekomposition;

Fig.6

eine Darstellung einer graphischen Benutzeroberfläche zur interaktiven Bereitstellung von Zusatzinformationen;

Fig.7

eine Darstellung eines ersten Schrittes in einem ersten Beispiel zur interaktiven Bereitstellung von Zusatzinformationen durch Setzen von Taktmarkierungen;

Fig.8

eine Darstellung eines zweiten Schrittes in einem ersten Beispiel zur interaktiven Bereitstellung von Zusatzinformationen durch Setzen von Taktmarkierungen;

Fig.9

eine Darstellung eines ersten Schrittes in einem zweiten Beispiel zur interaktiven Bereitstellung von Zusatzinformationen durch Anpassung des Verstärkungsfaktors und

Fig.10

eine Darstellung eines zweiten Schrittes in einem zweiten Beispiel zur interaktiven Bereitstellung von Zusatzinformationen durch Anpassung des Verstärkungsfaktors.

Show in detail

Fig.1

a schematic representation of the individual steps of the inventive method;

Fig.2

a schematic representation of deployment alternatives for an input signal;

Figure 3

a schematic representation of the decomposition of the input signal in frequency ranges;

Figure 4

a schematic representation of transforming the frequency ranges;

Figure 5

a schematic representation of the steps for note recognition by harmonic decomposition;

Figure 6

a representation of a graphical user interface for the interactive provision of additional information;

Figure 7

a representation of a first step in a first example for the interactive provision of additional information by setting clock marks;

Figure 8

a representation of a second step in a first example for the interactive provision of additional information by setting clock marks;

Figure 9

a representation of a first step in a second example for the interactive provision of additional information by adjusting the gain and

Figure 10

a representation of a second step in a second example for the interactive provision of additional information by adjusting the gain.

1 shows a schematic representation of the individual steps of the inventive method.

The acoustic signal is detected by a recording component or imported from a data carrier and provided in the form of an input signal ES for further processing. This input signal ES is decomposed in a subband coder SC into individual frequency bands, which are subsequently each supplied to a frequency-optimized first transformation TF1 and a time-optimized second transformation TF2. These transformation processes can also extract information from the original input signal ES in parallel and use it for the transformation process.

The results of the two transformations are combined in a transformation processor TP-possibly in feedback with the first transformation TF1 and the second transformation TF2-to form a coefficient file.

On the basis of this coefficient file, the harmonic decomposition HD is performed to detect patterns inherent in the input signal ES. It can be used for harmonic decomposition HD predetermined coefficients that are stored for example in a memory or supplied via external media.

The identified patterns are made exportable via a graphical conversion for a graphical interface. An example of this is the conversion into notes and, for example, the printout of a score. If a representation is made on a graphical user interface, parameters can be interactively changed or be given as well as further selections or modifications.

For the transfer of files an interface EX / IM is used. Moreover, after a format conversion, the acoustic representation of the patterns may be transmitted via an audio output, e.g. connected to a synthesizer done.

FIG. 2 shows a schematic representation of provision alternatives for the input signal ES. The input signal can be provided by various sources. These include timely or real-time recording as well as the use of stored data. For example, signals in Wave format and files from Audio CDs can be used directly. Files in the formats MPx (MP3, MP4) or WMA or any other format are first converted to wave files by decoders. These are commercially available function libraries, e.g. for MP3 from the Fraunhofer Institute, available on the Internet. Alternatively, the coefficients of MP3 or comparable formats may be arranged directly or via a pre-treatment (e.g., scaling) into one or more layers of the pyramid decomposition of the signal. Decoders for other formats, e.g. Ogg or WMA, are provided on the Internet, e.g. on www.microsoft.com.

A recording buffer AP is part of a Sigalaufnahmeverfahren on the computer, such as Microsoft DirectX. This allows, for example, recordings of signals via a microphone connected to the computer.

The decomposition of the input signal ES into frequency ranges in the subband Coder SC is shown schematically in FIG.

• o für Tiefpass {0.25, 0.5, 0.25} oder {0.05, 0.2, 0.4, 0.2, 0.05} und
• o für Hochpässe Filterkerne deren Mittelwert der Koeffizienten Null (0.0) ergibt, z.B. {-1, 2, -1}.

The input signal ES provided as a wave file is divided into subregions or subbands SBB by means of suitable high-pass filters HP and low-pass filters TP and by reducing the sampling rate, for example by halving the data rate HDR. Typically, each sub-band SBB contains a band-pass filtered version of the input signal ES. Examples of filter cores are

• o for lowpass {0.25, 0.5, 0.25} or {0.05, 0.2, 0.4, 0.2, 0.05} and
• o For high pass filter kernels whose mean value of the coefficients is zero (0.0), eg {-1, 2, -1}.

Optionally, the high pass filters may also be omitted, thereby producing a series of low pass filtered subbands.

4 illustrates the transformation of the frequency ranges in a schematic representation. The individual subbands SBB are subjected to the two differently optimized transformations TF1 and TF2 and subsequently stored in different layers TFL0, TFL1, ... TFLN. The signal stored in the layers TFL0, TFL1, ... TFLN of the output quantity is for example a mixture of the transformation output signals and a pyramid decomposition of the next higher level of the pyramid. Depending on the specific application and types of acoustic input signals ES to be processed, a different type of decomposition or a multiple pyramid decomposition can also be carried out.

5 shows a schematic representation of the steps for note recognition by harmonic decomposition HD. The information contained in the various layers TFL0, TFL1, ... TFLN are combined in a filter FI and then subjected to harmonic decomposition event extraction, where pattern recognition and modeling takes place. For this purpose, a multiplicity of approaches described above can be used according to the invention. The results of the harmonic decomposition HD are represented graphically in the form of notes, for example, so that a selection or specification of information can be made by a user or other methods, which in turn find their way into the step of harmonic decomposition HD.

An example of a graphical user interface for interactively providing additional information is shown in FIG. The surface provides, inter alia, a gain control 1 and a manually changeable clock marker 2 for setting clocks.

The application of the clock marker 2 is explained in FIG. 7 in a first step of a first example for the interactive provision of additional information by setting clock markings. This approach allows a determination of all measures in the entire song. By the clock marker 2, a clock in the song is identified and displayed graphically by a diamond 3 in the top line. The actuation of a function element then leads to the conversion of the events into standard music notes, wherein the automatically set clocks are marked by triangles 4 in the top line. Improvements to this method can still do so be achieved that the soundtracks, especially the drum track, can be used to fine-tune the clocks. Nevertheless, due to variations in the music being played, variations in recording speed or drift effects, calculated clocks and actual patterns in the recording may fall apart, as indicated by arrows in the example within the dashed area.

By manually adjusting the timing mark this falling apart can be corrected again, as shown in Fig.8.

9 shows a representation of a first step in a second example for the interactive provision of additional information by adjusting the amplification factor. In this example, the threshold controller is selected with a threshold greater than 0 so that only note events greater than the threshold are displayed. Some relevant areas are marked by ellipses.

In these areas, after changing the setting of the threshold controller, further information becomes visible, as shown in FIG. If the Threshold slider is set to zero, all note events become visible and all detected events are displayed. By varying the threshold controller, it is thus possible to make adjustments to the result without the entire method having to be carried out again from the beginning.

Claims (14)

Method for pattern assignment for acoustic recordings with the steps Providing a signal representing an acoustic recording; - decomposing the signal into frequency ranges; Transforming the frequency ranges for spectral decomposition into at least one coefficient file; - performing a harmonic decomposition of the coefficient file; and - pattern assignment; characterized in that
when transforming the frequency ranges, in particular in each case for all frequency ranges, at least a first transformation optimized with respect to the frequency resolution and - An optimized with respect to the time resolution second transformation takes place. Method according to claim 1,
characterized in that
When transforming the frequency ranges, an optimized selection of the coefficients from the results of the first transformation and the second transformation and / or a mixture of the coefficients from the results of the first transformation and the second transformation takes place. Method according to claim 2,
characterized in that
when transforming the frequency ranges - the first transformation with a longer time window and the second transformation takes place with a shorter time window, in particular wherein the selection is made on the basis of the ratio of the real parts of the first and second transformations. Method according to claim 2,
characterized in that
when transforming the frequency ranges the selection or mixture based on the frequency-dependent slope of the phase signal in each case for the results of the first transformation and the second transformation takes place. Method according to claim 2,
characterized in that
in transforming the frequency ranges, the selection or blending is done by comparing the results of the first transform and the second transform with a set of predetermined coefficients. Method according to one of the preceding claims,
characterized in that
the first transformation and / or second transformation takes place according to one of the following principles Discrete Fourier transform, Fast Fourier Transformation, Wavelet transformation, - sinus transformation, - cosine transformation. Method according to one of the preceding claims,
characterized in that
When transforming the frequency ranges for each transformation, an aggregate of the results, in particular the temporal integral for a frequency, are taken into account. Method according to one of the preceding claims,
characterized in that
the decomposition of the signal takes place according to the principle of division into octaves and / or pyramid decomposition. Method according to one of the preceding claims,
characterized in that
while performing the harmonic decomposition a comparison with predetermined coefficients, in particular with minimization of the residual, or a comparison with coefficients from a previous analysis of the signal, in particular by deriving coefficients using a characteristic basic profile, he follows. Method according to one of the preceding claims,
characterized in that
when performing the harmonic decomposition, an interaction with a user occurs, in particular by entering additional information. Method according to one of the preceding claims,
characterized in that
when performing the harmonic decomposition original and / or synthetic frequency components, in particular upper frequency components are used. Computer program product with program code stored on a machine-readable carrier or embodied by an electromagnetic wave for carrying out the method according to one of claims 1 to 11. Apparatus for carrying out a method according to one of claims 1 to 11
with at least a recording component for recording an acoustic signal, a subband coder for dividing the signal into individual frequency ranges, a transformation processor for the spectral decomposition of the frequency ranges into at least one coefficient file, - an export interface for exporting the coefficient file, characterized in that
the transformation processor is associated with a first transformation stage and a second transformation stage, wherein the first transformation stage causes an optimized frequency resolution and the second transformation stage effects an optimized time resolution. A coefficient file for use in a method according to any one of claims 1 to 11
marked by
the coefficients of the spectral decomposition of the acoustic signal and associated information to the signal statistics.

Images