EP1280138A1 - Method for audio signals analysis - Google Patents

Abstract

The method involves generating a series of short-time spectra, forming non-linear versions of the spectra in a sound amplitude stimulation plane and in a rhythm stimulation plane, extracting coherent time events from the audio signal and modeling the residual signal of the audio signal. The spectra can be generated by Fourier transformations, wavelet transformations or a hybrid method.

Description

Field of the Invention

The invention relates to a method for analyzing audio signals. Analogous to the way the human brain works, the present method examines the audio signals for frequency and time coherence. By extracting these coherences, data streams of the signals can be separated.

State of the art

The human brain reduces data streams that are supplied by the cochlea, the retina or other sensors. Acoustic information, for example, is reduced to less than 0.1 percent on the way to the neocortex.

Data reduction in analogy to the human brain therefore offers two advantages. On the one hand, you can get a strong compression, on the other hand, when data streams are reduced, only information that would have been removed in the brain and is therefore inaudible is lost.

Psychoacoustic models try to imitate the phenomena of this reduction, cf. Auditory Perception - A New Analysis and Synthesis, Richard M. Warren, 1999 Cambridge University Press, but due to the principle, in principle make very poor results in direct comparison.

The type of data reduction can be explained with the help of information theory. Neural networks try to maximize signal entropy. This process is extremely complicated and can hardly be described analytically, and can actually only be modeled through learning networks.

A major disadvantage of this known method is the very slow convergence, so that it cannot be implemented satisfactorily even on modern computers.

The object of the invention is therefore to provide a method with which acoustic data streams (audio signals) can be analyzed and decomposed with little computation effort so that the separated signals can on the one hand be very well compressed or otherwise processed, but on the other hand as small as possible Have loss of information.

Description of the invention

This object is achieved by a method for analyzing audio signals according to claim 1.

The following terms are used in the description of the invention.

A short-term spectrum of a signal a (t) is a two-dimensional representation S (f, t) in phase space with the coordinates f (frequency) and t (time).

wobei S ⁺ das konjugierte Spektrum bezeichnet. Weist diese Funktion vorhersagbares Verhalten für t=0 bzw. f=0 aus, so spricht man von Frequenzkohärenz respektive Zeitkohärenz. Diese Aussage betrifft das gesammte Kurzzeitspektrum S; will man, wie im folgenden, etwas über lokale Kohärenz erfahren, so zieht man nur einen Ausschnitt von S zur Bewertung heran.The definition of coherence used relates to characteristic properties of the autocorrelation function A _s of short-term spectra S :

where S ⁺ denotes the conjugate spectrum. If this function shows predictable behavior for t = 0 or f = 0 , one speaks of frequency coherence or time coherence. This statement concerns the entire short-term spectrum S; if you want to learn something about local coherence, as in the following, you only use a section of S for evaluation.

als frequenzabhängige komplexwertige Bewertung h(f), die man als Frequenzantwort bezeichnet:

$$\mathit{\text{h}}\text{(}\mathit{\text{f}}\text{) = |}\mathit{\text{h}}\text{(}\mathit{\text{f}}\text{) |}{\mathit{\text{e}}}^{\mathit{\text{i arg(}}}{\text{}}^{\mathit{\text{h}}}{\text{}}^{\text{(}}{\text{}}^{\mathit{\text{f}}}{\text{}}^{\text{))}}\mathit{\text{=}}\text{:}\mathit{\text{g}}\text{(}\mathit{\text{f}}\text{)}{\mathit{\text{e}}}^{\mathit{\text{iφ}}}{\text{}}^{\text{(}}{\text{}}^{\mathit{\text{f}}}{\text{}}^{\text{)}}$$

Filters are defined by their effect in the frequency domain. The filter operator F̂ acts on the Fourier transform

as a frequency-dependent complex valuation h (f) , which is called the frequency response:

$$\mathit{\text{H}}\text{(}\mathit{\text{f}}\text{) = |}\mathit{\text{H}}\text{(}\mathit{\text{f}}\text{) |}{\mathit{\text{e}}}^{\mathit{\text{i arg (}}}{}^{\mathit{\text{H}}}{}^{\text{(}}{}^{\mathit{\text{f}}}{}^{\text{))}}\mathit{\text{=}}\text{:}\mathit{\text{G}}\text{(}\mathit{\text{f}}\text{)}{\mathit{\text{e}}}^{\mathit{\text{Iø}}}{}^{\text{(}}{}^{\mathit{\text{f}}}{}^{\text{)}}$$

The frequency-dependent real quantities g (f) and φ ( f ) are referred to as the amplitude and phase response.

Applying the inverse Fourier transform to the operator definition shows that the filter acts as a convolution in space with F̂ ^-1 [ h ( f )]. This convolution can be described as a scalar product with translationally symmetric vectors V ( t ). A set of filters with different h _n ( f ) thus provides a short-term spectrum according to the definition above. In the case of bandpass filters in which h ( f ) practically disappears to a finite interval, a bank of filters can be used to display short-term Fourier spectra or wavelet spectra. In the first case, the different h _n ( f ) arise by shifting a predetermined h ( f ), in the second case by scaling the frequency axis. With Fourier spectra, the h _n ( f ) have a constant bandwidth, whereas with Wavelet spectra, they have constant quality (constant Q).

Parts of the phase space that have the same type of coherence and are connected are summarized in streams and events . Currents relate to frequency coherence, events to temporal coherence. An example of a current is a unison melody line of an instrument that is not interrupted. An event can be a drum beat, but also the consonants in a vocal line.

The method according to the invention is based on the coherence analysis of audio signals. As in the human brain, a distinction is made between two coherent situations in the signals: firstly, temporal coherence in the form of simultaneity and rhythm, and secondly, coherence in the frequency domain, which is represented by overtone spectra and leads to the perception of a certain pitch. This reduces the complex audio data to rhythm and tonality, which significantly reduces the need for control data.

In order to start the data processing, a series of short-term spectra must first be created, which are required for further analysis. The excitation of the pitch layer is then generated with a non-linear image; Another nonlinear mapping shows the excitation of the rhythm layer. Then the coherent extraction takes place Frequency currents and the coherent temporal events. Finally, the remaining signal is modeled.

The separated streams can be excellently compressed due to their low entropy. In the optimal case, a compression rate of over 1: 100 can be achieved without losses being audible. A possible compression process is described after the separation process.

The steps of the method according to the invention and advantageous embodiments and various applications are described below.

Generation of short-term spectra

The short-term spectra are advantageously generated by means of short-term Fourier transformation, wavelet transformation or by means of a hybrid method from wavelet transformation and Fourier transformation.

The Fourier transformation can be used to generate a short-term spectrum by using a window function w ( t ) which is localized in time by t ₀ = 0:

The window function significantly influences the bandwidth of the individual filters, which has a constant value regardless of f . The frequency resolution is therefore the same across the entire frequency axis. The generation of a short-term spectrum by means of Fourier transform offers the advantage that fast algorithms (FFT, fast Fourier transform) are known for the discrete Fourier transform.

und

Die Transformation ergibt sich dann zu: $$\mathit{\text{S}}\text{(}{\mathit{\text{t}}}_{\text{0}}\text{,}\mathit{\text{f}}\text{)=〈}\mathit{\text{f*M}}\text{((}{\mathit{\text{t-t}}}_{\text{0}}\text{)*}\mathit{\text{f}}\text{)|}\mathit{\text{a}}\text{(}\mathit{\text{t}}\text{)〉}$$

The wavelet transformation (WT) is obtained by defining a mother wavelet M ( t ) with the properties

and

The transformation then results in: $$\mathit{\text{S}}\text{(}{\mathit{\text{<figure-callout id="0" label="t" filenames="imgf0003.png,imgf0004.png" state="{{state}}">t</figure-callout>}}}_{\text{0}}\text{.}\mathit{\text{f}}\text{) = <}\mathit{\text{f * M}}\text{((}{\mathit{\text{tt}}}_{\text{0}}\text{) *}\mathit{\text{f}}\text{) |}\mathit{\text{a}}\text{(}\mathit{\text{t}}\text{)>}$$

. Wegen ihrer logarithmischen Unterteilung hat diese Transformation den großen Vorteil, die Frequenzauflösung des menschlichen Gehörs nachzubilden. Schnelle Wavelettransformationen beruhen auf der Auswertung einer allgemeinen WT auf einem dyadischen Phasenraumgitter.The frequency axis is divided logarithmically homogeneously, so that log ( f ) is usefully considered as a new frequency axis. The wavelet transformation is equivalent to a bank of filters with

, Because of its logarithmic division, this transformation has the great advantage of simulating the frequency resolution of the human ear. Fast wavelet transformations are based on the evaluation of a general WT on a dyadic phase space grating.

, auf einem diskreten Zeitraster, wie es nach der Digitalisierung im Rechner vorliegt. Außerdem verwendet man die Operationen Ĥ und T̂, die den beiden Filtern entsprechen. Um das Verfahren rekursiv anzuwenden, muß die Signalrate halbiert werden, was der Operator D̂ durch entfernen aller ungeraden n erreicht. Umgekehrt fügt Û nach jedem diskreten Signalwert eine Null ein, um die Signalrate zu verdoppeln. Man kann dann die von der dyadischen WT erzeugten Bänder von der größten Frequenz an durchnumerieren : $${\mathit{\text{B}}}_{\mathit{\text{m}}}\text{(}\mathit{\text{n}}\text{) =}\widehat{\mathit{\text{D}}}\widehat{\mathit{\text{H}}}\text{(}\widehat{\mathit{\text{D}}}\widehat{\mathit{\text{T}}}{\text{)}}^{\mathit{\text{m}}}\mathit{\text{a}}\text{(}\mathit{\text{n}}\text{Δ}\mathit{\text{t}}\text{).}$$

The advantages of Fourier and wavelet transformation can be brought together using hybrid methods. Here, a dyadic WT is first performed by recursively halving the frequency spectrum with complementary high and low pass filters. A signal a ( nΔt ), n ∈ is required for implementation

, on a discrete time grid, as it is present in the computer after digitization. In addition, using the operations of H and T, which correspond to the two filters. In order to apply the method recursively, the signal rate must be halved, which the operator D̂ achieves by removing all odd n . Conversely, Û inserts a zero after each discrete signal value to double the signal rate. The bands generated by the dyadic WT can then be numbered from the highest frequency: $${\mathit{\text{B}}}_{\mathit{\text{m}}}\text{(}\mathit{\text{n}}\text{) =}\stackrel{}{\mathit{\text{D}}}\stackrel{}{\mathit{\text{H}}}\text{(}\stackrel{}{\mathit{\text{D}}}\stackrel{}{\mathit{\text{T}}}{\text{)}}^{\mathit{\text{m}}}\mathit{\text{a}}\text{(}\mathit{\text{n}}\text{Δ}\mathit{\text{t}}\text{).}$$

The high computing speed is due to the recursive evaluation of the band B _m over B _{m -1} . The scaling of the frequency axis is logarithmic. In order to increase the resolution of the transformation, each band signal B _m ( n ) can be subdivided further linearly with a discrete Fourier transformation. The individual Fourier spectra must be mirrored in their frequency axis, because the operator D̂ to Ĥ flips the upper part of the spectrum down. The result is a piecewise linear approximation of a logarithmically resolved spectrum. Depending on the window used for the discrete Fourier transformation, the resolution can reach very high values.

Nonlinear pitch excitation

The pitch (pitch) is defined by the frequency perceived by the brain of a tonal event as being in accordance with a sine wave offered for comparison, its frequency f . The pitch scale is advantageously logarithmized to the frequency resolution to do justice to human hearing. Such a scale can be mapped linearly on musical note numbers.

mit p = a log(f)+b und a, b Abbildungskonstanten dar, der sein Maximum bei p _max annimmt. Das Maximum gibt die zum Zeitpunkt t dominante Tonhöhe an.The pitch excitation layer (PEL) represents a time-dependent state PEL _t ( p ) ∈

with p = a log ( f ) + b and a , b represent imaging constants that assume their maximum at p _max . The maximum indicates the dominant pitch at time t .

Further local maxima also show existing pitches for polyphonic (polyphonic) signals. The PEL mimics pitch excitation in the cortex of the human brain by analyzing frequency coherence.

There are various possibilities for generating the pitch excitation. Among other things, neural networks come into question. For example, neural networks with a feedback element and detection inertia of the ART (Adaptive Resonance Theory) type can be used. One such model for expectation-driven current separation is in a simple form in Pitch-based Streaming in Auditory Perception, Stephen Grossberg, in: Musical Networks - Parallel Distributed Perception and Performance, Niall Griffith, Peter M. Todd (Editors), 1999 MIT Press, Cambridge , have been described.

A simpler and therefore particularly suitable option is the use of a deterministic mapping from the short-term spectrum in the PEL. It is advantageous to split this image into two partial images. In a first figure, the logarithm of the spectral amount is taken:

The second figure consists of different parts. First, the correlation of L ( t, f ) with an ideal overtone spectrum is calculated. Then spectral echoes of a tone are suppressed in the PEL, which correspond to the position of possible overtones.

In order to increase the contrast and suppress less pronounced parts of the spectrum, it is advantageous to inhibit the spectrum laterally. This lateral inhibition can be calculated after L ( t, f ), after correlation or after

Echo cancellation can be performed. For lateral inhibition, a non-linear mapping can be used, based on nature.

In order to reduce the effort, it is advantageous to carry out the lateral inhibition using a linear image. The entire second mapping of pitch excitation thus becomes a linear mapping and can be written as a product of matrices. In a preferred embodiment, a first matrix H carries out the lateral inhibition; the contrast of the spectrum is increased in order to provide an optimal starting basis for the following correlation matrix K. The correlation matrix is a matrix that contains all possible overtone positions and thus produces a correspondingly large output at the point with maximum agreement of the overtone spectrum. Then lateral inhibition is performed again. Then, with a "decision matrix" U, the spectral echoes of a tone in the PEL are suppressed, which correspond to the position of possible overtones. Finally, lateral inhibition is carried out again. Depending on the shape of the individual images, it is necessary to connect a matrix M upstream or downstream in order to free the spectral vector from the mean.

   wobei die α _j so gewählt werden, daß

Falls die Kurzzeitspektren mit reinen Fourier- oder Wavelettransformationen ermittelt wurden, ist

a,b sind nach dem zu analysierenden Spektralausschnitt zu wählen, P ist die Anzahl zu korrelierener Obertöne. Die verwendeten Konstanten ergeben sich aus der Lage der interessanten Daten im Spektrum und können relativ frei gewählt werden. Die Anzahl der Obertöne sollte sich zwischen etwa 5 und 20 bewegen, da dies der Zahl der wirklich vorkommenden Obertöne entspricht. Die Konstante p wird empirisch ermittelt. Sie kompensiert die Breite der spektralen Bänder. Für die Hybridmethode kann die Korrelationsmatrix entsprechend stückweise konstruiert werden.In a preferred embodiment, the matrices can have the following shape. The size of the correlation matrix K ⁱ _j corresponds to the length of the discrete spectrum and is designated N. Then the entries can have the form

where the α _j are chosen so that

If the short-term spectra were determined using pure Fourier or wavelet transformations,

a, b are to be selected according to the spectral section to be analyzed, P is the number of overtones to be correlated. The constants used result from the position of the interesting data in the spectrum and can be chosen relatively freely. The number of overtones should be between about 5 and 20, since this corresponds to the number of overtones that actually occur. The constant p is determined empirically. It compensates for the width of the spectral bands. For the hybrid method, the correlation matrix can be constructed piece by piece.

unterdrückt werden:

mit δ_{0 l} dem Kronecker-Symbol; die α _j werden so gewählt, daß

The spectral echoes, which correspond to the position of possible overtones, can be measured with the matrix U $\genfrac{}{}{0ex}{}{\mathit{\text{i}}}{\mathit{\text{j}}}$

can be suppressed:

with δ _{0 l} the Kronecker symbol; the α _j are chosen so that

wählen, wobei die Konstanten s > 0 und ρ₁ > ρ₂ empirisch zu bestimmen sind; die α _j werden so gewählt, daß

The matrix H ⁱ _j can be used for lateral inhibition

choose, whereby the constants s > 0 and ρ ₁ > ρ _{2 are} to be determined empirically; the α _j are chosen so that

The spectral vector must be free of mean values for the above matrices to work correctly.

verwenden:

wobei

die N -dimensionale Identitätsmatrix bezeichnet und E $\genfrac{}{}{0ex}{}{\mathit{\text{i}}}{\mathit{\text{j}}}$

=1, i,j =1,...,N. Definiert man =

MHM , so läßt sich der lineare Anteil der PEL-Abbildung schreiben als

The matrix M $\genfrac{}{}{0ex}{}{\mathit{\text{i}}}{\mathit{\text{j}}}$

use:

in which

denotes the N -dimensional identity matrix and E $\genfrac{}{}{0ex}{}{\mathit{\text{i}}}{\mathit{\text{j}}}$

= 1, i, j = 1, ..., N. If one defines =

MHM , the linear part of the PEL mapping can be written as

To calculate the excitation layer, the logarithmic spectrum must be represented with A : $$\text{PL (}\mathit{\text{t, p}}\text{)}\mathit{\text{= AL}}\text{(}\mathit{\text{t, f}}\text{).}$$

The pitch spectrum generated in this way shows clear characteristics for all tonal events occurring in the audio signal. In order to separate the events, a large number of such pitch spectra can be generated at the same time, all of which inhibit one another, so that a different coherence current is manifested in each spectrum. If you assign each of these Pitch spectra to a copy of his frequency spectrum, you can even generate an expectation-controlled excitation in the pitch spectrum via a feedback in these. Such an ART stream network is ideally suited to model properties of human perception.

It is advantageous to recognize the currents by searching temporally related local maxima on the pitch axis and to calculate the pitch data therefrom as a time series. This stream data will later be used to extract the coherent data.

Nonlinear rhythm stimulation

Sudden changes on the timeline of the short-term spectrum, so-called transients, are the basis for rhythmic sensations and represent the most striking temporal coherence within a short time window.

The rhythmic excitation should react to events with strong temporal coherence with low frequency resolution and relatively high time resolution. It is advisable to recalculate a second spectrum with a lower frequency resolution for this purpose.

To reduce the effort, it is advantageous to use the existing spectrum for this purpose. The basis for the linear mapping into the rhythm excitation layer (REL, rhythm excitation layer) is then the logarithmic spectrum L ( t, f ). The illustration to be applied can be described in two steps.

zur Frequenzrauschunterdrückung die Gestalt

mit

In a first step, the frequency components are averaged in order to obtain a better signal / noise ratio. In a preferred embodiment, which is adapted to the matrices described above, the matrix R $\genfrac{}{}{0ex}{}{\mathit{\text{i}}}{\mathit{\text{j}}}$

the shape for frequency noise suppression

With

The constants a, b are to be selected according to the spectral section to be analyzed as above, in order to be able to compare the PEL with the REL. The constant σ controls the frequency smear and thus the noise suppression.

mit 0 < β < 1 und σ₁ > σ₂ > 0 als empirisch bestimmbaren Parametern.In the human brain, there can only be a temporal correlation over a very short interval. A differential correlation can therefore be carried out in the second step of rhythm stimulation without losing essential information. The operator Ĉ for this mapping is reproduced here analytically and continuously, but can be discretized using standard methods.

with 0 <β <1 and σ ₁ > σ ₂ > 0 as empirically determinable parameters.

gegeben ist. Der Betrag von RL gibt Aufschluß über das Auftreten und den Frequenzbereich von Transienten.The two operators commutate so that the composite mapping through into the rhythm layer $$\text{RL (}\mathit{\text{t, p}}\text{) =}\stackrel{}{\mathit{\text{C}}}\mathit{\text{RL}}\text{(}\mathit{\text{t, f}}\text{)}$$

given is. The amount of RL provides information about the occurrence and frequency range of transients.

Extraction of the coherent frequency currents

Since the PEL streams are well localized in the frequency domain, a filter structure is used to separate the stream from the rest of the data from the audio stream. A filter with a variable center frequency is advantageously used for this. It is particularly advantageous if the pitch information from the PEL level is converted into a frequency trajectory and thus the center frequency of the bandpass filter is controlled. A signal of low bandwidth is thus generated for each overtone, which can then be processed by adding to the total current, but can also be described by means of an amplitude envelope for each overtone and pitch curve.

To delete the signal from the data stream, it must be subtracted. A phase shift can be introduced through the filter. In this case it is necessary perform a phase adjustment after the extraction. This is advantageously achieved by multiplying the extracted signal by a complex value envelope of the amount 1. The envelope is used to achieve phase compensation by means of optimization, for example by minimizing the quadratic error.

It is advantageous to also use the envelope curve to adjust the amplitude of the extracted signal. The pitch information is known from the PEL, so that a corresponding sinusoid can be synthesized which, apart from the missing amplitude information and a certain phase deviation, exactly describes the partial tone of the current.

wobei f(t) den Frequenzverlauf aus der PEL und n die Nummer der harmonischen Komponente bezeichnet. Diese Hüllkurve muß jetzt sowohl die Amplitude anpassen als auch die Phasenverschiebung kompensieren. Das Orginalsignal kann dabei als Referenz genommen werden, um den Fehler der Anpassung zu messen und zu minimieren. Dabei reicht es aus, den Fehler lokal zu reduzieren und sich schrittweise durch die gesamte Hüllkurve zu arbeiten.In a preferred embodiment, the sinusoid S ( t ) can have the following form:

where f ( t ) denotes the frequency response from the PEL and n the number of the harmonic component. This envelope must now both adjust the amplitude and compensate for the phase shift. The original signal can be used as a reference to measure and minimize the error of the adjustment. It is sufficient to reduce the error locally and work through the entire envelope step by step.

mit B _n (t) den komplexwertigen Frequenzantwort des n-ten Filters. In diesem Fall repräsentiert S(t) den kompletten extrahierten Strom und weist keine Phasenverschiebung auf, da diese durch die komplexen Koeffizienten bereits korrigiert wurde. Obige Formel gilt jedoch nur für näherungsweise orthogonale h _n (f), im allgemeinen Fall ist ein Korrekturglied zu ergänzen.If a filter bank has already been used to generate the PEL, this opens up another advantageous possibility for frequency selection of the currents. The required frequency weighting B ( f, t ) for the entire overtone structure can be calculated at any time from the known frequency curve f ( t ). From the known frequency responses h _n ( f ), the coefficients can be calculated from which the current S ( t ) can be extracted:

with B _n ( t ) the complex frequency response of the nth filter. In this case, S ( t ) represents the complete extracted current and has no phase shift, since this has already been corrected by the complex coefficients. However, the above formula applies only for approximately orthogonal h _n ( f ), in the general case a correction element must be added.

Extract coherent temporal events

erhalten werden; das Signal a(t) wird mit H(f) frequenzbewertet und mit W(t) ausgeschnitten.In contrast to the PEL currents, the REL events are poorly localized in the frequency domain, but are rather sharply defined in the period. The extraction strategy should be chosen accordingly. First, a rough frequency evaluation takes place, which is derived from the event blur in the REL. Since no particular precision is required here, it is advantageous to use FFT filters, analysis filter banks or similar tools for the evaluation, but where there should be no dispersion in the pass band. The next step accordingly requires a period evaluation. The event is advantageously separated by multiplication with a window function. The choice of window function must be determined empirically and can also be done adaptively. This allows the extracted event to go through

be preserved; the signal a ( t ) is weighted with H ( f ) and cut out with W ( t ).

Modeling the residual signal

After extraction of the coherent frequency currents and temporal events, the residual signal (residuals) of the audio stream no longer contains any parts that have coherences that can be recognized by the ear, only the frequency distribution is still perceived. It is therefore advantageous to statistically model these parts. Two methods prove to be particularly advantageous for this.

In a first method, several bands are used that contain frequency-localized noise. A frequency analysis of the residual signal provides the mixing ratio; the synthesis then consists of a time-dependent weighted addition of the bands.

bezeichnet man das n-te Moment der Zufallsfolge a _k . Aus den Momenten läßt sich die Verteilungsfunktion der Zufallsfolge berechnen und dann eine äquivalente Folge neu erzeugen. Die Anzahl der analysierten Momente sollte wesentlich kleiner sein als die Länge K der Folge. Genaue Werte erschließen sich durch Hörexperimente.In a second method, the signal is described by its statistical moments. The development over time of these moments is recorded and can be used for resynthesis. The individual statistical moments are at certain time intervals calculated. Advantageously, the interval windows overlap by 50% in the analysis and are then added with a triangular window evaluated in the resynthesis in order to compensate for the overlap. With

is the nth moment of the random sequence a _k . The distribution function of the random sequence can be calculated from the moments and then an equivalent sequence can be generated again. The number of moments analyzed should be significantly smaller than the length K of the sequence. Exact values are revealed through listening experiments.

applications

The method described above can advantageously be used for compressing audio data. For this purpose, a method according to the invention is provided with the steps according to claim 20.

The streams and events separated by the extraction have low entropy and can therefore advantageously be compressed very efficiently. It is advantageous to first transform the signals into a representation suitable for compression.

First, an adaptive differential coding of the PEL currents can take place. From the extraction of the currents, a frequency trajectory is obtained for each stream and an amplitude envelope for each harmonic component present. A double differential scheme is advantageously used to effectively store this data. The data are sampled at regular intervals. A sampling rate of approximately 20 Hz is preferably used. The frequency trajectory is logarithmized to do justice to the tonal resolution of the hearing and quantized on this logarithmic scale. In a preferred embodiment, the resolution is approximately 1/100 halftone. The value of the start frequency and then only the differences from the previous value are advantageously stored explicitly. A dynamic bit adaptation can be used, which generates practically no data at stable frequency positions, such as long tones.

The envelopes can be coded similarly. Here too, the amplitude information is interpreted logarithmically in order to achieve a higher adapted resolution. After the envelope of the fundamental frequency has been coded analogously to the frequency trajectory, everyone becomes Overtone the amplitude start value stored. Since the course of the overtone amplitudes is strongly correlated with the fundamental tone amplitudes, the difference information of the fundamental tone amplitude is advantageously assumed as a change in the overtone amplitude and only the difference to this estimated value is stored. In the case of overtone envelopes, this means that there is only significant data volume if the overtone characteristics change significantly. This further increases the density of information.

The events extracted from the REL layer have little temporal coherence due to their temporal location. It is therefore advantageous to use a time-localized coding and to save the events in their period representation. The events are often very similar to one another. It is therefore advantageous to determine a set of base vectors (transients) by analyzing typical audio data, in which the events can be described by a few coefficients. These coefficients can be quantized and then provide an efficient representation of the data. The basis vectors are preferably determined using neural networks, in particular vector quantization networks, as are known, for example, from Neural Networks, Rüdiger Brause, 1995 BG Teubner Stuttgart.

Because of their statistical character, the residuals can be modeled, as described above, by a time series of moments or by amplitude profiles of band noise. A low sampling rate is sufficient for this type of data. Analogous to the coding of the PEL streams, differential coding with adaptive bit depth adjustment can also be used here, with which the residuals contribute only minimally to the data stream.

As soon as the data has been transformed into a suitable representation, statistical data compression can be carried out by maximizing entropy. LZW or Huffmann processes are particularly suitable.

The signals separated according to the above procedure are also very suitable for manipulating the time base (time stretching), the key (pitch shifting) or the formant structure, whereby formant is to be understood as the range of the sound spectrum in which sound energy is concentrated regardless of the pitch. For these manipulations, the synthesis parameters have to be changed in a suitable manner during the resynthesis of the audio data. For this purpose, methods are provided according to the invention with the steps according to claims 25-28.

The PEL streams are advantageously adapted to a new time base by adapting the time markings of their envelope points or their trajectory points from the PEL in accordance with the new time base. All other parameters can remain unchanged. To change the key, the logarithmic frequency trajectory is shifted along the frequency axis. In order to change the formant structure, a frequency envelope is interpolated from the overtone amplitudes of the PEL currents. This interpolation can preferably be done by averaging over time. This gives a spectrum whose frequency envelope gives the formant structure. This frequency envelope can be shifted independently of the base frequency.

The events of the REL layer remain invariant when the key and formant structure change. If the time base is changed, the time of the events is adjusted accordingly.

Like the REL events, the global residuals remain invariant when the key changes. If the time base is manipulated, the synthesis window length can be adapted in the case of moment encoding. If the residuals are modeled with noise bands, the envelope base points for the noise bands can be adjusted accordingly if the time base is manipulated. The noise band display is preferably used for formant correction. In this case, the band frequency can be adjusted in accordance with the formant shift.

Another notable application is the notation of the audio data in notation. For this purpose, a method according to the invention is provided with the steps according to claim 29. In the process, the PEL currents are first grouped according to their overtone characteristics. The group criterion is provided by a trainable vector quantizer that learns from given examples. A group generated in this way can then be converted into a notation by the frequency trajectories. The pitches can be quantized, for example, in the twelve-tone system and with properties such as vibrato, legato or the like. be provided.

To notate the percussive instruments, coincidences of REL events with low-frequency PEL events or residuals must be recognized. For this purpose, conventional neural networks are preferably used for pattern recognition tasks, as are also described, for example, in Neural Networks, Rüdiger Brause, 1995 BG Teubner Stuttgart. The percussion beats identified in this way are then inserted into the notation.

Claim 30 provides, according to the invention, a method with which track separation of audio signals can be carried out in an advantageous manner. The PEL currents are grouped according to their overtone characteristics and then synthesized separately. To do this, however, certain correlations between REL events, PEL currents and residuals must be recognized, since these are to be combined into a resynthesized track corresponding to the instrument. This relationship can only be determined deterministically to a limited extent; it is therefore preferred to use neural networks as mentioned above for this pattern recognition.

Once the tracks have been separated, they can be edited separately and mixed again. In addition to many other options, individual instruments can be analyzed or replaced, and voices can be hidden or amplified.

It is advantageous to use the method for analyzing audio signals for the global and local identification of audio signals, for which, according to the invention, a method with the steps according to claim 31 or 32 is provided. This identification is based on features that are also available to human perception as recognition features. Different types of recognition can be obtained with different criteria.

In order to uniquely identify a piece of music as a piece stored in a database, the relative position and type, i.e. to compare the internal structure, the currents and events. The inner structure of the melody line, for example, means features such as intervals and long-lasting tones. This comparison with a database can be carried out deterministically and advantageously initially limited to the interval sequences. If there is still no clear identification, additional criteria can be used.

In order to determine the title of a piece of music regardless of the artist or recording circumstances, one has to find dominant structures in the material. These structures can be identified deterministically by frequent repetitions or particularly high signal components. The more such features match a comparison or reference piece, whereby changes in the time base, key or phrasing are permissible, the greater the likelihood that the piece of music examined matches the comparison piece. The comparison of melody lines can advantageously focus on the sequence of the sustained tones and also here only on the sequence the intervals. It is often sufficient to only roughly evaluate and include the rhythmic information, since this information can depend heavily on the interpreter.

The method according to the invention for analyzing audio data can advantageously be used to identify a singing voice in an audio signal. For this, a method according to the invention is provided with the steps according to claim 33. To identify the singer of a piece of music, one advantageously characterizes his voice via the formant structure. As described above, the typical formant layer can be interpolated from the PEL streams. When comparing the formant structures with a database, the selection of possible singers can be greatly restricted, ideally even the singer can be clearly identified.

With all of the identification methods mentioned above, it is advantageous to use a hashing scheme at the beginning in order to limit the selection by means of a checksum comparison with the database and only then to carry out the detailed check.

The method according to the invention for analyzing audio signals can also be used for the restoration of old or technically poor audio data. Typical problems of such recordings are noise, crackling, hum, poor mixing ratios, missing highs or basses. To suppress noise, one identifies (usually manually) the undesired components in the residual level, which are then deleted without falsifying the other data. Crackling is eliminated in an analog way from the REL level and hum from the PEL level. The mixing ratios can be edited by track separation, treble and bass can be re-synthesized with the PEL, REL and residual information.

Figur 1

ein Wavelet-Filterbankspektrum einer Gesangslinie,

Figur 2

ein Kurzzeit-Fourierspektrum der Gesangslinie aus Figur 1,

Figur 3

eine Matrix der linearen Abbildung vom Fourierspektrum zum PEL,

Figur 4

eine Anregung der Tonhöhe im PEL, berechnet aus Figur 2,

Figur 5

eine Anregung im REL, berechnet aus Figur 2.

The method according to the invention for analyzing audio data is explained below using the exemplary embodiment illustrated in the figures. It shows

Figure 1

a wavelet filter bank spectrum of a vocal line,

Figure 2

a short-term Fourier spectrum of the vocal line from Figure 1,

Figure 3

a matrix of the linear mapping from the Fourier spectrum to the PEL,

Figure 4

an excitation of the pitch in the PEL, calculated from FIG. 2,

Figure 5

a suggestion in the REL, calculated from FIG. 2.

There are several options for generating the short-term spectra. FIG. 1 shows a short-term spectrum of a constant Q filter bank, which corresponds to a wavelet transformation. Fourier transforms offer an alternative; FIG. 2 shows a short-term Fourier spectrum that was generated using a fast Fourier transformation.

In a preferred embodiment, the contrast of the spectrum with lateral inhibition is increased to excite the pitch layer. Then a correlation with an ideal overtone spectrum takes place. The resulting spectrum is again laterally inhibited. Subsequently, the pitch layer is freed from weak echoes of the overtones with a decision matrix and laterally inhibited again. This mapping can be chosen linearly. FIG. 3 contains a possible mapping matrix from the Fourier spectrum from FIG. 2 to the PEL.

After the excitation of the pitch layer, different dominant pitches can be seen, as for example in FIG. 4.

In order to stimulate the rhythm layer, frequency noise suppression can be carried out first and then a time correlation can be carried out. If this excitation is carried out for FIG. 2, an excitation in the REL as in FIG. 5 can be obtained.

Claims (34)

Process for analyzing audio signals a) generation of a series of short-term spectra, b) nonlinear mapping of the short-term spectra into the pitch excitation layer (PEL), c) nonlinear mapping of the short-term spectra into the rhythm excitation layer (REL), d) extraction of the coherent frequency currents from the audio signal, e) extraction of the coherent temporal events from the audio signal, f) modeling the residual signal of the audio signal. Method according to Claim 1, in which the short-term spectra are generated by means of short-term Fourier transformation, by means of wavelet transformation or by means of a hybrid method from wavelet transformation and Fourier transformation. Method according to one of the preceding claims, in which the mapping into the pitch excitation layer consists of correlating the logarithm of the spectral amount with a predetermined ideal overtone spectrum, suppressing spectral echoes which correspond to the positions of possible overtones, and then separating the frequency currents. Method according to Claim 3, in which lateral inhibition is carried out according to at least one of the logarithm, correlation and suppression of the echoes. The method of claim 4, wherein the correlation, echo cancellation, and lateral inhibition are linear maps. Method according to one of claims 3-5, in which the separation of the frequency currents is carried out with a neural network. Method according to one of claims 3-5, in which the separation of the frequency currents is achieved by searching for temporally related local maxima and calculating the pitch data as a time series. Method according to one of the preceding claims, in which the mapping into the rhythm excitation layer consists of a linear mapping for frequency noise suppression and for temporal correlation, which is applied to the logarithm of the spectral amount. A method according to claim 8, in which the temporal correlation matrix is given by a differential correlation. Method according to one of the preceding claims, in which the extraction of a frequency current from the audio signal is carried out with a filter with a variable center frequency. A method according to claim 10, in which the center frequency of the filter is controlled via frequency trajectories from the pitch excitation layer. A method according to claim 10 or 11, in which the extracted signal is multiplied by a complex envelope, in order to adjust the phase using an optimization method. A method according to claim 12, in which the complex valued envelope is used to adjust the amplitude of the signal using an optimization method. Method according to one of claims 1-9, in which the frequency currents are calculated as a development according to the band signals of a filter bank, the coefficients being given by projections of a frequency evaluation onto the frequency responses of the filter bank. Method according to one of the preceding claims, in which the extraction of the temporal events consists of a frequency evaluation and a period evaluation. The method of claim 15, in which the frequency evaluation is carried out with an FFT filter or an analysis filter bank. Method according to one of the preceding claims, in which the residual signal is statistically modeled. A method according to claim 17, in which a plurality of bands with frequency-localized noise are used for the modeling, which are added according to a frequency analysis with a time-dependent weighting. A method according to claim 17, in which the modeling of the residual signal is carried out by calculating a distribution function from the statistical moments at predetermined time intervals. Method according to claim 19, in which the interval windows overlap by 50% and are then added in the resynthesis with a triangular window. Method for compressing audio signals by separating the audio signal according to one of the previous methods and then compressing the PEL streams, REL events and the residual signal. 22. The method of claim 21, wherein the compression comprises the steps of: a) adaptive double differential coding of the PEL currents, b) time-localized coding of the REL events, c) adaptive differential coding of the residual signal, d) statistical compression of the data from steps a), b) and c) by maximizing entropy. The method of claim 22, in which the events for the REL coding are given as a linear combination of a finite set of base vectors. Method according to one of claims 22 or 23, in which the final compression is carried out using the LZW or Huffmann method. Method for manipulating the time base of signals which have been separated by the method according to claim 18 by a) determination of the envelopes or trajectories of the PEL currents and the envelopes of the noise bands, b) adjustment of the time markings of the envelope points or trajectory points, c) adjusting the timing of events, d) Adjustment of the envelope support points of the noise bands. Method for manipulating the time base of signals which have been separated using one of the methods according to claims 19 or 20 a) determination of the envelopes or trajectories of the PEL currents, b) adjustment of the time markings of the envelope points or trajectory points, c) adjusting the timing of events, d) Adaptation of the synthesis window lengths for the moment coding. Method for manipulating the key of signals which have been separated using a method according to claims 1-20, by shifting the logarithmic frequency trajectories along the frequency axis. Method for manipulating a formant structure of signals which have been separated by the method according to claim 18 by a) determination of the overtone amplitudes of PEL currents, b) interpolation of a frequency envelope from the overtone amplitudes, c) shifting the frequency envelope, d) Adjustment of the band frequencies in the noise band representation according to the formant shift. Procedure for notation of audio data in musical notation a) separation of the audio signal according to one of the methods 1-20, b) grouping the PEL currents according to their overtone characteristics into at least one group using a trainable vector quantizer, c) identification of the percussive instruments by comparing REL events with low-frequency PEL events or residual signal components using a neural network, d) Converting the frequency trajectories of each group and the percussion beats into notations. Process for the separation of audio data by a) separation of the audio signal according to one of the methods 1-20, b) grouping of the PEL currents according to their overtone characteristics by means of a trainable vector quantizer, c) Identification of PEL currents, REL events and residual signal components belonging to a group by means of a neural network, d) Resynthesis of the related currents, events and residual signal components in one track for each group. Method for identifying an audio signal by separating the signal according to one of claims 1 - 20 and then comparing the relative positions and types of streams and events with a database. Method for identifying an audio signal by separating the signal according to one of claims 1 - 20 and then comparing dominant structures with a database. Method for identifying a voice in an audio signal by extrapolating the formant layer from the PEL streams by separating the signal according to one of claims 1-20 and then comparing it to a database. Method according to one of Claims 31 to 33, in which a hashing scheme is used to restrict the selection after the separation of the signal, and a checksum comparison is therefore carried out with the database.

Images