EP1579426B1 - Method for transmitting audio signals according to the prioritizing pixel transmission method - Google Patents

Method for transmitting audio signals according to the prioritizing pixel transmission method Download PDF

Info

Publication number
EP1579426B1
EP1579426B1 EP03762456A EP03762456A EP1579426B1 EP 1579426 B1 EP1579426 B1 EP 1579426B1 EP 03762456 A EP03762456 A EP 03762456A EP 03762456 A EP03762456 A EP 03762456A EP 1579426 B1 EP1579426 B1 EP 1579426B1
Authority
EP
European Patent Office
Prior art keywords
audio signal
values
groups
priority
array
Prior art date
Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
Expired - Lifetime
Application number
EP03762456A
Other languages
German (de)
French (fr)
Other versions
EP1579426A1 (en
Inventor
Gerd Mossakowski
Current Assignee (The listed assignees may be inaccurate. Google has not performed a legal analysis and makes no representation or warranty as to the accuracy of the list.)
Telekom Deutschland GmbH
Original Assignee
T Mobile Deutschland GmbH
Priority date (The priority date is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the date listed.)
Filing date
Publication date
Application filed by T Mobile Deutschland GmbH filed Critical T Mobile Deutschland GmbH
Priority to SI200331788T priority Critical patent/SI1579426T1/en
Publication of EP1579426A1 publication Critical patent/EP1579426A1/en
Application granted granted Critical
Publication of EP1579426B1 publication Critical patent/EP1579426B1/en
Priority to CY20101100315T priority patent/CY1109952T1/en
Anticipated expiration legal-status Critical
Expired - Lifetime legal-status Critical Current

Links

Classifications

    • G—PHYSICS
    • G10—MUSICAL INSTRUMENTS; ACOUSTICS
    • G10L—SPEECH ANALYSIS OR SYNTHESIS; SPEECH RECOGNITION; SPEECH OR VOICE PROCESSING; SPEECH OR AUDIO CODING OR DECODING
    • G10L19/00—Speech or audio signals analysis-synthesis techniques for redundancy reduction, e.g. in vocoders; Coding or decoding of speech or audio signals, using source filter models or psychoacoustic analysis
    • G10L19/02—Speech or audio signals analysis-synthesis techniques for redundancy reduction, e.g. in vocoders; Coding or decoding of speech or audio signals, using source filter models or psychoacoustic analysis using spectral analysis, e.g. transform vocoders or subband vocoders
    • G10L19/022—Blocking, i.e. grouping of samples in time; Choice of analysis windows; Overlap factoring

Definitions

  • the invention relates to a method for transmitting audio signals according to the method of prioritizing pixel transmission according to the preamble of patent claim 1.
  • these pixels, and the pixel values used for the calculation of the prioritization are transferred or stored.
  • a pixel gets a high priority if the differences to its neighboring pixels are very large.
  • the current pixel values are shown on the display.
  • the not yet transferred pixels are calculated from the already transmitted pixels. In principle, these methods can also be used for transmission of audio signals.
  • the object of the invention is to provide a method for transmitting audio signals, which works as lossless as possible even at low transmission bandwidths.
  • the audio signal is first decomposed into a number n of spectral components.
  • the decomposed audio signal is stored in a two-dimensional array with a plurality of fields, with frequency and time as dimensions and the amplitude as the value to be entered in the field.
  • groups are formed from each individual field and at least two fields of the array adjacent to this field, and a priority is assigned to the individual groups, the priority of a group being greater the larger the amplitudes of the group values are and / or the greater Amplitude differences of the values of a group are and / or the closer the group is to the current time.
  • the new method is based essentially on the foundations of Shannon. Accordingly, signals can be transmitted without loss, if they are scanned at twice the frequency. This means that the sound can be split into individual sine waves of different amplitude and frequency. Accordingly, acoustic signals can be reproduced unambiguously by transmitting the individual frequency components, including the amplitudes and phases, without losses. In this case, particular use is also made of the fact that the frequently occurring sound sources, e.g. Musical instruments, human voice, consist of Resonanzkörpem whose resonant frequency does not change or only slowly.
  • the sound is recorded, converted into electrical signals and divided into its frequency components. This can be done either by FFT (Fast-Fourier Transformation) or by n-single frequency-selecting filters. If n-single filters are used, each filter absorbs only a single frequency or a narrow frequency band (similar to the hair in the human ear). Thus, one has at each time the frequency, and the amplitude value at this frequency. In this case, the number n can assume different values in accordance with the terminal characteristics. The larger n is, the better the audio signal can be reproduced. Thus, n is a parameter with which the quality of the audio transmission can be scaled.
  • FFT Fast-Fourier Transformation
  • the amplitude values are buffered in the fields of a 2-dimensonal array.
  • the first dimension of the array corresponds to the time axis and the second dimension corresponds to the frequency.
  • each sample with respective amplitude value and phase is uniquely determined and can be stored in the associated field of the array as an imaginary number.
  • the speech signal is thus represented in three acoustic dimensions (parameters) in the array: the time eg in milliseconds (ms), perceptually perceived as duration, as the first dimension of the array, the frequency in hertz (Hz), perceptually perceived as pitch, as the second dimension of the array and the energy (or intensity) of the signal, perceived perceptually as volume or intensity, which is stored as a numerical value in the corresponding field of the array.
  • the frequency of the image height, the time of the image width and the amplitude of the audio signal (intensity) corresponds to the color value.
  • groups are formed from adjacent values and prioritized.
  • Each field considered individually, forms a group together with at least one but preferably several adjacent fields.
  • the groups consist of the position value defined by time and frequency, the amplitude value at the position value, and the amplitude values of the surrounding values corresponding to a predetermined shape (see FIG. 2 of the applications DE 101 13 880.6 and DE 101 52 612.1 ).
  • those groups receive a very high priority, which are close to the current time, and / or whose amplitude values are very large in comparison to the other groups and / or in which the amplitude values within the group differ greatly from one another.
  • the pixel group values are sorted in descending order and stored or transmitted in that order.
  • the width of the array (time axis) preferably has only a limited extent (eg 5 seconds), ie there are always only signal sections of eg 5 seconds processed. After this time (eg 5 seconds), the array is filled with the values of the subsequent signal section.
  • the values of the individual groups are received in the receiver.
  • the groups are again entered in a corresponding array.
  • DE 101 13 880.6 and DE 101 52 612.1 can then be generated from the transmitting groups again the three-dimensional spectral representation. The more groups received, the more accurate the reconstruction becomes.
  • the not yet transferred array values are calculated by interpolation from the already transmitted array values.
  • a corresponding audio signal is then generated in the receiver, which can then be converted into sound.
  • n frequency generators can be used whose signals are added to an output signal. This parallel construction of n generators ensures good scalability.
  • the clock rate can be drastically reduced by parallel processing, so that the lower the power consumption, the playback time is increased in mobile devices.
  • FPGA's or ASIC's simple design could be used.
  • the described method is not limited to audio signals.
  • the method can be used effectively wherever multiple sensors (sound sensors, light sensors, tactile sensors, etc.) are used, which continuously measure signals that can then be displayed in an array (nth order).

Abstract

A method for the transmission of audio signals between a transmitter and at least one receiver operates according to the prioritizing pixel transmission method. The audio signal is first broken down into a number of spectral fractions. The broken-down audio signal is stored in a two-dimensional array with a plurality of fields. The dimensions to be registered in the field are frequency and time; the value to be registered in the field is amplitude. Groups are then formed from the individual fields and a priority is assigned to the individual groups, in which the priority will be gauged as higher if the amplitudes of the group values are higher, and/or if the amplitude differences of the values of one group are higher, and/or if the groups are closer to actual time. Finally, the groups are transmitted to the receiver according to the order of their established priority.

Description

Die Erfindung betrifft ein Verfahren zur Übertragung von Audiosignalen nach dem Verfahren der priorisierenden Pixelübertragung nach dem Oberbegriff des Patentanspruchs 1.The invention relates to a method for transmitting audio signals according to the method of prioritizing pixel transmission according to the preamble of patent claim 1.

Zur Zeit existiert eine Vielzahl verschiedener Verfahren zur komprimierten Übertragung von Audiosignalen. Im wesentlichen existieren folgende Verfahren.

  • Reduzierung der Abtastrate, z.B. 3 kHz anstelle von 44 kHz
  • Nichtlineare Übertragung der Abtastwerte, z.B. bei ISDN Übertragung
  • Benutzung von vorher abgespeicherten Akustiksequenzen, z.B. MIDI oder Stimmnachbildung
  • Verwendung von Markov Modellen zur Korrektur von Übertragungsfehlern
At present, there are a variety of different methods for compressed transmission of audio signals. Essentially, the following methods exist.
  • Reduction of the sampling rate, eg 3 kHz instead of 44 kHz
  • Non-linear transmission of the samples, eg for ISDN transmission
  • Use of previously stored acoustic sequences, eg MIDI or voice replication
  • Using Markov models to correct transmission errors

Ein weiteres bekanntes Verfahren zur Audiokodierung mittels einer MDCT-basierten Transformationskodierung wird in der Patentschrift US-A-2002/007273 offenbart.Another known method for audio coding by means of an MDCT-based transformation coding is described in the patent US-A-2002/007273 disclosed.

Die Gemeinsamkeiten der bekannten Verfahren liegen darin, dass auch bei niedrigeren Übertragungsraten eine befriedigende Sprachverständlichkeit vorhanden ist. Dieses wird im wesentlichen durch Mittelwertbildungen erreicht. Jedoch ergeben unterschiedliche Stimmen der Quelle ähnlich klingende Stimmen in der Senke, so dass z.B. Stimmungsschwankungen, die in einem normalen Gespräch erkennbar sind, nicht mehr übertragen werden. Dadurch ergibt sich eine deutliche Einschränkung in der Kommunikationsqualität.The common features of the known methods are that even at lower transmission rates a satisfactory speech intelligibility is available. This is achieved essentially by averaging. However, different voices of the source give similar sounding voices in the well such that e.g. Mood swings, which are recognizable in a normal conversation, are no longer transmitted. This results in a significant restriction in the quality of communication.

Verfahren zur Komprimierung und Dekomprimierung von Bild- oder Videodaten mittels priorisierter Pixelübertragung sind in den deutschen Patentanmeldungen DE 101 13 880.6 (entspricht PCT/DE02/00987 ) und DE 101 52 612.1 (entspricht PCT/DE02/00995 ) beschrieben. Bei diesen Verfahren werden z.B. digitale Bild- oder Videodaten bearbeitet, die aus einem Array einzelner Bildpunkte (Pixel) bestehen, wobei jedes Pixel einen sich zeitlich verändernden Pixelwert aufweist, der Farb- oder Helligkeitsinformation des Pixels beschreibt. Erfindungsgemäß wird jedem Pixel bzw. jeder Pixelgruppe eine Priorität zugeordnet und die Pixel entsprechend ihrer Priorisierung in einem Prioritätenarray abgelegt. Dieses Array enthält zu jedem Zeitpunkt, die nach der Priorisierung sortierten Pixelwerte. Entsprechend der Priorisierung werden diese Pixel, und die für die Berechnung der Priorisierung benutzten Pixelwerte, übertragen bzw. abgespeichert. Ein Pixel bekommt eine hohe Priorität, wenn die Unterschiede zu seinen benachbarten Pixel sehr groß sind. Zur Rekonstruktion werden die jeweils aktuellen Pixelwerte auf dem Display dargestellt. Die noch nicht übertragenen Pixel werden aus den schon übertragenen Pixel berechnet. Diese Verfahren lassen sich prinzipiell auch für eine Übertragung von Audiosignalen verwenden.Methods for compression and decompression of image or video data by means of prioritized pixel transmission are described in the German patent applications DE 101 13 880.6 (equivalent to PCT / DE02 / 00987 ) and DE 101 52 612.1 (equivalent to PCT / DE02 / 00995 ). In these methods, for example, digital image or Editing video data consisting of an array of individual pixels, each pixel having a temporally varying pixel value describing color or brightness information of the pixel. According to the invention, a priority is assigned to each pixel or each pixel group and the pixels are stored in a priority array in accordance with their prioritization. At any given time, this array contains the pixel values sorted after prioritization. According to the prioritization, these pixels, and the pixel values used for the calculation of the prioritization, are transferred or stored. A pixel gets a high priority if the differences to its neighboring pixels are very large. For reconstruction, the current pixel values are shown on the display. The not yet transferred pixels are calculated from the already transmitted pixels. In principle, these methods can also be used for transmission of audio signals.

Die Aufgabe der Erfindung besteht deshalb darin, ein Verfahren zur Übertragung von Audiosignalen anzugeben, das auch bei niedrigen Übertragungsbandbreiten möglichst verlustfrei arbeitet.The object of the invention is to provide a method for transmitting audio signals, which works as lossless as possible even at low transmission bandwidths.

Diese Aufgabe wird erfindungsgemäß durch die Merkmale des Patentanspruchs 1 gelöst.This object is achieved by the features of claim 1.

Gemäß der Erfindung wird das Audiosignal zunächst in eine Anzahl n von spektralen Anteilen zerlegt. Das zerlegte Audiosignals wird in einem zweidimensionalen Array mit einer Vielzahl von Feldern gespeichert, mit Frequenz und Zeit als Dimensionen und der Amplitude als jeweils einzutragenden Wert im Feld. Dann werden aus jedem einzelnen Feld und mindestens zwei zu diesem Feld benachbarten Feldern des Arrays Gruppen gebildet, und den einzelnen Gruppen eine Priorität zugeordnet, wobei die Priorität einer Gruppe umso größer gewählt wird, je größer die Amplituden der Gruppenwerte sind und/oder je größer die Amplitudenunterschiede der Werte einer Gruppe sind und/oder je näher die Gruppe an der aktuellen Zeit liegt.According to the invention, the audio signal is first decomposed into a number n of spectral components. The decomposed audio signal is stored in a two-dimensional array with a plurality of fields, with frequency and time as dimensions and the amplitude as the value to be entered in the field. Then, groups are formed from each individual field and at least two fields of the array adjacent to this field, and a priority is assigned to the individual groups, the priority of a group being greater the larger the amplitudes of the group values are and / or the greater Amplitude differences of the values of a group are and / or the closer the group is to the current time.

Schließlich werden die Gruppen in der Reihenfolge ihrer Priorität an den Empfänger übertragen.Finally, the groups are transmitted to the recipient in order of priority.

Das neue Verfahren beruht Im wesentlichen auf den Grundlagen von Shannon. Demnach lassen sich Signale verlustfrei übertragen, wenn man sie mit der doppelten Frequenz abtastet. Das bedeutet, dass der Schall in einzelne Sinusschwingung unterschiedlicher Amplitude und Frequenz zerlegbar ist. Demnach lassen sich akustische Signale eindeutig durch Übertragung der einzelnen Frequenzanteile, inklusive der Amplituden und Phasen, ohne Verluste wieder herstellen. Hierbei wird auch insbesondere ausgenutzt, dass die häufig vorkommenden Schallquellen, z.B. Musikinstrumente, menschliche Stimme, aus Resonanzkörpem bestehen, deren Resonanzfrequenz sich nicht bzw. nur langsam ändert.The new method is based essentially on the foundations of Shannon. Accordingly, signals can be transmitted without loss, if they are scanned at twice the frequency. This means that the sound can be split into individual sine waves of different amplitude and frequency. Accordingly, acoustic signals can be reproduced unambiguously by transmitting the individual frequency components, including the amplitudes and phases, without losses. In this case, particular use is also made of the fact that the frequently occurring sound sources, e.g. Musical instruments, human voice, consist of Resonanzkörpem whose resonant frequency does not change or only slowly.

Vorteilhafte Ausgestaltungen und Weiterbildungen der Erfindung sind in den abhängigen Patentansprüchen angegeben.Advantageous embodiments and further developments of the invention are specified in the dependent claims.

Ein Ausführungsbeispiel der Erfindung wird nachfolgen beschrieben. Hierbei sei insbesondere auch auf die Beschreibung und die Zeichnungen der älteren Patentanmeldungen DE 101 13 880.6 und DE 101 52 612.1 Bezug genommen.An embodiment of the invention will be described below. In this case, in particular, the description and drawings of the earlier patent applications DE 101 13 880.6 and DE 101 52 612.1 Referenced.

Zunächst wird der Schall aufgenommen, in elektrische Signale umgewandelt und in seine Frequenzanteile zerlegt. Dieses kann entweder durch FFT (Fast-Fourier Transformation) oder durch n-einzelne frequenzselektierende Filter geschehen. Werden n-einzelne Filter verwendet, so nimmt jeder Filter nur eine einzelne Frequenz, bzw. ein schmales Frequenzband, auf (ähnlich den Härchen im menschlichen Ohr). Somit hat man zu jedem Zeitpunkt die Frequenz, und den Amplitudenwert bei dieser Frequenz. Dabei kann die Zahl n entsprechend der Endgeräteeigenschaften unterschiedliche Werte annehmen Je größer n ist, desto besser kann das Audiosignal reproduziert werden. Somit ist n ein Parameter mit dem die Qualität der Audioübertragung skaliert werden kann.First, the sound is recorded, converted into electrical signals and divided into its frequency components. This can be done either by FFT (Fast-Fourier Transformation) or by n-single frequency-selecting filters. If n-single filters are used, each filter absorbs only a single frequency or a narrow frequency band (similar to the hair in the human ear). Thus, one has at each time the frequency, and the amplitude value at this frequency. In this case, the number n can assume different values in accordance with the terminal characteristics. The larger n is, the better the audio signal can be reproduced. Thus, n is a parameter with which the quality of the audio transmission can be scaled.

Die Amplitudenwerte werden in den Feldern eines 2-dimensonalen Arrays zwischengespeichert.
Dabei entspricht die erste Dimension des Arrays der Zeitachse und die zweite Dimension der Frequenz. Damit ist jeder Abtastwert mit jeweiliger Amplitudenwert und Phase eindeutig bestimmt und kann im zugeordneten Feld des Arrays als Imaginäre Zahl abgespeichert werden. Das Sprachsignal wird somit in drei akustischen Dimensionen (Parametern) im Array dargestellt: Die Zeit z.B. in Millisekunden (ms), perzeptiv als Dauer wahrgenommen, als die erste Dimension des Arrays, die Frequenz in Hertz (Hz), perzeptiv als Tonhöhe wahrgenommen, als die zweite Dimension des Arrays und die Energie (bzw. Intensität) des Signals, perzeptiv als Lautstärke bzw. Intensität wahrgenommen, welche als Zahlenwert im entsprechenden Feld des Arrays gespeichert wird.The amplitude values are buffered in the fields of a 2-dimensonal array.
The first dimension of the array corresponds to the time axis and the second dimension corresponds to the frequency. Thus, each sample with respective amplitude value and phase is uniquely determined and can be stored in the associated field of the array as an imaginary number. The speech signal is thus represented in three acoustic dimensions (parameters) in the array: the time eg in milliseconds (ms), perceptually perceived as duration, as the first dimension of the array, the frequency in hertz (Hz), perceptually perceived as pitch, as the second dimension of the array and the energy (or intensity) of the signal, perceived perceptually as volume or intensity, which is stored as a numerical value in the corresponding field of the array.

Im Vergleich zu den Anmeldungen DE 101 13 880.6 und DE 101 52 612.1 entspricht z.B. die Frequenz der Bildhöhe, die Zeit der Bildbreite und die Amplitude des Audiosignals (Intensität) dem Farbwert.Compared to the applications DE 101 13 880.6 and DE 101 52 612.1 For example, the frequency of the image height, the time of the image width and the amplitude of the audio signal (intensity) corresponds to the color value.

Ähnlich dem Verfahren der Priorisierung von Pixelgruppen bei der Bild/Videokodierung werden aus benachbarten Werten Gruppen gebildet und diese priorisiert. Jedes Feld für sich betrachtet bildet zusammen mit mindestens einem, vorzugsweise jedoch mehreren benachbarten Feldern eine Gruppe. Die Gruppen bestehen aus dem Positionswert, definiert durch Zeit und Frequenz, dem Amplitudenwert am Positionswert, und die Amplitudenwerte der umliegenden Werte entsprechend einer vorher festgelegten Form (siehe Figur 2 der Anmeldungen DE 101 13 880.6 und DE 101 52 612.1 ). Dabei bekommen insbesondere diejenigen Gruppen eine sehr hohe Priorität, die nahe der aktuellen Zeit liegen, und/oder deren Amplitudenwerte im Vergleich zu den anderen Gruppen sehr groß sind und/oder bei denen sich die Amplitudenwerte innerhalb der Gruppe stark voneinander unterscheiden. Die Pixelgruppenwerte werden absteigend sortiert und in dieser Reihenfolge gespeichert bzw. übertragen.
Die Breite des Arrays (Zeitachse) besitzt vorzugsweise nur eine begrenzte Ausdehnung (z.B. 5 Sekunden), d.h. es werden immer nur Signalabschnitte von z.B. 5 Sekunden Länge verarbeitet. Nach dieser Zeit (z.B. 5 Sekunden) wird das Array mit den Werten des nachfolgenden Signalabschnitts gefüllt.Similar to the method of prioritizing pixel groups in the image / video encoding, groups are formed from adjacent values and prioritized. Each field, considered individually, forms a group together with at least one but preferably several adjacent fields. The groups consist of the position value defined by time and frequency, the amplitude value at the position value, and the amplitude values of the surrounding values corresponding to a predetermined shape (see FIG. 2 of the applications DE 101 13 880.6 and DE 101 52 612.1 ). In particular, those groups receive a very high priority, which are close to the current time, and / or whose amplitude values are very large in comparison to the other groups and / or in which the amplitude values within the group differ greatly from one another. The pixel group values are sorted in descending order and stored or transmitted in that order.
The width of the array (time axis) preferably has only a limited extent (eg 5 seconds), ie there are always only signal sections of eg 5 seconds processed. After this time (eg 5 seconds), the array is filled with the values of the subsequent signal section.

Entsprechend der oben beschriebenen Priorisierungsparameter (Amplitude, zeitnahe Position und Amplitudenunterschiede zu benachbarten Werten) werden die Werte der einzelnen Gruppen im Empfänger empfangen.According to the prioritization parameters described above (amplitude, timely position and amplitude differences to adjacent values), the values of the individual groups are received in the receiver.

Beim Empfänger werden die Gruppen wieder in ein entsprechendes Array eingetragen. Entsprechend der Patentanmeldungen DE 101 13 880.6 und DE 101 52 612.1 kann dann aus den übertragenden Gruppen wieder die dreidimensionale Spektraldarstellung erzeugt werden. Je mehr Gruppen empfangen wurden, umso genauer wird die Rekonstruktion. Die noch nicht übertragenen Arraywerte werden mittels Interpolation aus den schon übertragenen Arraywerten berechnet. Aus dem so erzeugten Array wird dann im Empfänger ein entsprechendes Audiosignal generiert, welches dann in Schall umgewandelt werden kann.
Zur Synthese des Audiosignals können z.B. n Frequenzgeneratoren verwenden werden, deren Signale zu einem Ausgangssignal addiert werden. Durch diesen parallelen Aufbau von n Generatoren ist eine gute Skalierbarkeit gegeben. Zudem kann die Taktrate durch parallele Verarbeitung drastisch reduziert werden, so das durch ein geringeren Energieverbrauch die Wiedergabezeit bei mobilen Endgeräten erhöht wird. Für den parallelen Einsatz könnten z.B. FPGA's oder ASIC's einfacher Bauart benutzt werden.At the receiver, the groups are again entered in a corresponding array. According to the patent applications DE 101 13 880.6 and DE 101 52 612.1 can then be generated from the transmitting groups again the three-dimensional spectral representation. The more groups received, the more accurate the reconstruction becomes. The not yet transferred array values are calculated by interpolation from the already transmitted array values. From the array thus generated, a corresponding audio signal is then generated in the receiver, which can then be converted into sound.
For the synthesis of the audio signal, for example, n frequency generators can be used whose signals are added to an output signal. This parallel construction of n generators ensures good scalability. In addition, the clock rate can be drastically reduced by parallel processing, so that the lower the power consumption, the playback time is increased in mobile devices. For parallel use, for example FPGA's or ASIC's simple design could be used.

Das beschriebene Verfahren ist nicht auf Audiosignale beschränkt. Das Verfahren kann insbesondere überall dort effektiv zur Anwendung kommen, wo mehrere Sensoren (Schallsensoren, Lichtsensoren, Tastsensoren, usw.) verwendet werden, die kontinuierlich Signale messen, die dann in einem Array (n-ter Ordnung) dargestellt werden können.The described method is not limited to audio signals. In particular, the method can be used effectively wherever multiple sensors (sound sensors, light sensors, tactile sensors, etc.) are used, which continuously measure signals that can then be displayed in an array (nth order).

Die Vorteile gegenüber bisherigen Systemen liegen in der flexiblen Einsetzbarkeit bei erhöhten Kompressionsraten. Durch Benutzung eines Arrays, welches aus unterschiedlichen Quellen gespeist wird, erhält man automatisch eine Synchronisation der unterschiedlichen Quellen. Eine entsprechende Synchronisation muss bei herkömmlichen Verfahren durch besondere Protokolle, bzw. Maßnahmen gesichert werden. Insbesondere bei Videoübertragung mit großen Laufzeiten, z.B. Satellitenverbindungen, wo Ton und Bild über verschiedene Kanäle übertragen werden, fällt häufig eine fehlende Synchronisation der Lippen zu der Sprache auf. So etwas kann durch das beschriebene Verfahren beseitigt werdenThe advantages over previous systems lie in the flexible use at elevated compression rates. By using an array, which is fed from different sources, you automatically get one Synchronization of different sources. A corresponding synchronization must be secured by special protocols or measures in conventional methods. In particular, in video transmission with long durations, such as satellite connections, where sound and image are transmitted over different channels, is often a lack of synchronization of the lips to the language. Such a thing can be eliminated by the described method

Da das gleiche Grundprinzip der priorisierenden Pixelgruppenübertragung sowohl bei Sprache, Bild und Videoübertragung genutzt werden kann, ist ein starker Synergieeffekt bei der Implementierung nutzbar. Außerdem kann auf diese Weise eine einfache Synchronisation zwischen Sprache und Bildern erfolgen. Außerdem könnte beliebig zwischen Bild- und Audioauflösung skaliert werden.Since the same basic principle of the prioritizing pixel group transmission can be used both in voice, image and video transmission, a strong synergy effect in the implementation can be used. In addition, a simple synchronization between speech and images can be done in this way. In addition, it would be possible to scale arbitrarily between image and audio resolution.

Betrachtet man eine einzelne Audioübertragung nach dem neuen Verfahren, so ergibt sich bei Sprache eine natürlichere Wiedergabe, da die für jeden Menschen typischen Frequenzanteile (-gruppen) mit höchster Priorität und damit verlustfrei übertragen werden.Considering a single audio transmission according to the new method, there is a more natural reproduction in speech, since the frequency components (groups) typical for each human are transmitted with the highest priority and thus lossless.

Claims (7)

  1. Method for the transmission of audio signals between a transmitter and at least one receiver by a method of prioritising pixel transmission, characterised by the steps of:
    a) separating the audio signal into a number n of spectral fractions at frequency and amplitude at a point in time,
    b) storing the separated audio signal in a two-dimensional array having a plurality of fields, with frequency and time as dimensions and the amplitude as the value to be entered at any given time in the field,
    c) forming groups from each individual field and at least two fields of the array adjacent to the latter field,
    d) assigning a priority to the individual groups, the priority of a group being greater, the greater are the amplitudes of the values of a group and/or the greater are the amplitude differences of the values of a group and/or the closer the group is to a current time, and
    e) transmitting the groups in order of priority sorted in descending order to the receiver.
  2. Method according to claim 1, characterised in that the whole audio signal is present as an audio file and is processed and transmitted as a whole.
  3. Method according to claim 1, characterised in that in each case only a portion of the audio signal is processed and transmitted.
  4. Method according to one of claims 1 to 3, characterised in that the audio signal is separated into its spectral fractions by FFT.
  5. Method according to one of claims 1 to 3, characterised in that the audio signal is separated into its spectral fractions by a number n of frequency-selecting filters.
  6. Method according to one of claims 1 to 5, characterised in that in the receiver the groups transmitted according to their priority are assigned to a corresponding array, the as yet untransmitted values of the array being calculated from the existing values by interpolation.
  7. Method according to one of claims 1 to 6, characterised in that from the values present in the receiver and calculated an electrical signal is generated and converted to an audio signal.
EP03762456A 2002-07-08 2003-07-07 Method for transmitting audio signals according to the prioritizing pixel transmission method Expired - Lifetime EP1579426B1 (en)

Priority Applications (2)

Application Number Priority Date Filing Date Title
SI200331788T SI1579426T1 (en) 2002-07-08 2003-07-07 Method for transmitting audio signals according to the prioritizing pixel transmission method
CY20101100315T CY1109952T1 (en) 2002-07-08 2010-04-06 METHOD FOR THE TRANSFER OF AUDIO SIGNALS IN ACCORDANCE WITH THE METHOD OF HISTORICAL TRANSMISSION OF HOUSES

Applications Claiming Priority (3)

Application Number Priority Date Filing Date Title
DE10230809A DE10230809B4 (en) 2002-07-08 2002-07-08 Method for transmitting audio signals according to the method of prioritizing pixel transmission
DE10230809 2002-07-08
PCT/DE2003/002258 WO2004006224A1 (en) 2002-07-08 2003-07-07 Method for transmitting audio signals according to the prioritizing pixel transmission method

Publications (2)

Publication Number Publication Date
EP1579426A1 EP1579426A1 (en) 2005-09-28
EP1579426B1 true EP1579426B1 (en) 2010-01-06

Family

ID=29796219

Family Applications (1)

Application Number Title Priority Date Filing Date
EP03762456A Expired - Lifetime EP1579426B1 (en) 2002-07-08 2003-07-07 Method for transmitting audio signals according to the prioritizing pixel transmission method

Country Status (16)

Country Link
US (1) US7603270B2 (en)
EP (1) EP1579426B1 (en)
JP (1) JP4637577B2 (en)
CN (1) CN1323385C (en)
AT (1) ATE454695T1 (en)
AU (1) AU2003250775A1 (en)
CY (1) CY1109952T1 (en)
DE (2) DE10230809B4 (en)
DK (1) DK1579426T3 (en)
ES (1) ES2339237T3 (en)
HK (1) HK1081714A1 (en)
PL (1) PL207103B1 (en)
PT (1) PT1579426E (en)
RU (1) RU2322706C2 (en)
SI (1) SI1579426T1 (en)
WO (1) WO2004006224A1 (en)

Families Citing this family (3)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
JP3469567B2 (en) * 2001-09-03 2003-11-25 三菱電機株式会社 Acoustic encoding device, acoustic decoding device, acoustic encoding method, and acoustic decoding method
DE102007017254B4 (en) * 2006-11-16 2009-06-25 Fraunhofer-Gesellschaft zur Förderung der angewandten Forschung e.V. Device for coding and decoding
EP3121814A1 (en) * 2015-07-24 2017-01-25 Sound object techology S.A. in organization A method and a system for decomposition of acoustic signal into sound objects, a sound object and its use

Family Cites Families (29)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
JP2914974B2 (en) * 1987-02-27 1999-07-05 株式会社日立製作所 Variable rate audio signal transmission method and transmission system
US5253326A (en) * 1991-11-26 1993-10-12 Codex Corporation Prioritization method and device for speech frames coded by a linear predictive coder
JP3153933B2 (en) * 1992-06-16 2001-04-09 ソニー株式会社 Data encoding device and method and data decoding device and method
US5517511A (en) * 1992-11-30 1996-05-14 Digital Voice Systems, Inc. Digital transmission of acoustic signals over a noisy communication channel
US5675705A (en) * 1993-09-27 1997-10-07 Singhal; Tara Chand Spectrogram-feature-based speech syllable and word recognition using syllabic language dictionary
JP2797959B2 (en) * 1994-03-12 1998-09-17 日本ビクター株式会社 Multidimensional image compression / expansion method
JPH1083623A (en) * 1996-09-10 1998-03-31 Sony Corp Signal recording method, signal recorder, recording medium and signal processing method
US5886276A (en) * 1997-01-16 1999-03-23 The Board Of Trustees Of The Leland Stanford Junior University System and method for multiresolution scalable audio signal encoding
SE9700772D0 (en) * 1997-03-03 1997-03-03 Ericsson Telefon Ab L M A high resolution post processing method for a speech decoder
US6144937A (en) * 1997-07-23 2000-11-07 Texas Instruments Incorporated Noise suppression of speech by signal processing including applying a transform to time domain input sequences of digital signals representing audio information
US6351730B2 (en) * 1998-03-30 2002-02-26 Lucent Technologies Inc. Low-complexity, low-delay, scalable and embedded speech and audio coding with adaptive frame loss concealment
US6584509B2 (en) * 1998-06-23 2003-06-24 Intel Corporation Recognizing audio and video streams over PPP links in the absence of an announcement protocol
JP3522137B2 (en) * 1998-12-18 2004-04-26 富士通株式会社 Variable rate encoding / decoding device
JP3797836B2 (en) * 1999-12-09 2006-07-19 株式会社東芝 Remote maintenance system
DE10008055A1 (en) * 2000-02-22 2001-08-30 Infineon Technologies Ag Data compression method
JP3576936B2 (en) * 2000-07-21 2004-10-13 株式会社ケンウッド Frequency interpolation device, frequency interpolation method, and recording medium
JP3576935B2 (en) * 2000-07-21 2004-10-13 株式会社ケンウッド Frequency thinning device, frequency thinning method and recording medium
JP2002135122A (en) * 2000-10-19 2002-05-10 Nec Corp Audio signal coding apparatus
US6952669B2 (en) * 2001-01-12 2005-10-04 Telecompression Technologies, Inc. Variable rate speech data compression
DE10113880B4 (en) * 2001-03-21 2004-04-29 T-Mobile Deutschland Gmbh Method for compressing and decompressing video data
WO2002078322A1 (en) * 2001-03-21 2002-10-03 T-Mobile Deutschland Gmbh Method for compression and decompression of image data
DE10152612B4 (en) * 2001-03-21 2006-02-23 T-Mobile Deutschland Gmbh Method for compressing and decompressing image data
US7136418B2 (en) * 2001-05-03 2006-11-14 University Of Washington Scalable and perceptually ranked signal coding and decoding
US7079658B2 (en) * 2001-06-14 2006-07-18 Ati Technologies, Inc. System and method for localization of sounds in three-dimensional space
JP3463752B2 (en) * 2001-07-25 2003-11-05 三菱電機株式会社 Acoustic encoding device, acoustic decoding device, acoustic encoding method, and acoustic decoding method
US20030236674A1 (en) * 2002-06-19 2003-12-25 Henry Raymond C. Methods and systems for compression of stored audio
DE10229706A1 (en) * 2002-07-02 2004-12-09 T-Mobile Deutschland Gmbh Method for managing memory space in a storage medium of a digital terminal device for data storage according to the method of prioritized pixel transmission
DE10229976B4 (en) * 2002-07-03 2007-06-28 T-Mobile Deutschland Gmbh Method for encrypting and decrypting digital data transmitted or stored according to the method of prioritized pixel transmission
US7359979B2 (en) * 2002-09-30 2008-04-15 Avaya Technology Corp. Packet prioritization and associated bandwidth and buffer management techniques for audio over IP

Also Published As

Publication number Publication date
RU2322706C2 (en) 2008-04-20
ATE454695T1 (en) 2010-01-15
CN1666255A (en) 2005-09-07
JP4637577B2 (en) 2011-02-23
PT1579426E (en) 2010-04-08
DE50312330D1 (en) 2010-02-25
WO2004006224A1 (en) 2004-01-15
PL207103B1 (en) 2010-11-30
JP2005532580A (en) 2005-10-27
RU2005102935A (en) 2005-10-27
HK1081714A1 (en) 2006-05-19
ES2339237T3 (en) 2010-05-18
DE10230809A1 (en) 2004-01-29
EP1579426A1 (en) 2005-09-28
DK1579426T3 (en) 2010-05-17
US7603270B2 (en) 2009-10-13
AU2003250775A1 (en) 2004-01-23
DE10230809B4 (en) 2008-09-11
US20060015346A1 (en) 2006-01-19
CY1109952T1 (en) 2014-09-10
CN1323385C (en) 2007-06-27
SI1579426T1 (en) 2010-05-31
PL374146A1 (en) 2005-10-03

Similar Documents

Publication Publication Date Title
DE60208426T2 (en) DEVICE FOR SIGNAL CODING, SIGNAL DECODING AND SYSTEM FOR DISTRIBUTING AUDIO DATA
EP0290581B1 (en) Process for transmitting digital audio-signals
DE19604273C2 (en) Method and device for performing a search in a code book with regard to the coding of a sound signal, cell communication system, cell network element and mobile cell transmitter / receiver unit
EP1979901B1 (en) Method and arrangements for audio signal encoding
DE2656044C2 (en) Method and device for coding and transmission of a speech signal
DE60012198T2 (en) ENCODING THE CORD OF THE SPECTRUM BY VARIABLE TIME / FREQUENCY RESOLUTION
DE60103086T2 (en) IMPROVEMENT OF SOURCE DELIVERY SYSTEMS BY ADAPTIVE TRANSPOSITION
EP0205200A1 (en) Method for transmitting audio and additional information in digital form
EP1023777B1 (en) Method and device for limiting a stream of audio data with a scaleable bit rate
EP1979899A1 (en) Method and arrangements for encoding audio signals
DE60311891T2 (en) AUDIO CODING
EP0527374A2 (en) Audio signal coding method with 32 kbit/s
DE2609297C3 (en) Transmission system for voice signals
EP1579426B1 (en) Method for transmitting audio signals according to the prioritizing pixel transmission method
DE4020656A1 (en) METHOD FOR TRANSMITTING A SIGNAL
DE2836738B2 (en) Method for recording sound signals produced when playing a key-operated musical instrument and the associated playing expressions, in particular the volume, corresponding data signal sequences on a recording medium and devices for carrying out the method
DE2051589C3 (en) Electric synthesizer
DE2826818A1 (en) METHOD AND DEVICE FOR GENERATING AN ARTIFICIAL VOICE SIGNAL
DE69732870T2 (en) System for coding and decoding audio signals
EP1834322B1 (en) Analog signal encoding method
AT247018B (en) Arrangement for obtaining a digitized representation of natural language
AT500636A2 (en) METHOD FOR CODING ONE-DIMENSIONAL DIGITAL SIGNALS
AT509439B1 (en) METHOD AND MEANS FOR SCALABLE IMPROVEMENT OF THE QUALITY OF A SIGNAL CODING METHOD
DE3218755A1 (en) CIRCUIT ARRANGEMENT FOR THE ELECTRONIC VOICE SYNTHESIS
DE838170C (en) Multiple electric generator

Legal Events

Date Code Title Description
PUAI Public reference made under article 153(3) epc to a published international application that has entered the european phase

Free format text: ORIGINAL CODE: 0009012

17P Request for examination filed

Effective date: 20041217

AK Designated contracting states

Kind code of ref document: A1

Designated state(s): AT BE BG CH CY CZ DE DK EE ES FI FR GB GR HU IE IT LI LU MC NL PT RO SE SI SK TR

AX Request for extension of the european patent

Extension state: AL LT LV MK

DAX Request for extension of the european patent (deleted)
GRAP Despatch of communication of intention to grant a patent

Free format text: ORIGINAL CODE: EPIDOSNIGR1

GRAS Grant fee paid

Free format text: ORIGINAL CODE: EPIDOSNIGR3

GRAA (expected) grant

Free format text: ORIGINAL CODE: 0009210

AK Designated contracting states

Kind code of ref document: B1

Designated state(s): AT BE BG CH CY CZ DE DK EE ES FI FR GB GR HU IE IT LI LU MC NL PT RO SE SI SK TR

REG Reference to a national code

Ref country code: GB

Ref legal event code: FG4D

Free format text: NOT ENGLISH

REG Reference to a national code

Ref country code: CH

Ref legal event code: EP

REG Reference to a national code

Ref country code: IE

Ref legal event code: FG4D

REF Corresponds to:

Ref document number: 50312330

Country of ref document: DE

Date of ref document: 20100225

Kind code of ref document: P

REG Reference to a national code

Ref country code: RO

Ref legal event code: EPE

REG Reference to a national code

Ref country code: PT

Ref legal event code: SC4A

Free format text: AVAILABILITY OF NATIONAL TRANSLATION

Effective date: 20100330

REG Reference to a national code

Ref country code: CH

Ref legal event code: NV

Representative=s name: LUCHS & PARTNER PATENTANWAELTE

REG Reference to a national code

Ref country code: NL

Ref legal event code: T3

Ref country code: GR

Ref legal event code: EP

Ref document number: 20100400762

Country of ref document: GR

REG Reference to a national code

Ref country code: DK

Ref legal event code: T3

REG Reference to a national code

Ref country code: ES

Ref legal event code: FG2A

Ref document number: 2339237

Country of ref document: ES

Kind code of ref document: T3

REG Reference to a national code

Ref country code: SK

Ref legal event code: T3

Ref document number: E 7088

Country of ref document: SK

REG Reference to a national code

Ref country code: HU

Ref legal event code: AG4A

Ref document number: E007767

Country of ref document: HU

PLBE No opposition filed within time limit

Free format text: ORIGINAL CODE: 0009261

STAA Information on the status of an ep patent application or granted ep patent

Free format text: STATUS: NO OPPOSITION FILED WITHIN TIME LIMIT

26N No opposition filed

Effective date: 20101007

PG25 Lapsed in a contracting state [announced via postgrant information from national office to epo]

Ref country code: MC

Free format text: LAPSE BECAUSE OF NON-PAYMENT OF DUE FEES

Effective date: 20100731

PG25 Lapsed in a contracting state [announced via postgrant information from national office to epo]

Ref country code: DE

Free format text: LAPSE BECAUSE OF NON-PAYMENT OF DUE FEES

Effective date: 20110201

REG Reference to a national code

Ref country code: DE

Ref legal event code: R119

Ref document number: 50312330

Country of ref document: DE

Effective date: 20110201

REG Reference to a national code

Ref country code: SK

Ref legal event code: MM4A

Ref document number: E 7088

Country of ref document: SK

Effective date: 20110707

PG25 Lapsed in a contracting state [announced via postgrant information from national office to epo]

Ref country code: SK

Free format text: LAPSE BECAUSE OF NON-PAYMENT OF DUE FEES

Effective date: 20110707

PG25 Lapsed in a contracting state [announced via postgrant information from national office to epo]

Ref country code: LU

Free format text: LAPSE BECAUSE OF NON-PAYMENT OF DUE FEES

Effective date: 20100707

PG25 Lapsed in a contracting state [announced via postgrant information from national office to epo]

Ref country code: TR

Free format text: LAPSE BECAUSE OF FAILURE TO SUBMIT A TRANSLATION OF THE DESCRIPTION OR TO PAY THE FEE WITHIN THE PRESCRIBED TIME-LIMIT

Effective date: 20100106

REG Reference to a national code

Ref country code: FR

Ref legal event code: PLFP

Year of fee payment: 14

REG Reference to a national code

Ref country code: FR

Ref legal event code: PLFP

Year of fee payment: 15

REG Reference to a national code

Ref country code: FR

Ref legal event code: PLFP

Year of fee payment: 16

PGFP Annual fee paid to national office [announced via postgrant information from national office to epo]

Ref country code: RO

Payment date: 20210625

Year of fee payment: 19

Ref country code: CZ

Payment date: 20210624

Year of fee payment: 19

PGFP Annual fee paid to national office [announced via postgrant information from national office to epo]

Ref country code: NL

Payment date: 20210721

Year of fee payment: 19

PGFP Annual fee paid to national office [announced via postgrant information from national office to epo]

Ref country code: CY

Payment date: 20210630

Year of fee payment: 19

Ref country code: EE

Payment date: 20210720

Year of fee payment: 19

Ref country code: FI

Payment date: 20210720

Year of fee payment: 19

Ref country code: AT

Payment date: 20210720

Year of fee payment: 19

Ref country code: BG

Payment date: 20210719

Year of fee payment: 19

Ref country code: FR

Payment date: 20210722

Year of fee payment: 19

Ref country code: IT

Payment date: 20210730

Year of fee payment: 19

Ref country code: IE

Payment date: 20210726

Year of fee payment: 19

PGFP Annual fee paid to national office [announced via postgrant information from national office to epo]

Ref country code: GR

Payment date: 20210720

Year of fee payment: 19

Ref country code: HU

Payment date: 20210628

Year of fee payment: 19

Ref country code: SK

Payment date: 20210701

Year of fee payment: 19

Ref country code: SI

Payment date: 20210628

Year of fee payment: 19

Ref country code: SE

Payment date: 20210721

Year of fee payment: 19

Ref country code: BE

Payment date: 20210721

Year of fee payment: 19

Ref country code: CH

Payment date: 20210723

Year of fee payment: 19

Ref country code: DK

Payment date: 20210721

Year of fee payment: 19

Ref country code: ES

Payment date: 20210819

Year of fee payment: 19

Ref country code: GB

Payment date: 20210722

Year of fee payment: 19

PGFP Annual fee paid to national office [announced via postgrant information from national office to epo]

Ref country code: PT

Payment date: 20210701

Year of fee payment: 19

PG25 Lapsed in a contracting state [announced via postgrant information from national office to epo]

Ref country code: CZ

Free format text: LAPSE BECAUSE OF NON-PAYMENT OF DUE FEES

Effective date: 20220707

REG Reference to a national code

Ref country code: DK

Ref legal event code: EBP

Effective date: 20220731

REG Reference to a national code

Ref country code: CH

Ref legal event code: PL

Ref country code: SE

Ref legal event code: EUG

REG Reference to a national code

Ref country code: NL

Ref legal event code: MM

Effective date: 20220801

REG Reference to a national code

Ref country code: AT

Ref legal event code: MM01

Ref document number: 454695

Country of ref document: AT

Kind code of ref document: T

Effective date: 20220707

GBPC Gb: european patent ceased through non-payment of renewal fee

Effective date: 20220707

REG Reference to a national code

Ref country code: BE

Ref legal event code: MM

Effective date: 20220731

PG25 Lapsed in a contracting state [announced via postgrant information from national office to epo]

Ref country code: SE

Free format text: LAPSE BECAUSE OF NON-PAYMENT OF DUE FEES

Effective date: 20220708

Ref country code: RO

Free format text: LAPSE BECAUSE OF NON-PAYMENT OF DUE FEES

Effective date: 20220707

Ref country code: PT

Free format text: LAPSE BECAUSE OF NON-PAYMENT OF DUE FEES

Effective date: 20230109

Ref country code: LI

Free format text: LAPSE BECAUSE OF NON-PAYMENT OF DUE FEES

Effective date: 20220731

Ref country code: FR

Free format text: LAPSE BECAUSE OF NON-PAYMENT OF DUE FEES

Effective date: 20220731

Ref country code: FI

Free format text: LAPSE BECAUSE OF NON-PAYMENT OF DUE FEES

Effective date: 20220707

Ref country code: CH

Free format text: LAPSE BECAUSE OF NON-PAYMENT OF DUE FEES

Effective date: 20220731

Ref country code: AT

Free format text: LAPSE BECAUSE OF NON-PAYMENT OF DUE FEES

Effective date: 20220707

REG Reference to a national code

Ref country code: SI

Ref legal event code: KO00

Effective date: 20230314

PG25 Lapsed in a contracting state [announced via postgrant information from national office to epo]

Ref country code: SI

Free format text: LAPSE BECAUSE OF NON-PAYMENT OF DUE FEES

Effective date: 20220708

Ref country code: HU

Free format text: LAPSE BECAUSE OF NON-PAYMENT OF DUE FEES

Effective date: 20220708

Ref country code: GR

Free format text: LAPSE BECAUSE OF NON-PAYMENT OF DUE FEES

Effective date: 20230209

Ref country code: GB

Free format text: LAPSE BECAUSE OF NON-PAYMENT OF DUE FEES

Effective date: 20220707

Ref country code: CY

Free format text: LAPSE BECAUSE OF NON-PAYMENT OF DUE FEES

Effective date: 20220707

Ref country code: BE

Free format text: LAPSE BECAUSE OF NON-PAYMENT OF DUE FEES

Effective date: 20220731

PG25 Lapsed in a contracting state [announced via postgrant information from national office to epo]

Ref country code: NL

Free format text: LAPSE BECAUSE OF NON-PAYMENT OF DUE FEES

Effective date: 20220801

PG25 Lapsed in a contracting state [announced via postgrant information from national office to epo]

Ref country code: IT

Free format text: LAPSE BECAUSE OF NON-PAYMENT OF DUE FEES

Effective date: 20220707

Ref country code: IE

Free format text: LAPSE BECAUSE OF NON-PAYMENT OF DUE FEES

Effective date: 20220707

Ref country code: DK

Free format text: LAPSE BECAUSE OF NON-PAYMENT OF DUE FEES

Effective date: 20220731

REG Reference to a national code

Ref country code: SK

Ref legal event code: MM4A

Ref document number: E 7088

Country of ref document: SK

Effective date: 20220707

REG Reference to a national code

Ref country code: ES

Ref legal event code: FD2A

Effective date: 20230825

REG Reference to a national code

Ref country code: EE

Ref legal event code: MM4A

Ref document number: E004255

Country of ref document: EE

Effective date: 20220731

PG25 Lapsed in a contracting state [announced via postgrant information from national office to epo]

Ref country code: PT

Free format text: LAPSE BECAUSE OF EXPIRATION OF PROTECTION

Effective date: 20230718

Ref country code: ES

Free format text: LAPSE BECAUSE OF NON-PAYMENT OF DUE FEES

Effective date: 20220708

Ref country code: EE

Free format text: LAPSE BECAUSE OF NON-PAYMENT OF DUE FEES

Effective date: 20220731

PG25 Lapsed in a contracting state [announced via postgrant information from national office to epo]

Ref country code: SK

Free format text: LAPSE BECAUSE OF NON-PAYMENT OF DUE FEES

Effective date: 20220707