EP2441072B1 - Audioverarbeitung - Google Patents

Audioverarbeitung Download PDF

Info

Publication number
EP2441072B1
EP2441072B1 EP09779676.7A EP09779676A EP2441072B1 EP 2441072 B1 EP2441072 B1 EP 2441072B1 EP 09779676 A EP09779676 A EP 09779676A EP 2441072 B1 EP2441072 B1 EP 2441072B1
Authority
EP
European Patent Office
Prior art keywords
audio
audio channel
channel
time difference
acoustic space
Prior art date
Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
Active
Application number
EP09779676.7A
Other languages
English (en)
French (fr)
Other versions
EP2441072A1 (de
Inventor
Jussi Virolainen
Jussi Mutanen
Kai Samposalo
Current Assignee (The listed assignees may be inaccurate. Google has not performed a legal analysis and makes no representation or warranty as to the accuracy of the list.)
Nokia Technologies Oy
Original Assignee
Nokia Technologies Oy
Priority date (The priority date is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the date listed.)
Filing date
Publication date
Application filed by Nokia Technologies Oy filed Critical Nokia Technologies Oy
Publication of EP2441072A1 publication Critical patent/EP2441072A1/de
Application granted granted Critical
Publication of EP2441072B1 publication Critical patent/EP2441072B1/de
Active legal-status Critical Current
Anticipated expiration legal-status Critical

Links

Images

Classifications

    • G—PHYSICS
    • G10—MUSICAL INSTRUMENTS; ACOUSTICS
    • G10L—SPEECH ANALYSIS TECHNIQUES OR SPEECH SYNTHESIS; SPEECH RECOGNITION; SPEECH OR VOICE PROCESSING TECHNIQUES; SPEECH OR AUDIO CODING OR DECODING
    • G10L19/00—Speech or audio signals analysis-synthesis techniques for redundancy reduction, e.g. in vocoders; Coding or decoding of speech or audio signals, using source filter models or psychoacoustic analysis
    • G10L19/008—Multichannel audio signal coding or decoding using interchannel correlation to reduce redundancy, e.g. joint-stereo, intensity-coding or matrixing
    • G—PHYSICS
    • G10—MUSICAL INSTRUMENTS; ACOUSTICS
    • G10L—SPEECH ANALYSIS TECHNIQUES OR SPEECH SYNTHESIS; SPEECH RECOGNITION; SPEECH OR VOICE PROCESSING TECHNIQUES; SPEECH OR AUDIO CODING OR DECODING
    • G10L25/00—Speech or voice analysis techniques not restricted to a single one of groups G10L15/00 - G10L21/00
    • G10L25/03—Speech or voice analysis techniques not restricted to a single one of groups G10L15/00 - G10L21/00 characterised by the type of extracted parameters
    • G10L25/06—Speech or voice analysis techniques not restricted to a single one of groups G10L15/00 - G10L21/00 characterised by the type of extracted parameters the extracted parameters being correlation coefficients
    • H—ELECTRICITY
    • H04—ELECTRIC COMMUNICATION TECHNIQUE
    • H04M—TELEPHONIC COMMUNICATION
    • H04M3/00—Automatic or semi-automatic exchanges
    • H04M3/42—Systems providing special services or facilities to subscribers
    • H04M3/56—Arrangements for connecting several subscribers to a common circuit, i.e. affording conference facilities
    • H04M3/568—Arrangements for connecting several subscribers to a common circuit, i.e. affording conference facilities audio processing specific to telephonic conferencing, e.g. spatial distribution, mixing of participants
    • H—ELECTRICITY
    • H04—ELECTRIC COMMUNICATION TECHNIQUE
    • H04S—STEREOPHONIC SYSTEMS 
    • H04S7/00—Indicating arrangements; Control arrangements, e.g. balance control
    • H04S7/30—Control circuits for electronic adaptation of the sound field

Definitions

  • Embodiments of the present invention relate to audio processing.
  • they relate to differential processing of audio channels that share a remote acoustic space.
  • An acoustic space may comprise a plurality of locations for audio capture. Audio captured at one location may also be captured at another location in the acoustic space resulting in correlated audio channels. The correlation between the audio channels should be respected if the quality of the captured audio is to be maintained when processed at a remote location.
  • US patent no. 4,890,065 A discloses a time delay corrector that corrects the relative time delay or phase error between two signals, for example, stereo audio signals.
  • a window of no correction is established between two threshold levels. So long as a signal representative of the relative phase error on a relatively fast integration basis does not logically exceed the thresholds which define this window, no rapid phase error or time delay correction is accomplished.
  • the window of zero correction thereby prevents any time delay or phase error corrections which might otherwise result from the normal phase fluctuations inherent in the two correlated stereo signals, thereby preserving the stereo imaging and information content of those signals.
  • detection signals logically exceed the thresholds of the window and a phase correction is rapidly attained.
  • a method comprising: determining a time difference between audio data of at least a first audio channel and a second audio channel of the same acoustic space, wherein the time difference is determined by correlating audio data of the first audio channel as a consequence of an event in the acoustic space against audio data of the second audio channel as a consequence of the event in the acoustic space; and enabling a corrective time shift between the first audio channel and the second audio channel when the time difference exceeds a threshold, wherein the corrective time shift selectively introduces a segment of additional audio to an audio channel or discards a segment of audio from an audio channel.
  • an apparatus comprising: means for determining a time difference between audio data of at least a first audio channel and a second audio channel of the same acoustic space, wherein the time difference is determined by correlating audio data of the first audio channel as a consequence of an event in the acoustic space against audio data of the second audio channel as a consequence of the event in the acoustic space; and means for enabling a corrective time shift between the first audio channel and the second audio channel when the time difference exceeds a threshold, wherein the corrective time shift selectively introduces a segment of additional audio to an audio channel or discards a segment of audio from an audio channel.
  • a computer program product embodied in a computer-readable storage medium, comprising computer program instructions for causing performance of the actions according to the method described in the foregoing.
  • An acoustic space 2 comprises a plurality of locations for audio capture wherein the characteristics of the space, the audio in the space and the locations for audio capture result in the audio input from the plurality of locations being correlated. That is the same audio is captured at different locations within an acoustic space 2.
  • the server 20 may provide via separate downlink audio channels 12 audio which may, for example, have been captured by another client apparatus remote from the acoustic space 2.
  • the audio is received from the server 20 at the client apparatuses 4 which reproduce using their audio output devices 8 the audio from the remote client apparatus.
  • the path to each different source of audio may have a different delay caused for example by different properties of the downlink audio channels 12. As a consequence the sources would without intervention produce the same audio output at different times which could result in a poor experience for a user within the acoustic space 2 as the user would hear the same audio from different locations within the acoustic space 2 but at different times.
  • the server 20 may be configured to control compensation for the different delays in the paths to the different sources of audio in the acoustic space 2. The compensation may, for example, occur at the server 20 for downlink audio channels 12 or the compensation may occur at the client apparatuses 4.
  • the server 20 may receive via separate uplink audio channels 10 audio captured by the separate audio input devices 6 of the client apparatus 4 of the acoustic space 2. There may therefore apparently be several sources of the same audio from the acoustic space 2. The path from each different source of audio to the server 20 may have a different delay caused for example by different properties of the uplink audio channels 10. As a consequence the sources would without intervention provide the same audio output at different times to the server 20 for further processing and, for example, delivery to the remote client apparatus. This could result in a poor experience for a user at the remote client apparatus.
  • the server 20 may be configured to control compensation for the different delays in the paths to the different sources of audio from the acoustic space 2. The compensation or alignment correction may, for example, occur at the server 20 for uplink audio channels 12 or the compensation may occur at the client apparatuses 4.
  • the client apparatuses 4 comprise an audio output device 8 such as a loudspeaker.
  • the audio 7 is broadcast by the audio output devices 8 1 , 8 2 , 8 3 of the respective client apparatuses 4 1 , 4 2 , 4 3 within the acoustic space 2.
  • Each of the respective client apparatus 4 1 , 4 2 , 4 3 receives audio data, used for creating audio 7, from the server 20 via respective downlink audio channels 12 1 , 12 2 , 12 3 .
  • the downlink audio channels are communication channels that provide audio data. They may for example provide the audio data from the server 20 via a packet switched network which may, for example, include a cellular radio communications network, or for example via Wireless Local Area Network (WLAN) or Bluetooth.
  • WLAN Wireless Local Area Network
  • the client apparatuses 4 1 , 4 2 , 4 3 each comprise a respective audio downlink buffer 14 1 , 14 2 , 14 3 associated with the respective downlink audio channel.
  • a first audio downlink buffer 14 1 stores the audio data received via the first downlink audio channel 12 1 .
  • a second audio downlink buffer 14 2 stores the audio data received via the second downlink audio channel 12 2 .
  • a third audio downlink buffer 14 3 stores the audio data received via the third downlink audio channel 12 3 .
  • the client apparatuses 4 comprise an audio input device 6 such as a microphone.
  • the audio 3 from the audio source 5 is captured by the audio input devices 6 1 , 6 2 , 6 3 of the respective client apparatuses 4 1 , 4 2 , 4 3 within the acoustic space 2.
  • Each of the respective client apparatus 4 1 , 4 2 , 4 3 provide the audio captured by their audio input devices to the server 20 via respective uplink audio channels 10 1 , 10 2 , 10 3 .
  • the uplink audio channels are communication channels that provide audio data. They may for example provide the audio data to the server 20 via a packet switched network which may, for example, include a cellular radio communications network.
  • the server apparatus 20 comprises a plurality of audio upink buffers 24 1 , 24 2 , 24 3 associated with the respective uplink audio channels.
  • a first audio uplink buffer 24 1 stores the audio data received via the first uplink audio channel 10 1 before it is processed by processing circuitry 22.
  • a second audio uplink buffer 24 2 stores the audio data received via the second uplink audio channel 10 2 before it is processed by processing circuitry 22.
  • a third audio uplink buffer 24 3 stores the audio data received via the third uplink audio channel 10 3 before it is processed by processing circuitry 22.
  • the processing circuitry 22 processes the audio data received from the buffers 24 and provides as a result of the processing output audio data 26 that is transmitted to another remote client apparatus.
  • Fig. 4 schematically illustrates an alignment correction process 40 which may be performed by the server 20.
  • a time difference between at least a first uplink audio channel and a second audio uplink channel of the same acoustic space 2 is determined.
  • a corrective time shift between the first uplink audio channel and the second uplink audio channel is enabled when the time difference exceeds a threshold.
  • the corrective time shift may be a discrete predetermined unit of time.
  • the corrective time shift may be achieved by introducing a segment of additional audio to an audio channel or discarding a segment of audio from an audio channel.
  • the segment of audio may be an audio frame such as a silent frame.
  • the server 20 may enable application of a forward time shift to the second audio channel by adding audio data to the second audio channel when the time difference exceeds a lag threshold.
  • the lag threshold may have a value equivalent to at least half the duration of the audio data added so that adding data reduces the magnitude of the time difference.
  • the audio data added may be a frame, for example, a silent frame and the threshold may be half a frame duration.
  • the server 20 may enable application of a backward time shift to the second audio channel by discarding audio data of the second audio channel when the time difference exceeds a lead threshold.
  • the second lead threshold has a value equivalent to at least half the duration of the audio data discarded so that adding data reduces the magnitude of the time difference.
  • the audio data added discarded be a frame, for example, a silent frame and the threshold may be half a frame duration.
  • the audio uplink buffers 24 are illustrated in Figure 2 as being located in the server apparatus 20, the purpose of the audio uplink buffers is to store the audio data before combined processing. Therefore, in other implementations the audio uplink buffers 24 may reside in the client apparatuses 4.
  • the use of audio uplink buffers 24 in the client apparatuses 4 may be in addition to or as an alternative to using audio uplink buffers in the server apparatus 20.
  • the corrective time shift between the first uplink audio channel and the second uplink audio channel is enabled at remote audio uplink buffers in the client apparatuses 4 by sending a command from the server apparatus 20 to the client apparatuses 4.
  • the server 20 after determining that a corrective shift is necessary and enabling the corrective shift at block 44, can send a command 45 to a client apparatus 4.
  • the client apparatus 4 then makes the corrective shift itself as illustrated by block 48.
  • the corrective shift may, for example, be achieved by selectively adding data to or discarding data from the audio downlink buffers 14 and/or uplink buffers 24 if present in the clients.
  • the data may be an audio frame such as a silent frame.
  • the corrective shift is sized to reduce the time difference between the first uplink audio channel and a second audio uplink channel to below the threshold.
  • the audio downlink buffers 14 are illustrated in Figure 3 as being located in the client apparatuses 4, the purpose of the audio downlink buffers 14 is to store the audio data before rendering. Therefore, in other implementations the audio downlink buffers 14 may reside in the server apparatus 20. The use of audio downlink buffers 14 in the server apparatus 20 may be in addition to or as an alternative to using audio downlink buffers in the client apparatuses 4. Referring to Figure 5B , at block 44, the corrective time shift between the first uplink audio channel and the second uplink audio channel may then be enabled at local audio downlink buffers 14 in the server client apparatus 20.
  • the time difference may be determined by correlating audio data of the first audio channel as a consequence of an event in the acoustic space 2 against audio data of the second audio channel as a consequence of the event in the acoustic space 2 to find the time difference corresponding to a maximum value of correlation function between audio signals of the first and second audio channels.
  • a noise event in the acoustic space 2 should be captured by all the input devices 6 in the acoustic space 2 and appear as a common characteristic in the uplink audio channels 10 from the acoustic space.
  • the characteristic may be identified in one audio channel, for example, the reference audio channel and then correlated against the remaining plurality of input audio channels 10 from the acoustic space 2 to determine the time shifts between the remaining plurality of audio channels and the reference channel.
  • the audio may be transmitted in the input audio channel 10 as a pulse code modulated (PCM) audio signal and the characteristic identified by correlation is then a characteristic of the PCM signal.
  • the audio may be encoded at the client apparatus 4 to include one or more time variable parameters.
  • the characteristic identified by correlation may be a characteristic of such a time variable parameter.
  • An example of a parameter is a Voice Activity Detection (VAD) decision (indicating that respective signal segment is either active or inactive speech) or a pitch lag value.
  • VAD Voice Activity Detection
  • the server 20 may, in some embodiments, trigger an audio output event within the acoustic space 2 for the purpose determining the time difference. For example, it may send an audio beacon to the client apparatuses 4 of the acoustic space 2 for output in the acoustic space 2.
  • the audio beacon may be detectable by signal processing but inaudible to a normal human.
  • Fig. 4, 5A and 5B have been described as processes that determine a time difference between a first audio channel and a second audio channel of the same acoustic space 2.
  • the first audio channel may be designated as a reference channel for the acoustic space 2.
  • the processes of Figs 4, 5A and 5B may then be used to determine a time difference between the reference audio channel and a respective one of the remaining plurality of audio channels for the acoustic space 2.
  • the processes of Figs 4, 5A and 5B may then be used to enable a corrective time shift of the remaining plurality of audio channels relative to the reference audio channel.
  • the reference audio channel may be fixed so that it is always the first audio channel.
  • the reference channel may be any of the plurality of audio channels and the identity of the reference channel may vary over time.
  • the reference audio channel may be selected so that the distribution of time differences for an acoustic space are distributed fairly evenly as lags and leads about the reference audio channel. That is, the reference channel is chosen as the audio channel that suffers the median or near the median delay of the plurality of audio channels.
  • the reference may also be defined for example as a channel that has the highest average energy in a predetermined time window or a channel that has signal-to-noise ratio (SNR) meeting predetermined criteria.
  • the reference channel selection logic may also take into account the reference channel selected for one or more preceding segments of audio signal in order to avoid frequently changing the reference signal.
  • a reference input audio channel 10 is defined for the acoustic space 2.
  • the audio uplink buffers 24 for each of the different input audio channels 10 associated with the acoustic space 2 are accessed to determine their fill rates.
  • Fill rate is a rate of speed at which a buffer receives audio data.
  • buffer fill levels indicating the duration of audio signal stored in respective buffers may be determined.
  • the process moves to block 58.
  • the process at block 54 may use the evaluation of buffer fill levels buffer fill rates in order to decide on the next step. In this case, if the fill level for the buffer 24 s associated with the subject input audio channel 10 s is greater than the fill level for the buffer 24 r associated with the reference audio channel 10 r , the process moves to block 56. If the fill level for the buffer 24 s associated with the subject input audio channel 10 s is less than the fill level for the buffer 24 r associated with the reference audio channel 10 r , the process moves to block 58.
  • a backward time shift is applied to the subject audio channel 10 s at block 60 by discarding audio data from the audio uplink buffer 24 s of the subject audio channel 10s.
  • the lead threshold has a value equivalent to at least half the duration of the audio data discarded so that removing data reduces the magnitude of the time difference.
  • the audio data discarded may be a frame, for example, a silent frame and the threshold may be half a frame duration.
  • a forward time shift is applied to the subject audio channel 10 s at block 62 by adding audio data to the audio uplink buffer 24 s of the subject audio channel if there is sufficient room in the audio uplink buffer 24 s associated with the subject audio channel 10 s .
  • a flag may be set for that buffer 24 s .
  • the forward time shift is then delayed until there is sufficient room in the audio uplink buffer 24 s for the extra audio data to be added.
  • the lag threshold (L) may have a value equivalent to at least half the duration of the audio data added so that adding data reduces the magnitude of the time difference.
  • the audio data added may be a frame, for example, a silent frame, a copy of a preceding or a following frame, or a frame derived based on at least one preceding frame and/or at least one following frame, and the threshold may be half a frame duration.
  • a flag may be set for that buffer. The backward time shift is then delayed until there is sufficient data in the audio uplink buffer 24 s for the audio data to be discarded.
  • the lead threshold (K2) has a value equivalent to at least half the duration of the audio data discarded so that removing data reduces the magnitude of the time difference.
  • the audio data discarded may be a frame, for example, a silent frame, a copy of a preceding or a following frame, or a frame derived based on at least one preceding frame and/or at least one following frame, and the threshold may be half a frame duration.
  • a forward time shift is applied to the subject audio channel 10 s at block 66 by adding audio data to the audio uplink buffer 24 s of the subject audio channel 10 s .
  • the lag threshold (L2) may have a value equivalent to at least half the duration of the audio data added so that adding data reduces the magnitude of the time difference.
  • the audio data added may be a frame, for example, a silent frame, a copy of a preceding or a following frame, or a frame derived based on at least one preceding frame and/or at least one following frame, and the threshold may be half a frame duration.
  • the addition of data to a buffer or the removal of data from a buffer may be delayed until a predetermined condition is satisfied.
  • a predetermined condition is the availability of data for removal and the availability of space in the buffer for the addition of data.
  • An alternative or additional condition may relate to audio quality. That is audio data is added or removed at moments when there is less impact on the quality of the audio. For example, audio data may be added or removed during silence or unvoiced speech, or a data segment substantially corresponding to current pitch lag (i.e. pitch cycle length) may be removed or repeated.
  • the method 40 may also be augmented by enabling suppression of the subject audio channel when the time difference between the reference audio channel and the subject audio channel exceeds a suppression threshold.
  • the suppression threshold may be a size that indicates that alignment correction by forward or backward shifting the subject audio channel is not possible, which may happen for example when audio buffer size is small compared to the time difference between the channels.
  • the time difference exceeds the suppression threshold or thresholds (as there may be different thresholds for lag and lead) the subject audio channel is suppressed. This may, for example, be achieved by setting the gain for that channel at or close to zero so that no data is used for that channel.
  • the time difference between the signals has significant effect on the perceived quality of the mixed signal.
  • the listener may perceive the delayed version of the signal as an echo. Echo disturbance increases when the delay increases and when the level of the delayed signal increases.
  • Time difference dependent level correction for the signals may be applied before they are mixed to preserve signal quality in cases when the delay cannot be compensated.
  • the delay increases, the delayed signal is attenuated more to preserve the quality of the mixed signal. Note that in this approach, suppression of a signal may take place if the time difference is large enough to cause the attenuation to be increased to a level that substantially corresponds to suppression of a signal.
  • a set of one or more predetermined thresholds may be used together with a respective set of attenuation factors, and when the time difference exceeds a first threshold (but does not exceed a second threshold having the closest value to that of the first threshold among the set of thresholds), the respective attenuation factor is selected.
  • the delay dependent attenuation may be adapted according to acoustics of the environment where the signals are recorded. When the acoustics is dry the delay may be more disturbing requiring more attenuation than when the acoustics are more echoic when same delay might not be perceived as disturbing.
  • Fig 7 schematically illustrates a method for controlling the alignment correction process 40 when there are a plurality N of subject audio channels.
  • the alignment correction process 40 should be carried out for the N-1 subject audio channels using the same reference audio channel 10 r .
  • a free-running clock or timer is compared against a trigger value t. If the clock has reached the trigger value, the process moves to block 74. At block 74 the clock is re-set to zero and then the process moves onto the alignment correction process 40.
  • the process moves to block 76 where the trigger t is decreased or reset.
  • the process moves to block 76 where the trigger t is increased.
  • This process 70 ensures that the alignment correction process occurs intermittently over a continuous period.
  • the intermission period controlled by the threshold t may vary. It may for example be initially 5 seconds but may decrease to, for example, 0.5 seconds or increase to 60 seconds.
  • the process 70 is designed to adapt the threshold t so that it decreases whenever alignment correction of the channels is more likely to be necessary and increases whenever alignment correction of the channels is less likely to be necessary.
  • Fig 8 schematically illustrates an apparatus 80 suitable for performing the alignment correction process 40.
  • the apparatus 80 is a server apparatus 20.
  • it may be a module for a server 20.
  • module' refers to a unit or apparatus that excludes certain parts/components that would be added by an end manufacturer or a user.
  • Implementation of the apparatus 80 may be in hardware alone (a circuit, a processor%), have certain aspects in software including firmware alone or can be a combination of hardware and software (including firmware).
  • the apparatus 80 comprises a plurality of audio uplink buffers 24 each of which is associated with an input audio channel 10, a processor 82 and a memory 84.
  • the buffers 24 may be implemented using memory 84.
  • the components are operationally coupled and any number or combination of intervening elements can exist (including no intervening elements)
  • the processor 82 is configured to read from and write to the memory 84.
  • the processor 82 may also comprise an output interface via which data and/or commands are output by the processor and an input interface via which data and/or commands are input to the processor 82.
  • the memory 84 stores a computer program 86 comprising computer program instructions that control the operation of the apparatus 80 when loaded into the processor 82.
  • the computer program instructions 86 provide the logic and routines that enables the apparatus 80 to perform the methods illustrated in Figs 4 to 7 .
  • the processor 82 by reading the memory 84 is able to load and execute the computer program 86.
  • the computer program instructions provide: computer readable program means for determining a time difference between at least a first audio channel and a second audio channel of the same acoustic space; and computer readable program means for enabling a corrective time shift between the first audio channel and the second audio channel when the time difference exceeds a threshold.
  • the computer program may arrive at the apparatus 80 via any suitable delivery mechanism 88.
  • the delivery mechanism 88 may be, for example, a computer-readable storage medium, a computer program product, a memory device, a record medium such as a CD-ROM or DVD, an article of manufacture that tangibly embodies the computer program 86.
  • the delivery mechanism may be a signal configured to reliably transfer the computer program 86.
  • the apparatus 80 may propagate or transmit the computer program 86 as a computer data signal.
  • memory 84 is illustrated as a single component it may be implemented as one or more separate components some or all of which may be integrated/removable and/or may provide permanent/semi-permanent/ dynamic/cached storage.
  • references to 'computer-readable storage medium', 'computer program product', 'tangibly embodied computer program' etc. or a 'controller', 'computer', 'processor' etc. should be understood to encompass not only computers having different architectures such as single /multi- processor architectures and sequential (Von Neumann)/parallel architectures but also specialized circuits such as field-programmable gate arrays (FPGA), application specific circuits (ASIC), signal processing devices and other devices.
  • References to computer program, instructions, code etc. should be understood to encompass software for a programmable processor or firmware such as, for example, the programmable content of a hardware device whether instructions for a processor, or configuration settings for a fixed-function device, gate array or programmable logic device etc.
  • Fig 9 schematically illustrates functional components of the apparatus 80.
  • the functional components include the audio uplink buffers, processing circuitry 90 configured to determine a time difference between at least a first audio channel and a second audio channel of the same acoustic space; comparison circuitry 92 configured to compare the determined time difference with a threshold; control circuitry 94 configured to apply a corrective time shift between the first audio channel and the second audio channel when the comparison circuitry determines that the time difference exceeds a threshold.
  • the apparatus 80 may perform some or all of the methods illustrated in Figs 4 to 7 .
  • the first acoustic space 2 1 is remote from the server 20 and comprises three distinct client apparatuses 4 1 , 4 2 , 4 3 .
  • the client apparatuses 4 1 , 4 2 , 4 3 have audio input devices and audio output devices defining audio channels for the first acoustic space 2 1 .
  • the client apparatuses 4 1 , 4 2 , 4 3 form a distributed system in which each client apparatus 4 1 , 4 2 , 4 3 separately connects to the server 20.
  • the second acoustic space 2 2 is remote from the server 20 and the first acoustic space 2 1 and comprises four distinct client apparatuses 4 4 , 4 5 , 4 6 , 4 7 .
  • the client apparatuses 4 4 , 4 5 , 4 6 , 4 7 have audio input devices and audio output devices defining audio channels for the second acoustic space 2 2 .
  • the client apparatus 4 4 , 4 5 , 4 6 , 4 7 form a distributed system in which one of the client devices 4 4 is Master and the other client apparatuses 4 5 , 4 6 , 4 7 are Slaves.
  • the Slaves 4 5 , 4 6 , 4 7 communicate with the Master 4 4 and the Master 4 4 communicates with the server 20.
  • Master 4 4 may perform alignment correction for Slaves 4 5 , 4 6 and 4 7 .
  • Master 4 4 may just forward these signals to server 20, which may perform alignment correction to these signals.
  • the server 20 may perform the alignment correction process 40 for the audio channels associated with the first acoustic space and/or perform the alignment correction process 40 for the audio channels associated with the second acoustic space.
  • the blocks illustrated in the Figs 4 to 7 may represent steps in a method and/or sections of code in the computer program 86.
  • the illustration of a particular order to the blocks does not necessarily imply that there is a required or preferred order for the blocks and the order and arrangement of the block may be varied. Furthermore, it may be possible for some steps to be omitted.

Landscapes

  • Engineering & Computer Science (AREA)
  • Physics & Mathematics (AREA)
  • Mathematical Physics (AREA)
  • Computational Linguistics (AREA)
  • Signal Processing (AREA)
  • Health & Medical Sciences (AREA)
  • Audiology, Speech & Language Pathology (AREA)
  • Human Computer Interaction (AREA)
  • Acoustics & Sound (AREA)
  • Multimedia (AREA)
  • Telephonic Communication Services (AREA)
  • Stereophonic System (AREA)

Claims (12)

  1. Verfahren in einem Server (20) zum Verarbeiten von Audiodaten in einer Vielzahl von Uplink-Audiokanälen (10), die an einer jeweiligen Vielzahl von Clientgeräten (4) an unterschiedlichen Orten in einem akustischen Raum (2) erfasst werden, zur Übermittlung an ein entferntes Clientgerät, wobei das Verfahren Folgendes umfasst:
    Ermitteln einer Zeitdifferenz zwischen Audiodaten mindestens eines ersten Audiokanals und eines zweiten Audiokanals desselben akustischen Raums (2), wobei die Zeitdifferenz durch Korrelieren von Audiodaten des ersten Audiokanals als Folge eines Ereignisses im akustischen Raum (2) gegen Audiodaten des zweiten Audiokanals als Folge des Ereignisses im akustischen Raum (2) ermittelt wird; und
    Ermöglichen einer korrigierenden Zeitversetzung zwischen dem ersten Audiokanal und dem zweiten Audiokanal, wenn die Zeitdifferenz einen Schwellenwert überschreitet,
    dadurch gekennzeichnet, dass die korrigierende Zeitversetzung selektiv ein Segment von zusätzlichem Ton in einen Audiokanal in einem Uplink-Audiopuffer einführt oder ein Segment von Ton aus einem Audiokanal im Uplink-Audiopuffer aussondert, wobei die korrigierende Zeitversetzung so groß ist, dass die Zeitdifferenz zwischen dem ersten Audiokanal und dem zweiten Audiokanal unter den Schwellenwert reduziert wird.
  2. Verfahren nach Anspruch 1, wobei die Audiodaten ein Audiosignal oder einen Parameter eines codierten Audiosignals umfassen.
  3. Verfahren nach Anspruch 1 oder 2, wobei das Ereignis ein ausgelöstes Audioausgabeereignis innerhalb des akustischen Raums zum Zweck des Ermittelns der Zeitdifferenz ist.
  4. Verfahren nach einem der vorhergehenden Ansprüche, wobei das Ermitteln einer Zeitdifferenz zwischen mindestens einem ersten Audiokanal und einem zweiten Audiokanal desselben akustischen Raums intermittierend pro erste Zeitspanne erfolgt.
  5. Verfahren nach einem der vorhergehenden Ansprüche, das Ermitteln einer Zeitdifferenz zwischen anderen Audiokanälen als dem ersten Audiokanal und dem zweiten Audiokanal zu einer Zeit, die eine andere ist als diejenige, wenn die Zeitdifferenz zwischen dem ersten Audiokanal und dem zweiten Audiokanal ermittelt wird, umfasst.
  6. Verfahren nach einem der vorhergehenden Ansprüche, wobei das Ermöglichen der korrigierenden Zeitversetzung Setzen eines Flags, das das Implementieren der korrigierenden Zeitversetzung so lange verzögert, bis eine zusätzliche, vorher ermittelte Bedingung erfüllt ist, einschließt.
  7. Verfahren nach einem der vorhergehenden Ansprüche, wobei die Zeitdifferenz anzeigt, ob der zweite Audiokanal dem ersten Audiokanal nach- oder voreilt, wobei das Verfahren Folgendes umfasst:
    Ermöglichen der Anwendung einer Zeitversetzung nach vorne zum zweiten Audiokanal durch Hinzufügen von Audiodaten zum zweiten Audiokanal, wenn die Zeitdifferenz einen ersten Lag-Schwellenwert überschreitet;
    Ermöglichen der Anwendung einer Zeitversetzung nach hinten zum zweiten Audiokanal durch Aussondern von Audiodaten des zweiten Audiokanals, wenn die Zeitdifferenz einen ersten Lead-Schwellenwert überschreitet.
  8. Verfahren nach Anspruch 7, wobei der erste Lag-Schwellenwert einen Wert, der gleich mindestens der halben Dauer der hinzugefügten Audiodaten ist, aufweist und wobei der zweite Lead-Schwellenwert einen Wert, der gleich mindestens der halben Dauer der ausgesonderten Audiodaten ist, aufweist.
  9. Server-Vorrichtung (20), die dazu ausgelegt ist, Audiodaten in einer Vielzahl von Uplink-Audiokanälen (10), die an einer jeweiligen Vielzahl von Clientgeräten (4) an unterschiedlichen Orten in einem akustischen Raum (2) erfasst werden, zur Übermittlung an ein entferntes Clientgerät zu verarbeiten, wobei die Server-Vorrichtung (20) Folgendes umfasst:
    Mittel zum Ermitteln einer Zeitdifferenz zwischen Audiodaten mindestens eines ersten Audiokanals und eines zweiten Audiokanals desselben akustischen Raums (2), wobei die Zeitdifferenz durch Korrelieren von Audiodaten des ersten Audiokanals als Folge eines Ereignisses im akustischen Raum (2) gegen Audiodaten des zweiten Audiokanals als Folge des Ereignisses im akustischen Raum (2) ermittelt wird; und
    Mittel zum Ermöglichen einer korrigierenden Zeitversetzung zwischen dem ersten Audiokanal und dem zweiten Audiokanal, wenn die Zeitdifferenz einen Schwellenwert überschreitet,
    dadurch gekennzeichnet, dass die korrigierende Zeitversetzung selektiv ein Segment von zusätzlichem Ton in einen Audiokanal in einem Uplink-Audiopuffer einführt oder ein Segment von Ton aus einem Audiokanal im Uplink-Audiopuffer aussondert, wobei die korrigierende Zeitversetzung so groß ist, dass die Zeitdifferenz zwischen dem ersten Audiokanal und dem zweiten Audiokanal unter den Schwellenwert reduziert wird.
  10. Vorrichtung nach Anspruch 9, wobei der erste Kanal ein Referenzaudiokanal ist, der zum Ermitteln der Zeitdifferenz zwischen dem Referenzaudiokanal und einer Vielzahl von Audiokanälen verwendet wird.
  11. Vorrichtung nach Anspruch 9 oder 10, die Mittel zum Durchführen einer korrigierenden Zeitversetzung zwischen dem ersten Audiokanal und dem zweiten Audiokanal, wenn die Zeitdifferenz einen Schwellenwert überschreitet, umfasst.
  12. Computerprogrammprodukt, das in einem computerlesbaren Speichermedium enthalten ist und Computerprogrammbefehle zum Bewirken der Durchführung der Vorgänge nach einem der Ansprüche 1 bis 8 umfasst.
EP09779676.7A 2009-06-08 2009-06-08 Audioverarbeitung Active EP2441072B1 (de)

Applications Claiming Priority (1)

Application Number Priority Date Filing Date Title
PCT/EP2009/057050 WO2010142320A1 (en) 2009-06-08 2009-06-08 Audio processing

Publications (2)

Publication Number Publication Date
EP2441072A1 EP2441072A1 (de) 2012-04-18
EP2441072B1 true EP2441072B1 (de) 2019-02-20

Family

ID=42008501

Family Applications (1)

Application Number Title Priority Date Filing Date
EP09779676.7A Active EP2441072B1 (de) 2009-06-08 2009-06-08 Audioverarbeitung

Country Status (3)

Country Link
US (1) US9008321B2 (de)
EP (1) EP2441072B1 (de)
WO (1) WO2010142320A1 (de)

Families Citing this family (14)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
US9025780B2 (en) * 2009-08-14 2015-05-05 Koninklijke Kpn N.V. Method and system for determining a perceived quality of an audio system
WO2013064860A1 (en) * 2011-10-31 2013-05-10 Nokia Corporation Audio scene rendering by aligning series of time-varying feature data
US9282420B2 (en) * 2012-06-25 2016-03-08 Calgary Scientific Inc. Method and system for multi-channel mixing for transmission of audio over a network
WO2014072772A1 (en) * 2012-11-12 2014-05-15 Nokia Corporation A shared audio scene apparatus
US8914007B2 (en) * 2013-02-27 2014-12-16 Nokia Corporation Method and apparatus for voice conferencing
US9503491B2 (en) * 2013-03-15 2016-11-22 Echostar Technologies L.L.C. Playback stall avoidance in adaptive media streaming
JP2015222847A (ja) * 2014-05-22 2015-12-10 富士通株式会社 音声処理装置、音声処理方法および音声処理プログラム
CN106033671B (zh) * 2015-03-09 2020-11-06 华为技术有限公司 确定声道间时间差参数的方法和装置
JP6981409B2 (ja) * 2016-06-22 2021-12-15 日本電気株式会社 処理装置、信号処理システム、処理方法、およびプログラム
GB2566978A (en) 2017-09-29 2019-04-03 Nokia Technologies Oy Processing audio signals
GB2568940A (en) * 2017-12-01 2019-06-05 Nokia Technologies Oy Processing audio signals
US10743107B1 (en) * 2019-04-30 2020-08-11 Microsoft Technology Licensing, Llc Synchronization of audio signals from distributed devices
JP7404664B2 (ja) * 2019-06-07 2023-12-26 ヤマハ株式会社 音声処理装置及び音声処理方法
US20230232178A1 (en) * 2022-01-19 2023-07-20 Meta Platforms Technologies, Llc Modifying audio data transmitted to a receiving device to account for acoustic parameters of a user of the receiving device

Family Cites Families (13)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
US4890065A (en) * 1987-03-26 1989-12-26 Howe Technologies Corporation Relative time delay correction system utilizing window of zero correction
US6130949A (en) * 1996-09-18 2000-10-10 Nippon Telegraph And Telephone Corporation Method and apparatus for separation of source, program recorded medium therefor, method and apparatus for detection of sound source zone, and program recorded medium therefor
US7024575B2 (en) 2002-11-14 2006-04-04 Intel Corporation Apparatus and method for time synchronization of a plurality of multimedia streams
JP4765289B2 (ja) * 2003-12-10 2011-09-07 ソニー株式会社 音響システムにおけるスピーカ装置の配置関係検出方法、音響システム、サーバ装置およびスピーカ装置
US7838620B2 (en) 2005-06-17 2010-11-23 Eastman Chemical Company Thermoformed sheet(s) comprising polyester compositions which comprise cyclobutanediol
EP1897355A1 (de) 2005-06-30 2008-03-12 Nokia Corporation System für eine konferenzverbindung und entsprechende einrichtungen, verfahren und programmprodukte
US20070106498A1 (en) 2005-11-10 2007-05-10 Nokia Corporation Mobile communication terminal and method therefor
US7333865B1 (en) * 2006-01-03 2008-02-19 Yesvideo, Inc. Aligning data streams
US7512629B2 (en) 2006-07-13 2009-03-31 International Business Machines Corporation Consistent and unbiased cardinality estimation for complex queries with conjuncts of predicates
US8218460B2 (en) 2006-12-27 2012-07-10 Laura Laaksonen Network entity, method and computer program product for mixing signals during a conference session
US7973857B2 (en) 2006-12-27 2011-07-05 Nokia Corporation Teleconference group formation using context information
US8767975B2 (en) * 2007-06-21 2014-07-01 Bose Corporation Sound discrimination method and apparatus
US8781137B1 (en) * 2010-04-27 2014-07-15 Audience, Inc. Wind noise detection and suppression

Non-Patent Citations (2)

* Cited by examiner, † Cited by third party
Title
BORONAT F ET AL: "Multimedia group and inter-stream synchronization techniques: A comparative study", INFORMATION SYSTEMS, PERGAMON PRESS, OXFORD, GB, vol. 34, no. 1, 1 March 2009 (2009-03-01), pages 108 - 131, XP025644936, ISSN: 0306-4379, [retrieved on 20080524], DOI: 10.1016/J.IS.2008.05.001 *
STEFAN WEHR IGOR KOZINTSEV RAINER LIENHART WALTER KELLERMANN: "Synchronization of Acoustic Sensors for Distributed Ad-Hoc Audio Networks and its Use for Blind Source Separation", MULTIMEDIA SOFTWARE ENGINEERING, 2004. PROCEEDINGS. IEEE SIXTH INTERNA TIONAL SYMPOSIUM ON MIAMI, FL, USA 13-15 DEC. 2004, PISCATAWAY, NJ, USA,IEEE, 13 December 2004 (2004-12-13), pages 18 - 25, XP010757227, ISBN: 978-0-7695-2217-3 *

Also Published As

Publication number Publication date
US9008321B2 (en) 2015-04-14
EP2441072A1 (de) 2012-04-18
US20120170760A1 (en) 2012-07-05
WO2010142320A1 (en) 2010-12-16

Similar Documents

Publication Publication Date Title
EP2441072B1 (de) Audioverarbeitung
KR101843010B1 (ko) 더킹 제어를 위한 메타데이터
KR100938032B1 (ko) V o I P 용 적응성 디-지터 버퍼
KR101126056B1 (ko) 요해도에 영향을 미치지 않고 문장 내의 토크스퍼트의 재생 타이밍을 변경하는 방법 및 장치
EP3273620B1 (de) Steuerung miteinander verbundener multimediavorrichtungen
EP3504861B1 (de) Audioübertragung mit kompensation der zeitdauer der spracherkennung
TW201707498A (zh) 用於使用個人通信裝置之虛擬會議系統的方法及系統
US10546581B1 (en) Synchronization of inbound and outbound audio in a heterogeneous echo cancellation system
US20180307555A1 (en) Transfer of data with check bits
KR20160075664A (ko) 복수-경로 오디오 프로세싱
US10867615B2 (en) Voice recognition with timing information for noise cancellation
US10290303B2 (en) Audio compensation techniques for network outages
US20120250640A1 (en) Communication device, and communication system
KR101597768B1 (ko) 입체 음향을 이용한 다자간 인터랙티브 통화 시스템 및 방법
US20120009892A1 (en) Wireless communication system and method for adjusting used depth of buffer unit in wireless communication system
CN113055312B (zh) 基于同步以太网的多路音频拾音方法和系统
CN109327724B (zh) 音视频同步播放方法和装置
US20020172352A1 (en) Non-embedded acoustic echo cancellation
US11741933B1 (en) Acoustic signal cancelling
WO2023170677A1 (en) Acoustic signal cancelling
GB2596107A (en) Managing network jitter for multiple audio streams
CN113535115A (zh) 音频播放方法、装置、电子设备及存储介质
JP2014011749A (ja) 音声信号受信装置、揺らぎ吸収バッファ制御方法、音声信号再生システム
JPH02272849A (ja) 音声パケット通信装置

Legal Events

Date Code Title Description
PUAI Public reference made under article 153(3) epc to a published international application that has entered the european phase

Free format text: ORIGINAL CODE: 0009012

17P Request for examination filed

Effective date: 20111111

AK Designated contracting states

Kind code of ref document: A1

Designated state(s): AT BE BG CH CY CZ DE DK EE ES FI FR GB GR HR HU IE IS IT LI LT LU LV MC MK MT NL NO PL PT RO SE SI SK TR

DAX Request for extension of the european patent (deleted)
RAP1 Party data changed (applicant data changed or rights of an application transferred)

Owner name: NOKIA CORPORATION

RAP1 Party data changed (applicant data changed or rights of an application transferred)

Owner name: NOKIA TECHNOLOGIES OY

17Q First examination report despatched

Effective date: 20160728

STAA Information on the status of an ep patent application or granted ep patent

Free format text: STATUS: EXAMINATION IS IN PROGRESS

GRAP Despatch of communication of intention to grant a patent

Free format text: ORIGINAL CODE: EPIDOSNIGR1

STAA Information on the status of an ep patent application or granted ep patent

Free format text: STATUS: GRANT OF PATENT IS INTENDED

INTG Intention to grant announced

Effective date: 20180813

GRAJ Information related to disapproval of communication of intention to grant by the applicant or resumption of examination proceedings by the epo deleted

Free format text: ORIGINAL CODE: EPIDOSDIGR1

STAA Information on the status of an ep patent application or granted ep patent

Free format text: STATUS: EXAMINATION IS IN PROGRESS

GRAR Information related to intention to grant a patent recorded

Free format text: ORIGINAL CODE: EPIDOSNIGR71

GRAS Grant fee paid

Free format text: ORIGINAL CODE: EPIDOSNIGR3

STAA Information on the status of an ep patent application or granted ep patent

Free format text: STATUS: GRANT OF PATENT IS INTENDED

GRAA (expected) grant

Free format text: ORIGINAL CODE: 0009210

STAA Information on the status of an ep patent application or granted ep patent

Free format text: STATUS: THE PATENT HAS BEEN GRANTED

INTC Intention to grant announced (deleted)
AK Designated contracting states

Kind code of ref document: B1

Designated state(s): AT BE BG CH CY CZ DE DK EE ES FI FR GB GR HR HU IE IS IT LI LT LU LV MC MK MT NL NO PL PT RO SE SI SK TR

INTG Intention to grant announced

Effective date: 20190114

REG Reference to a national code

Ref country code: GB

Ref legal event code: FG4D

REG Reference to a national code

Ref country code: CH

Ref legal event code: EP

REG Reference to a national code

Ref country code: DE

Ref legal event code: R096

Ref document number: 602009057074

Country of ref document: DE

REG Reference to a national code

Ref country code: AT

Ref legal event code: REF

Ref document number: 1099244

Country of ref document: AT

Kind code of ref document: T

Effective date: 20190315

REG Reference to a national code

Ref country code: IE

Ref legal event code: FG4D

REG Reference to a national code

Ref country code: LT

Ref legal event code: MG4D

REG Reference to a national code

Ref country code: NL

Ref legal event code: MP

Effective date: 20190220

PG25 Lapsed in a contracting state [announced via postgrant information from national office to epo]

Ref country code: SE

Free format text: LAPSE BECAUSE OF FAILURE TO SUBMIT A TRANSLATION OF THE DESCRIPTION OR TO PAY THE FEE WITHIN THE PRESCRIBED TIME-LIMIT

Effective date: 20190220

Ref country code: LT

Free format text: LAPSE BECAUSE OF FAILURE TO SUBMIT A TRANSLATION OF THE DESCRIPTION OR TO PAY THE FEE WITHIN THE PRESCRIBED TIME-LIMIT

Effective date: 20190220

Ref country code: FI

Free format text: LAPSE BECAUSE OF FAILURE TO SUBMIT A TRANSLATION OF THE DESCRIPTION OR TO PAY THE FEE WITHIN THE PRESCRIBED TIME-LIMIT

Effective date: 20190220

Ref country code: NO

Free format text: LAPSE BECAUSE OF FAILURE TO SUBMIT A TRANSLATION OF THE DESCRIPTION OR TO PAY THE FEE WITHIN THE PRESCRIBED TIME-LIMIT

Effective date: 20190520

Ref country code: PT

Free format text: LAPSE BECAUSE OF FAILURE TO SUBMIT A TRANSLATION OF THE DESCRIPTION OR TO PAY THE FEE WITHIN THE PRESCRIBED TIME-LIMIT

Effective date: 20190620

PG25 Lapsed in a contracting state [announced via postgrant information from national office to epo]

Ref country code: IS

Free format text: LAPSE BECAUSE OF FAILURE TO SUBMIT A TRANSLATION OF THE DESCRIPTION OR TO PAY THE FEE WITHIN THE PRESCRIBED TIME-LIMIT

Effective date: 20190620

Ref country code: BG

Free format text: LAPSE BECAUSE OF FAILURE TO SUBMIT A TRANSLATION OF THE DESCRIPTION OR TO PAY THE FEE WITHIN THE PRESCRIBED TIME-LIMIT

Effective date: 20190520

Ref country code: GR

Free format text: LAPSE BECAUSE OF FAILURE TO SUBMIT A TRANSLATION OF THE DESCRIPTION OR TO PAY THE FEE WITHIN THE PRESCRIBED TIME-LIMIT

Effective date: 20190521

Ref country code: LV

Free format text: LAPSE BECAUSE OF FAILURE TO SUBMIT A TRANSLATION OF THE DESCRIPTION OR TO PAY THE FEE WITHIN THE PRESCRIBED TIME-LIMIT

Effective date: 20190220

Ref country code: NL

Free format text: LAPSE BECAUSE OF FAILURE TO SUBMIT A TRANSLATION OF THE DESCRIPTION OR TO PAY THE FEE WITHIN THE PRESCRIBED TIME-LIMIT

Effective date: 20190220

Ref country code: HR

Free format text: LAPSE BECAUSE OF FAILURE TO SUBMIT A TRANSLATION OF THE DESCRIPTION OR TO PAY THE FEE WITHIN THE PRESCRIBED TIME-LIMIT

Effective date: 20190220

RAP2 Party data changed (patent owner data changed or rights of a patent transferred)

Owner name: NOKIA TECHNOLOGIES OY

REG Reference to a national code

Ref country code: AT

Ref legal event code: MK05

Ref document number: 1099244

Country of ref document: AT

Kind code of ref document: T

Effective date: 20190220

PG25 Lapsed in a contracting state [announced via postgrant information from national office to epo]

Ref country code: CZ

Free format text: LAPSE BECAUSE OF FAILURE TO SUBMIT A TRANSLATION OF THE DESCRIPTION OR TO PAY THE FEE WITHIN THE PRESCRIBED TIME-LIMIT

Effective date: 20190220

Ref country code: RO

Free format text: LAPSE BECAUSE OF FAILURE TO SUBMIT A TRANSLATION OF THE DESCRIPTION OR TO PAY THE FEE WITHIN THE PRESCRIBED TIME-LIMIT

Effective date: 20190220

Ref country code: SK

Free format text: LAPSE BECAUSE OF FAILURE TO SUBMIT A TRANSLATION OF THE DESCRIPTION OR TO PAY THE FEE WITHIN THE PRESCRIBED TIME-LIMIT

Effective date: 20190220

Ref country code: ES

Free format text: LAPSE BECAUSE OF FAILURE TO SUBMIT A TRANSLATION OF THE DESCRIPTION OR TO PAY THE FEE WITHIN THE PRESCRIBED TIME-LIMIT

Effective date: 20190220

Ref country code: EE

Free format text: LAPSE BECAUSE OF FAILURE TO SUBMIT A TRANSLATION OF THE DESCRIPTION OR TO PAY THE FEE WITHIN THE PRESCRIBED TIME-LIMIT

Effective date: 20190220

Ref country code: IT

Free format text: LAPSE BECAUSE OF FAILURE TO SUBMIT A TRANSLATION OF THE DESCRIPTION OR TO PAY THE FEE WITHIN THE PRESCRIBED TIME-LIMIT

Effective date: 20190220

Ref country code: DK

Free format text: LAPSE BECAUSE OF FAILURE TO SUBMIT A TRANSLATION OF THE DESCRIPTION OR TO PAY THE FEE WITHIN THE PRESCRIBED TIME-LIMIT

Effective date: 20190220

REG Reference to a national code

Ref country code: DE

Ref legal event code: R097

Ref document number: 602009057074

Country of ref document: DE

PG25 Lapsed in a contracting state [announced via postgrant information from national office to epo]

Ref country code: PL

Free format text: LAPSE BECAUSE OF FAILURE TO SUBMIT A TRANSLATION OF THE DESCRIPTION OR TO PAY THE FEE WITHIN THE PRESCRIBED TIME-LIMIT

Effective date: 20190220

PLBE No opposition filed within time limit

Free format text: ORIGINAL CODE: 0009261

STAA Information on the status of an ep patent application or granted ep patent

Free format text: STATUS: NO OPPOSITION FILED WITHIN TIME LIMIT

PG25 Lapsed in a contracting state [announced via postgrant information from national office to epo]

Ref country code: AT

Free format text: LAPSE BECAUSE OF FAILURE TO SUBMIT A TRANSLATION OF THE DESCRIPTION OR TO PAY THE FEE WITHIN THE PRESCRIBED TIME-LIMIT

Effective date: 20190220

26N No opposition filed

Effective date: 20191121

PG25 Lapsed in a contracting state [announced via postgrant information from national office to epo]

Ref country code: MC

Free format text: LAPSE BECAUSE OF FAILURE TO SUBMIT A TRANSLATION OF THE DESCRIPTION OR TO PAY THE FEE WITHIN THE PRESCRIBED TIME-LIMIT

Effective date: 20190220

REG Reference to a national code

Ref country code: CH

Ref legal event code: PL

GBPC Gb: european patent ceased through non-payment of renewal fee

Effective date: 20190608

PG25 Lapsed in a contracting state [announced via postgrant information from national office to epo]

Ref country code: SI

Free format text: LAPSE BECAUSE OF FAILURE TO SUBMIT A TRANSLATION OF THE DESCRIPTION OR TO PAY THE FEE WITHIN THE PRESCRIBED TIME-LIMIT

Effective date: 20190220

REG Reference to a national code

Ref country code: BE

Ref legal event code: MM

Effective date: 20190630

PG25 Lapsed in a contracting state [announced via postgrant information from national office to epo]

Ref country code: TR

Free format text: LAPSE BECAUSE OF FAILURE TO SUBMIT A TRANSLATION OF THE DESCRIPTION OR TO PAY THE FEE WITHIN THE PRESCRIBED TIME-LIMIT

Effective date: 20190220

PG25 Lapsed in a contracting state [announced via postgrant information from national office to epo]

Ref country code: GB

Free format text: LAPSE BECAUSE OF NON-PAYMENT OF DUE FEES

Effective date: 20190608

Ref country code: IE

Free format text: LAPSE BECAUSE OF NON-PAYMENT OF DUE FEES

Effective date: 20190608

PG25 Lapsed in a contracting state [announced via postgrant information from national office to epo]

Ref country code: LU

Free format text: LAPSE BECAUSE OF NON-PAYMENT OF DUE FEES

Effective date: 20190608

Ref country code: CH

Free format text: LAPSE BECAUSE OF NON-PAYMENT OF DUE FEES

Effective date: 20190630

Ref country code: LI

Free format text: LAPSE BECAUSE OF NON-PAYMENT OF DUE FEES

Effective date: 20190630

Ref country code: BE

Free format text: LAPSE BECAUSE OF NON-PAYMENT OF DUE FEES

Effective date: 20190630

PG25 Lapsed in a contracting state [announced via postgrant information from national office to epo]

Ref country code: FR

Free format text: LAPSE BECAUSE OF NON-PAYMENT OF DUE FEES

Effective date: 20190630

PG25 Lapsed in a contracting state [announced via postgrant information from national office to epo]

Ref country code: CY

Free format text: LAPSE BECAUSE OF FAILURE TO SUBMIT A TRANSLATION OF THE DESCRIPTION OR TO PAY THE FEE WITHIN THE PRESCRIBED TIME-LIMIT

Effective date: 20190220

PG25 Lapsed in a contracting state [announced via postgrant information from national office to epo]

Ref country code: HU

Free format text: LAPSE BECAUSE OF FAILURE TO SUBMIT A TRANSLATION OF THE DESCRIPTION OR TO PAY THE FEE WITHIN THE PRESCRIBED TIME-LIMIT; INVALID AB INITIO

Effective date: 20090608

Ref country code: MT

Free format text: LAPSE BECAUSE OF FAILURE TO SUBMIT A TRANSLATION OF THE DESCRIPTION OR TO PAY THE FEE WITHIN THE PRESCRIBED TIME-LIMIT

Effective date: 20190220

PG25 Lapsed in a contracting state [announced via postgrant information from national office to epo]

Ref country code: MK

Free format text: LAPSE BECAUSE OF FAILURE TO SUBMIT A TRANSLATION OF THE DESCRIPTION OR TO PAY THE FEE WITHIN THE PRESCRIBED TIME-LIMIT

Effective date: 20190220

P01 Opt-out of the competence of the unified patent court (upc) registered

Effective date: 20230527

PGFP Annual fee paid to national office [announced via postgrant information from national office to epo]

Ref country code: DE

Payment date: 20230502

Year of fee payment: 15