EP3449643B1 - Verfahren und system zum senden eines 360°-audiosignals - Google Patents

Verfahren und system zum senden eines 360°-audiosignals Download PDF

Info

Publication number
EP3449643B1
EP3449643B1 EP17725294.7A EP17725294A EP3449643B1 EP 3449643 B1 EP3449643 B1 EP 3449643B1 EP 17725294 A EP17725294 A EP 17725294A EP 3449643 B1 EP3449643 B1 EP 3449643B1
Authority
EP
European Patent Office
Prior art keywords
sound signal
microphones
ambisonic
format
signal processing
Prior art date
Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
Active
Application number
EP17725294.7A
Other languages
English (en)
French (fr)
Other versions
EP3449643A1 (de
Inventor
Delphine Devallez
Frédéric AMADU
Current Assignee (The listed assignees may be inaccurate. Google has not performed a legal analysis and makes no representation or warranty as to the accuracy of the list.)
Arkamys SA
Original Assignee
Arkamys SA
Priority date (The priority date is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the date listed.)
Filing date
Publication date
Application filed by Arkamys SA filed Critical Arkamys SA
Publication of EP3449643A1 publication Critical patent/EP3449643A1/de
Application granted granted Critical
Publication of EP3449643B1 publication Critical patent/EP3449643B1/de
Active legal-status Critical Current
Anticipated expiration legal-status Critical

Links

Images

Classifications

    • H—ELECTRICITY
    • H04—ELECTRIC COMMUNICATION TECHNIQUE
    • H04S—STEREOPHONIC SYSTEMS 
    • H04S7/00—Indicating arrangements; Control arrangements, e.g. balance control
    • H04S7/30—Control circuits for electronic adaptation of the sound field
    • H04S7/302—Electronic adaptation of stereophonic sound system to listener position or orientation
    • H—ELECTRICITY
    • H04—ELECTRIC COMMUNICATION TECHNIQUE
    • H04R—LOUDSPEAKERS, MICROPHONES, GRAMOPHONE PICK-UPS OR LIKE ACOUSTIC ELECTROMECHANICAL TRANSDUCERS; ELECTRIC HEARING AIDS; PUBLIC ADDRESS SYSTEMS
    • H04R1/00—Details of transducers, loudspeakers or microphones
    • H04R1/20—Arrangements for obtaining desired frequency or directional characteristics
    • H04R1/32—Arrangements for obtaining desired frequency or directional characteristics for obtaining desired directional characteristic only
    • H04R1/40—Arrangements for obtaining desired frequency or directional characteristics for obtaining desired directional characteristic only by combining a number of identical transducers
    • H04R1/406—Arrangements for obtaining desired frequency or directional characteristics for obtaining desired directional characteristic only by combining a number of identical transducers microphones
    • H—ELECTRICITY
    • H04—ELECTRIC COMMUNICATION TECHNIQUE
    • H04R—LOUDSPEAKERS, MICROPHONES, GRAMOPHONE PICK-UPS OR LIKE ACOUSTIC ELECTROMECHANICAL TRANSDUCERS; ELECTRIC HEARING AIDS; PUBLIC ADDRESS SYSTEMS
    • H04R3/00—Circuits for transducers
    • H04R3/005—Circuits for transducers for combining the signals of two or more microphones
    • H—ELECTRICITY
    • H04—ELECTRIC COMMUNICATION TECHNIQUE
    • H04S—STEREOPHONIC SYSTEMS 
    • H04S2400/00—Details of stereophonic systems covered by H04S but not provided for in its groups
    • H04S2400/15—Aspects of sound capture and related signal processing for recording or reproduction
    • H—ELECTRICITY
    • H04—ELECTRIC COMMUNICATION TECHNIQUE
    • H04S—STEREOPHONIC SYSTEMS 
    • H04S2420/00—Techniques used stereophonic systems covered by H04S but not provided for in its groups
    • H04S2420/11—Application of ambisonics in stereophonic audio systems
    • H—ELECTRICITY
    • H04—ELECTRIC COMMUNICATION TECHNIQUE
    • H04S—STEREOPHONIC SYSTEMS 
    • H04S7/00—Indicating arrangements; Control arrangements, e.g. balance control
    • H04S7/30—Control circuits for electronic adaptation of the sound field
    • H04S7/302—Electronic adaptation of stereophonic sound system to listener position or orientation
    • H04S7/303—Tracking of listener position or orientation
    • H04S7/304—For headphones

Definitions

  • the present invention relates to the field of sound signal processing.
  • 3D audio has been reserved for sound professionals and researchers.
  • the purpose of this technology is to capture as much spatial information as possible during recording, then restore it to the listener and give them a feeling of immersion in the sound scene.
  • the interest is growing for videos filmed at 360 ° and reproduced via a virtual reality headset for a complete immersion in the image: the user can turn his head and explore the visual scene all around from him.
  • the most compact solution is the use of a network of microphones, such as for example the Eigenmike from mh acoustics, the Soundfield from TSL Products, and the TetraMic from Core Sound.
  • the document WO 2005/015954 shows a method for converting the signals of an array of microphones placed in a spherical configuration into an ambisonic format. Equipped with four to thirty-two microphones, these products are expensive and therefore reserved for professional use. Recent research has reduced the number of microphones ( Palacino, JD, & Nicol, R. (2013). "Spatial sound pick-up with a low number of microphones.” ICA 2013. Montreal, Canada .), and microphones of reduced size and cost can be used such as those available in mobile phones.
  • the shape of the microphone networks, a polyhedron remains standardized, however, from the dodecahedron for the EigenMike to the tetrahedron for the Soundfield and the TetraMic.
  • the present invention intends to remedy the drawbacks of the prior art by proposing a method for processing the sound signal making it possible to capture the sound signal in all directions, then to restore said sound signal.
  • the present invention relates, in its most general sense, to a method for processing the sound signal, according to claim 1.
  • the matrix calculation involves a matrix H calculated by the method of least squares from the measured directivities of the N microphones and the ideal directivities of the ambisonic components.
  • said microphones are arranged in a circle on a plane, spaced at an angle equal to 360 ° / N or at each corner of a portable telephone.
  • said method uses four microphones spaced at an angle of 90 ° to the horizontal.
  • said method implements a filter bandpass filter from 100 Hz to 6 kHz.
  • the order R of the ambisonic type format is equal to one.
  • said information relating to the orientation of the head of a user listening to the sound signal is captured by a sensor in a mobile phone or by a sensor located in a headset or a virtual reality headset.
  • the data in ambisonic format are transformed into data in binaural format.
  • the present invention also relates to a sound signal processing system according to claim 9.
  • the present invention relates to a method for processing the sound signal, according to claim 1.
  • said microphones are arranged in a circle on a plane, spaced at an angle equal to 360 ° / N or at each corner of a portable telephone.
  • the method according to the present invention uses four microphones spaced at an angle of 90 ° to the horizontal.
  • the order R of the ambisonic type format is equal to one.
  • the first step of the method according to the present invention consists in recording the sound signal.
  • N microphones are used, N being a natural integer greater than or equal to three, said microphones being arranged in a circle on a plane, spaced at an angle equal to 360 ° / N or at each corner of a mobile telephone.
  • N is equal to four and the microphones are spaced 90 ° apart.
  • These microphones are arranged in a circle on a plane.
  • the radius of said circle is two centimeters, and the microphones are omnidirectional.
  • the sound signal is picked up by said microphones and digitized. It is a synchronous capture.
  • the second step of the method according to the present invention consists in encoding said four sampled digital signals, in an ambisonic type format of order R, R being a natural integer greater than or equal to one.
  • the ambisonic format is a standardized format for audio coding in several dimensions.
  • the order R is equal to one.
  • This order 1 makes it possible to represent the sound with the following concepts: Front - Rear and Left - Right.
  • the Figures 4a, 4b and 4c represent the ideal components W, Y and X of an order 1 ambisonic format (on a horizontal plane).
  • the Figures 5a, 5b and 5c illustrate the approximate W, Y and X components of a first order ambisonic format.
  • the Figure 2 illustrates the treatments applied in the context of the second step of the method according to the present invention.
  • Hanning windows are used with an overlap by implementing a “ overlap-add” type function .
  • This matrix includes weighting coefficients for each microphone signal and each frequency.
  • the method according to the present invention implements a filter bandpass filter from 100 Hz to 6 kHz. This eliminates the lower part and the acute part.
  • the impulse responses of the N microphones are measured, in the present case of the four microphones, with a source positioned every 5 ° or every 10 ° around the microphone array.
  • the frequency responses of the N microphones are obtained as a function of the angles measured, or in other words the directivities of the N microphones as a function of the frequency.
  • the microphones' responses are then placed in a matrix C.
  • H NOT ⁇ V P D ⁇ V
  • N the number of microphones (four in the present embodiment)
  • D the number of angular source positions measured (108 in the present embodiment)
  • V the number of ambisonic channels (three in the present example)
  • C DxN denotes the directivities of the microphones
  • H NxV denotes the matrix which transforms the directivities of the microphones into the desired directivities
  • P DxV denotes the directivities prescribed by the ambisonic format (W, X and Y in the present example of production).
  • H NxV P DxV / C DxN for each frequency index k if C DxN is invertible.
  • C DxN is not invertible.
  • a least squares method is implemented to solve for C 108x4 .
  • H 4x3 P 108x3
  • the matrix H is defined once for the future uses of the network of microphones considered. Then, with each use, a matrix multiplication is carried out in the frequency domain.
  • Said matrix H has as many lines as there are microphones, therefore four in the present embodiment, and as many columns as required by the order of the ambisonic format used, therefore three columns in the present embodiment, in which the order 1 is implemented horizontally.
  • Out In x H, where H denotes the previously calculated matrix, In denotes the input (audio channels coming from the network of microphones, passed in the frequency domain) and Out denotes the output (Out being reconverted in the temporal domain for get ambisonic format).
  • the method according to the present invention implements, during this second step, an algorithm called least squares algorithm for each frequency, with for example 512 frequency points.
  • data is obtained in the ambisonic format (in the present exemplary embodiment the signals W, X and Y).
  • the third step of the method according to the present invention consists in restoring the sound signal, by means of a transformation of the data in ambisonic format into two binaural channels.
  • the information relating to the orientation of the head of the user listening to the sound signal is collected and used. This can be realized by a sensor in a mobile phone, a headset or a virtual reality headset.
  • This orientation information consists of a vector comprising three angle values, in Anglo-Saxon terminology “ pitch”,”yaw” and “ roll”.
  • the angle value " yaw" is used on a plane.
  • the ambisonic format is transformed into eight audio channels corresponding to a virtual placement of eight speakers, each placed at 45 ° around the user.
  • the Figure 6 represents the placement of eight virtual speakers, each placed at 45 ° around a user.
  • W, X and Y are the data relating to the ambisonic format
  • ⁇ n represents the horizontal angle of the n th loudspeaker.
  • W, X and Y are the data relating to the ambisonic format
  • ⁇ n represents the horizontal angle of the n th loudspeaker.
  • HRTF head-related transfer function
  • IIR Infinite Impulse Response
  • ⁇ n ⁇ n - ⁇ .
  • the Figure 3 represents the different stages of the process according to the present invention.
  • the present invention also relates to a sound signal processing system according to claim 9.
  • This sound signal processing system includes at least one computing unit and one memory unit.

Landscapes

  • Health & Medical Sciences (AREA)
  • Otolaryngology (AREA)
  • Physics & Mathematics (AREA)
  • Engineering & Computer Science (AREA)
  • Acoustics & Sound (AREA)
  • Signal Processing (AREA)
  • General Health & Medical Sciences (AREA)
  • Stereophonic System (AREA)
  • Circuit For Audible Band Transducer (AREA)

Claims (9)

  1. Verfahren zur Verarbeitung des Audiosignals, die folgenden Schritte beinhaltend:
    • Synchrones Erfassen eines Eingangsaudiosignals (Sentrée) mithilfe von N Mikrofonen, wobei N eine natürliche ganze Zahl größer oder gleich drei ist;
    • Codieren des Eingangsaudiosignals (Sentrée) in ein Tondatenformat (D), wobei das Codieren einen Unterschritt des Umwandelns des Eingangssignals in ein Raumklangformat R. Ordnung ist, wobei R eine natürliche ganze Zahl größer oder gleich eins ist, wobei der Unterschritt des Umwandelns in ein Raumklangformat mithilfe einer schnellen Fourier-Transformation, einer Matrixmultiplikation, einer schnellen umgekehrten Fourier-Transformation, und mithilfe eines Bandpassfilters durchgeführt wird; und
    • Wiedergeben eines Ausgangsaudiosignals (Ssortie) mithilfe einer digitalen Verarbeitung der Tondaten (D);
    und dadurch gekennzeichnet, dass die Matrixmultiplikation eine Matrix H einbeziehen lässt, die durch die Methode der kleinsten Quadrate aus den gemessenen Richtwirkungen der N Mikrofone und den idealen Richtwirkungen der Raumklangkomponenten berechnet wird.
  2. Verfahren zur Verarbeitung des Audiosignals nach Anspruch 1, dadurch gekennzeichnet, dass die Mikrofone in einem Kreis auf einer Ebene, entsprechend einem Winkel gleich 360°/N beabstandet, oder an jeder Ecke eines tragbaren Telefons angeordnet sind.
  3. Verfahren zur Verarbeitung des Audiosignals nach Anspruch 2, dadurch gekennzeichnet, dass es vier Mikrofone anwendet, die entsprechend einem Winkel von 90° zur Horizontalen beabstandet sind.
  4. Verfahren zur Verarbeitung des Audiosignals nach einem der vorstehenden Ansprüche, dadurch gekennzeichnet, dass es ein Bandpassfilter anwendet, das von 100 Hz bis 6 kHz filtert.
  5. Verfahren zur Verarbeitung des Audiosignals nach einem der vorstehenden Ansprüche, dadurch gekennzeichnet, dass die R. Ordnung des Raumklangformats gleich eins ist.
  6. Verfahren zur Verarbeitung des Audiosignals nach einem der vorstehenden Ansprüche, dadurch gekennzeichnet, dass im Laufe des Wiedergabeschrittes eine Information in Bezug auf die Ausrichtung des Kopfes eines Nutzers, der das Audiosignal anhört, ausgewertet wird.
  7. Verfahren zur Verarbeitung des Audiosignals nach Anspruch 6, dadurch gekennzeichnet, dass das Erfassen der Information in Bezug auf die Ausrichtung des Kopfes eines Nutzers, der das Audiosignal anhört, durch einen Sensor innerhalb eines Telefons, eines Kopfhörers oder einer Brille für virtuelle Realität realisiert wird.
  8. Verfahren zur Verarbeitung des Audiosignals nach einem der vorstehenden Ansprüche, dadurch gekennzeichnet, dass im Laufe des Wiedergabeschrittes die Daten im Raumklangformat in Daten im Binärformat umgewandelt werden.
  9. System zur Verarbeitung des Audiosignals, Mittel beinhaltend zum:
    • Synchronen Erfassen eines Eingangsaudiosignals (Sentrée) mithilfe von N Mikrofonen, wobei N eine natürliche ganze Zahl größer oder gleich drei ist;
    • Codieren des Eingangsaudiosignals (Sentrée) in ein Tondatenformat (D), wobei das Codieren einen Unterschritt des Umwandelns des Eingangssignals in ein Raumklangformat R. Ordnung ist, wobei R eine natürliche ganze Zahl größer oder gleich eins ist, wobei die Mittel zum Umwandeln in ein Raumklangformat mithilfe einer schnellen Fourier-Transformation, einer Matrixmultiplikation, einer schnellen umgekehrten Fourier-Transformation, und mithilfe eines Bandpassfilters angewendet werden; und
    • Wiedergeben eines Ausgangsaudiosignals (Ssortie) mithilfe einer digitalen Verarbeitung der Tondaten (D);
    und dadurch gekennzeichnet, dass die Matrixmultiplikation eine Matrix H einbeziehen lässt, die durch die Methode der kleinsten Quadrate aus den gemessenen Richtwirkungen der N Mikrofone und den idealen Richtwirkungen der Raumklangkomponenten berechnet wird.
EP17725294.7A 2016-04-26 2017-04-20 Verfahren und system zum senden eines 360°-audiosignals Active EP3449643B1 (de)

Applications Claiming Priority (2)

Application Number Priority Date Filing Date Title
FR1653684A FR3050601B1 (fr) 2016-04-26 2016-04-26 Procede et systeme de diffusion d'un signal audio a 360°
PCT/FR2017/050935 WO2017187053A1 (fr) 2016-04-26 2017-04-20 Procédé et système de diffusion d'un signal audio à 360°

Publications (2)

Publication Number Publication Date
EP3449643A1 EP3449643A1 (de) 2019-03-06
EP3449643B1 true EP3449643B1 (de) 2020-06-10

Family

ID=56943619

Family Applications (1)

Application Number Title Priority Date Filing Date
EP17725294.7A Active EP3449643B1 (de) 2016-04-26 2017-04-20 Verfahren und system zum senden eines 360°-audiosignals

Country Status (5)

Country Link
US (1) US10659902B2 (de)
EP (1) EP3449643B1 (de)
CN (1) CN109661824A (de)
FR (1) FR3050601B1 (de)
WO (1) WO2017187053A1 (de)

Family Cites Families (10)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
AUPO099696A0 (en) * 1996-07-12 1996-08-08 Lake Dsp Pty Limited Methods and apparatus for processing spatialised audio
US6021206A (en) * 1996-10-02 2000-02-01 Lake Dsp Pty Ltd Methods and apparatus for processing spatialised audio
NZ502603A (en) * 2000-02-02 2002-09-27 Ind Res Ltd Multitransducer microphone arrays with signal processing for high resolution sound field recording
FR2858512A1 (fr) * 2003-07-30 2005-02-04 France Telecom Procede et dispositif de traitement de donnees sonores en contexte ambiophonique
GB0906269D0 (en) * 2009-04-09 2009-05-20 Ntnu Technology Transfer As Optimal modal beamformer for sensor arrays
US9986356B2 (en) * 2012-02-15 2018-05-29 Harman International Industries, Incorporated Audio surround processing system
US9736609B2 (en) * 2013-02-07 2017-08-15 Qualcomm Incorporated Determining renderers for spherical harmonic coefficients
US9959875B2 (en) * 2013-03-01 2018-05-01 Qualcomm Incorporated Specifying spherical harmonic and/or higher order ambisonics coefficients in bitstreams
CN104424953B (zh) * 2013-09-11 2019-11-01 华为技术有限公司 语音信号处理方法与装置
FR3018015B1 (fr) 2014-02-25 2016-04-29 Arkamys Procede et systeme d'egalisation acoustique automatise

Non-Patent Citations (1)

* Cited by examiner, † Cited by third party
Title
None *

Also Published As

Publication number Publication date
FR3050601B1 (fr) 2018-06-22
WO2017187053A1 (fr) 2017-11-02
EP3449643A1 (de) 2019-03-06
US10659902B2 (en) 2020-05-19
CN109661824A (zh) 2019-04-19
FR3050601A1 (fr) 2017-10-27
US20190132695A1 (en) 2019-05-02

Similar Documents

Publication Publication Date Title
EP1836876B1 (de) Verfahren und vorrichtung zur individualisierung von hrtfs durch modellierung
EP1563485B1 (de) Verfahren zur verarbeitung von audiodateien und erfassungsvorrichtung zur anwendung davon
EP2000002B1 (de) Verfahren und einrichtung zur effizienten binauralen raumklangerzeugung im transformierten bereich
EP1992198B1 (de) Optimierung des binauralen raumklangeffektes durch mehrkanalkodierung
EP1946612B1 (de) Hrtfs-individualisierung durch modellierung mit finiten elementen gekoppelt mit einem korrekturmodell
CN104581610B (zh) 一种虚拟立体声合成方法及装置
EP1586220B1 (de) Verfahren und einrichtung zur steuerung einer wiedergabeeinheitdurch verwendung eines mehrkanalsignals
FR2995754A1 (fr) Calibration optimisee d'un systeme de restitution sonore multi haut-parleurs
WO2007110520A1 (fr) Procede de synthese binaurale prenant en compte un effet de salle
EP2009891B1 (de) Übertragung eines Audiosignals in einem immersiven Audiokonferenzsystem
FR3065137A1 (fr) Procede de spatialisation sonore
EP2995095B1 (de) Vorrichtung und verfahren zum komprimieren eines satzes an n-binauralen raumimpulsantworten
EP3025514B1 (de) Klangverräumlichung mit raumwirkung
WO2021212287A1 (zh) 音频信号处理方法、音频处理装置及录音设备
EP3449643B1 (de) Verfahren und system zum senden eines 360°-audiosignals
EP2987339B1 (de) Verfahren zur akustischen wiedergabe eines numerischen audiosignals
EP3384688B1 (de) Aufeinanderfolgende dekompositionen von audiofiltern
US10764684B1 (en) Binaural audio using an arbitrarily shaped microphone array
WO2024240950A1 (fr) Procédé pour générer une scène audio dans un système de spatialisation binaurale
WO2019020437A1 (fr) Procédé et système de traitement d'un signal audio incluant un encodage au format ambisonique

Legal Events

Date Code Title Description
STAA Information on the status of an ep patent application or granted ep patent

Free format text: STATUS: UNKNOWN

STAA Information on the status of an ep patent application or granted ep patent

Free format text: STATUS: THE INTERNATIONAL PUBLICATION HAS BEEN MADE

PUAI Public reference made under article 153(3) epc to a published international application that has entered the european phase

Free format text: ORIGINAL CODE: 0009012

STAA Information on the status of an ep patent application or granted ep patent

Free format text: STATUS: REQUEST FOR EXAMINATION WAS MADE

17P Request for examination filed

Effective date: 20181113

AK Designated contracting states

Kind code of ref document: A1

Designated state(s): AL AT BE BG CH CY CZ DE DK EE ES FI FR GB GR HR HU IE IS IT LI LT LU LV MC MK MT NL NO PL PT RO RS SE SI SK SM TR

AX Request for extension of the european patent

Extension state: BA ME

DAV Request for validation of the european patent (deleted)
DAX Request for extension of the european patent (deleted)
GRAP Despatch of communication of intention to grant a patent

Free format text: ORIGINAL CODE: EPIDOSNIGR1

STAA Information on the status of an ep patent application or granted ep patent

Free format text: STATUS: GRANT OF PATENT IS INTENDED

INTG Intention to grant announced

Effective date: 20191126

GRAS Grant fee paid

Free format text: ORIGINAL CODE: EPIDOSNIGR3

GRAA (expected) grant

Free format text: ORIGINAL CODE: 0009210

STAA Information on the status of an ep patent application or granted ep patent

Free format text: STATUS: THE PATENT HAS BEEN GRANTED

AK Designated contracting states

Kind code of ref document: B1

Designated state(s): AL AT BE BG CH CY CZ DE DK EE ES FI FR GB GR HR HU IE IS IT LI LT LU LV MC MK MT NL NO PL PT RO RS SE SI SK SM TR

REG Reference to a national code

Ref country code: GB

Ref legal event code: FG4D

Free format text: NOT ENGLISH

REG Reference to a national code

Ref country code: CH

Ref legal event code: EP

Ref country code: AT

Ref legal event code: REF

Ref document number: 1280121

Country of ref document: AT

Kind code of ref document: T

Effective date: 20200615

REG Reference to a national code

Ref country code: DE

Ref legal event code: R096

Ref document number: 602017018008

Country of ref document: DE

REG Reference to a national code

Ref country code: IE

Ref legal event code: FG4D

Free format text: LANGUAGE OF EP DOCUMENT: FRENCH

REG Reference to a national code

Ref country code: LT

Ref legal event code: MG4D

PG25 Lapsed in a contracting state [announced via postgrant information from national office to epo]

Ref country code: FI

Free format text: LAPSE BECAUSE OF FAILURE TO SUBMIT A TRANSLATION OF THE DESCRIPTION OR TO PAY THE FEE WITHIN THE PRESCRIBED TIME-LIMIT

Effective date: 20200610

Ref country code: NO

Free format text: LAPSE BECAUSE OF FAILURE TO SUBMIT A TRANSLATION OF THE DESCRIPTION OR TO PAY THE FEE WITHIN THE PRESCRIBED TIME-LIMIT

Effective date: 20200910

Ref country code: LT

Free format text: LAPSE BECAUSE OF FAILURE TO SUBMIT A TRANSLATION OF THE DESCRIPTION OR TO PAY THE FEE WITHIN THE PRESCRIBED TIME-LIMIT

Effective date: 20200610

Ref country code: SE

Free format text: LAPSE BECAUSE OF FAILURE TO SUBMIT A TRANSLATION OF THE DESCRIPTION OR TO PAY THE FEE WITHIN THE PRESCRIBED TIME-LIMIT

Effective date: 20200610

REG Reference to a national code

Ref country code: NL

Ref legal event code: MP

Effective date: 20200610

PG25 Lapsed in a contracting state [announced via postgrant information from national office to epo]

Ref country code: RS

Free format text: LAPSE BECAUSE OF FAILURE TO SUBMIT A TRANSLATION OF THE DESCRIPTION OR TO PAY THE FEE WITHIN THE PRESCRIBED TIME-LIMIT

Effective date: 20200610

Ref country code: LV

Free format text: LAPSE BECAUSE OF FAILURE TO SUBMIT A TRANSLATION OF THE DESCRIPTION OR TO PAY THE FEE WITHIN THE PRESCRIBED TIME-LIMIT

Effective date: 20200610

Ref country code: HR

Free format text: LAPSE BECAUSE OF FAILURE TO SUBMIT A TRANSLATION OF THE DESCRIPTION OR TO PAY THE FEE WITHIN THE PRESCRIBED TIME-LIMIT

Effective date: 20200610

Ref country code: BG

Free format text: LAPSE BECAUSE OF FAILURE TO SUBMIT A TRANSLATION OF THE DESCRIPTION OR TO PAY THE FEE WITHIN THE PRESCRIBED TIME-LIMIT

Effective date: 20200910

REG Reference to a national code

Ref country code: AT

Ref legal event code: MK05

Ref document number: 1280121

Country of ref document: AT

Kind code of ref document: T

Effective date: 20200610

PG25 Lapsed in a contracting state [announced via postgrant information from national office to epo]

Ref country code: NL

Free format text: LAPSE BECAUSE OF FAILURE TO SUBMIT A TRANSLATION OF THE DESCRIPTION OR TO PAY THE FEE WITHIN THE PRESCRIBED TIME-LIMIT

Effective date: 20200610

Ref country code: AL

Free format text: LAPSE BECAUSE OF FAILURE TO SUBMIT A TRANSLATION OF THE DESCRIPTION OR TO PAY THE FEE WITHIN THE PRESCRIBED TIME-LIMIT

Effective date: 20200610

PG25 Lapsed in a contracting state [announced via postgrant information from national office to epo]

Ref country code: PT

Free format text: LAPSE BECAUSE OF FAILURE TO SUBMIT A TRANSLATION OF THE DESCRIPTION OR TO PAY THE FEE WITHIN THE PRESCRIBED TIME-LIMIT

Effective date: 20201012

Ref country code: ES

Free format text: LAPSE BECAUSE OF FAILURE TO SUBMIT A TRANSLATION OF THE DESCRIPTION OR TO PAY THE FEE WITHIN THE PRESCRIBED TIME-LIMIT

Effective date: 20200610

Ref country code: CZ

Free format text: LAPSE BECAUSE OF FAILURE TO SUBMIT A TRANSLATION OF THE DESCRIPTION OR TO PAY THE FEE WITHIN THE PRESCRIBED TIME-LIMIT

Effective date: 20200610

Ref country code: IT

Free format text: LAPSE BECAUSE OF FAILURE TO SUBMIT A TRANSLATION OF THE DESCRIPTION OR TO PAY THE FEE WITHIN THE PRESCRIBED TIME-LIMIT

Effective date: 20200610

Ref country code: RO

Free format text: LAPSE BECAUSE OF FAILURE TO SUBMIT A TRANSLATION OF THE DESCRIPTION OR TO PAY THE FEE WITHIN THE PRESCRIBED TIME-LIMIT

Effective date: 20200610

Ref country code: EE

Free format text: LAPSE BECAUSE OF FAILURE TO SUBMIT A TRANSLATION OF THE DESCRIPTION OR TO PAY THE FEE WITHIN THE PRESCRIBED TIME-LIMIT

Effective date: 20200610

Ref country code: SM

Free format text: LAPSE BECAUSE OF FAILURE TO SUBMIT A TRANSLATION OF THE DESCRIPTION OR TO PAY THE FEE WITHIN THE PRESCRIBED TIME-LIMIT

Effective date: 20200610

Ref country code: AT

Free format text: LAPSE BECAUSE OF FAILURE TO SUBMIT A TRANSLATION OF THE DESCRIPTION OR TO PAY THE FEE WITHIN THE PRESCRIBED TIME-LIMIT

Effective date: 20200610

PG25 Lapsed in a contracting state [announced via postgrant information from national office to epo]

Ref country code: PL

Free format text: LAPSE BECAUSE OF FAILURE TO SUBMIT A TRANSLATION OF THE DESCRIPTION OR TO PAY THE FEE WITHIN THE PRESCRIBED TIME-LIMIT

Effective date: 20200610

Ref country code: SK

Free format text: LAPSE BECAUSE OF FAILURE TO SUBMIT A TRANSLATION OF THE DESCRIPTION OR TO PAY THE FEE WITHIN THE PRESCRIBED TIME-LIMIT

Effective date: 20200610

Ref country code: IS

Free format text: LAPSE BECAUSE OF FAILURE TO SUBMIT A TRANSLATION OF THE DESCRIPTION OR TO PAY THE FEE WITHIN THE PRESCRIBED TIME-LIMIT

Effective date: 20201010

REG Reference to a national code

Ref country code: DE

Ref legal event code: R097

Ref document number: 602017018008

Country of ref document: DE

PLBE No opposition filed within time limit

Free format text: ORIGINAL CODE: 0009261

STAA Information on the status of an ep patent application or granted ep patent

Free format text: STATUS: NO OPPOSITION FILED WITHIN TIME LIMIT

PG25 Lapsed in a contracting state [announced via postgrant information from national office to epo]

Ref country code: DK

Free format text: LAPSE BECAUSE OF FAILURE TO SUBMIT A TRANSLATION OF THE DESCRIPTION OR TO PAY THE FEE WITHIN THE PRESCRIBED TIME-LIMIT

Effective date: 20200610

26N No opposition filed

Effective date: 20210311

PG25 Lapsed in a contracting state [announced via postgrant information from national office to epo]

Ref country code: SI

Free format text: LAPSE BECAUSE OF FAILURE TO SUBMIT A TRANSLATION OF THE DESCRIPTION OR TO PAY THE FEE WITHIN THE PRESCRIBED TIME-LIMIT

Effective date: 20200610

PG25 Lapsed in a contracting state [announced via postgrant information from national office to epo]

Ref country code: MC

Free format text: LAPSE BECAUSE OF FAILURE TO SUBMIT A TRANSLATION OF THE DESCRIPTION OR TO PAY THE FEE WITHIN THE PRESCRIBED TIME-LIMIT

Effective date: 20200610

GBPC Gb: european patent ceased through non-payment of renewal fee

Effective date: 20210420

PG25 Lapsed in a contracting state [announced via postgrant information from national office to epo]

Ref country code: LU

Free format text: LAPSE BECAUSE OF NON-PAYMENT OF DUE FEES

Effective date: 20210420

REG Reference to a national code

Ref country code: BE

Ref legal event code: MM

Effective date: 20210430

PG25 Lapsed in a contracting state [announced via postgrant information from national office to epo]

Ref country code: GB

Free format text: LAPSE BECAUSE OF NON-PAYMENT OF DUE FEES

Effective date: 20210420

Ref country code: CH

Free format text: LAPSE BECAUSE OF NON-PAYMENT OF DUE FEES

Effective date: 20210430

Ref country code: LI

Free format text: LAPSE BECAUSE OF NON-PAYMENT OF DUE FEES

Effective date: 20210430

PG25 Lapsed in a contracting state [announced via postgrant information from national office to epo]

Ref country code: IE

Free format text: LAPSE BECAUSE OF NON-PAYMENT OF DUE FEES

Effective date: 20210420

PG25 Lapsed in a contracting state [announced via postgrant information from national office to epo]

Ref country code: IS

Free format text: LAPSE BECAUSE OF FAILURE TO SUBMIT A TRANSLATION OF THE DESCRIPTION OR TO PAY THE FEE WITHIN THE PRESCRIBED TIME-LIMIT

Effective date: 20201010

PG25 Lapsed in a contracting state [announced via postgrant information from national office to epo]

Ref country code: BE

Free format text: LAPSE BECAUSE OF NON-PAYMENT OF DUE FEES

Effective date: 20210430

P01 Opt-out of the competence of the unified patent court (upc) registered

Effective date: 20230512

PG25 Lapsed in a contracting state [announced via postgrant information from national office to epo]

Ref country code: CY

Free format text: LAPSE BECAUSE OF FAILURE TO SUBMIT A TRANSLATION OF THE DESCRIPTION OR TO PAY THE FEE WITHIN THE PRESCRIBED TIME-LIMIT

Effective date: 20200610

PG25 Lapsed in a contracting state [announced via postgrant information from national office to epo]

Ref country code: HU

Free format text: LAPSE BECAUSE OF FAILURE TO SUBMIT A TRANSLATION OF THE DESCRIPTION OR TO PAY THE FEE WITHIN THE PRESCRIBED TIME-LIMIT; INVALID AB INITIO

Effective date: 20170420

Ref country code: GR

Free format text: LAPSE BECAUSE OF FAILURE TO SUBMIT A TRANSLATION OF THE DESCRIPTION OR TO PAY THE FEE WITHIN THE PRESCRIBED TIME-LIMIT

Effective date: 20200610

PG25 Lapsed in a contracting state [announced via postgrant information from national office to epo]

Ref country code: MK

Free format text: LAPSE BECAUSE OF FAILURE TO SUBMIT A TRANSLATION OF THE DESCRIPTION OR TO PAY THE FEE WITHIN THE PRESCRIBED TIME-LIMIT

Effective date: 20200610

PG25 Lapsed in a contracting state [announced via postgrant information from national office to epo]

Ref country code: TR

Free format text: LAPSE BECAUSE OF FAILURE TO SUBMIT A TRANSLATION OF THE DESCRIPTION OR TO PAY THE FEE WITHIN THE PRESCRIBED TIME-LIMIT

Effective date: 20200610

PG25 Lapsed in a contracting state [announced via postgrant information from national office to epo]

Ref country code: MT

Free format text: LAPSE BECAUSE OF FAILURE TO SUBMIT A TRANSLATION OF THE DESCRIPTION OR TO PAY THE FEE WITHIN THE PRESCRIBED TIME-LIMIT

Effective date: 20200610

PGFP Annual fee paid to national office [announced via postgrant information from national office to epo]

Ref country code: DE

Payment date: 20250429

Year of fee payment: 9

PGFP Annual fee paid to national office [announced via postgrant information from national office to epo]

Ref country code: FR

Payment date: 20250430

Year of fee payment: 9