EP3114859A1 - Modélisation structurale de la réponse impulsionnelle relative à la tête - Google Patents

Modélisation structurale de la réponse impulsionnelle relative à la tête

Info

Publication number
EP3114859A1
EP3114859A1 EP15713262.2A EP15713262A EP3114859A1 EP 3114859 A1 EP3114859 A1 EP 3114859A1 EP 15713262 A EP15713262 A EP 15713262A EP 3114859 A1 EP3114859 A1 EP 3114859A1
Authority
EP
European Patent Office
Prior art keywords
pinna
model
torso
hrir
audio
Prior art date
Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
Granted
Application number
EP15713262.2A
Other languages
German (de)
English (en)
Other versions
EP3114859B1 (fr
Inventor
C. Phillip Brown
Matthew Fellers
Regunathan Radhakrishnan
Current Assignee (The listed assignees may be inaccurate. Google has not performed a legal analysis and makes no representation or warranty as to the accuracy of the list.)
Dolby Laboratories Licensing Corp
Original Assignee
Dolby Laboratories Licensing Corp
Priority date (The priority date is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the date listed.)
Filing date
Publication date
Application filed by Dolby Laboratories Licensing Corp filed Critical Dolby Laboratories Licensing Corp
Publication of EP3114859A1 publication Critical patent/EP3114859A1/fr
Application granted granted Critical
Publication of EP3114859B1 publication Critical patent/EP3114859B1/fr
Active legal-status Critical Current
Anticipated expiration legal-status Critical

Links

Classifications

    • H—ELECTRICITY
    • H04—ELECTRIC COMMUNICATION TECHNIQUE
    • H04S—STEREOPHONIC SYSTEMS 
    • H04S7/00—Indicating arrangements; Control arrangements, e.g. balance control
    • H04S7/30—Control circuits for electronic adaptation of the sound field
    • H04S7/302—Electronic adaptation of stereophonic sound system to listener position or orientation
    • H04S7/303—Tracking of listener position or orientation
    • H04S7/304—For headphones
    • H—ELECTRICITY
    • H04—ELECTRIC COMMUNICATION TECHNIQUE
    • H04S—STEREOPHONIC SYSTEMS 
    • H04S7/00—Indicating arrangements; Control arrangements, e.g. balance control
    • H04S7/30—Control circuits for electronic adaptation of the sound field
    • H—ELECTRICITY
    • H04—ELECTRIC COMMUNICATION TECHNIQUE
    • H04S—STEREOPHONIC SYSTEMS 
    • H04S1/00—Two-channel systems
    • H04S1/007—Two-channel systems in which the audio signals are in digital form
    • H—ELECTRICITY
    • H04—ELECTRIC COMMUNICATION TECHNIQUE
    • H04S—STEREOPHONIC SYSTEMS 
    • H04S2400/00—Details of stereophonic systems covered by H04S but not provided for in its groups
    • H04S2400/11—Positioning of individual sound objects, e.g. moving airplane, within a sound field
    • H—ELECTRICITY
    • H04—ELECTRIC COMMUNICATION TECHNIQUE
    • H04S—STEREOPHONIC SYSTEMS 
    • H04S2420/00—Techniques used stereophonic systems covered by H04S but not provided for in its groups
    • H04S2420/01—Enhancing the perception of the sound image or of the spatial distribution using head related transfer functions [HRTF's] or equivalents thereof, e.g. interaural time difference [ITD] or interaural level difference [ILD]

Definitions

  • FIG. 16 illustrates front/back tilt error as a function of the TILT parameter, under an embodiment.
  • HRTF CJiff (L, az) HRTF c (az)- HRTF c (-az)
  • the direction of b is thus dependent on a, which is dependent on the angle of elevation ⁇ ;
  • s is the unit vector in the direction of the source 1002 (which is the rectangular- to-polar conversion of the source elevation and azimuth);
  • d is the specified vector from the center 1008 of the torso 1004 to the ear 1006, where the position of the ear is specified with respect to the head sphere.
  • the vector d 2 is a vector that is orthogonal to d, and lies in the plane formed by s and d. It should be noted that a can be estimated as a function of ⁇ , according to Eq. 11:
  • the delay AT L p due to the filter was found to be 17 samples for a 44.1 kHz sample rate.
  • a diffusion network is applied to the torso reflection impulse response, conditioned on the elevation. For elevations near or below the horizon (elevation ⁇ 0 degrees) the signal will arrive tangentially (or near tangentially) to the torso and any acoustic energy that arrives at the ear will be heavily diffuse due to the acoustic scattering of the wave-front reflecting from the torso. This is modeled in the system with a diffusion network of which the degree of diffusion applied varies as a function of elevation as shown in FIG. 13.
  • FIG. 13 illustrates diffusion as a function of elevation for a diffusion network applied to a torso reflection impulse response, under an embodiment.
  • the average contains torso reflection components for frequencies below 2 kHz. Since the model contains a dedicated tool to apply torso reflection, the torso reflection components are removed from the front/back difference magnitude response. This may be accomplished by forcing the magnitude response to 0 dB below 2 kHz. A smooth cross-fade is applied between this frequency range, and the non-affected frequency range. The cross- fade is applied between 2 and 4 kHz. Likewise for elevations that would boost the gain above 0 dB at Nyquist, the gain is faded down such that the gain is 0 dB at Nyquist. This fade is applied between 20 to 22.05 kHz (for a sample rate of 44.1 kHz).
  • the structural HRIR model models the frequency location of pinna notches as function of elevation and azimuth.
  • the ILD and ITD cues are not sufficient to localize objects in 3D space.
  • the ITD and ILD values are identical as one varies the elevation from -45 to 225 degrees assuming an inter-aural coordinate system as described above. This set of points is usually referred to as the cone of confusion. To resolve two locations on the cone of confusion, one relies on the frequency locations of various pinna notches. The frequency location of the pinna notch is dependent on the source elevation at a given azimuth.
  • 'd' is the distance of the reflecting structure of pinna from the ear-canal entrance
  • 'c' is the speed of sound
  • 'f is frequency at which destructive interference happens resulting in a notch in the spectrum.

Abstract

L'invention concerne un procédé permettant de créer une réponse impulsionnelle relative à la tête (HRIR) destinée à être utilisée dans un rendu audio pour la lecture par le biais d'écouteurs, ledit procédé consistant à recevoir des paramètres de localisation pour un son comprenant l'azimut (az), l'élévation (el) et la portée (range) par rapport à la tête d'un auditeur; à appliquer un modèle de tête sphérique aux paramètres entrés d'azimut, d'élévation et de portée pour générer des valeurs HRIR binaurales; à calculer un modèle de pavillon au moyen des paramètres d'azimut et d'élévation à appliquer aux valeurs HRIR binaurales pour générer des valeurs HRIR modélisées de pavillon; à calculer un modèle de torse au moyen des paramètres d'azimut et d'élévation à appliquer aux valeurs HRIR modélisées du pavillon pour générer des valeurs HRIR modélisées de pavillon et de torse; et à calculer un modèle de champ proche au moyen des paramètres d'azimut et de portée à appliquer aux valeurs HRIR modélisées de pavillon et de torse pour générer des valeurs HRIR modélisées de pavillon, de torse et de champ proche.
EP15713262.2A 2014-03-06 2015-03-04 Modélisation structurale de la réponse impulsionnelle relative à la tête Active EP3114859B1 (fr)

Applications Claiming Priority (2)

Application Number Priority Date Filing Date Title
US201461948849P 2014-03-06 2014-03-06
PCT/US2015/018812 WO2015134658A1 (fr) 2014-03-06 2015-03-04 Modélisation structurale de la réponse impulsionnelle relative à la tête

Publications (2)

Publication Number Publication Date
EP3114859A1 true EP3114859A1 (fr) 2017-01-11
EP3114859B1 EP3114859B1 (fr) 2018-05-09

Family

ID=52780017

Family Applications (1)

Application Number Title Priority Date Filing Date
EP15713262.2A Active EP3114859B1 (fr) 2014-03-06 2015-03-04 Modélisation structurale de la réponse impulsionnelle relative à la tête

Country Status (3)

Country Link
US (1) US10142761B2 (fr)
EP (1) EP3114859B1 (fr)
WO (1) WO2015134658A1 (fr)

Cited By (1)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN112425187A (zh) * 2018-05-18 2021-02-26 诺基亚技术有限公司 用于实现头部跟踪耳机的方法和装置

Families Citing this family (42)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
WO2014087195A1 (fr) * 2012-12-05 2014-06-12 Nokia Corporation Appareil de sélection de microphone basée sur une orientation
WO2015134658A1 (fr) * 2014-03-06 2015-09-11 Dolby Laboratories Licensing Corporation Modélisation structurale de la réponse impulsionnelle relative à la tête
GB2544458B (en) 2015-10-08 2019-10-02 Facebook Inc Binaural synthesis
WO2017161383A2 (fr) * 2016-01-26 2017-09-21 Ferrer Julio Système et procédé de synchronisation en temps réel de contenu multimédia par l'intermédiaire de dispositifs multiples et de systèmes de haut-parleurs
US9591427B1 (en) * 2016-02-20 2017-03-07 Philip Scott Lyren Capturing audio impulse responses of a person with a smartphone
US10331750B2 (en) 2016-08-01 2019-06-25 Facebook, Inc. Systems and methods to manage media content items
CN106231528B (zh) * 2016-08-04 2017-11-10 武汉大学 基于分段式多元线性回归的个性化头相关传递函数生成系统及方法
US9980077B2 (en) 2016-08-11 2018-05-22 Lg Electronics Inc. Method of interpolating HRTF and audio output apparatus using same
CN109891913B (zh) * 2016-08-24 2022-02-18 领先仿生公司 用于通过保留耳间水平差异来促进耳间水平差异感知的系统和方法
US9913061B1 (en) 2016-08-29 2018-03-06 The Directv Group, Inc. Methods and systems for rendering binaural audio content
US10848899B2 (en) * 2016-10-13 2020-11-24 Philip Scott Lyren Binaural sound in visual entertainment media
WO2018073759A1 (fr) 2016-10-19 2018-04-26 Audible Reality Inc. Système et procédé de génération d'une image audio
EP3547718A4 (fr) 2016-11-25 2019-11-13 Sony Corporation Dispositif de reproduction, procédé de reproduction, dispositif de traitement d'informations, procédé de traitement d'informations, et programme
KR102502383B1 (ko) * 2017-03-27 2023-02-23 가우디오랩 주식회사 오디오 신호 처리 방법 및 장치
US10880649B2 (en) 2017-09-29 2020-12-29 Apple Inc. System to move sound into and out of a listener's head using a virtual acoustic system
US10206055B1 (en) * 2017-12-28 2019-02-12 Verizon Patent And Licensing Inc. Methods and systems for generating spatialized audio during a virtual experience
US10390171B2 (en) 2018-01-07 2019-08-20 Creative Technology Ltd Method for generating customized spatial audio with head tracking
KR102483470B1 (ko) * 2018-02-13 2023-01-02 한국전자통신연구원 다중 렌더링 방식을 이용하는 입체 음향 생성 장치 및 입체 음향 생성 방법, 그리고 입체 음향 재생 장치 및 입체 음향 재생 방법
US10186247B1 (en) * 2018-03-13 2019-01-22 The Nielsen Company (Us), Llc Methods and apparatus to extract a pitch-independent timbre attribute from a media signal
US20210112287A1 (en) * 2018-04-11 2021-04-15 Lg Electronics Inc. Method and apparatus for transmitting or receiving metadata of audio in wireless communication system
CN109005496A (zh) * 2018-07-26 2018-12-14 西北工业大学 一种hrtf中垂面方位增强方法
WO2020044244A1 (fr) 2018-08-29 2020-03-05 Audible Reality Inc. Système et procédé de commande d'un moteur audio tridimensionnel
US10856097B2 (en) 2018-09-27 2020-12-01 Sony Corporation Generating personalized end user head-related transfer function (HRTV) using panoramic images of ear
US11503423B2 (en) * 2018-10-25 2022-11-15 Creative Technology Ltd Systems and methods for modifying room characteristics for spatial audio rendering over headphones
US10798515B2 (en) 2019-01-30 2020-10-06 Facebook Technologies, Llc Compensating for effects of headset on head related transfer functions
US11113092B2 (en) * 2019-02-08 2021-09-07 Sony Corporation Global HRTF repository
DE112020001017T5 (de) * 2019-03-01 2021-11-18 Google LLC (n.d.Ges.d.Staates Delaware) Verfahren zur Modellierung der akustischen Effekte des menschlichen Kopfes
US11451907B2 (en) 2019-05-29 2022-09-20 Sony Corporation Techniques combining plural head-related transfer function (HRTF) spheres to place audio objects
US11347832B2 (en) 2019-06-13 2022-05-31 Sony Corporation Head related transfer function (HRTF) as biometric authentication
US20220295213A1 (en) * 2019-08-02 2022-09-15 Sony Group Corporation Signal processing device, signal processing method, and program
WO2021041140A1 (fr) * 2019-08-27 2021-03-04 Anagnos Daniel P Dispositif de casque d'écoute pour reproduire un son tridimensionnel dans celui-ci, et procédé associé
CN112449262A (zh) * 2019-09-05 2021-03-05 哈曼国际工业有限公司 用于实现头相关传递函数的自适应的方法及系统
US11212631B2 (en) * 2019-09-16 2021-12-28 Gaudio Lab, Inc. Method for generating binaural signals from stereo signals using upmixing binauralization, and apparatus therefor
US11146908B2 (en) 2019-10-24 2021-10-12 Sony Corporation Generating personalized end user head-related transfer function (HRTF) from generic HRTF
US11070930B2 (en) 2019-11-12 2021-07-20 Sony Corporation Generating personalized end user room-related transfer function (RRTF)
EP3879856A1 (fr) * 2020-03-13 2021-09-15 FRAUNHOFER-GESELLSCHAFT zur Förderung der angewandten Forschung e.V. Appareil et procédé de synthèse d'une source sonore étendue spatialement à l'aide d'éléments d'informations de repère
GB2598960A (en) * 2020-09-22 2022-03-23 Nokia Technologies Oy Parametric spatial audio rendering with near-field effect
CN113068112B (zh) * 2021-03-01 2022-10-14 深圳市悦尔声学有限公司 声场重现中仿真系数向量信息的获取算法及其应用
WO2023059838A1 (fr) * 2021-10-08 2023-04-13 Dolby Laboratories Licensing Corporation Suivi de tête d'audio binaural ajusté
CN113821190B (zh) * 2021-11-25 2022-03-15 广州酷狗计算机科技有限公司 音频播放方法、装置、设备及存储介质
US11770670B2 (en) * 2022-01-13 2023-09-26 Meta Platforms Technologies, Llc Generating spatial audio and cross-talk cancellation for high-frequency glasses playback and low-frequency external playback
CN115412808B (zh) * 2022-09-05 2024-04-02 天津大学 基于个性化头相关传递函数的虚拟听觉重放方法及系统

Family Cites Families (34)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
US4817149A (en) 1987-01-22 1989-03-28 American Natural Sound Company Three-dimensional auditory display apparatus and method utilizing enhanced bionic emulation of human binaural sound localization
DE3840766C2 (de) 1987-12-10 1993-11-18 Goerike Rudolf Stereophone Aufnahmevorrichtung
EP0746960B1 (fr) 1994-02-25 1999-08-04 Henrik Moller Synthese binaurale, fonctions de transfert concernant une tete, et leurs utilisations
US5729612A (en) 1994-08-05 1998-03-17 Aureal Semiconductor Inc. Method and apparatus for measuring head-related transfer functions
US7085393B1 (en) 1998-11-13 2006-08-01 Agere Systems Inc. Method and apparatus for regularizing measured HRTF for smooth 3D digital audio
GB2337676B (en) 1998-05-22 2003-02-26 Central Research Lab Ltd Method of modifying a filter for implementing a head-related transfer function
GB9813973D0 (en) 1998-06-30 1998-08-26 Univ Stirling Interactive directional hearing aid
US6996244B1 (en) * 1998-08-06 2006-02-07 Vulcan Patents Llc Estimation of head-related transfer functions for spatial sound representative
US6223090B1 (en) 1998-08-24 2001-04-24 The United States Of America As Represented By The Secretary Of The Air Force Manikin positioning for acoustic measuring
GB2351213B (en) 1999-05-29 2003-08-27 Central Research Lab Ltd A method of modifying one or more original head related transfer functions
GB2369976A (en) 2000-12-06 2002-06-12 Central Research Lab Ltd A method of synthesising an averaged diffuse-field head-related transfer function
IL141822A (en) 2001-03-05 2007-02-11 Haim Levy A method and system for imitating a 3D audio environment
US20030202665A1 (en) 2002-04-24 2003-10-30 Bo-Ting Lin Implementation method of 3D audio
US7333622B2 (en) 2002-10-18 2008-02-19 The Regents Of The University Of California Dynamic binaural sound capture and reproduction
DE60336398D1 (de) 2003-10-10 2011-04-28 Harman Becker Automotive Sys System und Verfahren zur Bestimmung der Position einer Schallquelle
JP2005223713A (ja) 2004-02-06 2005-08-18 Sony Corp 音響再生装置、音響再生方法
US8638946B1 (en) 2004-03-16 2014-01-28 Genaudio, Inc. Method and apparatus for creating spatialized sound
US20060013409A1 (en) 2004-07-16 2006-01-19 Sensimetrics Corporation Microphone-array processing to generate directional cues in an audio signal
AU2005282680A1 (en) 2004-09-03 2006-03-16 Parker Tsuhako Method and apparatus for producing a phantom three-dimensional sound space with recorded sound
US20090041254A1 (en) 2005-10-20 2009-02-12 Personal Audio Pty Ltd Spatial audio simulation
GB0601110D0 (en) 2006-01-19 2006-03-01 Cho Youngjae Additional pinna and pinna-hollow filler for the head-related transfer function
JP5285626B2 (ja) 2007-03-01 2013-09-11 ジェリー・マハバブ 音声空間化及び環境シミュレーション
KR100818660B1 (ko) 2007-03-22 2008-04-02 광주과학기술원 근거리 모델을 위한 3차원 음향 생성 장치
JP5752414B2 (ja) 2007-06-26 2015-07-22 コーニンクレッカ フィリップス エヌ ヴェ バイノーラル型オブジェクト指向オーディオデコーダ
UA101542C2 (ru) 2008-12-15 2013-04-10 Долби Лабораторис Лайсензин Корпорейшн Виртуализатор окружающего звука с динамическим сжатием диапазона и способ
US8428269B1 (en) 2009-05-20 2013-04-23 The United States Of America As Represented By The Secretary Of The Air Force Head related transfer function (HRTF) enhancement for improved vertical-polar localization in spatial audio systems
JP5533248B2 (ja) 2010-05-20 2014-06-25 ソニー株式会社 音声信号処理装置および音声信号処理方法
CN101909236A (zh) 2010-07-12 2010-12-08 华南理工大学 用于近场hrtf测量的球形正十二面体声源及设计方法
RU2589377C2 (ru) * 2010-07-22 2016-07-10 Конинклейке Филипс Электроникс Н.В. Система и способ для воспроизведения звука
US8644520B2 (en) 2010-10-14 2014-02-04 Lockheed Martin Corporation Morphing of aural impulse response signatures to obtain intermediate aural impulse response signals
US9154896B2 (en) 2010-12-22 2015-10-06 Genaudio, Inc. Audio spatialization and environment simulation
US9131305B2 (en) * 2012-01-17 2015-09-08 LI Creative Technologies, Inc. Configurable three-dimensional sound system
JP5954147B2 (ja) * 2012-12-07 2016-07-20 ソニー株式会社 機能制御装置およびプログラム
WO2015134658A1 (fr) * 2014-03-06 2015-09-11 Dolby Laboratories Licensing Corporation Modélisation structurale de la réponse impulsionnelle relative à la tête

Cited By (2)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN112425187A (zh) * 2018-05-18 2021-02-26 诺基亚技术有限公司 用于实现头部跟踪耳机的方法和装置
CN112425187B (zh) * 2018-05-18 2022-09-02 诺基亚技术有限公司 用于空间音频渲染的装置、方法和存储介质

Also Published As

Publication number Publication date
WO2015134658A1 (fr) 2015-09-11
US20170094440A1 (en) 2017-03-30
US10142761B2 (en) 2018-11-27
EP3114859B1 (fr) 2018-05-09

Similar Documents

Publication Publication Date Title
EP3114859B1 (fr) Modélisation structurale de la réponse impulsionnelle relative à la tête
US11681490B2 (en) Binaural rendering for headphones using metadata processing
KR101627652B1 (ko) 바이노럴 렌더링을 위한 오디오 신호 처리 장치 및 방법
CN107018460B (zh) 具有头部跟踪的双耳头戴式耳机呈现
EP3311593B1 (fr) Reproduction audio binaurale
US9560467B2 (en) 3D immersive spatial audio systems and methods
KR101627647B1 (ko) 바이노럴 렌더링을 위한 오디오 신호 처리 장치 및 방법
US9197977B2 (en) Audio spatialization and environment simulation
US8270616B2 (en) Virtual surround for headphones and earbuds headphone externalization system
US9769589B2 (en) Method of improving externalization of virtual surround sound
US10341799B2 (en) Impedance matching filters and equalization for headphone surround rendering
US20120213375A1 (en) Audio Spatialization and Environment Simulation
JP2020506639A (ja) オーディオ信号処理方法及び装置
JP2018500816A (ja) ヘッドホンを通じて頭部外面化３ｄオーディオを生成するシステム及び方法
Frank How to make Ambisonics sound good
US20210067891A1 (en) Headphone Device for Reproducing Three-Dimensional Sound Therein, and Associated Method
US20240056760A1 (en) Binaural signal post-processing
Avendano Virtual spatial sound
Kim et al. 3D Sound Techniques for Sound Source Elevation in a Loudspeaker Listening Environment

Legal Events

Date Code Title Description
STAA Information on the status of an ep patent application or granted ep patent

Free format text: STATUS: THE INTERNATIONAL PUBLICATION HAS BEEN MADE

PUAI Public reference made under article 153(3) epc to a published international application that has entered the european phase

Free format text: ORIGINAL CODE: 0009012

STAA Information on the status of an ep patent application or granted ep patent

Free format text: STATUS: REQUEST FOR EXAMINATION WAS MADE

17P Request for examination filed

Effective date: 20161006

AK Designated contracting states

Kind code of ref document: A1

Designated state(s): AL AT BE BG CH CY CZ DE DK EE ES FI FR GB GR HR HU IE IS IT LI LT LU LV MC MK MT NL NO PL PT RO RS SE SI SK SM TR

AX Request for extension of the european patent

Extension state: BA ME

DAV Request for validation of the european patent (deleted)
DAX Request for extension of the european patent (deleted)
GRAP Despatch of communication of intention to grant a patent

Free format text: ORIGINAL CODE: EPIDOSNIGR1

STAA Information on the status of an ep patent application or granted ep patent

Free format text: STATUS: GRANT OF PATENT IS INTENDED

INTG Intention to grant announced

Effective date: 20180102

GRAS Grant fee paid

Free format text: ORIGINAL CODE: EPIDOSNIGR3

GRAA (expected) grant

Free format text: ORIGINAL CODE: 0009210

STAA Information on the status of an ep patent application or granted ep patent

Free format text: STATUS: THE PATENT HAS BEEN GRANTED

AK Designated contracting states

Kind code of ref document: B1

Designated state(s): AL AT BE BG CH CY CZ DE DK EE ES FI FR GB GR HR HU IE IS IT LI LT LU LV MC MK MT NL NO PL PT RO RS SE SI SK SM TR

REG Reference to a national code

Ref country code: GB

Ref legal event code: FG4D

REG Reference to a national code

Ref country code: CH

Ref legal event code: EP

Ref country code: AT

Ref legal event code: REF

Ref document number: 998594

Country of ref document: AT

Kind code of ref document: T

Effective date: 20180515

REG Reference to a national code

Ref country code: IE

Ref legal event code: FG4D

REG Reference to a national code

Ref country code: DE

Ref legal event code: R096

Ref document number: 602015010933

Country of ref document: DE

REG Reference to a national code

Ref country code: NL

Ref legal event code: MP

Effective date: 20180509

REG Reference to a national code

Ref country code: LT

Ref legal event code: MG4D

PG25 Lapsed in a contracting state [announced via postgrant information from national office to epo]

Ref country code: ES

Free format text: LAPSE BECAUSE OF FAILURE TO SUBMIT A TRANSLATION OF THE DESCRIPTION OR TO PAY THE FEE WITHIN THE PRESCRIBED TIME-LIMIT

Effective date: 20180509

Ref country code: SE

Free format text: LAPSE BECAUSE OF FAILURE TO SUBMIT A TRANSLATION OF THE DESCRIPTION OR TO PAY THE FEE WITHIN THE PRESCRIBED TIME-LIMIT

Effective date: 20180509

Ref country code: FI

Free format text: LAPSE BECAUSE OF FAILURE TO SUBMIT A TRANSLATION OF THE DESCRIPTION OR TO PAY THE FEE WITHIN THE PRESCRIBED TIME-LIMIT

Effective date: 20180509

Ref country code: BG

Free format text: LAPSE BECAUSE OF FAILURE TO SUBMIT A TRANSLATION OF THE DESCRIPTION OR TO PAY THE FEE WITHIN THE PRESCRIBED TIME-LIMIT

Effective date: 20180809

Ref country code: NO

Free format text: LAPSE BECAUSE OF FAILURE TO SUBMIT A TRANSLATION OF THE DESCRIPTION OR TO PAY THE FEE WITHIN THE PRESCRIBED TIME-LIMIT

Effective date: 20180809

Ref country code: LT

Free format text: LAPSE BECAUSE OF FAILURE TO SUBMIT A TRANSLATION OF THE DESCRIPTION OR TO PAY THE FEE WITHIN THE PRESCRIBED TIME-LIMIT

Effective date: 20180509

PG25 Lapsed in a contracting state [announced via postgrant information from national office to epo]

Ref country code: HR

Free format text: LAPSE BECAUSE OF FAILURE TO SUBMIT A TRANSLATION OF THE DESCRIPTION OR TO PAY THE FEE WITHIN THE PRESCRIBED TIME-LIMIT

Effective date: 20180509

Ref country code: NL

Free format text: LAPSE BECAUSE OF FAILURE TO SUBMIT A TRANSLATION OF THE DESCRIPTION OR TO PAY THE FEE WITHIN THE PRESCRIBED TIME-LIMIT

Effective date: 20180509

Ref country code: LV

Free format text: LAPSE BECAUSE OF FAILURE TO SUBMIT A TRANSLATION OF THE DESCRIPTION OR TO PAY THE FEE WITHIN THE PRESCRIBED TIME-LIMIT

Effective date: 20180509

Ref country code: GR

Free format text: LAPSE BECAUSE OF FAILURE TO SUBMIT A TRANSLATION OF THE DESCRIPTION OR TO PAY THE FEE WITHIN THE PRESCRIBED TIME-LIMIT

Effective date: 20180810

Ref country code: RS

Free format text: LAPSE BECAUSE OF FAILURE TO SUBMIT A TRANSLATION OF THE DESCRIPTION OR TO PAY THE FEE WITHIN THE PRESCRIBED TIME-LIMIT

Effective date: 20180509

REG Reference to a national code

Ref country code: AT

Ref legal event code: MK05

Ref document number: 998594

Country of ref document: AT

Kind code of ref document: T

Effective date: 20180509

PG25 Lapsed in a contracting state [announced via postgrant information from national office to epo]

Ref country code: CZ

Free format text: LAPSE BECAUSE OF FAILURE TO SUBMIT A TRANSLATION OF THE DESCRIPTION OR TO PAY THE FEE WITHIN THE PRESCRIBED TIME-LIMIT

Effective date: 20180509

Ref country code: RO

Free format text: LAPSE BECAUSE OF FAILURE TO SUBMIT A TRANSLATION OF THE DESCRIPTION OR TO PAY THE FEE WITHIN THE PRESCRIBED TIME-LIMIT

Effective date: 20180509

Ref country code: AT

Free format text: LAPSE BECAUSE OF FAILURE TO SUBMIT A TRANSLATION OF THE DESCRIPTION OR TO PAY THE FEE WITHIN THE PRESCRIBED TIME-LIMIT

Effective date: 20180509

Ref country code: EE

Free format text: LAPSE BECAUSE OF FAILURE TO SUBMIT A TRANSLATION OF THE DESCRIPTION OR TO PAY THE FEE WITHIN THE PRESCRIBED TIME-LIMIT

Effective date: 20180509

Ref country code: DK

Free format text: LAPSE BECAUSE OF FAILURE TO SUBMIT A TRANSLATION OF THE DESCRIPTION OR TO PAY THE FEE WITHIN THE PRESCRIBED TIME-LIMIT

Effective date: 20180509

Ref country code: PL

Free format text: LAPSE BECAUSE OF FAILURE TO SUBMIT A TRANSLATION OF THE DESCRIPTION OR TO PAY THE FEE WITHIN THE PRESCRIBED TIME-LIMIT

Effective date: 20180509

Ref country code: SK

Free format text: LAPSE BECAUSE OF FAILURE TO SUBMIT A TRANSLATION OF THE DESCRIPTION OR TO PAY THE FEE WITHIN THE PRESCRIBED TIME-LIMIT

Effective date: 20180509

REG Reference to a national code

Ref country code: DE

Ref legal event code: R097

Ref document number: 602015010933

Country of ref document: DE

PG25 Lapsed in a contracting state [announced via postgrant information from national office to epo]

Ref country code: SM

Free format text: LAPSE BECAUSE OF FAILURE TO SUBMIT A TRANSLATION OF THE DESCRIPTION OR TO PAY THE FEE WITHIN THE PRESCRIBED TIME-LIMIT

Effective date: 20180509

Ref country code: IT

Free format text: LAPSE BECAUSE OF FAILURE TO SUBMIT A TRANSLATION OF THE DESCRIPTION OR TO PAY THE FEE WITHIN THE PRESCRIBED TIME-LIMIT

Effective date: 20180509

PLBE No opposition filed within time limit

Free format text: ORIGINAL CODE: 0009261

STAA Information on the status of an ep patent application or granted ep patent

Free format text: STATUS: NO OPPOSITION FILED WITHIN TIME LIMIT

26N No opposition filed

Effective date: 20190212

PG25 Lapsed in a contracting state [announced via postgrant information from national office to epo]

Ref country code: SI

Free format text: LAPSE BECAUSE OF FAILURE TO SUBMIT A TRANSLATION OF THE DESCRIPTION OR TO PAY THE FEE WITHIN THE PRESCRIBED TIME-LIMIT

Effective date: 20180509

PG25 Lapsed in a contracting state [announced via postgrant information from national office to epo]

Ref country code: MC

Free format text: LAPSE BECAUSE OF FAILURE TO SUBMIT A TRANSLATION OF THE DESCRIPTION OR TO PAY THE FEE WITHIN THE PRESCRIBED TIME-LIMIT

Effective date: 20180509

REG Reference to a national code

Ref country code: CH

Ref legal event code: PL

PG25 Lapsed in a contracting state [announced via postgrant information from national office to epo]

Ref country code: AL

Free format text: LAPSE BECAUSE OF FAILURE TO SUBMIT A TRANSLATION OF THE DESCRIPTION OR TO PAY THE FEE WITHIN THE PRESCRIBED TIME-LIMIT

Effective date: 20180509

Ref country code: LU

Free format text: LAPSE BECAUSE OF NON-PAYMENT OF DUE FEES

Effective date: 20190304

REG Reference to a national code

Ref country code: BE

Ref legal event code: MM

Effective date: 20190331

PG25 Lapsed in a contracting state [announced via postgrant information from national office to epo]

Ref country code: IE

Free format text: LAPSE BECAUSE OF NON-PAYMENT OF DUE FEES

Effective date: 20190304

Ref country code: CH

Free format text: LAPSE BECAUSE OF NON-PAYMENT OF DUE FEES

Effective date: 20190331

Ref country code: LI

Free format text: LAPSE BECAUSE OF NON-PAYMENT OF DUE FEES

Effective date: 20190331

PG25 Lapsed in a contracting state [announced via postgrant information from national office to epo]

Ref country code: BE

Free format text: LAPSE BECAUSE OF NON-PAYMENT OF DUE FEES

Effective date: 20190331

PG25 Lapsed in a contracting state [announced via postgrant information from national office to epo]

Ref country code: TR

Free format text: LAPSE BECAUSE OF FAILURE TO SUBMIT A TRANSLATION OF THE DESCRIPTION OR TO PAY THE FEE WITHIN THE PRESCRIBED TIME-LIMIT

Effective date: 20180509

PG25 Lapsed in a contracting state [announced via postgrant information from national office to epo]

Ref country code: PT

Free format text: LAPSE BECAUSE OF FAILURE TO SUBMIT A TRANSLATION OF THE DESCRIPTION OR TO PAY THE FEE WITHIN THE PRESCRIBED TIME-LIMIT

Effective date: 20180910

Ref country code: MT

Free format text: LAPSE BECAUSE OF NON-PAYMENT OF DUE FEES

Effective date: 20190304

PG25 Lapsed in a contracting state [announced via postgrant information from national office to epo]

Ref country code: CY

Free format text: LAPSE BECAUSE OF FAILURE TO SUBMIT A TRANSLATION OF THE DESCRIPTION OR TO PAY THE FEE WITHIN THE PRESCRIBED TIME-LIMIT

Effective date: 20180509

PG25 Lapsed in a contracting state [announced via postgrant information from national office to epo]

Ref country code: IS

Free format text: LAPSE BECAUSE OF FAILURE TO SUBMIT A TRANSLATION OF THE DESCRIPTION OR TO PAY THE FEE WITHIN THE PRESCRIBED TIME-LIMIT

Effective date: 20180909

PG25 Lapsed in a contracting state [announced via postgrant information from national office to epo]

Ref country code: HU

Free format text: LAPSE BECAUSE OF FAILURE TO SUBMIT A TRANSLATION OF THE DESCRIPTION OR TO PAY THE FEE WITHIN THE PRESCRIBED TIME-LIMIT; INVALID AB INITIO

Effective date: 20150304

PG25 Lapsed in a contracting state [announced via postgrant information from national office to epo]

Ref country code: MK

Free format text: LAPSE BECAUSE OF FAILURE TO SUBMIT A TRANSLATION OF THE DESCRIPTION OR TO PAY THE FEE WITHIN THE PRESCRIBED TIME-LIMIT

Effective date: 20180509

PGFP Annual fee paid to national office [announced via postgrant information from national office to epo]

Ref country code: FR

Payment date: 20230222

Year of fee payment: 9

PGFP Annual fee paid to national office [announced via postgrant information from national office to epo]

Ref country code: GB

Payment date: 20230222

Year of fee payment: 9

Ref country code: DE

Payment date: 20230221

Year of fee payment: 9

P01 Opt-out of the competence of the unified patent court (upc) registered

Effective date: 20230513