EP2546806A3 - Image based rendering for AR - enabling user generation of 3D content - Google Patents

Image based rendering for AR - enabling user generation of 3D content Download PDF

Info

Publication number
EP2546806A3
EP2546806A3 EP12175950.0A EP12175950A EP2546806A3 EP 2546806 A3 EP2546806 A3 EP 2546806A3 EP 12175950 A EP12175950 A EP 12175950A EP 2546806 A3 EP2546806 A3 EP 2546806A3
Authority
EP
European Patent Office
Prior art keywords
real
marker
augmented reality
content
image based
Prior art date
Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
Granted
Application number
EP12175950.0A
Other languages
German (de)
French (fr)
Other versions
EP2546806B1 (en
EP2546806A2 (en
Inventor
Rahul Swaminathan
Renato Agurto
Simon BURKARD
Andreas Grothe
Ingo Wolf
Current Assignee (The listed assignees may be inaccurate. Google has not performed a legal analysis and makes no representation or warranty as to the accuracy of the list.)
Deutsche Telekom AG
Original Assignee
Technische Universitaet Berlin
Deutsche Telekom AG
Priority date (The priority date is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the date listed.)
Filing date
Publication date
Application filed by Technische Universitaet Berlin, Deutsche Telekom AG filed Critical Technische Universitaet Berlin
Priority to EP12175950.0A priority Critical patent/EP2546806B1/en
Publication of EP2546806A2 publication Critical patent/EP2546806A2/en
Publication of EP2546806A3 publication Critical patent/EP2546806A3/en
Application granted granted Critical
Publication of EP2546806B1 publication Critical patent/EP2546806B1/en
Active legal-status Critical Current
Anticipated expiration legal-status Critical

Links

Classifications

    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
    • G06T19/00—Manipulating 3D models or images for computer graphics
    • G06T19/006—Mixed reality
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
    • G06T15/00—3D [Three Dimensional] image rendering
    • G06T15/10—Geometric effects
    • G06T15/20—Perspective computation
    • G06T15/205—Image-based rendering
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
    • G06T17/00—Three dimensional [3D] modelling, e.g. data description of 3D objects

Landscapes

  • Engineering & Computer Science (AREA)
  • Physics & Mathematics (AREA)
  • Theoretical Computer Science (AREA)
  • Computer Graphics (AREA)
  • General Physics & Mathematics (AREA)
  • Geometry (AREA)
  • Software Systems (AREA)
  • Computing Systems (AREA)
  • Computer Hardware Design (AREA)
  • General Engineering & Computer Science (AREA)
  • Processing Or Creating Images (AREA)

Abstract

The present invention relates to an augmented reality system and a method of augmented reality, and in particular to an augmented reality system and a method of augmented reality allowing for measurement, 3D modeling and visualization of real world objects. The method for generating a 3D model representation of a real-world object comprises the steps of: obtaining a picture of a real-world object together with a marker, the object and the marker being embedded in a real scene; detecting the marker in the picture; measuring the dimensions of a selected sector in 3D using the information provided by the marker detected in the second step; creating a 3D model representation of the real-world object.
EP12175950.0A 2011-07-11 2012-07-11 Image based rendering for ar - enabling user generation of 3d content Active EP2546806B1 (en)

Priority Applications (1)

Application Number Priority Date Filing Date Title
EP12175950.0A EP2546806B1 (en) 2011-07-11 2012-07-11 Image based rendering for ar - enabling user generation of 3d content

Applications Claiming Priority (2)

Application Number Priority Date Filing Date Title
EP11173488 2011-07-11
EP12175950.0A EP2546806B1 (en) 2011-07-11 2012-07-11 Image based rendering for ar - enabling user generation of 3d content

Publications (3)

Publication Number Publication Date
EP2546806A2 EP2546806A2 (en) 2013-01-16
EP2546806A3 true EP2546806A3 (en) 2017-06-28
EP2546806B1 EP2546806B1 (en) 2019-05-08

Family

ID=46458385

Family Applications (1)

Application Number Title Priority Date Filing Date
EP12175950.0A Active EP2546806B1 (en) 2011-07-11 2012-07-11 Image based rendering for ar - enabling user generation of 3d content

Country Status (1)

Country Link
EP (1) EP2546806B1 (en)

Families Citing this family (19)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
WO2014182545A1 (en) * 2013-05-04 2014-11-13 Vupad Partners, Llc Virtual object scaling in augmented reality environment
US9728010B2 (en) * 2014-12-30 2017-08-08 Microsoft Technology Licensing, Llc Virtual representations of real-world objects
JP6618700B2 (en) * 2015-04-03 2019-12-11 株式会社サイバーウォーカー Information distribution method, information distribution program, and information distribution apparatus
CN114327043A (en) 2016-02-02 2022-04-12 索尼公司 Information processing apparatus, information processing method, and recording medium
JP6642153B2 (en) * 2016-03-16 2020-02-05 富士通株式会社 Three-dimensional measurement program, three-dimensional measurement method, and three-dimensional measurement system
EP3246879A1 (en) 2016-05-20 2017-11-22 Thomson Licensing Method and device for rendering an image of a scene comprising a real object and a virtual replica of the real object
DK180470B1 (en) 2017-08-31 2021-05-06 Apple Inc Systems, procedures, and graphical user interfaces for interacting with augmented and virtual reality environments
DK201870347A1 (en) 2018-01-24 2019-10-08 Apple Inc. Devices, Methods, and Graphical User Interfaces for System-Wide Behavior for 3D Models
CN110647301B (en) * 2018-06-27 2023-08-08 光宝电子(广州)有限公司 Augmented reality system and image display method thereof
CN109345614B (en) * 2018-09-20 2023-04-07 山东师范大学 Deep reinforcement learning-based animation simulation method for AR augmented reality large-screen interaction
US10573019B1 (en) * 2018-09-25 2020-02-25 Ebay Inc. Augmented reality digital content search and sizing techniques
EP3706076B1 (en) 2019-03-07 2021-02-24 Siemens Healthcare GmbH Method and device to determine the dimensions and distance of a number of objects in an environment
CN109949400B (en) * 2019-03-22 2023-02-03 南京可居网络科技有限公司 Shadow estimation and reconstruction method suitable for AR virtual soft-loading synthesis
CN109960872B (en) * 2019-03-22 2023-02-03 南京可居网络科技有限公司 AR virtual soft package matching management system and working method thereof
WO2021173824A1 (en) * 2020-02-28 2021-09-02 Magic Leap, Inc. 3d models for displayed 2d elements
JP6867071B1 (en) * 2020-05-08 2021-04-28 アイタックソリューションズ株式会社 Measuring device, measuring program and measuring method
JP6821222B1 (en) * 2020-05-08 2021-01-27 アイタックソリューションズ株式会社 Measuring device, measuring program and measuring method
JP6867070B1 (en) * 2020-05-08 2021-04-28 アイタックソリューションズ株式会社 Measuring device, measuring program and measuring method
CN113538452B (en) * 2021-05-27 2022-11-01 华中科技大学 Furniture layout and three-dimensional visualization method, device and equipment

Citations (2)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
WO2002069272A2 (en) * 2001-01-26 2002-09-06 Zaxel Systems, Inc. Real-time virtual viewpoint in simulated reality environment
EP1507235A1 (en) * 2003-08-15 2005-02-16 Werner G. Lonsing Method and apparatus for producing composite images which contain virtual objects

Patent Citations (2)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
WO2002069272A2 (en) * 2001-01-26 2002-09-06 Zaxel Systems, Inc. Real-time virtual viewpoint in simulated reality environment
EP1507235A1 (en) * 2003-08-15 2005-02-16 Werner G. Lonsing Method and apparatus for producing composite images which contain virtual objects

Non-Patent Citations (2)

* Cited by examiner, † Cited by third party
Title
ANDREAS HARTL ET AL: "Rapid reconstruction of small objects on mobile phones", COMPUTER VISION AND PATTERN RECOGNITION WORKSHOPS (CVPRW), 2011 IEEE COMPUTER SOCIETY CONFERENCE ON, IEEE, 20 June 2011 (2011-06-20), pages 20 - 27, XP031926543, ISBN: 978-1-4577-0529-8, DOI: 10.1109/CVPRW.2011.5981789 *
TOMMASO CHIARELLI ET AL: "A tool to achieve correct and precise measurements for oral implant planning and simulation", MEDICAL MEASUREMENTS AND APPLICATIONS, 2009. MEMEA 2009. IEEE INTERNATIONAL WORKSHOP ON, IEEE, PISCATAWAY, NJ, USA, 29 May 2009 (2009-05-29), pages 246 - 251, XP031492207, ISBN: 978-1-4244-3598-2 *

Also Published As

Publication number Publication date
EP2546806B1 (en) 2019-05-08
EP2546806A2 (en) 2013-01-16

Similar Documents

Publication Publication Date Title
EP2546806A3 (en) Image based rendering for AR - enabling user generation of 3D content
EP2402906A3 (en) Apparatus and method for providing 3D augmented reality
CN102568026B (en) Three-dimensional enhancing realizing method for multi-viewpoint free stereo display
WO2014153139A3 (en) Systems and methods for displaying a three-dimensional model from a photogrammetric scan
EP2378488A3 (en) Various methods and apparatuses for achieving augmented reality
EP2369553A3 (en) Three-dimensional template transformation method and apparatus
EP2662838A3 (en) Apparatus and method for augmented reality
GB2507830A8 (en) System and Method of Image Augmentation
EP4254334A3 (en) System and method for valve quantification
EP2755187A3 (en) 3d-animation effect generation method and system
EP2506222A3 (en) Augmented reality apparatus, method and program
WO2015031854A3 (en) Method and apparatus for representing physical scene
EP2509326A3 (en) Analysis of 3D video
EP2703873A3 (en) Information providing method and information providing vehicle therefor
WO2009157668A3 (en) Method and apparatus for outputting and displaying image data
WO2011029209A3 (en) Method and apparatus for generating and processing depth-enhanced images
WO2010138870A3 (en) Graphics processing unit with deferred vertex shading
JP2016512363A5 (en)
MX2011009232A (en) Three-dimensional visualization of images in the earth's subsurface.
JP2015125774A5 (en)
EP2418859A3 (en) Method for controlling depth of image and mobile terminal using the method
EP2459959A4 (en) Position and orientation calibration method and apparatus
WO2010102288A3 (en) Methods, systems, and computer readable media for shader-lamps based physical avatars of real and virtual people
WO2010118729A3 (en) Method for representing an animated object
WO2012108721A3 (en) Device and method for providing augmented reality using image information

Legal Events

Date Code Title Description
PUAI Public reference made under article 153(3) epc to a published international application that has entered the european phase

Free format text: ORIGINAL CODE: 0009012

AK Designated contracting states

Kind code of ref document: A2

Designated state(s): AL AT BE BG CH CY CZ DE DK EE ES FI FR GB GR HR HU IE IS IT LI LT LU LV MC MK MT NL NO PL PT RO RS SE SI SK SM TR

AX Request for extension of the european patent

Extension state: BA ME

PUAL Search report despatched

Free format text: ORIGINAL CODE: 0009013

AK Designated contracting states

Kind code of ref document: A3

Designated state(s): AL AT BE BG CH CY CZ DE DK EE ES FI FR GB GR HR HU IE IS IT LI LT LU LV MC MK MT NL NO PL PT RO RS SE SI SK SM TR

AX Request for extension of the european patent

Extension state: BA ME

RIC1 Information provided on ipc code assigned before grant

Ipc: G06T 17/00 20060101AFI20170522BHEP

Ipc: G06T 19/00 20110101ALI20170522BHEP

Ipc: G06T 15/20 20110101ALI20170522BHEP

STAA Information on the status of an ep patent application or granted ep patent

Free format text: STATUS: REQUEST FOR EXAMINATION WAS MADE

17P Request for examination filed

Effective date: 20171206

RBV Designated contracting states (corrected)

Designated state(s): AL AT BE BG CH CY CZ DE DK EE ES FI FR GB GR HR HU IE IS IT LI LT LU LV MC MK MT NL NO PL PT RO RS SE SI SK SM TR

RIC1 Information provided on ipc code assigned before grant

Ipc: G06T 15/20 20110101ALI20180802BHEP

Ipc: G06T 19/00 20110101ALI20180802BHEP

Ipc: G06T 17/00 20060101AFI20180802BHEP

GRAP Despatch of communication of intention to grant a patent

Free format text: ORIGINAL CODE: EPIDOSNIGR1

STAA Information on the status of an ep patent application or granted ep patent

Free format text: STATUS: GRANT OF PATENT IS INTENDED

INTG Intention to grant announced

Effective date: 20180921

RIN1 Information on inventor provided before grant (corrected)

Inventor name: AGURTO, RENATO

Inventor name: SWAMINATHAN, RAHUL

Inventor name: WOLF, INGO

Inventor name: GROTHE, ANDREAS

Inventor name: BURKARD, SIMON

GRAS Grant fee paid

Free format text: ORIGINAL CODE: EPIDOSNIGR3

GRAA (expected) grant

Free format text: ORIGINAL CODE: 0009210

STAA Information on the status of an ep patent application or granted ep patent

Free format text: STATUS: THE PATENT HAS BEEN GRANTED

RAP1 Party data changed (applicant data changed or rights of an application transferred)

Owner name: DEUTSCHE TELEKOM AG

AK Designated contracting states

Kind code of ref document: B1

Designated state(s): AL AT BE BG CH CY CZ DE DK EE ES FI FR GB GR HR HU IE IS IT LI LT LU LV MC MK MT NL NO PL PT RO RS SE SI SK SM TR

REG Reference to a national code

Ref country code: GB

Ref legal event code: FG4D

REG Reference to a national code

Ref country code: CH

Ref legal event code: EP

Ref country code: AT

Ref legal event code: REF

Ref document number: 1131397

Country of ref document: AT

Kind code of ref document: T

Effective date: 20190515

REG Reference to a national code

Ref country code: DE

Ref legal event code: R096

Ref document number: 602012059784

Country of ref document: DE

Ref country code: IE

Ref legal event code: FG4D

REG Reference to a national code

Ref country code: NL

Ref legal event code: MP

Effective date: 20190508

REG Reference to a national code

Ref country code: LT

Ref legal event code: MG4D

PG25 Lapsed in a contracting state [announced via postgrant information from national office to epo]

Ref country code: NO

Free format text: LAPSE BECAUSE OF FAILURE TO SUBMIT A TRANSLATION OF THE DESCRIPTION OR TO PAY THE FEE WITHIN THE PRESCRIBED TIME-LIMIT

Effective date: 20190808

Ref country code: FI

Free format text: LAPSE BECAUSE OF FAILURE TO SUBMIT A TRANSLATION OF THE DESCRIPTION OR TO PAY THE FEE WITHIN THE PRESCRIBED TIME-LIMIT

Effective date: 20190508

Ref country code: HR

Free format text: LAPSE BECAUSE OF FAILURE TO SUBMIT A TRANSLATION OF THE DESCRIPTION OR TO PAY THE FEE WITHIN THE PRESCRIBED TIME-LIMIT

Effective date: 20190508

Ref country code: LT

Free format text: LAPSE BECAUSE OF FAILURE TO SUBMIT A TRANSLATION OF THE DESCRIPTION OR TO PAY THE FEE WITHIN THE PRESCRIBED TIME-LIMIT

Effective date: 20190508

Ref country code: NL

Free format text: LAPSE BECAUSE OF FAILURE TO SUBMIT A TRANSLATION OF THE DESCRIPTION OR TO PAY THE FEE WITHIN THE PRESCRIBED TIME-LIMIT

Effective date: 20190508

Ref country code: PT

Free format text: LAPSE BECAUSE OF FAILURE TO SUBMIT A TRANSLATION OF THE DESCRIPTION OR TO PAY THE FEE WITHIN THE PRESCRIBED TIME-LIMIT

Effective date: 20190908

Ref country code: ES

Free format text: LAPSE BECAUSE OF FAILURE TO SUBMIT A TRANSLATION OF THE DESCRIPTION OR TO PAY THE FEE WITHIN THE PRESCRIBED TIME-LIMIT

Effective date: 20190508

Ref country code: SE

Free format text: LAPSE BECAUSE OF FAILURE TO SUBMIT A TRANSLATION OF THE DESCRIPTION OR TO PAY THE FEE WITHIN THE PRESCRIBED TIME-LIMIT

Effective date: 20190508

Ref country code: AL

Free format text: LAPSE BECAUSE OF FAILURE TO SUBMIT A TRANSLATION OF THE DESCRIPTION OR TO PAY THE FEE WITHIN THE PRESCRIBED TIME-LIMIT

Effective date: 20190508

PG25 Lapsed in a contracting state [announced via postgrant information from national office to epo]

Ref country code: GR

Free format text: LAPSE BECAUSE OF FAILURE TO SUBMIT A TRANSLATION OF THE DESCRIPTION OR TO PAY THE FEE WITHIN THE PRESCRIBED TIME-LIMIT

Effective date: 20190809

Ref country code: LV

Free format text: LAPSE BECAUSE OF FAILURE TO SUBMIT A TRANSLATION OF THE DESCRIPTION OR TO PAY THE FEE WITHIN THE PRESCRIBED TIME-LIMIT

Effective date: 20190508

Ref country code: RS

Free format text: LAPSE BECAUSE OF FAILURE TO SUBMIT A TRANSLATION OF THE DESCRIPTION OR TO PAY THE FEE WITHIN THE PRESCRIBED TIME-LIMIT

Effective date: 20190508

Ref country code: BG

Free format text: LAPSE BECAUSE OF FAILURE TO SUBMIT A TRANSLATION OF THE DESCRIPTION OR TO PAY THE FEE WITHIN THE PRESCRIBED TIME-LIMIT

Effective date: 20190808

REG Reference to a national code

Ref country code: AT

Ref legal event code: MK05

Ref document number: 1131397

Country of ref document: AT

Kind code of ref document: T

Effective date: 20190508

PG25 Lapsed in a contracting state [announced via postgrant information from national office to epo]

Ref country code: RO

Free format text: LAPSE BECAUSE OF FAILURE TO SUBMIT A TRANSLATION OF THE DESCRIPTION OR TO PAY THE FEE WITHIN THE PRESCRIBED TIME-LIMIT

Effective date: 20190508

Ref country code: SK

Free format text: LAPSE BECAUSE OF FAILURE TO SUBMIT A TRANSLATION OF THE DESCRIPTION OR TO PAY THE FEE WITHIN THE PRESCRIBED TIME-LIMIT

Effective date: 20190508

Ref country code: CZ

Free format text: LAPSE BECAUSE OF FAILURE TO SUBMIT A TRANSLATION OF THE DESCRIPTION OR TO PAY THE FEE WITHIN THE PRESCRIBED TIME-LIMIT

Effective date: 20190508

Ref country code: EE

Free format text: LAPSE BECAUSE OF FAILURE TO SUBMIT A TRANSLATION OF THE DESCRIPTION OR TO PAY THE FEE WITHIN THE PRESCRIBED TIME-LIMIT

Effective date: 20190508

Ref country code: AT

Free format text: LAPSE BECAUSE OF FAILURE TO SUBMIT A TRANSLATION OF THE DESCRIPTION OR TO PAY THE FEE WITHIN THE PRESCRIBED TIME-LIMIT

Effective date: 20190508

Ref country code: DK

Free format text: LAPSE BECAUSE OF FAILURE TO SUBMIT A TRANSLATION OF THE DESCRIPTION OR TO PAY THE FEE WITHIN THE PRESCRIBED TIME-LIMIT

Effective date: 20190508

REG Reference to a national code

Ref country code: DE

Ref legal event code: R097

Ref document number: 602012059784

Country of ref document: DE

PG25 Lapsed in a contracting state [announced via postgrant information from national office to epo]

Ref country code: SM

Free format text: LAPSE BECAUSE OF FAILURE TO SUBMIT A TRANSLATION OF THE DESCRIPTION OR TO PAY THE FEE WITHIN THE PRESCRIBED TIME-LIMIT

Effective date: 20190508

Ref country code: IT

Free format text: LAPSE BECAUSE OF FAILURE TO SUBMIT A TRANSLATION OF THE DESCRIPTION OR TO PAY THE FEE WITHIN THE PRESCRIBED TIME-LIMIT

Effective date: 20190508

Ref country code: MC

Free format text: LAPSE BECAUSE OF FAILURE TO SUBMIT A TRANSLATION OF THE DESCRIPTION OR TO PAY THE FEE WITHIN THE PRESCRIBED TIME-LIMIT

Effective date: 20190508

REG Reference to a national code

Ref country code: CH

Ref legal event code: PL

PLBE No opposition filed within time limit

Free format text: ORIGINAL CODE: 0009261

STAA Information on the status of an ep patent application or granted ep patent

Free format text: STATUS: NO OPPOSITION FILED WITHIN TIME LIMIT

PG25 Lapsed in a contracting state [announced via postgrant information from national office to epo]

Ref country code: TR

Free format text: LAPSE BECAUSE OF FAILURE TO SUBMIT A TRANSLATION OF THE DESCRIPTION OR TO PAY THE FEE WITHIN THE PRESCRIBED TIME-LIMIT

Effective date: 20190508

26N No opposition filed

Effective date: 20200211

REG Reference to a national code

Ref country code: BE

Ref legal event code: MM

Effective date: 20190731

PG25 Lapsed in a contracting state [announced via postgrant information from national office to epo]

Ref country code: PL

Free format text: LAPSE BECAUSE OF FAILURE TO SUBMIT A TRANSLATION OF THE DESCRIPTION OR TO PAY THE FEE WITHIN THE PRESCRIBED TIME-LIMIT

Effective date: 20190508

PG25 Lapsed in a contracting state [announced via postgrant information from national office to epo]

Ref country code: CH

Free format text: LAPSE BECAUSE OF NON-PAYMENT OF DUE FEES

Effective date: 20190731

Ref country code: BE

Free format text: LAPSE BECAUSE OF NON-PAYMENT OF DUE FEES

Effective date: 20190731

Ref country code: LI

Free format text: LAPSE BECAUSE OF NON-PAYMENT OF DUE FEES

Effective date: 20190731

Ref country code: LU

Free format text: LAPSE BECAUSE OF NON-PAYMENT OF DUE FEES

Effective date: 20190711

Ref country code: SI

Free format text: LAPSE BECAUSE OF FAILURE TO SUBMIT A TRANSLATION OF THE DESCRIPTION OR TO PAY THE FEE WITHIN THE PRESCRIBED TIME-LIMIT

Effective date: 20190508

PG25 Lapsed in a contracting state [announced via postgrant information from national office to epo]

Ref country code: IE

Free format text: LAPSE BECAUSE OF NON-PAYMENT OF DUE FEES

Effective date: 20190711

PG25 Lapsed in a contracting state [announced via postgrant information from national office to epo]

Ref country code: CY

Free format text: LAPSE BECAUSE OF FAILURE TO SUBMIT A TRANSLATION OF THE DESCRIPTION OR TO PAY THE FEE WITHIN THE PRESCRIBED TIME-LIMIT

Effective date: 20190508

PG25 Lapsed in a contracting state [announced via postgrant information from national office to epo]

Ref country code: IS

Free format text: LAPSE BECAUSE OF FAILURE TO SUBMIT A TRANSLATION OF THE DESCRIPTION OR TO PAY THE FEE WITHIN THE PRESCRIBED TIME-LIMIT

Effective date: 20190908

PG25 Lapsed in a contracting state [announced via postgrant information from national office to epo]

Ref country code: HU

Free format text: LAPSE BECAUSE OF FAILURE TO SUBMIT A TRANSLATION OF THE DESCRIPTION OR TO PAY THE FEE WITHIN THE PRESCRIBED TIME-LIMIT; INVALID AB INITIO

Effective date: 20120711

Ref country code: MT

Free format text: LAPSE BECAUSE OF FAILURE TO SUBMIT A TRANSLATION OF THE DESCRIPTION OR TO PAY THE FEE WITHIN THE PRESCRIBED TIME-LIMIT

Effective date: 20190508

PG25 Lapsed in a contracting state [announced via postgrant information from national office to epo]

Ref country code: MK

Free format text: LAPSE BECAUSE OF FAILURE TO SUBMIT A TRANSLATION OF THE DESCRIPTION OR TO PAY THE FEE WITHIN THE PRESCRIBED TIME-LIMIT

Effective date: 20190508

PGFP Annual fee paid to national office [announced via postgrant information from national office to epo]

Ref country code: GB

Payment date: 20230724

Year of fee payment: 12

PGFP Annual fee paid to national office [announced via postgrant information from national office to epo]

Ref country code: FR

Payment date: 20230724

Year of fee payment: 12

Ref country code: DE

Payment date: 20230720

Year of fee payment: 12