EP1321894A3 - Apparatus and method for representing 3-dimensional objects unsing depth images - Google Patents

Apparatus and method for representing 3-dimensional objects unsing depth images Download PDF

Info

Publication number
EP1321894A3
EP1321894A3 EP02258159A EP02258159A EP1321894A3 EP 1321894 A3 EP1321894 A3 EP 1321894A3 EP 02258159 A EP02258159 A EP 02258159A EP 02258159 A EP02258159 A EP 02258159A EP 1321894 A3 EP1321894 A3 EP 1321894A3
Authority
EP
European Patent Office
Prior art keywords
reference images
octreelmage
depthlmage
formats
image
Prior art date
Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
Granted
Application number
EP02258159A
Other languages
German (de)
French (fr)
Other versions
EP1321894A2 (en
EP1321894B1 (en
Inventor
In-Kyu Park
Alexander Olegovich Room 16 Zhirkov
Mahn-jin 307-602 Satbyeol Maeul Woobang Apt. Han
Current Assignee (The listed assignees may be inaccurate. Google has not performed a legal analysis and makes no representation or warranty as to the accuracy of the list.)
Samsung Electronics Co Ltd
Original Assignee
Samsung Electronics Co Ltd
Priority date (The priority date is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the date listed.)
Filing date
Publication date
Priority claimed from KR10-2002-0067970A external-priority patent/KR100446635B1/en
Application filed by Samsung Electronics Co Ltd filed Critical Samsung Electronics Co Ltd
Publication of EP1321894A2 publication Critical patent/EP1321894A2/en
Publication of EP1321894A3 publication Critical patent/EP1321894A3/en
Application granted granted Critical
Publication of EP1321894B1 publication Critical patent/EP1321894B1/en
Anticipated expiration legal-status Critical
Expired - Fee Related legal-status Critical Current

Links

Classifications

    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
    • G06T9/00—Image coding
    • G06T9/001—Model-based coding, e.g. wire frame
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
    • G06T15/00—3D [Three Dimensional] image rendering
    • G06T15/10—Geometric effects
    • G06T15/20—Perspective computation
    • G06T15/205—Image-based rendering
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
    • G06T17/00—Three dimensional [3D] modelling, e.g. data description of 3D objects
    • G06T17/005—Tree description, e.g. octree, quadtree
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
    • G06T9/00—Image coding
    • G06T9/40—Tree coding, e.g. quadtree, octree

Abstract

A family of node structures for representing 3-dimensional objects using depth image are provided. These node structures can be adopted into MPEG-4 AFX for conventional polygonal 3D representations. Main formats of the family are Depthlmage, PointTexture and Octreelmage. Depthlmage represents an object by a union of its reference images and corresponding depth maps. PointTexture represents the object as a set of colored points parameterized by projection onto a regular 2D grid. Octreelmage converts the same data into hierarchical octree-structured voxel model, set of compact reference images and a tree of voxel-image correspondence indices. Depthlmage and Octreelmage have animated versions, where reference images are replaced by videostreams. DIBR formats are very convenient for 3D model construction from 3D range-scanning and multiple source video data. MPEG-4 framework allows construction of a wide variety of representations from the main DIBR formats, providing flexible tools for effective work with 3D models. Compression of the DIBR formats is achieved by application of image (video) compression techniques to depth maps and reference images (videostreams).
EP02258159A 2001-11-27 2002-11-27 Apparatus and method for representing 3-dimensional objects using depth images Expired - Fee Related EP1321894B1 (en)

Applications Claiming Priority (10)

Application Number Priority Date Filing Date Title
US33316701P 2001-11-27 2001-11-27
US333167P 2001-11-27
US36254502P 2002-03-08 2002-03-08
US362545P 2002-03-08
US37656302P 2002-05-01 2002-05-01
US376563P 2002-05-01
US39530402P 2002-07-12 2002-07-12
US395304P 2002-07-12
KR2002067970 2002-11-04
KR10-2002-0067970A KR100446635B1 (en) 2001-11-27 2002-11-04 Apparatus and method for depth image-based representation of 3-dimensional object

Publications (3)

Publication Number Publication Date
EP1321894A2 EP1321894A2 (en) 2003-06-25
EP1321894A3 true EP1321894A3 (en) 2005-01-05
EP1321894B1 EP1321894B1 (en) 2011-11-09

Family

ID=27532386

Family Applications (1)

Application Number Title Priority Date Filing Date
EP02258159A Expired - Fee Related EP1321894B1 (en) 2001-11-27 2002-11-27 Apparatus and method for representing 3-dimensional objects using depth images

Country Status (4)

Country Link
EP (1) EP1321894B1 (en)
JP (2) JP3957620B2 (en)
CN (1) CN1261912C (en)
CA (1) CA2413056C (en)

Families Citing this family (60)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
KR100519780B1 (en) * 2004-02-17 2005-10-07 삼성전자주식회사 Method and apparatus for encoding and decoding 3D volume data
KR100695142B1 (en) 2004-03-08 2007-03-14 삼성전자주식회사 Adaptive nth power of 2 n-ary tree generating method, and method and apparatus for encoding and decoding 3D volume data using it
JP4759291B2 (en) * 2004-03-08 2011-08-31 三星電子株式会社 Adaptive binary tree generation method, and method and apparatus for encoding / decoding three-dimensional volume data using the same
KR100707206B1 (en) * 2005-04-11 2007-04-13 삼성전자주식회사 Depth Image-based Representation method for 3D objects, Modeling method and apparatus using it, and Rendering method and apparatus using the same
JP4650750B2 (en) * 2005-12-16 2011-03-16 株式会社Ｉｈｉ 3D shape data storage and display method and apparatus, and 3D shape measurement method and apparatus
CN101331381B (en) 2005-12-16 2011-08-24 株式会社Ihi Three-dimensional shape data aligning method and device
DE112006003363B4 (en) 2005-12-16 2016-05-04 Ihi Corporation Method and apparatus for identifying the self-position, and method and apparatus for measuring a three-dimensional shape
JP4368880B2 (en) * 2006-01-05 2009-11-18 シャープ株式会社 Image processing apparatus, image forming apparatus, image processing method, image processing program, and computer-readable recording medium
CN1946180B (en) * 2006-10-27 2010-05-12 北京航空航天大学 Three dimension mode compression coding method based on octree
WO2008123823A1 (en) * 2007-04-04 2008-10-16 Telefonaktiebolaget Lm Ericsson (Publ) Vector-based image processing
JP5380792B2 (en) * 2007-06-15 2014-01-08 株式会社Ｉｈｉ Object recognition method and apparatus
US8279222B2 (en) 2008-03-14 2012-10-02 Seiko Epson Corporation Processing graphics data for a stereoscopic display
US8456517B2 (en) * 2008-07-09 2013-06-04 Primesense Ltd. Integrated processor for 3D mapping
CN102106152A (en) 2008-07-24 2011-06-22 皇家飞利浦电子股份有限公司 Versatile 3-D picture format
EP2328337A4 (en) 2008-09-02 2011-08-10 Huawei Device Co Ltd 3d video communicating means, transmitting apparatus, system and image reconstructing means, system
CN101668219B (en) * 2008-09-02 2012-05-23 华为终端有限公司 Communication method, transmitting equipment and system for 3D video
KR20100128233A (en) * 2009-05-27 2010-12-07 삼성전자주식회사 Method and apparatus for processing video image
US20120050480A1 (en) * 2010-08-27 2012-03-01 Nambi Seshadri Method and system for generating three-dimensional video utilizing a monoscopic camera
US9983685B2 (en) 2011-01-17 2018-05-29 Mediatek Inc. Electronic apparatuses and methods for providing a man-machine interface (MMI)
US8670023B2 (en) * 2011-01-17 2014-03-11 Mediatek Inc. Apparatuses and methods for providing a 3D man-machine interface (MMI)
WO2013067674A1 (en) * 2011-11-07 2013-05-16 Thomson Licensing Predictive position encoding
CN104781852B (en) * 2012-09-21 2020-09-15 欧克里德私人有限公司 Computer drawing method for rendering three-dimensional scene
US20140092439A1 (en) * 2012-09-28 2014-04-03 Scott A. Krig Encoding images using a 3d mesh of polygons and corresponding textures
CA2909006C (en) * 2013-04-08 2022-07-26 Thomson Licensing Method for encoding and method for decoding a look-up table and corresponding devices
GB2515343B (en) * 2013-06-21 2018-02-07 Toshiba Res Europe Limited Methods and systems for generating a three dimensional representation of a subject
CN105791793A (en) * 2014-12-17 2016-07-20 光宝电子(广州)有限公司 Image processing method and electronic device
CA2999193A1 (en) * 2015-09-23 2017-03-30 Koninklijke Philips N.V. Generation of triangle mesh for a three dimensional image
GB2558314B (en) 2017-01-02 2020-07-29 Canon Kk Improved attribute mapping to encode and decode 3D models
US10373365B2 (en) * 2017-04-10 2019-08-06 Intel Corporation Topology shader technology
EP3429212A1 (en) * 2017-07-13 2019-01-16 Thomson Licensing A method and apparatus for encoding/decoding the geometry of a point cloud representing a 3d object
EP3428887A1 (en) * 2017-07-13 2019-01-16 Thomson Licensing Method and device for encoding a point cloud
EP3429207A1 (en) * 2017-07-13 2019-01-16 Thomson Licensing A method and apparatus for encoding/decoding a colored point cloud representing the geometry and colors of a 3d object
EP3429209A1 (en) * 2017-07-13 2019-01-16 Thomson Licensing A method and apparatus for encoding/decoding a point cloud representing a 3d object
EP3429211A1 (en) * 2017-07-13 2019-01-16 Thomson Licensing A method and apparatus for encoding/decoding the colors of a colored point cloud whose geometry is represented by an octree-based structure
EP3429208A1 (en) * 2017-07-13 2019-01-16 Thomson Licensing A method and apparatus for encoding/decoding a colored point cloud representing the geometry and colors of a 3d object
EP3467789A1 (en) * 2017-10-06 2019-04-10 Thomson Licensing A method and apparatus for reconstructing a point cloud representing a 3d object
US11062461B2 (en) * 2017-11-16 2021-07-13 Zoox, Inc. Pose determination from contact points
EP3716218A4 (en) * 2017-11-22 2021-03-31 Panasonic Intellectual Property Corporation of America Three-dimensional data encoding method, three-dimensional data decoding method, three-dimensional data encoding device and three-dimensional data decoding device
EP3493158A1 (en) * 2017-12-04 2019-06-05 Thomson Licensing A method and apparatus for encoding/decoding the geometry of a point cloud representing a 3d object
US10719982B2 (en) * 2017-12-26 2020-07-21 Htc Corporation Surface extrction method, apparatus, and non-transitory computer readable storage medium thereof
CN108460826B (en) * 2017-12-28 2022-04-15 深圳市创梦天地科技有限公司 3D model processing method and terminal
US11651523B2 (en) 2017-12-28 2023-05-16 Nokia Technologies Oy Apparatus, a method and a computer program for volumetric video
TWI815842B (en) * 2018-01-16 2023-09-21 日商索尼股份有限公司 Image processing device and method
CN111566703B (en) * 2018-01-17 2023-10-20 索尼公司 Image processing apparatus and method
US10853975B2 (en) 2018-01-26 2020-12-01 Sony Corporation Hybrid projection-based point cloud texture coding
US10748306B2 (en) * 2018-03-19 2020-08-18 Fuji Xerox Co., Ltd. Encoding device, decoding device, and non-transitory computer readable medium
CN110297597B (en) * 2018-03-23 2023-06-13 中国石油化工股份有限公司 Storage method and reading method of seismic data
EP3553745B1 (en) 2018-04-09 2021-09-01 BlackBerry Limited Methods and devices for binary entropy coding of point clouds
MX2020014247A (en) * 2018-06-25 2021-05-12 Huawei Tech Co Ltd Hybrid geometric coding of point clouds.
CN110796726B (en) * 2018-08-01 2022-04-05 深圳朗呈医疗科技有限公司 Three-dimensional imaging method and device and terminal equipment
CN109215119B (en) * 2018-09-18 2023-06-20 创新先进技术有限公司 Method and device for establishing three-dimensional model of damaged vehicle
JP2022051967A (en) * 2019-01-08 2022-04-04 ソニーグループ株式会社 Information processing device and information processing method
CN110070613B (en) * 2019-04-26 2022-12-06 东北大学 Large three-dimensional scene webpage display method based on model compression and asynchronous loading
CN111699684B (en) * 2019-06-14 2022-05-06 深圳市大疆创新科技有限公司 Three-dimensional data point coding and decoding method and device
CN112950753B (en) * 2019-12-11 2023-09-26 腾讯科技（深圳）有限公司 Virtual plant display method, device, equipment and storage medium
CN111161365B (en) * 2019-12-30 2023-05-09 珠海金山数字网络科技有限公司 Compression method and device for bone animation data
CN112991549A (en) * 2021-03-31 2021-06-18 上海商汤临港智能科技有限公司 Point cloud data processing method and device, computer equipment and storage medium
CN113297701B (en) * 2021-06-10 2022-12-20 清华大学深圳国际研究生院 Simulation data set generation method and device for multiple industrial part stacking scenes
CN113673116B (en) * 2021-09-01 2022-03-08 上海交通大学 Three-dimensional quasi-transportation acceleration method aiming at uniform geometric variable block method
CN113487690B (en) * 2021-09-08 2021-11-19 北京医百科技有限公司 Depth map processing method, system and storage medium

Non-Patent Citations (8)

* Cited by examiner, † Cited by third party
Title
"PDAM OF ISO/IEC 14496-1/AMD4", INTERNATIONAL ORGANIZATION FOR STANDARDIZATION - ORGANISATION INTERNATIONALE DE NORMALISATION, N4415, December 2001 (2001-12-01), pages I - X,10, XP001089813 *
BAYAKOVSKI Y ET AL: "Depth image-based representations for static and animated 3D objects", PROCEEDINGS 2002 INTERNATIONAL CONFERENCE ON IMAGE PROCESSING. ICIP 2002. ROCHESTER, NY, SEPT. 22 - 25, 2002, IEEE, US, vol. VOL. 2 OF 3, 22 September 2002 (2002-09-22), pages 25 - 28, XP010607501, ISBN: 0-7803-7622-6 *
DUAN J ET AL: "COMPRESSION OF THE LAYERED DEPTH IMAGE", DATA COMPRESSION CONFERENCE, PROCEEDINGS. DCC, IEEE COMPUTER SOCIETY PRESS, LOS ALAMITOS, CA, US, 27 March 2001 (2001-03-27), pages 331 - 340, XP001089854 *
OLIVEIRA M M ET AL: "RELIEF TEXTURE MAPPING", COMPUTER GRAPHICS. SIGGRAPH 2000 CONFERENCE PROCEEDINGS. NEW ORLEANS, LA, JULY 23 - 28, 2000, COMPUTER GRAPHICS PROCEEDINGS. SIGGRAPH, NEW YORK, NY : ACM, US, 23 July 2000 (2000-07-23), pages 359 - 368, XP001003575, ISBN: 1-58113-208-5 *
SCHMITT F ET AL: "3D color object reconstruction from 2D image sequences", PROC. OF INTERNATIONAL CONFERENCE ON IMAGE PROCESSING, 1999. ICIP 99. KOBE, JAPAN 24-28 OCT. 1999, PISCATAWAY, NJ, USA,IEEE, US, 24 October 1999 (1999-10-24), pages 65 - 69, XP010368848, ISBN: 0-7803-5467-2 *
SHADE J ET AL: "Layered depth images", COMPUTER GRAPHICS. SIGGRAPH 98 CONFERENCE PROCEEDINGS. ORLANDO, FL, JULY 19- - 24, 1998, COMPUTER GRAPHICS PROCEEDINGS. SIGGRAPH, NEW YORK, NY : ACM, US, 19 July 1998 (1998-07-19), pages 231 - 242, XP002270434, ISBN: 0-89791-999-8 *
TORIYAMA H: "CONTEXT NUMBER REDUCTION FOR ENTROPY CODING OF OCTREE REPRESENTED 3-D OBJECTS", IEICE TRANSACTIONS ON INFORMATION AND SYSTEMS, INSTITUTE OF ELECTRONICS INFORMATION AND COMM. ENG. TOKYO, JP, vol. E80-D, no. 2, 1 February 1997 (1997-02-01), pages 243 - 249, XP000723712, ISSN: 0916-8532 *
ZHIRKOV A: "Binary Volumetric Octree Representation for Image-Based Rendering", PROC. OF GRAPHICON'2001, 10 September 2001 (2001-09-10) - 17 September 2001 (2001-09-17), NIZHNY NOVGOROD, RUSSIA, XP002303951, Retrieved from the Internet <URL:http://graphics.cs.msu.su/en/publications/text/Zhirkov_2001.pdf> [retrieved on 20041102] *

Also Published As

Publication number Publication date
EP1321894A2 (en) 2003-06-25
JP4629005B2 (en) 2011-02-09
JP2003296755A (en) 2003-10-17
CA2413056C (en) 2009-02-10
CN1261912C (en) 2006-06-28
CA2413056A1 (en) 2003-05-27
EP1321894B1 (en) 2011-11-09
JP3957620B2 (en) 2007-08-15
CN1432969A (en) 2003-07-30
JP2006318503A (en) 2006-11-24

Similar Documents

Publication Publication Date Title
EP1321894A3 (en) Apparatus and method for representing 3-dimensional objects unsing depth images
EP1321893A3 (en) Node structure for representing 3-dimensional objects using depth image
CN103810685B (en) A kind of super-resolution processing method of depth map
Sinha et al. Interactive 3D architectural modeling from unordered photo collections
CA2581273C (en) System and method for processing video images
JP5128827B2 (en) Modeling apparatus and method based on 3D points
US6862024B2 (en) Enhancing textured range images using a 2D editor
US20030197806A1 (en) Single lens 3D camera
CA2391670A1 (en) Hierarchical image-based representation of still and animated three-dimensional object, method and apparatus for using this representation for the object rendering
RU2001118222A (en) Hierarchical image-based representation of a fixed and animated three-dimensional object, method and apparatus for using this representation to visualize an object
EP0952552A3 (en) Method for generating 2-D images from 3-D video data
WO2005010825A3 (en) Method and sytem for the three-dimensional surface reconstruction of an object
JP3964749B2 (en) A method for representing image-based rendering information in a 3D scene
KR102456302B1 (en) CityGML-based building object information generation method using 3D geometric object information, building object information generation system, and computer program therefor
US6963671B2 (en) Method for determining distances to a surface from a range image
US6792205B2 (en) Method for generating a textured range image
CA2423114A1 (en) Method for conducting analysis of two-dimensional images
JP2007507945A (en) 3D visualization
CN110148206B (en) Multi-space fusion method
CA2514655A1 (en) Apparatus and method for depth image-based representation of 3-dimensional object
KR20130003992A (en) Method of converting 3d image data and apparatus for the same
WO2017182448A1 (en) Method and device for determining a 3d model
KR100370366B1 (en) The image converting method using vanishing line
JP4102091B2 (en) Method for determining an image according to the magnitude of a gradient from a range image
JP2004013869A (en) Apparatus for generating three-dimensional shape, method therefor, and its program

Legal Events

Date Code Title Description
PUAI Public reference made under article 153(3) epc to a published international application that has entered the european phase

Free format text: ORIGINAL CODE: 0009012

17P Request for examination filed

Effective date: 20021224

AK Designated contracting states

Designated state(s): AT BE BG CH CY CZ DE DK EE ES FI FR GB GR IE IT LI LU MC NL PT SE SK TR

AX Request for extension of the european patent

Extension state: AL LT LV MK RO SI

PUAL Search report despatched

Free format text: ORIGINAL CODE: 0009013

RIC1 Information provided on ipc code assigned before grant

Ipc: 7G 06T 17/00 B

Ipc: 7G 06T 1/00 A

Ipc: 7G 06T 15/20 B

AK Designated contracting states

Kind code of ref document: A3

Designated state(s): AT BE BG CH CY CZ DE DK EE ES FI FR GB GR IE IT LI LU MC NL PT SE SK TR

AX Request for extension of the european patent

Extension state: AL LT LV MK RO SI

AKX Designation fees paid

Designated state(s): BE DE ES FR GB NL

17Q First examination report despatched

Effective date: 20050829

RTI1 Title (correction)

Free format text: APPARATUS AND METHOD FOR REPRESENTING 3-DIMENSIONAL OBJECTS USING DEPTH IMAGES

GRAP Despatch of communication of intention to grant a patent

Free format text: ORIGINAL CODE: EPIDOSNIGR1

GRAS Grant fee paid

Free format text: ORIGINAL CODE: EPIDOSNIGR3

GRAA (expected) grant

Free format text: ORIGINAL CODE: 0009210

AK Designated contracting states

Kind code of ref document: B1

Designated state(s): BE DE ES FR GB NL

REG Reference to a national code

Ref country code: GB

Ref legal event code: FG4D

REG Reference to a national code

Ref country code: DE

Ref legal event code: R096

Ref document number: 60241496

Country of ref document: DE

Effective date: 20120119

REG Reference to a national code

Ref country code: ES

Ref legal event code: FG2A

Ref document number: 2374066

Country of ref document: ES

Kind code of ref document: T3

Effective date: 20120213

REG Reference to a national code

Ref country code: NL

Ref legal event code: T3

PLBE No opposition filed within time limit

Free format text: ORIGINAL CODE: 0009261

STAA Information on the status of an ep patent application or granted ep patent

Free format text: STATUS: NO OPPOSITION FILED WITHIN TIME LIMIT

RAP2 Party data changed (patent owner data changed or rights of a patent transferred)

Owner name: SAMSUNG ELECTRONICS CO., LTD.

26N No opposition filed

Effective date: 20120810

REG Reference to a national code

Ref country code: DE

Ref legal event code: R097

Ref document number: 60241496

Country of ref document: DE

Effective date: 20120810

REG Reference to a national code

Ref country code: FR

Ref legal event code: PLFP

Year of fee payment: 14

PGFP Annual fee paid to national office [announced via postgrant information from national office to epo]

Ref country code: FR

Payment date: 20150625

Year of fee payment: 14

PGFP Annual fee paid to national office [announced via postgrant information from national office to epo]

Ref country code: GB

Payment date: 20151026

Year of fee payment: 14

Ref country code: DE

Payment date: 20151028

Year of fee payment: 14

PGFP Annual fee paid to national office [announced via postgrant information from national office to epo]

Ref country code: ES

Payment date: 20151105

Year of fee payment: 14

Ref country code: NL

Payment date: 20151022

Year of fee payment: 14

Ref country code: BE

Payment date: 20151022

Year of fee payment: 14

PG25 Lapsed in a contracting state [announced via postgrant information from national office to epo]

Ref country code: BE

Free format text: LAPSE BECAUSE OF NON-PAYMENT OF DUE FEES

Effective date: 20161130

REG Reference to a national code

Ref country code: DE

Ref legal event code: R119

Ref document number: 60241496

Country of ref document: DE

REG Reference to a national code

Ref country code: NL

Ref legal event code: MM

Effective date: 20161201

GBPC Gb: european patent ceased through non-payment of renewal fee

Effective date: 20161127

REG Reference to a national code

Ref country code: FR

Ref legal event code: ST

Effective date: 20170731

PG25 Lapsed in a contracting state [announced via postgrant information from national office to epo]

Ref country code: NL

Free format text: LAPSE BECAUSE OF NON-PAYMENT OF DUE FEES

Effective date: 20161201

PG25 Lapsed in a contracting state [announced via postgrant information from national office to epo]

Ref country code: FR

Free format text: LAPSE BECAUSE OF NON-PAYMENT OF DUE FEES

Effective date: 20161130

PG25 Lapsed in a contracting state [announced via postgrant information from national office to epo]

Ref country code: GB

Free format text: LAPSE BECAUSE OF NON-PAYMENT OF DUE FEES

Effective date: 20161127

Ref country code: DE

Free format text: LAPSE BECAUSE OF NON-PAYMENT OF DUE FEES

Effective date: 20170601

REG Reference to a national code

Ref country code: BE

Ref legal event code: MM

Effective date: 20161130

PG25 Lapsed in a contracting state [announced via postgrant information from national office to epo]

Ref country code: ES

Free format text: LAPSE BECAUSE OF FAILURE TO SUBMIT A TRANSLATION OF THE DESCRIPTION OR TO PAY THE FEE WITHIN THE PRESCRIBED TIME-LIMIT

Effective date: 20111109

REG Reference to a national code

Ref country code: ES

Ref legal event code: FD2A

Effective date: 20181116

PG25 Lapsed in a contracting state [announced via postgrant information from national office to epo]

Ref country code: ES

Free format text: LAPSE BECAUSE OF FAILURE TO SUBMIT A TRANSLATION OF THE DESCRIPTION OR TO PAY THE FEE WITHIN THE PRESCRIBED TIME-LIMIT

Effective date: 20161128