EP2835794A1 - Verfahren zum Warnen des Fahrers eines Kraftfahrzeugs abhängig von einer ermittelten Zeit bis zur Kollision, Kamerasystem und Kraftfahrzeug - Google Patents

Verfahren zum Warnen des Fahrers eines Kraftfahrzeugs abhängig von einer ermittelten Zeit bis zur Kollision, Kamerasystem und Kraftfahrzeug Download PDF

Info

Publication number
EP2835794A1
EP2835794A1 EP14170016.1A EP14170016A EP2835794A1 EP 2835794 A1 EP2835794 A1 EP 2835794A1 EP 14170016 A EP14170016 A EP 14170016A EP 2835794 A1 EP2835794 A1 EP 2835794A1
Authority
EP
European Patent Office
Prior art keywords
ttc1
ttcx
measured values
motor vehicle
fitting function
Prior art date
Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
Granted
Application number
EP14170016.1A
Other languages
English (en)
French (fr)
Other versions
EP2835794B1 (de
Inventor
Rammos Perikles
Robert Voros
Current Assignee (The listed assignees may be inaccurate. Google has not performed a legal analysis and makes no representation or warranty as to the accuracy of the list.)
Connaught Electronics Ltd
Original Assignee
Connaught Electronics Ltd
Priority date (The priority date is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the date listed.)
Filing date
Publication date
Application filed by Connaught Electronics Ltd filed Critical Connaught Electronics Ltd
Publication of EP2835794A1 publication Critical patent/EP2835794A1/de
Application granted granted Critical
Publication of EP2835794B1 publication Critical patent/EP2835794B1/de
Active legal-status Critical Current
Anticipated expiration legal-status Critical

Links

Images

Classifications

    • G—PHYSICS
    • G08—SIGNALLING
    • G08G—TRAFFIC CONTROL SYSTEMS
    • G08G1/00—Traffic control systems for road vehicles
    • G08G1/16—Anti-collision systems
    • G08G1/166—Anti-collision systems for active traffic, e.g. moving vehicles, pedestrians, bikes

Definitions

  • the invention relates to a method for warning the driver of a motor vehicle of an imminent collision with a target object located in an environmental region of the motor vehicle and approaching the motor vehicle, wherein a temporal sequence of images of the environmental region is provided by means of a camera of the motor vehicle, wherein a measured value for a degree of risk with respect to the collision is respectively determined from at least a subset of the images based on the respective image by means of an image processing device, and wherein the warning of the driver is effected depending on the measured values.
  • the invention relates to a camera system for performing such a method as well as to a motor vehicle with such a camera system.
  • Camera-based collision warning systems are already prior art.
  • the interest is in particular directed to a so-called "cross traffic alert" system, which serves for warning of possible cross traffic based on images provided by means of a camera.
  • Such camera systems can for example be used in particularly advantageous manner at garage exits, in parking out of a parking space or else on intersections, i.e. overall in road situations, in which the driver has restricted sight to the cross traffic.
  • a camera is mounted on a front bumper of the motor vehicle and/or a camera in the rear area of the motor vehicle (for example on the rear bumper or on the tailgate).
  • This camera has a relatively wide opening angle in a range of values from 160° to 200° and therefore is able to provide images, in which the cross traffic is depicted. These images are communicated to an electronic image processing device processing the images.
  • the image processing device can for example identify target objects - for example other vehicles, pedestrians, cyclists and the like - and track them over the sequence of images.
  • the images can also be displayed on a display.
  • time to collision i.e. that time the target object requires for reaching the motor vehicle, based on the sequence of images of a camera.
  • This time to collision presents a degree of risk, which is proportional to the existent risk of collision.
  • a measured value for the time to collision can respectively be determined to each image or to each n-th image (n>1).
  • these measured values have a relatively great fluctuation over the time such that additional filtering of the measured values is required.
  • the determination of the time to collision based on images of a camera is for example already known from the document US 2009/0148986 A1 .
  • the measured values have to be subjected to additional filtering in the prior art in order to be able to provide usable results for the time to collision.
  • the so-called Kalman filter is used, as it is for example described in the document US 2010/0191391 A1 .
  • a Kalman filter is associated with relatively great disadvantages.
  • a relatively high computational power is required for implementing a Kalman filter, but which is only available in restricted manner in particular in motor vehicles.
  • the computational power is in particular not present in so-called embedded systems.
  • a Kalman filter presents an algorithm excessively complex for the present application, which optionally also can cause a delay in the calculation of the current time to collision.
  • One object of the invention is in providing a method, a camera system as well as a motor vehicle, in which measures are taken, which ensure that the current degree of risk with respect to the collision can be determined particularly precisely and with least computational effort.
  • this object is solved by a method, by a motor vehicle as well as by a camera system having the features according to the respective independent claims.
  • Advantageous implementations of the invention are the subject matter of the dependent claims, of the description and of the figures.
  • a method according to the invention serves for warning the driver of a motor vehicle of an imminent (possible) collision with a target object located in an environmental region of the motor vehicle and approaching the motor vehicle.
  • a temporal sequence of images of the environmental region is provided by means of a camera of the motor vehicle.
  • To at least a subset of the images for example to each image or to each n-th image (n>1) - a measured value for a degree of risk with respect to the collision is respectively determined based on the respective image. This determination is effected by means of an electronic image processing device.
  • a fitting function is determined based on the respectively current measured value as well as based on a plurality of previous measured values (for example a predetermined number of previous measured values), which satisfies a predetermined optimization criterion (for example "least squares") with respect to the measured values, and that the current degree of risk is determined based on the fitting function for warning the driver.
  • a predetermined number of previous measured values for example a predetermined number of previous measured values
  • a predetermined optimization criterion for example "least squares”
  • the degree of risk can therefore be determined particularly precisely and with least computational effort by using also the previous measured values and thus the measurement history in addition to the current measured value, in order to determine a fitting function or fitting curve.
  • This fitting function thus considers a plurality of measured values and preferably is continuously adapted and updated with the respectively new current measured value.
  • the method according to the invention can also be implemented in an embedded system, which, as is known, only has restricted computational power. In particular in motor vehicles, this proves particularly advantageous.
  • the degree of risk for example the above mentioned time to collision (TTC, or time to crossing) and/or a distance to the possible collision can be determined based on the images. Both the TTC and the distance to the collision represent a reliable measure of the degree of risk.
  • TTC time to collision
  • the degree of risk can also include a position of the target object in the image and/or in the three-dimensional world and/or a target angle relative to the motor vehicle.
  • the fitting function can preferably be determined by means of a regression analysis method.
  • a regression analysis method it has proven advantageous if the linear or square regression is used. In this manner, the computational effort can further be reduced.
  • the target object moves with a constant velocity relatively to the motor vehicle such that a linear function is determined as the fitting function.
  • this assumption proves advantageous. Namely, in this system, it can be assumed with high probability that the cross traffic moves substantially with constant velocity relatively to the motor vehicle. This assumption reduces the computational effort to a minimum.
  • a non-linear fitting function can also be determined, in which variation of the relative velocity is also taken into account.
  • the measured values are preferably each weighted with an associated weighting factor. Then, the fitting function is determined based on the weighted measured values.
  • the method according to the invention can each be optimally adapted to very different applications and different road situations by correspondingly adjusting the weighting factors.
  • the method can be universally used for different driver assistance systems.
  • the weighting factors are determined depending on a position of the target object in the respective image. This embodiment is based on the realization that the accuracy of the measured values depends on the position of the target object in the image frame. The more precise measured values can therefore be weighted with greater weighting factors than the less precise measured values in order to be able to overall determine a very precise fitting function.
  • the measured values constitute a measurement sequence of measured values, in which the measured values are ordered corresponding to an order of the images within the sequence.
  • the measurement sequence begins with the respectively current measured value and terminates with an oldest measured value.
  • the weighting factors are determined depending on the position of the respective measured value within the measurement sequence. In other words, the weighting factors depend on the "age" of the respective measured value. This embodiment is based on the fact that the recent measured values are considerably more important than the older measured values and therefore are to be prioritized with respect to the older measured values.
  • a fitting function can be provided, which allows precise estimation of the degree of risk in the respectively current road situation.
  • the weighting factors can each be calculated from at least one partial weighting factor, which is determined based on a weighting function having a (global) maximum for the current measured value and a (global) minimum for an oldest measured value.
  • This weighting function can for example be an exponential function, for example the e -t function.
  • the weighting factors, with which the respective measured values are weighted are each calculated from at least a first and a second partial weighting factor, for example by multiplication of the first partial weighting factor by the second partial weighting factor.
  • the first partial weighting factor can be determined depending on a position of the target object in the respective image.
  • the second partial weighting factor in turn is preferably determined depending on the position of the respective measured value within the measurement sequence and thus depending on the "age" of the respective measured value.
  • the following method can also be performed: After update of the fitting function based on a new measured value, the measured values used for the fitting function (for example all of the measured values, which have been considered in the current fitting function) can be examined to the effect if a deviation of the respective measured values from the current fitting function is less or greater than a preset threshold value.
  • This threshold value can be the above mentioned threshold value or another threshold value.
  • a new fitting function can then be determined exclusively based on those measured values, the deviation of which is less than the threshold value. In this embodiment, thus, former measured values can also be filtered out as outliers such that the accuracy of the fitting function can be further improved.
  • the invention relates to a camera system formed for performing a method according to the invention.
  • a motor vehicle according to the invention in particular a passenger car, includes a camera system according to the invention.
  • a motor vehicle 1 illustrated in Fig. 1 is for example a passenger car.
  • the motor vehicle 1 includes a camera system 2, which is formed as a cross traffic alert system and thus as a collision warning system in the embodiment, which warns the driver of the motor vehicle 1 of cross traffic.
  • the camera system 2 includes at least one camera 3, 4, which is attached to the motor vehicle 1.
  • two cameras 3, 4 can be provided, wherein the invention is not restricted to a certain number of cameras 3, 4 and the number of the cameras 3, 4 can be arbitrary.
  • a first camera 3 is disposed on the front bumper of the motor vehicle 1.
  • a second camera 4 is disposed in the rear area, for example on the rear bumper or on a tailgate of the motor vehicle 1.
  • the cameras 3, 4 each provide a temporal sequence of images and communicate these images to a central electronic image processing device 5 processing the received images and being able to provide very different functionalities in the motor vehicle 1 based on the images.
  • a warning device 6 which can for example include a display and/or a speaker.
  • a possible road situation is shown, in which the driver can be assisted by the camera system 2 of the vehicle 1.
  • the motor vehicle 1 is in a garage exit 7, which is bounded by respective walls 8, 9 on both sides.
  • the garage exit 7 goes to a road 10, which extends perpendicularly to the motor vehicle 1.
  • a further vehicle 11 travels on the road 10, which moves towards the motor vehicle 1 according to the arrow representation 12. The vehicle 11 then moves past the garage exit 7 and thus passes the garage exit 7.
  • the image processing device 5 determines measured values for the time to collision (TTC), which represents a degree of risk.
  • TTC time to collision
  • the TTC means a period of time, which the vehicle 11 requires for reaching the motor vehicle 1 or for passing the garage exit 7.
  • the TTC describes a period of time until a possible collision, which would occur if the motor vehicle 1 would be on a collision trajectory of the vehicle 11.
  • the TTC describes a period of time that the vehicle 11 requires for reaching a line 13 presenting an extension of the lateral flank of the motor vehicle 1 and extending perpendicularly to the direction of travel 12 of the vehicle 11.
  • the TTC can also be calculated if the vehicle 11 will prospectively pass the motor vehicle 1 and thus collision will prospectively not occur.
  • the image processing device 5 calculates a measured value for the TTC to each or to each n-th image (n>1).
  • a measurement sequence of measured values is formed over the sequence of images:
  • the target object 11 is identified in the respective image, and the position thereof in the respective image is determined. Based on the position and the relative velocity, then, the TTC can be determined.
  • Fig. 2 an exemplary progression of the measured values TTC1 to TTCx (in seconds) is shown depending on the time t and thus over the sequence of images.
  • the progression of the determined measured values TTC1 to TTCx is denoted by 14 in Fig. 2 .
  • the measured values TTC1 to TTCx have a relatively great standard deviation and thus a relatively great fluctuation.
  • a fitting function is determined, which is exemplarily shown in Fig. 2 and denoted by 15.
  • a linear fitting function 15 is used; however, the invention is not restricted to such a linear function.
  • a regression analysis method is used, such as in particular the linear or square regression.
  • the "least squares" method can also be used.
  • the first partial weighting factor W1 depends on the position of the target object 11 in the respective image.
  • a first heuristic weighting function 16 can be defined, which represents the dependency of the partial weighting factor W1 on a position P of the target object 11 in the image.
  • this first weighting function 16 can also be a three-dimensional function considering both image directions. Namely, the accuracy of the TTC is correlated with the position P such that the more precise measured values TTC1 to TTCx can be prioritized by corresponding weighting.
  • the respective second partial weighting factor W2 in turn depends on the position of the respective measured value within the measurement sequence TTC1 to TTCx.
  • a second weighting function 17 is defined, which is preferably a heuristic function and is exemplarily illustrated in Fig. 4 . As is apparent from Fig. 4 , the relation applies that the second partial weighting factor W2 is higher at the "recent" measured values than at the "older” measured values.
  • the second weighting function 17 can be a linear, a polynomial or an exponential function. Therein, an exponential function is shown in Fig.
  • the second weighting function 17 has a global maximum 18 for the respectively current measured value TTC1 and a global minimum 19 for the oldest measured value TTCx.
  • the first and/or the second weighting function 16, 17 can be a heuristic function.
  • weighting factors W total can optionally also be normalized, for example such that the sum of all of the weighting factors is equal to "1".
  • filtering of outliers can also be performed to prevent adverse affectation of the fitting function 15.
  • the respectively new measured value TTC1 can be examined to the effect if the deviation thereof from the current (not yet adapted) fitting function 15 is less than a preset threshold value. If it is detected that the current measured value TTC1 is greater than the threshold value, thus, this measured value TTC1 can be ignored such that adaptation of the fitting function 15 based on this new measured value TTC1 is not effected.
  • all of the previous measured values TTC1 to TTCx can also be examined to the effect if the deviation thereof from the newly adapted fitting function 15 is less or greater than a preset threshold value.
  • measured values TTC1 to TTCx are detected, the deviation of which is greater than the threshold value, thus, these measured values TTC1 to TTCx can be filtered out, and a new fitting function 15 can be determined exclusively based on those measured values TTC1 to TTCx, the deviation of which is less than the threshold value. This new fitting function 15 can then be used for the determination of the current TTC.

Landscapes

  • Physics & Mathematics (AREA)
  • General Physics & Mathematics (AREA)
  • Traffic Control Systems (AREA)
EP14170016.1A 2013-08-09 2014-05-27 Verfahren zum Warnen des Fahrers eines Kraftfahrzeugs abhängig von einer ermittelten Zeit bis zur Kollision, Kamerasystem und Kraftfahrzeug Active EP2835794B1 (de)

Applications Claiming Priority (1)

Application Number Priority Date Filing Date Title
DE201310013253 DE102013013253A1 (de) 2013-08-09 2013-08-09 Verfahren zum Warnen des Fahrers eines Kraftfahrzeugs abhängig von einer ermittelten Zeit bis zur Kollision, Kamerasystem und Kraftfahrzeug

Publications (2)

Publication Number Publication Date
EP2835794A1 true EP2835794A1 (de) 2015-02-11
EP2835794B1 EP2835794B1 (de) 2016-05-25

Family

ID=50792389

Family Applications (1)

Application Number Title Priority Date Filing Date
EP14170016.1A Active EP2835794B1 (de) 2013-08-09 2014-05-27 Verfahren zum Warnen des Fahrers eines Kraftfahrzeugs abhängig von einer ermittelten Zeit bis zur Kollision, Kamerasystem und Kraftfahrzeug

Country Status (2)

Country Link
EP (1) EP2835794B1 (de)
DE (1) DE102013013253A1 (de)

Cited By (1)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
EP3290283A1 (de) * 2016-09-06 2018-03-07 Delphi Technologies, Inc. Querverkehrdetektionssystem für automatisiertes fahrzeug

Families Citing this family (1)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
DE102019205542A1 (de) 2018-05-09 2019-11-14 Ford Global Technologies, Llc Verfahren und Vorrichtung zur bildhaften Information über Querverkehr auf einer Anzeigevorrichtung eines gefahrenen Fahrzeugs

Citations (5)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
US20090143986A1 (en) 2004-04-08 2009-06-04 Mobileye Technologies Ltd Collision Warning System
US20100191391A1 (en) 2009-01-26 2010-07-29 Gm Global Technology Operations, Inc. multiobject fusion module for collision preparation system
US20110128138A1 (en) * 2009-11-30 2011-06-02 Fujitsu Ten Limited On-vehicle device and recognition support system
JP2012234293A (ja) * 2011-04-28 2012-11-29 Toyota Motor Corp ドライバ状態判定装置
US20120314071A1 (en) * 2011-04-27 2012-12-13 Mobileye Technologies Ltd. Pedestrian collision warning system

Family Cites Families (5)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
DE19858297A1 (de) * 1998-12-17 2000-06-29 Daimler Chrysler Ag Verfahren zur Korrektur eines Signals wenigstens eines Sensors, aufgrund dessen eine Bahnkurve ermittelbar ist, auf der sich das Fahrzeug bewegt
JP3515926B2 (ja) * 1999-06-23 2004-04-05 本田技研工業株式会社 車両の周辺監視装置
DE19944542C2 (de) * 1999-09-17 2003-01-23 Daimler Chrysler Ag Verfahren zur fahrzeugseitigen Bestimmung des Fahrstreckenverlaufs
DE102007034196A1 (de) * 2007-07-23 2009-01-29 Robert Bosch Gmbh Verfahren und Vorrichtung für die Spurerfassung mit einem Fahrerassistenzsystem
DE102007049706A1 (de) * 2007-10-17 2009-04-23 Robert Bosch Gmbh Verfahren zur Schätzung der Relativbewegung von Video-Objekten und Fahrerassistenzsystem für Kraftfahrzeuge

Patent Citations (6)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
US20090143986A1 (en) 2004-04-08 2009-06-04 Mobileye Technologies Ltd Collision Warning System
US20100191391A1 (en) 2009-01-26 2010-07-29 Gm Global Technology Operations, Inc. multiobject fusion module for collision preparation system
US20110128138A1 (en) * 2009-11-30 2011-06-02 Fujitsu Ten Limited On-vehicle device and recognition support system
US20120314071A1 (en) * 2011-04-27 2012-12-13 Mobileye Technologies Ltd. Pedestrian collision warning system
JP2012234293A (ja) * 2011-04-28 2012-11-29 Toyota Motor Corp ドライバ状態判定装置
EP2704124A1 (de) * 2011-04-28 2014-03-05 Toyota Jidosha Kabushiki Kaisha Vorrichtung zur beurteilung des zustands eines fahrers

Non-Patent Citations (2)

* Cited by examiner, † Cited by third party
Title
DOUGLAS C MONTGOMERY ET AL, 17 April 2012 (2012-04-17), XP055141604, ISBN: 978-0-47-054281-1, Retrieved from the Internet <URL:http://proquest.safaribooksonline.com/book/math-and-science/9780470542811> [retrieved on 20140919] *
TOMOTAKA WADA ET AL: "Pedestrian Oriented Vehicular Collision Avoidance Support System: P-VCASS", IEICE TRANSACTIONS ON FUNDAMENTALS OF ELECTRONICS,COMMUNICATIONS AND COMPUTER SCIENCES, ENGINEERING SCIENCES SOCIETY, TOKYO, JP, vol. E93A, no. 4, 1 April 2010 (2010-04-01), pages 679 - 688, XP001555134, ISSN: 0916-8508, DOI: 10.1587/TRANSFUN.E93.A.679 *

Cited By (2)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
EP3290283A1 (de) * 2016-09-06 2018-03-07 Delphi Technologies, Inc. Querverkehrdetektionssystem für automatisiertes fahrzeug
US10089875B2 (en) 2016-09-06 2018-10-02 Delphi Technologies, Inc. Automated vehicle cross-traffic detection system

Also Published As

Publication number Publication date
DE102013013253A1 (de) 2015-02-12
EP2835794B1 (de) 2016-05-25

Similar Documents

Publication Publication Date Title
US10155515B2 (en) Travel control device
US9889858B2 (en) Confidence estimation for predictive driver assistance systems based on plausibility rules
US9159023B2 (en) System for predicting a driver&#39;s intention to change lanes
US8831867B2 (en) Device and method for driver assistance
JP4420011B2 (ja) 物体検知装置
EP2778007B1 (de) Verfahren und System zur Beurteilung von anormalem Fahrverhalten von Fahrzeugen auf einer Straße
KR102005253B1 (ko) 극도로 빠르게 접근하는 차량들에 응답하는 차선 보조 시스템
US20140297170A1 (en) Driving support system
EP3156766A1 (de) Fahrspurauswahlvorrichtung, fahrzeugsteuerungssystem und fahrspurauswahlverfahren
CN108027422A (zh) 借助于汽车传感器自动检测危险偏离车辆
EP2394256A1 (de) Kollisionswarngerät
CN109427213B (zh) 用于车辆的防碰撞装置、方法及非暂时性存储介质
CN104071157B (zh) 车载设备
CN107004367B (zh) 车辆的行驶控制装置以及行驶控制方法
KR20170030814A (ko) 차선 유지 보조 시스템 및 차선 유지 보조 방법
JP2010093610A (ja) 自動車の外界認識装置及び外界認識方法
WO2018146314A1 (en) Improvements in or relating to driver assistance systems
EP3196076A1 (de) Elektronische spiegelvorrichtung
JP2014078056A (ja) 車両用領域識別装置、そのプログラム及びその方法
CN111645674B (zh) 车辆控制装置
EP2835794B1 (de) Verfahren zum Warnen des Fahrers eines Kraftfahrzeugs abhängig von einer ermittelten Zeit bis zur Kollision, Kamerasystem und Kraftfahrzeug
WO2011090414A1 (en) An object collision warning system and method for a motor vehicle
JP6169270B2 (ja) 外部車両の走行状態を識別する方法及び装置
US11383713B2 (en) Assist apparatus and assist method
JP6455292B2 (ja) 物体検出装置及び物体検出方法

Legal Events

Date Code Title Description
PUAI Public reference made under article 153(3) epc to a published international application that has entered the european phase

Free format text: ORIGINAL CODE: 0009012

17P Request for examination filed

Effective date: 20140527

AK Designated contracting states

Kind code of ref document: A1

Designated state(s): AL AT BE BG CH CY CZ DE DK EE ES FI FR GB GR HR HU IE IS IT LI LT LU LV MC MK MT NL NO PL PT RO RS SE SI SK SM TR

AX Request for extension of the european patent

Extension state: BA ME

R17P Request for examination filed (corrected)

Effective date: 20150807

RBV Designated contracting states (corrected)

Designated state(s): AL AT BE BG CH CY CZ DE DK EE ES FI FR GB GR HR HU IE IS IT LI LT LU LV MC MK MT NL NO PL PT RO RS SE SI SK SM TR

GRAP Despatch of communication of intention to grant a patent

Free format text: ORIGINAL CODE: EPIDOSNIGR1

INTG Intention to grant announced

Effective date: 20151215

GRAS Grant fee paid

Free format text: ORIGINAL CODE: EPIDOSNIGR3

GRAA (expected) grant

Free format text: ORIGINAL CODE: 0009210

AK Designated contracting states

Kind code of ref document: B1

Designated state(s): AL AT BE BG CH CY CZ DE DK EE ES FI FR GB GR HR HU IE IS IT LI LT LU LV MC MK MT NL NO PL PT RO RS SE SI SK SM TR

REG Reference to a national code

Ref country code: GB

Ref legal event code: FG4D

REG Reference to a national code

Ref country code: CH

Ref legal event code: EP

REG Reference to a national code

Ref country code: IE

Ref legal event code: FG4D

Ref country code: AT

Ref legal event code: REF

Ref document number: 802883

Country of ref document: AT

Kind code of ref document: T

Effective date: 20160615

REG Reference to a national code

Ref country code: DE

Ref legal event code: R096

Ref document number: 602014002058

Country of ref document: DE

REG Reference to a national code

Ref country code: LT

Ref legal event code: MG4D

REG Reference to a national code

Ref country code: NL

Ref legal event code: MP

Effective date: 20160525

PG25 Lapsed in a contracting state [announced via postgrant information from national office to epo]

Ref country code: NL

Free format text: LAPSE BECAUSE OF FAILURE TO SUBMIT A TRANSLATION OF THE DESCRIPTION OR TO PAY THE FEE WITHIN THE PRESCRIBED TIME-LIMIT

Effective date: 20160525

Ref country code: NO

Free format text: LAPSE BECAUSE OF FAILURE TO SUBMIT A TRANSLATION OF THE DESCRIPTION OR TO PAY THE FEE WITHIN THE PRESCRIBED TIME-LIMIT

Effective date: 20160825

Ref country code: LT

Free format text: LAPSE BECAUSE OF FAILURE TO SUBMIT A TRANSLATION OF THE DESCRIPTION OR TO PAY THE FEE WITHIN THE PRESCRIBED TIME-LIMIT

Effective date: 20160525

Ref country code: FI

Free format text: LAPSE BECAUSE OF FAILURE TO SUBMIT A TRANSLATION OF THE DESCRIPTION OR TO PAY THE FEE WITHIN THE PRESCRIBED TIME-LIMIT

Effective date: 20160525

REG Reference to a national code

Ref country code: AT

Ref legal event code: MK05

Ref document number: 802883

Country of ref document: AT

Kind code of ref document: T

Effective date: 20160525

PG25 Lapsed in a contracting state [announced via postgrant information from national office to epo]

Ref country code: LV

Free format text: LAPSE BECAUSE OF FAILURE TO SUBMIT A TRANSLATION OF THE DESCRIPTION OR TO PAY THE FEE WITHIN THE PRESCRIBED TIME-LIMIT

Effective date: 20160525

Ref country code: PT

Free format text: LAPSE BECAUSE OF FAILURE TO SUBMIT A TRANSLATION OF THE DESCRIPTION OR TO PAY THE FEE WITHIN THE PRESCRIBED TIME-LIMIT

Effective date: 20160926

Ref country code: RS

Free format text: LAPSE BECAUSE OF FAILURE TO SUBMIT A TRANSLATION OF THE DESCRIPTION OR TO PAY THE FEE WITHIN THE PRESCRIBED TIME-LIMIT

Effective date: 20160525

Ref country code: GR

Free format text: LAPSE BECAUSE OF FAILURE TO SUBMIT A TRANSLATION OF THE DESCRIPTION OR TO PAY THE FEE WITHIN THE PRESCRIBED TIME-LIMIT

Effective date: 20160826

Ref country code: SE

Free format text: LAPSE BECAUSE OF FAILURE TO SUBMIT A TRANSLATION OF THE DESCRIPTION OR TO PAY THE FEE WITHIN THE PRESCRIBED TIME-LIMIT

Effective date: 20160525

Ref country code: ES

Free format text: LAPSE BECAUSE OF FAILURE TO SUBMIT A TRANSLATION OF THE DESCRIPTION OR TO PAY THE FEE WITHIN THE PRESCRIBED TIME-LIMIT

Effective date: 20160525

REG Reference to a national code

Ref country code: FR

Ref legal event code: PLFP

Year of fee payment: 3

PG25 Lapsed in a contracting state [announced via postgrant information from national office to epo]

Ref country code: IT

Free format text: LAPSE BECAUSE OF FAILURE TO SUBMIT A TRANSLATION OF THE DESCRIPTION OR TO PAY THE FEE WITHIN THE PRESCRIBED TIME-LIMIT

Effective date: 20160525

Ref country code: BE

Free format text: LAPSE BECAUSE OF NON-PAYMENT OF DUE FEES

Effective date: 20160531

PG25 Lapsed in a contracting state [announced via postgrant information from national office to epo]

Ref country code: DK

Free format text: LAPSE BECAUSE OF FAILURE TO SUBMIT A TRANSLATION OF THE DESCRIPTION OR TO PAY THE FEE WITHIN THE PRESCRIBED TIME-LIMIT

Effective date: 20160525

Ref country code: CZ

Free format text: LAPSE BECAUSE OF FAILURE TO SUBMIT A TRANSLATION OF THE DESCRIPTION OR TO PAY THE FEE WITHIN THE PRESCRIBED TIME-LIMIT

Effective date: 20160525

Ref country code: EE

Free format text: LAPSE BECAUSE OF FAILURE TO SUBMIT A TRANSLATION OF THE DESCRIPTION OR TO PAY THE FEE WITHIN THE PRESCRIBED TIME-LIMIT

Effective date: 20160525

Ref country code: SK

Free format text: LAPSE BECAUSE OF FAILURE TO SUBMIT A TRANSLATION OF THE DESCRIPTION OR TO PAY THE FEE WITHIN THE PRESCRIBED TIME-LIMIT

Effective date: 20160525

Ref country code: RO

Free format text: LAPSE BECAUSE OF FAILURE TO SUBMIT A TRANSLATION OF THE DESCRIPTION OR TO PAY THE FEE WITHIN THE PRESCRIBED TIME-LIMIT

Effective date: 20160525

REG Reference to a national code

Ref country code: IE

Ref legal event code: MM4A

PG25 Lapsed in a contracting state [announced via postgrant information from national office to epo]

Ref country code: SM

Free format text: LAPSE BECAUSE OF FAILURE TO SUBMIT A TRANSLATION OF THE DESCRIPTION OR TO PAY THE FEE WITHIN THE PRESCRIBED TIME-LIMIT

Effective date: 20160525

Ref country code: AT

Free format text: LAPSE BECAUSE OF FAILURE TO SUBMIT A TRANSLATION OF THE DESCRIPTION OR TO PAY THE FEE WITHIN THE PRESCRIBED TIME-LIMIT

Effective date: 20160525

Ref country code: PL

Free format text: LAPSE BECAUSE OF FAILURE TO SUBMIT A TRANSLATION OF THE DESCRIPTION OR TO PAY THE FEE WITHIN THE PRESCRIBED TIME-LIMIT

Effective date: 20160525

Ref country code: BE

Free format text: LAPSE BECAUSE OF FAILURE TO SUBMIT A TRANSLATION OF THE DESCRIPTION OR TO PAY THE FEE WITHIN THE PRESCRIBED TIME-LIMIT

Effective date: 20160525

REG Reference to a national code

Ref country code: DE

Ref legal event code: R097

Ref document number: 602014002058

Country of ref document: DE

PG25 Lapsed in a contracting state [announced via postgrant information from national office to epo]

Ref country code: MC

Free format text: LAPSE BECAUSE OF FAILURE TO SUBMIT A TRANSLATION OF THE DESCRIPTION OR TO PAY THE FEE WITHIN THE PRESCRIBED TIME-LIMIT

Effective date: 20160525

PLBE No opposition filed within time limit

Free format text: ORIGINAL CODE: 0009261

STAA Information on the status of an ep patent application or granted ep patent

Free format text: STATUS: NO OPPOSITION FILED WITHIN TIME LIMIT

26N No opposition filed

Effective date: 20170228

REG Reference to a national code

Ref country code: FR

Ref legal event code: PLFP

Year of fee payment: 4

PG25 Lapsed in a contracting state [announced via postgrant information from national office to epo]

Ref country code: SI

Free format text: LAPSE BECAUSE OF FAILURE TO SUBMIT A TRANSLATION OF THE DESCRIPTION OR TO PAY THE FEE WITHIN THE PRESCRIBED TIME-LIMIT

Effective date: 20160525

Ref country code: IE

Free format text: LAPSE BECAUSE OF NON-PAYMENT OF DUE FEES

Effective date: 20160527

REG Reference to a national code

Ref country code: CH

Ref legal event code: PL

PG25 Lapsed in a contracting state [announced via postgrant information from national office to epo]

Ref country code: LI

Free format text: LAPSE BECAUSE OF NON-PAYMENT OF DUE FEES

Effective date: 20170531

Ref country code: CH

Free format text: LAPSE BECAUSE OF NON-PAYMENT OF DUE FEES

Effective date: 20170531

REG Reference to a national code

Ref country code: FR

Ref legal event code: PLFP

Year of fee payment: 5

PG25 Lapsed in a contracting state [announced via postgrant information from national office to epo]

Ref country code: HU

Free format text: LAPSE BECAUSE OF FAILURE TO SUBMIT A TRANSLATION OF THE DESCRIPTION OR TO PAY THE FEE WITHIN THE PRESCRIBED TIME-LIMIT; INVALID AB INITIO

Effective date: 20140527

PG25 Lapsed in a contracting state [announced via postgrant information from national office to epo]

Ref country code: HR

Free format text: LAPSE BECAUSE OF FAILURE TO SUBMIT A TRANSLATION OF THE DESCRIPTION OR TO PAY THE FEE WITHIN THE PRESCRIBED TIME-LIMIT

Effective date: 20160525

Ref country code: MK

Free format text: LAPSE BECAUSE OF FAILURE TO SUBMIT A TRANSLATION OF THE DESCRIPTION OR TO PAY THE FEE WITHIN THE PRESCRIBED TIME-LIMIT

Effective date: 20160525

Ref country code: CY

Free format text: LAPSE BECAUSE OF FAILURE TO SUBMIT A TRANSLATION OF THE DESCRIPTION OR TO PAY THE FEE WITHIN THE PRESCRIBED TIME-LIMIT

Effective date: 20160525

Ref country code: MT

Free format text: LAPSE BECAUSE OF NON-PAYMENT OF DUE FEES

Effective date: 20160531

Ref country code: LU

Free format text: LAPSE BECAUSE OF NON-PAYMENT OF DUE FEES

Effective date: 20160527

Ref country code: IS

Free format text: LAPSE BECAUSE OF FAILURE TO SUBMIT A TRANSLATION OF THE DESCRIPTION OR TO PAY THE FEE WITHIN THE PRESCRIBED TIME-LIMIT

Effective date: 20160525

PG25 Lapsed in a contracting state [announced via postgrant information from national office to epo]

Ref country code: BG

Free format text: LAPSE BECAUSE OF FAILURE TO SUBMIT A TRANSLATION OF THE DESCRIPTION OR TO PAY THE FEE WITHIN THE PRESCRIBED TIME-LIMIT

Effective date: 20160525

PG25 Lapsed in a contracting state [announced via postgrant information from national office to epo]

Ref country code: TR

Free format text: LAPSE BECAUSE OF FAILURE TO SUBMIT A TRANSLATION OF THE DESCRIPTION OR TO PAY THE FEE WITHIN THE PRESCRIBED TIME-LIMIT

Effective date: 20160525

Ref country code: AL

Free format text: LAPSE BECAUSE OF FAILURE TO SUBMIT A TRANSLATION OF THE DESCRIPTION OR TO PAY THE FEE WITHIN THE PRESCRIBED TIME-LIMIT

Effective date: 20160525

P01 Opt-out of the competence of the unified patent court (upc) registered

Effective date: 20230528

PGFP Annual fee paid to national office [announced via postgrant information from national office to epo]

Ref country code: FR

Payment date: 20230523

Year of fee payment: 10

Ref country code: DE

Payment date: 20230510

Year of fee payment: 10

PGFP Annual fee paid to national office [announced via postgrant information from national office to epo]

Ref country code: GB

Payment date: 20230519

Year of fee payment: 10