EP2374117A1 - Vehicle tracking system, vehicle infrastructure provided with vehicle tracking system and method for tracking vehicles - Google Patents
Vehicle tracking system, vehicle infrastructure provided with vehicle tracking system and method for tracking vehiclesInfo
- Publication number
- EP2374117A1 EP2374117A1 EP09771423A EP09771423A EP2374117A1 EP 2374117 A1 EP2374117 A1 EP 2374117A1 EP 09771423 A EP09771423 A EP 09771423A EP 09771423 A EP09771423 A EP 09771423A EP 2374117 A1 EP2374117 A1 EP 2374117A1
- Authority
- EP
- European Patent Office
- Prior art keywords
- vehicle
- message
- facility
- sensor nodes
- infrastructure
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Granted
Links
Classifications
-
- G—PHYSICS
- G08—SIGNALLING
- G08G—TRAFFIC CONTROL SYSTEMS
- G08G1/00—Traffic control systems for road vehicles
- G08G1/01—Detecting movement of traffic to be counted or controlled
- G08G1/0104—Measuring and analyzing of parameters relative to traffic conditions
Definitions
- Vehicle tracking system vehicle infrastructure provided with vehicle tracking system and method for tracking vehicles.
- the present invention relates to a vehicle tracking system.
- the present invention relates to a vehicle infrastructure provided with a vehicle tracking system.
- the present invention further relates to a method for tracking vehicles.
- PeMS traffic performance measurement system
- US5801943 describes a wide area surveillance system for application to large road networks.
- the system employs smart sensors to identify plural individual vehicles in the network. These vehicles are tracked on an individual basis, and the system derives the behavior of the vehicle. Furthermore, the system derives traffic behavior on a local basis, across roadway links, and in sections of the network. Processing in the system is divided into multiple processing layers, with geographical separation of tasks.
- the vehicle tracking system comprises a plurality of sensor nodes that each provide a message indicative for an occupancy status of a detection area of a vehicle infrastructure monitored by said sensor node, a message interpretator including a vehicle database facility with state information of vehicles present at the vehicle infrastructure, and a database updating facility for updating the database facility on the basis of messages provided by the sensor nodes, characterized in that said sensor nodes are arranged in the vehicle infrastructure at a density of at least 0.2 per square meter.
- the vehicle tracking system in the vehicle tracking system according to the present invention vehicles can be tracked with relatively simple and cheap means. Smart sensors are not necessary. It is sufficient that the sensor nodes sense an occupancy state, i.e. whether a detection area associated with the sensor node is occupied by a vehicle or not and that they merely provide a message that indicates whether the occupancy state is changed.
- the relatively cheap and simple construction of the sensor nodes contributes to an economically feasible application in vehicle tracking systems for large vehicle infrastructures.
- the message may additionally include the value of the occupancy state after the change was detected.
- the plurality of sensor nodes arranged in the vehicle infrastructure having at least the above-mentioned density provide a course image of the vehicles present at the vehicle infrastructure.
- the database update facility comprises an association facility for associating the messages provided by the sensor nodes with the state information present in the vehicle data base facility, a state updating facility for updating the state information on the basis of the messages associated therewith.
- the association facility selects for which state information the received messages are relevant, and provides the selected messages to the state updating facility. In this way the state updating facility can operate more efficiently, than in case no selection takes place.
- the present invention is in particular suitable for tracking vehicles.
- suitable sensor elements to be used in the sensor nodes are for example magneto restrictive sensors. These sensors determine whether their associated detection area is occupied by detection of a perturbation of the earth magnetic field.
- magnetic loop sensors may be used, which detect a change of inductance caused by the presence of ferromagnetic material.
- each sensor node is provided with a wireless transmission facility that transmits the preprocessed data, e.g. the occupance status or an indication of a change thereof to a data to a receiver facility coupled to the message interpreter.
- a wireless transmission facility that transmits the preprocessed data, e.g. the occupance status or an indication of a change thereof to a data to a receiver facility coupled to the message interpreter.
- the absence of wiring towards the message interpreter makes the installation easier and cost effective.
- the sensor nodes provide their message at an event basis, e.g. if a perturbation of the earth magnetic field exceeds a threshold value. This reduces communication load of the message interpreter and minimizes power consumption of the sensor nodes.
- the density of the sensor nodes are at least 0.6 per square meter. Due to the relatively high density of the sensor nodes in the traffic infrastructure in said embodiment an individual failure of a sensor or of an individual sensor does not have serious consequences for the estimation of the states of the traffic participants. Accordingly, any return transmission from the message interpreter to the sensor nodes to actively verify the occupancy state is superfluous, which is also favorable for a low power consumption of the sensor nodes.
- the vehicle tracking system may in addition to the plurality of sensor nodes arranged in the traffic infrastructure comprise one or more cameras.
- a camera may be used for example if a perturbation of the earth magnetic field can not be measured. This is the case for example if (parts of) the infra structure comprises metal components e.g. a bridge.
- the detection areas of the sensor elements are complementary.
- the detection areas may overlap, or spaces may exist between the detection areas, but it is required that the detection area of the sensor be smaller than the vehicles to be tracked.
- the sensor elements are point detectors.
- the sensor nodes can be either randomly distributed over the vehicle infrastructure or placed in a pattern optimized for the vehicle tracking problem in hand.
- the vehicle tracking system comprises a plurality of system modules, each module comprising a respective subset of the plurality of sensor nodes for monitoring a respective section of the vehicle infrastructure and a respective message interpreter, the vehicle tracking system has a communication facility for enabling system modules of mutually neighboring sections to exchange state and detection information. In this way the vehicle tracking system can be easily expanded if required.
- a new system module need only to communicate with the system modules arranged for neighboring sections. Neighboring sections may be arranged in one dimensional scheme, e.g. in case of a narrow road.
- the new module may communicate with other modules neighboring in various directions.
- the system modules merely need to exchange state information and vehicle -detection information (i.e. the unprocessed sensor signals) in a limited subarea of the respective sections, the amount of communication between the system modules is modest resulting in a scalable vehicle tracking system.
- the association facility associates the messages provided by the sensor nodes or neighboring system modules with the state information present in the vehicle data base facility. In other words the association facility determines the probability that the detections are caused by a particular vehicle for which state information is present in the vehicle data base facility. If the messages cannot be associated with state information of an already identified vehicle here or in the neighboring system module, a new entry may be added to the database. Alternatively, the entry for the new vehicle may be added by a separate procedure.
- the vehicle infrastructure may have an access with a vehicle identification facility that provides for an identification of every vehicle that enters the infrastructure.
- the individual sensor nodes do not need to provide other information than an occupancy status of their associated detection area.
- the sensor node may associate its own signal with a color, shape, or other signature of the tracked vehicles to facilitate or obviate association by the message interpreter.
- An association facility for associating the detection signals obtained on asynchronous basis with state information of a particular vehicle may be based on one of the following methods.
- MHT Multiple Hypothesis Tracker
- Gating comprises forming a gate around the predicted measurement of a vehicle.
- the size and shape of the gate are chosen in such a way that unlikely messages are precluded to be associated with this particular vehicle -track.
- the method determines a statistical, quadratic distance from vehicle i.
- a measurement y is associated with the state-vector x 01 of vehicle i if , with G some constant threshold and equal to:
- Threshold G Various methods can be used for finding the Threshold G.
- this data association method is not suitable for associating event based messages.
- the Nearest Neighbor method also uses a gate, but it can handle overlapping gates.
- the sum of all possible combinations to associate a certain measurement to a certain track is analyzed.
- the chosen combination associates the most measurements to a track for a minimum sum of distances.
- JPDA Probabilistic Data Association
- a further data association method is the Markov chain Monte Carlo data association (MCMCDA). All observations are used to classify and cluster them. To that end the whole set of observations is divided into a number of partitions represented by the set w. This is done n m c times resulting in n m c sets of w, i.e. possible partitions. The set of w with the highest probability, given the number of vehicles in the previous sample instant, is chosen and the state-vectors of the tracks a are updated according the partitioned observation. The computational time can be decreased by not using the total history of observations, but by using a moving horizon.
- a downside of this method is that each observation can belong to at most one vehicle and, making this method unsuitable for event-based state- estimation.
- the step of associating may comprise - initializing (S40) a vehicle index (i), retrieving (S41) the current state known for the vehicle with that index from a vehicle database facility, determining (S42) a probability that the vehicle with that index caused the detection reported by the message D, incrementing (S43) the vehicle index, determining (S44) whether the vehicle index is less than the number of vehicles, if the outcome of the determination is positive repeating steps S41 to S43 with the incremented vehicle index, and if the outcome of the determination is negative, determining (S45) which vehicle caused the detection reported by the message D with the highest probability. - returning (S46) the index of the vehicle identified in step S45.
- the message interpreter does not need to send verification messages to the sensor nodes to verify correct operation of the transmission. Accordingly a one-way message traffic from the sensor nodes to the message interpreters is sufficient for a correct operation of the method.
- the state of a vehicle can also be estimated at a point in time later than the last message, but before a new message has arrived. In that case the error covariance matrix is bounded, as it is known that the state change of the vehicle must be within the detection boundaries of the sensor node.
- Figure 1 shows a first view of an embodiment of a vehicle infrastructure provided with a vehicle tracking system according to the invention
- Figure 2 shows a second view of an embodiment of a vehicle infrastructure provided with a vehicle tracking system according to the invention
- Figure 3 shows another view of an embodiment of a vehicle tracking system according to the invention
- Figure 4 schematically shows a part of the vehicle infrastructure that is provided with a plurality of sensor nodes
- Figure 5 schematically shows a signal flow in a sensor node
- Figure 6 schematically shows a possible hardware implementation of a sensor node
- Figure 7 shows a possible method carried out by a sensor node
- Figure 8 shows a signal flow in a message interpretor
- Figure 9 shows a possible hardware implementation of a message interpretor
- Figure 10 shows neighboring infrastructure regions, with specially handled subregions marked
- Figure 11 shows an overview of a method carried out by the message interpreter
- Figure 12 shows a first detail of the method of Figure 11
- Figure 13 shows a second detail of the method of Figure 11
- Figure 14 shows an example of a vehicle to be detected at a reference position and orientation and at a different position and orientation
- Figure 15 shows a definition of a set S and the equidistant sampled set ⁇
- Figure 16 shows detection of a vehicle at multiple detection points
- Figure 17 shows a definition of the set O n of possible positions o'k for a single detection point
- Figure 18 shows a definition of the set ON of possible positions o'k for multiple detection points
- Figure 19 shows a derivation of ON(q ) given 2 detections and 2 different samples of q
- Figure 20 shows (Left) determination of - ⁇ , (right) the vehicle's possible position set On given d n and q,
- Figure 21 shows (left) the mean of all Gaussians from f(o I zi, ⁇ ) and f(o I Z2, ⁇ ); (right) The selection of means of the Gaussians from f(o I zi, ⁇ ) and f(o I Z2, ⁇ ), of which their mean ⁇ m is close or in CN( ⁇ ),
- Figure 22 shows an association result with event-based data-association
- Figure 23 shows an association result with Nearest Neighbor data-association
- Figure 24 shows time sampling of a signal y(t)
- Figure 25 shows event sampling of a signal y(t)
- Figure 26 shows event sampling: Send-on-Delta
- Figure 27 shows the Gaussian function
- Figure 28 shows a top view of the Gaussian function
- Figure 29 shows an approximation of A H (y ⁇ ) as a sum of Gaussian functions
- Figure 30 shows position, speed and acceleration of a simulated vehicle
- Figure 31 shows a position estimation error for various methods
- Figure 32 shows a speed estimation speed for various methods
- Figure 33 shows a factor of increase in estimation error after z k , or _y fa .
- Figure 1 and 2 show a first and a second view of an embodiment of a vehicle infrastructure 80 provided with a vehicle tracking system.
- the vehicle infrastructure is intended to allow stationary and/or moving vehicles 70 thereon, e.g. a road or a parking place.
- the vehicle infrastructure may be part of a public or private space, e.g. a recreational park.
- the vehicle tracking system comprises a plurality of sensor nodes 10 that each provide a message indicative for an occupancy status of a detection area of the vehicle infrastructure monitored by said sensor node 10. As shown therein the sensor nodes are randomly distributed over the vehicle infrastructure.
- the vehicle tracking system comprises a message interpretator MI, each comprising a vehicle database facility, an association facility and a state updating facility.
- Each message interpretator is responsible for handling messages D from a respective section 8OA, 8OB, 8OC, 8OD of the vehicle infrastructure 80.
- Figure 3 is another schematic view of the vehicle tracking system.
- Figure 3 shows how sensor nodes 10 transmit (detection) messages to a message interpreter MI in their neighborhood.
- the message interpreters MI may also communicate to each other via a communication channel 60 to indicate that a vehicle crosses a boundary between their respective sections and to exchange a status of such a vehicle.
- the vehicle tracking system comprises a plurality of system modules MDl, MD2, MD3. Although three modules are shown in this example, any number of system modules is possible, dependent on the application. For example for an isolated vehicle infra structure, e.g. an intersection of roads a single module may be applicable, while on a long road thousands of modules may be present.
- Each module MDl, MD2, MD3 comprises a respective subset of the plurality of sensor nodes 10 for monitoring a respective section of the vehicle infrastructure and a respective message interpreter MI.
- the vehicle tracking system further has a communication facility 60 for enabling system modules MDl, MD2, MD3 of mutually neighboring sections to exchange state information.
- messages from the sensor nodes are directly transmitted to a message interpretor.
- the sensor nodes may form a network that routes the messages to the message interpreters. In that case the transmitters may have a short transmission range.
- Figure 4 schematically shows a part of the vehicle infrastructure that is provided with a plurality of sensor nodes j having position c ⁇ .
- the sensor nodes have a detection area with radius R.
- a vehicle i is present at the infrastructure having a position (V x , vv). In this case if the vehicle substantially covers the detection area the sensor node indicates that the detection area is occupied as indicated in gray. Otherwise the sensor node indicates that the detection area is not occupied (white).
- the fraction of the detection area that should be covered before an occupied status is detected may deviate from the above-mentioned 50% depending on the type of vehicle.
- Figure 5 schematically illustrates the signal flow for the sensor node 10, having sensor element 12, a processing unit 14 (with memory), and a radio link 16.
- the sensor element 12 is capable of sensing the proximity of the vehicles to be tracked.
- the processing unit 14 determines if a vehicle is present or absent on the basis of the signals from the sensor element 12. If an occupancy status of the detection area of the sensor changes, the processing unit 14 initiates a transmission of a message D indicating the new occupancy status.
- the message may include a time stamp indicative of the time t at which the new occupancy status occurred.
- the sensor nodes may transmit occupancy status information on a periodical basis for example. However, an event-based transmission enables a lower power use.
- the message D sent should reach at least one message interpreter MI.
- the sensor element 12 is a magnetoresistive component, which measures the disturbance on the earth magnetic field induced by the vehicles.
- a magnetic rod or loop antenna may be used to detect the occupancy by a vehicle.
- Figure 6 shows a possible implementation of the hardware involved for the sensor node 10 of Figure 5.
- the sensor element 12 is coupled via an A/D converter 13 to a microcontroller 14 that has access to a memory 15, and that further controls a radio transmitter 16 coupled to an antenna 17.
- Figure 7 schematically shows a method performed by a sensor node to generate a message indicative for occupancy status of a detection area of the sensor node.
- Step Sl initialization
- Step S2 input from the A/D converter
- Step S3 offset is removed from the sensed value.
- step S4 it is determined whether the occupancy state of the detection area as reported by the last message transmitted by the sensor node was ON (vehicle was present in the detection range) or OFF (no vehicle present in the detection range. This occupancy state is internally stored in the sensor node.
- step S5 it is determined whether a signal value v obtained from the A/D converter, and indicative for an occupied status of the detection area is below a first predetermined value TL. If this is not the case program flow continues with step S2. If however the value is lower than said first predetermined value then program flow continues with step S6. In step S6 it is verified whether the signal value v remains below the first predetermined value TL for a first predetermined time period. During step S6 the retrieval of input from the A/D convertor is continued. If the signal value v returns to a value higher then said predetermined value TL before the end of said predetermined time-period then processing flow continues with step S2. Otherwise the value for the occupancy state is internally saved as unoccupied in step S7, and a message signaling this is transmitted in step S8.
- step S9 it is determined whether the signal value v obtained from the A/D converter, and indicative for an occupied status of the detection area is above a second predetermined value TH.
- the second predetermined value TH is higher than the first predetermined value TL. If this is not the case program flow continues with step S2. If however the value is higher than said second predetermined value TH then program flow continues with step SlO.
- step SlO it is verified whether the signal value v remains above the second predetermined value TH for a second predetermined time period, which may be equal to the first predetermined time period. During step SlO the retrieval of input from the A/D converter is continued.
- step S2 If the signal value v returns to a value lower then said predetermined value TH before the end of said predetermined time-period then processing flow continues with step S2. Otherwise the value for the occupancy state is internally saved as occupied in step SlI, and a message signaling this is transmitted in step S 12.
- FIG 8 illustrates the signal flow in a message interpreter MI.
- a radio receiver 20 receives the binary "vehicle present" signals D (optionally with timestamp) from the sensor nodes 10 via the radio link and runs a model based state estimator algorithm to calculate the motion states of the vehicles individually (i.e. each vehicle is represented in the message interpreter).
- the sensor density may be chosen dependent on the required accuracy of the estimation. If a very accurate vehicle tracking is desired multiple sensors per vehicle area may be present.
- the message interpreter MI has a vehicle database facility 32, 34 that comprises state information of vehicles present at the vehicle infrastructure.
- the message interpreter MI further has a sensor map 45describing the spatial location of the sensor nodes 10.
- the sensor nodes may transmit their location, or their position could even be derived by a localization method for wireless sensor networks.
- the message interpreter MI further has an association facility 40 for associating the messages D provided by the sensor nodes 10 with the state information present in the vehicle data base facility 32, 34.
- the association facility 40 may associate the messages received with state information for example with one of the methods Gating, Nearest Neighbor (NN), (Joint) Probabilistic Data Association ((J)DPA), Multiple Hypothesis Tracker (MHT) and the MCMCDA.
- the message interpreter further has a state updating facility 50 for updating the state information on the basis of the messages D associated therewith by the association facility 40. Once the messages D are associated with a particular vehicle the state of that vehicle in a local vehicle data base is updated by the state updating facility 50.
- the association facility 40 and the state updating facility 50 together form a database updating facility DBU.
- a global map builder 65 may exchange this updated information with global map builders of neighboring message interpreters via network interface 60 (wired or wireless) and to receive close to border detections.
- network interface 60 wireless or wireless
- Other uses are also possible to exchange the motion state of crossing vehicles (e.g. to calculate system level features like vehicle density and average velocity, but these are independent from the motion state estimation).
- a message interpreter MI shown in Figure 9, consists of a radio receiver 20, coupled to antenna 22, a processing unit 24 (with memory 28) and a network interface 65, as well as a real-time clock 26.
- a real-time clock may be part of the sensor node, and the sensor node may embed a time-stamp indicative for time at which an event was detected in the message.
- a message interpretor will have a more reliable clock, as it can be more reliable synchronized with a reference clock.
- the network interface 65 couples the message interpreter MI via the communication channel 60 to other message interpreters.
- the microcontroller 24 of Figure 9 processes the received messages D.
- the memory 28 stores the local and global vehicle map and the sensor map as well as the software for carrying out the data association and state estimation tasks.
- separate memories may be present for storing each of these maps and for storing the software.
- dedicated hardware may be present to perform one or more of these tasks.
- the result of the processing i.e. the estimation of the motion states of all sensed vehicles
- the result of the processing is present in the memory of the message interpreters in a distributed way.
- Message interpreters may run additional (cooperative) algorithms to deduct higher level motion characteristics and/or estimate additional vehicle characteristics (e.g. geometry).
- the vehicle tracking system may comprise only a single message interpreter MI.
- MI message interpreter
- the global map builder is superfluous, and local vehicle map is identical to the global vehicle map.
- each message interpreter MI for a respective module comprises hardware as described with reference to Figure 8 and 9. Operation of the message interpreter is further illustrated with respect to Figures 10-13
- Figure 10 schematically shows a part of a vehicle infrastructure having sections Rj 1, Rj, Rj+1.
- a vehicle moves in a direction indicated by arrow X from Rj i, via Rj, to Rj+1.
- FIG 11 shows an overview of a method for detecting the vehicle performed by the message interpreter for section Rj , using the messages obtained from the sensor nodes.
- step S20 the method waits for a message D from a sensor node.
- program flow continues with step S21, where the time t associated with the message is registered.
- the registered time t associated with the message may be a time-stamp embedded in the message or a time read from an internal clock of the message interpreter.
- step S22 it is verified whether the detection is made by a sensor node in a location of section Rj that neighbors one of the neighboring sections Rj i or R+i.
- step S23 the event is communicated via the communication network interface to the message interpreter for that neighboring section.
- step S24 it is determined which vehicle O in the vehicle data base facility is responsible for the detected event. An embodiment of a method used to carry out step S24 is described in more detail in Figure 12. After the responsible vehicle O is identified in Step 25, i.e. an association is made with existing vehicle state information, it is determined in Step 26 whether it is present in the section Rj. If that is the case, control flow continues with Step S27, where the state of vehicle O is estimated. Otherwise control flow returns to step S20. A procedure for estimating the state is described in more detail with reference to Figure 13.
- step S28 it is determined whether the state information implies that the vehicle O has a position in a neighboring section Rj i or Rj+1. In that case the updated state information is transmitted in step S29 to the message interpreter for the neighboring section and control flow returns to step S20. Otherwise the control flow returns immediately to Step S20.
- a method to associate a message D at time t, with a vehicle O is now described in more detail with reference to Figure 12.
- the current state known for the vehicle with that index i is retrieved from the vehicle database facility.
- a probability is determined that the vehicle O caused the detection reported by the message D at time t.
- the vehicle index i is incremented in step S43 and if it is determined in step S44 that i is less than the number of vehicles, the steps S41 to S43 are repeated. Otherwise in step S45 it is determined which vehicle caused the detection reported by the message D at time t with the highest probability.
- the index of that vehicle is returned as the result if the method.
- a method to estimate (update the present estimation of) the state of a vehicle is now described in more detail with reference to Figure 13.
- step S60 the messages Di,...,D n associated with vehicle O are selected.
- step S61 a probability density function is constructed on the basis of the associated messages Di,...,D n .
- step S62 the current state So and time to for vehicle O is retrieved from the vehicle database.
- step S63 it is determined whether the time for which the state S of the vehicle O has to be determined is greater than the time to associated with the current state So. If that is the case, the state S (determined by the estimation method) is the state update of SO to t, performed in step S65. If that is not the case, then the message D relates to a detection preceding the detection that resulted in the earlier estimation for state SO. In that case the state SO is updated using the detection D by the state estimation method in step S64
- the word "comprising” does not exclude other elements or steps, and the indefinite article “a” or “an” does not exclude a plurality. A single component or other unit may fulfill the functions of several items recited in the claims.
- multiple target tracking [1-3] one aims to track all the objects/targets, which are moving in a certain area.
- R defines the set of real numbers whereas the set R + defines the non-negative real numbers.
- the set Z defines the integer values and Z + defines the set of non-negative integer numbers.
- the variable 0 is used either as null, the null- vector or the null-matrix. Its size will become clear from the context.
- Vector x(Z) e R" is defined as a vector depending on time t and is sampled using some sampling method.
- the time t at sampling instant k e Z + is defined as t k e H .
- the variables ⁇ ⁇ e R , i ⁇ e R" and x o k e R ⁇ xk+1 are defined as:
- transpose, inverse and determinant of a matrix Ae R" x " are denoted as A ⁇ , A “1 and I A I respectively.
- E[x I u] The conditional expectation of x given a vector u is denoted as E[x I u] .
- the definitions of E[x] , E[x I u] and cov(x) can be found in [6] sections B4 and B7.
- Gaussian function (0.05 * (1 + 0.05 * (1 + 0.05 * (1 + 0.05 * (1 + 0.05 * (1 + 0.05 * (1 + 0.05 * (1 + 0.05 * (1 + 0.05 * (1 + 0.05 * (1 + 0.05 * (1 + 0.05 * (1 + 0.05 * (1 + 0.05 * (1 + 0.05 * (1 + 0.05 * (1 + 0.05 * (1 + 0.05 * (1 + 0.05 * (1 + 0.05 * (1 + 0.05 * (1 + 0.05 * (1 + ⁇ R"
- each object Beside the positions o and ⁇ each object also has a certain shape or geometry which covers a certain set of positions in R x , i.e. the grey area of Figure
- To define the vectors A 1 we equidistant sample the rectangular box defined by C 0 using a grid with a distance r .
- Each A 1 is a grid point within the set S as graphically depicted in Figure 15.
- the aim is to estimate position, speed and rotation of the object in the case that its acceleration and rotational speed are unknown. Therefore the object's state- vector s(t) G R 5 and process-noise w(t) G R 2 are defined as:
- the vectors d (x 1 , y') ⁇ and ⁇ ' are the i' h object's position- and rotation-vector respectively.
- T' represents the i' h object's rotation-matrix dependent on ⁇ ' .
- the dynamical process of object i with state-vector s' , process-noise w 1 and measurement-vector rri is defined with the following state-space model:
- the objects are observed in R by a camera or a network of sensors. For that
- M 'detection' points are marked within R and collected in the set D c R .
- the position of a detection point is denoted as d e D .
- the k' h detection of the system generates the observation vector z k ' e (R x , R ⁇ if the edge of the i' object covers one of the detection points d k e D at time t k :
- the system does not know which object was detected for it can be any object. As a result the system will not generate z k ' but a general observation vector z k e (R x , R ⁇ , which is yet to be associated with an object. Therefore, due to the k' h detection, the observation vector z k is generated whenever one of the E object covers a detection point d k e D at time t k :
- Figure 16 shows an example of object i which is detected by multiple detection points. The co variance £ of each detection point is also indicated.
- the sampling method of the observation vectors Z 0 k is a form of event sampling [4, 5, 7]. For a new observation vector is sampled whenever an event, i.e. object detection, takes place. With these event samples all N objects are to be tracked. To accomplish that three methods are needed. The first one is the association of the new observation-vector z k to an object i and therefore denote it with z k ' ⁇ Suppose that all associated observation-vectors z n ' are collected in the set Z ⁇ e ⁇ z o k ⁇ ⁇ Then the second method is to estimate m k ' from the observation-set Z k ' .
- Z ⁇ is defined as the set with all observation-vectors from z o k that were associated with object i .
- the set Z k ' e [z o k ] is defined as the set of all observation-vectors Z n which were associated with object i , from which their detection point is still covered by the object.
- At time step k we have the observation- set Z ⁇ _ ⁇ and the observation z k was associated to object i , i.e.
- Z k ' is defined as: With this definition of Z k ' the approach for estimating m k ' , i.e. p[m k ' ⁇ Z k ' ), is given. For clarity we assume that the object's shape is rectangular and that all its detection points are denoted with d ⁇ , with ne N c [0, k] .
- the first step is to position the object on each detection point d ⁇ and mirror its set S into the set O n , as shown in Figure 17 for a single detection. This way we transform the points that are covered by the object, into possible vectors of the object's position o k ' ⁇ O n given that it is detected at the detection point d n .
- the second step is to turn all sets O n simultaneously around their detection point d n .
- For o ⁇ must be inside all the sets O n , Vn e N , and therefore thus inside the intersection of all sets O n , Vn e N , which is denoted as O N .
- the detection point at time-step n are defined as d n e R xy . Meaning that the objects orientation is not directly. However, because every observation vector z n e Z detects the object for one and the same ⁇ , the PDF p(m ⁇ Z) is approximated by sampling in ⁇ , i.e.:
- the main aspect of equation (17) is to determine p(o ⁇ Z, ⁇ ) .
- O n ( ⁇ ) e R To do that we define the set O n ( ⁇ ) e R to be equal to all possible object positions o , given that the object is detected at position d n e z ⁇ (e Z) and that the object's rotation is equal to ⁇ .
- the determination of O n ( ⁇ ) e R is presented in the n the next section. Therefore, if one object is detected at multiple detection points d ⁇ , Vn e N , then the set of all possible object positions o given a certain ⁇ equals O ⁇ ( ⁇ ) :
- Both p ⁇ o ⁇ Z, ⁇ ) and a are related to the set O N ⁇ ) due to the fact that it O N (theta) defines the set of possible object positions o for a given ⁇ .
- O N theta
- both p(m I Z) is calculated according to (6).
- the rest of this section is divided into two parts.
- the first part derives the probability function based on a single detection, i.e. f(o ⁇ z n , ⁇ ) .
- the second part derives the probability function based on a multiple detections, i.e. g(o ⁇ Z, ⁇ ) .
- Figure 20 (right) graphically depicts the determination of O n from the set ⁇ for a given ⁇ and detection point d n .
- Equation (19) is solved with the following Proposition and the fact that
- G(x,a + b,C) G(x-b,a,C) : Proposition 1. Let there exist two Gaussian functions of the random vectors xe R" and me. R* and the matrix Fe R « x " ; G(x,u,U) and G(m,Fx,M) . Then they have the following property:
- Equation (22) If N contains m elements, then calculating equation (22) would result in K m products of m Gaussian functions and sum them afterwards. This would take too much processing power if m is large. That is why equation (22) is calculated differently.
- each detection point d n defines a rectangular set denoted with C ⁇ ( ⁇ ) dependent on rotation ⁇ .
- the intersection of all these rectangular sets is defined with the set C N ( ⁇ ) .
- the first set, O ⁇ ( ⁇ ) shown in Figure 17 defines all possible object positions o based on a single detection at d ⁇ .
- the second set, i.e. O N ( ⁇ ) shown in Figure 18, defines all possible object positions o based on all detections at d n , ⁇ /ne N .
- O N ( ⁇ ) c C N ( ⁇ ) .
- the calculation of (26) is done by applying the following two propositions.
- the first one i.e. Proposition 2
- the second one i.e. Proposition 3, proofs that a product of Gaussians results in a single Gaussian.
- Proposition 2 The product of a summation of Gaussians can be written into a summation of a product of Gaussian:
- Equation (30) is substituted into equation (16) together with f(o ⁇ z ⁇ , ⁇ ) of (27) to calculate p ⁇ o ⁇ Z, ⁇ ) and a, . Substituted these results into (13) gives:
- the PDF p(m I Z) also gives us the probability that a new observation vector is generated by an certain object i . This is discussed in the next section.
- the total probability that a new observation vector z k is generated by object i is equal to the total probability of the measurement- vector m k ' given the observation set .
- This probability we can use which is equal to equation (41).
- the definition of a PDF is that its total probability, i.e. its integral from -oo to ⁇ , is equal to 1.
- To make sure that of equation (31) has a total probability of 1 it is divided by its true probability ⁇
- y 1 and K' are equal to ⁇ and K respectively, which define the approximation of the function as shown in (6.1).
- the simulation case is made such that it contains two interesting situation.
- the objects are tracked using two different association methods.
- the first one is a combination of Gating and detection association of 6.
- the second one is a combination of Gating and Nearest Neighbor.
- This paper presents a method for estimating the position- and rotation- vector of objects from spatially, distributed detections of that object. Each detection is generated at the event that the edge of an object crosses a detection point. From the estimation method a detection associator is also designed. This association method calculates the probability that a new detection was generated by an object i .
- An example of a parking lot shows that the detection association method has no incorrect associated detections in the case that two vehicles cross each other both in parallel as well as orthogonal. If the association method of Nearest Neighbor was used, a large amount of incorrect associated detections were noticed, resulting in a higher state- estimation error.
- the data- assimilation can be further improved with two adjustments.
- the first one is replacing the set S with S E only at the time-instants that the observation vector is received.
- the second improvement is to take the detection points that have not detected anything also in account.
- Equation (53) is equal to (37) for:
- a vector x(t) e R" is defined to depend on time J e R and is sampled using some sampling method. Two different sampling methods are discussed. The first one is time sampling in which samples are generated whenever time Z equals some predefined value. This is either synchronous in time or asynchronous. In the synchronous case the time between two samples is constant and defined as Z s e R + . If the time Z at sampling instant k a e Z + is defined as , with , we define:
- a transition- matrix is defined to relate the vector M(ZJ e R* to a vector x(Z 2 ) e R ⁇ as follows: x
- the transpose, inverse and determinant of a matrix Ae R" x " are denoted as A ⁇ , A "1 and I A I respectively.
- the i' h and maximum eigenvalue of a square matrix A are denoted as A 1 (A) and ⁇ max (A) respectively.
- Ae R" x " and Be R" x " are positive definite, denoted with A >- 0 and ByO, then A>- B denotes A-ByO.
- a ⁇ O denotes A is positive semi- definite.
- PDF probability density function
- Gaussian The Gaussian function (shortly noted as Gaussian) of vectors xe R" and MeR" and matrix is defined as G(x,u,P) :
- the set PDF is defined as ⁇ y (x) : R" — > ⁇ 0, v] with Ve R defined as the Lebesque measure [8] of the set Y , i.e.: 3 Event sampling
- time sampling in which the sampling instant is defined at time for some .
- ⁇ (t) is sampled at t it is denoted as y k .
- This method is formalized by defining the observation vector a at sampling instant k a _ l . Let us define the set containing all the values that t can take between and
- time sampling defines that the next sampling instant, i.e. k a , takes place whenever present time t exceeds the set . Therefore is defined as:
- a well designed H, (z k _, , t) should contain the set of all e e possible values that y(t) can take in between the event instants k e — 1 and k e . Meaning that if t k _, ⁇ t ⁇ t k , then y(t) e H, (z k _, , t) .
- the state vector ⁇ (t) of this system is to be estimated from the observation vectors ⁇ Notice that the estimated states are usually required at all synchronous time samples k , with , e.g., as input to a controller that runs synchronously in time.
- our goal is to construct an event-based state-estimator (EBSE) that provides an estimate of x(t) not only at the event instants t, but also at the sampling instants t, . Therefore, we define a new set of sampling instants t as the a combination of sampling instants due to event sampling, i.e. k e , and time sampling, i.e. k a :
- the estimator calculates the PDF of the state-vector X n given all the observations until t n . This results in a hybrid state-estimator, for at time t n an event can either occur or not, which further implies that measurement data is received or not, respectively. In both cases the estimated state must be updated (not predicted) with all information until t n . Therefore, depending on t n a different PDF must be calculated, i.e.:
- the PDFs of (9) can be described as the Gaussian G(x ⁇ , x ⁇ l ⁇ , P ⁇ l ⁇ ) .
- the square root of the eigenvalues of P nin i.e. define the shape of this Gaussian function. Together with x ⁇ t ⁇ they indicate the bound which surrounds 63% of the possible values for X n .
- This is graphically depicted in Figure 27 for the ID case and Figure 29 for a 2D case, in a top view. The smaller the eigenvalues A 1 (P n ⁇ n ) are, the smaller the estimation-error is.
- the problem of interest in this paper is to construct a state-estimator suitable for the general event sampling method introduced in Section 3 and which is computationally tractable. Furthermore, it is desirable to guarantee that P n ⁇ n has bounded eigenvalues for all n .
- the EBSE estimates X n given the received observation vectors until time t n . Notice that due to the definition of event sampling we can extract information of all the measurement vectors J 0n . For with I 1 e ⁇ t O ⁇ ⁇ and it follows that:
- the variables of (22) are: the e variables depend on and its approximation. As an example these variables are calculated for the method "Send-on-Delta" with ye R .
- Equation (25) is explicitly solved by applying Proposition 1:
- the third step is to approximate (27) as a single Gaussian to retrieve a computationally tractable algorithm.
- the estimate of X n in (27) is described with M n Gaussians.
- M n equals M n-1 N , meaning that M n increases after each sample instant and with it also the processing demand of the EBSE increases.
- Step3 state approximation p(x ⁇ I y O ⁇ e F 0 n ) of (27) is approximated as a single Gaussian with an equal expectation and co variance matrix, i.e.: ⁇ j
- the first two estimators are the EBSE and the asynchronous Kalman filter (AKF) of [13].
- A 0 ⁇ [m] .
- N 5
- the AKF estimates the states only at the event instants .
- the states at are calculated by applying the prediction- step of (14b).
- the third estimator is based on the quantized Kalman filter (QKF) introduced in [21] that uses synchronous time sampling of .
- the QKF can deal with quantized data, which also results in less data transfer, and therefore can be considered as an alternative to EBSE.
- QKF y is the quantized version of with quantization level 0.1 , which corresponds to the "Send-on-Delta" method.
- sampling efficiency ⁇ is also important due to the increased interest in WSNs. For these systems communication is expensive and one
- Figure 33 shows that for the EBSE ⁇ ⁇ 1 at all instants n .
- Tj of the QKF converges to 1. Meaning that for f > 5.5 the estimation error does not change after an update and new samples are mostly used to bound .
- the last aspect on which the three estimators are compared is the total amount of processing time which was needed to estimate all state-vectors.
- Ae R" x " and Be R" xm are defined as the state-space matrices for the time-continuous counterpart of (7). Then it is known [22] that for any sampling period % > 0 , A ⁇ and B ⁇ of (7) are obtained from their corresponding continuous- time matrices A and B as follows: Using (41) one obtains:
Landscapes
- Chemical & Material Sciences (AREA)
- Analytical Chemistry (AREA)
- Physics & Mathematics (AREA)
- General Physics & Mathematics (AREA)
- Traffic Control Systems (AREA)
- Radar Systems Or Details Thereof (AREA)
Abstract
Description
Claims
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| EP09771423.2A EP2374117B1 (en) | 2008-12-12 | 2009-12-11 | Vehicle tracking system, vehicle infrastructure provided with vehicle tracking system and method for tracking vehicles |
Applications Claiming Priority (3)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| EP08171580A EP2196972A1 (en) | 2008-12-12 | 2008-12-12 | Object tracking system, object infrastructure provided with object tracking system and method for tracking objects |
| EP09771423.2A EP2374117B1 (en) | 2008-12-12 | 2009-12-11 | Vehicle tracking system, vehicle infrastructure provided with vehicle tracking system and method for tracking vehicles |
| PCT/NL2009/050758 WO2010068106A1 (en) | 2008-12-12 | 2009-12-11 | Vehicle tracking system, vehicle infrastructure provided with vehicle tracking system and method for tracking vehicles |
Publications (2)
| Publication Number | Publication Date |
|---|---|
| EP2374117A1 true EP2374117A1 (en) | 2011-10-12 |
| EP2374117B1 EP2374117B1 (en) | 2017-08-30 |
Family
ID=40756284
Family Applications (2)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| EP08171580A Withdrawn EP2196972A1 (en) | 2008-12-12 | 2008-12-12 | Object tracking system, object infrastructure provided with object tracking system and method for tracking objects |
| EP09771423.2A Active EP2374117B1 (en) | 2008-12-12 | 2009-12-11 | Vehicle tracking system, vehicle infrastructure provided with vehicle tracking system and method for tracking vehicles |
Family Applications Before (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| EP08171580A Withdrawn EP2196972A1 (en) | 2008-12-12 | 2008-12-12 | Object tracking system, object infrastructure provided with object tracking system and method for tracking objects |
Country Status (2)
| Country | Link |
|---|---|
| EP (2) | EP2196972A1 (en) |
| WO (1) | WO2010068106A1 (en) |
Families Citing this family (2)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| US7991550B2 (en) * | 2006-02-03 | 2011-08-02 | GM Global Technology Operations LLC | Method and apparatus for on-vehicle calibration and orientation of object-tracking systems |
| CN110542885B (en) * | 2019-08-13 | 2021-09-21 | 北京理工大学 | Millimeter wave radar target tracking method in complex traffic environment |
Family Cites Families (4)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| US5801943A (en) * | 1993-07-23 | 1998-09-01 | Condition Monitoring Systems | Traffic surveillance and simulation apparatus |
| US6697103B1 (en) * | 1998-03-19 | 2004-02-24 | Dennis Sunga Fernandez | Integrated network for monitoring remote objects |
| AUPQ684600A0 (en) * | 2000-04-11 | 2000-05-11 | Safehouse International Limited | An object monitoring system |
| WO2006068463A1 (en) * | 2004-12-24 | 2006-06-29 | Ultrawaves Design Holding B.V. | Intelligent distributed image processing |
-
2008
- 2008-12-12 EP EP08171580A patent/EP2196972A1/en not_active Withdrawn
-
2009
- 2009-12-11 EP EP09771423.2A patent/EP2374117B1/en active Active
- 2009-12-11 WO PCT/NL2009/050758 patent/WO2010068106A1/en not_active Ceased
Non-Patent Citations (1)
| Title |
|---|
| See references of WO2010068106A1 * |
Also Published As
| Publication number | Publication date |
|---|---|
| EP2374117B1 (en) | 2017-08-30 |
| WO2010068106A1 (en) | 2010-06-17 |
| EP2196972A1 (en) | 2010-06-16 |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| EP2370965B1 (en) | Traffic information unit, traffic information system, vehicle management system, vehicle and method of controlling a vehicle | |
| Vasuhi et al. | Target tracking using interactive multiple model for wireless sensor network | |
| Subhan et al. | Kalman filter‐based hybrid indoor position estimation technique in bluetooth networks | |
| CN105467382A (en) | SVM (Support Vector Machine)-based multi-sensor target tracking data fusion algorithm and system thereof | |
| Yu et al. | Interacting multiple model filter-based distributed target tracking algorithm in underwater wireless sensor networks | |
| Finke et al. | Cooperative control via task load balancing for networked uninhabited autonomous vehicles | |
| Tsaramirsis et al. | Smart parking: An IoT application for smart city | |
| Gloudemans et al. | Interstate-24 motion: Closing the loop on smart mobility | |
| CN114287023A (en) | Multi-sensor learning system for traffic prediction | |
| Ghods et al. | Real-time estimation of turning movement counts at signalized intersections using signal phase information | |
| EP2374117B1 (en) | Vehicle tracking system, vehicle infrastructure provided with vehicle tracking system and method for tracking vehicles | |
| Yin et al. | Queue intensity adaptive signal control for isolated intersection based on vehicle trajectory data | |
| Ng et al. | LoRa-based indoor positioning in dynamic industrial environments using deep Gaussian process regression and temporal-based enhancements | |
| CN114916059B (en) | WiFi fingerprint sparse map extension method based on interval random logarithmic shadow model | |
| Gratzer et al. | Short-term collision estimation by stochastic predictions in multi-agent intersection traffic | |
| Piperigkos et al. | Robust cooperative sparse representation solutions for detecting and mitigating spoofing attacks in autonomous vehicles | |
| Worrall et al. | A probabilistic method for detecting impending vehicle interactions | |
| Schubert et al. | Unifying Bayesian networks and IMM filtering for improved multiple model estimation | |
| Baek et al. | Method for estimating population OD matrix based on probe vehicles | |
| CN117453843A (en) | Data processing method and device, storage medium and electronic equipment | |
| Duarte et al. | Chaotic statistical downscaling (CSD): Application and comparison in the Bogotá river basin | |
| Claessens et al. | Context driven tracking using particle filters | |
| Piperigkos et al. | Cooperative plug-and-play-KalmanNet for 4D situational awareness in autonomous driving | |
| Nantes et al. | Bayesian inference of traffic volumes based on Bluetooth data | |
| Sutarto et al. | Modeling, identification, estimation, and simulation of urban traffic flow in Jakarta and Bandung |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PUAI | Public reference made under article 153(3) epc to a published international application that has entered the european phase |
Free format text: ORIGINAL CODE: 0009012 |
|
| 17P | Request for examination filed |
Effective date: 20110705 |
|
| AK | Designated contracting states |
Kind code of ref document: A1 Designated state(s): AT BE BG CH CY CZ DE DK EE ES FI FR GB GR HR HU IE IS IT LI LT LU LV MC MK MT NL NO PL PT RO SE SI SK SM TR |
|
| DAX | Request for extension of the european patent (deleted) | ||
| RAP1 | Party data changed (applicant data changed or rights of an application transferred) |
Owner name: NEDERLANDSE ORGANISATIE VOOR TOEGEPAST- NATUURWETE |
|
| GRAP | Despatch of communication of intention to grant a patent |
Free format text: ORIGINAL CODE: EPIDOSNIGR1 |
|
| INTG | Intention to grant announced |
Effective date: 20170322 |
|
| RIN1 | Information on inventor provided before grant (corrected) |
Inventor name: SIJS, JORIS Inventor name: THEEUWES, JOHANNES, ADRIANUS, CORNELIS Inventor name: DOODEMAN, GERARDUS, JOHANNES, NICOLAAS Inventor name: PAPP, ZOLTAN Inventor name: NELISSE, MARTIN, WILLEM Inventor name: DRIESSEN, BART |
|
| GRAS | Grant fee paid |
Free format text: ORIGINAL CODE: EPIDOSNIGR3 |
|
| GRAA | (expected) grant |
Free format text: ORIGINAL CODE: 0009210 |
|
| AK | Designated contracting states |
Kind code of ref document: B1 Designated state(s): AT BE BG CH CY CZ DE DK EE ES FI FR GB GR HR HU IE IS IT LI LT LU LV MC MK MT NL NO PL PT RO SE SI SK SM TR |
|
| REG | Reference to a national code |
Ref country code: GB Ref legal event code: FG4D |
|
| REG | Reference to a national code |
Ref country code: CH Ref legal event code: EP |
|
| REG | Reference to a national code |
Ref country code: AT Ref legal event code: REF Ref document number: 924283 Country of ref document: AT Kind code of ref document: T Effective date: 20170915 |
|
| REG | Reference to a national code |
Ref country code: IE Ref legal event code: FG4D |
|
| REG | Reference to a national code |
Ref country code: DE Ref legal event code: R096 Ref document number: 602009048071 Country of ref document: DE |
|
| REG | Reference to a national code |
Ref country code: NL Ref legal event code: FP |
|
| REG | Reference to a national code |
Ref country code: FR Ref legal event code: PLFP Year of fee payment: 9 |
|
| REG | Reference to a national code |
Ref country code: LT Ref legal event code: MG4D |
|
| REG | Reference to a national code |
Ref country code: AT Ref legal event code: MK05 Ref document number: 924283 Country of ref document: AT Kind code of ref document: T Effective date: 20170830 |
|
| PG25 | Lapsed in a contracting state [announced via postgrant information from national office to epo] |
Ref country code: SE Free format text: LAPSE BECAUSE OF FAILURE TO SUBMIT A TRANSLATION OF THE DESCRIPTION OR TO PAY THE FEE WITHIN THE PRESCRIBED TIME-LIMIT Effective date: 20170830 Ref country code: AT Free format text: LAPSE BECAUSE OF FAILURE TO SUBMIT A TRANSLATION OF THE DESCRIPTION OR TO PAY THE FEE WITHIN THE PRESCRIBED TIME-LIMIT Effective date: 20170830 Ref country code: FI Free format text: LAPSE BECAUSE OF FAILURE TO SUBMIT A TRANSLATION OF THE DESCRIPTION OR TO PAY THE FEE WITHIN THE PRESCRIBED TIME-LIMIT Effective date: 20170830 Ref country code: NO Free format text: LAPSE BECAUSE OF FAILURE TO SUBMIT A TRANSLATION OF THE DESCRIPTION OR TO PAY THE FEE WITHIN THE PRESCRIBED TIME-LIMIT Effective date: 20171130 Ref country code: HR Free format text: LAPSE BECAUSE OF FAILURE TO SUBMIT A TRANSLATION OF THE DESCRIPTION OR TO PAY THE FEE WITHIN THE PRESCRIBED TIME-LIMIT Effective date: 20170830 Ref country code: LT Free format text: LAPSE BECAUSE OF FAILURE TO SUBMIT A TRANSLATION OF THE DESCRIPTION OR TO PAY THE FEE WITHIN THE PRESCRIBED TIME-LIMIT Effective date: 20170830 |
|
| PG25 | Lapsed in a contracting state [announced via postgrant information from national office to epo] |
Ref country code: LV Free format text: LAPSE BECAUSE OF FAILURE TO SUBMIT A TRANSLATION OF THE DESCRIPTION OR TO PAY THE FEE WITHIN THE PRESCRIBED TIME-LIMIT Effective date: 20170830 Ref country code: IS Free format text: LAPSE BECAUSE OF FAILURE TO SUBMIT A TRANSLATION OF THE DESCRIPTION OR TO PAY THE FEE WITHIN THE PRESCRIBED TIME-LIMIT Effective date: 20171230 Ref country code: BG Free format text: LAPSE BECAUSE OF FAILURE TO SUBMIT A TRANSLATION OF THE DESCRIPTION OR TO PAY THE FEE WITHIN THE PRESCRIBED TIME-LIMIT Effective date: 20171130 Ref country code: GR Free format text: LAPSE BECAUSE OF FAILURE TO SUBMIT A TRANSLATION OF THE DESCRIPTION OR TO PAY THE FEE WITHIN THE PRESCRIBED TIME-LIMIT Effective date: 20171201 Ref country code: ES Free format text: LAPSE BECAUSE OF FAILURE TO SUBMIT A TRANSLATION OF THE DESCRIPTION OR TO PAY THE FEE WITHIN THE PRESCRIBED TIME-LIMIT Effective date: 20170830 |
|
| PG25 | Lapsed in a contracting state [announced via postgrant information from national office to epo] |
Ref country code: PL Free format text: LAPSE BECAUSE OF FAILURE TO SUBMIT A TRANSLATION OF THE DESCRIPTION OR TO PAY THE FEE WITHIN THE PRESCRIBED TIME-LIMIT Effective date: 20170830 Ref country code: DK Free format text: LAPSE BECAUSE OF FAILURE TO SUBMIT A TRANSLATION OF THE DESCRIPTION OR TO PAY THE FEE WITHIN THE PRESCRIBED TIME-LIMIT Effective date: 20170830 Ref country code: CZ Free format text: LAPSE BECAUSE OF FAILURE TO SUBMIT A TRANSLATION OF THE DESCRIPTION OR TO PAY THE FEE WITHIN THE PRESCRIBED TIME-LIMIT Effective date: 20170830 Ref country code: RO Free format text: LAPSE BECAUSE OF FAILURE TO SUBMIT A TRANSLATION OF THE DESCRIPTION OR TO PAY THE FEE WITHIN THE PRESCRIBED TIME-LIMIT Effective date: 20170830 |
|
| PG25 | Lapsed in a contracting state [announced via postgrant information from national office to epo] |
Ref country code: IT Free format text: LAPSE BECAUSE OF FAILURE TO SUBMIT A TRANSLATION OF THE DESCRIPTION OR TO PAY THE FEE WITHIN THE PRESCRIBED TIME-LIMIT Effective date: 20170830 Ref country code: SK Free format text: LAPSE BECAUSE OF FAILURE TO SUBMIT A TRANSLATION OF THE DESCRIPTION OR TO PAY THE FEE WITHIN THE PRESCRIBED TIME-LIMIT Effective date: 20170830 Ref country code: SM Free format text: LAPSE BECAUSE OF FAILURE TO SUBMIT A TRANSLATION OF THE DESCRIPTION OR TO PAY THE FEE WITHIN THE PRESCRIBED TIME-LIMIT Effective date: 20170830 Ref country code: EE Free format text: LAPSE BECAUSE OF FAILURE TO SUBMIT A TRANSLATION OF THE DESCRIPTION OR TO PAY THE FEE WITHIN THE PRESCRIBED TIME-LIMIT Effective date: 20170830 |
|
| REG | Reference to a national code |
Ref country code: DE Ref legal event code: R097 Ref document number: 602009048071 Country of ref document: DE |
|
| PLBE | No opposition filed within time limit |
Free format text: ORIGINAL CODE: 0009261 |
|
| STAA | Information on the status of an ep patent application or granted ep patent |
Free format text: STATUS: NO OPPOSITION FILED WITHIN TIME LIMIT |
|
| REG | Reference to a national code |
Ref country code: CH Ref legal event code: PL |
|
| 26N | No opposition filed |
Effective date: 20180531 |
|
| PG25 | Lapsed in a contracting state [announced via postgrant information from national office to epo] |
Ref country code: SI Free format text: LAPSE BECAUSE OF FAILURE TO SUBMIT A TRANSLATION OF THE DESCRIPTION OR TO PAY THE FEE WITHIN THE PRESCRIBED TIME-LIMIT Effective date: 20170830 |
|
| REG | Reference to a national code |
Ref country code: IE Ref legal event code: MM4A |
|
| PG25 | Lapsed in a contracting state [announced via postgrant information from national office to epo] |
Ref country code: MT Free format text: LAPSE BECAUSE OF NON-PAYMENT OF DUE FEES Effective date: 20171211 Ref country code: LU Free format text: LAPSE BECAUSE OF NON-PAYMENT OF DUE FEES Effective date: 20171211 |
|
| REG | Reference to a national code |
Ref country code: BE Ref legal event code: MM Effective date: 20171231 |
|
| PG25 | Lapsed in a contracting state [announced via postgrant information from national office to epo] |
Ref country code: IE Free format text: LAPSE BECAUSE OF NON-PAYMENT OF DUE FEES Effective date: 20171211 |
|
| PG25 | Lapsed in a contracting state [announced via postgrant information from national office to epo] |
Ref country code: BE Free format text: LAPSE BECAUSE OF NON-PAYMENT OF DUE FEES Effective date: 20171231 Ref country code: CH Free format text: LAPSE BECAUSE OF NON-PAYMENT OF DUE FEES Effective date: 20171231 Ref country code: LI Free format text: LAPSE BECAUSE OF NON-PAYMENT OF DUE FEES Effective date: 20171231 |
|
| PG25 | Lapsed in a contracting state [announced via postgrant information from national office to epo] |
Ref country code: HU Free format text: LAPSE BECAUSE OF FAILURE TO SUBMIT A TRANSLATION OF THE DESCRIPTION OR TO PAY THE FEE WITHIN THE PRESCRIBED TIME-LIMIT; INVALID AB INITIO Effective date: 20091211 Ref country code: MC Free format text: LAPSE BECAUSE OF FAILURE TO SUBMIT A TRANSLATION OF THE DESCRIPTION OR TO PAY THE FEE WITHIN THE PRESCRIBED TIME-LIMIT Effective date: 20170830 |
|
| PG25 | Lapsed in a contracting state [announced via postgrant information from national office to epo] |
Ref country code: CY Free format text: LAPSE BECAUSE OF NON-PAYMENT OF DUE FEES Effective date: 20170830 |
|
| PG25 | Lapsed in a contracting state [announced via postgrant information from national office to epo] |
Ref country code: MK Free format text: LAPSE BECAUSE OF FAILURE TO SUBMIT A TRANSLATION OF THE DESCRIPTION OR TO PAY THE FEE WITHIN THE PRESCRIBED TIME-LIMIT Effective date: 20170830 |
|
| PG25 | Lapsed in a contracting state [announced via postgrant information from national office to epo] |
Ref country code: TR Free format text: LAPSE BECAUSE OF FAILURE TO SUBMIT A TRANSLATION OF THE DESCRIPTION OR TO PAY THE FEE WITHIN THE PRESCRIBED TIME-LIMIT Effective date: 20170830 |
|
| PG25 | Lapsed in a contracting state [announced via postgrant information from national office to epo] |
Ref country code: PT Free format text: LAPSE BECAUSE OF FAILURE TO SUBMIT A TRANSLATION OF THE DESCRIPTION OR TO PAY THE FEE WITHIN THE PRESCRIBED TIME-LIMIT Effective date: 20170830 |
|
| P01 | Opt-out of the competence of the unified patent court (upc) registered |
Effective date: 20230522 |
|
| PGFP | Annual fee paid to national office [announced via postgrant information from national office to epo] |
Ref country code: DE Payment date: 20251211 Year of fee payment: 17 |
|
| PGFP | Annual fee paid to national office [announced via postgrant information from national office to epo] |
Ref country code: GB Payment date: 20251219 Year of fee payment: 17 |
|
| PGFP | Annual fee paid to national office [announced via postgrant information from national office to epo] |
Ref country code: NL Payment date: 20251219 Year of fee payment: 17 Ref country code: FR Payment date: 20251222 Year of fee payment: 17 |