EP3640679B1 - A method for assigning ego vehicle to a lane - Google Patents

A method for assigning ego vehicle to a lane Download PDF

Info

Publication number
EP3640679B1
EP3640679B1 EP18200390.5A EP18200390A EP3640679B1 EP 3640679 B1 EP3640679 B1 EP 3640679B1 EP 18200390 A EP18200390 A EP 18200390A EP 3640679 B1 EP3640679 B1 EP 3640679B1
Authority
EP
European Patent Office
Prior art keywords
vehicle
lane
likelihood
computer
momentary
Prior art date
Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
Active
Application number
EP18200390.5A
Other languages
German (de)
French (fr)
Other versions
EP3640679A1 (en
Inventor
Viktor Runemalm
Markus Hammarsten
Jonatan Silvlin
Albin Manhof
Current Assignee (The listed assignees may be inaccurate. Google has not performed a legal analysis and makes no representation or warranty as to the accuracy of the list.)
Zenuity AB
Original Assignee
Zenuity AB
Priority date (The priority date is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the date listed.)
Filing date
Publication date
Application filed by Zenuity AB filed Critical Zenuity AB
Priority to EP18200390.5A priority Critical patent/EP3640679B1/en
Priority to US16/600,719 priority patent/US11776392B2/en
Priority to CN201910977906.1A priority patent/CN111123908A/en
Publication of EP3640679A1 publication Critical patent/EP3640679A1/en
Application granted granted Critical
Publication of EP3640679B1 publication Critical patent/EP3640679B1/en
Active legal-status Critical Current
Anticipated expiration legal-status Critical

Links

Images

Classifications

    • GPHYSICS
    • G01MEASURING; TESTING
    • G01SRADIO DIRECTION-FINDING; RADIO NAVIGATION; DETERMINING DISTANCE OR VELOCITY BY USE OF RADIO WAVES; LOCATING OR PRESENCE-DETECTING BY USE OF THE REFLECTION OR RERADIATION OF RADIO WAVES; ANALOGOUS ARRANGEMENTS USING OTHER WAVES
    • G01S19/00Satellite radio beacon positioning systems; Determining position, velocity or attitude using signals transmitted by such systems
    • G01S19/01Satellite radio beacon positioning systems transmitting time-stamped messages, e.g. GPS [Global Positioning System], GLONASS [Global Orbiting Navigation Satellite System] or GALILEO
    • G01S19/13Receivers
    • GPHYSICS
    • G05CONTROLLING; REGULATING
    • G05DSYSTEMS FOR CONTROLLING OR REGULATING NON-ELECTRIC VARIABLES
    • G05D1/00Control of position, course or altitude of land, water, air, or space vehicles, e.g. automatic pilot
    • G05D1/02Control of position or course in two dimensions
    • G05D1/021Control of position or course in two dimensions specially adapted to land vehicles
    • G05D1/0231Control of position or course in two dimensions specially adapted to land vehicles using optical position detecting means
    • G05D1/0234Control of position or course in two dimensions specially adapted to land vehicles using optical position detecting means using optical markers or beacons
    • G05D1/0236Control of position or course in two dimensions specially adapted to land vehicles using optical position detecting means using optical markers or beacons in combination with a laser
    • GPHYSICS
    • G08SIGNALLING
    • G08GTRAFFIC CONTROL SYSTEMS
    • G08G1/00Traffic control systems for road vehicles
    • G08G1/01Detecting movement of traffic to be counted or controlled
    • G08G1/0104Measuring and analyzing of parameters relative to traffic conditions
    • G08G1/0137Measuring and analyzing of parameters relative to traffic conditions for specific applications
    • BPERFORMING OPERATIONS; TRANSPORTING
    • B60VEHICLES IN GENERAL
    • B60WCONJOINT CONTROL OF VEHICLE SUB-UNITS OF DIFFERENT TYPE OR DIFFERENT FUNCTION; CONTROL SYSTEMS SPECIALLY ADAPTED FOR HYBRID VEHICLES; ROAD VEHICLE DRIVE CONTROL SYSTEMS FOR PURPOSES NOT RELATED TO THE CONTROL OF A PARTICULAR SUB-UNIT
    • B60W30/00Purposes of road vehicle drive control systems not related to the control of a particular sub-unit, e.g. of systems using conjoint control of vehicle sub-units, or advanced driver assistance systems for ensuring comfort, stability and safety or drive control systems for propelling or retarding the vehicle
    • B60W30/10Path keeping
    • B60W30/12Lane keeping
    • GPHYSICS
    • G01MEASURING; TESTING
    • G01CMEASURING DISTANCES, LEVELS OR BEARINGS; SURVEYING; NAVIGATION; GYROSCOPIC INSTRUMENTS; PHOTOGRAMMETRY OR VIDEOGRAMMETRY
    • G01C21/00Navigation; Navigational instruments not provided for in groups G01C1/00 - G01C19/00
    • G01C21/26Navigation; Navigational instruments not provided for in groups G01C1/00 - G01C19/00 specially adapted for navigation in a road network
    • G01C21/28Navigation; Navigational instruments not provided for in groups G01C1/00 - G01C19/00 specially adapted for navigation in a road network with correlation of data from several navigational instruments
    • GPHYSICS
    • G01MEASURING; TESTING
    • G01CMEASURING DISTANCES, LEVELS OR BEARINGS; SURVEYING; NAVIGATION; GYROSCOPIC INSTRUMENTS; PHOTOGRAMMETRY OR VIDEOGRAMMETRY
    • G01C21/00Navigation; Navigational instruments not provided for in groups G01C1/00 - G01C19/00
    • G01C21/26Navigation; Navigational instruments not provided for in groups G01C1/00 - G01C19/00 specially adapted for navigation in a road network
    • G01C21/34Route searching; Route guidance
    • G01C21/36Input/output arrangements for on-board computers
    • G01C21/3626Details of the output of route guidance instructions
    • G01C21/3658Lane guidance
    • GPHYSICS
    • G05CONTROLLING; REGULATING
    • G05DSYSTEMS FOR CONTROLLING OR REGULATING NON-ELECTRIC VARIABLES
    • G05D1/00Control of position, course or altitude of land, water, air, or space vehicles, e.g. automatic pilot
    • G05D1/0088Control of position, course or altitude of land, water, air, or space vehicles, e.g. automatic pilot characterized by the autonomous decision making process, e.g. artificial intelligence, predefined behaviours
    • GPHYSICS
    • G05CONTROLLING; REGULATING
    • G05DSYSTEMS FOR CONTROLLING OR REGULATING NON-ELECTRIC VARIABLES
    • G05D1/00Control of position, course or altitude of land, water, air, or space vehicles, e.g. automatic pilot
    • G05D1/02Control of position or course in two dimensions
    • G05D1/021Control of position or course in two dimensions specially adapted to land vehicles
    • G05D1/0212Control of position or course in two dimensions specially adapted to land vehicles with means for defining a desired trajectory
    • G05D1/0214Control of position or course in two dimensions specially adapted to land vehicles with means for defining a desired trajectory in accordance with safety or protection criteria, e.g. avoiding hazardous areas
    • GPHYSICS
    • G05CONTROLLING; REGULATING
    • G05DSYSTEMS FOR CONTROLLING OR REGULATING NON-ELECTRIC VARIABLES
    • G05D1/00Control of position, course or altitude of land, water, air, or space vehicles, e.g. automatic pilot
    • G05D1/02Control of position or course in two dimensions
    • G05D1/021Control of position or course in two dimensions specially adapted to land vehicles
    • G05D1/0212Control of position or course in two dimensions specially adapted to land vehicles with means for defining a desired trajectory
    • G05D1/0221Control of position or course in two dimensions specially adapted to land vehicles with means for defining a desired trajectory involving a learning process
    • GPHYSICS
    • G05CONTROLLING; REGULATING
    • G05DSYSTEMS FOR CONTROLLING OR REGULATING NON-ELECTRIC VARIABLES
    • G05D1/00Control of position, course or altitude of land, water, air, or space vehicles, e.g. automatic pilot
    • G05D1/02Control of position or course in two dimensions
    • G05D1/021Control of position or course in two dimensions specially adapted to land vehicles
    • G05D1/0212Control of position or course in two dimensions specially adapted to land vehicles with means for defining a desired trajectory
    • G05D1/0223Control of position or course in two dimensions specially adapted to land vehicles with means for defining a desired trajectory involving speed control of the vehicle
    • GPHYSICS
    • G05CONTROLLING; REGULATING
    • G05DSYSTEMS FOR CONTROLLING OR REGULATING NON-ELECTRIC VARIABLES
    • G05D1/00Control of position, course or altitude of land, water, air, or space vehicles, e.g. automatic pilot
    • G05D1/02Control of position or course in two dimensions
    • G05D1/021Control of position or course in two dimensions specially adapted to land vehicles
    • G05D1/0231Control of position or course in two dimensions specially adapted to land vehicles using optical position detecting means
    • GPHYSICS
    • G05CONTROLLING; REGULATING
    • G05DSYSTEMS FOR CONTROLLING OR REGULATING NON-ELECTRIC VARIABLES
    • G05D1/00Control of position, course or altitude of land, water, air, or space vehicles, e.g. automatic pilot
    • G05D1/02Control of position or course in two dimensions
    • G05D1/021Control of position or course in two dimensions specially adapted to land vehicles
    • G05D1/0231Control of position or course in two dimensions specially adapted to land vehicles using optical position detecting means
    • G05D1/0238Control of position or course in two dimensions specially adapted to land vehicles using optical position detecting means using obstacle or wall sensors
    • G05D1/024Control of position or course in two dimensions specially adapted to land vehicles using optical position detecting means using obstacle or wall sensors in combination with a laser
    • GPHYSICS
    • G05CONTROLLING; REGULATING
    • G05DSYSTEMS FOR CONTROLLING OR REGULATING NON-ELECTRIC VARIABLES
    • G05D1/00Control of position, course or altitude of land, water, air, or space vehicles, e.g. automatic pilot
    • G05D1/02Control of position or course in two dimensions
    • G05D1/021Control of position or course in two dimensions specially adapted to land vehicles
    • G05D1/0231Control of position or course in two dimensions specially adapted to land vehicles using optical position detecting means
    • G05D1/0246Control of position or course in two dimensions specially adapted to land vehicles using optical position detecting means using a video camera in combination with image processing means
    • G05D1/0251Control of position or course in two dimensions specially adapted to land vehicles using optical position detecting means using a video camera in combination with image processing means extracting 3D information from a plurality of images taken from different locations, e.g. stereo vision
    • GPHYSICS
    • G05CONTROLLING; REGULATING
    • G05DSYSTEMS FOR CONTROLLING OR REGULATING NON-ELECTRIC VARIABLES
    • G05D1/00Control of position, course or altitude of land, water, air, or space vehicles, e.g. automatic pilot
    • G05D1/02Control of position or course in two dimensions
    • G05D1/021Control of position or course in two dimensions specially adapted to land vehicles
    • G05D1/0257Control of position or course in two dimensions specially adapted to land vehicles using a radar
    • GPHYSICS
    • G05CONTROLLING; REGULATING
    • G05DSYSTEMS FOR CONTROLLING OR REGULATING NON-ELECTRIC VARIABLES
    • G05D1/00Control of position, course or altitude of land, water, air, or space vehicles, e.g. automatic pilot
    • G05D1/02Control of position or course in two dimensions
    • G05D1/021Control of position or course in two dimensions specially adapted to land vehicles
    • G05D1/0276Control of position or course in two dimensions specially adapted to land vehicles using signals provided by a source external to the vehicle
    • G05D1/0278Control of position or course in two dimensions specially adapted to land vehicles using signals provided by a source external to the vehicle using satellite positioning signals, e.g. GPS
    • GPHYSICS
    • G05CONTROLLING; REGULATING
    • G05DSYSTEMS FOR CONTROLLING OR REGULATING NON-ELECTRIC VARIABLES
    • G05D1/00Control of position, course or altitude of land, water, air, or space vehicles, e.g. automatic pilot
    • G05D1/02Control of position or course in two dimensions
    • G05D1/021Control of position or course in two dimensions specially adapted to land vehicles
    • G05D1/0276Control of position or course in two dimensions specially adapted to land vehicles using signals provided by a source external to the vehicle
    • G05D1/0285Control of position or course in two dimensions specially adapted to land vehicles using signals provided by a source external to the vehicle using signals transmitted via a public communication network, e.g. GSM network
    • GPHYSICS
    • G06COMPUTING; CALCULATING OR COUNTING
    • G06VIMAGE OR VIDEO RECOGNITION OR UNDERSTANDING
    • G06V20/00Scenes; Scene-specific elements
    • G06V20/50Context or environment of the image
    • G06V20/56Context or environment of the image exterior to a vehicle by using sensors mounted on the vehicle
    • G06V20/588Recognition of the road, e.g. of lane markings; Recognition of the vehicle driving pattern in relation to the road
    • GPHYSICS
    • G08SIGNALLING
    • G08GTRAFFIC CONTROL SYSTEMS
    • G08G1/00Traffic control systems for road vehicles
    • G08G1/123Traffic control systems for road vehicles indicating the position of vehicles, e.g. scheduled vehicles; Managing passenger vehicles circulating according to a fixed timetable, e.g. buses, trains, trams
    • G08G1/133Traffic control systems for road vehicles indicating the position of vehicles, e.g. scheduled vehicles; Managing passenger vehicles circulating according to a fixed timetable, e.g. buses, trains, trams within the vehicle ; Indicators inside the vehicles or at stops
    • G08G1/137Traffic control systems for road vehicles indicating the position of vehicles, e.g. scheduled vehicles; Managing passenger vehicles circulating according to a fixed timetable, e.g. buses, trains, trams within the vehicle ; Indicators inside the vehicles or at stops the indicator being in the form of a map
    • GPHYSICS
    • G08SIGNALLING
    • G08GTRAFFIC CONTROL SYSTEMS
    • G08G1/00Traffic control systems for road vehicles
    • G08G1/16Anti-collision systems
    • G08G1/167Driving aids for lane monitoring, lane changing, e.g. blind spot detection

Definitions

  • the present invention relates to system, methods and arrangements for determining a position of a vehicle and in particular for a solution for determining in which lane the vehicle is in and assigning the vehicle to the lane.
  • Satellite based positioning systems Global Navigation Satellite Systems, GNSS
  • GPS Global Positioning system
  • GLONASS Globalnaya Navigazionnaya Sputnikovaya
  • Galileo Beidou
  • other regional systems are often not accurate enough to rely on solely for determining a position of a moving vehicle.
  • Chinese patent document CN107221195 discusses an anti-collision solution for vehicle lane prediction and lane level map wherein a lane level is determined using a unique link ID number for each lane and including vehicle travel history into the algorithm for determining the current lane of the vehicle.
  • US patent document US2016/0335507 discloses a method and system for vehicle localization.
  • the method involves identifying expected lane lines based on a lane level map and a position of the vehicle. Further, a set of observed lines are identified based on images captured by an imaging device of the vehicle and the position of the vehicle. Then, the expected lines are matched to the observed lines in order to determine a likelihood score based on the matching, and finally, a best vehicle pose is determined based on the likelihood score
  • a method for assigning a vehicle to a lane in a road for a vehicle comprising obtaining an estimated pose, E, of the vehicle, obtaining an uncertainty value, V, to the estimated pose of the vehicle, projecting the pose, E, onto nearby lanes in a map, obtaining position uncertainty, U, of each lane, creating a distribution, P, by combining uncertainty V and U for each nearby lane, creating a current momentary likelihood hypothesis for the vehicle for being in each lane by evaluating distribution, P, with pose, E, for each projection, adjusting the current momentary likelihood for each lane with a prior determined earlier; and comparing all the adjusted momentary likelihoods and determining the lane with the highest likelihood.
  • the step of adjusting current momentary likelihoods may comprise, for each likelihood from a previous iteration: checking for logical connection associated to a current momentary likelihood and adjusting the current momentary likelihood with the value of the likelihood from the previous iteration if the connection is within a set distance from the pose, E, and if the current momentary likelihood has not been adjusted previously; and adjusting current momentary likelihoods, that was not adjusted in previous step, with a minimum value;
  • the set distance may be set to shorter than 10 meters, or more preferably shorter than 5 meters, or even more preferably 2 meters or shorter.
  • Nearby lanes may be located within 20 meters from the estimated pose, or more preferably within 10 meters from the estimated pose, or even more preferably within 5 meters or shorter from the estimated pose.
  • the method may further comprise a step of normalizing the current likelihoods.
  • the method may further comprise a step of setting all likelihoods below a threshold to a fixed minimum likelihood value.
  • the step of normalizing the current likelihoods may comprise determining the maximum value of the most probably current likelihood and subtracting this maximum value from all current likelihoods.
  • each lanes probabilistic value drops to zero at some distance away from the centre of the respective lane.
  • the step of adjusting momentary likelihood may comprise checking logical connection between lane hypotheses in the map.
  • the pose may comprise a position in map coordinates and a direction of travel.
  • Acquiring estimated pose of the vehicle may comprise acquiring sensor data from at least one of a Global Navigation Satellite System, camera, vehicle parameters, and Lidar.
  • the method is advantageously implemented in autonomous vehicles or advanced driver-assistance systems vehicles.
  • a vehicle control system comprising at least one processor, at least one memory for storing instructions and data, at least one sensor interface, and at least one communication interface.
  • the processor is arranged to execute instruction sets stored in the memory to acquire data via the sensor interface and execute a method of obtaining an estimated pose, E, of the vehicle, obtaining an uncertainty value, V, to the estimated pose of the vehicle, projecting the pose, E, onto nearby lanes in a map, obtaining position uncertainty, U, of each lane, creating a distribution, P, by combining uncertainty V and U for each nearby lane, creating a current momentary likelihood hypothesis for the vehicle for being in each lane by evaluating distribution, P, with pose, E, for each projection, adjusting the current momentary likelihood for each lane with a prior determined earlier; comparing all the adjusted momentary likelihoods and determining the lane with the highest likelihood, and sending control parameters on the communication interface for controlling the vehicle in accordance with the determined most probable lane.
  • the step of adjusting current momentary likelihoods may comprise, for each likelihood from a previous iteration: checking for logical connection associated to a current momentary likelihood and adjusting the current momentary likelihood with the value of the likelihood from the previous iteration if the connection is within a set distance from the pose, E, and if the current momentary likelihood has not been adjusted previously; and adjusting current momentary likelihoods that was not adjusted in previous step, with a minimum value;
  • a vehicle comprising a vehicle control system, controlling driving functions of the vehicle, comprising at least one processor, at least one memory for storing instructions and data, at least one sensor input interface, at least one sensor for acquiring an estimation of vehicle position, wherein the vehicle control system is arranged to execute instructions in the processing unit for executing the above method.
  • Still another embodiment is provided, a computer program product arranged to execute the above method.
  • the proposed solution makes it possible to achieve an increased accuracy of determining in which lane a vehicle is in. With the proposed solution more accurate position data may be obtained also for situations with many lanes present at a local area as well as with lanes provided over each other.
  • reference numeral 100 generally denotes a vehicle, or sometimes denoted as ego-vehicle, that may be autonomous, manually driven or a combination of these, i.e. semi-autonomous.
  • the vehicle 100 comprise any suitable parts for propulsion and controlling the vehicle such as an engine or electrical motor(s), control electronics for controlling and monitoring the engine/motor, propulsion, steering, gear box, breaks, lights, user interface dashboard, wheels and so on as understood by the skilled person.
  • the vehicle have electric control circuitry 210, as will be described in more detail in relation to Fig. 2 , and one or more sensors 101, 102 for providing data such as related to position, speed, direction of travel, temperature, road surface characteristics, and so on.
  • Position sensors may include sensors receiving signals from one or several satellite based positioning systems, e.g. from GPS satellites 104 via an antenna 102 and/or one or more cameras 101 providing images of the road and surrounding areas. These images may be processed to determine vehicle position relative road boundaries, other vehicles and so on.
  • the vehicle may also be provided with an IMU (Inertial Measurement Unit) (not shown) comprising accelerometers and gyroscopes for assisting in tracking the vehicles position and movements.
  • Other sensors that may be used are for instance Lidar (light detection and ranging) and radar for keeping track of distances and relative speed to other vehicles around the vehicle; these sensors have not been illustrated in Fig. 1 but may be housed within the vehicle or combined in sensor unit 101.
  • the antenna may be used for both communication with external networks and also for receiving GPS signals; however, separate antennas may be provided for external communication and GPS signals.
  • the vehicle may further be connected to external network(s) 150 via for instance a wireless link 103.
  • the same or some other wireless link may be used to communicate with other vehicles in the vicinity of the vehicle or with local infrastructure elements.
  • Cellular communication technologies may be used for long range communication such as to external networks and if the cellular communication technology used have low latency it may also be used for communication between vehicles, V2V, and/or vehicle to infrastructure, V2X. Examples of cellular radio technologies are GSM, GPRS, EDGE, LTE, 5G, 5G NR, and so on, also including future cellular solutions. However, in some solutions mid to short range communication technologies are used such as Wireless Local Area (LAN), e.g. IEEE 802.11 based solutions. ETSI is working on cellular standards for vehicle communication and for instance 5G is considered as a suitable solution due to the low latency and efficient handling of high bandwidths and communication channels.
  • LAN Wireless Local Area
  • a vehicle control circuitry/system 210 is used for controlling the vehicle and is illustrated in Fig. 2 .
  • Control circuitry may comprise at least one processor 201 with a number of modules 250, 260, and 270 for performing different operations, at least one computer readable memory or storage medium 202 for storing data and/or instruction sets for operating different functions in the processor.
  • processor 201 with a number of modules 250, 260, and 270 for performing different operations
  • computer readable memory or storage medium 202 for storing data and/or instruction sets for operating different functions in the processor.
  • one or more communication interfaces 205 and/or one or more antenna interfaces 207 may be provided and furthermore, also one or more sensor interfaces 206 may be provided for acquiring data from sensors within the vehicle.
  • the modules of the processor may be a creation module 250, an adjustment module 260 and a comparing module 270 each arranged to execute instruction sets for different steps of a process for determining the ego-lane of a vehicle.
  • the communication/antenna interface may also provide the possibility to acquire sensor data directly or via dedicated sensor control circuitry in the vehicle: for instance GPS data may be acquired through the antenna interface, some sensors in the vehicle may communicate with the control circuitry using a local network setup, such as CAN bus, I2C, Ethernet, optical fibres, and so on.
  • the communication interface may be arranged to communicate with other control functions of the vehicle and may thus be seen as control interface also; however, it separate control interface (not shown) may be provided.
  • Local communication within the vehicle may also be of a wireless type with protocols such as WiFi, LoRa, Zigbee, Bluetooth, or similar mid/short range technologies.
  • the processor 201 may for instance be a microprocessor, digital signal processor, graphical processing unit (GPU), embedded processor, field programmable gate array (FPGA), or ASIC (Application specific integrated circuit).
  • the storage unit 202 may be a volatile or non-volatile computer readable memory and arranged to store instructions or instruction sets for execution by the processor. Instruction sets are preferably stored in a non-volatile memory such as solid state (SSD), magnetic disk drive storage, optical storage such as CD, DVD, or Bluray, or persistent solid state memory technology such as flash memory or memory card.
  • SSD solid state
  • the storage unit may also comprise a combination of storage types.
  • the method may be realized in a computer program product and/or stored in a computer-readable storage medium.
  • the memory may be non-transitory or transitory computer-readable storage medium which stores one or more programs configured to be executed by one or more processors of an electronic device with or without a display apparatus and one or more input devices.
  • the computer program product may be delivered on a storage medium such as for instance SSD, magnetic storage, optical storage or delivered on a network connection as a signal with a suitable protocol, for instance Ethernet using Internet Protocol (IP) or wirelessly with suitable radio protocol such as cellular technologies or short or medium range local area network technologies.
  • IP Internet Protocol
  • Fig. 3 illustrates an exemplary lane situation where two roads 301, 302 are close to each other or overlap partially or entirely.
  • Each road in this case comprise two lanes each 310, 311, 312, 313 and the vehicle 100 is travelling on one lane 310.
  • the control circuitry 100 need to determine where the vehicle is travelling and on which lane it is at a specific time. Due to uncertainties in determination of location, speed, and yaw of the vehicle, and map accuracy there is only a finite accuracy of the necessary parameters to determine the location/pose of the vehicle in such a way as to safely control the vehicle when in autonomous or semi-autonomous driving mode.
  • Determination of the location of the vehicle has built in uncertainties, for instance due to noise in the GPS signal, accuracy of speed measurement, accuracy of yaw measurement, camera resolution, latency issues in processing, map accuracy, and so on.
  • parameters related to vehicle pose i.e. pose comprise a two dimensional position on a map and a direction of travel.
  • the two dimensions on the map are surface coordinates excluding the height information.
  • the location data may comprise height information as well which could be used in below analysis in determining where the vehicle is in relation to available lanes/roads.
  • Radar and Lidar signals may also be used for determining the position of the vehicle. For instance with the use of Radar and/or Lidar maps of static objects such as signs, barriers, houses and so on. Also known landmarks may be used to position the vehicle on a map; images of the surrounding area may be analysed to identify objects and landmarks which may be compared to known objects and landmarks.
  • Fig. 4 an example of uncertainty, U, 405, 406 of each lane 310, 311 position is plotted, in this case this may for instance be a Gaussian distribution, as seen in Fig. 4 , with centre on lane centre 402, 401 of the road 301 and then with decreasing likelihood towards the lane sides; however, other statistical distributions may be used to describe the uncertainties of the lane position. In a similar manner the different uncertainties of the parameters may be determined or estimated. It can be noted in Fig. 4 that the statistical distributions overlap 410 and thus there is some probability that the vehicle is in another lane as compared to the centre position of the distribution.
  • the lane uncertainty, U may depend on map uncertainties, lane size and so on.
  • the map is more reliable than in other areas due to having been more thoroughly investigated, e.g. using a plurality of vehicles to map out these areas; this may for instance be the case for high traffic areas close to cities or other densely populated areas.
  • Some maps are built upon feedback from vehicles travelling in the area.
  • the processor in the processing system is arranged to iteratively execute instruction sets stored in the memory for acquiring sensor data via the sensors interface and operating a method for assigning a vehicle to a lane by determining the most likely lane in which the vehicle is located, the so called ego-lane. This may be done by first creating 501 a momentary assignment-likelihood for each nearby lane within a pre-determined distance from an estimated pose of the vehicle, adjusting 502 the momentary likelihood for each lane with a prior, which prior is determined from a previous iteration, comparing 503 all the adjusted momentary likelihoods and determining the most probable lane.
  • the pre-determined distance may be set to for instance 5 meters, 10 meters, or 20 meters, but it should be understood that other distances may be used including shorter or longer distances.
  • Roads or road sections that have a logical connection with where the vehicle was in a previous iteration will be used in the creation of likelihood values.
  • the processor may be arranged to send 504 control parameters on the communication/control interface for controlling the vehicle in accordance with the determined geometry of the most probable lane. This process is iterated often enough to ensure safe determination of in which lane the vehicle is in, for instance with a frequency of 1 - 200 times per second depending on speed of vehicle and computational power.
  • the step of creating the momentary-likelihoods for the current iteration may comprise acquiring 601 the estimated pose, E, of the vehicle, i.e. the position on a map, with or without height information, and with direction of travel.
  • the estimated pose may be obtained by using the sensor data such as data from a global navigation satellite system and/or based on determination of fixed landmarks using other sensors.
  • the estimated pose may be expressed in either a global coordinate frame or a local map-based coordinate frame.
  • the method comprise determining 602 an uncertainty value, V, to the estimated pose of the vehicle.
  • the pose uncertainty may be expressed for instance as a Gaussian around a most likely pose.
  • the uncertainty V is depending for instance on sensors signals, map uncertainty, update frequency of sensor, update frequency of iteration of algorithms, and speed of vehicle.
  • V may be either a set static value or a dynamic value determined for each iteration or at some intervals.
  • E is projected 603 onto nearby lanes in a map, for instance by finding the closest point from E to nearby lanes.
  • the projection for each lane advantageously also comprise the lane direction.
  • Uncertainty of lane may be expressed as a Gaussian and the uncertainty of lane may be a static value or a dynamic value depending on the map source and other factors.
  • the step of creating the distribution, P may be performed by combining the two Gaussians for V and lane.
  • the method further comprises creating a momentary likelihood hypothesis for the vehicle being in each lane by evaluating 605 distribution P with pose E and creating a momentary likelihood hypothesis for the vehicle for being in each lane.
  • the evaluation may be performed by subtracting E from each projection, P, and storing the results in a list.
  • this momentary likelihood for each lane is adjusted with a prior value determined from a previous iteration, for example if it was determined that the vehicle was assigned to a lane in the last iteration then that prior value have a high value but if the lane was not occupied previously the prior will be low.
  • the system steps through all likelihoods from the last iteration and for each likelihood from the previous iteration (prior), the system determines all logical connections to the current prior in the step, i.e. the current likelihood in the list of likelihoods from the last iteration.
  • Each logical connection associated to a momentary likelihood are adjusted with the current prior likelihood if the connection is within a set distance to the pose of the vehicle and no prior has been assigned before to the logical connection. This set distance may be for instance 2 meter, 5 meters, or 10 meters. It should be understood that the set distance is not limited to the exemplified distances it may be larger or smaller. Adjusting all momentary likelihoods which were not adjusted above, with a minimum likelihood value. A connection is deemed logical if there is a connection on the map between lanes.
  • the most likely lane may be found and the vehicle may be assigned to that lane.
  • non-zero values are preferably normalized, for instance so that all the non-zero values add up to a reference value, such as 1, 100 %, or any other suitable value used in statistical analysis to determine the overall probability.
  • Another way of normalizing the likelihood values is to for each likelihood subtract the maximum likelihood in the created likelihoods, i.e. normalized likelihoods equals non-normalized likelihood subtracted by maximum value of non-normalized likelihoods if operating in a log scale.
  • This normalized probability value for each lane is set as prior for that lane in next iteration. For likelihoods below a threshold, the likelihood value is set to a minimum likelihood value. For possible lanes with no prior value a prior for that lane is set to a small non-zero value, for instance -12 using a log scale in the calculations. It should be understood that the normalization step may be done before the comparison step.
  • step 7 and 6 may be performed in different temporal order.
  • Sensor and/or determined data may be low and/or high pass filtered in any suitable way in order to reduce fluctuations and spurious data and provide a smoother data set for controlling the vehicle.
  • the analysis may be used to determine the suitable direction of travel of the vehicle by predicting and determining a suitable path forward knowing the current lane and destination. This may be used for determining a suitable route both on short and long term basis, i.e. operational and tactical respectively.
  • One such parameter to plan for is fuel/energy consumption.
  • a probabilistic ego-lane assignment may be calculated. Firstly, compute projections to all lanes in the vicinity of the ego-vehicle. Based on the uncertainty of the ego-vehicle pose and the road, evaluate the distribution and assign a likelihood to all lanes to which a projection may be made. Secondly, utilize the prior history of lanes and update the likelihood of the lanes which have a prior. All new lanes which can be reached from any of the prior lanes, given the logical connections of the map, update their probability with the most likely prior from which the lane can be reached. These two steps are then repeated iteratively continuously updating the position of the vehicle relative the roads/lanes.
  • the distribution for evaluating the projections is at least a 3-dimensional distribution, which incorporates the surface position (x, y) on a map and the heading (orientation/yaw), i.e. the direction of travel.
  • parts of the described solution may be implemented either in the vehicle, in a system located external the vehicle, or in a combination of internal and external the vehicle; for instance in a server in communication with the vehicle, a so called cloud solution.
  • sensor data may be sent to an external system and that system performs the steps to determine the most probable lane and send back information indicating the lane and other relevant parameters used in controlling the vehicle.

Description

    Technical field
  • The present invention relates to system, methods and arrangements for determining a position of a vehicle and in particular for a solution for determining in which lane the vehicle is in and assigning the vehicle to the lane.
  • Background
  • The last years the development of technology for autonomous vehicles has exploded and many different solutions are being developed, both within sensor technologies and control solutions for controlling the vehicle and determining objects in the surrounding area as well as the location of the vehicle relative roads, other vehicles, objects and so on to ensure a safe travel of the vehicle. Today, development is ongoing in both autonomous driving (AD) and advanced driver-assistance systems (ADAS). There exists a number of different technical areas within this field. One such area is how to position the vehicle with accuracy and consistency since this is an essential safety feature for the vehicle when moving within traffic. Of fundamental importance to autonomous driving is an accurate model of the roadway and its lanes as the vehicle moves.
  • Satellite based positioning systems (Global Navigation Satellite Systems, GNSS), like for instance Global Positioning system (GPS), Globalnaya Navigazionnaya Sputnikovaya Sistema (GLONASS), Galileo, Beidou, and other regional systems are often not accurate enough to rely on solely for determining a position of a moving vehicle.
  • These GNSS based solutions have even less accuracy in determining height/horizontal information.
  • When utilizing maps to navigate and estimate adjacent lane geometry the ability to estimate current ego lane from an estimated position plays an important role. For instance if two roads are placed on top of each other and one is turning left while the other is turning right the ability to estimate the lane geometry is directly correlated to the ego lane assignment. Further it has been found that utilizing height information from GNSS is unstable - especially when going under another road where the reception of satellite signals are reduced- and does not provide the accuracy needed to separate the roads from each other.
  • Therefore, current solutions utilize a number of different sensors to increase the reliability of map position such as image processing, Lidar, vehicle travelling parameters such as speed, and so on. However, even when the position and the instantaneous direction of the vehicle is known the forward travel of the vehicle can still be difficult to predict or estimate, for instance in above example with overlapping roads or when different road segments are located close to each other. One such as example is for instance when the vehicle is approaching a highway exit or similar branching of the road.
  • Chinese patent document CN107221195 discusses an anti-collision solution for vehicle lane prediction and lane level map wherein a lane level is determined using a unique link ID number for each lane and including vehicle travel history into the algorithm for determining the current lane of the vehicle.
  • US patent document US2016/0335507 discloses a method and system for vehicle localization. In more detail, the method involves identifying expected lane lines based on a lane level map and a position of the vehicle. Further, a set of observed lines are identified based on images captured by an imaging device of the vehicle and the position of the vehicle. Then, the expected lines are matched to the observed lines in order to determine a likelihood score based on the matching, and finally, a best vehicle pose is determined based on the likelihood score
  • Summary
  • It is an object to obviate at least some of the above disadvantages and provide improved node(s) and method(s) for assigning a vehicle to a lane in road.
  • This is provided in a number of embodiments, such as a method for assigning a vehicle to a lane in a road for a vehicle, wherein the method comprise obtaining an estimated pose, E, of the vehicle, obtaining an uncertainty value, V, to the estimated pose of the vehicle, projecting the pose, E, onto nearby lanes in a map, obtaining position uncertainty, U, of each lane, creating a distribution, P, by combining uncertainty V and U for each nearby lane, creating a current momentary likelihood hypothesis for the vehicle for being in each lane by evaluating distribution, P, with pose, E, for each projection, adjusting the current momentary likelihood for each lane with a prior determined earlier; and comparing all the adjusted momentary likelihoods and determining the lane with the highest likelihood.
  • In the method, the step of adjusting current momentary likelihoods may comprise, for each likelihood from a previous iteration: checking for logical connection associated to a current momentary likelihood and adjusting the current momentary likelihood with the value of the likelihood from the previous iteration if the connection is within a set distance from the pose, E, and if the current momentary likelihood has not been adjusted previously; and adjusting current momentary likelihoods, that was not adjusted in previous step, with a minimum value;
  • The set distance may be set to shorter than 10 meters, or more preferably shorter than 5 meters, or even more preferably 2 meters or shorter.
  • Nearby lanes may be located within 20 meters from the estimated pose, or more preferably within 10 meters from the estimated pose, or even more preferably within 5 meters or shorter from the estimated pose.
  • The method may further comprise a step of normalizing the current likelihoods. The method may further comprise a step of setting all likelihoods below a threshold to a fixed minimum likelihood value. The step of normalizing the current likelihoods may comprise determining the maximum value of the most probably current likelihood and subtracting this maximum value from all current likelihoods.
  • The maximum of the likelihood value for each lane is found in respective lane centre and each lanes probabilistic value drops to zero at some distance away from the centre of the respective lane.
  • The step of adjusting momentary likelihood may comprise checking logical connection between lane hypotheses in the map. The pose may comprise a position in map coordinates and a direction of travel. Acquiring estimated pose of the vehicle may comprise acquiring sensor data from at least one of a Global Navigation Satellite System, camera, vehicle parameters, and Lidar.
  • The method is advantageously implemented in autonomous vehicles or advanced driver-assistance systems vehicles.
  • Another embodiment is provided in a vehicle control system comprising at least one processor, at least one memory for storing instructions and data, at least one sensor interface, and at least one communication interface. The processor is arranged to execute instruction sets stored in the memory to acquire data via the sensor interface and execute a method of obtaining an estimated pose, E, of the vehicle, obtaining an uncertainty value, V, to the estimated pose of the vehicle, projecting the pose, E, onto nearby lanes in a map, obtaining position uncertainty, U, of each lane, creating a distribution, P, by combining uncertainty V and U for each nearby lane, creating a current momentary likelihood hypothesis for the vehicle for being in each lane by evaluating distribution, P, with pose, E, for each projection, adjusting the current momentary likelihood for each lane with a prior determined earlier; comparing all the adjusted momentary likelihoods and determining the lane with the highest likelihood, and sending control parameters on the communication interface for controlling the vehicle in accordance with the determined most probable lane.
  • In the vehicle control system, the step of adjusting current momentary likelihoods may comprise, for each likelihood from a previous iteration: checking for logical connection associated to a current momentary likelihood and adjusting the current momentary likelihood with the value of the likelihood from the previous iteration if the connection is within a set distance from the pose, E, and if the current momentary likelihood has not been adjusted previously; and adjusting current momentary likelihoods that was not adjusted in previous step, with a minimum value;
  • Yet another embodiment is provided, a vehicle comprising a vehicle control system, controlling driving functions of the vehicle, comprising at least one processor, at least one memory for storing instructions and data, at least one sensor input interface, at least one sensor for acquiring an estimation of vehicle position, wherein the vehicle control system is arranged to execute instructions in the processing unit for executing the above method.
  • Still another embodiment is provided, a computer program product arranged to execute the above method.
  • The proposed solution makes it possible to achieve an increased accuracy of determining in which lane a vehicle is in. With the proposed solution more accurate position data may be obtained also for situations with many lanes present at a local area as well as with lanes provided over each other.
  • This has the advantages of providing an efficient way of providing a high likelihood of the vehicle being in a lane and thus providing a possibility for the system to determining a suitable route forward knowing the destination. This may be done both on a short and long term perspective, i.e. an operational and strategic level.
  • Brief description of the drawings
  • In the following the invention will be described in a non-limiting way and in more detail with reference to exemplary embodiments illustrated in the enclosed drawings, in which:
  • Fig. 1
    is a schematic diagram illustrating an example vehicle;
    Fig. 2
    is a schematic block diagram illustrating exemplary control circuitry;
    Fig. 3
    is a schematic diagram illustrating an exemplary lane situation;
    Fig. 4
    is a schematic diagram illustrating an exemplary likelihood distribution;
    Fig. 5
    is a schematic block diagram illustrating an exemplary method; and
    Fig. 6
    is a schematic block diagram illustrating an exemplary method.
    Detailed description
  • In Fig. 1 reference numeral 100 generally denotes a vehicle, or sometimes denoted as ego-vehicle, that may be autonomous, manually driven or a combination of these, i.e. semi-autonomous. The vehicle 100 comprise any suitable parts for propulsion and controlling the vehicle such as an engine or electrical motor(s), control electronics for controlling and monitoring the engine/motor, propulsion, steering, gear box, breaks, lights, user interface dashboard, wheels and so on as understood by the skilled person. However, for autonomous or semi-autonomous driving the vehicle have electric control circuitry 210, as will be described in more detail in relation to Fig. 2, and one or more sensors 101, 102 for providing data such as related to position, speed, direction of travel, temperature, road surface characteristics, and so on. Position sensors may include sensors receiving signals from one or several satellite based positioning systems, e.g. from GPS satellites 104 via an antenna 102 and/or one or more cameras 101 providing images of the road and surrounding areas. These images may be processed to determine vehicle position relative road boundaries, other vehicles and so on. The vehicle may also be provided with an IMU (Inertial Measurement Unit) (not shown) comprising accelerometers and gyroscopes for assisting in tracking the vehicles position and movements. Other sensors that may be used are for instance Lidar (light detection and ranging) and radar for keeping track of distances and relative speed to other vehicles around the vehicle; these sensors have not been illustrated in Fig. 1 but may be housed within the vehicle or combined in sensor unit 101. The antenna may be used for both communication with external networks and also for receiving GPS signals; however, separate antennas may be provided for external communication and GPS signals.
  • The vehicle may further be connected to external network(s) 150 via for instance a wireless link 103. The same or some other wireless link may be used to communicate with other vehicles in the vicinity of the vehicle or with local infrastructure elements. Cellular communication technologies may be used for long range communication such as to external networks and if the cellular communication technology used have low latency it may also be used for communication between vehicles, V2V, and/or vehicle to infrastructure, V2X. Examples of cellular radio technologies are GSM, GPRS, EDGE, LTE, 5G, 5G NR, and so on, also including future cellular solutions. However, in some solutions mid to short range communication technologies are used such as Wireless Local Area (LAN), e.g. IEEE 802.11 based solutions. ETSI is working on cellular standards for vehicle communication and for instance 5G is considered as a suitable solution due to the low latency and efficient handling of high bandwidths and communication channels.
  • A vehicle control circuitry/system 210 is used for controlling the vehicle and is illustrated in Fig. 2. Control circuitry may comprise at least one processor 201 with a number of modules 250, 260, and 270 for performing different operations, at least one computer readable memory or storage medium 202 for storing data and/or instruction sets for operating different functions in the processor. Furthermore, optionally, depending on functionality provided in the control circuitry one or more communication interfaces 205 and/or one or more antenna interfaces 207 may be provided and furthermore, also one or more sensor interfaces 206 may be provided for acquiring data from sensors within the vehicle. The modules of the processor may be a creation module 250, an adjustment module 260 and a comparing module 270 each arranged to execute instruction sets for different steps of a process for determining the ego-lane of a vehicle. It should be appreciated that the communication/antenna interface may also provide the possibility to acquire sensor data directly or via dedicated sensor control circuitry in the vehicle: for instance GPS data may be acquired through the antenna interface, some sensors in the vehicle may communicate with the control circuitry using a local network setup, such as CAN bus, I2C, Ethernet, optical fibres, and so on. The communication interface may be arranged to communicate with other control functions of the vehicle and may thus be seen as control interface also; however, it separate control interface (not shown) may be provided. Local communication within the vehicle may also be of a wireless type with protocols such as WiFi, LoRa, Zigbee, Bluetooth, or similar mid/short range technologies.
  • The processor 201 may for instance be a microprocessor, digital signal processor, graphical processing unit (GPU), embedded processor, field programmable gate array (FPGA), or ASIC (Application specific integrated circuit). The storage unit 202 may be a volatile or non-volatile computer readable memory and arranged to store instructions or instruction sets for execution by the processor. Instruction sets are preferably stored in a non-volatile memory such as solid state (SSD), magnetic disk drive storage, optical storage such as CD, DVD, or Bluray, or persistent solid state memory technology such as flash memory or memory card. The storage unit may also comprise a combination of storage types. The method may be realized in a computer program product and/or stored in a computer-readable storage medium. The memory may be non-transitory or transitory computer-readable storage medium which stores one or more programs configured to be executed by one or more processors of an electronic device with or without a display apparatus and one or more input devices. The computer program product may be delivered on a storage medium such as for instance SSD, magnetic storage, optical storage or delivered on a network connection as a signal with a suitable protocol, for instance Ethernet using Internet Protocol (IP) or wirelessly with suitable radio protocol such as cellular technologies or short or medium range local area network technologies.
  • Fig. 3 illustrates an exemplary lane situation where two roads 301, 302 are close to each other or overlap partially or entirely. Each road in this case comprise two lanes each 310, 311, 312, 313 and the vehicle 100 is travelling on one lane 310. At some point when the vehicle is travelling along a lane 310 it will be close to other lanes for instance lane 312 and the control circuitry 100 need to determine where the vehicle is travelling and on which lane it is at a specific time. Due to uncertainties in determination of location, speed, and yaw of the vehicle, and map accuracy there is only a finite accuracy of the necessary parameters to determine the location/pose of the vehicle in such a way as to safely control the vehicle when in autonomous or semi-autonomous driving mode. Determination of the location of the vehicle has built in uncertainties, for instance due to noise in the GPS signal, accuracy of speed measurement, accuracy of yaw measurement, camera resolution, latency issues in processing, map accuracy, and so on. In one embodiment parameters related to vehicle pose is used, i.e. pose comprise a two dimensional position on a map and a direction of travel. The two dimensions on the map are surface coordinates excluding the height information. However, it should be appreciated that the location data may comprise height information as well which could be used in below analysis in determining where the vehicle is in relation to available lanes/roads.
  • Radar and Lidar signals may also be used for determining the position of the vehicle. For instance with the use of Radar and/or Lidar maps of static objects such as signs, barriers, houses and so on. Also known landmarks may be used to position the vehicle on a map; images of the surrounding area may be analysed to identify objects and landmarks which may be compared to known objects and landmarks.
  • In Fig. 4, an example of uncertainty, U, 405, 406 of each lane 310, 311 position is plotted, in this case this may for instance be a Gaussian distribution, as seen in Fig. 4, with centre on lane centre 402, 401 of the road 301 and then with decreasing likelihood towards the lane sides; however, other statistical distributions may be used to describe the uncertainties of the lane position. In a similar manner the different uncertainties of the parameters may be determined or estimated. It can be noted in Fig. 4 that the statistical distributions overlap 410 and thus there is some probability that the vehicle is in another lane as compared to the centre position of the distribution. The lane uncertainty, U, may depend on map uncertainties, lane size and so on. For instance in some areas the map is more reliable than in other areas due to having been more thoroughly investigated, e.g. using a plurality of vehicles to map out these areas; this may for instance be the case for high traffic areas close to cities or other densely populated areas. Some maps are built upon feedback from vehicles travelling in the area.
  • The processor in the processing system is arranged to iteratively execute instruction sets stored in the memory for acquiring sensor data via the sensors interface and operating a method for assigning a vehicle to a lane by determining the most likely lane in which the vehicle is located, the so called ego-lane. This may be done by first creating 501 a momentary assignment-likelihood for each nearby lane within a pre-determined distance from an estimated pose of the vehicle, adjusting 502 the momentary likelihood for each lane with a prior, which prior is determined from a previous iteration, comparing 503 all the adjusted momentary likelihoods and determining the most probable lane. The pre-determined distance may be set to for instance 5 meters, 10 meters, or 20 meters, but it should be understood that other distances may be used including shorter or longer distances. Roads or road sections that have a logical connection with where the vehicle was in a previous iteration will be used in the creation of likelihood values. Optionally, the processor may be arranged to send 504 control parameters on the communication/control interface for controlling the vehicle in accordance with the determined geometry of the most probable lane. This process is iterated often enough to ensure safe determination of in which lane the vehicle is in, for instance with a frequency of 1 - 200 times per second depending on speed of vehicle and computational power.
  • The step of creating the momentary-likelihoods for the current iteration may comprise acquiring 601 the estimated pose, E, of the vehicle, i.e. the position on a map, with or without height information, and with direction of travel. The estimated pose may be obtained by using the sensor data such as data from a global navigation satellite system and/or based on determination of fixed landmarks using other sensors. The estimated pose may be expressed in either a global coordinate frame or a local map-based coordinate frame.
  • Furthermore, the method comprise determining 602 an uncertainty value, V, to the estimated pose of the vehicle. The pose uncertainty may be expressed for instance as a Gaussian around a most likely pose. The uncertainty V is depending for instance on sensors signals, map uncertainty, update frequency of sensor, update frequency of iteration of algorithms, and speed of vehicle. V may be either a set static value or a dynamic value determined for each iteration or at some intervals.
  • E is projected 603 onto nearby lanes in a map, for instance by finding the closest point from E to nearby lanes. The projection for each lane advantageously also comprise the lane direction. Creating 604 a distribution, P, by combining uncertainty of V with uncertainty of lane, U, for each lane. Uncertainty of lane may be expressed as a Gaussian and the uncertainty of lane may be a static value or a dynamic value depending on the map source and other factors. The step of creating the distribution, P, may be performed by combining the two Gaussians for V and lane. Depending on in which coordinate system and/or if working in decimal/logarithmic systems this combining of lane and pose uncertainties may be done in different ways for example by simply adding the two gaussians to each other. The method further comprises creating a momentary likelihood hypothesis for the vehicle being in each lane by evaluating 605 distribution P with pose E and creating a momentary likelihood hypothesis for the vehicle for being in each lane. The evaluation may be performed by subtracting E from each projection, P, and storing the results in a list.
  • In general, this momentary likelihood for each lane is adjusted with a prior value determined from a previous iteration, for example if it was determined that the vehicle was assigned to a lane in the last iteration then that prior value have a high value but if the lane was not occupied previously the prior will be low.
  • For example, the system steps through all likelihoods from the last iteration and for each likelihood from the previous iteration (prior), the system determines all logical connections to the current prior in the step, i.e. the current likelihood in the list of likelihoods from the last iteration. Each logical connection associated to a momentary likelihood are adjusted with the current prior likelihood if the connection is within a set distance to the pose of the vehicle and no prior has been assigned before to the logical connection. This set distance may be for instance 2 meter, 5 meters, or 10 meters. It should be understood that the set distance is not limited to the exemplified distances it may be larger or smaller. Adjusting all momentary likelihoods which were not adjusted above, with a minimum likelihood value. A connection is deemed logical if there is a connection on the map between lanes.
  • All lanes with a logical connection or within a certain distance from the last occupied lane will get a non-zero value to indicate that there is some likelihood for the vehicle to be in that lane; however, for many of the lanes the likelihood is very small. By comparing all the adjusted momentary likelihoods it is possible to determine the most probable lane where the vehicle is, i.e. determining the lane with the highest adjusted likelihood or probability.
  • By comparing the adjusted likelihoods, the most likely lane may be found and the vehicle may be assigned to that lane.
  • Before using the likelihood values in next iteration all non-zero values are preferably normalized, for instance so that all the non-zero values add up to a reference value, such as 1, 100 %, or any other suitable value used in statistical analysis to determine the overall probability. Another way of normalizing the likelihood values is to for each likelihood subtract the maximum likelihood in the created likelihoods, i.e. normalized likelihoods equals non-normalized likelihood subtracted by maximum value of non-normalized likelihoods if operating in a log scale. This normalized probability value for each lane is set as prior for that lane in next iteration. For likelihoods below a threshold, the likelihood value is set to a minimum likelihood value. For possible lanes with no prior value a prior for that lane is set to a small non-zero value, for instance -12 using a log scale in the calculations. It should be understood that the normalization step may be done before the comparison step.
  • The method in one embodiment may be described in a short step-by-step example:
    1. 1) Find the closest point (projection) from "E" onto all nearby lanes, and form the set "projection_set" which contains all projections. Each projection also contains information of the lane direction (lane heading).
    2. 2) Create distribution "P" from uncertainty "V" for "E" together with lane uncertainty (for example by using Gaussian distributions).
    3. 3) Create momentary likelihood for each projection by evaluating ((projection - "E") for each projection in projection_set) in "P".
    4. 4) For each likelihood from the previous iteration (prior), starting with the highest and working downwards, do the following:
      • Find all logical connections from current prior "CP"
      • For each logical connection, including the prior, associated to a momentary likelihood: Adjust the momentary likelihoods with the "CP" likelihood if the connection is physically close to "E", and if no prior has been added.
    5. 5) Use a minimum likelihood value to adjust all momentary likelihoods which were not adjusted in the step 4).
    6. 6) Normalize all likelihoods (likelihoods - max of (likelihoods)) and set all likelihood values lower than a threshold to a minimum likelihood value.
    7. 7) Sort list with respect to likelihood and extract the most probable lane.
    8. 8) Go to 1)
  • However, it should be understood that the order of some of the steps may be interchanged, for example step 7 and 6 may be performed in different temporal order.
  • This approach is in line with Bayesian probability theory operating with probability distributions.
  • Sensor and/or determined data may be low and/or high pass filtered in any suitable way in order to reduce fluctuations and spurious data and provide a smoother data set for controlling the vehicle.
  • Also the analysis may be used to determine the suitable direction of travel of the vehicle by predicting and determining a suitable path forward knowing the current lane and destination. This may be used for determining a suitable route both on short and long term basis, i.e. operational and tactical respectively. One such parameter to plan for is fuel/energy consumption.
  • To summarize: given a map with the attributes: logical connections between lanes, a geometric description of the centre line, and a position of the ego-vehicle relative to the map, a probabilistic ego-lane assignment may be calculated. Firstly, compute projections to all lanes in the vicinity of the ego-vehicle. Based on the uncertainty of the ego-vehicle pose and the road, evaluate the distribution and assign a likelihood to all lanes to which a projection may be made. Secondly, utilize the prior history of lanes and update the likelihood of the lanes which have a prior. All new lanes which can be reached from any of the prior lanes, given the logical connections of the map, update their probability with the most likely prior from which the lane can be reached. These two steps are then repeated iteratively continuously updating the position of the vehicle relative the roads/lanes.
  • The distribution for evaluating the projections is at least a 3-dimensional distribution, which incorporates the surface position (x, y) on a map and the heading (orientation/yaw), i.e. the direction of travel.
  • With this solution the height information of the lanes is not needed to determine the most probably lane for which the vehicle is occupying reducing the uncertainty of the position and/or pose and increasing the safety.
  • It should be understood that parts of the described solution may be implemented either in the vehicle, in a system located external the vehicle, or in a combination of internal and external the vehicle; for instance in a server in communication with the vehicle, a so called cloud solution. For instance, sensor data may be sent to an external system and that system performs the steps to determine the most probable lane and send back information indicating the lane and other relevant parameters used in controlling the vehicle.
  • It should be noted that the word "comprising" does not exclude the presence of other elements or steps than those listed and the words "a" or "an" preceding an element do not exclude the presence of a plurality of such elements. It should further be noted that any reference signs do not limit the scope of the claims, that the invention may be at least in part implemented by means of both hardware and software, and that several "means" or "units" may be represented by the same item of hardware.
  • The above mentioned and described embodiments are only given as examples and should not be limiting to the present invention. Other solutions, uses, objectives, and functions within the scope of the invention as claimed in the below described patent embodiments should be apparent for the person skilled in the art.
  • Abbreviations
  • Wi-Fi
    Wireless Fidelity (registered trademark of Wi-Fi alliance) - based on IEEE 802.11 protocol
    LoRa
    Long Range radio (registered trademark of LoRa alliance)
    Zigbee
    Wireless communication protocol for loT and similar applications, handled by the Zigbee Alliance.
    GSM
    Global System for Mobile
    GPRS
    General Packet Radio Service
    EDGE
    Enhanced Data rates for Global Evolution
    LTE
    Long term evolution
    IP
    Internet Protocol
    Radar
    Radio detection and ranging
    Lidar
    Light detection and ranging

Claims (16)

  1. A computer-implemented method for assigning a vehicle (100) to a lane in a road (301), wherein the method comprise:
    - obtaining (601) an estimated pose, E, of the vehicle;
    - characterized in that the method further comprises:
    - obtaining (602) an uncertainty value, V, to the estimated pose of the vehicle;
    - projecting (603) the pose, E, onto nearby lanes in a map;
    - obtaining position uncertainty, U, of each lane;
    - creating a distribution, P, by combining (604) uncertainty V and U for each nearby lane;
    - creating a current momentary likelihood hypothesis for the vehicle for being in each lane by evaluating (605) distribution, P, with pose, E, for each projection;
    - adjusting (502) the current momentary likelihood for each lane with a prior determined earlier; and
    - comparing (503) all the adjusted momentary likelihoods and determining the lane with the highest likelihood.
  2. The computer-implemented method according to claim 1, wherein the step of adjusting current momentary likelihoods comprise, for each likelihood from a previous iteration:
    - checking for logical connection associated to a current momentary likelihood and adjusting the current momentary likelihood with the value of the likelihood from the previous iteration if the connection is within a set distance from the pose, E, and if the current momentary likelihood has not been adjusted previously; and
    - adjusting current momentary likelihoods, that was not adjusted in previous step, with a minimum value;
  3. The computer-implemented method according to claim 2, wherein the set distance is set to shorter than 10 meters, or more preferably shorter than 5 meters, or even more preferably 2 meters or shorter.
  4. The The computer-implemented method according to claim 1, wherein nearby lanes are located within 20 meters from the estimated pose, or more preferably within 10 meters from the estimated pose, or even more preferably within 5 meters or shorter from the estimated pose.
  5. The computer-implemented method according to claim 1 or 2, further comprising a step of normalizing the current likelihoods.
  6. The computer-implemented method according to claim 5, further comprising a step of setting all likelihoods below a threshold to a fixed minimum likelihood value.
  7. The computer-implemented method according to claim 5, wherein the step of normalizing the current likelihoods comprise determining the maximum value of the most probably current likelihood and subtracting this maximum value from all current likelihoods.
  8. The computer-implemented method according to any of the preceding claims, wherein a maximum of the likelihood value for each lane is found in respective lane centre (401, 402) and each lanes probabilistic value drops to zero at some distance away from the centre of the respective lane.
  9. The computer-implemented method according to claim 2, wherein the step of adjusting momentary likelihood comprise checking logical connection between lane hypotheses in the map.
  10. The computer-implemented flhe- method according to claim 1, wherein the pose comprise a position in map coordinates and a direction of travel.
  11. The computer-implemented flhe- method according to any of the preceding claims, wherein acquiring estimated pose of the vehicle comprise acquiring sensor (101, 102) data from at least one of a Global Navigation Satellite System, camera, vehicle parameters, and Lidar.
  12. The computer-implemented method according any preceding claims, wherein the vehicle is an autonomous vehicle or an advanced driver-assistance systems vehicle.
  13. A vehicle control system (210) comprising at least one processor (201), at least one memory (202) for storing instructions and data, at least one sensor interface (206), and at least one communication interface (205), wherein the processor is arranged to execute instruction sets stored in the memory, acquire data via the sensor interface and:
    - obtain (601) an estimated pose, E, of the vehicle;
    - characterized in that the processor is further arranged to:
    - obtain (602) an uncertainty value, V, to the estimated pose of the vehicle;
    - project (603) the pose, E, onto nearby lanes in a map;
    - obtain position uncertainty, U, of each lane;
    - create a distribution, P, by combining (604) uncertainty V and U for each nearby lane;
    - create a current momentary likelihood hypothesis for the vehicle for being in each lane by evaluating (605) distribution, P, with pose, E, for each projection;
    - adjust (502) the current momentary likelihood for each lane with a prior determined earlier; and
    compare (503) all the adjusted momentary likelihoods and determining the lane with the highest likelihood.
  14. The vehicle control system according to claim 13, wherein the step of adjusting current momentary likelihoods comprise, for each likelihood from a previous iteration:
    - checking for logical connection associated to a current momentary likelihood and adjusting the current momentary likelihood with the value of the likelihood from the previous iteration if the connection is within a set distance from the pose, E, and if the current momentary likelihood has not been adjusted previously; and
    - adjusting current momentary likelihoods that was not adjusted in previous step, with a minimum value.
  15. A vehicle (100) comprising a
    - a vehicle control system (210), controlling driving functions of the vehicle, comprising at least one processor (201), at least one memory (202) for storing instructions and data, at least one sensor input interface (206);
    - at least one sensor (101, 102) for acquiring an estimation of vehicle position;
    wherein the vehicle control system is arranged to execute instructions in the processing unit for executing the method according to claims 1 to 12.
  16. A computer program product comprising instructions which, when the program is executed by a computer, cause the computer to carry out the computer-implemented method of claim 1.
EP18200390.5A 2018-10-15 2018-10-15 A method for assigning ego vehicle to a lane Active EP3640679B1 (en)

Priority Applications (3)

Application Number Priority Date Filing Date Title
EP18200390.5A EP3640679B1 (en) 2018-10-15 2018-10-15 A method for assigning ego vehicle to a lane
US16/600,719 US11776392B2 (en) 2018-10-15 2019-10-14 Method for assigning ego vehicle to a lane
CN201910977906.1A CN111123908A (en) 2018-10-15 2019-10-15 Method for allocating autonomous vehicles to lanes

Applications Claiming Priority (1)

Application Number Priority Date Filing Date Title
EP18200390.5A EP3640679B1 (en) 2018-10-15 2018-10-15 A method for assigning ego vehicle to a lane

Publications (2)

Publication Number Publication Date
EP3640679A1 EP3640679A1 (en) 2020-04-22
EP3640679B1 true EP3640679B1 (en) 2023-06-07

Family

ID=63857812

Family Applications (1)

Application Number Title Priority Date Filing Date
EP18200390.5A Active EP3640679B1 (en) 2018-10-15 2018-10-15 A method for assigning ego vehicle to a lane

Country Status (3)

Country Link
US (1) US11776392B2 (en)
EP (1) EP3640679B1 (en)
CN (1) CN111123908A (en)

Families Citing this family (1)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
DE102022102201A1 (en) 2022-01-31 2023-08-03 Bayerische Motoren Werke Aktiengesellschaft Method for supporting a user of a vehicle during automated lateral guidance of the vehicle on a road with a junction, computing device and driver assistance system

Family Cites Families (22)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
JPH0278907A (en) * 1988-09-16 1990-03-19 Hitachi Ltd Navigation system using map data and location system for moving body
DE102005007802A1 (en) * 2005-02-21 2006-08-24 Robert Bosch Gmbh Method for object plausibility in driver assistance systems
JP4934167B2 (en) * 2009-06-18 2012-05-16 クラリオン株式会社 Position detection apparatus and position detection program
US8452535B2 (en) * 2010-12-13 2013-05-28 GM Global Technology Operations LLC Systems and methods for precise sub-lane vehicle positioning
US8457827B1 (en) * 2012-03-15 2013-06-04 Google Inc. Modifying behavior of autonomous vehicle based on predicted behavior of other vehicles
KR101491256B1 (en) * 2013-05-28 2015-02-06 현대자동차주식회사 Apparatus and Method for Recognizing for Traffic Lane using Wireless Communication
DE102014003343A1 (en) * 2014-03-07 2014-09-18 Daimler Ag Method for determining a lane change requirement of a system vehicle
US9483700B1 (en) * 2015-05-13 2016-11-01 Honda Motor Co., Ltd. System and method for lane vehicle localization with lane marking detection and likelihood scoring
US11125566B2 (en) * 2015-07-16 2021-09-21 Ford Global Technologies, Llc Method and apparatus for determining a vehicle ego-position
DE102015214338A1 (en) * 2015-07-29 2017-02-02 Volkswagen Aktiengesellschaft Determining an arrangement information for a vehicle
US10152882B2 (en) * 2015-11-30 2018-12-11 Nissan North America, Inc. Host vehicle operation using remote vehicle intention prediction
KR102456626B1 (en) * 2016-01-15 2022-10-18 현대자동차주식회사 Apparatus and method for traffic lane recognition in automatic steering control of vehilcles
WO2018015811A1 (en) * 2016-07-21 2018-01-25 Mobileye Vision Technologies Ltd. Crowdsourcing and distributing a sparse map, and lane measurements for autonomous vehicle navigation
EP3291197A1 (en) * 2016-08-30 2018-03-07 Volvo Car Corporation Method and system for establishing whether a road section is suitable for autonomous vehicle driving
KR101866075B1 (en) * 2016-10-20 2018-06-08 현대자동차주식회사 Apparatus and method for estmating lane
CN108303103B (en) * 2017-02-07 2020-02-07 腾讯科技(深圳)有限公司 Method and device for determining target lane
US11193782B2 (en) * 2017-03-27 2021-12-07 Mitsubishi Electric Corporation Vehicle position estimation apparatus
CN107221195B (en) 2017-05-26 2020-03-17 重庆长安汽车股份有限公司 Automobile lane prediction method and lane level map
JP2019016246A (en) * 2017-07-10 2019-01-31 株式会社Soken Travelling lane recognition device
DE102017123842A1 (en) * 2017-10-13 2019-04-18 Connaught Electronics Ltd. Lane detection method
US10824162B2 (en) * 2018-01-03 2020-11-03 Uatc, Llc Low quality pose lane associator
EP3759562B1 (en) * 2018-03-02 2022-12-28 DeepMap Inc. Camera based localization for autonomous vehicles

Also Published As

Publication number Publication date
US11776392B2 (en) 2023-10-03
EP3640679A1 (en) 2020-04-22
CN111123908A (en) 2020-05-08
US20200118426A1 (en) 2020-04-16

Similar Documents

Publication Publication Date Title
US10471955B2 (en) Stop sign and traffic light alert
CN107015559B (en) Probabilistic inference of target tracking using hash weighted integration and summation
CN106546977B (en) Vehicle radar sensing and localization
US10133947B2 (en) Object detection using location data and scale space representations of image data
CN107430819B (en) Vehicle information processing device and vehicle information processing method
US10520949B2 (en) Method and device for localizing a vehicle in its surroundings
KR101755944B1 (en) Autonomous driving method and system for determing position of car graft on gps, uwb and v2x
US20170344844A1 (en) Information processing apparatus and information processing method
US10782411B2 (en) Vehicle pose system
US20220157168A1 (en) V2X with 5G/6G Image Exchange and AI-Based Viewpoint Fusion
US20210190537A1 (en) Method and system for generating and updating digital maps
JP6903598B2 (en) Information processing equipment, information processing methods, information processing programs, and mobiles
CN114174137A (en) Source lateral offset of ADAS or AD features
CN109029418A (en) A method of vehicle is positioned in closed area
US20220205804A1 (en) Vehicle localisation
US20210323577A1 (en) Methods and systems for managing an automated driving system of a vehicle
EP3640679B1 (en) A method for assigning ego vehicle to a lane
US20230090338A1 (en) Method and system for evaluation and development of automated driving system features or functions
JP2023068009A (en) Map information creation method
CN115817466A (en) Collision risk assessment method and device
JP2023531826A (en) Method, apparatus and computer program for vehicle
JP2022160281A (en) Vehicle, server, system, method, storage medium and program
US20230031485A1 (en) Device and method for generating lane information
US20230202497A1 (en) Hypothesis inference for vehicles
JP7241582B2 (en) MOBILE POSITION DETECTION METHOD AND MOBILE POSITION DETECTION SYSTEM

Legal Events

Date Code Title Description
PUAI Public reference made under article 153(3) epc to a published international application that has entered the european phase

Free format text: ORIGINAL CODE: 0009012

STAA Information on the status of an ep patent application or granted ep patent

Free format text: STATUS: THE APPLICATION HAS BEEN PUBLISHED

AK Designated contracting states

Kind code of ref document: A1

Designated state(s): AL AT BE BG CH CY CZ DE DK EE ES FI FR GB GR HR HU IE IS IT LI LT LU LV MC MK MT NL NO PL PT RO RS SE SI SK SM TR

AX Request for extension of the european patent

Extension state: BA ME

STAA Information on the status of an ep patent application or granted ep patent

Free format text: STATUS: REQUEST FOR EXAMINATION WAS MADE

17P Request for examination filed

Effective date: 20201021

RBV Designated contracting states (corrected)

Designated state(s): AL AT BE BG CH CY CZ DE DK EE ES FI FR GB GR HR HU IE IS IT LI LT LU LV MC MK MT NL NO PL PT RO RS SE SI SK SM TR

RIC1 Information provided on ipc code assigned before grant

Ipc: G01C 21/16 20060101ALI20221014BHEP

Ipc: G08G 1/16 20060101ALI20221014BHEP

Ipc: G01C 21/36 20060101ALI20221014BHEP

Ipc: G01C 21/28 20060101ALI20221014BHEP

Ipc: B60W 30/12 20200101ALI20221014BHEP

Ipc: G08G 1/137 20060101ALI20221014BHEP

Ipc: G06V 20/56 20220101ALI20221014BHEP

Ipc: G01S 19/13 20100101AFI20221014BHEP

GRAP Despatch of communication of intention to grant a patent

Free format text: ORIGINAL CODE: EPIDOSNIGR1

STAA Information on the status of an ep patent application or granted ep patent

Free format text: STATUS: GRANT OF PATENT IS INTENDED

INTG Intention to grant announced

Effective date: 20230102

GRAS Grant fee paid

Free format text: ORIGINAL CODE: EPIDOSNIGR3

GRAA (expected) grant

Free format text: ORIGINAL CODE: 0009210

STAA Information on the status of an ep patent application or granted ep patent

Free format text: STATUS: THE PATENT HAS BEEN GRANTED

AK Designated contracting states

Kind code of ref document: B1

Designated state(s): AL AT BE BG CH CY CZ DE DK EE ES FI FR GB GR HR HU IE IS IT LI LT LU LV MC MK MT NL NO PL PT RO RS SE SI SK SM TR

REG Reference to a national code

Ref country code: GB

Ref legal event code: FG4D

REG Reference to a national code

Ref country code: CH

Ref legal event code: EP

Ref country code: AT

Ref legal event code: REF

Ref document number: 1576775

Country of ref document: AT

Kind code of ref document: T

Effective date: 20230615

REG Reference to a national code

Ref country code: DE

Ref legal event code: R096

Ref document number: 602018050767

Country of ref document: DE

P01 Opt-out of the competence of the unified patent court (upc) registered

Effective date: 20230525

REG Reference to a national code

Ref country code: LT

Ref legal event code: MG9D

REG Reference to a national code

Ref country code: NL

Ref legal event code: MP

Effective date: 20230607

PG25 Lapsed in a contracting state [announced via postgrant information from national office to epo]

Ref country code: SE

Free format text: LAPSE BECAUSE OF FAILURE TO SUBMIT A TRANSLATION OF THE DESCRIPTION OR TO PAY THE FEE WITHIN THE PRESCRIBED TIME-LIMIT

Effective date: 20230607

Ref country code: NO

Free format text: LAPSE BECAUSE OF FAILURE TO SUBMIT A TRANSLATION OF THE DESCRIPTION OR TO PAY THE FEE WITHIN THE PRESCRIBED TIME-LIMIT

Effective date: 20230907

Ref country code: ES

Free format text: LAPSE BECAUSE OF FAILURE TO SUBMIT A TRANSLATION OF THE DESCRIPTION OR TO PAY THE FEE WITHIN THE PRESCRIBED TIME-LIMIT

Effective date: 20230607

PGFP Annual fee paid to national office [announced via postgrant information from national office to epo]

Ref country code: GB

Payment date: 20230920

Year of fee payment: 6

REG Reference to a national code

Ref country code: AT

Ref legal event code: MK05

Ref document number: 1576775

Country of ref document: AT

Kind code of ref document: T

Effective date: 20230607

PG25 Lapsed in a contracting state [announced via postgrant information from national office to epo]

Ref country code: RS

Free format text: LAPSE BECAUSE OF FAILURE TO SUBMIT A TRANSLATION OF THE DESCRIPTION OR TO PAY THE FEE WITHIN THE PRESCRIBED TIME-LIMIT

Effective date: 20230607

Ref country code: NL

Free format text: LAPSE BECAUSE OF FAILURE TO SUBMIT A TRANSLATION OF THE DESCRIPTION OR TO PAY THE FEE WITHIN THE PRESCRIBED TIME-LIMIT

Effective date: 20230607

Ref country code: LV

Free format text: LAPSE BECAUSE OF FAILURE TO SUBMIT A TRANSLATION OF THE DESCRIPTION OR TO PAY THE FEE WITHIN THE PRESCRIBED TIME-LIMIT

Effective date: 20230607

Ref country code: LT

Free format text: LAPSE BECAUSE OF FAILURE TO SUBMIT A TRANSLATION OF THE DESCRIPTION OR TO PAY THE FEE WITHIN THE PRESCRIBED TIME-LIMIT

Effective date: 20230607

Ref country code: HR

Free format text: LAPSE BECAUSE OF FAILURE TO SUBMIT A TRANSLATION OF THE DESCRIPTION OR TO PAY THE FEE WITHIN THE PRESCRIBED TIME-LIMIT

Effective date: 20230607

Ref country code: GR

Free format text: LAPSE BECAUSE OF FAILURE TO SUBMIT A TRANSLATION OF THE DESCRIPTION OR TO PAY THE FEE WITHIN THE PRESCRIBED TIME-LIMIT

Effective date: 20230908

PGFP Annual fee paid to national office [announced via postgrant information from national office to epo]

Ref country code: FR

Payment date: 20230921

Year of fee payment: 6

PG25 Lapsed in a contracting state [announced via postgrant information from national office to epo]

Ref country code: FI

Free format text: LAPSE BECAUSE OF FAILURE TO SUBMIT A TRANSLATION OF THE DESCRIPTION OR TO PAY THE FEE WITHIN THE PRESCRIBED TIME-LIMIT

Effective date: 20230607

PG25 Lapsed in a contracting state [announced via postgrant information from national office to epo]

Ref country code: SK

Free format text: LAPSE BECAUSE OF FAILURE TO SUBMIT A TRANSLATION OF THE DESCRIPTION OR TO PAY THE FEE WITHIN THE PRESCRIBED TIME-LIMIT

Effective date: 20230607

PG25 Lapsed in a contracting state [announced via postgrant information from national office to epo]

Ref country code: IS

Free format text: LAPSE BECAUSE OF FAILURE TO SUBMIT A TRANSLATION OF THE DESCRIPTION OR TO PAY THE FEE WITHIN THE PRESCRIBED TIME-LIMIT

Effective date: 20231007

PG25 Lapsed in a contracting state [announced via postgrant information from national office to epo]

Ref country code: SM

Free format text: LAPSE BECAUSE OF FAILURE TO SUBMIT A TRANSLATION OF THE DESCRIPTION OR TO PAY THE FEE WITHIN THE PRESCRIBED TIME-LIMIT

Effective date: 20230607

Ref country code: SK

Free format text: LAPSE BECAUSE OF FAILURE TO SUBMIT A TRANSLATION OF THE DESCRIPTION OR TO PAY THE FEE WITHIN THE PRESCRIBED TIME-LIMIT

Effective date: 20230607

Ref country code: RO

Free format text: LAPSE BECAUSE OF FAILURE TO SUBMIT A TRANSLATION OF THE DESCRIPTION OR TO PAY THE FEE WITHIN THE PRESCRIBED TIME-LIMIT

Effective date: 20230607

Ref country code: PT

Free format text: LAPSE BECAUSE OF FAILURE TO SUBMIT A TRANSLATION OF THE DESCRIPTION OR TO PAY THE FEE WITHIN THE PRESCRIBED TIME-LIMIT

Effective date: 20231009

Ref country code: IS

Free format text: LAPSE BECAUSE OF FAILURE TO SUBMIT A TRANSLATION OF THE DESCRIPTION OR TO PAY THE FEE WITHIN THE PRESCRIBED TIME-LIMIT

Effective date: 20231007

Ref country code: EE

Free format text: LAPSE BECAUSE OF FAILURE TO SUBMIT A TRANSLATION OF THE DESCRIPTION OR TO PAY THE FEE WITHIN THE PRESCRIBED TIME-LIMIT

Effective date: 20230607

Ref country code: CZ

Free format text: LAPSE BECAUSE OF FAILURE TO SUBMIT A TRANSLATION OF THE DESCRIPTION OR TO PAY THE FEE WITHIN THE PRESCRIBED TIME-LIMIT

Effective date: 20230607

Ref country code: AT

Free format text: LAPSE BECAUSE OF FAILURE TO SUBMIT A TRANSLATION OF THE DESCRIPTION OR TO PAY THE FEE WITHIN THE PRESCRIBED TIME-LIMIT

Effective date: 20230607

PGFP Annual fee paid to national office [announced via postgrant information from national office to epo]

Ref country code: DE

Payment date: 20230919

Year of fee payment: 6

PG25 Lapsed in a contracting state [announced via postgrant information from national office to epo]

Ref country code: PL

Free format text: LAPSE BECAUSE OF FAILURE TO SUBMIT A TRANSLATION OF THE DESCRIPTION OR TO PAY THE FEE WITHIN THE PRESCRIBED TIME-LIMIT

Effective date: 20230607