EP3531069A1 - Kontextbasierte wegplanung für die vektornavigation in hexagonalen raumkarten - Google Patents

Kontextbasierte wegplanung für die vektornavigation in hexagonalen raumkarten Download PDF

Info

Publication number
EP3531069A1
EP3531069A1 EP18206380.0A EP18206380A EP3531069A1 EP 3531069 A1 EP3531069 A1 EP 3531069A1 EP 18206380 A EP18206380 A EP 18206380A EP 3531069 A1 EP3531069 A1 EP 3531069A1
Authority
EP
European Patent Office
Prior art keywords
objects
hexagonal grid
contexts
identified
hexagonal
Prior art date
Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
Granted
Application number
EP18206380.0A
Other languages
English (en)
French (fr)
Other versions
EP3531069B1 (de
Inventor
Sounak Dey
Arijit Mukherjee
Aritra SARKAR
Current Assignee (The listed assignees may be inaccurate. Google has not performed a legal analysis and makes no representation or warranty as to the accuracy of the list.)
Tata Consultancy Services Ltd
Original Assignee
Tata Consultancy Services Ltd
Priority date (The priority date is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the date listed.)
Filing date
Publication date
Application filed by Tata Consultancy Services Ltd filed Critical Tata Consultancy Services Ltd
Publication of EP3531069A1 publication Critical patent/EP3531069A1/de
Application granted granted Critical
Publication of EP3531069B1 publication Critical patent/EP3531069B1/de
Active legal-status Critical Current
Anticipated expiration legal-status Critical

Links

Images

Classifications

    • G—PHYSICS
    • G01—MEASURING; TESTING
    • G01C—MEASURING DISTANCES, LEVELS OR BEARINGS; SURVEYING; NAVIGATION; GYROSCOPIC INSTRUMENTS; PHOTOGRAMMETRY OR VIDEOGRAMMETRY
    • G01C21/00—Navigation; Navigational instruments not provided for in groups G01C1/00 - G01C19/00
    • G01C21/20—Instruments for performing navigational calculations
    • G01C21/206—Instruments for performing navigational calculations specially adapted for indoor navigation
    • G—PHYSICS
    • G05—CONTROLLING; REGULATING
    • G05D—SYSTEMS FOR CONTROLLING OR REGULATING NON-ELECTRIC VARIABLES
    • G05D1/00—Control of position, course, altitude or attitude of land, water, air or space vehicles, e.g. using automatic pilots
    • G05D1/02—Control of position or course in two dimensions
    • G05D1/021—Control of position or course in two dimensions specially adapted to land vehicles
    • G05D1/0212—Control of position or course in two dimensions specially adapted to land vehicles with means for defining a desired trajectory
    • G05D1/0221—Control of position or course in two dimensions specially adapted to land vehicles with means for defining a desired trajectory involving a learning process
    • G—PHYSICS
    • G01—MEASURING; TESTING
    • G01C—MEASURING DISTANCES, LEVELS OR BEARINGS; SURVEYING; NAVIGATION; GYROSCOPIC INSTRUMENTS; PHOTOGRAMMETRY OR VIDEOGRAMMETRY
    • G01C21/00—Navigation; Navigational instruments not provided for in groups G01C1/00 - G01C19/00
    • G01C21/005—Navigation; Navigational instruments not provided for in groups G01C1/00 - G01C19/00 with correlation of navigation data from several sources, e.g. map or contour matching
    • G—PHYSICS
    • G01—MEASURING; TESTING
    • G01C—MEASURING DISTANCES, LEVELS OR BEARINGS; SURVEYING; NAVIGATION; GYROSCOPIC INSTRUMENTS; PHOTOGRAMMETRY OR VIDEOGRAMMETRY
    • G01C21/00—Navigation; Navigational instruments not provided for in groups G01C1/00 - G01C19/00
    • G01C21/20—Instruments for performing navigational calculations
    • G—PHYSICS
    • G01—MEASURING; TESTING
    • G01C—MEASURING DISTANCES, LEVELS OR BEARINGS; SURVEYING; NAVIGATION; GYROSCOPIC INSTRUMENTS; PHOTOGRAMMETRY OR VIDEOGRAMMETRY
    • G01C21/00—Navigation; Navigational instruments not provided for in groups G01C1/00 - G01C19/00
    • G01C21/26—Navigation; Navigational instruments not provided for in groups G01C1/00 - G01C19/00 specially adapted for navigation in a road network
    • G01C21/34—Route searching; Route guidance
    • G01C21/3407—Route searching; Route guidance specially adapted for specific applications
    • G01C21/3415—Dynamic re-routing, e.g. recalculating the route when the user deviates from calculated route or after detecting real-time traffic data or accidents
    • G—PHYSICS
    • G01—MEASURING; TESTING
    • G01C—MEASURING DISTANCES, LEVELS OR BEARINGS; SURVEYING; NAVIGATION; GYROSCOPIC INSTRUMENTS; PHOTOGRAMMETRY OR VIDEOGRAMMETRY
    • G01C21/00—Navigation; Navigational instruments not provided for in groups G01C1/00 - G01C19/00
    • G01C21/38—Electronic maps specially adapted for navigation; Updating thereof
    • G01C21/3863—Structures of map data
    • G01C21/387—Organisation of map data, e.g. version management or database structures
    • G01C21/3881—Tile-based structures
    • G—PHYSICS
    • G05—CONTROLLING; REGULATING
    • G05D—SYSTEMS FOR CONTROLLING OR REGULATING NON-ELECTRIC VARIABLES
    • G05D1/00—Control of position, course, altitude or attitude of land, water, air or space vehicles, e.g. using automatic pilots
    • G05D1/0088—Control of position, course, altitude or attitude of land, water, air or space vehicles, e.g. using automatic pilots characterized by the autonomous decision making process, e.g. artificial intelligence, predefined behaviours

Definitions

  • the disclosure herein generally relates to path planning and navigational techniques, and, more particularly, to context based path planning for vector navigation in hexagonal spatial maps.
  • Robots (in general machines) today are being made artificially intelligent (AI) by using techniques like machine learning, deep learning, and the like.
  • AI artificially intelligent
  • main drawbacks of these genres of techniques are: firstly, they need huge volume of training data to be fed into the system so that the system learns a model about the data and its features; then this trained system can identify or take decision for new incoming data of same category.
  • some features of the training data changes then the system need to be re-trained in order to work with new types of data. Since the volume of the required training data is huge and every time the system needs to be re-trained with new type of similar data, this kind of artificial intelligence technologies are not only very complex and time consuming but they also consume a lot of energy.
  • Embodiments of the present disclosure present technological improvements as solutions to one or more of the above-mentioned technical problems recognized by the inventors in conventional systems. For example, in one aspect, there is provided a processor implemented method for context based path planning for vector navigation in hexagonal spatial maps.
  • the method comprises: obtaining and representing a two-dimensional (2D) environment into a hexagonal grid map, wherein the hexagonal grid map comprises a plurality of hexagonal grid cells; identifying one or more objects based on a comparison of Red Green Blue (RGB) value associated with a plurality of two or more contiguous cells from the plurality of hexagonal grid cells, wherein the one or more objects are identified based on number of hexagonal centers and distance of center of each hexagonal grid cell from center of one or more hexagonal grid cells from the plurality of hexagonal grid cells; identifying one or more candidate contexts based on a tuple created for the identified one or more objects; iteratively performing a comparison of the hexagonal grid map with one or more pre-stored hexagonal grid maps, wherein the hexagonal grid map is transformed to one or more variations (e.g., using rotation technique, affine transformation technique, topological transformation technique, and the like) and iteratively compared with the one or more pre-stored hexagonal grid maps obtained from a map database to
  • the method further comprises generating an object database for each of the one or more objects based on an annotation, side of bounding square, a template shape, and a variance in the RGB value; and generating a context database for each of the one or more probable contexts with the identified one or more objects in a tuple.
  • the one or more objects are identified as one or more predefined objects when the RGB value of the one or more objects is equal to a predefined threshold. In another embodiment, the one or more objects are identified as new objects when the RGB value of the one or more objects is greater than or less than the predefined threshold. In an embodiment, size and shape of the one or more objects are determined based on the distance (e.g., Euclidean distance between two hexagonal centers).
  • the method further comprises identifying one or more common objects that indicate an intersection of two or more contexts; and generating a path based on one or more navigable objects identified from the one or more common objects from the intersection of the two or more contexts.
  • the one or more contexts and the one or more objects may be identified using information obtained from one or more sensors.
  • the method may further comprise training a robot using the dynamically planned path, the one or more contexts and the one or more objects identified for subsequent path planning and vector navigation; and generating a database comprising information pertaining to the subsequent path planning and vector navigation.
  • the method may further comprise performing a comparison of the hexagonal grid map with the one or more candidate contexts to obtain a maximum SSQ; and identifying an environment as a new environment or a pre-defined environment based on the comparison of the maximum SSQ with a pre-defined threshold.
  • a system for context based path planning for vector navigation in hexagonal spatial maps comprises a memory storing instructions; one or more communication interfaces; and one or more hardware processors coupled to the memory via the one or more communication interfaces, wherein the one or more hardware processors are configured by the instructions to: obtain and represent a two-dimensional (2D) environment into a hexagonal grid map, wherein the hexagonal grid map comprises a plurality of hexagonal grid cells; identify one or more objects based on a comparison of Red Green Blue (RGB) value associated with a plurality of two or more contiguous cells from the plurality of hexagonal grid cells, wherein the one or more objects are identified based on number of hexagonal centers and distance of center of each hexagonal grid cell from center of one or more hexagonal grid cells from the plurality of hexagonal grid cells; identifying one or more candidate contexts based on a tuple created for the identified one or more objects; iteratively perform a comparison of the hexagonal grid map with one or more pre-stored
  • RGB Red Green Blue
  • the hardware processors are further configured by the instructions to generate an object database for each of the one or more objects based on an annotation, side of bounding square, a template shape, and a variance in the RGB value; and generate a context database for each of the one or more probable contexts with the identified one or more objects in a tuple.
  • the one or more objects are identified as one or more predefined objects when the RGB value of the one or more objects is equal to a predefined threshold. In another embodiment, the one or more objects are identified as new objects when the RGB value of the one or more objects is greater than or less than the predefined threshold. In an embodiment, size and shape of the one or more objects are determined based on the distance (e.g., Euclidean distance between two hexagonal centers).
  • the hardware processors are further configured by the instructions to identify one or more common objects that indicate an intersection of two or more contexts; and generate a path based on one or more navigable objects identified from the one or more common objects from the intersection of the two or more contexts.
  • the one or more contexts and the one or more objects may be identified using information obtained from one or more sensors.
  • the hardware processors are further configured by the instructions to train a robot using the dynamically planned path, the one or more contexts and the one or more objects identified for subsequent path planning and vector navigation; and generate a database comprising information pertaining to the subsequent path planning and vector navigation.
  • the method may further comprise performing a comparison of the hexagonal grid map with the one or more candidate contexts to obtain a maximum SSQ; and identifying an environment as a new environment or a pre-defined environment based on the comparison of the maximum SSQ with a pre-defined threshold.
  • one or more non-transitory machine readable information storage mediums comprising one or more instructions.
  • the one or more instructions which when executed by one or more hardware processors causes to perform a method comprising: obtaining and representing a two-dimensional (2D) environment into a hexagonal grid map, wherein the hexagonal grid map comprises a plurality of hexagonal grid cells; identifying one or more objects based on a comparison of Red Green Blue (RGB) value associated with a plurality of two or more contiguous cells from the plurality of hexagonal grid cells, wherein the one or more objects are identified based on number of hexagonal centers and distance of center of each hexagonal grid cell from center of one or more hexagonal grid cells from the plurality of hexagonal grid cells; identifying one or more candidate contexts based on a tuple created for the identified one or more objects; iteratively performing a comparison of the hexagonal grid map with one or more pre-stored hexagonal grid maps, wherein the hexagonal grid map is transformed to one or more variations (e.
  • RGB Red Green
  • the instructions further cause generating an object database for each of the one or more objects based on an annotation, side of bounding square, a template shape, and a variance in the RGB value; and generating a context database for each of the one or more probable contexts with the identified one or more objects in a tuple.
  • the one or more objects are identified as one or more predefined objects when the RGB value of the one or more objects is equal to a predefined threshold. In another embodiment, the one or more objects are identified as new objects when the RGB value of the one or more objects is greater than or less than the predefined threshold. In an embodiment, size and shape of the one or more objects are determined based on the distance (e.g., Euclidean distance between two hexagonal centers).
  • the instructions further cause identifying one or more common objects that indicate an intersection of two or more contexts; and generating a path based on one or more navigable objects identified from the one or more common objects from the intersection of the two or more contexts.
  • the one or more contexts and the one or more objects may be identified using information obtained from one or more sensors.
  • the instructions further cause training a robot using the dynamically planned path, the one or more contexts and the one or more objects identified for subsequent path planning and vector navigation; and generating a database comprising information pertaining to the subsequent path planning and vector navigation.
  • the instructions further cause performing a comparison of the hexagonal grid map with the one or more candidate contexts to obtain a maximum SSQ; and identifying an environment as a new environment or a pre-defined environment based on the comparison of the maximum SSQ with a pre-defined threshold.
  • robots are being made artificially intelligent by using techniques for example, machine learning, deep learning etc., for navigation applications which require huge volume of training data to be able to learn and adopt the data and its features.
  • these type of techniques have several challenges in terms of volume of training data required, and system requiring to undergo re-training in order to work with any new data types which leads to complexity, consumption of time and energy and more computational power.
  • Mammalian brains for example, human brains, can be trained on very minimal set of data and can re-train itself with again very minimal set of data. With these, it can identify and handle different contexts and semantics of same data. And all these are done very quickly and by consuming much less energy than conventional AI systems.
  • Brain stores spatial and temporal information differently but handles them seamlessly by quickly switching contexts.
  • embodiments thereof focus on spatial information representation like brain and have shown that how efficiently that can be used for intelligent path planning for robotic navigation. Following are the key challenge areas:
  • Robots need to do a lot of real time path planning for easy navigation. This is a very time consuming and complex task for them. This becomes more complex in case of dynamic environments. Moreover, all the path planning algorithms use square grids to represent the environment and plan the path accordingly. On the other hand, brain represents environment by a superposition of multiple hexagonal grid tessellation.
  • Embodiments of the present disclosure provide systems and methods for creating and using spatial context of environments for robotic path planning in different environments. Brain does this very efficiently for path planning. By repeating spatial information or by finding spatial similarities (within a threshold limit) can create a spatial context and that again can be used for easy navigation between spatial environments with different context.
  • embodiments implement utilization of brain techniques for representing environmental maps and then planning path of navigation for a robot in that map.
  • the environment is converted to or represented in the form of a hexagonal grid map.
  • the embodiments of the present disclosure provides systems and methods to create object database and context database for robots.
  • the embodiments of the present disclosure further enable identifying spatial similarity between two (or more) such maps and introduce a quantitative measure for that spatial similarity namely Spatial Similarity Quotient (SSQ). By repeating more spatial information the present disclosure proposes to create contexts out of that information and then use them for seamless navigation in contextually different environments.
  • SSQ Spatial Similarity Quotient
  • maps are usually represented in square grids or in terms of occupancy grids but this way they are not brain-like.
  • the present disclosure focusses on hexagonal grid structure map to be in line with brain.
  • Path planning algorithms usually do not work for hexagonal grids.
  • the present disclosure also adapts to existing algorithms to work in hexagonal grids.
  • traditional path planning and navigation systems do not focus on spatial similarity of environments.
  • the present disclosure focuses on path planning and navigation systems that utilizes spatial similarity of environments to measure spatial similarity between two environments. Based on this similarity, the system calculates probable spatial distribution of dynamic objects for each environment.
  • the present disclosure also implemented semantic context stitching for longer inter-context path planning.
  • FIGS. 1 through 9 where similar reference characters denote corresponding features consistently throughout the figures, there are shown preferred embodiments and these embodiments are described in the context of the following exemplary system and/or method.
  • FIG. 1 illustrates an exemplary block diagram of a system 100 for context based path planning for vector navigation in hexagonal spatial maps according to an embodiment of the present disclosure.
  • the system 100 includes one or more processors 104, communication interface device(s) or input/output (I/O) interface(s) 106, and one or more data storage devices or memory 102 operatively coupled to the one or more processors 104.
  • the one or more processors 104 may be one or more software processing modules and/or hardware processors.
  • the hardware processors can be implemented as one or more microprocessors, microcomputers, microcontrollers, digital signal processors, central processing units, state machines, logic circuitries, and/or any devices that manipulate signals based on operational instructions.
  • the processor(s) is configured to fetch and execute computer-readable instructions stored in the memory.
  • the device 100 can be implemented in a variety of computing systems, such as laptop computers, notebooks, hand-held devices, workstations, mainframe computers, servers, a network cloud and the like.
  • the I/O interface device(s) 106 can include a variety of software and hardware interfaces, for example, a web interface, a graphical user interface, and the like and can facilitate multiple communications within a wide variety of networks N/W and protocol types, including wired networks, for example, LAN, cable, etc., and wireless networks, such as WLAN, cellular, or satellite.
  • the I/O interface device(s) can include one or more ports for connecting a number of devices to one another or to another server.
  • the memory 102 may include any computer-readable medium known in the art including, for example, volatile memory, such as static random access memory (SRAM) and dynamic random access memory (DRAM), and/or non-volatile memory, such as read only memory (ROM), erasable programmable ROM, flash memories, hard disks, optical disks, and magnetic tapes.
  • volatile memory such as static random access memory (SRAM) and dynamic random access memory (DRAM)
  • non-volatile memory such as read only memory (ROM), erasable programmable ROM, flash memories, hard disks, optical disks, and magnetic tapes.
  • ROM read only memory
  • the memory 102 may store training data that is learnt by the system 100, which gets utilized by the one or more hardware processors 104 (or by the system 100) to enable learning of the data patterns, evaluate the learning pattern, and train the system 100 using the training data to perform the methodology described herein.
  • FIG. 2 illustrates an exemplary method for context based path planning for vector navigation in hexagonal spatial maps using the system 100 of FIG. 1 according to an embodiment of the present.
  • the system(s) 100 comprises one or more data storage devices or the memory 102 operatively coupled to the one or more hardware processors 104 and is configured to store instructions for execution of steps of the method by the one or more processors 104.
  • the steps of the method of the present disclosure will now be explained with reference to the components of the system 100 as depicted in FIG. 1 , and the flow diagram of FIG. 2 .
  • FIG. 3A depicts an exemplary view of a hexagonal representation of map in brain in accordance with an example embodiment as authored by G. Buzsáki and E. I. Moser, literature titled "Memory, navigation and theta rhythm in the hippocampal-entorhinal system," Nature neuroscience, vol. 16, no. 2, pp. 130-138, 2013 .
  • Path planning for a robot is a compute intensive task. For a dynamic environment this is more cumbersome where position and orientation of obstacles changes often.
  • mammal's (particularly, human brain) brain does the same task very quickly and efficiently. It does not re-plan blindly every time it faces a new environment. Instead, it tries to find out spatial similarity between environments and try to understand the similarity of context and then intelligently re-plan for the affected part of the previously planned path or use previously planned path as is either by doing some transformational geometric operations.
  • the one or more hardware processors 104 obtain a two-dimension (2D) environment and represent the two-dimensional (2D) environment into a hexagonal grid map, wherein the hexagonal grid map comprises a plurality of hexagonal grid cells.
  • hexagons are the only regular polygons that can tessellate a 2-Dimensional space. Unlike triangles and squares, hexagons have few advantages, for example hexagons have two more degrees of freedom for movement and hexagons occupy the space with more efficiency than other two polygons (e.g., triangles and squares) of equal area.
  • neural map of 2-dimensional space is represented via regular hexagonal tessellation in brain.
  • the first hurdle is to create an efficient representation for hexagons. A minimum of two parameters are required to represent a 2-D space, for example, the X-Y co-ordinate in Cartesian coordinate system or Rad-Ang in Polar coordinate system.
  • the Cartesian coordinates are efficient as it is based on an orthonormal basis vectors.
  • overlaying hexagonal grids on a Cartesian grid cells give rise to aliasing problems.
  • the data structure can be array based which requires translating to hexagonal grid before display; or, it can be a list of hexagonal grid centers with all calculations performed in a 60 degree rotated basis vector from the fundamental Y-axis (or X-axis), which are translated to fundamental orthonormal X-Y coordinates before rendering on screen.
  • FIG. 3B depicts creation of dual of contiguous hexagonal centers used as basic building block for creating hexagonal tessellation of an environment in accordance with an example embodiment of the present disclosure.
  • the regular hexagon grid cells have a diameter (denoted by d), which is the scaling factor or the resolution (vertex to vertex pixel distance) of the map.
  • the 6 Moore/von-Neumann [refer] neighbour cells of (a,b) are ⁇ (a+1,b), (a,b+1), (a-1,b+1), (a-1,b), (a,b-1), (a+1,b-1) ⁇ .
  • the Moore/von-Neumann neighbor cells technique is referenced from literature titled " Introduction to cellular automata," 2005 authored by J. Schiff . Below are description for Moore neighbours and Von-Neumann neighbours:
  • the vertex-sharing corner cells has a distance ratio of 2 : 1 with respect to the side-sharing cells.
  • Von-Neumann neighbours For square grid, only the four side-sharing cells are considered as Von-Neumann neighbours of a given cell. The center of these cells are all at an equal distance from the center of the current cell.
  • FIG. 4 depicts a 2D environment in accordance with an embodiment of the present disclosure.
  • FIG. 5A-5C depict representation of the 2D environment into a hexagonal grid map at one or more zoom levels based on different center to center distances between hexagonal grid cells in accordance with an example embodiment of the present disclosure.
  • the one or more hardware processors 104 identify one or more objects based on a comparison of Red Green Blue (RGB) value associated with a plurality of two or more contiguous cells from the plurality of hexagonal grid cells.
  • RGB Red Green Blue
  • the one or more objects are identified based on number of hexagonal centers and distance of each hexagonal center from one or more hexagonal centers. In an embodiment, size and shape of the one or more objects are determined based on the distance. In an example embodiment, the distance herein refers at least one of Euclidean or Manhattan distance between two hexagonal centers.
  • the average RGB value for each hexagon grid cell are stored, with the hexagon center in a-b coordinate. Thus each hexagonal center may be associated with average RGB values. Contiguous hex cells having similar RGB values may represent an object because it is be assumed assume the colour and shape of the object may not change much.
  • the system 100 like a new learner, receives image(s) of environment(s) and then finds out what are different objects (as per RGB scheme above) in that map and annotates them with type of enumeration.
  • the image will also provide an idea about the size, shape of the object in hexagon domain in terms of accumulated distance of hexagonal centers and in terms of number of hexagonal centers.
  • a tuple e.g., 9 tuple
  • a tuple may be created as illustrated by way of example below: ⁇ Annotation, Side of bounding square, Template shape, color (r, g, b, variance in r, variance in g, variance in b)
  • shape can be predefined enumerations depending upon the scope of the environment.
  • the last item in the 9-tuple is color which contains the RGB values of the object in the image/environment.
  • the color experience may change and thus corresponding variance in value of r, g and b may get updated and stored.
  • the value of r, g, b goes beyond a threshold then it may construe to a new color or new object.
  • the one or more objects are identified as one or more predefined objects when the RGB value of the one or more objects is equal to a predefined threshold.
  • the one or more objects are identified as new objects when the RGB value of the one or more objects is greater than or less than the predefined threshold.
  • one such 9-tuple can be: ⁇ Obj_type_1, 20 pi X 35 pi, Round, Grey (127, 127, 127, 10, 14, 18)>
  • Such a tuple (e.g., 9-tuple) are stored in a database (e.g., an object database).
  • a database e.g., an object database
  • the pre-defined threshold for variance values here is taken as 20 and that is why all the variance values are within that tolerance limit.
  • the one or more hardware processors 104 Upon identifying the one or more objects, at step 206, the one or more hardware processors 104 identify one or more candidate contexts based on the tuple (e.g., 9-tuple) created for the identified one or more objects. In an embodiment of the present disclosure, at step 208, the one or more hardware processors 104 iteratively perform a comparison of the hexagonal grid map with one or more pre-stored hexagonal grid maps. In an embodiment, more specifically, the represented hexagonal grid map is transformed to one or more variations (e.g., rotated at one or more angles) and iteratively compared with the one or more pre-stored hexagonal grid maps (obtained from a map database) to determine a Spatial Similarity Quotient (SSQ).
  • SSQ Spatial Similarity Quotient
  • FIGS. 1 through 5C depicts a representation of the hexagonal grid map transformed to one or more variations for comparison with the one or more pre-stored hexagonal grid maps (also known as contexts) to determine the Spatial Similarity Quotient (SSQ) in accordance with an example embodiment of the present disclosure.
  • the hexagonal grid map is transformed to one or more variations using at least one of a rotation technique, an affine transformation technique, a topological transformation technique, and the like. More specifically, in an example embodiment, as depicted in FIG. 6 , the hexagonal grid map is rotated at one or more angles for comparison with the one or more pre-stored hexagonal grid maps.
  • the one or more transformed variations of the hexagonal grid map is compared with the one or more pre-stored hexagonal grid maps for determination or computation of SSQ.
  • the SSQ is also referred as 'quantitative measure of similarity', wherein the Spatial Similarity Quotient is indicative of degree of overlap of the hexagonal grid map with at least one of the one or more pre-stored hexagonal grid maps.
  • FIG. 7 with reference to, FIGS. 1 through 6 , illustrating a graphical representation depicting a comparison between original and rotated version of the hexagonal grid map and a maximum similarity found at 200 degree in accordance with an example embodiment of the present disclosure.
  • FIG. 7 illustrates a graphical representation of a Spatial Similarity Quotient (SSQ) comparison between hexagonal grid maps of FIGS. 5B and 6 in accordance with an example embodiment of the present disclosure.
  • SSQ Spatial Similarity Quotient
  • the one or more hardware processors 104 identify one or more contexts from the one or more candidate contexts based on the determined Spatial Similarity Quotient.
  • the system 100 identifies which of the objects in the "object data base" i.e., Type O database are present in the environment and with what probability. This means if there are n-objects in Type O database then each environment or context C is represented a n-tuple where each element in the tuple is probability value of the object n being in that context. So one context C i will be a set ⁇ Pi ⁇ where i run from 1 to n and P i denotes the probability of i-th object present at that context C i . In an embodiment, there may be some probabilities with non-zero and with rest being zero as not all of the objects are present in one environment.
  • BedRoom say C k
  • it contains some Type O objects annotated like Bed, Bedlamp, AC (say O2, O8, O10), etc.
  • Another context can be a DrawingRoom (say C j ) where some Type O objects for example, Chair, Table, TV (say O5, O9, O7) etc., are present.
  • the system 100 gathers experience with this kind of environments, more confidently the system 100 can identify exact set of Type O objects for each different context.
  • the set of contexts may be referred as 'Type C database' .
  • each entry in the Type C database is a context.
  • the mapping database is practically a matrix containing probability of an object being in different context i.e., columns are objects from Type B database and rows are environments or contexts i.e., entries from Type C database.
  • a sample matrix of the context database may be represented by way of example below: NoOfObs for Object 1 NoOfObs for Object 2 NoOfObs for Object m NoOfObs for Context 1 Prob of Obj 1 being in Contxt 1 Prob of Obj 2 being in Contxt 1 Prob of Obj m being in Contxt 1 NoOfObs for Context 2 Prob of Obj 1 being in Contxt 2 Prob of Obj 2 being in Contxt 2 Prob of Obj m being in Contxt 2 NoOfObs for Context 3...
  • the value of the probabilities in each cell may keep on changing as life-time of the system 100 increases i.e., they change as more and more environments/objects and repetition of environment/objects are processed by the system 100.
  • the one or more hardware processors 104 dynamically plan a path for vector navigation (e.g., a start point and an end point) within the hexagonal grid map based on the one or more objects, and the identified one or more contexts.
  • a path for vector navigation e.g., a start point and an end point
  • the system 100 determines if any of the objects (of Type O) is present in that scene. If a certain set of Type O objects are found then the context from Type C is retrieved. However, the set of Type O may have a one-to-many relationship with Context from Type C. Thus multiple matches from Type C can be retrieved for a given scene. For e.g., ball and man from Type O together can be in environments 'room' and 'playground' from Type C.
  • the system 100 compares semantic similarity between these probable contexts and the newly arrived context.
  • Geometric transformation, hexagonal rotation, scaling up and down techniques, etc. may be applied on the full or part of the map to identify the Maximum Spatial Similarity Quotient (SSQ) with current map and the probable Type C contexts.
  • SSQ Maximum Spatial Similarity Quotient
  • a certain threshold value of SSQ (as per accuracy requirement of the use case or end application) is defined, and a comparison is performed between the probable contexts (or candidate contexts) and the hexagonal grid map under consideration.
  • the one or more hardware processors 104 perform a comparison of the hexagonal grid map with the one or more candidate contexts to obtain a maximum SSQ, and identify an environment as a new environment or a pre-defined environment based on the comparison of the maximum SSQ with a pre-defined threshold.
  • the system 100 identifies the current hexagonal grid map with corresponding subset of Type O objects as a new environment i.e., one member of Type C core database increases.
  • the maximum SSQ is greater than the threshold, then it is a pre-defined environment, else, when the maximum SSQ is less than or equal to the threshold, then it is a new environment.
  • each hexagonal center co-ordinate vector [ a i b i ] is multiplied by a rotation matrix R resulting into new rotated co-ordinate vectors a i ⁇ b i ⁇
  • R is represented by way of example expression below:
  • R Cos ⁇ ⁇ Sin ⁇ Sin ⁇ Cos ⁇
  • the ⁇ for which a minimum value of SSQ ⁇ is obtained would denote the angle of rotation for which maximum similarity between two map exist.
  • the target path is known for one of the maps and it is intended to find a direct translation of the path for the other map without executing the path planning algorithm.
  • the plot SSQ ⁇ versus ⁇ as shown in FIG. 7 shows that these two maps are spatially similar at an angle 200° for which SSQ ⁇ is minimum.
  • the translated path is shown in FIG. 8 with dotted line representation.
  • Context C1 If however the SSQ is found to be higher than the pre-defined threshold thus conveying a semantic match with a context, say Context C1, then the corresponding details of the context is updated with readings from this new entry i.e., Context C1 is updated with the belief value/probability and number of readings. Similarly, the corresponding entries in the Type O objects (that are present in the Context C1) are also updated i.e., variance for each object is updated.
  • Type S database consisting of 3-tuple provided by way of example below: ⁇ a, b, ObjectProbability>, where a and b are hexagonal center coordinates in (a, b) coordinate system defined above and ObjectProbability is the combined probability of finding any object among those defined in Type O database.
  • Type S database has multiple entries each corresponding to these 3-tuples.
  • (1 -ObjectProbability ) can be thought of as probability of the hexagonal grid cell being free for robotic motion. This means if (1 -ObjectProbability ) is zero then the hexagonal grid cell is completely free for navigation and there is no obstacle in that cell in the hexagonal grid map.
  • the tuple may represent as below: ⁇ 10, 12, Chair, 0.4, 50>
  • time-evolution is found to be a regular feature then it can be used for creating a stable time-model for that environment.
  • time stamp inside the tuples and they may represent as below:
  • these tuples are created based on multiple situations (here observations) the system 100 experiences in its lifetime or training. More the training/experience are, more refined or believable are the probability values of these tuples against an annotation.
  • the robot may have a fair idea about where to look for a TV in a Room (most probable place for a TV is wall) or it will know in advance that most probable place to look for a Table Lamp in a Room is at the corner. This information is useful for path planning in a given context because location of the objects are well known with some known level of confidence.
  • the present disclosure provides context based vector navigation for each hexagonal grip map wherein SSQ is identified with known environments as discussed above. Then a start and goal location is accepted on the current map from a user. This input may either be as specific hexagonal grid cells, or 2 identifiable objects on the map (e.g., a box, and a rack). Then the system 100 dynamically plans path (using D* Lite for known space or Q-Learning technique for unknown space) and at each step it takes input from the user (emulating robot sensors) of the actual dynamic objects present. The path may further be adjusted based on the chosen navigation algorithm.
  • the system 100 enables to load features of selected environment (both fixed and dynamic object probabilities).
  • the system 100 plans path using D* Lite or Q-Learning algorithm taking into account current map and loaded environment.
  • the chosen environment aka a context of Type A is a probabilistic model from the training set and the current incoming map is a map with ground truth which needs to be overlapped belief wise so that no object is overlooked.
  • the system 100 creates a probable environment map by a weighted overlap of the chosen environment with the given map. And path is planned based on objects in the probable environment map.
  • Type S map For example, say in a given Type S map, system calculates path from a start point to a goal point. Since Type S map actually provides past experience (using historical data stored in the memory for subsequent updation of the databases 108A-N (object database, context database, map database)) about position of objects and free paths in the corresponding environment, then based on a threshold of allowable objects (say 40%) the system offers three path options with object(s) probabilities 10%, 20% and 30 % respectively. Now given no idea about present snapshot of the same environment, the system may choose the first path with 10% obstacle probability. It is assumed that this path is also the shortest path.
  • object database object database, context database, map database
  • the system 100 gets the present snapshot of the environment and finds that there is an object in this 10% path, then it rejects the path and optimally selects one of the rest two paths based on minimal travel cost and obstacle probability. This enables the system 100 to further train a robot using the dynamically planned path, the one or more contexts and the one or more objects identified for subsequent path planning subsequent and vector navigation, and to further generate a database that comprises information pertaining to the subsequent path planning and vector navigation for increased efficiency and easy of navigation in future.
  • the system 100 eventually identifies a correlation or associativity between objects in a given context using the Type O and Type C database.
  • a sofa is usually associated with a table in a context or a mirror is usually associated with a wall in another context are some exemplary associativity and if these information are understood over experience by the system 100 then these information can be used efficiently during path planning. For example, if a chair is found in a context then there are high chance that a table is somewhere in vicinity/proximity and this information can be used by a robot to look for path beyond the probable position of table instead of actually identifying the obstacle named Table. More the training/experience are, more refined, strong and believable are the correlation/associativity between these objects.
  • the system 100 can be used to plan path across multiple contexts or environments which is not possible by present or traditional robotic path planning systems.
  • Navigation or path planning across contexts requires identifying common objects that fall in the intersection of both contexts. These objects that are there in the common intersection are the connecting points for inter-context path planning.
  • not all objects can be feasibly used for navigation, but only those that fall in the boundary of both contexts or are contiguous in both contexts can be used for navigation or path planning.
  • the system 100 (or the one or more hardware processors 104 identify one or more common objects (e.g., door, wall, window, etc.) that indicate an intersection of two or more contexts (e.g., kitchen, dining, and hall), and generate a path based on one or more navigable objects (e.g., door) identified from the one or more common objects from the intersection of the two or more contexts.
  • one or more common objects e.g., door, wall, window, etc.
  • two or more contexts e.g., kitchen, dining, and hall
  • the system 100 For example, for navigating from bed room to road outside, the system 100 identifies that it needs to cover three context namely bedroom, dining room and road. Now in the intersection of bedroom and dining room there may be objects like door, chair, etc. But door may be used for navigation across bedroom context to dining room context as door falls in the boundary of both maps.
  • This kind of context stitching and inter-context path planning can be used for robots and cars for city navigation. It is to be noted that not only the intersection of context is required but also a physical snapshot of current position / map is required for successful navigation. To elaborate using previous example, if the current snapshot of bedroom shows that the door common between bedroom and dining room is closed for some reason then though door is identified as most feasible object of passing through but the path planning may fail.
  • system 100 may also identify the one or more objects, the one or more contexts based on information obtained from one or more sensors. This is elaborated further by way of example below:
  • FIG. 9 illustrates an exemplary block diagram depicting an implementation of the system 100 in accordance with an example embodiment of the present disclosure.
  • the hardware device can be any kind of device which can be programmed including e.g. any kind of computer like a server or a personal computer, or the like, or any combination thereof.
  • the device may also include means which could be e.g. hardware means like e.g. an application-specific integrated circuit (ASIC), a field-programmable gate array (FPGA), or a combination of hardware and software means, e.g.
  • ASIC application-specific integrated circuit
  • FPGA field-programmable gate array
  • the means can include both hardware means and software means.
  • the method embodiments described herein could be implemented in hardware and software.
  • the device may also include software means.
  • the embodiments may be implemented on different hardware devices, e.g. using a plurality of CPUs.
  • the embodiments herein can comprise hardware and software elements.
  • the embodiments that are implemented in software include but are not limited to, firmware, resident software, microcode, etc.
  • the functions performed by various modules described herein may be implemented in other modules or combinations of other modules.
  • a computer-usable or computer readable medium can be any apparatus that can comprise, store, communicate, propagate, or transport the program for use by or in connection with the instruction execution system, apparatus, or device.
  • a computer-readable storage medium refers to any type of physical memory on which information or data readable by a processor may be stored.
  • a computer-readable storage medium may store instructions for execution by one or more processors, including instructions for causing the processor(s) to perform steps or stages consistent with the embodiments described herein.
  • the term "computer-readable medium” should be understood to include tangible items and exclude carrier waves and transient signals, i.e., be non-transitory. Examples include random access memory (RAM), read-only memory (ROM), volatile memory, nonvolatile memory, hard drives, CD ROMs, DVDs, flash drives, disks, and any other known physical storage media.

Landscapes

  • Engineering & Computer Science (AREA)
  • Radar, Positioning & Navigation (AREA)
  • Remote Sensing (AREA)
  • Automation & Control Theory (AREA)
  • Physics & Mathematics (AREA)
  • General Physics & Mathematics (AREA)
  • Aviation & Aerospace Engineering (AREA)
  • Databases & Information Systems (AREA)
  • Business, Economics & Management (AREA)
  • Health & Medical Sciences (AREA)
  • Artificial Intelligence (AREA)
  • Evolutionary Computation (AREA)
  • Game Theory and Decision Science (AREA)
  • Medical Informatics (AREA)
  • Image Analysis (AREA)
  • Control Of Position, Course, Altitude, Or Attitude Of Moving Bodies (AREA)
  • Navigation (AREA)
EP18206380.0A 2018-02-23 2018-11-15 Kontextbasierte wegplanung für die vektornavigation in hexagonalen raumkarten Active EP3531069B1 (de)

Applications Claiming Priority (1)

Application Number Priority Date Filing Date Title
IN201821007023 2018-02-23

Publications (2)

Publication Number Publication Date
EP3531069A1 true EP3531069A1 (de) 2019-08-28
EP3531069B1 EP3531069B1 (de) 2021-01-27

Family

ID=64331706

Family Applications (1)

Application Number Title Priority Date Filing Date
EP18206380.0A Active EP3531069B1 (de) 2018-02-23 2018-11-15 Kontextbasierte wegplanung für die vektornavigation in hexagonalen raumkarten

Country Status (3)

Country Link
US (1) US10948918B2 (de)
EP (1) EP3531069B1 (de)
JP (1) JP6691202B2 (de)

Cited By (4)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
WO2021098079A1 (zh) * 2019-11-21 2021-05-27 大连理工大学 一种利用双目立体相机构建栅格地图的方法
CN112880663A (zh) * 2021-01-19 2021-06-01 西北工业大学 一种考虑累积误差的auv强化学习路径规划方法
CN113203420A (zh) * 2021-05-06 2021-08-03 浙江大学 一种基于变密度搜索空间的工业机器人动态路径规划方法
CN113607171A (zh) * 2021-08-04 2021-11-05 清华大学建筑设计研究院有限公司 一种疏散路径规划方法、装置、设备及存储介质

Families Citing this family (8)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
US11126199B2 (en) * 2018-04-16 2021-09-21 Baidu Usa Llc Learning based speed planner for autonomous driving vehicles
US11586914B2 (en) * 2019-01-11 2023-02-21 Arizona Board Of Regents On Behalf Of Arizona State University Systems and methods for evaluating perception systems for autonomous vehicles using quality temporal logic
CN110631601B (zh) * 2019-11-13 2021-04-27 中国电子科技集团公司第二十八研究所 一种基于非显示拓扑矢量地图的路径规划方法
CN111811514B (zh) * 2020-07-03 2023-06-09 大连海事大学 一种基于正六边形栅格跳点搜索算法的路径规划方法
CN112067011B (zh) * 2020-08-24 2024-04-26 安庆师范大学 一种基于大规模多中心问题的路径规划方法
US11356801B1 (en) * 2020-12-25 2022-06-07 Mapsted Corp. Method and system for targeted calibration
CN112985405B (zh) * 2021-02-18 2023-06-09 湖南国科微电子股份有限公司 一种机器人全覆盖路径规划方法、装置、设备及介质
CN112807691B (zh) * 2021-02-22 2022-07-29 腾讯科技（深圳）有限公司 一种寻路方法、系统及存储介质和终端设备

Citations (1)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
EP2570772A1 (de) * 2011-09-16 2013-03-20 Deutsches Zentrum für Luft- und Raumfahrt e.V. Verfahren zur Lokalisation und Kartierung von Fußgängern und Robotern unter Verwendung von drahtlosen Zugangspunkten

Family Cites Families (9)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
US4874936A (en) * 1988-04-08 1989-10-17 United Parcel Service Of America, Inc. Hexagonal, information encoding article, process and system
US4998010A (en) * 1988-04-08 1991-03-05 United Parcel Service Of America, Inc. Polygonal information encoding article, process and system
JP2003266349A (ja) * 2002-03-18 2003-09-24 Sony Corp 位置認識方法、その装置、そのプログラム、その記録媒体及び位置認識装置搭載型ロボット装置
KR100791382B1 (ko) * 2006-06-01 2008-01-07 삼성전자주식회사 로봇의 이동 경로에 따라 소정 영역의 특성에 관한 정보를수집하고 분류하는 방법 및 상기 영역 특성에 따라제어되는 로봇, 상기 영역 특성을 이용한 ｕｉ 구성 방법및 장치
WO2011033100A1 (de) 2009-09-18 2011-03-24 Deutsches Zentrum Fuer Luft- Und Raumfahrt E.V. Verfahren zur erstellung einer karte bezüglich ortsbezogener angaben über die wahrscheinlichkeit der zukünftigen bewegung einer person
US8942917B2 (en) 2011-02-14 2015-01-27 Microsoft Corporation Change invariant scene recognition by an agent
JP2013186615A (ja) * 2012-03-07 2013-09-19 Brother Ind Ltd 自動搬送体、自動搬送体制御装置、自動搬送システム、自動搬送方法、及び自動搬送プログラム
CN103413313B (zh) 2013-08-19 2016-08-10 国家电网公司 基于电力机器人的双目视觉导航系统及方法
US20190122146A1 (en) * 2017-10-23 2019-04-25 Artificial Intelligence Foundation, Inc. Dynamic and Intuitive Aggregation of a Training Dataset

Patent Citations (1)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
EP2570772A1 (de) * 2011-09-16 2013-03-20 Deutsches Zentrum für Luft- und Raumfahrt e.V. Verfahren zur Lokalisation und Kartierung von Fußgängern und Robotern unter Verwendung von drahtlosen Zugangspunkten

Non-Patent Citations (4)

* Cited by examiner, † Cited by third party
Title
G. BUZSAKI; E. I. MOSER: "Memory, navigation and theta rhythm in the hippocampal-entorhinal system", NATURE NEUROSCIENCE, vol. 16, no. 2, 2013, pages 130 - 138
JAMES GORMAN: "Grid Cells: "Crystals of the Brain"", 29 April 2013 (2013-04-29), XP055589488, Retrieved from the Internet <URL:https://www.nytimes.com/2013/04/30/science/how-grid-cells-in-brain-help-map-out-space.html> [retrieved on 20190516] *
JISHA V R ET AL: "Frontier Based Goal Seeking for Robots in Unknown Environments", JOURNAL OF INTELLIGENT AND ROBOTIC SYSTEMS ; THEORY AND APPLICATIONS - (INCORPORATING MECHATRONIC SYSTEMS ENGINEERING), KLUWER ACADEMIC PUBLISHERS, DO, vol. 67, no. 3 - 4, 1 March 2012 (2012-03-01), pages 229 - 254, XP035100106, ISSN: 1573-0409, DOI: 10.1007/S10846-012-9658-9 *
SIMON LE GLOANNEC ET AL: "Unknown Area Exploration with an Autonomous Robot using Markov Decision Processes", HAL ARCHIVES-OUVERTES.FR, 31 August 2010 (2010-08-31), XP055589490, Retrieved from the Internet <URL:https://hal.archives-ouvertes.fr/hal-01438198/document> [retrieved on 20190516] *

Cited By (6)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
WO2021098079A1 (zh) * 2019-11-21 2021-05-27 大连理工大学 一种利用双目立体相机构建栅格地图的方法
US11315318B2 (en) 2019-11-21 2022-04-26 Dalian University Of Technology Method for constructing grid map by using binocular stereo camera
CN112880663A (zh) * 2021-01-19 2021-06-01 西北工业大学 一种考虑累积误差的auv强化学习路径规划方法
CN113203420A (zh) * 2021-05-06 2021-08-03 浙江大学 一种基于变密度搜索空间的工业机器人动态路径规划方法
CN113607171A (zh) * 2021-08-04 2021-11-05 清华大学建筑设计研究院有限公司 一种疏散路径规划方法、装置、设备及存储介质
CN113607171B (zh) * 2021-08-04 2023-05-26 清华大学建筑设计研究院有限公司 一种疏散路径规划方法、装置、设备及存储介质

Also Published As

Publication number Publication date
EP3531069B1 (de) 2021-01-27
JP2019169125A (ja) 2019-10-03
US10948918B2 (en) 2021-03-16
US20190265711A1 (en) 2019-08-29
JP6691202B2 (ja) 2020-04-28

Similar Documents

Publication Publication Date Title
EP3531069B1 (de) Kontextbasierte wegplanung für die vektornavigation in hexagonalen raumkarten
Shrestha et al. Learned map prediction for enhanced mobile robot exploration
Lyu et al. Robot path planning by leveraging the graph-encoded Floyd algorithm
Saeedi et al. Neural network-based multiple robot simultaneous localization and mapping
Zhu et al. Global path planning of mobile robots using a memetic algorithm
Fabrizi et al. Augmenting topology-based maps with geometric information
Rashid et al. Multi-robot localization and orientation estimation using robotic cluster matching algorithm
Haj Darwish et al. Using the Bees Algorithm for wheeled mobile robot path planning in an indoor dynamic environment
Ly et al. Autonomous exploration, reconstruction, and surveillance of 3d environments aided by deep learning
Li et al. A new path planning method based on concave polygon convex decomposition and artificial bee colony algorithm
Zhang et al. Building metric-topological map to efficient object search for mobile robot
Keshavarzi et al. Scenegen: Generative contextual scene augmentation using scene graph priors
Wu et al. ST-FMT*: A fast optimal global motion planning for mobile robot
Best et al. Decentralised self-organising maps for the online orienteering problem with neighbourhoods
Feng et al. Floorplannet: Learning topometric floorplan matching for robot localization
Kim et al. CT-Loc: Cross-domain visual localization with a channel-wise transformer
Kaleci et al. Constructing topological map from metric map using spectral clustering
Soni et al. Multi-robot unknown area exploration using frontier trees
EP4216015A1 (de) Systeme und verfahren zur objekterkennung unter verwendung einer geometrischen semantischen kartenbasierten roboternavigation
Faigl et al. Inspection planning in the polygonal domain by Self-Organizing Map
Valentin et al. Motion strategies for exploration and map building under uncertainty with multiple heterogeneous robots
Taniguchi et al. Planning on topological map using omnidirectional images and spherical CNNs
Zhou et al. Topological segmentation for indoor environments from grid maps using an improved NJW algorithm
Jun et al. Towards a realistic indoor world reconstruction: Preliminary results for an object-oriented 3D RGB-D mapping
Hržica et al. Active vision for 3D indoor scene reconstruction using a 3D camera on a pan-tilt mechanism

Legal Events

Date Code Title Description
PUAI Public reference made under article 153(3) epc to a published international application that has entered the european phase

Free format text: ORIGINAL CODE: 0009012

STAA Information on the status of an ep patent application or granted ep patent

Free format text: STATUS: THE APPLICATION HAS BEEN PUBLISHED

AK Designated contracting states

Kind code of ref document: A1

Designated state(s): AL AT BE BG CH CY CZ DE DK EE ES FI FR GB GR HR HU IE IS IT LI LT LU LV MC MK MT NL NO PL PT RO RS SE SI SK SM TR

AX Request for extension of the european patent

Extension state: BA ME

STAA Information on the status of an ep patent application or granted ep patent

Free format text: STATUS: REQUEST FOR EXAMINATION WAS MADE

17P Request for examination filed

Effective date: 20200227

RBV Designated contracting states (corrected)

Designated state(s): AL AT BE BG CH CY CZ DE DK EE ES FI FR GB GR HR HU IE IS IT LI LT LU LV MC MK MT NL NO PL PT RO RS SE SI SK SM TR

GRAP Despatch of communication of intention to grant a patent

Free format text: ORIGINAL CODE: EPIDOSNIGR1

STAA Information on the status of an ep patent application or granted ep patent

Free format text: STATUS: GRANT OF PATENT IS INTENDED

RIC1 Information provided on ipc code assigned before grant

Ipc: G01C 21/00 20060101ALI20200710BHEP

Ipc: G01C 21/20 20060101AFI20200710BHEP

INTG Intention to grant announced

Effective date: 20200810

GRAS Grant fee paid

Free format text: ORIGINAL CODE: EPIDOSNIGR3

GRAA (expected) grant

Free format text: ORIGINAL CODE: 0009210

STAA Information on the status of an ep patent application or granted ep patent

Free format text: STATUS: THE PATENT HAS BEEN GRANTED

AK Designated contracting states

Kind code of ref document: B1

Designated state(s): AL AT BE BG CH CY CZ DE DK EE ES FI FR GB GR HR HU IE IS IT LI LT LU LV MC MK MT NL NO PL PT RO RS SE SI SK SM TR

REG Reference to a national code

Ref country code: GB

Ref legal event code: FG4D

REG Reference to a national code

Ref country code: CH

Ref legal event code: EP

REG Reference to a national code

Ref country code: AT

Ref legal event code: REF

Ref document number: 1358762

Country of ref document: AT

Kind code of ref document: T

Effective date: 20210215

REG Reference to a national code

Ref country code: IE

Ref legal event code: FG4D

REG Reference to a national code

Ref country code: DE

Ref legal event code: R096

Ref document number: 602018012245

Country of ref document: DE

REG Reference to a national code

Ref country code: NL

Ref legal event code: FP

REG Reference to a national code

Ref country code: LT

Ref legal event code: MG9D

REG Reference to a national code

Ref country code: AT

Ref legal event code: MK05

Ref document number: 1358762

Country of ref document: AT

Kind code of ref document: T

Effective date: 20210127

PG25 Lapsed in a contracting state [announced via postgrant information from national office to epo]

Ref country code: LT

Free format text: LAPSE BECAUSE OF FAILURE TO SUBMIT A TRANSLATION OF THE DESCRIPTION OR TO PAY THE FEE WITHIN THE PRESCRIBED TIME-LIMIT

Effective date: 20210127

Ref country code: PT

Free format text: LAPSE BECAUSE OF FAILURE TO SUBMIT A TRANSLATION OF THE DESCRIPTION OR TO PAY THE FEE WITHIN THE PRESCRIBED TIME-LIMIT

Effective date: 20210527

Ref country code: NO

Free format text: LAPSE BECAUSE OF FAILURE TO SUBMIT A TRANSLATION OF THE DESCRIPTION OR TO PAY THE FEE WITHIN THE PRESCRIBED TIME-LIMIT

Effective date: 20210427

Ref country code: GR

Free format text: LAPSE BECAUSE OF FAILURE TO SUBMIT A TRANSLATION OF THE DESCRIPTION OR TO PAY THE FEE WITHIN THE PRESCRIBED TIME-LIMIT

Effective date: 20210428

Ref country code: HR

Free format text: LAPSE BECAUSE OF FAILURE TO SUBMIT A TRANSLATION OF THE DESCRIPTION OR TO PAY THE FEE WITHIN THE PRESCRIBED TIME-LIMIT

Effective date: 20210127

Ref country code: FI

Free format text: LAPSE BECAUSE OF FAILURE TO SUBMIT A TRANSLATION OF THE DESCRIPTION OR TO PAY THE FEE WITHIN THE PRESCRIBED TIME-LIMIT

Effective date: 20210127

Ref country code: BG

Free format text: LAPSE BECAUSE OF FAILURE TO SUBMIT A TRANSLATION OF THE DESCRIPTION OR TO PAY THE FEE WITHIN THE PRESCRIBED TIME-LIMIT

Effective date: 20210427

PG25 Lapsed in a contracting state [announced via postgrant information from national office to epo]

Ref country code: AT

Free format text: LAPSE BECAUSE OF FAILURE TO SUBMIT A TRANSLATION OF THE DESCRIPTION OR TO PAY THE FEE WITHIN THE PRESCRIBED TIME-LIMIT

Effective date: 20210127

Ref country code: RS

Free format text: LAPSE BECAUSE OF FAILURE TO SUBMIT A TRANSLATION OF THE DESCRIPTION OR TO PAY THE FEE WITHIN THE PRESCRIBED TIME-LIMIT

Effective date: 20210127

Ref country code: LV

Free format text: LAPSE BECAUSE OF FAILURE TO SUBMIT A TRANSLATION OF THE DESCRIPTION OR TO PAY THE FEE WITHIN THE PRESCRIBED TIME-LIMIT

Effective date: 20210127

Ref country code: PL

Free format text: LAPSE BECAUSE OF FAILURE TO SUBMIT A TRANSLATION OF THE DESCRIPTION OR TO PAY THE FEE WITHIN THE PRESCRIBED TIME-LIMIT

Effective date: 20210127

Ref country code: SE

Free format text: LAPSE BECAUSE OF FAILURE TO SUBMIT A TRANSLATION OF THE DESCRIPTION OR TO PAY THE FEE WITHIN THE PRESCRIBED TIME-LIMIT

Effective date: 20210127

PG25 Lapsed in a contracting state [announced via postgrant information from national office to epo]

Ref country code: IS

Free format text: LAPSE BECAUSE OF FAILURE TO SUBMIT A TRANSLATION OF THE DESCRIPTION OR TO PAY THE FEE WITHIN THE PRESCRIBED TIME-LIMIT

Effective date: 20210527

REG Reference to a national code

Ref country code: DE

Ref legal event code: R097

Ref document number: 602018012245

Country of ref document: DE

PG25 Lapsed in a contracting state [announced via postgrant information from national office to epo]

Ref country code: CZ

Free format text: LAPSE BECAUSE OF FAILURE TO SUBMIT A TRANSLATION OF THE DESCRIPTION OR TO PAY THE FEE WITHIN THE PRESCRIBED TIME-LIMIT

Effective date: 20210127

Ref country code: EE

Free format text: LAPSE BECAUSE OF FAILURE TO SUBMIT A TRANSLATION OF THE DESCRIPTION OR TO PAY THE FEE WITHIN THE PRESCRIBED TIME-LIMIT

Effective date: 20210127

Ref country code: SM

Free format text: LAPSE BECAUSE OF FAILURE TO SUBMIT A TRANSLATION OF THE DESCRIPTION OR TO PAY THE FEE WITHIN THE PRESCRIBED TIME-LIMIT

Effective date: 20210127

PG25 Lapsed in a contracting state [announced via postgrant information from national office to epo]

Ref country code: DK

Free format text: LAPSE BECAUSE OF FAILURE TO SUBMIT A TRANSLATION OF THE DESCRIPTION OR TO PAY THE FEE WITHIN THE PRESCRIBED TIME-LIMIT

Effective date: 20210127

Ref country code: RO

Free format text: LAPSE BECAUSE OF FAILURE TO SUBMIT A TRANSLATION OF THE DESCRIPTION OR TO PAY THE FEE WITHIN THE PRESCRIBED TIME-LIMIT

Effective date: 20210127

Ref country code: SK

Free format text: LAPSE BECAUSE OF FAILURE TO SUBMIT A TRANSLATION OF THE DESCRIPTION OR TO PAY THE FEE WITHIN THE PRESCRIBED TIME-LIMIT

Effective date: 20210127

PLBE No opposition filed within time limit

Free format text: ORIGINAL CODE: 0009261

STAA Information on the status of an ep patent application or granted ep patent

Free format text: STATUS: NO OPPOSITION FILED WITHIN TIME LIMIT

26N No opposition filed

Effective date: 20211028

PG25 Lapsed in a contracting state [announced via postgrant information from national office to epo]

Ref country code: ES

Free format text: LAPSE BECAUSE OF FAILURE TO SUBMIT A TRANSLATION OF THE DESCRIPTION OR TO PAY THE FEE WITHIN THE PRESCRIBED TIME-LIMIT

Effective date: 20210127

Ref country code: AL

Free format text: LAPSE BECAUSE OF FAILURE TO SUBMIT A TRANSLATION OF THE DESCRIPTION OR TO PAY THE FEE WITHIN THE PRESCRIBED TIME-LIMIT

Effective date: 20210127

PG25 Lapsed in a contracting state [announced via postgrant information from national office to epo]

Ref country code: SI

Free format text: LAPSE BECAUSE OF FAILURE TO SUBMIT A TRANSLATION OF THE DESCRIPTION OR TO PAY THE FEE WITHIN THE PRESCRIBED TIME-LIMIT

Effective date: 20210127

PG25 Lapsed in a contracting state [announced via postgrant information from national office to epo]

Ref country code: IT

Free format text: LAPSE BECAUSE OF FAILURE TO SUBMIT A TRANSLATION OF THE DESCRIPTION OR TO PAY THE FEE WITHIN THE PRESCRIBED TIME-LIMIT

Effective date: 20210127

PG25 Lapsed in a contracting state [announced via postgrant information from national office to epo]

Ref country code: IS

Free format text: LAPSE BECAUSE OF FAILURE TO SUBMIT A TRANSLATION OF THE DESCRIPTION OR TO PAY THE FEE WITHIN THE PRESCRIBED TIME-LIMIT

Effective date: 20210527

PG25 Lapsed in a contracting state [announced via postgrant information from national office to epo]

Ref country code: MC

Free format text: LAPSE BECAUSE OF FAILURE TO SUBMIT A TRANSLATION OF THE DESCRIPTION OR TO PAY THE FEE WITHIN THE PRESCRIBED TIME-LIMIT

Effective date: 20210127

PG25 Lapsed in a contracting state [announced via postgrant information from national office to epo]

Ref country code: LU

Free format text: LAPSE BECAUSE OF NON-PAYMENT OF DUE FEES

Effective date: 20211115

Ref country code: BE

Free format text: LAPSE BECAUSE OF NON-PAYMENT OF DUE FEES

Effective date: 20211130

REG Reference to a national code

Ref country code: BE

Ref legal event code: MM

Effective date: 20211130

PG25 Lapsed in a contracting state [announced via postgrant information from national office to epo]

Ref country code: IE

Free format text: LAPSE BECAUSE OF NON-PAYMENT OF DUE FEES

Effective date: 20211115

PG25 Lapsed in a contracting state [announced via postgrant information from national office to epo]

Ref country code: CY

Free format text: LAPSE BECAUSE OF FAILURE TO SUBMIT A TRANSLATION OF THE DESCRIPTION OR TO PAY THE FEE WITHIN THE PRESCRIBED TIME-LIMIT

Effective date: 20210127

P01 Opt-out of the competence of the unified patent court (upc) registered

Effective date: 20230526

PG25 Lapsed in a contracting state [announced via postgrant information from national office to epo]

Ref country code: HU

Free format text: LAPSE BECAUSE OF FAILURE TO SUBMIT A TRANSLATION OF THE DESCRIPTION OR TO PAY THE FEE WITHIN THE PRESCRIBED TIME-LIMIT; INVALID AB INITIO

Effective date: 20181115

PGFP Annual fee paid to national office [announced via postgrant information from national office to epo]

Ref country code: NL

Payment date: 20231124

Year of fee payment: 6

PGFP Annual fee paid to national office [announced via postgrant information from national office to epo]

Ref country code: GB

Payment date: 20231121

Year of fee payment: 6

PGFP Annual fee paid to national office [announced via postgrant information from national office to epo]

Ref country code: FR

Payment date: 20231123

Year of fee payment: 6

Ref country code: DE

Payment date: 20231127

Year of fee payment: 6

Ref country code: CH

Payment date: 20231201

Year of fee payment: 6

PG25 Lapsed in a contracting state [announced via postgrant information from national office to epo]

Ref country code: MK

Free format text: LAPSE BECAUSE OF FAILURE TO SUBMIT A TRANSLATION OF THE DESCRIPTION OR TO PAY THE FEE WITHIN THE PRESCRIBED TIME-LIMIT

Effective date: 20210127