WO2020003319A1 - Techniques de localisation - Google Patents
Techniques de localisation Download PDFInfo
- Publication number
- WO2020003319A1 WO2020003319A1 PCT/IL2019/050718 IL2019050718W WO2020003319A1 WO 2020003319 A1 WO2020003319 A1 WO 2020003319A1 IL 2019050718 W IL2019050718 W IL 2019050718W WO 2020003319 A1 WO2020003319 A1 WO 2020003319A1
- Authority
- WO
- WIPO (PCT)
- Prior art keywords
- particle filter
- sensor
- input
- location
- map
- Prior art date
Links
Classifications
-
- G—PHYSICS
- G01—MEASURING; TESTING
- G01C—MEASURING DISTANCES, LEVELS OR BEARINGS; SURVEYING; NAVIGATION; GYROSCOPIC INSTRUMENTS; PHOTOGRAMMETRY OR VIDEOGRAMMETRY
- G01C21/00—Navigation; Navigational instruments not provided for in groups G01C1/00 - G01C19/00
- G01C21/20—Instruments for performing navigational calculations
-
- G—PHYSICS
- G01—MEASURING; TESTING
- G01C—MEASURING DISTANCES, LEVELS OR BEARINGS; SURVEYING; NAVIGATION; GYROSCOPIC INSTRUMENTS; PHOTOGRAMMETRY OR VIDEOGRAMMETRY
- G01C21/00—Navigation; Navigational instruments not provided for in groups G01C1/00 - G01C19/00
- G01C21/20—Instruments for performing navigational calculations
- G01C21/206—Instruments for performing navigational calculations specially adapted for indoor navigation
Definitions
- the present invention in some embodiments thereof, relates to a method of location using inputs from multiple sensors, and, more particularly, but not exclusively, to a method for converging candidate locations to a more accurate location and, more particularly, but not exclusively, to a method for fusing image -based navigation with additional location inputs to obtain a more accurate location, and more particularly, but not exclusively to a particle filter method for converging candidate locations,
- the present invention in some embodiments thereof, relates to a method of location using inputs from multiple sensors, and, more particularly, but not exclusively, to a method for converging candidate locations to a more accurate location and, more particularly, but not exclusively, to a method for fusing image -based navigation with additional location inputs to obtain a more accurate location, and more particularly, but not exclusively to a particle filter method for converging candidate locations.
- a localization method including obtaining a map of a Region Of Interest (ROI), obtaining a first input from a first sensor and a second input from a second sensor, providing the first input and the second input to a processor, using the processor to estimate a location based on the first input and the second input, wherein the processor uses a particle filter method to estimate the location.
- ROI Region Of Interest
- the first input includes images from a camera.
- the second input includes data associated which is unavailable in some area in the ROI.
- the particle filter method is a modified particle filter method in which associates a likelihood with a candidate location based upon the first input and the second input.
- the particle filter method is a modified particle filter method further including performing soft-init.
- performing soft-init is used to solve a state called“kidnapped-robot”.
- the soft-init includes adding a number of particles, the number in a range of 1-10% of a total number of particles, when the particle filter method performs re-sampling.
- the adding a number of particles includes adding particles associated with candidate locations having a probability above a threshold probability, the probability based on a consideration selected from a group consisting of the candidate location can image a light source in the ROI, the candidate location can image a sign in the ROI, and the candidate location is in an elevator and an altitude change has been detected.
- the particle filter method is a modified particle filter method further including removing a fraction of the particles each re-sample, the fraction in a range of 1-25% of a total number of the particles.
- the particle filter method is a modified particle filter method further including using elevation change data as a particle filter map constraint.
- the particle filter method is a modified particle filter method further including using one or more distinct environmental features in a particle filter map, the distinct features selected from a group consisting of a light, a ceiling light, a sign.
- the particle filter method is a modified particle filter method further including using both angular bias and angular drift as part of a particle state.
- the particle filter method is a modified particle filter method further including adapting a number of initial particles to a navigation scenario.
- the particle filter method is a modified particle filter method further including using pedometry based on one or more data inputs selected from a group consisting of optical flow, distance-to-object ranging, and device orientation.
- one of the first input and the second input includes a light level input.
- At least one of the first input and the second input includes a sensor in a smart phone or tablet.
- At least one of the first input and the second input includes a sensor installed in a car.
- At least one of the first input and the second input includes a sensor selected from a group consisting of a GPS receiver, a GNSS receiver, a WiFi receiver, a Bluetooth receiver, a Bluetooth Low Energy (BLE) receiver, a 3G receiver, a 4G receiver, a 5G receiver, an acceleration sensor, a pedometer, an odometer, an attitude sensor, a MEMS sensor, a magnetometer, a pressure sensor, a light sensor, an audio sensor, a microphone, a camera, a multi-lens camera, a Time-Of-Flight (TOF) camera, a range-finder sensor, an ultrasonic range-finder, a Lidar, an RFID sensor, and a NFC sensor.
- the particle filter method adapts a weight of a candidate location based upon associating a change in light level to proximity to a door of a building or proximity to a window.
- the particle filter method adapts a weight of a candidate location based upon a map of WiFi reception strength.
- the particle filter method adapts a weight of a candidate location based upon associating a change in GPS signal reception level to proximity to a door of a building or proximity to a window.
- the particle filter method adapts a weight of a candidate location based upon associating a vertical acceleration with an elevator or an escalator or stairs.
- the particle filter method adapts a weight of a candidate location based upon associating a change in pressure with an elevator or an escalator or stairs.
- the particle filter method adapts a weight of a candidate location based upon associating a change in magnetic field with a magnometer placed in proximity to a door of a building.
- the particle filter method includes producing initial candidate locations, and iteratively improving accuracy of the candidate locations, and wherein at least some of the candidate locations are cancelled during at least one iteration.
- the method is used for navigation in an area where GNSS signals are not received.
- the method is used for navigation in a car park.
- the method is used for navigation in a tunnel.
- a method of mapping a Region Of Interest including obtaining first sensor data from a first sensor and second sensor data from a second sensor, providing the first sensor data and the second sensor data to a processor, using the processor to estimate a location based on the first sensor data and the second sensor data, and sending the location to a mapping application.
- mapping application further including using the mapping application to display at least one of the first sensor data and the second sensor data.
- a localization method including a) obtaining a map of a Region Of Interest (ROI), b) obtaining a first input from a first sensor, c) providing the first input to a processor, d) using the processor to estimate a location based on the first input, e) moving from the location and repeating (b)-(d), and further including f) obtaining a second input from a second sensor, g) providing the second input to the processor, h) using the processor to estimate a location based on the second input in addition to the first input, thereby increasing accuracy of the estimating the location.
- ROI Region Of Interest
- the processor uses a particle filter method to estimate the location.
- the second sensor provides input intermittently.
- the second sensor provides input only in specific areas of the ROI.
- some embodiments of the present invention may be embodied as a system, method or computer program product. Accordingly, some embodiments of the present invention may take the form of an entirely hardware embodiment, an entirely software embodiment (including firmware, resident software, micro-code, etc.) or an embodiment combining software and hardware aspects that may all generally be referred to herein as a“circuit,”“module” or“system.” Furthermore, some embodiments of the present invention may take the form of a computer program product embodied in one or more computer readable medium(s) having computer readable program code embodied thereon. Implementation of the method and/or system of some embodiments of the invention can involve performing and/or completing selected tasks manually, automatically, or a combination thereof. Moreover, according to actual instrumentation and equipment of some embodiments of the method and/or system of the invention, several selected tasks could be implemented by hardware, by software or by firmware and/or by a combination thereof, e.g., using an operating system.
- a data processor such as a computing platform for executing a plurality of instructions.
- the data processor includes a volatile memory for storing instructions and/or data and/or a non-volatile storage, for example, a magnetic hard-disk and/or removable media, for storing instructions and/or data.
- a network connection is provided as well.
- a display and/or a user input device such as a keyboard or mouse are optionally provided as well.
- the computer readable medium may be a computer readable signal medium or a computer readable storage medium.
- a computer readable storage medium may be, for example, but not limited to, an electronic, magnetic, optical, electromagnetic, infrared, or semiconductor system, apparatus, or device, or any suitable combination of the foregoing.
- a computer readable storage medium may be any tangible medium that can contain, or store a program for use by or in connection with an instruction execution system, apparatus, or device.
- a computer readable signal medium may include a propagated data signal with computer readable program code embodied therein, for example, in baseband or as part of a carrier wave. Such a propagated signal may take any of a variety of forms, including, but not limited to, electro magnetic, optical, or any suitable combination thereof.
- a computer readable signal medium may be any computer readable medium that is not a computer readable storage medium and that can communicate, propagate, or transport a program for use by or in connection with an instruction execution system, apparatus, or device.
- Program code embodied on a computer readable medium and/or data used thereby may be transmitted using any appropriate medium, including but not limited to wireless, wireline, optical fiber cable, RF, etc., or any suitable combination of the foregoing.
- Computer program code for carrying out operations for some embodiments of the present invention may be written in any combination of one or more programming languages, including an object oriented programming language such as Java, Smalltalk, C++ or the like and conventional procedural programming languages, such as the "C" programming language or similar programming languages.
- the program code may execute entirely on the user's computer, partly on the user's computer, as a stand-alone software package, partly on the user's computer and partly on a remote computer or entirely on the remote computer or server.
- the remote computer may be connected to the user's computer through any type of network, including a local area network (LAN) or a wide area network (WAN), or the connection may be made to an external computer (for example, through the Internet using an Internet Service Provider).
- LAN local area network
- WAN wide area network
- Internet Service Provider for example, AT&T, MCI, Sprint, EarthLink, MSN, GTE, etc.
- These computer program instructions may also be stored in a computer readable medium that can direct a computer, other programmable data processing apparatus, or other devices to function in a particular manner, such that the instructions stored in the computer readable medium produce an article of manufacture including instructions which implement the function/act specified in the flowchart and/or block diagram block or blocks.
- the computer program instructions may also be loaded onto a computer, other programmable data processing apparatus, or other devices to cause a series of operational steps to be performed on the computer, other programmable apparatus or other devices to produce a computer implemented process such that the instructions which execute on the computer or other programmable apparatus provide processes for implementing the functions/acts specified in the flowchart and/or block diagram block or blocks.
- Some of the methods described herein are generally designed only for use by a computer, and may not be feasible or practical for performing purely manually, by a human expert.
- a human expert who wanted to manually perform similar tasks such as determining his location indoors, or navigating indoors, might be expected to use completely different methods, e.g., making use of expert knowledge and/or the pattern recognition capabilities of the human brain, which may be more efficient than manually going through the steps of the methods described herein.
- FIGETRE 1A is a simplified block diagram illustration of a mobile device for navigation and/or localization according to an example embodiment of the invention
- FIGETRE 1B is a simplified block diagram illustration of a device for producing maps according to an example embodiment of the invention.
- FIGETRE 1C is a simplified block diagram illustration of a system for producing maps according to an example embodiment of the invention.
- FIGETRE 1D is a simplified block diagram illustration of a system for vehicle navigation according to an example embodiment of the invention.
- FIGETRE 1E is a simplified block diagram illustration of a system for producing maps for vehicles according to an example embodiment of the invention
- FIGURE 1F is a simplified flowchart illustration of a method for navigation and/or localization according to an example embodiment of the invention
- FIGURE 1G is a simplified flowchart illustration of a method for producing maps according to an example embodiment of the invention.
- FIGURE 1H is a simplified flowchart illustration of a method for navigation and/or localization according to an example embodiment of the invention
- FIGURE II is a simplified flowchart illustration of a method for navigation and/or localization according to an example embodiment of the invention.
- FIGURES 2A-2C are simplified illustrations of progressing from building level to room level and to a higher accuracy of“seat” level;
- FIGURES 3A-3C are simplified illustrations of using a particle filter to estimate a location according to an example embodiment of the invention.
- FIGURES 4A-4B are simplified illustrations of applying a threshold to a color image of ceiling lights according to an example embodiment of the invention.
- FIGURES 4C-4D are simplified illustrations of contour detection of light sources on an image and determining center mass points according to an example embodiment of the invention
- FIGURE 5 is a simplified illustration of a map used in mapping RF and visual data according to an example embodiment of the invention
- FIGURE 6 is a simplified illustration of light source mapping according to an example embodiment of the invention.
- FIGURE 7 is a simplified illustration of a screenshot of a navigating application according to an example embodiment of the invention.
- FIGURES 8A-8B are simplified illustrations of tracking according to an example embodiment of the invention in comparison to the ground truth;
- FIGURE 9 is a simplified illustration of a screenshot of a navigating application displaying a large area, according to an example embodiment of the invention.
- FIGURE 10 is a graph of an accuracy evaluation of a path according to an example embodiment of the invention and a path according to Google map;
- FIGURES 11A-12C are simplified illustrations of using a particle filter to estimate a location according to an example embodiment of the invention.
- FIGURE 11D is a graph showing atmospheric pressure measurements over time of according to an example embodiment of the invention.
- FIGURE 12 is a simplified illustration of a multicolor map used according to an example embodiment of the invention
- FIGURE 13 is a simplified illustration of a multicolor map used according to an example embodiment of the invention
- FIGURE 14 is a simplified illustration of a map showing a comparison of a location determined by Google’s fused location service and a location method used according to an example embodiment of the invention
- FIGURES 15A-15B are simplified illustrations of a potential advantage of using indoor/outdoor determination according to an example embodiment of the invention.
- FIGURE 16 is a simplified illustration of tracking according to an example embodiment of the invention in comparison to a Lidar-based ground truth
- FIGURE 17 is a graph showing height value error over time according to an example embodiment of the invention.
- FIGURE 18 is a graph showing position error over time according to an example embodiment of the invention.
- FIGURE 19 is a graph showing position error over time according to an example embodiment of the invention.
- FIGURE 20 is a graph showing position error over a path according to an example embodiment of the invention.
- FIGURE 21 is a graph showing position error according to an example embodiment of the invention.
- FIGURE 22 is a graph showing linear acceleration as captured by a smartphone positioned in a car according to an example embodiment of the invention.
- FIGURE 23 is a color image of a contemporary parking-lot with speed bumps, color column markings, and location codes
- FIGURE 24 is an example parking lot map
- FIGURES 25A-25C are screen capture illustrations of a mapping tool implemented on a smartphone according to an example embodiment of the invention.
- FIGURE 26 is a photograph of a tunnel including visual landmarks according to an example embodiment of the invention.
- FIGURES 27 A and 27B are images of a highway according to an example embodiment of the invention. DESCRIPTION OF SPECIFIC EMBODIMENTS OF THE INVENTION
- the present invention in some embodiments thereof, relates to a method of location using inputs from multiple sensors, and, more particularly, but not exclusively, to a method for converging candidate locations to a more accurate location and, more particularly, but not exclusively, to a method for fusing image -based navigation with additional location inputs to obtain a more accurate location, and more particularly, but not exclusively to a particle filter method for converging candidate locations.
- An aspect of some embodiments is related to calculating a location of a device by producing candidate locations, followed by iteratively improving accuracy of the candidate locations, or reducing spread of the candidate locations.
- the location of the device is optionally tracked over time, especially if and/or when the device is moving.
- device localization is optionally used for indoor navigation, or for navigation in an environment which can provide more than one data source for use in calculating a location.
- the device moves, collecting sensor data from various locations, and the fact that the data is from different locations potentially improves the accuracy of the location.
- An aspect of some embodiments is related to fusing data from different sources to potentially reduce time to converge spread of the candidate locations and/or increase accuracy.
- one of the data sources is an intermittent data source, which is not available all the time and/or not available at every location.
- a particle filter algorithm is used to evaluate the candidate locations. It may be said that the data sources provide hints as to the correct location of the device. Some hints“nudge” the candidate locations toward a correct location. Some non-limiting examples of such hints can be odometer or pedometer readings, which enable calculation of candidate location similarly to a dead-reckoning (DR) method, by providing information how far the device has travelled. Some hints can significantly increase likelihood of a candidate location or rule out the likelihood of a candidate location.
- DR dead-reckoning
- Some non-limiting examples of such hints can be sensing when the device passes over a speed bump - that increases the likelihood of candidate locations which are near to locations of one or more known speed bump(s) in an environment, and decreases the likelihood of candidate locations where a map indicates that there are no speed bumps.
- An aspect of some embodiments is related to calculating a location of a device by starting fusing data at“room level” accuracy (-5-10 meters), which is available using some methods (e.g. WiFi maps) with data from additional sensors, such as described herein, to increase accuracy to “seat level” accuracy (sub-meter).
- a number of initial candidate locations is optionally produced.
- candidate locations are optionally used produce an estimated location.
- the estimated location is optionally calculated based on applying a method for calculating the estimated location. Some non-limiting examples of such methods include: using an average; using a weighted average, optionally where a weight of a candidate location may be based on its likelihood; selecting a most-likely candidate; iteratively calculating values of the candidate locations; adding additional candidate locations; and removing candidate locations, optionally based on least-likely or least-weighted.
- the weighting each particle is evaluated with respect to one or more of the sensory data and/or map constraints, and a weight of the particle is updated accordingly.
- the weights are optionally sampling-rate dependent.
- the sensory data constraints include constraint from one or more of any one of the sensors listed herein, or of sensors included in a smartphone, tablet, vehicle on-board sensors, and so on.
- the constraint is optionally that a location is more likely to be in an area where elevation change is possible (stairs, elevation, parking lot ramp) and less likely where such elevation change is unlikely (elevation change of more than 3 meters in a floor of a building where ceiling height is 3 meters).
- the constraint is optionally that a location is more likely to be in a parking lot area where a speed bump exists less likely where no such bump exists.
- each particle’s weight, or grade is updated according to how suitable the particle location and/or orientation is to the sensor input.
- a weight change is additive. In some embodiments a weight change is multiplicative.
- a change is optionally effected so that when there are two particles which are close by with respect to location, and optionally also orientation, the change brings their weight closer together, that is, a ratio of their weights is made closer to 1.
- a number of initial candidate locations is optionally controlled. In some embodiments, the number of initial candidate locations is optionally reduced relative to previously known methods such as described in above-mentioned article [YBM17] by Roi Yozevitch and Boaz Ben-Moshe titled“Advanced particle filter methods”.
- a number of initial candidate locations is optionally controlled based on a computing power of a device to be used to determine location.
- a computing power of a device to be used to determine location By limiting an initial number of particles the computing load is limited, and the method potentially enables calculating a location at a rate which is useful to a moving user.
- the number of candidate locations, or particles in a case of particle filer is reduced by 1-10 candidate locations per iteration.
- the number of candidate locations, or particles in a case of particle filter is reduced by 1-25% of the current candidate locations, per iteration.
- the number of candidate locations is optionally reduced during the process of iteration, relative to previously known methods such as described in above-mentioned article [YBM17] by Roi Yozevitch and Boaz Ben-Moshe titled“Advanced particle filter methods”.
- a method of calculation includes making a soft-init, or soft initialization.
- soft-init and soft-initialization are used herein interchangeably, to refer to the process described below.
- a soft-init is optionally used to re-start calculation of the location of the device.
- the soft- init includes adding a few, even just 1-10, particles somewhere, optionally even randomly distributed, within the Region-Of-Interest (ROI).
- ROI Region-Of-Interest
- the new particles are optionally not dependent on locations of other particles, and can potentially re-start the particle filter in converging to a correct location.
- a source for one or more of the localization inputs such as a sensor, stops providing data
- a soft-init is optionally used to re-start calculation of the location of the device.
- a method of calculation includes taking into account intermittent data input.
- a method of calculation optionally includes optionally compensating for intermittent data input.
- the device if the device has been calculated to be moving, the device is optionally located as if it is continuing its movement during pauses and stops in incoming data.
- a map is optionally used in relation to localizing a device.
- a map optionally defines a ROI in which the localization is to be made, or at least started or initialized.
- a map optionally defines constraints upon possible location of the device, typically in two dimension (2D) or three dimensions (3D).
- 2D two dimension
- 3D three dimensions
- Localizing or tracking a device is performed by using data input from one or more sensors.
- such sensors are optionally built into a smart phone, a tablet, a car, or a smart camera, optionally coupled with associated computing abilities.
- a non-limiting example of such sensors includes:
- a smart phone or tablet camera for example capturing image(s) of an environment, including features such a ceiling light, advertising, bill boards, street signs, store signs, doors, stairs, floor markings, wall colors, windows, hydrants, and additional identifiable features.
- the image(s) is processed to identify such features.
- a relative direction and/or distance from the camera to the features is optionally calculated.
- a simple sensing of relative light or darkness can suffice to determine whether the device is indoors, outdoors, or just passing through a door from outside to inside or vice versa.
- a distance measurement sensor for example a pair of lenses for measuring distance, Lidar (an acronym of light detection and ranging or of light imaging, detection, and ranging), ultrasonic range detector, camera auto-focus, applying camera image deep-learning to an image, and so on.
- Lidar an acronym of light detection and ranging or of light imaging, detection, and ranging
- ultrasonic range detector for example a pair of lenses for measuring distance
- camera auto-focus for example a camera auto-focus
- applying camera image deep-learning to an image and so on.
- WiFi is often used for localization.
- WiFi, Bluetooth (BLE), 3G, 4G and 5G signals can be used for locating, optionally based on RF fingerprinting or an RF map.
- GPS signals are often not received, or not received in a sufficient number or clarity to be used for standard GPS localization n.
- partial reception can optionally be used to indicate that the receiving device is close to a specific location, such as under a skylight, close to a window, and so on.
- locations in a GPS- restricted environment are optionally marked on a map, or reception of a GPS signal is compared to a map, and a constraint of proximity to a skylight or window or door is optionally deduced.
- - Acceleration sensors and/or inertial sensors are optionally used to determine movement or rate of movement from an initial location.
- Odometry in some embodiments odometry is optionally used.
- a rate of advance and/or direction of advance from a location is optionally deduced by counting steps, tire rotation (optionally converted to distance), calculating optical flow, and so on.
- any one or more of the above-mentioned sensors potentially provides input to a localization system.
- a camera such as a smartphone camera is selected as a first source of data, potentially providing candidate locations for placing on a map of a ROI, and data from other sensors is optionally used to calculate probability of likelihood of a candidate location or of candidate locations.
- sensors include: a GPS receiver; a GNSS receiver; a WiFi receiver; a Bluetooth receiver; a Bluetooth Low Energy (BLE) receiver; a 3G receiver; a 4G receiver; a 5G receiver; an acceleration sensor; a pedometer; an odometer; an attitude sensor; a MEMS sensor; a magnetometer; a pressure sensor; a light sensor; an audio sensor; a microphone; a camera; a multi-lens camera; a Time-Of-Flight (TOF) camera; a range-finder sensor; an ultrasonic range-finder; a Lidar; an RFID sensor; and a NFC sensor.
- Various sensors can provide different types of data.
- Some data is continuously available, and can be consistent. For example odometer data, pedometer data.
- Some data is available intermittently, based on location. Such data is available at some location, and not available at others. Some non-limiting examples of location-dependent data include: more intense light associated with windows and doors, GNSS or GPS signals received when passing near an entrance to a building or wen passing under a skylight.
- time-dependent data Some data is available intermittently, based on time. Such data is available at some time, and not available at others. Some non-limiting examples of time-dependent data include: more intense light associated with windows and doors during daylight hours, and not at night.
- low quality data Some data can be called low quality data.
- Some non-limiting examples of low quality data includes low quality images. Low quality images can be at lower resolution than maximal resolution available from current smartphones. However, even low quality images can potentially provide sensory data which can potentially assist in accurate location. By way of a non-limiting example, even a low quality image can provide directions, vectors, to ceiling lights, and assist in evaluating candidate locations.
- initial maps of a ROI are optionally obtained from a source for such maps.
- maps may be found on the Internet, as maps of airports, shopping malls, museums, parking lots, cities, and so on.
- paper maps are optionally scanned and optionally processed in order to provide digital maps.
- maps are obtained and/or updated according to methods described herein, optionally producing maps of visual landmarks such as ceiling lights, signs, and additional visual and non-visual features described herein.
- constructing a light source map is optionally done as described below, with reference to the“exemplary positioning embodiments”, with reference to the“visual landmark mapping” section therein, and with reference to Figure 6 and its description.
- a map is produced by a process of Simultaneous Localization And Mapping, optionally as described in SLAM [6].
- producing a map includes:
- an initial map for example a building plan (e.g., mall map, fire-escape map, an operator-made map, or even a blank map;
- a building plan e.g., mall map, fire-escape map, an operator-made map, or even a blank map;
- Locating landmarks e.g., lights & signs
- ray intersections optionally as presented in the“exemplary positioning embodiments”.
- mapping device has ranging capability, a SLAM like map is optionally produced;
- corrections to the path are optionally performed base on loop- closure, potentially increasing map accuracy
- a location of the elevation change is optionally recorded, and associated with a corresponding color, in a sense described with reference to Figure 12 below, representing different types of areas in the map, such as accessible areas, fixed inaccessible areas, partially-accessible areas, dynamic inaccessible areas and stairs or elevator.
- An aspect of some embodiments includes mapping environments.
- locations of a device are optionally collected, and additional data is optionally also collected.
- the locations and/or data are optionally used to produce a map and/or update a map.
- the map is optionally updated as to: more-exact locations where GPS may be received, corrections to locations calculated by WiFi, updating locations of signs or adding locations of new signs to a map, and so on.
- a map produced by using data from one or more device(s) navigating a ROI is optionally sent to devices for use in localization.
- signs are used to provide location.
- signs contain identifying information like text and drawings which make the signs highly identifiable.
- a number of similar signs in an environment such as a mall or a park, may be quite few, sometimes only one or two, compared, for example, with a number of ceiling lights.
- the signs may provide good candidate locations, as correctly identifying a unique sign can provide quality input to grade candidate locations. Even when there are a few similar signs, correctly identifying the signs can provide quality input to grade candidate locations, although several candidate locations may be provided with a high grade.
- Some signs provide significant assistance in estimating location - for example room numbers or store numbers, column number in a parking lot, and so on.
- image processing is optionally used to decipher number or text in a sign.
- captured images are optionally processed to determine whether a sign appears in the captured image, and optionally processed to determine what is written in the sign.
- a map which includes data about signs potentially associate each sign with one or more features.
- Some example features include geometric features which can assist location, such as a location of the sign, an orientation to which the sign is facing or from which it can be seen, and an actual size of the sign.
- Some example identifying features include text on a sign, font, colors, drawings, and so on.
- mapping environments it is noted that when a device or system is used for navigation or localization, the device optionally sends data about signs it sees and/or detects, to a central system which can enhance maps with data about signs.
- a map of an environment can optionally be updated whenever a localization or tracking application is used in that environment, becoming a“collective memory” of the environment.
- an object detection model Given an image or a video stream, an object detection model can identify which of a known set of objects might be present and provide information about their positions within the image. Such an object model is optionally applied with reference to signs.
- images are optionally filtered, for example with Python, optionally by the COCO API.
- the images are optionally filtered to use the images which contain legible text, optionally even filtering that such text is printed by machine.
- TensorFlow’s Object Detection API is used for image recognition
- the Tensorflow API provides some different pre-trained Deep leaning models (model previously trained over multiple huge datasets).
- indoors localization and/or tracking is performed, optionally using a camera and associated computing circuitry (e.g. on a smartphone or tablet).
- the localization a localization and/or tracking algorithm is based on a modified particle filter which combines visual landmarks with additional data inputs such as RF finger printing, odometry, and map constraints.
- the localization is potentially well suited for using a low resolution camera (e. g. 1 Megapixel) to track dominant landmarks such as lights and potentially achieve an accuracy of sub-meter 2D, 2.5D or 3D positioning at image capture rates of 10-30 Hz, which are typical and even lower than smartphone video rates.
- a low resolution camera e. g. 1 Megapixel
- Such an algorithm potentially works at a fairly low energy consumption suitable for smartphones.
- example embodiments of the invention have operated successfully using less than 100 milliwatts.
- the energy consumption is expected to be hardware dependent, yet easily suitable for operation on smartphones or tablets or other battery-dependent devices without causing undue drainage of the battery.
- the method optionally uses input of ceiling light(s) images and a ceiling light map.
- the method detects passing through doors, under a skylight or near a window by one or more of detecting changes in light intensity, light color.
- the method detects passing through a magnetometer.
- Passing under a skylight and/or passing near a window by receiving a GPS signal or a partial GPS signal.
- the method includes image processing to read signs such as shop signs, mall signs, parking lot signs, and so on.
- the method includes detecting obtaining a WiFi signal at an entrance.
- a parking lot typically includes features which sensors can detect.
- parking lot localization and/or tracking is performed, optionally using a camera on a smartphone, or a camera serving as a car-mounted system, and associated computing circuitry (e.g. in the smartphone or in the car).
- some features which can be detected and used as input, optionally combined with visual landmarks include:
- a tollbooth at an entry (visual detection); speed bumps at specific locations in a parking lot; layout of traffic lanes within a parking lot; relative height difference from known-height locations - e. g. by knowing where an entrance is and measuring barometric pressure difference to detect how many levels up or down, by sensing an incline and a distance travelled along the incline (odometry plus sensor attitude/direction); image processing for detecting parking-lot-specific markings such as numbers and/or letters and/or colors marked on columns; loss of GPS signal at entrance; obtaining a WiFi signal at an entrance.
- Figure 1A is a simplified block diagram illustration of a mobile device for navigation and/or localization according to an example embodiment of the invention.
- Figure 1A shows a mobile device 102, including a processor 102, sensors 104, and the processor having a communication channel 103 with the sensors 104.
- the mobile device 102 may be a smartphone, a tablet, or even a dedicated mobile navigation device.
- the sensors 104 may include a camera 106 (such sensors are very usual in smartphones and such devices), as well as other sensors 108.
- Figure 1B is a simplified block diagram illustration of a device for producing maps according to an example embodiment of the invention.
- Figure 1B shows a mobile device 110, including a processor 112, sensors 114, a mapping user interface 120, and the processor 112 having a communication channel 103 with the sensors 114 and a communication channel 119 with the mapping user interface 120.
- the mobile device 110 may be a smartphone, a tablet, or even a dedicated mobile mapping device.
- the sensors 114 may include a camera 116 (such sensors are very usual in smartphones and such devices), as well as other sensors 118.
- Figure 1C is a simplified block diagram illustration of a system for producing maps according to an example embodiment of the invention.
- Figure 1C shows a mobile device 125, having communication capability 134 with a mapping server 135.
- Figure 1C shows the mobile device 135 including a processor 127, sensors 128, a mapping user interface 131, and the processor 127 having a communication channel 132 with the sensors 128 and a communication channel 133 with the mapping user interface 131.
- the mobile device 125 may be a smartphone, a tablet, or even a dedicated mobile mapping device.
- the sensors 128 may include a camera 129 (such sensors are very usual in smartphones and such devices), as well as other sensors 130.
- FIG. 1D is a simplified block diagram illustration of a system for vehicle navigation according to an example embodiment of the invention.
- Figure 1D shows a mobile device 141, having communication capabilityl42 with a vehicle 140.
- Figure 1D shows the mobile device 141 including a processor 147, sensors 148, a mapping user interface 150, the processor 147 having a communication channel 149 with the sensors 148 and a communication channel 151 with the mapping user interface 150.
- Figure 1D also shows the vehicle 140 including a processor 143 and sensors 144, the processor 143 having a communication channel 145 with the sensors 144.
- mapping user interface 150 and its communication channel may be included in the vehicle 140.
- mapping user interface and its communication channel may be included in both the mobile device 141 and the vehicle 140.
- the mobile device 141 may optionally receive images from the vehicle’s camera or cameras, which may have better view outside the vehicle 140. In some embodiments the mobile device 141 may even be placed in a mobile device mount in the vehicle 141, and be used with images peripheral to the car.
- Figure 1E is a simplified block diagram illustration of a system for producing maps for vehicles according to an example embodiment of the invention.
- Figure 1E shows a vehicle 155, having communication capabilityl57 with a mapping server
- Figure 1E shows the vehicle 155 including a processor 160, sensors 161, with the processor 160 having a communication channel 162 with the sensors 161.
- Figure 1E also shows an optional mobile device 158, having communication capabilityl59 with the vehicle 155.
- Figure 1E shows the mobile device 158 including a processor 164, sensors 165, an optional mapping user interface 168, the processor 164 having a communication channel 166 with the sensors 165 and a communication channel 167 with the mapping user interface 168.
- mapping user interface 164 and its communication channel may be included in the vehicle 155.
- mapping user interface and its communication channel may be included in both the mobile device 158 and the vehicle 155.
- the mobile device 158 may have communication capability 169 with the mapping server 156.
- Figure 1F is a simplified flowchart illustration of a method for navigation and/or localization according to an example embodiment of the invention.
- the method of Figure 1F includes:
- ROI Region Of Interest
- the processor uses a particle filter method to estimate the location.
- Figure 1G is a simplified flowchart illustration of a method for producing maps according to an example embodiment of the invention.
- the method of Figure 1G includes:
- mapping application (1878).
- Figure 1H is a simplified flowchart illustration of a method for navigation and/or localization according to an example embodiment of the invention.
- the method of Figure 1H includes:
- ROI Region Of Interest
- Figure II is a simplified flowchart illustration of a method for navigation and/or localization according to an example embodiment of the invention.
- the method of Figure II includes:
- ROI Region Of Interest
- the processor uses a particle filter method to estimate the location.
- the second sensor provides input intermittently.
- the second sensor provides input only in specific areas of the ROI.
- the exemplary positioning embodiments below describe a general framework for positioning and navigation which improves expected accuracy over known methods such as using WLAN and cellular information.
- Figures 2A-2C are simplified illustrations of progressing from building level to room level and to a higher accuracy of“seat” level.
- Figure 2A is meant to illustrate a“building level” location accuracy, capable of estimating a location at an accuracy corresponding to a portion of a building. This can correspond to -10 accuracy.
- Figure 2B is meant to illustrate a“room level” location accuracy, capable of estimating a location at an accuracy corresponding to a specific room in a building. This can correspond to 2-5 meter accuracy.
- Figure 2C is meant to illustrate a“seat level” location accuracy, capable of estimating a location at an accuracy corresponding to, by way of a non-limiting example, a specific seat around a specific table, or a specific location within a room. This can correspond to sub-meter accuracy.
- An example embodiment method uses a modified particle filter which combines one or more of RF finger-printing, odometry, visual landmarks and map constraints.
- the potential accuracy improvement is achieved by using a camera, optionally a low resolution camera, to track dominant landmarks such as lights.
- a camera optionally a low resolution camera
- Use of “glowing-markers” potentially allows one to accurately map relatively complex indoor buildings, optionally using a compact representation.
- an example method as described in above-mentioned U.S. Patent Application Number 14/418,106 titled“Navigation method and device” was implemented and tested on Android-based mobile devices. The tests indicated a robust sub-meter 3D positioning at a frame capture rate of 10 - 30Hz with a fairly low energy consumption.
- High sampling rate typically required for a natural and intuitive navigation results, especially for highly dynamic devices.
- Low energy consumption is an important property for most mobile (or battery operated) devices, it usually requires the use of low computing power methods (e.g., the use of visual based navigation methods are usually impractical for mobile devices - due to their high computing power requirements for image and video processing algorithms).
- Minimal dedicated infrastructure preferably, a solution should work without any need for additional infrastructure.
- BYOD “Bring your own device”
- a need for reliable indoor positioning and navigation service is motivated by several applications including: Location Based Services,“where have I parked my car?”,“Where are my friends”, Augmented Reality gaming (for example a game such as“Pokemon Go”) and even search and rescue.
- An indoor positioning system is described which focuses on positioning methods for smart phones, optionally Commercial Off The Shelf (COTS) devices.
- COTS Commercial Off The Shelf
- an embodiment is described which is potentially useful for mass market scenarios in which an I P S potentially work on existing mobile devices (i.e., smart-phones), in some embodiments even without additional dedicated infrastructure.
- WLAN finger-printing this common method mainly uses WiFi or BlueTooth ( BLE ) beacon- signals in order to approximate a user position.
- the method uses a preprocessing stage (finger-printing) in which a site-survey is performed - storing the signal strength of received signals in many locations of a Region of Interest (ROI). Then, the location of the user can be approximated by comparing the current set of RF signals with the finger-printing data-set.
- ROI Region of Interest
- Pedestrian odometry this method uses step counting combined with the device approximated orientation in order to compute the user relative path, see: [Harl3], [BH13].
- Map Matching and Dead Reckoning this method is commonly used for GNSS-based (Global Navigation Satellite System - based) road position, in which the fact that the vehicle needs to be on the road implies significant constraints which can be used to reduce a search space.
- a related method can be used for indoor positioning - assuming we have the map constrains, see: [BK+ 08], [LSVW11], [BW13]
- Some positioning systems combine a few of the above methods in order to allow a better accuracy.
- major IPS providers such as Google and Apple provide a solution for a room or building level accuracy with expected error larger than 5 meters, therefore, even a user’s floor is often not automatically determined, see [LL17], [TSJK+17] for IPS accuracy evaluation.
- This application describes methods and systems for accurate indoor navigation.
- standard (C OT S) android mobile devices are used.
- C OT S standard
- the described system potentially allows a 3D sub-meter accuracy while maintaining one or more of the properties listed above (Accuracy, High sampling rate, Low energy consumption, Privacy and potentially without adding dedicated infrastructure).
- WLAN technologies refers not only to wireless LAN but also to other RF technologies, like: Blue-tooth/BLE, 3G/4G and RF - ID. Nevertheless, deployment of additional infrastructures such as RF - ID or RF -beacons are time consuming and often have high implementation costs, hence some embodiments of the present invention rely on existing infrastructures.
- WLAN technology is being deployed in public places, such as: shopping malls, industrial buildings, airports and hospitals.
- GNSS e.g., GPS
- W LAN based positioning systems can provide room level localization, usually with an accuracy of about 5 - 10 meters. Those systems typically use WLAN“Finger Printing” signal-map. A simple estimation can use the ratio of the current WLAN signal scan and the RF signal map.
- the particle filter localization algorithm also known as Monte Carlo Localization [FBDT99] is a variant of the Bayesian filter family [C+03].
- each x ⁇ is signed with a corresponding weight w t L that describes a belief state and evaluated proportionally to the likelihood of the Bayesian function p(z ⁇ x) where z is the sensory measurement.
- some preliminary terms are defined:
- ROI Environment
- 2D floor map
- building map - which may optionally contain several floors (i.e., 2.5 D), in some embodiments, one can also address the map as a 3D representation of the building.
- a set P of n particles is distributed on the environment map, and assigned with some initial weights.
- the set P is optionally uniformly distributed.
- Action A movement-function that describes a position change of a device. In robotics, for example, odometry is used, while for mobile devices carried by a person a pedometer (step counter with orientation) is optionally used.
- a Sense-function maps sensor data to a particle weight.
- Re-sample A process where the particles with negligible weights are replaced by new particles in the proximity of the particles with higher weights.
- An actual implementation of the resampling method is usually a key factor in the performance of a particle filter, in particular, the resampling method potentially affects convergence properties of the localization method.
- the particle filter optionally reports the current optimal position, for example by simply reporting the’’best” particle or by performing some kind of weighted average over the particles.
- reporting just a position or location is insufficient.
- a localization algorithm is optionally required to estimate an accuracy of the proposed location. Using the particles’ current distribution - one can use a method for estimating the expected accuracy.
- Algorithm 1 Simplified Particle Filter Algorithm For localization.
- FIGS. 3A-3C are simplified illustrations of using a particle filter to estimate a location according to an example embodiment of the invention.
- Figure 3A shows an initial distribution of particles, corresponding to an initial distribution of estimated locations.
- the initial distribution is a uniform distribution.
- the initial distribution is limited to being with in Region Of Interest (ROI).
- ROI Region Of Interest
- the initial distribution is limited to being, for example, within a building, or to being not at inaccessible areas.
- Figure 3B shows particle translation due to total action vector marked as an arrow 305.
- Figure 3C shows a convergence of the particles.
- a process of extracting visual landmarks includes the following two steps: Image processing and Extraction.
- Figures 4A-4B are simplified illustrations of applying a threshold to a color image of ceiling lights according to an example embodiment of the invention.
- Figure 4A shows an example color image which includes lights 305
- Figure 4B shows a resulting, optionally binary, optionally black and white, image of Figure 4A, showing the light 305 after a threshold has been applied.
- a contour y of a landmark that appears in lb see [Can86], [XB12]).
- Figures 4C-4D are simplified illustrations of contour detection of light sources on an image and of determining center mass points according to an example embodiment of the invention.
- Figure 4C shows an example color image which includes lights, showing optional contours 402 of the lights.
- Figure 4D shows locations 404 of centers-of-mass of the lights.
- additional geometric properties of each light source are optionally analyzed, including, for example center, radius and a compact representation of a perimeter (contour) of the light source.
- a first step includes calculating an Intrinsic matrix K3 x 3 of the visual sensor which is known as “Camera Calibration”.
- the intrinsic matrix encapsulates the sensor’s focal length on both axes fx, fy, it’s center c and the sensor’s skewness [ZhaOO], [HS97].
- the relative vector is then: where w is a scale factor of mapping R 2 to R 3 which is unknown. Hence, v E R 3 can be calculated up to a scale.
- one rotates the acquired vector c by the device self-orientation in order to align it with world coordinate system.
- device orientation is extracted from commonly-used MEMS sensors (e. g android smartphone orientation sensor).
- the system described herein optionally implements pedometry (for example using a step-counter) and/or optical flow.
- the rough position estimation is optionally to define a region- of-interest.
- data fusion is addressed by using a (optionally modified) particles filter method.
- a particle filter with the initial area of the filter covering all the evaluation map and converging to the most likely Probability Density Function.
- some particles are periodically spread outside the ROI to overcome ’kidnap robot’ situation.
- the algorithm used is described in [YBM17].
- Algorithm 2 Evaluating high accuracy location
- a real world localization scenario often includes complex sensory data: outliers, inaccurate and partial sensory information are common. Moreover, human factors can influence the sensor measurements and therefore contradict the pure Bayesian inference filter. In this section we present several improvements for the generic particle filter that aim to over- come such problems.
- barometer data is used.
- the barometer data is optionally smoothed, optionally using a Kalman filter, detecting events of elevation change.
- the particles can then be moved in the z axis using the filtered data. Such a method allows one to spread the particles filter over a height of a few floors, for example during an initialization stage.
- “vertical corridors” are optionally defined in the map (stairs, escalators, elevators) through which an elevation change may occur. The fact that the area of such “vertical corridors” is potentially results in a rapid convergence.
- Barometric pressure is sampled at a rate suitable for tracking a specific object.
- sampling may be done every few seconds or even every second.
- sampling may be done every 1 or more minutes. For example, in a vicinity of the“vertical corridors” sampling may be performed more often than when far from the“vertical corridors”.
- the algorithm is optionally kept both simple and efficient by using only a 2.5D representation of the building, for example using a 2D map for each floor and an optional absolute world elevation or relative elevation.
- the soft init is optionally based on discrete analysis of possible locations - for example using lights or signs patterns.
- a naive registration of the largest image- light to each relevant light in the database is performed.
- the naive registration defines a global 3D vector for each such registration. Note that each registration pair allows us to approximate the vector length by matching the size of the light in the image and in the database.
- this new set of particles includes at least one particle which is rather close to the real location - and therefore has a high probability to increase its weight.
- a re-sampling simplifies the process -only a few particles are changed/replaced each re-sample, and the rest of the particles are evaluated and their weight updated accordingly, potentially allowing a shorter convergence time, or a shorter iteration time, and potentially less particles.
- an elevation is added to a particle filter map constraint, potentially enabling more accurate localization, and potentially enabling more rapid convergence, at least in scenarios where elevation is a factor.
- an enriched map is used - for example a map which includes, in addition to structures such as walls and/or doors, also features which are simple to locate, such as lights and/or signs.
- a simplified map is used - for example a map which includes only or mostly features which are simple to locate, such as lights and/or signs.
- the particle filter method includes using an accurate compass input, using one or more of an angular bias and angular drift or change as part of a particle’s state.
- Compass sensor may have an initial bias.
- compass angle is optionally maintained using a gyro sensor or optical-vision-based method, and potentially an angular drift may develop.
- an angle based (bias) shift and an angular drift are computed by the algorithm - just as the location (x, y, z) is computed - as part of the particle state.
- the particle filter method includes using a flexible number of particles, enabling to adapt the number of particles to a navigation scenario. The number of particles is optionally adapted to the probability space represented by the set of the particles.
- additional particles are optionally added.
- the additional particles are optionally added in locations according to sensed data and/or in vicinity of high weight particles.
- the number of particles may be reduced.
- a particle with a lowest weight is optionally tested with respect to a particle with the highest weight, or with respect to a particle with the average weight - if a ratio of their weights is under some threshold value the lowest particle is removed.
- a lowest weight particle is removed from the set of particles.
- the lowest weight particle is optionally replaced with a new particle.
- the new particle is assigned an initial weight of an average particle and/or a 50-th percentile weight.
- two particles replace a particle removed.
- no new particle is added.
- pedometry input optionally includes measuring optical flow and/or distance ranging with suitable sensors, which provides a more precise“action” as used in particle filter terminology.
- pedometry input optionally includes device orientation, which provides a more precise“action” as used in particle filter terminology.
- a few particles are initialized each time the algorithm re samples.
- a few new particles are optionally located at locations where there are no current particles, yet there is a likelihood for a particle to be.
- Some non-limiting examples include: (i) initializing according to a light pattern (ii) initializing according to a mapped land mark such as a sign (iii) initializing in an elevator - based on detecting an altitude change.
- Soft-init potentially overcomes a situation called the“Kidnapped robot problem”, where a localization algorithm loses touch with the ground truth. Adding a few particles in likely places, can lead a particle filter method to converge on those particles, and correct its own deviation from the ground truth.
- Reporting expected error and confidence the genetic particle filter reports some kind of combined position at one or more of the rounds. In some cases both the expected error and the confidence of the positioning are reported, potentially allowing higher level filters to be applied to the reported positions.
- Some embodiments include collecting benchmark data (e.g. environmental map, RF map and visual landmark map). Such collecting is potentially useful for a system to work efficiently and to achieve high accuracy results as well as robustness.
- benchmark data e.g. environmental map, RF map and visual landmark map
- an environmental map data includes a 2D map of a building’s floor (e.g. a mall).
- the map describes possible locations the map includes, such as walls, shops (in case of mall map), doors, steps, elevators area. Data about staircases and elevator area can be used to determine floor changing along with sensor fusion. Wall location descriptions help eliminate false location estimation in two ways: non-realistic positions such as within a wall, and previous estimation cannot be evaluated again if they passed through a wall in the action step of the algorithm.
- Creating the environmental map is optionally a first step in the pre- processing process, it may be performed manually by measuring the environment, by using an existing map, or by collection from users of a localization app which navigate the environment and send data to a data collector, to produce maps and/or update maps - such can be called social mapping.
- a digital map may be created in any convenient form.
- a whole map data is optionally split in a way such that sub-maps each describes a single floor of a multi floor building.
- an additional step of the process is collection of RF data.
- RF data can consist of 3G/4G mobile signals, Bluetooth/BLE beacons, WiFi at 2.4/5.0 Ghz transmission.
- RF data collection may optionally be a semi-autonomous process where raw positioning data measurements collected are saved with respect to their collection elapsed time. Given n RF data measurements that were recorded at time ti along a vector p on an environmental map, the estimated
- Figure 5 is a simplified illustration of a map used in mapping RF and visual data according to an example embodiment of the invention.
- Figure 5 shows a display 502 of an example mapping application, including a map 504, and controls 506 for a user to provide input.
- Figure 5 shoes additional controls 508 typical of a smartphone or a tablet.
- Figure 5 illustrates that just as a smartphone or tablet can be used for localization, they can be used for mapping. A synergy is potentially produced when using a same device or type of device to map a region when a similar device will later be used for localization.
- Figure 5 shows a path 510 along which the mapping device was carried, while mapping visual data and/or RF data.
- the map 504 displays dots 512 where ceiling lights, for example, were identified and mapped as visual landmarks.
- the process of visual landmark mapping resembles the RF mapping task.
- the visual landmarks are then registered (mapped), optionally using real world coordinates, relative to the environmental map. Note that the task of mapping a complex building such as a shopping-mall may include mapping hundreds of light sources, and may be time consuming.
- Figure 6 is a simplified illustration of light source mapping according to an example embodiment of the invention.
- Figure 6 shows a device such as, for example, a smartphone 602, carried along a path 604.
- the smartphone 602 includes a camera 606, which captures images of a light source 608, while also recording time and/or location.
- the location recorded while mapping may optionally be recorded using various means, such as pedometry, odometry, RF mapping, WiFi location, GPS if available, optical flow odometry, Google’s Tango phones, and so on.
- Figure 6 also shows a motion vector 610 indicating the direction the smartphone 602 is moving.
- Figure 6 depicts a semi-automated mapping method that is both simple, efficient and applicable for mobile devices.
- the light source 608 is spotted and tracked along the path 604, the position of the light source 608 is optionally estimated as a weighted average of intersections of the 3D vectors to that light source 608.
- a simple tracking algorithm that compares two consequent frames and looks for a minimal distance change between a previous spotted landmark and landmarks spotted in the new frame. In cases where the minimal distance does not exceed some threshold, a match is declared, otherwise a new landmark Id is assigned.
- a more sophisticated tracking algorithm may include a 2D Kalman’s Filter, (see [BW01], [LWWL10]).
- a direction or vector to the light source 608 is calculated for each image.
- a location of the light source is optionally calculated by calculating an intersection of the vectors to the light source 608. It is noted that the more images are used, the more accurate the location of the light source cn potentially be.
- Figure 7 is a simplified illustration of a screenshot of a navigating application according to an example embodiment of the invention.
- Figure 7 shows a screenshot of an example embodiment of a mapping navigating application.
- images of two detected and tracked lights 702a 702b are shown.
- the two light sources 702a 702b are mapped to map locations 704a 704b with respect to a user position 706, or the mapping device’s position 706.
- a barometer sensor is rather common in many smartphones.
- the current air pressure can be used in order to detect height changes at sub meter accuracy.
- Using a Kalman filter we were able to detect changes in height (i.e., detecting movement from one floor to another).
- convergence and/or accuracy potentially depends on a number of image frames included in a calculation.
- a high frame capture and calculation rate for example 30 frames-per-second (fps)
- algorithm convergence potentially occurs within 2 seconds or less.
- convergence rate potentially depends on a processor speed.
- convergence rate potentially depends on an environment in which the localization is taking place. An environment with many similar feature can take longer than an environment with one or more distinct- from-each-other features.
- Performing data fusion that is, using input from sensors in addition to a camera, potentially increase speed of convergence.
- a high fps (say 30) sensory input convergence may take from 2 to. 60 seconds.
- the time is scenario dependent - in case of detectable signs - a sub- second convergence is possible, while in case of no detectable objects - the convergence may optionally wait for a user movement such as a floor change or simply a walk through a mapped region.
- Figure 8A shows a path as computed by the“Goln” algorithm
- Figure 8 B shows the ground truth
- Figures 8A and 8B present a part of the actual competition evaluation of the”GoIn” system.
- mapping a shopping mall (with three floors and about a hundred stores) required about one hour in which the RF fingerprinting and the light mapping were performed simultaneously. Then, in the localization part those maps were used for positioning (as shown in Figures 9, 10).
- mapping an average size shopping mall can be done in a matter of minutes to hours, depending on the hardware, including testing and map validation.
- the level of accuracy may vary between light based high accuracy (0.5-2 meters) and low accuracy (5-10 meters) when the visual sensor is blocked. Using relative simple image analysis we are able to differ between the high accuracy and the low accuracy cases.
- image analysis typically enables sub-meter accuracy and often enables a sub-feet accuracy using multiple images. It is noted that accuracy is potentially affected by a distance from a feature which is imaged, typically closer enables a better accuracy. In cases of features located far away from the user (say 50 meters) an expected accuracy may be 2-3 meters.
- Figure 9 is a simplified illustration of a screenshot of a navigating application displaying a large area, according to an example embodiment of the invention.
- Figure 9 shows a screen shot from the“Goln” application.
- Each blue dot 902 represents a single light source (a few hundred lights were mapped in each of three floors).
- a red line 904 represents a computed path with a meter accuracy (on average).
- a green circle 906 presents the WiFi location and a red circle 908 represents the light-based position.
- a Google-maps reported position 910 is presented in light blue - with an average error larger than 15 meter.
- Figure 10 is a graph of an accuracy evaluation of a path according to an example embodiment of the invention and a path according to Google map.
- the graph of Figure 10 shows an accuracy evaluation of Goln’s path 1006 (blue) and Google-map’s path 1008 (orange).
- the path 1006 computed by the Goln app was within 1 meter accuracy during at least half of the evaluation, yet: whenever the phone’s camera was blocked the expected accuracy went to 3-6 meters.
- the graph of Figure 10 has an X-axis 1002 in meters, and a Y-axis 1004 in meters.
- the paths 1006 and 1008 present a specific walk in a real mall (PETAH TIKVA) and the locations reported by Google (less accurate) and an example embodiment of the invention - the Goln method (more accurate).
- the example embodiment of the localization method uses light sources as landmarks.
- the example embodiment algorithm was able to improve the accuracy from“room-level” to a’’seat-level” in cases where the visual sensor was able to detect those landmarks.
- the suggested algorithm was implemented as an Android application and tested in several real-life scenarios both for localization and mapping. In general, the accuracy of the localization depends on the ability of the visual sensor to detect visual landmarks (without being blocked by a user’s body).
- Lessons learned include performing a“camera-switch”, in which one camera on one side of a smartphone or tablet is replaced by another camera on another side, while continuing to use the tracking method.
- The“camera-switch” is used for potentially improving visual path tracking (i.e., optical flow or visual pedometry) to allow a smooth and continuous path approximation even in cases of relatively long scenarios of a blocked camera.
- the exemplary positioning embodiments below describe a vision based navigation system designed for indoor localization.
- An example embodiment framework works as a standalone 3D positioning system by fusing a sophisticated optical-flow pedometry with map-constraints using an advanced particle filter.
- An example embodiment method potentially requires no personal calibration and potentially works on standard smart-phones, optionally with a relatively low energy consumption.
- LBS location base services
- first responders first responders
- autonomous robotics indoor navigation
- LBS related applications mainly target smart-phone users navigating in a shopping mall [FNI13], [GLN09]
- first responders may be using a foot mounded pedometer (see [JSPG10], [KR08a], [KR08b]).
- JSPG10 foot mounded pedometer
- the suggested solution should be able to work in an’’off-line” mode (i.e.,“flight- mode” or“standalone” mode).
- This Example presents a smartphone indoor positioning system (I P S) based on software developed using recent AR and MR (Augmented Reality and Mixed Reality) tools such as Google’s ARCore or Apple’s ARKit.
- the AR tools were used to develop visual pedometry (scaled optical flow) sensors, which were fused with an improved version of a localization particle filter to produce an accurate and robust solution for various indoor positioning applications.
- the Example method enables a simple and efficient mapping solution that, combined with the improved version of the localization particle filter, allows 1-2 meter positioning accuracy in most standard indoor scenarios.
- a user global position can be retrieved from existing geolocation services (e.g., Google Maps Geolocation API). Such user location is commonly approximated using RF signals (4G - 3 G, WLAN, BLE ) and even global navigation satellite system (GNSS ). The accuracy of such methods is considered to be“building level” (10-30 meters) or“room level” (5-10 meters).
- geolocation services e.g., Google Maps Geolocation API.
- RF signals 4G - 3 G, WLAN, BLE
- GNSS global navigation satellite system
- the accuracy of such methods is considered to be“building level” (10-30 meters) or“room level” (5-10 meters).
- a user relative position is optionally computed using a pedometer.
- a smartphone-based pedometer is composed of two virtual sensors: (i) A“Step-counter”: which detects discrete step- events. (ii) An orientation sensor: which approximates the user global / relative direction. Combined, the two sensors enable a step-based relative path computation. Naturally such a method tends to drift over time and steps or distance.
- Map The particle filter methods estimate the internal state in a given area.
- an input of this algorithm is a 2D, 2.5D or 3D map of the region.
- Such a map preferably include as many constraints as possible (for example walls and tables).
- the map constraint is one of the parameters that determine each particle’s grade, since particles with an impossible location on the map are usually downgraded.
- Particle At the beginning of the localization process we“spread” a set of particles P on the map.
- Each particle xi E P has one or more of the following attributes: location: ⁇ x, y, z >, orientation: w and grade: g.
- location ⁇ x, y, z >
- orientation w
- grade g.
- all particle locations and orientations are optionally modified, as well as their grades.
- the sum of P particles’ grade is optionally 1 in each step.
- the grade of each particle is optionally set higher when its location on the map seems more likely to represent the internal state.
- Move function (Action function): With each step the particles in the map are relocated according to the internal movement in some embodiments, for each step we calculate a movement vector (optionally in 3D) and a difference in orientation, then move the particles accordingly.
- the movement of each step is optionally provided by a mobile pedometer (for example a step counter, optionally with orientation) as commonly used in smart-phone.
- Sense function Sensors of the device are optionally used to determine each particle grade.
- a sense method or function predicts each particle’s sense for each step, and grades it with respect to a correlation between the particle prediction and the internal sense.
- the sense function can compute distances to a nearest wall (forward and back, right and left) and then compare the computed distance to a distance of each particle to a nearest wall in the map and optionally change the particle grade in an amount corresponding to the correlation.
- Re-sampling A process of choosing a new set of particles P’ from P.
- the re-sampling process can be done by various methods and one purpose of the re-sampling is to choose particles with a high weight or grade over particles with lower weight or grade.
- Random noise To prevent the convergence of the particles from happening too fast (and by that risk missing a true location).
- particles are optionally moved by a small random noise on the map. In some embodiments this is done by moving each particle in a small radius from its original location.
- Algorithm 1 described below explains a process of the particle filter method using a mobile pedometry sensor.
- Input A black and white 2D, 2.5D or 3D map of the navigation area.
- P a random location ⁇ x, y > is set, optionally
- Algorithm 1 is described as:
- Init generate a set P of n particles . For every xi EP a random location ⁇ x, y, z> orientation ⁇ w > and grade g is set, optionally in a uniform distribution over the map.
- a black and white map is used in order to present geo-constraints used by the particle filter.
- the above algorithm is relatively time efficient, however its precision may be insufficient in some cases, for example in large areas or areas with few constraints.
- the next section describes a particle filter based algorithm with advance methods to improve the results accuracy.
- the improved mapping and the adjusted sense function are both possible due to new AR smart phone technology.
- the next subsections explain the improved mapping process and the advanced particle filter algorithm.
- AR augmented reality
- AR algorithms have the following features:
- one or more of the first two features enable estimating a user movement and orientation in real time, optionally even for each step taken by the user, and are optionally used to improve pedometer-sensed data.
- the third feature is optionally exploited to improve the mapping and the particle filter sense function, as will be explain in detail in the following subsections.
- a method for detecting floor change.
- An error of “wrong floor” is significant for a user and may cause significant error related to wrong constraints applied by a“wrong map”.
- one or both of a barometer sensor and a 3D optical-flow were used in order to estimate elevation of a user. Both methods are relatively sensitive to changes in elevation, yet both also tend to drift.
- 3D optical-flow methods are typically not able to detect vertical movement in an elevator, where the change in elevation is not seen.
- the following floor - change filter is used, which is based on rapid changes in barometer readings (for simplicity we assume that the barometer sampling rate is fixed):
- the particles are initially randomly spread among all floors.
- the improved algorithm may optionally fuse the 3D optical-flow sensor reading with the barometer sensor, in some cases using a Kalman filter.
- Figures 13A-13C are simplified illustrations of using a particle filter to estimate a location according to an example embodiment of the invention.
- Figure 13D is a graph showing atmospheric pressure measurements over time of according to an example embodiment of the invention.
- Figures 11A-11C show a map and particles displayed upon the map, at three different stages.
- Figure 11A shows an initial state, where the particles are uniformly distributed.
- Figure 11B shows how, using a short motion vector, the particles begin to organize clusters.
- Figure 11C shows the particles converged to a single position cluster.
- Figure 11D shows a graph having an X-axis of time and a Y-axis of barometric pressure (in PSI).
- Figure 11D shows barometric pressure over time. The detection of floor change enabled the algorithm to converge efficiently to the right 3D location.
- the advanced particle filter algorithm uses a map of the region of interest.
- a map is assembled by an example embodiment system using the following technique:
- the colors are placed on the map.
- one or more of the following constraints are represented according to the following logic:
- color B Inaccessible area, such as walls, fixed barriers, etc. optionally as sensed by the AR tool.
- color C partially accessible area, adjacent to an inaccessible area, optionally having a fixed width.
- color D Inaccessible area, one that could not have been identified by the AR tool due to its lack of vertical-surfaces-shape. This area may optionally be colored manually.
- the generated map is an input for the particle filter algorithm, and is later also used to determine the particles grade.
- Figure 12 is a simplified illustration of a multicolor map used according to an example embodiment of the invention.
- Figure 12 shows a 2D multicolor map example used in the advanced algorithm.
- the Color white (marked by the letter A) represents accessible areas
- the black color (marked as B ) represent fixed inaccessible areas (in this case walls)
- the color pink (marked as C) represents adjacent partially-accessible areas (near walls)
- the brown (marked as D ) represents dynamic inaccessible areas (tables in this case)
- the yellow part (marked as E) represents stairs.
- FIG. 13 is a simplified illustration of a multicolor map used according to an example embodiment of the invention.
- Figure 13 shows a multicolor map example used in the advanced algorithm.
- the color white is (marked by the letter A) represent accessible areas
- the black color (marked as B) represent the fixed areas (in this case walls)
- the grey (marked as C) represents dynamic inaccessible areas (tables in this case)
- the yellow part (marked as D) represent stairs and elevators.
- the map of Figure 13 includes two floors.
- the map of Figure 12 is used as a 2.5D map.
- Such a map can optionally benefit from elevation-related sensors such as barometric pressure or GPS, if available.
- the naive and the advanced particle filter algorithm differ in their sense functions. While the naive algorithm simply evaluates the weight of the particles according to their map location (a particle in a B or A area), the advanced algorithm may optionally perform an actual sensing, to determine how far each particle is from a ground truth.
- the sensing performed by the AR measurement tool measures, for example, a real distance from a nearest vertical obstacle, and compares the sensed distance to a calculated distance of each particle to a nearest B area in the same direction on the map. This comparison provides an ability to re-weight the particles in a more precise way. Note that the existence of the C area provides more flexibility with respect to inaccurate measurements.
- IMU inertial measurement unit
- pedometer which detects the device’ s global orientation and counts’’steps”.
- IMU inertial measurement unit
- Such a method potentially introduces inaccuracy both in the distance measured and in the orientation (i.e., some steps are larger than other, the device orientation is only loosely correlated with the walking orientation).
- velocity estimation is based on optical flow with plan and range detection, such as described in [VDA+18], in order to estimate the user movement, optionally at a high sampling rate.
- VDA+18 optical flow with plan and range detection
- each particle-state may also include additional data, to estimate the compass original bias, and/or current drift.
- each particle starts with an initial, optionally Gaussian, random value of compass bias.
- each new particle is optionally assigned a compass-related state according to values of its nearest neighbors, with some minor noise.
- Each particle optionally uses the smartphone’s compass-measured data combined with its bias and drift for the move function.
- Particle Filter (PF) algorithms [TBF05] rely on an assumption of a continuous flow of incoming data from sensors. However, the sampling process cannot be guaranteed. Many localization problems include sparse sensing scenarios, i.e., scenarios where data from a sensor is obsolete. In context of the current Example, assuming a localization algorithm is vision based, blocking the camera can cause very serious ramifications. Since sampling only works well given the correct weights which are obtained from the real world via the sensors.
- the particle filter when such a scenario is detected, optionally reacts accordingly by adding random noisy movement to one or more of the particles, relative to a previously measured movement or pace. Such a reaction provides more scattered particles that potentially solve a momentary uncertainty.
- an adjustable particle filter is used, that adjusts the number of particles according to a size of an expected probabilistic space.
- a large region of possible solutions e.g., init- stage over a few floors
- a large set of particles may be used, yet later on, when the particle filter tends to converge, the number is optionally reduced - enabling a better practical runtime and/or lower memory usage.
- Kidnapped Robot is a name of a well-known problem [TFBD01], which in context of localization, navigation, tracking, refers to a situation when the algorithm completely loses track of the real world location and the evaluation function performs badly.
- geolocation services e.g., Google Maps Geolocation API
- the geolocation services are used as an anchor to the truth, see Figure 14. In case our system reports an extremely different location from the one the used geolocation service reported, we reboot the system based on the geolocation service report, respectively to the service expected accuracy.
- Figure 14 is a simplified illustration of a map showing a comparison of a location determined by Google’s fused location service and a location method used according to an example embodiment of the invention
- Figure 14 shows Google’s fused location service location marked as a blue dot 1402, and its accuracy is marked with a light blue circle 1402.
- the particle filter location determined by an example localization algorithm used with reference to the exemplary positioning embodiments is marked with a green dot 1406.
- GNSS global system for Mobile communications
- an estimation is performed whether phone has LOS (line of site) to a navigation satellite, see [YMW16] for more details.
- a method is based on the phone’s light sensor, as in daylight the outdoor light is usually stronger than the indoor light (even on cloudy days) while at night-time the opposite happens. Using such a method provides an additional sense evaluation, optionally for use as a constraint and/or particle weight consideration.
- FIGS 15A-15B are simplified illustrations of a potential advantage of using indoor/outdoor determination according to an example embodiment of the invention.
- Figure 15A shows particle initialization marked on a map, without use of indoor / outdoor sensing data.
- Figure 15A shows particles 1502 scattered on a map, within a circle 1504 having a specific initial radius.
- Figure 15B shows that an ability to classify between indoor and outdoor enables an example embodiment algorithm to dramatically decrease an overall area of possible solutions - potentially leading to a faster converges time and/or better accuracy.
- Figure 15B shows particles 1512 scattered on a map, within a circle 1514 having a specific initial radius, but only indoors, with respect to a building 1516 also shown on the map.
- Figure 16 is a simplified illustration of tracking according to an example embodiment of the invention in comparison to a Lidar-based ground truth.
- Figure 16 shows a 2D evaluation of a path 1602 (shown in green) as detected by the STEPS system with respect to the ground truth ( GT) path 1604 (shown in blue). The evaluation was performed at specific points in time, and error lines 1606 are shown in red.
- Figure 17 is a graph showing height value error over time according to an example embodiment of the invention.
- the graph of Figure 17 has an X-axis 1704 of time in arbitrary units and a Y-axis 1702 of relative elevation in meters.
- Figure 17 shows a z-converge process - in which a floor-position, was accurately found after about 40 seconds.
- Figure 17 includes a first line 1707 showing floor position, a second line 1708 showing the ground truth, and a third line 1706 showing the z-error, or difference between estimated elevation shown by the first line 1707 and ground truth shown by the second line 1708.
- Figures 18 and 19, described below, show the different convergence nature of a particle filter regarding a 3D case (when the floor is unknown) and a 2D case (when the floor is given).
- Figure 18 is a graph showing position error over time according to an example embodiment of the invention.
- the graph of Figure 18 has an X-axis 1804 of time in arbitrary units and a Y-axis 1802 of relative position in meters.
- Figure 18 includes a first line 1806 showing X-Y error and a second line 1808 showing Z error.
- Figure 18 shows a test case lasting about 120 seconds.
- the elevation, or Z component of the location converged from a height error of about 4 meters to a sub-meter error.
- the horizontal error also reduced, or converged, over time.
- Figure 19 is a graph showing position error over time according to an example embodiment of the invention.
- the graph of Figure 19 has an X-axis 1904 of time in arbitrary units and a Y-axis 1802 of relative position in meters.
- Figure 19 includes a first line 1906 showing Z error and a second line 1908 showing X-Y error.
- Figure 19 shows Particle Filter 2D convergence in a case where the correct floor, is known. A know value for the Z axis or elevation, is therefore used.
- the second line 1908 shows that the X-Y error reduces from an error of 4.5 meters to about 1.3 meters within 10 seconds (about 15 steps). During the rest of the test The horizontal error is about 1 meter, while the vertical error is (on average) below half a meter.
- An example embodiment implementation of the localization algorithm in phones with a TOF (Time Of Flight) camera differs from the augmented reality based single-camera phones (e.g., Google’s ARC ore), potentially providing increased accuracy and/or convergence time.
- TOF Time Of Flight
- augmented reality based single-camera phones e.g., Google’s ARC ore
- a 3D point cloud potentially enables to implement a high-accuracy localization loop using range-error analysis.
- the TOF cameras typically have a relatively narrow error range - usually smaller than 5cm (see [FPB+ 16]).
- An example embodiment implementation shows an improvement of expected accuracy.
- Such implementation enables to improve the expected accuracy down to 10-20 centimeters.
- a 3D mapping of the region of interest can be performed and the localization particle filter algorithm can work on the 3D map.
- the particle filter localization algorithm was able to maintain a relative 4-12 meter accuracy (7.2 meter on average), see Figures 20, 21.
- the overall evaluation of the example embodiment algorithm provided first place in the competition.
- Figure 20 is a graph showing position error over a path according to an example embodiment of the invention.
- the graph of Figure 20 has an X-axis 2004 of relative position in meters and a Y-axis 2002 of relative position in meters.
- Figure 20 includes green dots 2006 showing ground truth, a blue line 2008 showing a calculated position, as calculated by the example embodiment, and red lines
- Figure 21 is a graph showing position error according to an example embodiment of the invention.
- the graph of Figure 21 has an X-axis 2004 of consecutive waypoints and a Y-axis 2002 of error in meters.
- Figure 21 shows an error for each of the 70 points along the evaluation process of the IPIN 2018 localization competition. At the last few points a large error was reported - due to a significant compass drift most probably generated by working elevator which was part of the evaluation path.
- Fig 21 shows the actual measured accuracy (error) as tested in the IPIN2018 localization competition for the“STEPS” group.
- the graph shows the error (in meters) with respect to the waypoints (there where about 70 such points).
- the exemplary parking embodiments below describe a novel approach to vehicle navigation, even in GNSS-denied environments.
- the approach fuses Dead Reckoning methods obtained from a smartphone and/or a vehicle’s on-board computer, and computes an accurate 2D, 2.5D or 3D position of the vehicle.
- An example embodiment algorithm is based on an advanced version of a particle filter algorithm which uses road-based events such as speed bumps, turns, altitude change and RF signals, optionally in addition to other sensors described herein, in order to estimate the vehicle’s location, potentially in real-time.
- An aspect of some example embodiments includes mapping the environment.
- the present exemplary parking embodiments describe an underground parking lot, but the approach can be applied to other scenarios such as, by way of some non-limiting examples, roads, and more specifically but not exclusively tunnel roads.
- smartphone and phone are used interchangeably with the terms computer such as a vehicle on-board computer, tablet, and similar computing devices.
- GNSS navigation is used everywhere in the vehicle industry. Coupled with Map Matching (MM) and Dead Reckoning (DR) techniques, it apparently enables a fairly accurate vehicle localization on top of mapped roads [1, 8]. However, GNSS-denied environments such as indoors, tunnel roads and parking lots create a real challenge for those navigation algorithms.
- Map Matching Map Matching
- DR Dead Reckoning
- Vision based navigation is yet another, less common approach, due to its complexity.
- light sources can serve as landmarks for IPS as was apparently suggested by [3, 4], who developed a vision-based indoor localization system for mobile robots, utilizing ceiling lamps as landmarks.
- a lion’s share of research in this field is for pedestrian navigation, mainly for shopping malls, since the commercial incentive is clear - Location Based Services (LBS) [2].
- LBS Location Based Services
- UAV Unmanned Aerial Vehicle
- the present description describes automobile 2,5D navigation in GNSS denied environments. However, additional devices can benefit from such navigation.
- a complementary sensor-based mechanism for accurate automobile navigation in underground parking-lots and freeway tunnels is described.
- the system and methods described harnesses, amongst other inputs, road-based events (e.g., crossing a speed-bump) as detected by a phone’s sensors in order to estimate the vehicle’s ego-location.
- the novelty of the system and methods described includes characterization and detection of the road- based events.
- the implementation offers a GNSS -level of accuracy in GNSS-denied environments, a feature unavailable today.
- a desired navigation scenario is as follows: a driver enters a vehicle with her phone and starts driving. The phone should always be able to present an exact location on roads, in tunnels and in parking lots.
- the naive navigation algorithm may rely on at least two information sources: the mobile device’s sensors and the vehicle itself. We start with the latter.
- Modem vehicle on-board computers can produce valuable information in real-time regarding the vehicle’s state.
- the exact absolute speed also known as Speed Over Ground (SoG) value and the wheel orientation.
- SoG Speed Over Ground
- a Course Over Ground (CoG) value is reported by a GNSS receiver and the phone’s absolute orientation is calculated from the IMU sensor.
- the phone orientation relative to the vehicle can be precisely computed. Once this figure is obtained, the vehicle’s orientation can be computed by the phone.
- the vehicle’s trajectory can be coarsely computed. This is called the Dead Reckoning (DR) approach.
- DR Dead Reckoning
- one or more of the following factors are optionally used - a Region Of Interest (ROI) map and data fusion of road-based events.
- ROI Region Of Interest
- map obtainment or mapping aspect of the exemplary parking embodiments.
- the data fusion is performed by a probabilistic particle filter algorithm.
- a probabilistic particle filter algorithm We start by describing the road-based events and their detection:
- Speed bumps Nearly all parking lots have speed bumps installed on them (see description of Figure 23 below).
- the speed bumps provide at least two characterizations: first, they are relativity easy to map (only a few bumps on each floor) and, second, they can be detected by sensing the accelerometer value.
- the accelerometer measurements may not be distinguishable between two bumps, so a ROI map is used to compare location of a bump and a sensory (acceleration) indication of a bump.
- Figure 22 demonstrates an accelerometer graph of a vehicle oven a speed bump.
- Figure 22 is a graph showing linear acceleration as captured by a smartphone positioned in a car according to an example embodiment of the invention.
- the graph of Figure 22 has an X-axis 2204 of time in arbitrary units and a Y-axis 2202 of time.
- Figure 19 includes a first red line 2206 showing X-axis acceleration, a second green line 2207 showing Y-axis acceleration, and a third blue line 2208 showing Z-axis acceleration.
- Figure 22 shows linear acceleration in three axes as captured by a smartphone positioned in a car which passed over a speed-bump. It is noted that detecting a speed bump in a vicinity where a speed bump is expected may optionally be done using less acceleration measurement, for example in only one or two dimensions, along one or two axes.
- An event of losing or retrieving a GNSS signal A features of indoor navigation is a lack of GNSS signals. From a different perspective, one can get input and information from event of losing or retrieving the GNSS signal. Losing a GNSS signal is also called a GNSS“lost fix” event. Entering a parking-lot is usually accompanied by a sharp GNSS signal degradation or loss. The parking-lot entrance position can be deduced from the GNSS position a just before the signal degradation event. A similar method can also be used when exiting the parking lot. The“fix retrieval” event potentially indicates a location of the exit.
- a useful and potentially accurate differential sensors is a barometer, also available in many contemporary smartphones.
- differential one means that an absolute height is not necessary to extract from a barometer.
- a floor shift is easy to spot as a change in atmospheric pressure.
- the floor of a parking lot can be determined. This is also true for determining levels halfway between floors.
- a road turn can be detected from the phone’s gyroscope sensor. Many parking-lots have spiral turns, mostly between floors. Those spirals are unmistakable for the gyroscope sensor.
- Leaving the vehicle event implies the vehicle is at a parking spot. This event can be deduced both by a phone and by the vehicle from loss of Bluetooth connectivity with the vehicle.
- a road facing camera even a low-resolution camera, can produce valuable information regarding position and orientation of visual landmarks such as signs, pillars, colored pillars, coded pillars and such landmarks.
- visual landmarks such as signs, pillars, colored pillars, coded pillars and such landmarks.
- two consecutive similar- looking pillars may be distinguished, for example by pillar numbers, colors and similar markings often used in parking lots.
- Position of the pillars is optionally determined, optionally in global coordinates, and a vehicle’s position can be computed with a high accuracy level, potentially limited only by the map accuracy.
- Figure 23 is a color image of a contemporary parking-lot with speed bumps, color column markings, and location codes.
- Figure 23 is a picture was taken at a typical parking lot in Israel.
- Figure 23 shows a speed bump 2302; a first column 2304 painted green marking a“green” portion of the parking lot; a second column 2306 painted red marking a“red” portion of the parking lot; markings 2308 on the columns - including unique identification codes on each column; pavement markings 2310; lights 2314 and signs 2311.
- Figure 23 also shows green rectangles 2312 displaying where a vision system detected the speed bumps 2302.
- the road-based events are fused using a particle filter to get an accurate position.
- the Particle Filter is a member of the non-parametric multi-modal Bayesian filter family.
- a PF estimates the posterior by a finite number of parameters also called particles.
- Each particle is represented by a belief function bel(x t ). This belief function serves as a weight
- the sense function, Zi.- t is n °t necessarily periodic - road-based events are discrete events that change the probability space. Taking a speed-bump event for example. As demonstrated in Figure 22, the event can be detected with high certainty. This means that the vehicle is located in the vicinity of one of the speed bumps. In other words, all the other guesses (particles) are optionally eliminated, and/or their likelihood is diminished. Usually the detection is not absolute and may be only probabilistic, since there may be two or more speed bump candidates. In some embodiments the algorithm does not eliminate all the non- speed-bump-location particles and preserves a small portion of such particles. This approach can also solve the“kidnapped robot” problem.
- An aspect of some embodiments includes mapping a parking lot. This section addresses a map obtainment problem; how can a parking-lot map be constructed? How can a multi-floor parking-lot map be constructed?
- Some parking-lots provide a detailed map along with an exact scale. Such a map is shown in Figure 24.
- Figure 24 is an example parking lot map.
- the map of Figure 24 shows detail down to a level of an individual parking stall, and may be geometrically accurate to at least that level, that is, ⁇ 1-2 meters.
- mapping process is optionally applied.
- a mapping process is optionally used to update a parking lot map (or other environment map).
- speed bumps in a parking lot may be moved around, or added, signs may be out up, taken down, changed, and so on.
- the mapping process optionally updates an electronic map.
- mapping process should be as simple and efficient as possible.
- mapping process a Simultaneous Localization And Mapping (SLAM [6]) algorithm where a portion of the users (drivers) also function as mappers.
- SLAM [6] Simultaneous Localization And Mapping
- a refined, more accurate trajectory can optionally be computed, using the detected road-based events as geographically fixed markers, optionally for fine alignment between various trajectories.
- a 2.5D Parking-Map The 3D path was used in order to construct a separate map for each floor. The road-based events were positioned in a corresponding floor-map.
- An advanced mapping algorithm optionally uses a computer vision algorithm.
- Google published its Vision API.
- the API apparently enables a user to get an image description, optionally in real time, using a pre-trained Google Artificial Neural Network (ANN).
- ANN Google Artificial Neural Network
- an“Exit” sign can be detected utilizing this framework.
- the tested parking-lot included three main floors with approximately 3,000 parking spots, three main entrances and a relatively complicated subdivision into about a dozen regions, see Figure 23.
- the tested parking lot has a complex shape and includes sub-floors, intermediate floors with a 1 meter height difference from main floors.
- the evaluating process was started with a 1 hour preliminary mapping stage.
- the preliminary stage we performed a 20 minute drive in the parking-lot while logging the car speed (via OBDII protocol), orientation and sensory data available on a standard mobile phone (including barometric pressure, Gyro, Accelerometer, Magnetometer).
- the mobile phone was positioned in a phone holder attached to the car, and during the drive a low resolution video (QVGA) was captured at a rate of 30 fps.
- QVGA low resolution video
- a full mapping stage lasted about three hours.
- We used the Parking-Map for the particle filter - enabling to report a 3D car position in real-time with a horizontal accuracy of 3-6 meters, with a sub-meter vertical error (floor detection was 100% accurate).
- the mapping results can be seen in Figures 25A-25C.
- FIGS. 25A-25C are screen capture illustrations of a mapping tool implemented on a smartphone according to an example embodiment of the invention.
- Figures 25A-25C show an example embodiment of a display 2502 in a mapping application.
- the display 2502 includes a top portion where paths 2504a 2504b 2504c (X-Y trajectories) are displayed and optional images of a compass 2506 showing compass directions; a middle portion where a graph shows paths 2508a 2508b 2508c showing elevation during the travel (Z-trajectory); and a bottom portion displaying application controls 2510.
- Figures 25A-25C show a mapping application based on an android phone with ARCoreSDK.
- the mapping application includes a vision-based mapping tool which potentially enables us a sub 1% error in mapping without even without performing corrections based on minimizing errors whenever a path forms a closed loop.
- localization errors are further minimized, by distributing errors when a path closes a loop.
- the example embodiment tool enables to perform a rapid mapping by simply driving through the parking lot.
- a dataset which includes the path is sent to a central server, which, by receiving more than one such dataset for a specific environment, optionally produces an improved map by averaging the data.
- the preliminary implementation includes a large set of parameters defining the“sense- weight” for the road-base events (e.g., what values defines a speed-bump?).
- the application appears to be robust, simple to use, and suitable for a wide range of scenarios.
- the coloring and numbering may appear to obviate the need for the location algorithm described here. Nevertheless, even colored and numbered parking lots create confusion. Moreover, the very same method can be generalized to other scenarios.
- Figure 26 shows a freeway tunnel, including light sources 2602 and signs 2604a 2604b.
- One of the signs 2604b displays a code number.
- the coded sign 2604b is, shown in the photograph marked by a red circle 2606.
- the navigation method described herein can be used as an aided lane- detection algorithm, which operates both where GNSS signals are available and where GNSS signals are available, and transitions between the GNSS-available and GNSS -unavailable areas.
- Many highways have light poles along the sides, sometimes on both sides.
- a center-of-mass of each light source is optionally detected using a simple brightness threshold as shown in Figures 27 A and 27B.
- Figures 27A and 27B are images of a highway according to an example embodiment of the invention.
- Figure 27B shows a color photograph of a road during darkness, when road lights 2702 are lit.
- Figure 27A shows the photograph of Figure 24B after applying a brightness threshold operation, in black and white.
- Figure 24A shows the road lights 2702 of Figure 27B as white spots 2704.
- compositions, method or structure may include additional ingredients, steps and/or parts, but only if the additional ingredients, steps and/or parts do not materially alter the basic and novel characteristics of the claimed composition, method or structure.
- the singular form“a”,“an” and“the” include plural references unless the context clearly dictates otherwise.
- the term“a unit” or“at least one unit” may include a plurality of units, including combinations thereof.
- range format is merely for convenience and brevity and should not be construed as an inflexible limitation on the scope of the invention. Accordingly, the description of a range should be considered to have specifically disclosed all the possible sub-ranges as well as individual numerical values within that range. For example, description of a range such as from 1 to 6 should be considered to have specifically disclosed sub-ranges such as from 1 to 3, from 1 to 4, from 1 to 5, from 2 to 4, from 2 to 6, from 3 to 6 etc., as well as individual numbers within that range, for example, 1, 2, 3, 4, 5, and 6. This applies regardless of the breadth of the range.
- a numerical range is indicated herein (for example“10-15”,“10 to 15”, or any pair of numbers linked by these another such range indication), it is meant to include any number (fractional or integral) within the indicated range limits, including the range limits, unless the context clearly dictates otherwise.
- the phrases“range/ranging/ranges between” a first indicate number and a second indicate number and“range/ranging/ranges from” a first indicate number “to”,“up to”,“until” or“through” (or another such range-indicating term) a second indicate number are used herein interchangeably and are meant to include the first and second indicated numbers and all the fractional and integral numbers therebetween.
Landscapes
- Engineering & Computer Science (AREA)
- Radar, Positioning & Navigation (AREA)
- Remote Sensing (AREA)
- Automation & Control Theory (AREA)
- Physics & Mathematics (AREA)
- General Physics & Mathematics (AREA)
- Navigation (AREA)
Abstract
Un procédé de localisation consiste à obtenir une carte d'une région d'intérêt (ROI), à obtenir une première entrée à partir d'un premier capteur et une seconde entrée à partir d'un second capteur, à fournir la première entrée et la seconde entrée à un processeur, à utiliser le processeur pour estimer un emplacement sur la base de la première entrée et de la seconde entrée, le processeur utilisant un procédé de filtre à particules pour estimer l'emplacement. L'invention concerne également un procédé de mappage d'une région d'intérêt, le procédé consistant à obtenir des premières données de capteur à partir d'un premier capteur et des secondes données de capteur à partir d'un second capteur, à fournir les premières données de capteur et les secondes données de capteur à un processeur, à utiliser le processeur pour estimer un emplacement sur la base des premières données de capteur et des secondes données de capteur, et à envoyer l'emplacement à une application de mappage. L'invention concerne aussi un appareil et des procédés associés.
Applications Claiming Priority (6)
Application Number | Priority Date | Filing Date | Title |
---|---|---|---|
US201862690953P | 2018-06-28 | 2018-06-28 | |
US201862690955P | 2018-06-28 | 2018-06-28 | |
US201862690958P | 2018-06-28 | 2018-06-28 | |
US62/690,953 | 2018-06-28 | ||
US62/690,958 | 2018-06-28 | ||
US62/690,955 | 2018-06-28 |
Publications (1)
Publication Number | Publication Date |
---|---|
WO2020003319A1 true WO2020003319A1 (fr) | 2020-01-02 |
Family
ID=68984715
Family Applications (1)
Application Number | Title | Priority Date | Filing Date |
---|---|---|---|
PCT/IL2019/050718 WO2020003319A1 (fr) | 2018-06-28 | 2019-06-28 | Techniques de localisation |
Country Status (1)
Country | Link |
---|---|
WO (1) | WO2020003319A1 (fr) |
Cited By (5)
Publication number | Priority date | Publication date | Assignee | Title |
---|---|---|---|---|
CN112762928A (zh) * | 2020-12-23 | 2021-05-07 | 重庆邮电大学 | 含有激光slam的odom与dm地标组合移动机器人及导航方法 |
DE102021116510A1 (de) | 2021-06-25 | 2022-12-29 | Cariad Se | Verfahren und Rechenvorrichtung zum Bereitstellen einer Wegnetzkarte eines Parkhauses |
US20230100851A1 (en) * | 2021-09-28 | 2023-03-30 | Here Global B.V. | Method, apparatus, and system for mapping a parking facility without location sensor data |
CN115965682A (zh) * | 2022-12-16 | 2023-04-14 | 镁佳(北京)科技有限公司 | 一种车辆可通行区域确定方法、装置及计算机设备 |
WO2024137372A1 (fr) * | 2022-12-20 | 2024-06-27 | Zebra Technologies Corporation | Systèmes et procédés de localisation |
Citations (2)
Publication number | Priority date | Publication date | Assignee | Title |
---|---|---|---|---|
US20130238236A1 (en) * | 2012-03-12 | 2013-09-12 | Google Inc. | Location correction |
WO2014020547A1 (fr) * | 2012-07-31 | 2014-02-06 | Indoorgo Navigation Systems Ltd. | Procédé et dispositif de navigation |
-
2019
- 2019-06-28 WO PCT/IL2019/050718 patent/WO2020003319A1/fr active Application Filing
Patent Citations (2)
Publication number | Priority date | Publication date | Assignee | Title |
---|---|---|---|---|
US20130238236A1 (en) * | 2012-03-12 | 2013-09-12 | Google Inc. | Location correction |
WO2014020547A1 (fr) * | 2012-07-31 | 2014-02-06 | Indoorgo Navigation Systems Ltd. | Procédé et dispositif de navigation |
Non-Patent Citations (3)
Title |
---|
BOAZ BEN MOSHE ET AL.: "Advanced particle filter methods.", HEURISTICS AND HYPER-HEURISTICS-PRINCIPLES AND APPLICATIONS, 31 December 2017 (2017-12-31), XP055648201, DOI: 10.5772/intechopen.69236 * |
BOAZ BEN MOSHE ET AL.: "GoIn-an accurate indoor navigation framework for mobile devices", MICROSOFT INDOOR LOCALIZATION COMPETITION, TECH. REP., 31 December 2017 (2017-12-31) * |
LANDA, VLAD ET AL.: "GoIn-An Accurate 3D InDoor Navigation Framework for Mobile Devices", 27 September 2018 (2018-09-27), pages 1 - 8, XP033447082, DOI: 10.1109/IPIN.2018.8533810 * |
Cited By (7)
Publication number | Priority date | Publication date | Assignee | Title |
---|---|---|---|---|
CN112762928A (zh) * | 2020-12-23 | 2021-05-07 | 重庆邮电大学 | 含有激光slam的odom与dm地标组合移动机器人及导航方法 |
DE102021116510A1 (de) | 2021-06-25 | 2022-12-29 | Cariad Se | Verfahren und Rechenvorrichtung zum Bereitstellen einer Wegnetzkarte eines Parkhauses |
DE102021116510B4 (de) | 2021-06-25 | 2024-05-08 | Cariad Se | Verfahren und Rechenvorrichtung zum Bereitstellen einer Wegnetzkarte eines Parkhauses |
US20230100851A1 (en) * | 2021-09-28 | 2023-03-30 | Here Global B.V. | Method, apparatus, and system for mapping a parking facility without location sensor data |
CN115965682A (zh) * | 2022-12-16 | 2023-04-14 | 镁佳(北京)科技有限公司 | 一种车辆可通行区域确定方法、装置及计算机设备 |
CN115965682B (zh) * | 2022-12-16 | 2023-09-01 | 镁佳(北京)科技有限公司 | 一种车辆可通行区域确定方法、装置及计算机设备 |
WO2024137372A1 (fr) * | 2022-12-20 | 2024-06-27 | Zebra Technologies Corporation | Systèmes et procédés de localisation |
Similar Documents
Publication | Publication Date | Title |
---|---|---|
US11959771B2 (en) | Creation and use of enhanced maps | |
WO2020003319A1 (fr) | Techniques de localisation | |
EP3570061B1 (fr) | Localisation de drones | |
KR102399591B1 (ko) | 관심 지역 및 입구의 위치를 결정하는 시스템 | |
US10281279B2 (en) | Method and system for global shape matching a trajectory | |
US9146113B1 (en) | System and method for localizing a trackee at a location and mapping the location using transitions | |
US10621861B2 (en) | Method and system for creating a lane-accurate occupancy grid map for lanes | |
US9448072B2 (en) | System and method for locating, tracking, and/or monitoring the status of personnel and/or assets both indoors and outdoors | |
US20170219359A1 (en) | Method and system for estimating uncertainty for offline map information aided enhanced portable navigation | |
US10444019B2 (en) | Generating map data | |
CN107110651A (zh) | 用于使用地图信息辅助的增强型便携式导航的方法和装置 | |
CN105378431A (zh) | 利用磁场异常进行户内定位-寻找 | |
EP3848674B1 (fr) | Signalisation d'emplacement par rapport à un véhicule autonome et un passager | |
AU2014277724B2 (en) | Locating, tracking, and/or monitoring personnel and/or assets both indoors and outdoors | |
CN117203492A (zh) | 地图匹配轨迹 | |
KR101523147B1 (ko) | 실내 측위 장치 및 방법 | |
Landa et al. | GoIn-An Accurate 3D InDoor Navigation Framework for Mobile Devices | |
JP6810723B2 (ja) | 情報処理装置、情報処理方法、およびプログラム | |
Alaoui et al. | Points of interest detection for map-aided PDR in combined outdoor-indoor spaces | |
Davidson | Algorithms for autonomous personal navigation systems | |
JP2020085783A (ja) | 歩行者用測位装置、歩行者用測位システム及び歩行者用測位方法 | |
US20240271938A1 (en) | Smartphone-based inertial odometry | |
Vourgidis et al. | A Prediction-Communication P2V Framework for Enhancing Vulnerable Road Users’ Safety | |
Wang | Indoor navigation for passengers in underground transit stations using smartphones |
Legal Events
Date | Code | Title | Description |
---|---|---|---|
121 | Ep: the epo has been informed by wipo that ep was designated in this application |
Ref document number: 19826352 Country of ref document: EP Kind code of ref document: A1 |
|
NENP | Non-entry into the national phase |
Ref country code: DE |
|
122 | Ep: pct application non-entry in european phase |
Ref document number: 19826352 Country of ref document: EP Kind code of ref document: A1 |