SG11201908974XA - Real time robust localization via visual inertial odometry - Google Patents

Real time robust localization via visual inertial odometry

Info

Publication number
SG11201908974XA
SG11201908974XA SG11201908974XA SG11201908974XA SG 11201908974X A SG11201908974X A SG 11201908974XA SG 11201908974X A SG11201908974X A SG 11201908974XA SG 11201908974X A SG11201908974X A SG 11201908974XA
Authority
SG
Singapore
Prior art keywords
frame
singapore
connexis
linear velocity
kth
Prior art date
Application number
Inventor
Zhengguo Li
Jinghong Zheng
Zijian Orson Zhu
Qinyuan Ren
Kong Wah Wan
Wei Yun Yau
Original Assignee
Agency Science Tech & Res
Priority date (The priority date is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the date listed.)
Filing date
Publication date
Application filed by Agency Science Tech & Res filed Critical Agency Science Tech & Res
Publication of SG11201908974XA publication Critical patent/SG11201908974XA/en

Links

Classifications

    • G—PHYSICS
    • G01—MEASURING; TESTING
    • G01C—MEASURING DISTANCES, LEVELS OR BEARINGS; SURVEYING; NAVIGATION; GYROSCOPIC INSTRUMENTS; PHOTOGRAMMETRY OR VIDEOGRAMMETRY
    • G01C21/00—Navigation; Navigational instruments not provided for in groups G01C1/00 - G01C19/00
    • G01C21/10—Navigation; Navigational instruments not provided for in groups G01C1/00 - G01C19/00 by using measurements of speed or acceleration
    • G01C21/12—Navigation; Navigational instruments not provided for in groups G01C1/00 - G01C19/00 by using measurements of speed or acceleration executed aboard the object being navigated; Dead reckoning
    • G01C21/16—Navigation; Navigational instruments not provided for in groups G01C1/00 - G01C19/00 by using measurements of speed or acceleration executed aboard the object being navigated; Dead reckoning by integrating acceleration or speed, i.e. inertial navigation
    • G01C21/165—Navigation; Navigational instruments not provided for in groups G01C1/00 - G01C19/00 by using measurements of speed or acceleration executed aboard the object being navigated; Dead reckoning by integrating acceleration or speed, i.e. inertial navigation combined with non-inertial navigation instruments
    • G01C21/1656—Navigation; Navigational instruments not provided for in groups G01C1/00 - G01C19/00 by using measurements of speed or acceleration executed aboard the object being navigated; Dead reckoning by integrating acceleration or speed, i.e. inertial navigation combined with non-inertial navigation instruments with passive imaging devices, e.g. cameras
    • G—PHYSICS
    • G01—MEASURING; TESTING
    • G01C—MEASURING DISTANCES, LEVELS OR BEARINGS; SURVEYING; NAVIGATION; GYROSCOPIC INSTRUMENTS; PHOTOGRAMMETRY OR VIDEOGRAMMETRY
    • G01C21/00—Navigation; Navigational instruments not provided for in groups G01C1/00 - G01C19/00
    • G01C21/10—Navigation; Navigational instruments not provided for in groups G01C1/00 - G01C19/00 by using measurements of speed or acceleration
    • G01C21/12—Navigation; Navigational instruments not provided for in groups G01C1/00 - G01C19/00 by using measurements of speed or acceleration executed aboard the object being navigated; Dead reckoning
    • G01C21/16—Navigation; Navigational instruments not provided for in groups G01C1/00 - G01C19/00 by using measurements of speed or acceleration executed aboard the object being navigated; Dead reckoning by integrating acceleration or speed, i.e. inertial navigation
    • G01C21/183—Compensation of inertial measurements, e.g. for temperature effects
    • G01C21/188—Compensation of inertial measurements, e.g. for temperature effects for accumulated errors, e.g. by coupling inertial systems with absolute positioning systems
    • G—PHYSICS
    • G05—CONTROLLING; REGULATING
    • G05D—SYSTEMS FOR CONTROLLING OR REGULATING NON-ELECTRIC VARIABLES
    • G05D1/00—Control of position, course, altitude or attitude of land, water, air or space vehicles, e.g. using automatic pilots
    • G05D1/02—Control of position or course in two dimensions
    • G05D1/021—Control of position or course in two dimensions specially adapted to land vehicles
    • G05D1/0231—Control of position or course in two dimensions specially adapted to land vehicles using optical position detecting means
    • G05D1/0246—Control of position or course in two dimensions specially adapted to land vehicles using optical position detecting means using a video camera in combination with image processing means
    • G05D1/0253—Control of position or course in two dimensions specially adapted to land vehicles using optical position detecting means using a video camera in combination with image processing means extracting relative motion information from a plurality of images taken successively, e.g. visual odometry, optical flow
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
    • G06T7/00—Image analysis
    • G06T7/20—Analysis of motion
    • G06T7/246—Analysis of motion using feature-based methods, e.g. the tracking of corners or segments
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
    • G06T7/00—Image analysis
    • G06T7/70—Determining position or orientation of objects or cameras
    • G06T7/73—Determining position or orientation of objects or cameras using feature-based methods
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
    • G06T2207/00—Indexing scheme for image analysis or image enhancement
    • G06T2207/30—Subject of image; Context of image processing
    • G06T2207/30244—Camera pose

Landscapes

  • Engineering & Computer Science (AREA)
  • Radar, Positioning & Navigation (AREA)
  • Remote Sensing (AREA)
  • Physics & Mathematics (AREA)
  • General Physics & Mathematics (AREA)
  • Computer Vision & Pattern Recognition (AREA)
  • Automation & Control Theory (AREA)
  • Theoretical Computer Science (AREA)
  • Multimedia (AREA)
  • Electromagnetism (AREA)
  • Aviation & Aerospace Engineering (AREA)
  • Image Analysis (AREA)
  • Studio Devices (AREA)
  • Devices For Executing Special Programs (AREA)

Abstract

Refinethe angular velocity and linear velocity of the kth frame using the k-lth frame and kth frame The linear velocity of the kth frame PUBLISHED UNDER THE PATENT COOPERATION TREATY (PCT) I iiimmommiononolomoniolliolommovois (10) International Publication Number WO 2018/182524 Al WIPO I PCT (12) INTERNATIONAL APPLICATION (19) World Intellectual Property Organization International Bureau (43) International Publication Date 04 October 2018 (04.10.2018) (51) International Patent Classification: G06T 7/73 (2017.01) G06T 7/246 (2017.01) GO1C 21/16 (2006.01) (21) International Application Number: PCT/SG2018/050153 (22) International Filing Date: 29 March 2018 (29.03.2018) (25) Filing Language: English (26) Publication Language: English (30) Priority Data: 10201702561Y 29 March 2017 (29.03.2017) SG (71) Applicant: AGENCY FOR SCIENCE, TECHNOLO- GY AND RESEARCH [SG/SG]; 1 Fusionopolis Way, #20-10 Connexis North Tower, Singapore 138632 (SG). (72) Inventors: LI, Zhengguo; c/o PKM, Institute for Info- comm Research, 1 Fusionopolis Way, #21-01 Connexis South Tower, Singapore 138632 (SG). ZHENG, Jinghong; c/o PKM, Institute for Infocomm Research, 1 Fusionopo- lis Way, #21-01 Connexis South Tower, Singapore 138632 (SG). ZHU, Zijian Orson; c/o PKM, Institute for Info- comm Research, 1 Fusionopolis Way, #21-01 Connexis South Tower, Singapore 138632 (SG). REN, Qinyuan; c/ o PKM, Institute for Infocomm Research, 1 Fusionopolis Way, #21-01 Connexis South Tower, Singapore 138632 (SG). WAN, Kong Wah; c/o PKM, Institute for Info- comm Research, 1 Fusionopolis Way, #21-01 Connexis South Tower, Singapore 138632 (SG). YAU, Wei Yun; c/o PKM, Institute for Infocomm Research, 1 Fusionopo- lis Way, #21-01 Connexis South Tower, Singapore 138632 (SG). (54) Title: REAL TIME ROBUST LOCALIZATION VIA VISUAL INERTIAL ODOMETRY 100 102 The Dose Of the k Fuse information of the k-ith frame via an adaptive EKF The linear velocity, rotation matrix and height of the k-/)th frame 104 r' 120 The angular velocity and IMU Estimate the inear velocity of the kth frame Ducar acceleration o the kth frame The angular velocity and linear velocity (tithe ic01 frame 108 r T Estimate the initial I pose of the kth frame The angular velocity o the kth frame FIG. 1 122 / Height The lei ht of the L sensor kill frame The information of the kth frame 110 Reduce drift of the kth rame using the nearest key frame The rotation matrix and the translation vector of the kth frame 112 ,-, The linear velocity, rotation Fuse inform ion of the Mb matrix, translation vector and frame via an adaptive EKF height of the kth frame (57) : A camera-based localization system is provided. The camera-based localization system may assist an unmanned vehicle to continue its operation in a GPS-denied environment with minimal increase in vehicular cost and payload. In one aspect, a method, a computer-readable medium, and an apparatus for localization via visual inertial odometry are provided. The apparatus may construct an optical flow based on feature points across a first video frame and a second video frame captured by a camera of the apparatus. The 00 apparatus may refine the angular velocity and the linear velocity corresponding to the second video frame via solving a quadratic opti- mization problem constructed based on the optical flow, the initial values of the angular velocity and the linear velocity corresponding 00 to the second video frame. The apparatus may estimate the pose of the apparatus based on the refined angular velocity and the refined O linear velocity. O [Continued on next page] WO 2018/182524 Al MIDEDIMOMOIDEIREEM3111011110101iMOIS (74) Agent: VIERING, JENTSCHURA & PARTNER LLP; P.O. Box 1088, Rochor Post Office, Rochor Road, Singa- pore 911833 (SG). (81) Designated States (unless otherwise indicated, for every kind of national protection available): AE, AG, AL, AM, AO, AT, AU, AZ, BA, BB, BG, BH, BN, BR, BW, BY, BZ, CA, CH, CL, CN, CO, CR, CU, CZ, DE, DJ, DK, DM, DO, DZ, EC, EE, EG, ES, FI, GB, GD, GE, GH, GM, GT, HN, HR, HU, ID, IL, IN, IR, IS, JO, JP, KE, KG, KH, KN, KP, KR, KW, KZ, LA, LC, LK, LR, LS, LU, LY, MA, MD, ME, MG, MK, MN, MW, MX, MY, MZ, NA, NG, NI, NO, NZ, OM, PA, PE, PG, PH, PL, PT, QA, RO, RS, RU, RW, SA, SC, SD, SE, SG, SK, SL, SM, ST, SV, SY, TH, TJ, TM, TN, TR, TT, TZ, UA, UG, US, UZ, VC, VN, ZA, ZM, ZW. (84) Designated States (unless otherwise indicated, for every kind of regional protection available): ARIPO (BW, GH, GM, KE, LR, LS, MW, MZ, NA, RW, SD, SL, ST, SZ, TZ, UG, ZM, ZW), Eurasian (AM, AZ, BY, KG, KZ, RU, TJ, TM), European (AL, AT, BE, BG, CH, CY, CZ, DE, DK, EE, ES, FI, FR, GB, GR, HR, HU, IE, IS, IT, LT, LU, LV, MC, MK, MT, NL, NO, PL, PT, RO, RS, SE, SI, SK, SM, TR), OAPI (BF, BJ, CF, CG, CI, CM, GA, GN, GQ, GW, KM, ML, MR, NE, SN, TD, TG). Published: — with international search report (Art. 21(3))
SG11201908974X 2017-03-29 2018-03-29 Real time robust localization via visual inertial odometry SG11201908974XA (en)

Applications Claiming Priority (2)

Application Number Priority Date Filing Date Title
SG10201702561Y 2017-03-29
PCT/SG2018/050153 WO2018182524A1 (en) 2017-03-29 2018-03-29 Real time robust localization via visual inertial odometry

Publications (1)

Publication Number Publication Date
SG11201908974XA true SG11201908974XA (en) 2019-10-30

Family

ID=63677524

Family Applications (2)

Application Number Title Priority Date Filing Date
SG10202110833PA SG10202110833PA (en) 2017-03-29 2018-03-29 Real time robust localization via visual inertial odometry
SG11201908974X SG11201908974XA (en) 2017-03-29 2018-03-29 Real time robust localization via visual inertial odometry

Family Applications Before (1)

Application Number Title Priority Date Filing Date
SG10202110833PA SG10202110833PA (en) 2017-03-29 2018-03-29 Real time robust localization via visual inertial odometry

Country Status (3)

Country Link
US (1) US11747144B2 (en)
SG (2) SG10202110833PA (en)
WO (1) WO2018182524A1 (en)

Cited By (1)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN111105460A (en) * 2019-12-26 2020-05-05 电子科技大学 RGB-D camera pose estimation method for indoor scene three-dimensional reconstruction

Families Citing this family (11)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN109523589B (en) * 2018-11-13 2021-06-08 浙江工业大学 Design method of more robust visual odometer
US20200226787A1 (en) * 2019-01-14 2020-07-16 Sony Corporation Information processing apparatus, information processing method, and program
CN110260861B (en) * 2019-06-13 2021-07-27 北京华捷艾米科技有限公司 Pose determination method and device and odometer
CN110288702B (en) * 2019-06-28 2021-05-18 联想(北京)有限公司 Data processing method and electronic equipment
WO2021081707A1 (en) * 2019-10-28 2021-05-06 深圳市大疆创新科技有限公司 Data processing method and apparatus, movable platform and computer-readable storage medium
US11173893B2 (en) * 2019-11-14 2021-11-16 Wipro Limited Method and system for detecting and compensating for mechanical fault in autonomous ground vehicle
CN111539982B (en) * 2020-04-17 2023-09-15 北京维盛泰科科技有限公司 Visual inertial navigation initialization method based on nonlinear optimization in mobile platform
CN111609868A (en) * 2020-05-29 2020-09-01 电子科技大学 Visual inertial odometer method based on improved optical flow method
CN111811501B (en) * 2020-06-28 2022-03-08 鹏城实验室 Trunk feature-based unmanned aerial vehicle positioning method, unmanned aerial vehicle and storage medium
EP4295207A1 (en) * 2021-02-17 2023-12-27 Alauda Aeronautics Pty Ltd Manned vertical take-off and landing aerial vehicle navigation
CN116940969A (en) * 2021-03-02 2023-10-24 创峰科技 Visual inertial SLAM system

Family Cites Families (50)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
US4442491A (en) * 1981-06-23 1984-04-10 General Dynamics Corporation Training evaluation process
US5672872A (en) * 1996-03-19 1997-09-30 Hughes Electronics FLIR boresight alignment
US5774832A (en) * 1996-04-19 1998-06-30 Honeywell Inc. Inertial navigation with gravity deflection compensation
US20060235610A1 (en) * 2005-04-14 2006-10-19 Honeywell International Inc. Map-based trajectory generation
US7536030B2 (en) * 2005-11-30 2009-05-19 Microsoft Corporation Real-time Bayesian 3D pose tracking
US8026931B2 (en) * 2006-03-16 2011-09-27 Microsoft Corporation Digital video effects
US20080195316A1 (en) * 2007-02-12 2008-08-14 Honeywell International Inc. System and method for motion estimation using vision sensors
US7659967B2 (en) * 2007-03-05 2010-02-09 Geoffrey Louis Barrows Translational optical flow sensor
JP2010019703A (en) * 2008-07-10 2010-01-28 Toyota Motor Corp Positioning device for mobile body
US20100305784A1 (en) * 2009-05-26 2010-12-02 Anderson Thomas E Embedded Ground Proximity Warning System for Helicopters
US8779971B2 (en) * 2010-05-24 2014-07-15 Robert J. Wellington Determining spatial orientation information of a body from multiple electromagnetic signals
US9014421B2 (en) * 2011-09-28 2015-04-21 Qualcomm Incorporated Framework for reference-free drift-corrected planar tracking using Lucas-Kanade optical flow
TWI469062B (en) * 2011-11-11 2015-01-11 Ind Tech Res Inst Image stabilization method and image stabilization device
US9466007B2 (en) * 2012-01-18 2016-10-11 Agency For Science, Technology And Research Method and device for image processing
US9025859B2 (en) * 2012-07-30 2015-05-05 Qualcomm Incorporated Inertial sensor aided instant autofocus
US9013617B2 (en) * 2012-10-12 2015-04-21 Qualcomm Incorporated Gyroscope conditioning and gyro-camera alignment
US9070202B2 (en) * 2013-03-14 2015-06-30 Nec Laboratories America, Inc. Moving object localization in 3D using a single camera
WO2014154533A1 (en) * 2013-03-27 2014-10-02 Thomson Licensing Method and apparatus for automatic keyframe extraction
US9674507B2 (en) * 2013-04-30 2017-06-06 Qualcomm Incorporated Monocular visual SLAM with general and panorama camera movements
US20140341465A1 (en) * 2013-05-16 2014-11-20 The Regents Of The University Of California Real-time pose estimation system using inertial and feature measurements
CN103345751A (en) 2013-07-02 2013-10-09 北京邮电大学 Visual positioning method based on robust feature tracking
US9953400B2 (en) * 2013-07-23 2018-04-24 Microsoft Technology Licensing, Llc Adaptive path smoothing for video stabilization
AU2013237718A1 (en) * 2013-10-04 2015-04-23 Canon Kabushiki Kaisha Method, apparatus and system for selecting a frame
ITTO20130835A1 (en) * 2013-10-16 2015-04-17 St Microelectronics Srl PROCEDURE FOR PRODUCING COMPACT DESCRIBERS FROM POINTS OF INTEREST OF DIGITAL IMAGES, SYSTEM, EQUIPMENT AND CORRESPONDENT COMPUTER PRODUCT
CN109885080B (en) * 2013-11-27 2022-09-20 宾夕法尼亚大学理事会 Autonomous control system and autonomous control method
US10096114B1 (en) * 2013-11-27 2018-10-09 Google Llc Determining multiple camera positions from multiple videos
JP2015104105A (en) * 2013-11-28 2015-06-04 キヤノン株式会社 Image acquisition apparatus and image acquisition method
CN104954631B (en) * 2014-03-25 2018-02-27 腾讯科技（深圳）有限公司 A kind of method for processing video frequency, device and system
US10401375B2 (en) * 2014-04-10 2019-09-03 SZ DJI Technology Co., Ltd. Method and device for measuring flight parameters of an unmanned aerial vehicle
WO2015165004A1 (en) * 2014-04-28 2015-11-05 Intel Corporation Heading estimation for determining a user's location
WO2016029970A1 (en) * 2014-08-29 2016-03-03 Toyota Motor Europe Nv/Sa Method and system for scanning an object using an rgb-d sensor
US9607388B2 (en) * 2014-09-19 2017-03-28 Qualcomm Incorporated System and method of pose estimation
FR3030058B1 (en) * 2014-12-11 2016-12-09 Airbus Helicopters REDUNDANT DEVICE FOR STEERING SENSORS FOR ROTATING CAR AIRCRAFT
US11768508B2 (en) * 2015-02-13 2023-09-26 Skydio, Inc. Unmanned aerial vehicle sensor activation and correlation system
WO2016187757A1 (en) 2015-05-23 2016-12-01 SZ DJI Technology Co., Ltd. Sensor fusion using inertial and image sensors
US10062010B2 (en) * 2015-06-26 2018-08-28 Intel Corporation System for building a map and subsequent localization
US9909866B2 (en) * 2015-11-05 2018-03-06 Raytheon Company Synthetic digital sextant for navigation
US10502840B2 (en) * 2016-02-03 2019-12-10 Qualcomm Incorporated Outlier detection for satellite positioning system using visual inertial odometry
US10093355B1 (en) * 2016-04-18 2018-10-09 Robo Industries, Inc. Vehicle automatic power and driving control
US10203209B2 (en) * 2016-05-25 2019-02-12 Regents Of The University Of Minnesota Resource-aware large-scale cooperative 3D mapping using multiple mobile devices
WO2017206179A1 (en) * 2016-06-03 2017-12-07 SZ DJI Technology Co., Ltd. Simple multi-sensor calibration
US10101174B2 (en) * 2016-08-22 2018-10-16 Rosemount Aerospace Inc. Air data aided inertial measurement unit
US10578457B2 (en) * 2016-09-15 2020-03-03 Syracuse University Robust and stable autonomous vision-inertial navigation system for unmanned vehicles
US10151588B1 (en) * 2016-09-28 2018-12-11 Near Earth Autonomy, Inc. Determining position and orientation for aerial vehicle in GNSS-denied situations
WO2018081348A1 (en) * 2016-10-26 2018-05-03 The Charles Stark Draper Laboratory, Inc. Vision-inertial navigation with variable contrast tracking residual
FR3060178A1 (en) * 2016-12-09 2018-06-15 Parrot Drones ELECTRONIC DEVICE FOR DRIVING A DRONE, DRONE, STEERING METHOD AND COMPUTER PROGRAM
CN110192122B (en) * 2017-01-24 2023-11-14 深圳市大疆创新科技有限公司 System and method for radar control on unmanned mobile platforms
US10816337B2 (en) * 2017-01-27 2020-10-27 Fa Drone Inc. Inertial measurement method, inertial measurement apparatus, and inertial measurement program
US10107627B2 (en) * 2017-03-08 2018-10-23 Northrop Grumman Systems Corporation Adaptive navigation for airborne, ground and dismount applications (ANAGDA)
US10496104B1 (en) * 2017-07-05 2019-12-03 Perceptin Shenzhen Limited Positional awareness with quadocular sensor in autonomous platforms

Cited By (1)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN111105460A (en) * 2019-12-26 2020-05-05 电子科技大学 RGB-D camera pose estimation method for indoor scene three-dimensional reconstruction

Also Published As

Publication number Publication date
US20200025570A1 (en) 2020-01-23
US11747144B2 (en) 2023-09-05
WO2018182524A1 (en) 2018-10-04
SG10202110833PA (en) 2021-11-29

Similar Documents

Publication Publication Date Title
SG11201908974XA (en) Real time robust localization via visual inertial odometry
SG11201907665QA (en) Intelligent rope or cable termination
SG11201900246TA (en) Determining drivability of objects for autonomous vehicles
SG11201905460SA (en) Data unsealing with a sealing enclave
SG11201804506RA (en) Systems and methods for rendering multiple levels of detail
SG11201808076UA (en) Facilitating vehicle driving and self-driving
SG11201811228SA (en) Systems and methods for displaying movement of vehicle on map
SG11201809866PA (en) Cryptographic applications for a blockchain system
SG11201808609YA (en) System and method for imaging a surface defect on an object
SG11201811318XA (en) Respiratory interface
SG11201909274WA (en) Annular motion simulation amusement park attraction
SG11201907368SA (en) Systems and methods for robotic mobile platforms
SG11201910030SA (en) Virtual reality mobile pod
SG11201903867YA (en) Antibodies directed against t cell immunoglobulin and mucin protein 3 (tim-3)
SG11201907770YA (en) Systems and methods for determining a parking region of vehicles
SG11201909134WA (en) Systems and methods of vehicle guidance
SG11201809219XA (en) Systems and methods for generating stereoscopic, augmented, and virtual reality images
SG11201804528YA (en) Multi-passenger ride vehicle
SG11201811270WA (en) Systems and methods for trajectory determination
SG11201807429UA (en) Real-time multimodal travel estimation and routing system
SG11201811330WA (en) Detecting objects in video data
SG11201804190YA (en) Method and system for blockchain variant using digital signatures
SG11201803050PA (en) Electronic device generating notification based on context data in response to speech phrase from user
SG11201809284UA (en) Mobile device connection apparatus
SG11201809293TA (en) Drive unit for a manipulator