EP3850581B1 - Systeme und verfahren zur vslam-skalenschätzung mittels eines optischen flusssensors auf einem roboter - Google Patents

Systeme und verfahren zur vslam-skalenschätzung mittels eines optischen flusssensors auf einem roboter

Info

Publication number
EP3850581B1
EP3850581B1 EP18933675.3A EP18933675A EP3850581B1 EP 3850581 B1 EP3850581 B1 EP 3850581B1 EP 18933675 A EP18933675 A EP 18933675A EP 3850581 B1 EP3850581 B1 EP 3850581B1
Authority
EP
European Patent Office
Prior art keywords
robotic device
optical flow
image frame
determining
wheel
Prior art date
Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
Active
Application number
EP18933675.3A
Other languages
English (en)
French (fr)
Other versions
EP3850581A4 (de
EP3850581A1 (de
Inventor
Jun Liu
Zixiang Wang
Weizhang LUO
Mei Wu FANG
Tiequan LUO
Current Assignee (The listed assignees may be inaccurate. Google has not performed a legal analysis and makes no representation or warranty as to the accuracy of the list.)
Qualcomm Inc
Original Assignee
Qualcomm Inc
Priority date (The priority date is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the date listed.)
Filing date
Publication date
Family has litigation
First worldwide family litigation filed litigation Critical https://patents.darts-ip.com/?family=69777386&utm_source=google_patent&utm_medium=platform_link&utm_campaign=public_patent_search&patent=EP3850581(B1) "Global patent litigation dataset” by Darts-ip is licensed under a Creative Commons Attribution 4.0 International License.
Application filed by Qualcomm Inc filed Critical Qualcomm Inc
Publication of EP3850581A1 publication Critical patent/EP3850581A1/de
Publication of EP3850581A4 publication Critical patent/EP3850581A4/de
Application granted granted Critical
Publication of EP3850581B1 publication Critical patent/EP3850581B1/de
Active legal-status Critical Current
Anticipated expiration legal-status Critical

Links

Classifications

    • G—PHYSICS
    • G06—COMPUTING OR CALCULATING; COUNTING
    • G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
    • G06T7/00—Image analysis
    • G06T7/20—Analysis of motion
    • G06T7/246—Analysis of motion using feature-based methods, e.g. the tracking of corners or segments
    • G—PHYSICS
    • G05—CONTROLLING; REGULATING
    • G05D—SYSTEMS FOR CONTROLLING OR REGULATING NON-ELECTRIC VARIABLES
    • G05D1/00—Control of position, course, altitude or attitude of land, water, air or space vehicles, e.g. using automatic pilots
    • G05D1/02—Control of position or course in two dimensions
    • G05D1/021—Control of position or course in two dimensions specially adapted to land vehicles
    • G05D1/0231—Control of position or course in two dimensions specially adapted to land vehicles using optical position detecting means
    • G05D1/0246—Control of position or course in two dimensions specially adapted to land vehicles using optical position detecting means using a video camera in combination with image processing means
    • G05D1/0253—Control of position or course in two dimensions specially adapted to land vehicles using optical position detecting means using a video camera in combination with image processing means extracting relative motion information from a plurality of images taken successively, e.g. visual odometry, optical flow
    • G—PHYSICS
    • G05—CONTROLLING; REGULATING
    • G05D—SYSTEMS FOR CONTROLLING OR REGULATING NON-ELECTRIC VARIABLES
    • G05D1/00—Control of position, course, altitude or attitude of land, water, air or space vehicles, e.g. using automatic pilots
    • G05D1/20—Control system inputs
    • G05D1/24—Arrangements for determining position or orientation
    • G05D1/243—Means capturing signals occurring naturally from the environment, e.g. ambient optical, acoustic, gravitational or magnetic signals
    • G05D1/2435—Extracting 3D information
    • G—PHYSICS
    • G06—COMPUTING OR CALCULATING; COUNTING
    • G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
    • G06T7/00—Image analysis
    • G06T7/20—Analysis of motion
    • G—PHYSICS
    • G06—COMPUTING OR CALCULATING; COUNTING
    • G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
    • G06T7/00—Image analysis
    • G06T7/20—Analysis of motion
    • G06T7/269—Analysis of motion using gradient-based methods
    • G—PHYSICS
    • G06—COMPUTING OR CALCULATING; COUNTING
    • G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
    • G06T7/00—Image analysis
    • G06T7/50—Depth or shape recovery
    • G06T7/55—Depth or shape recovery from multiple images
    • G06T7/579—Depth or shape recovery from multiple images from motion
    • G—PHYSICS
    • G06—COMPUTING OR CALCULATING; COUNTING
    • G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
    • G06T7/00—Image analysis
    • G06T7/70—Determining position or orientation of objects or cameras
    • G—PHYSICS
    • G06—COMPUTING OR CALCULATING; COUNTING
    • G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
    • G06T2207/00—Indexing scheme for image analysis or image enhancement
    • G06T2207/10—Image acquisition modality
    • G06T2207/10016—Video; Image sequence
    • G—PHYSICS
    • G06—COMPUTING OR CALCULATING; COUNTING
    • G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
    • G06T2207/00—Indexing scheme for image analysis or image enhancement
    • G06T2207/30—Subject of image; Context of image processing
    • G06T2207/30244—Camera pose
    • G—PHYSICS
    • G06—COMPUTING OR CALCULATING; COUNTING
    • G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
    • G06T2207/00—Indexing scheme for image analysis or image enhancement
    • G06T2207/30—Subject of image; Context of image processing
    • G06T2207/30248—Vehicle exterior or interior
    • G06T2207/30252—Vehicle exterior; Vicinity of vehicle

Definitions

  • Robotic devices have become increasingly commonplace for performing various tasks in a semi-autonomous or autonomous manner. Such robotic devices may be embodied in a variety of forms and used in a variety of applications, such as in automated vacuum cleaners, unmanned aerial vehicles, terrestrial vehicle, etc. Applications for which robotic devices may be employed may include entertainment applications (e.g., toy robots), utility applications in environments that are unfriendly to humans (e.g., space, deep water, cold temperature, radiation, chemical exposure, biohazards, etc.), dangerous tasks (e.g., defusing of explosives), operation in confined spaces (e.g., collapsed buildings) performing menial tasks (e.g., cleaning), etc.
  • entertainment applications e.g., toy robots
  • utility applications in environments that are unfriendly to humans (e.g., space, deep water, cold temperature, radiation, chemical exposure, biohazards, etc.)
  • dangerous tasks e.g., defusing of explosives
  • operation in confined spaces e.g., collapsed buildings
  • a robotic device may be provided with mapping techniques allowing the device to construct a map of its surroundings. Further, a robotic device may be configured to localize itself within the map, and thereby exhibit a relatively high degree of autonomy.
  • autonomous navigation may be complex due to temporal and spatial variations in the local environment, as well as changes in the position and motion of obstacles. Such complexities may create navigation problems for the robotic device using existing techniques.
  • Various aspects include methods, robotic devices implementing such methods, and processing devices within robotic vehicles for accomplishing visual simultaneous localization and mapping using data from an optical flow sensor.
  • Various embodiments include methods performed by a processor of a robotic device for improving navigation by employing localization and mapping techniques, such as visual simultaneous localization and mapping (VSLAM), leveraging information provided by an optical flow sensor to better plan a navigation path within the environment.
  • VSLAM visual simultaneous localization and mapping
  • the processor of the robotic device may be configured to receive a first image frame from an image sensor, receive a second image frame from the image sensor, generate homograph computation values based on the first and second image frames, and generate a homography matrix based on the homograph computation values, determine or generate a first transaction information structure (e.g., a three-element translation vector, etc.) based on the homography matrix or homograph computation values, receive optical flow sensor data from an optical flow sensor, determine or generate a second transaction information structure (e.g., an optical flow sensor transaction information structure) based on the optical flow sensor data, determine a scale estimation value based on the homograph computation values and the optical flow sensor data (e.g., based on the values in the first and second transaction information structures), and determine a robotic device pose based on the scale estimation value.
  • a first transaction information structure e.g., a three-element translation vector, etc.
  • the processor of the robotic device may be configured to also receive wheel transaction data from a wheel encoder.
  • the processor may determine or compute a third transaction information structure (e.g., a wheel encoder transaction information structure) based on the wheel transaction data.
  • the processor may determine whether the wheel transaction data is inaccurate or erroneous, or include outlier data (e.g., due to wheel slippage or spinning).
  • the processor may use the wheel transaction data to determine the scale estimation value when the wheel transaction data is not inaccurate, erroneous, or outlier, and use the optical flow sensor data to determine scale estimation value when the wheel transaction data is inaccurate or erroneous, or include outlier data.
  • the processor may use the values in the first and third transaction information structures to determine the scale estimation value when the wheel transaction data is not inaccurate, erroneous or outlier, and use the values in the first and second transaction information structures to determine scale estimation value when the wheel transaction data is inaccurate or erroneous, or include outlier data.
  • the various embodiments improve the performance and functioning of the robotic device.
  • the inclusion and use of an optical flow sensor in accordance with the various embodiments may allow the robotic device to better identify and respond to slippery conditions in which wheel rotations do not correlate well with actual device motions.
  • the inclusion and use of an optical flow sensor in accordance with the various embodiments may also allow the robotic device to better plan its navigation path within the environment and enable traveling over surfaces where wheel slippage may be anticipated.
  • robot device refers any of various types of robotic vehicles, robotic appliances, robots, etc. that include an onboard processing device configured to provide some autonomous or semi-autonomous navigation and maneuvering capabilities.
  • robotic devices include but are not limited to ground vehicles and robots (e.g., autonomous or semi-autonomous cars), robotic appliances (e.g., robotic vacuums and other cleaning robots), search and rescue robots, bomb detection and disarming robots, etc.) and combinations thereof.
  • the robotic device may be manned. In other embodiments, the robotic device may be unmanned.
  • the robotic device may include an onboard processing device configured to maneuver and/or navigate the robotic device without remote operating instructions (i.e., autonomously), such as from a human operator (e.g., via a remote computing device).
  • the robotic device may include an onboard processing device configured to receive some information or instructions, such as from a human operator (e.g., via a remote computing device), and autonomously maneuver and/or navigate the robotic device consistent with the received information or instructions.
  • the term "pose” refers to a value or information structure that identifies the position and orientation of a robotic device, or a component thereof, within its surrounding environment.
  • the robotic device pose may be specified by a two-dimensional position (e.g., x,y distances or dimensions in Cartesian coordinates) and a heading ( ⁇ ).
  • the robotic device may be configured to estimate pose (e.g., camera pose, robotic device pose, etc.) based on a rigid transformation (R,T), where R is a rotation matrix (e.g., a three-by-three matrix information structure, etc.) and T is a transaction (e.g., a three-element translation vector information structure, etc.).
  • R is a rotation matrix
  • T is a transaction (e.g., a three-element translation vector information structure, etc.).
  • the values for the rotation matrix (R) and/or transaction (T) may be determined via the robotic device processor performing a singular value decomposition (SVD) operation on a homography matrix (H).
  • the values of the rotation matrix (R) may be determined based on the orientation of the camera/robotic device.
  • the values of the transaction (T) may be determined based on the orientation and/or location of the camera/robotic device.
  • wheel encoder is used herein to refer to a sensor or device that collects and converts the angular position or motion of a wheel, shaft or axle of the robotic device to analog or digital output signals so as to provide information regarding the angle of rotation, number of rotations and/or rate of rotations of a wheel.
  • a wheel encoder may use mechanical, optical or magnetic sensors to detect rotational position changes, count rotations and/or determine the rate of rotation.
  • Information from a wheel encoder in combination with knowledge of the wheel diameter may be used by a processor of the robotic device to determine position, speed, and/or traveled distance information. Examples of wheel encoders include rotary encoders and shaft encoders.
  • the wheel encoder may include, or may be coupled to, a processor that is configured to generate a wheel encoder transaction (Tw) information structure based on the wheel encoder data.
  • Tw wheel encoder transaction
  • optical flow sensor refers to a sensor capable of measuring motion optically by detecting optical flow, and outputting measurements based on the optical flow or observed visual motion.
  • An optical flow may identify or define the pattern of apparent motion of objects, surfaces, and edges in a visual scene caused by the relative motion between an observer (e.g., robotic device, etc.) and a scene.
  • the optical flow sensor may output measurements or optical flow data/information that may be used to measure the motion of the robotic device across a surface.
  • the optical flow information may also be used to measure visual motion or relative motion between the robotic device and other objects in the vicinity of the robotic device.
  • the robotic device may use the optical flow information for motion detection, object segmentation, time-to-contact information, focus of expansion calculations, motion compensated encoding, stereo disparity measurements, and other similar information.
  • the optical flow sensor may be coupled to a processor that is configured to generate an optical flow sensor transaction (To) information structure based on the optical flow information.
  • the optical flow sensor may be an image sensor that is coupled to a processor that is programmed to run an optical flow algorithm.
  • the optical flow sensor may be vision chip that includes the image sensor and processor on the same chip or die.
  • the optical flow sensor may include an illumination source (e.g., a light-emitting diode (LED) laser) and an optical sensor configured to receive light from the source reflected by a surface.
  • LED light-emitting diode
  • a robotic device may employ VSLAM techniques to construct and update a map of an unknown environment while simultaneously keeping track of the robotic device's location within the environment.
  • a robotic device may be equipped with a monocular image sensor that captures images or frames of the environment.
  • the robotic device may identify prominent objects or features within the captured image, estimate the dimensions and scale of the features in the image, compare the identified features to each other and/or to features in test images having known dimensions and scale, and identify correspondences based on the comparisons.
  • Each correspondence may be a value set or an information structure that identifies a feature (or feature point) in one image as having a high probability of being the same feature in another image.
  • a correspondence may be a set of image points (e.g., a first point in a first image and a second point in a second image, etc.) that are in correspondence.
  • the robotic device may produce a homography matrix information structure based on the identified correspondences, and use the homography matrix to determine its pose (e.g., position, orientation, etc.) within the environment.
  • the robotic device may be equipped with sensors that gather movement or distance information useful for employing VSLAM techniques.
  • the robotic device may use such movement or distance information to determine a distance between captured images or frames, and use such distance information in conjunction with the homography matrix to estimate the dimensions and scale of the objects in the frames. This in turn allows the robotic device to determine its pose with a higher degree of precision and accuracy than if the pose is determined solely based on captured images.
  • a robotic device may use wheel transaction data from a wheel encoder for determining the distance that the robotic device traveled between images/frames.
  • the robotic device may also use wheel transaction data from a wheel encoder to determine a scale estimate value.
  • a wheel rotation sensor such as a wheel encoder
  • a wheel rotation sensor may be prone to erroneous output when the robotic device is moving on certain surfaces due to slippage, as may occur on an incline and/or slippery surfaces, or when there is resistance to wheel rotation (e.g., increased friction or another impediment to rotation).
  • the robotic device may not be able to accurately determine the scale, perspective, or dimensions of the objects/features in the images. This, in turn, may hamper the robotic device's ability to determine its pose in the environment with a high degree of confidence, precision or accuracy.
  • the robotic device may be equipped with an optical flow sensor that collects and/or produces optical flow sensor data that may be used by a processor to determine distance travelled.
  • the processor of the robotic device processor may be configured to receive a first image frame, receive a second image frame, generate homograph computation values based on the first image frame and the second image frame, receive optical flow sensor data from an optical flow sensor, determine a scale estimation value based on the homograph computation values and the optical flow sensor data, and use the scale estimation value to determine the robotic device pose.
  • the processor of the robotic device may be configured to intelligently select and use the optical flow sensor data to replace inaccurate data collected from the wheel rotation sensor.
  • the distance traveled information from the optical flow sensor may be used by the processor to correct or replace outlier data, or improve the inaccurate of traveled distance information that is collected from a wheel encoder and/or other sensors in the robotic device.
  • the processor of the robotic device may be configured to detect slippage in the wheel or wheel encoder, and intelligently determine whether to use information collected from the wheel encoder, the optical flow sensor, or a combination thereof based on the level, degree, or magnitude of the detected slippage.
  • the processor of the robotic device may be configured to capture optical flow images, detect feature points (e.g., corners, etc.) in the optical flow images, and apply the images/features point data to a Lucas-Kanade component/function to generate optical flow information.
  • the optical flow information may characterize a transition or transaction between two images captured by an image sensor (e.g., camera) of the robotic device.
  • the optical flow information may include information that identifies a speed (e.g., in pixels per unit time or pixels per image) and a direction of apparent movement for each feature point.
  • the processor of the robotic device may be configured to use the optical flow sensor data collected from the optical flow sensor to calculate or estimate the scale in an image or between two images.
  • the communication system 100 may include a robotic device 102, a base station 104, an access point 106, a communication network 108, and a network server computing device 110.
  • the robotic device 102 may be equipped with at least one spatial measurement device 103, such as a monocular camera that is suitable for obtaining the three-dimensional position of the robotic device 102 within the real word environment.
  • the robotic device 102 may also be equipped with a wheel encoder 105 that measures the rotational movement of the wheel(s) and/or an optical flow sensor 107 configured to detect/measure movement over a surface based upon optical flow and output optical flow information that can be used by a processor to determine distances traveled, velocity, pixel speed, direction of travel, etc.
  • the base station 104 may be configured to provide wireless communications over a wide area (e.g., macro cells), as well as small cells, which may include a micro cell, a femto cell, a pico cell, and other similar network access points.
  • the access point 106 may include access points configured to provide wireless communications over a relatively smaller area. Other examples of base stations and access points are also possible.
  • the robotic device 102 may communicate over the communication network 108 through wireless links provided by the base station 104 and the access point 106.
  • the robotic device 102 may communicate with the base station 104 over a wireless communication link 112, and with the access point 106 over a wireless communication link 114.
  • the wireless communication links 112 and 114 may include a plurality of carrier signals, frequencies, or frequency bands, each of which may include a plurality of logical channels.
  • the wireless communication links 112 and 114 may utilize one or more radio access technologies (RATs).
  • RATs radio access technologies
  • the robotic device 102 may move in an environment 120 and use one or more sensors, such as the spatial measurement device 103, to capture at least one image of a target 125 in the environment 120.
  • Navigation of a robotic device in various embodiments may be performed autonomously using systems that rely on VSLAM techniques. While the localization and mapping processes may be performed simultaneously, the processes may instead be performed sequentially, such as in a multiplexed fashion. For example, a VSLAM system may use the same data to both localize a robotic device within a map and to generate or update the map.
  • the VSLAM system may employ algorithms to solve the computational problem of constructing or updating a map of an unknown environment while simultaneously keeping track of a robotic device's location within the map.
  • the VSLAM system may track a set of points through sensor inputs (e.g., camera or video frames), and use estimated locations of the tracked points to calculate the pose from which the robotic device could have observed or measured the points while simultaneously constructing or updating a map based on the tracked point locations in three-dimensional space.
  • the robotic device may include a multicore processor configured to accomplish VSLAM by separating the localization and mapping functions onto different threads. For example, localization (i.e., current pose estimation) may be performed in real time on one thread, while the mapping thread runs processes in the background to minimize the differences between the tracked point locations and where the points are expected to be given the pose estimate (i.e., reprojection errors). On completion, the mapping thread may update the information used to track the set of points, and in turn the localization thread adds new observations to expand the map.
  • localization i.e., current pose estimation
  • the mapping thread runs processes in the background to minimize the differences between the tracked point locations and where the points are expected to be given the pose estimate (i.e., reprojection errors).
  • the mapping thread may update the information used to track the set of points, and in turn the localization thread adds new observations to expand the map.
  • the robotic device may be configured to apply VSLAM techniques to non-autonomous tasks.
  • a remotely-controlled device for bomb detection or search and rescue may implement VSLAM in a remote-control application to assist an operator to navigate around an environment.
  • the robotic device may include various operational modes, such as a mode for manual control and another mode for an autonomous control.
  • a robotic device in the form of a vehicle may be manually-driven during an initial mapping stage, and later configured for autonomous control.
  • FIG. 1B illustrates various components of a ground-based robotic device 102 that is suitable for implementing the various embodiments.
  • the robotic device 102 may utilize one or more wheels 130 driven by corresponding motors to provide locomotion to the robotic device 102.
  • the robotic device 102 may be configured to determine or identify a height above or distance between a surface 132 (or the bottom portions of the wheels 130) and one or more points in the body frame 134 of the robotic device 102.
  • the robotic device may be preprogrammed with the height information.
  • the robotic device may be configured to determine the height information based on information collected from one or more sensors (e.g., infrared sensor, etc.).
  • a processor of the robotic device 102 may be configured to identify feature points 136 on the surface 132 within images.
  • a processor within the optical flow sensor 107 or the robotic device 102 may be configured to determine the distance and direction traveled by the robotic device 102 based on the height information and the identified feature points.
  • the robotic device 102 illustrated in FIGS. 1A and 1B is presented as a nonlimiting example of a device that may utilize various embodiments, but is not intended to imply or require that various embodiments are limited to ground robotic devices.
  • various embodiments may be used with rotorcraft or winged robotic devices or water-borne robotic devices for applications in which an optical flow sensor can provide information regarding distances traveled, speed, etc.
  • FIG. 1C illustrates a computer architecture and various sensors that may be included in a robotic device 102 that is configured to employ VSLAM techniques in accordance with the various embodiments.
  • the robotic device 102 includes a main board component 152, and image sensor 154, a microcontroller unit (MCU) 156, a wheel encoder 105, an infrared (IR) sensor 160, an inertial measurement unit (IMU) 162, a bumper sensor 164, a laser distance sensor (LDS) 166, and an optical flow sensor 107.
  • MCU microcontroller unit
  • IR infrared
  • IMU inertial measurement unit
  • LDS laser distance sensor
  • the sensors 154, 160-168 may gather information that is useful for employing VSLAM techniques in the robotic device 102.
  • a wheel encoder 105 and/or an optical flow sensor 107 may collect information suitable for determining the distance that the robotic device 102 has traveled between images/frames.
  • the wheel encoder 105 and/or the optical flow sensor 107 may also collect information suitable for determining or generating a transaction information structure that is suitable for estimating scale.
  • the robotic device 102 may use this information in conjunction with information collected from other sensors 160-168 to better estimate the dimensions and scale of the objects in the frames captured by the image sensor 154. This in turn allows the robotic device 102 to determine its pose with a higher degree of precision and accuracy.
  • the optical flow sensor 107 may be configured to collect and/or produce optical flow sensor data that may be used to correct, replace, or improve inaccurate and/or outlier distance measurement data that is collected from other the sensors 154, 160-166 in the robotic device.
  • a processor in the main board component 152 and/or the MCU 156 may be configured to intelligently select and use the optical flow sensor data from the optical flow sensor 107 to replace inaccurate data collected from the wheel encoder 105.
  • the processor may be configured to detect slippage based on the information collected from the sensors 154, 160-168, and intelligently determine whether to use information collected from a wheel encoder 105, the optical flow sensor 107, or a combination thereof.
  • FIG. 2 illustrates various components of a robotic device 102 that could be configured to estimate scale and determine pose in accordance with the various embodiments.
  • the robotic device 102 may include a number of wheels 130, a body frame 134, at least one spatial measurement device 103, and an optical flow sensor 107.
  • the spatial measurement device(s) 103 may be the same as or similar to the spatial measurement device 103.
  • the spatial measurement device(s) 103 may include, for example, systems configured as raw range scan sensors, or feature-based systems configured to recognize landmarks from scans or images.
  • a laser-based scanner e.g., Light Detection and Ranging (LiDAR)
  • LiDAR Light Detection and Ranging
  • sonar-based system may be used to detect and measure distances to various features and landmarks in the environment.
  • features and landmarks may be identified within images taken by an image sensor (e.g., cameras, optical readers, etc.).
  • any of a number of other devices capable of detecting nearby features and landmarks may also be implemented as a spatial measurement device.
  • the body frame 134 may provide structural support for the motors and their associated wheels 130 as well as for the spatial measurement device 103.
  • some detailed aspects of the robotic device 102 are omitted such as wiring, frame structure interconnects, or other features that would be known to one of skill in the art.
  • the illustrated robotic device 102 has wheels 130, this is merely exemplary and various embodiments may include any variety of components to provide propulsion and maneuvering capabilities, such as treads, paddles, skids, or any combination thereof or of other components.
  • the robotic device 102 may further include a control unit 210 that may house various circuits and devices used to power and control the operation of the robotic device 102.
  • the control unit 210 may include a processor 220, a power component 230, sensors 240, one or more payload securing units 244, an output component 250, an input component 260, and a radio component 270.
  • the processor 220 may be configured with processor-executable instructions to control travel and other operations of the robotic device 102, including operations of various embodiments.
  • the processor 220 may include or be coupled to a navigation unit 222, a memory 224, a maneuvering data component 226, and an optical flow sensor 228.
  • the processor 220 and/or the navigation unit 222 may be configured to communicate with a server through a wireless connection (e.g., a cellular data network) to receive data useful in navigation, provide real-time position reports, and assess data.
  • a wireless connection e.g., a cellular data network
  • the processor 220 may also receive data from the optical flow sensor 228 and use such data to determine, identify or define the pattern of apparent motion of objects, surfaces, and edges in the environment caused by the motion of the robotic device 202.
  • the processor 220 may be configured to detect wheel slippage, and determine whether to use information collected from the wheel rotation sensor, the optical flow sensor 228. or a combination thereof based on the detected slippage.
  • the maneuvering data component 226 may be coupled to the processor 220 and/or the navigation unit 222, and may be configured to provide travel control-related information such as orientation, attitude, speed, heading, and similar information that the navigation unit 222 may use for navigation purposes, such as dead reckoning between Global Navigation Satellite System (GNSS).
  • GNSS Global Navigation Satellite System
  • the maneuvering data component 226 may receive input from one or more sensors 240, such as an IMU or similar sensors (e.g., accelerometer, a gyroscope, etc.).
  • the sensor(s) 240 providing input to the maneuvering data component 226 may include or overlap with the one or more spatial measurement devices 103.
  • the input to the maneuvering data component 226 may provide data regarding the orientation and accelerations of the robotic device 102 that may be used in navigation and positioning calculations, as well as providing data used in various embodiments.
  • the processor 220 may further receive additional information from one or more other types of sensors 240 (e.g., a camera, which may be a monocular camera).
  • the sensor(s) 240 may include one or more optical sensors capable of detecting infrared, ultraviolet, and/or other wavelengths of light.
  • the sensor(s) 240 may include and/or overlap with the range measurement device(s) 206.
  • the sensor(s) 240 may include at least one sensor that provides motion feedback to the processor 220, for example, a wheel encoder, a contact or pressure sensor that provides a signal indicative of contact with a surface, optical flow sensor, etc.
  • the sensor(s) may also include a radio frequency (RF) sensor, a barometer, a sonar emitter/detector, a radar emitter/detector, a microphone or another acoustic sensor, or another sensor that may provide information usable by the processor 220 for movement operations as well as navigation and positioning calculations.
  • RF radio frequency
  • the payload-securing units 244 may include an actuator motor that drives a gripping and release mechanism and related controls that are responsive to the control unit 210 to grip and release a payload in response to commands from the control unit 210.
  • the power component 230 may include one or more batteries that may provide power to various components, including the processor 220, the sensors 240, the payload-securing unit(s) 244, the drive motors coupled to the wheels 130, the output component 250, the input component 260, and the radio component 270.
  • the power component 230 may be rechargeable.
  • the processor 220 may be configured with processor-executable instructions to control the charging of the power component 230, such as by executing a charging control algorithm using a charge control circuit. Alternatively or in addition, the power component 230 may be configured to manage its own charging.
  • the processor 220 may be coupled to the output component 250, which may output control signals for managing motors that drive the wheels 130 and other components.
  • the robotic device 102 may be maneuvered by controlling individual motors driving the wheels 130 as the robotic device 102 progresses toward a destination.
  • the processor 220 may receive data from the navigation unit 222 and use such data in order to determine the present position and orientation of the robotic device 102, as well as the appropriate course towards the destination or intermediate sites.
  • the navigation unit 222 may include a GNSS receiver system (e.g., one or more global positioning system (GPS) receivers) enabling the robotic device 102 to navigate using GNSS signals.
  • GPS global positioning system
  • the navigation unit 222 may be equipped with radio navigation receivers for receiving navigation beacons or other signals from radio nodes, such as navigation beacons (e.g., very high frequency (VHF) omni-directional range (VOR) beacons), Wi-Fi access points, cellular network sites, radio station, remote computing devices, robotic vehicles, etc.
  • navigation beacons e.g., very high frequency (VHF) omni-directional range (VOR) beacons
  • Wi-Fi access points e.g., Wi-Fi access points, cellular network sites, radio station, remote computing devices, robotic vehicles, etc.
  • the radio component 270 may include a modem 274 and a transmit/receive antenna 272.
  • the radio component 270 may be configured to conduct wireless communications with a variety of wireless communication devices (e.g., a wireless communication device (WCD) 290), examples of which include a wireless telephony base station or cell tower (e.g., the base station 104), a network access point (e.g., the access point 106), a beacon, a smartphone, a tablet, or another computing device (e.g., 110) with which the robotic device 102 may communicate.
  • WCD wireless communication device
  • the processor 220 may establish a bi-directional wireless communication link 294 via the modem 274 and the antenna 272 of the radio component 270 and the wireless communication device 290 via a transmit/receive antenna 292.
  • the radio component 270 may be configured to support multiple connections with different wireless communication devices using different radio access technologies.
  • control unit 210 may be equipped with an input component 260, which may be used for a variety of applications.
  • the input component 260 may receive images or data from an onboard camera or sensor, or may receive electronic signals from other components (e.g., a payload).
  • control unit 210 While various components of the control unit 210 are illustrated in FIG. 2 as separate components, some or all of the components (e.g., the processor 220, the output component 250, the radio component 270, and other units) may be integrated in a single processing device 310, an example of which is illustrated in FIG. 3 .
  • FIG. 3 illustrates a processing device 310 that could be configured to be used in a robotic device.
  • the processing device 310 may be configured to be used in the robotic device and may be configured as or including a system-on-chip (SoC) 312.
  • SoC 312 may include (but is not limited to) a processor 314, a memory 316, a communication interface 318, and a storage memory interface 320.
  • the processing device 310 or the SoC 312 may further include a communication component 322, such as a wired or wireless modem, a storage memory 324, an antenna 326 for establishing a wireless communication link, and/or the like.
  • the processing device 310 or the SoC 312 may further include a hardware interface 328 configured to enable the processor 314 to communicate with and control various components of a robotic device.
  • the processor 314 may include any of a variety of processing devices, for example any number of processor cores.
  • SoC system-on-chip
  • processors e.g., 314
  • memory e.g., 316
  • communication interface e.g., 318
  • the SoC 312 may include a variety of different types of processors 314 and processor cores, such as a general purpose processor, a central processing unit (CPU), a digital signal processor (DSP), a graphics processing unit (GPU), an accelerated processing unit (APU), a subsystem processor of specific components of the processing device, such as an image processor for a camera subsystem or a display processor for a display, an auxiliary processor, a single-core processor, and a multicore processor.
  • the SoC 312 may further embody other hardware and hardware combinations, such as a field programmable gate array (FPGA), an application-specific integrated circuit (ASIC), other programmable logic device, discrete gate logic, transistor logic, performance monitoring hardware, watchdog hardware, and time references.
  • Integrated circuits may be configured such that the components of the integrated circuit reside on a single piece of semiconductor material, such as silicon.
  • the SoC 312 may include one or more processors 314.
  • the processing device 310 may include more than one SoC 312, thereby increasing the number of processors 314 and processor cores.
  • the processing device 310 may also include processors 314 that are not associated with a SoC 312 (i.e., external to the SoC 312).
  • Individual processors 314 may be multicore processors.
  • the processors 314 may each be configured for specific purposes that may be the same as or different from other processors 314 of the processing device 310 or SoC 312.
  • One or more of the processors 314 and processor cores of the same or different configurations may be grouped together.
  • a group of processors 314 or processor cores may be referred to as a multi-processor cluster.
  • the memory 316 of the SoC 312 may be a volatile or non-volatile memory configured for storing data and processor-executable instructions for access by the processor 314.
  • the processing device 310 and/or SoC 312 may include one or more memories 316 configured for various purposes.
  • One or more memories 316 may include volatile memories such as random access memory (RAM) or main memory, or cache memory.
  • the processing device 310 and the SoC 312 may be arranged differently and/or combined while still serving the functions of the various aspects.
  • the processing device 310 and the SoC 312 may not be limited to one of each of the components, and multiple instances of each component may be included in various configurations of the processing device 310.
  • a robotic device in various embodiments may utilize VSLAM processes in order to navigate and in order to carry out tasks.
  • VSLAM processes may include mapping the spatial information of landmarks within the environment, as well as representing the robotic device's pose (i.e., position and orientation) relative to the mapped environment.
  • VSLAM algorithms take account of a variety of parameters, such as sensor information, map representation, device dynamics, environmental dynamics, and the integration of sensor measurements and the robot's control system over time. Mapping the spatial information of the environment generally requires employing spatial sensors (e.g., spatial measurement device(s) 103) to which VSLAM algorithms may be applied. For example, landmarks may be extracted from images obtained by one or multiple cameras, which may be any of a variety of configurations (e.g., monocular, stereo vision, multiple camera, etc.).
  • Localization of the robotic device may be performed by obtaining sensory information regarding the position and orientation (i.e., pose) of the device within the generated map.
  • pose information may be determined using information obtained from wheel encoders, which measure the rotational movement of the wheel(s) and/or optical flow sensors, which measure distances traveled based on the flow of features observed in a light or image sensor.
  • the robotic device may also use a GPS sensor, which allows acquisition of pose information.
  • FIG. 4 illustrates an example navigation system that is suitable for use in accordance with the various embodiments.
  • the navigation system 400 may be implemented on a processor of a robotic device (e.g., 102, 200), such as a ground vehicle (e.g., car, vacuum robot, etc.), an aerial vehicle (e.g., UAV), etc.
  • a robotic device e.g., 102, 200
  • a ground vehicle e.g., car, vacuum robot, etc.
  • an aerial vehicle e.g., UAV
  • inputs may be received from multiple sensors in order to enable the system to perform localization, map generation, and path planning for VSLAM processes on the robotic device.
  • sensors 402 may include at least one IMU sensor for detecting orientation or other maneuvering data.
  • the sensor(s) 402 may also include at least one motion feedback sensor, such as a wheel encoder, optical flow sensor, pressure sensor, or other collision or contact-based sensor.
  • the sensor(s) 402 may include at least one image sensor, such as a visual camera, an infrared sensor, a sonar detector, etc.
  • Raw measurement data from the sensors 402 may be provided to a pre-processor(s) 404.
  • Pre-processing performed by the pre-processor 404 may include removal of outliers and noise suppression, as well as landmark/feature extraction, data association, etc.
  • the pre-processor(s) 404 may output processed measurements relating to position and orientation of the robotic device (e.g., acceleration, velocity, odometry information, etc.).
  • the pre-processor(s) 404 may represent multiple pre-processors that each correspond to a different sensor or group of sensors.
  • some pre-processors may be configured to receive and jointly process raw measurements from multiple sensors, or may be configured to receive and process raw measurements from a single sensor 402.
  • Processed measurements from each pre-processor(s) 404 may be provided to semantic information extractor(s) 406, which may obtain semantic information about each sensor(s) 402 associated with the corresponding pre-processor(s) 404.
  • the semantic information extractor(s) 406 may be configured to identify and record specific adverse events relating to the associated sensor(s) 402.
  • adverse events may include, for example, an outlier data measurement, a string of blank measurements from a sensor, data indicating a condition of the robotic device (e.g., device stuck, collision with obstacle, etc.), or any other measurement indicating diminished performance by one or more sensors.
  • the semantic information extractor(s) 406 may be configured to determine whether data from the wheel encoder (e.g., 105) includes errors or inaccurate information due to conditions such slippage or resistance. In some embodiments, the semantic information extractor(s) 406 may be configured to compute a confidence value that quantifies the level of confidence that the collected sensor data falls within an accuracy tolerance interval. Similarly, the semantic information extractor(s) 406 may be configured to compute an accuracy probability value that identifies the degree or likelihood that information collected from a sensor (e.g., a wheel encoder 105, etc.) is accurate.
  • a sensor e.g., a wheel encoder 105, etc.
  • the computed accuracy probability value may identify the likelihood that distance information collected from the wheel encoder over a time period is accurate (e.g., within one meter, decimeter, centimeter, millimeter, etc.) of the actual distance that the robotic device traveled during that time period.
  • the semantic information extractor(s) 406 may store and update semantic information for each associated sensor 402.
  • Such semantic information may be represented in any of a number of data structures, which may be stored in a database in local storage on the robotic device. Examples of data structures that may be used to represent the semantic information include, for example, a data table/array, tree, etc.
  • semantic information that is relatively dense across time and space may be represented in a data array.
  • semantic information based on recording outlier sensor data may be best represented in a data array.
  • semantic information that is relatively sparse across time and space may be represented in a data tree. For example, semantic information based on recording events in which the robotic device is stuck may be best represented in a data tree.
  • local storage may be a separate storage device, examples of which may include universal serial bus (USB) drives, memory cards, solid-state drives (SSDs), hard disk drives (HDDs), floppy disks, optical disks, magnetic tapes, and the like.
  • USB universal serial bus
  • SSDs solid-state drives
  • HDDs hard disk drives
  • floppy disks optical disks
  • magnetic tapes and the like.
  • the local storage may be a removable storage device such as a memory card, for example, a PC Card, CompactFlash, SmartMedia, Memory Stick, Memory Stick Duo, Memory Stick PRO Duo, Miniature Card, Multimedia Card (MMC), Reduced Size Multimedia Card (RS-MMC), MMCmicro Card (MMCmicro), PS2 card, Secure Digital (SD) card, SxS, Universal Flash Storage (UFS), miniSD, microSD, xD-Picture Card, Intelligent Stick (iStick), etc.
  • the local storage may be a partition or part of a memory device (e.g., 224).
  • An adverse event may be detected in various embodiments by comparing the processed measurement s from the corresponding pre-processor 406 with a current estimated pose for the robotic device.
  • feedback from a localization component 408 for VSLAM may provide the current estimated pose to each semantic information extractor(s) 406. If the difference between a pose indicated by the processed measurement(s) from the pre-processor 404 and the current estimated pose is greater than a pre-determined threshold, the semantic information extractor 406 may record an adverse event for the related one or more associated sensor(s) 402.
  • the semantic information extractor 406 may identify a timeslot during which the adverse event occurred within a time period by accessing a timeslot server 410.
  • the timeslot server 410 may maintain independent time period and timeslot instructions for each semantic information extractor 406.
  • the semantic information extractor 406 may update the stored semantic information for the related one or more associated sensor(s) 402.
  • the semantic information extractor 406 may periodically seek to identify patterns in the adverse event occurrence across timeslots and/or poses for each associated sensor 402, such as when each time period ends. Any such identified patterns may be provided to a weighting factor component 412, which may receive the processed measurement(s) from the corresponding pre-processor 404.
  • the weighting component 412 may adjust the weight given to data (i.e., processed measurement(s)) from the related associated sensor(s) 402 based on the identified pattern, confidence value, or accuracy probability value. Such adjustment may involve lowering the weighting factor of data from a sensor 402 for a specific time duration, or when the robotic device is located in a particular spatial region.
  • the processed measurements may be provided to the localization component 408.
  • the localization component 408 may generate a current estimated pose for the robotic device as part of VSLAM processes.
  • the location component 408 may provide such information to a mapping component 414 and to a path planning component 416.
  • the pre-processing component(s) 404 may provide data about the position of obstacles in the local environment to the mapping component 414.
  • the pre-processor(s) 404 that provide such obstacle position information, as well as their associated sensor(s) 402 may be either the same as or different from those used to provide processed measurements to the semantic information extractor(s) 406.
  • the object position information may be computed based on an identified point set and/or landmarks in the local environment.
  • the mapping component 414 may utilize the current estimated pose and obstacle position information to construct at least one map. For example, in using VSLAM techniques, the mapping component 414 may generate a map with spatial representations of the local environment for navigation, as well as update mapping information relative to the robotic device about features (e.g., landmarks).
  • the map(s) generated by the mapping component 414 may be provided to the path planning component 416.
  • the path planning component 416 may use the received map(s) and the current estimated pose information to select, create, or update a navigation path for the robotic device.
  • a processor of a robotic device e.g., the processor 220, etc.
  • hardware components and/or software components of the robotic device may be configured to capture and process an image using an image sensor.
  • an image sensor of the robotic device may capture images of target objects.
  • the target objects may be located or arranged such that the robotic device image sensor can easily acquire images of at least two of the target objects. If the image sensor is unable to obtain an image of two or more of the target objects, the processor may instruct the various actuators to move the robotic device to another location or pose and try again.
  • the processor may execute VSLAM using the captured images of the target objects. For example, the processor may identify features within each of the captured images and may attempt to track those features between subsequent images.
  • the processor may determine whether the rotation angle of the image sensor can be determined.
  • the processor may attempt to calculate the angle of rotation between the robotic device and the target objects. Because the dimension, position, and location of the target objects is known, and stored in a memory of the robotic device, the processor may be able to calculate the robotic device's angle of rotation with reference to the target objects using the captured images.
  • the processor may capture additional images of the target objects. Thus, if the processor is unable to determine the robotic device's angle of rotation with reference to the target objects, the robotic device may return to capturing images and performing VSLAM until an image is found from which the angle of rotation may be calculated.
  • the processor may select a captured image having two or more target objects in a field of view of the image.
  • the processor may determine an image scale based, at least in part, on a location of the target objects within the selected image. Because the dimension and position of the target objects relative to each other is already known, the processor of the robotic device may calculate the scale of the images.
  • the processor may compare the size and distribution of the target objects in the selected image to the stored dimension and position information for the target objects to determine the scale of the objects in the image.
  • the processor may determine whether the selected image includes more than two target objects. This determination may be made during the selection of the captured image, during feature identification, or thereafter. If the selected image contains more than two target objects, additional information may be gained from the image.
  • the processor may continue capturing images of the target objects.
  • the processor may determine a rotation angle correction in response to determining that the selected image includes more than two target objects.
  • homography matrix-based techniques may be employed by the processor to determine the rotation angle correction. If the target objects are not positioned in the same plane, then fundamental matrix techniques may be employed by the processor to determine the rotation angle correction.
  • FIG. 5A illustrates various operations and functional components in a robotic device 102 configured to employ VSLAM techniques that include VSLAM scale estimation using optical flow sensor in accordance with an embodiment.
  • the robotic device 102 includes a wheel encoder 105, an optical flow sensor 107, a homograph computation component 506, and a scale estimation component 508.
  • the homograph computation component 506 may be configured to generate or compute an initial homography matrix (H).
  • the homograph computation component 506 may be configured to work in conjunction with the processor of the robotic device to identify prominent objects/features in a first camera frame 502, identify prominent objects/features in a second camera frame 504, and compare the identified features to each other and/or to features in test images having known dimensions and scale.
  • the homograph computation component 506 may identify correspondences based on the comparisons.
  • the homograph computation component 506 may use the identified correspondences to generate or produce the initial homography matrix (H) or homograph computation.
  • the homograph computation component 506 may be configured to generate or compute a homography matrix (H) or value.
  • the homography matrix (H) may include a set of values, a vector, a vector set, a map, a linear map, a rectangular array, a multidimensional array, or other similar information structure(s).
  • the homography matrix may be a 3x3 matrix with 8 degrees of freedom (DoF) that is calculated from four (4) point correspondences (e.g., (p1, p1'), (p2, p2'), (p3, p3'), and (p4, p4')).
  • the projected coordinates p(x', y') of a point p(x,y) may be computed from the homography matrix (H) using the following formula: x ′ y ′ 1 ⁇ H x y 1
  • the homograph computation component 506 may be configured to perform a singular value decomposition (SVD) operation on a homography matrix (H) to compute or generate a transaction information structure (T) that identifies or estimates a location and/or an orientation for the robotic device based on the images.
  • SVD singular value decomposition
  • H homography matrix
  • T transaction information structure
  • the system may be configured to compute the affinity, affine map, affine coordinates, or affine transformation via a function call, such as "Mat getAffineTransform(const Point2f src[], const Point2f dst[])."
  • the scale estimation component 508 may be configured to receive a homograph computation (e.g., homography matrix (H), transaction (T), etc.) from the homograph computation component 506, wheel data from the wheel encoder 105, and optical flow information from the optical flow sensor 107.
  • a homograph computation e.g., homography matrix (H), transaction (T), etc.
  • the received wheel transaction data and/or optical flow information may include information identifying the direction and distance moved by the robotic device between a first image frame and a second image frame. In some embodiments, the received wheel transaction data and/or optical flow information may include information identifying the distance (e.g., Euclidean distance, etc.) that the robotic device traveled between the first image frame and the second image frame. In some embodiments, the received wheel transaction data and/or optical flow information may include information identifying the length of the shortest path between a first location where the first image frame was captured and a second location where the second image frame was captured. In some embodiments, the received wheel data may include a wheel encoder transaction (Tw) information structure. In some embodiments, the received optical flow information may include an optical flow sensor transaction (To) information structure.
  • Tw wheel encoder transaction
  • the received optical flow information may include an optical flow sensor transaction (To) information structure.
  • the scale estimation component 508 may use the information received from the homograph computation component 506 to generate a first transaction (T), use the information received from the wheel encoder 105 to generate a second transaction (Tw), and determine scale (s) based on the first and second
  • the computing device may be configured to compute the values included in the wheel encoder transaction (Tw) by invoking a function, such as getWheelOdom (position, orientation).
  • the computing device may integrate the position over a short time period corresponding to a camera frame the time between camera frames.
  • the homograph computation e.g., camera transaction T
  • optical flow information e.g., optical flow sensor transaction To
  • the scale estimation component 508 may use the information received from the homograph computation component 506 to generate a first transaction (T), use the information received from the optical flow sensor 107 to generate another transaction (To), and
  • a processor in the robotic device may use the determined scale to employ homography matrix-based techniques for determining robotic device pose, the rotation angle correction, and/or other similar information.
  • the processor may use the differences in scale between the objects/features in the first and second camera frames in conjunction with displacement information to determine its current location and orientation.
  • FIG. 5B illustrates a system for computing the transaction from time (t) to time (t+1) based on optical flow information.
  • the robotic device 102 moves from location (x,y) at time t to location (x+u, y+v) at time t+1.
  • the optical flow sensor captures a first image at location (u 1 , v 1 ), a second image at location (u 2 , v 2 ), a third image at location (u 3 , v 3 ), a fourth image at location (u 4 , v 4 ), and a fifth image at location (u 5 , v 5 ).
  • the track or displacement between time t and time t+1 is (u, v), which is represented by optical flow sensor transaction (To).
  • the system may compute the transaction (T) of the main camera of the robotic device over the same time period.
  • the scale (s) of the robotic device is To/T.
  • FIGS. 5C illustrates the trajectory and pose of a camera of a robotic device over time without a scale estimate.
  • the first pose may be described as: I 0 0 1 .
  • FIG. 5D illustrates the trajectory and pose of a camera of a robotic device over time with a scale (s) estimate.
  • the first pose may be described as: I 0 0 1 .
  • the pose at time t 1 is: R 1 sT 1 0 1 .
  • the pose at time t 2 is: R 1 sT 1 0 1 ⁇ R 2 sT 2 0 1 .
  • the pose at time t 3 is: R 1 sT 1 0 1 ⁇ R 2 sT 2 0 1 ⁇ R 3 sT 3 0 1 .
  • FIG. 6 illustrates a method 600 of determining the pose of a robotic device according to an embodiment.
  • a processor of a robotic device e.g., the processor 220, etc.
  • hardware components and/or software components of the robotic device may attempt to determine the robotic device pose for the robotic device using any number and combination of sensors of the robotic device.
  • the processor of the robotic device may receive or capture a first image frame of the environment, which may include one or more target objects.
  • the first image frame may be captured by a single (i.e., monoscopic) camera.
  • the processor of the robotic device may receive or capture a second image frame of one or more target objects.
  • the processor of the robotic device may generate homograph computation values based on the first and second image frames, and generate a homography matrix (H) information structure based on the homograph computation values.
  • H homography matrix
  • the processor of the robotic device may collect or receive optical flow sensor data from the optical flow sensor.
  • the processor of the robotic device may apply an optical flow sensor image to a Lucas-Kanade component/function to generate optical flow information that include a pixel speed value and a direction value for each feature point.
  • the processor may generate an optical flow sensor transaction (To) information structure.
  • the operation of receiving optical flow information in block 608 is shown in parallel with the operations of capturing images in blocks 602 and 604 to illustrate that the reception of motion data from the optical flow sensor, as well as other motion or distance sensors (e.g., a wheel encoder), may be may occur in parallel with, in sequence with, or at another time interval with respect to the image capturing operations.
  • reception of distance measurements from the optical flow sensor, and other distance measuring sensors may occur continuously, periodically, on-demand (e.g., in response to capturing the first and/or second images), and/or in response to a determination, such as in response to a processor determination that wheel slippage is or may occur.
  • distance measuring data from the optical flow sensor may be gathered continuously, such as with the information gathered and processed by a separate processor that stores the raw or process data in a memory accessible by the processor performing VSLAM.
  • the processor performing VSLAM may access stored position or distance data when needed to perform VSLAM calculations.
  • the distance measuring information received from the optical flow sensor may be received in response to capturing images to be processed for VSLAM navigation (e.g., in blocks 602 and 604), such as beginning when the first image is captured in block 602 and terminating when the second image is obtained in block 604.
  • a processor may determine whether wheel slippage is occurring (as described herein), and in response to determining that wheel slippage has occurred or is occurring, the processor may begin receiving or accessing data from the optical flow sensor.
  • a processor evaluating images used for VSLAM may include processing images to recognize or predict when wheel slippage could occur, such as detecting a steep incline, an obstacle to be overcome, a wet surface, and the like.
  • the processor may begin receiving data from the optical flow sensor.
  • the optical flow sensor may be active at all times, or may be activated when measurement data is required or demanded by the VSLAM-executing processor.
  • the processor of the robotic device may compute or determine a scale estimation value based on a homograph computation (e.g., transaction T) and optical flow information (e.g., optical flow sensor transaction To).
  • a homograph computation e.g., transaction T
  • optical flow information e.g., optical flow sensor transaction To
  • the processor of the robotic device may determine the robotic device pose (e.g., location, position, orientation, etc.) based on the computed or determined scale estimation value.
  • FIG. 7 illustrates a method 700 of determining the pose of a robotic device according to another embodiment.
  • a processor of a robotic device e.g., the processor 220, etc.
  • hardware components and/or software components of the robotic device may attempt to determine the robotic device pose for the robotic device using any number and combination of sensors of the robotic device.
  • the operations of the method 700 may be performed after the robotic device captures images of the target objects or capture the first and second image frames in blocks 602 and 604.
  • the processor of the robotic device may generate a homograph computation (e.g., camera transaction T) based on the first and second image frames.
  • a homograph computation e.g., camera transaction T
  • the processor of the robotic device may receive wheel transaction data from a wheel encoder.
  • the received wheel transaction data may include information regarding a circumference or size of a wheel, as well as information regarding the angle of rotation, number of rotations and/or rate of rotations of the wheel.
  • the received wheel transaction data may include information identifying a displacement, distance, length between the first image frame and the second image frame.
  • the received wheel transaction data may include a wheel encoder transaction (Tw) information structure.
  • Tw wheel encoder transaction
  • the received wheel transaction data may include any or all of component vectors, location information, orientation information, coordinate values, time values, tolerance values, confidence values, precision values, and/or other similar information suitable for use in determining displacement, distance, or length and/or the probability or likelihood that the received information is precise, accurate, or correct.
  • the processor of the robotic device may analyze the wheel transaction data to detect slippage.
  • the robotic device may detect that slippage occurred by determining whether there was continuous change in the data, with discontinuous data indicative of slippage.
  • the robotic device may determine that slippage has been detected when the difference between the collected wheel transaction data and the expected wheel transaction data exceed a threshold value.
  • the robotic device may determine the expected wheel transaction data based on any or all information collected from the sensors (e.g., sensors 105, 107, 154, 160-166, etc.).
  • the processor may compare motion estimates based on wheel transaction data to motion estimates based on accelerometer data over a period of time to determine whether distance or velocity estimates determined from the two sources of data are inconsistent, such as diverging by an amount that exceeds a threshold value.
  • the processor may compare motion or distance estimates based on wheel transaction data to motion or distance estimates based on collision avoidance sensors (e.g., ultrasound sonar or radar sensors) to determine whether distance or velocity estimates determined from the two sources of data are inconsistent, such as diverging by an amount that exceeds a threshold value.
  • collision avoidance sensors e.g., ultrasound sonar or radar sensors
  • the processor of the robotic device may be configured to detect slippage by determining that the wheel pose changed (or is changing continuously), but the corresponding images captured by the camera did not change (e.g., translation T from the camera is zero) or did not change by an amount consistent with the wheel rotations. In this case, the wheels moved without causing the robotic device to move significantly (i.e., the wheels are spinning in place).
  • the processor of the robotic device may be configured to detect slippage by determining that both the wheel pose and the camera pose have changed (or are changing), but the wheel encoder transaction (Tw) is not equal to the translation from the camera (T).
  • the processor of the robotic device may collect optical flow information from an optical flow sensor in block 707.
  • the processor of the robotic device may apply the optical flow sensor data to a Lucas-Kanade component/function.
  • the processor of the robotic device may determine a scale estimation value based on the homograph computation and the optical flow information.
  • the processor of the robotic device may determine the scale estimation value based on the homograph computation and the wheel transaction data in block 712.
  • the processor of the robotic device may determine the robotic device pose (e.g., of the robotic device, of the image sensor of the robotic device, etc.) based on the scale estimation value.
  • FIG. 8 illustrates a method 800 of determining the pose of a robotic device according to another embodiment.
  • a processor of a robotic device e.g., the processor 220, etc.
  • hardware components and/or software components of the robotic device may attempt to determine the robotic device pose for the robotic device using any number and combination of sensors of the robotic device.
  • the operations of the method 800 may be performed after the robotic device captures images of the target objects or capture the first and second image frames in blocks 702 and 704.
  • the processor of the robotic device may generate a homograph computation based on the first and second image frames.
  • the processor of the robotic device may receive wheel transaction data from a wheel encoder.
  • the processor of the robotic device may compute, generate or determine a first confidence value based on the received wheel transaction data.
  • the processor of the robotic device may receive optical flow information from an optical flow sensor.
  • the processor of the robotic device may apply the optical flow sensor data to a Lucas-Kanade component/function to generate optical flow information that characterizes a transition or transaction between the two image frames.
  • the processor of the robotic device may determine, compute, generate or determine a second confidence value based on the received optical flow information.
  • the processor of the robotic device may determine whether the first confidence value (generated based on the received wheel transaction data) exceeds the second confidence value (generated based on the received optical flow information).
  • the processor may determine a scale estimation value based on the homograph computation and the wheel transaction data in block 814.
  • the processor may determine a scale estimation value based on the homograph computation and the optical flow information in block 816.
  • the processor of the robotic device may compute, generate or determine the robotic device pose (e.g., of the robotic device, of the image sensor of the robotic device, etc.) based on the scale estimation value.
  • FIG. 9 illustrates a method 900 of determining the pose of a robotic device according to another embodiment.
  • a processor of a robotic device e.g., the processor 220, etc.
  • hardware components and/or software components of the robotic device may attempt to determine the robotic device pose for the robotic device using any number and combination of sensors of the robotic device.
  • the operations of the method 900 may be performed after the robotic device captures images of the target objects in block 502 or capture the first and second image frames in blocks 702 and 704.
  • the processor of the robotic device may compute, generate or determine a homograph computation based on the first and second image frames.
  • the processor of the robotic device may receive wheel transaction data from a wheel encoder.
  • the processor of the robotic device may compute, generate or determine a first scale estimation value based on the homograph computation and the wheel transaction data.
  • the processor of the robotic device may compute, generate or determine a first confidence value based on the first scale estimation value.
  • the processor of the robotic device may receive optical flow information from an optical flow sensor.
  • the processor of the robotic device may apply the optical flow sensor data to a Lucas-Kanade component/function to generate optical flow information that characterizes a transition or transaction between the two image frames.
  • the processor of the robotic device may compute, generate or determine a second scale estimation value based on the homograph computation and the optical flow information.
  • the processor of the robotic device may compute, generate or determine a second confidence value based on the second scale estimation value.
  • the processor of the robotic device may determine whether the first confidence value (based on the wheel transaction data) exceeds the second confidence value (based on the optical flow information).
  • the processor may compute, generate or determine the robotic device pose based on the first scale estimation value in block 918.
  • the processor may compute, generate or determine the robotic device pose based on the second scale estimation value in block 920.
  • FIG. 10 illustrates a method 1000 of determining scale estimation value and the pose of a robotic device in accordance with an embodiment so to improve navigation.
  • a processor of a robotic device e.g., the processor 220, etc.
  • hardware components and/or software components of the robotic device may attempt to determine the robotic device pose for the robotic device using any number and combination of sensors of the robotic device.
  • the operations of the method 1000 may be performed after the robotic device captures images of the target objects in block 502 or capture the first and second image frames in blocks 702 and 704.
  • the processor of the robotic device may compute, generate or determine a homograph computation based on the first and second image frames.
  • the processor of the robotic device may receive wheel transaction data from a wheel encoder.
  • the processor of the robotic device may receive optical flow information from an optical flow sensor.
  • the processor of the robotic device may apply the optical flow sensor data to a Lucas-Kanade component/function to generate optical flow information that characterizes a transition or transaction between the two image frames.
  • the processor of the robotic device may determine scale estimation value based on the homograph computation and a combination of the wheel transaction data and optical flow information (e.g., optical flow sensor transaction To). For example, the processor may perform sensor fusion operations to normalize and combine the wheel transaction data with the optical flow information, and determine the scale estimation value based on the homograph computation and the results of the sensor fusion operations.
  • optical flow information e.g., optical flow sensor transaction To
  • the processor of the robotic device may compute, generate or determine the robotic device pose based on the determined scale estimation value.
  • Various embodiments also improve the accuracy of the robotic device's VSLAM capabilities using a more accurately determined robotic device pose.
  • DSP digital signal processor
  • ASIC application specific integrated circuit
  • FPGA field programmable gate array
  • a general-purpose processor may be a microprocessor, but, in the alternative, the processor may be any conventional processor, controller, microcontroller, or state machine.
  • a processor may also be implemented as a combination of receiver smart objects, e.g., a combination of a DSP and a microprocessor, a plurality of microprocessors, one or more microprocessors in conjunction with a DSP core, or any other such configuration. Alternatively, some operations or methods may be performed by circuitry that is specific to a given function.
  • the functions described may be implemented in hardware, software, firmware, or any combination thereof. If implemented in software, the functions may be stored as one or more instructions or code on a non-transitory computer-readable storage medium or non-transitory processor-readable storage medium.
  • the operations of a method or algorithm disclosed herein may be embodied in a processor-executable software module or processor-executable instructions, which may reside on a non-transitory computer-readable or processor-readable storage medium.
  • Non-transitory computer-readable or processor-readable storage media may be any storage media that may be accessed by a computer or a processor.
  • non-transitory computer-readable or processor-readable storage media may include RAM, ROM, EEPROM, FLASH memory, CD-ROM or other optical disk storage, magnetic disk storage or other magnetic storage smart objects, or any other medium that may be used to store desired program code in the form of instructions or data structures and that may be accessed by a computer.
  • Disk and disc includes compact disc (CD), laser disc, optical disc, digital versatile disc (DVD), floppy disk, and Blu-ray disc where disks usually reproduce data magnetically, while discs reproduce data optically with lasers. Combinations of the above are also included within the scope of non-transitory computer-readable and processor-readable media.
  • the operations of a method or algorithm may reside as one or any combination or set of codes and/or instructions on a non-transitory processor-readable storage medium and/or computer-readable storage medium, which may be incorporated into a computer program product.

Landscapes

  • Engineering & Computer Science (AREA)
  • Physics & Mathematics (AREA)
  • General Physics & Mathematics (AREA)
  • Computer Vision & Pattern Recognition (AREA)
  • Theoretical Computer Science (AREA)
  • Multimedia (AREA)
  • Remote Sensing (AREA)
  • Automation & Control Theory (AREA)
  • Radar, Positioning & Navigation (AREA)
  • Aviation & Aerospace Engineering (AREA)
  • Electromagnetism (AREA)
  • Image Analysis (AREA)
  • Length Measuring Devices By Optical Means (AREA)
  • Control Of Position, Course, Altitude, Or Attitude Of Moving Bodies (AREA)

Claims (14)

  1. Computerimplementiertes Verfahren zum Durchführen von VSLAM (Visual Simultaneous Localization and Mapping) durch einen Prozessor eines Robotergeräts, das Folgendes beinhaltet:
    Empfangen (602) eines ersten Bildframe von einem monokularen Bildsensor;
    Empfangen (604) eines zweiten Bildframe von dem monokularen Bildsensor;
    Erzeugen (606) einer Homographberechnung auf der Basis des ersten Bildframe und des zweiten Bildframe, wobei die Homographberechnung eine Translation des Robotergeräts zwischen dem ersten Bildframe und dem zweiten Bildframe angibt;
    Empfangen (608) optischer Flussdaten von einem optischen Flusssensor, wobei die optischen Flussdaten Informationen enthalten, die eine Translation des Robotergeräts zwischen dem ersten Bildframe und dem zweiten Bildframe angeben; und
    Bestimmen (610) einer geschätzten Skala auf der Basis der Homographberechnung und der optischen Flussdaten.
  2. Verfahren nach Anspruch 1, wobei das Erzeugen der Homographberechnung auf der Basis des ersten Bildframe und des zweiten Bildframe Folgendes beinhaltet:
    Erzeugen einer Homographie-Matrix-Informationsstruktur; und
    Erzeugen einer Transaktionsinformationsstruktur durch Ausführen einer SVD-(Singular Value Decomposition)-Operation an der erzeugten Homographie-Matrix-Informationsstruktur, wobei die Transaktionsinformationsstruktur eine Translation des Robotergeräts zwischen dem ersten Bildframe und dem zweiten Bildframe angibt.
  3. Verfahren nach Anspruch 2, wobei das Erzeugen der Homographberechnung auf der Basis des ersten Bildframe und des zweiten Bildframe ferner Folgendes beinhaltet:
    Identifizieren von mindestens vier Merkmalspunkten im ersten Bildframe;
    Identifizieren von mindestens vier entsprechenden Merkmalspunkten im zweiten Bildframe; und
    Erzeugen der Homographie-Matrix-Informationsstruktur, so dass sie die identifizierten mindestens vier Merkmalspunkte und die identifizierten mindestens vier entsprechenden Merkmalspunkte enthält.
  4. Verfahren nach Anspruch 1, das ferner das Bestimmen einer Robotergerätepose des Robotergeräts auf der Basis der geschätzten Skala beinhaltet.
  5. Verfahren nach Anspruch 4, das ferner Folgendes beinhaltet:
    Empfangen von Radtransaktionsdaten von einem Radencoder;
    Bestimmen einer radbasierten geschätzten Skala auf der Basis der Homographberechnung und der Radtransaktionsdaten;
    Bestimmen eines ersten Konfidenzwerts auf der Basis der radbasierten geschätzten Skala; und
    Bestimmen eines zweiten Konfidenzwerts auf der Basis der geschätzten Skala,
    wobei das Bestimmen der Robotergerätepose auf der Basis der geschätzten Skala Folgendes beinhaltet:
    Bestimmen der Robotergerätepose auf der Basis der geschätzten Skala als Reaktion auf die Feststellung, dass der erste Konfidenzwert den zweiten Konfidenzwert nicht überschreitet.
  6. Verfahren nach Anspruch 5, wobei:
    das Erzeugen der Homographberechnung auf der Basis des ersten Bildframe und des zweiten Bildframe das Erzeugen einer Kameratransaktions-Informationsstruktur auf der Basis des ersten Bildframe und des zweiten Bildframe beinhaltet; und
    das Bestimmen der radbasierten geschätzten Skala auf der Basis der Homographberechnung und der Radtransaktionsdaten Folgendes beinhaltet:
    Erzeugen einer Radencodertransaktions-Informationsstruktur auf der Basis der Radtransaktionsdaten; und
    Bestimmen der radbasierten geschätzten Skala auf der Basis von in der Kameratransaktions-Informationsstruktur enthaltenen Werten und in der Radencodertransaktions-Informationsstruktur enthaltenen Werten.
  7. Verfahren nach Anspruch 1, das ferner Folgendes beinhaltet:
    Empfangen von Radtransaktionsdaten von einem Radencoder;
    Bestimmen eines über einen Zeitraum aufgetretenen Schlupfbetrags auf der Basis der Radtransaktionsdaten und/oder der optischen Flussdaten; und
    Feststellen, ob der bestimmte Schlupfbetrag einen Schlupfschwellenwert überschreitet,
    wobei das Bestimmen des geschätzten Maßstabs auf der Basis der Homographberechnung und der optischen Flussdaten Folgendes beinhaltet:
    Bestimmen der geschätzten Skala auf der Basis der Homographberechnung und der optischen Flussdaten als Reaktion auf die Feststellung, dass der bestimmte Schlupfbetrag den Schlupfschwellenwert überschreitet; und
    Bestimmen der geschätzten Skala auf der Basis der Homographberechnung und der Radtransaktionsdaten als Reaktion auf die Feststellung, dass der bestimmte Schlupfbetrag den Schlupfschwellenwert nicht überschreitet.
  8. Verfahren nach Anspruch 1, das ferner Folgendes beinhaltet:
    Empfangen von Radtransaktionsdaten von einem Radencoder;
    Feststellen, ob die Radtransaktionsdaten fehlerhafte oder Ausreißerdaten enthalten;
    wobei das Bestimmen der geschätzten Skala auf der Basis der Homographberechnung und der optischen Flussdaten das Bestimmen der geschätzten Skala auf der Basis der Homographberechnung und der optischen Flussdaten als Reaktion auf die Feststellung beinhaltet, dass die Radtransaktionsdaten fehlerhafte oder Ausreißerdaten enthalten.
  9. Verfahren nach Anspruch 1, das ferner Folgendes beinhaltet:
    Empfangen von Radtransaktionsdaten von einem Radencoder;
    Bestimmen eines ersten Konfidenzwerts auf der Basis der Radtransaktionsdaten;
    Bestimmen eines zweiten Konfidenzwerts auf der Basis der optischen Flussdaten; und
    Feststellen, ob der erste Konfidenzwert den zweiten Konfidenzwert überschreitet,
    wobei das Bestimmen der geschätzten Skala auf der Basis der Homographberechnung und der optischen Flussdaten Folgendes beinhaltet:
    Bestimmen der geschätzten Skala auf der Basis der Homographberechnung und der optischen Flussdaten als Reaktion auf die Feststellung, dass der erste Konfidenzwert den zweiten Konfidenzwert nicht überschreitet; und
    Bestimmen der geschätzten Skala auf der Basis der Homographberechnung und der Radtransaktionsdaten als Reaktion auf die Feststellung, dass der erste Konfidenzwert den zweiten Konfidenzwert überschreitet.
  10. Verfahren nach Anspruch 1, das ferner Folgendes beinhaltet:
    Anwenden der optischen Flussdaten auf eine Lucas-Kanade-Komponente, um optische Flussinformationen zu erzeugen, die einen Pixelgeschwindigkeitswert und einen Richtungswert für mindestens einen Merkmalspunkt umfassen; und
    Erzeugen einer optischen Flusssensortransaktions-Informationsstruktur auf der Basis der optischen Flussinformationen,
    wobei das Bestimmen der geschätzten Skala auf der Basis der Homographberechnung und der optischen Flussdaten das Bestimmen der geschätzten Skala auf der Basis der Homographberechnung und der erzeugten optischen Flusssensortransaktions-Informationsstruktur beinhaltet.
  11. Verarbeitungsgerät, konfiguriert zur Verwendung in einem Robotergerät, wobei das Verarbeitungsgerät konfiguriert ist zum:
    Empfangen eines ersten Bildframe von einem monokularen Bildsensor;
    Empfangen eines zweiten Bildframe von dem monokularen Bildsensor;
    Erzeugen einer Homographberechnung auf der Basis des ersten Bildframe und des zweiten Bildframe, wobei die Homographberechnung eine Translation des Robotergeräts zwischen dem ersten Bildframe und dem zweiten Bildframe angibt;
    Empfangen von optischen Flussdaten von einem optischen Flusssensor; und
    Bestimmen einer geschätzten Skala auf der Basis der Homographberechnung und der optischen Flussdaten.
  12. Verarbeitungsgerät nach Anspruch 11, wobei das Verarbeitungsgerät ferner konfiguriert ist zum Erzeugen der Homographberechnung auf der Basis des ersten Bildframe und des zweiten Bildframe durch:
    Erzeugen einer Homographie-Matrix-Informationsstruktur; und
    Erzeugen einer Transaktionsinformationsstruktur durch Durchführen einer SVD-(Singular Value Decomposition)-Operation an der erzeugten Homographie-Matrix-Informationsstruktur, wobei die Transaktionsinformationsstruktur eine Translation des Robotergeräts zwischen dem ersten Bildframe und dem zweiten Bildframe angibt.
  13. Verarbeitungsgerät nach Anspruch 12, wobei das Verarbeitungsgerät ferner konfiguriert ist zum Erzeugen der Homographberechnung auf der Basis des ersten Bildframe und des zweiten Bildframe durch:
    Identifizieren von mindestens vier Merkmalspunkten im ersten Bildframe;
    Identifizieren von mindestens vier entsprechenden Merkmalspunkten im zweiten Bildframe; und
    Erzeugen der Homographie-Matrix-Informationsstruktur, so dass sie die identifizierten mindestens vier Merkmalspunkte und die identifizierten mindestens vier entsprechenden Merkmalspunkte enthält.
  14. Robotergerät, das Folgendes umfasst:
    einen Speicher;
    einen Sensor; und
    das Verarbeitungsgerät nach Anspruch 11, das mit vom Prozessor ausführbaren Befehlen konfiguriert und mit dem Speicher und dem Sensor kommunikativ verbunden ist.
EP18933675.3A 2018-09-15 2018-09-15 Systeme und verfahren zur vslam-skalenschätzung mittels eines optischen flusssensors auf einem roboter Active EP3850581B1 (de)

Applications Claiming Priority (1)

Application Number Priority Date Filing Date Title
PCT/CN2018/105898 WO2020051923A1 (en) 2018-09-15 2018-09-15 Systems And Methods For VSLAM Scale Estimation Using Optical Flow Sensor On A Robotic Device

Publications (3)

Publication Number Publication Date
EP3850581A1 EP3850581A1 (de) 2021-07-21
EP3850581A4 EP3850581A4 (de) 2022-05-11
EP3850581B1 true EP3850581B1 (de) 2026-01-14

Family

ID=69777386

Family Applications (1)

Application Number Title Priority Date Filing Date
EP18933675.3A Active EP3850581B1 (de) 2018-09-15 2018-09-15 Systeme und verfahren zur vslam-skalenschätzung mittels eines optischen flusssensors auf einem roboter

Country Status (5)

Country Link
US (1) US12124270B2 (de)
EP (1) EP3850581B1 (de)
CN (1) CN112740274B (de)
TW (1) TWI827649B (de)
WO (1) WO2020051923A1 (de)

Families Citing this family (16)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
US11358274B2 (en) * 2019-06-13 2022-06-14 Lingdong Technology (Beijing) Co. Ltd Autonomous mobile robot with adjustable display screen
US20210065374A1 (en) * 2019-08-26 2021-03-04 Organize Everything Inc. System and method for extracting outlines of physical objects
TWI732579B (zh) * 2020-06-02 2021-07-01 中華學校財團法人中華科技大學 無人載具之智慧充電方法及系統
US11718320B1 (en) * 2020-08-21 2023-08-08 Aurora Operations, Inc. Using transmission sensor(s) in localization of an autonomous vehicle
CN112284380A (zh) * 2020-09-23 2021-01-29 深圳市富临通实业股份有限公司 一种基于光流和imu融合的非线性估计方法及系统
TWI739622B (zh) * 2020-10-05 2021-09-11 國立虎尾科技大學 多光點雷射測距儀以及利用多光點雷射測距儀判斷反射光束種類及中心點位置之方法
CN112414365B (zh) * 2020-12-14 2022-08-16 广州昂宝电子有限公司 位移补偿方法和设备及速度补偿方法和设备
CN114765667B (zh) * 2021-01-13 2025-09-09 安霸国际有限合伙企业 用于多视图拼接的固定图案校准
US11756231B2 (en) 2021-06-29 2023-09-12 Midea Group Co., Ltd. Method and apparatus for scale calibration and optimization of a monocular visual-inertial localization system
US11521332B1 (en) * 2021-06-29 2022-12-06 Midea Group Co., Ltd. Method and apparatus for optimization of a monocular visual-inertial localization system
KR102704310B1 (ko) * 2021-11-03 2024-09-05 네이버랩스 주식회사 단안 거리 추정 모델 학습 방법 및 시스템
CN114237242B (zh) * 2021-12-14 2024-02-23 北京云迹科技股份有限公司 基于光学编码器对机器人进行控制的方法及装置
CN114347064B (zh) * 2022-01-31 2022-09-20 深圳市云鼠科技开发有限公司 基于光流的机器人碰撞检测方法、装置、计算机设备及存储介质
CN118163088A (zh) * 2022-12-08 2024-06-11 优思美地(上海)机器人科技有限公司 机器人运动信息的确定方法、系统、机器人和介质
CN116734908B (zh) * 2023-06-13 2025-10-03 传周半导体科技(苏州)有限公司 一种编码器芯片偏位响应测试系统
CN118357927B (zh) * 2024-06-14 2024-11-12 深圳优艾智合机器人科技有限公司 机器人的机械臂控制方法、电子设备及存储介质

Family Cites Families (18)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
US10027952B2 (en) * 2011-08-04 2018-07-17 Trx Systems, Inc. Mapping and tracking system with features in three-dimensional space
WO2016026039A1 (en) * 2014-08-18 2016-02-25 Verity Studios Ag Invisible track for an interactive mobile robot system
US10365363B2 (en) * 2015-05-08 2019-07-30 Humatics Corporation Mobile localization using sparse time-of-flight ranges and dead reckoning
CN106325118A (zh) 2015-06-30 2017-01-11 芋头科技（杭州）有限公司 机器人活跃程度智能控制系统及方法
US10173324B2 (en) 2015-11-16 2019-01-08 Abb Schweiz Ag Facilitating robot positioning
CN106803261A (zh) 2015-11-20 2017-06-06 沈阳新松机器人自动化股份有限公司 机器人相对位姿估计方法
EP3187953B1 (de) * 2015-12-30 2020-03-18 Honda Research Institute Europe GmbH Autonome arbeitsmaschine wie ein autonomer rasenmäher
US9811917B2 (en) * 2016-01-14 2017-11-07 Texas Instruments Incorporated Confidence estimation for optical flow under low light conditions
AU2017267983B2 (en) * 2016-05-20 2022-03-31 Magic Leap, Inc. Method and system for performing convolutional image transformation estimation
US20180043952A1 (en) 2016-08-12 2018-02-15 Spin Master Ltd. Spherical mobile robot with shifting weight steering
CN106529587B (zh) * 2016-08-26 2019-06-25 北京航空航天大学 基于目标点识别的视觉航向识别方法
TW201826993A (zh) 2016-12-09 2018-08-01 美商泰華施股份有限公司 具有基於環境之操作速度變化的機器人清潔裝置
CN106598339A (zh) * 2016-12-22 2017-04-26 武汉华星光电技术有限公司 触控显示器及具有该触控显示器的电子设备
JP7052788B2 (ja) * 2017-03-14 2022-04-12 日本電気株式会社 カメラパラメータ推定装置、カメラパラメータ推定方法、及びプログラム
US10705525B2 (en) * 2017-04-07 2020-07-07 Nvidia Corporation Performing autonomous path navigation using deep neural networks
CN107610175A (zh) * 2017-08-04 2018-01-19 华南理工大学 基于半直接法和滑动窗口优化的单目视觉slam算法
EP3474230B1 (de) * 2017-10-18 2020-07-22 Tata Consultancy Services Limited Systeme und verfahren für monokularen visuellen slam auf basis von kantenpunkten
CN108469823B (zh) * 2018-04-04 2020-03-20 浙江大学 一种基于单应性的移动机器人编队跟随方法

Also Published As

Publication number Publication date
TWI827649B (zh) 2024-01-01
CN112740274B (zh) 2024-08-09
EP3850581A4 (de) 2022-05-11
US12124270B2 (en) 2024-10-22
WO2020051923A1 (en) 2020-03-19
TW202013251A (zh) 2020-04-01
CN112740274A (zh) 2021-04-30
EP3850581A1 (de) 2021-07-21
US20210232151A1 (en) 2021-07-29

Similar Documents

Publication Publication Date Title
EP3850581B1 (de) Systeme und verfahren zur vslam-skalenschätzung mittels eines optischen flusssensors auf einem roboter
US11720100B2 (en) Systems and methods for utilizing semantic information for navigation of a robotic device
CN108051002B (zh) 基于惯性测量辅助视觉的运输车空间定位方法及系统
Langelaan State estimation for autonomous flight in cluttered environments
US10133278B2 (en) Apparatus of controlling movement of mobile robot mounted with wide angle camera and method thereof
Cherubini et al. Autonomous visual navigation and laser-based moving obstacle avoidance
Kanade et al. Real-time and 3D vision for autonomous small and micro air vehicles
KR101539270B1 (ko) 충돌회피 및 자율주행을 위한 센서융합 기반 하이브리드 반응 경로 계획 방법, 이를 수행하기 위한 기록 매체 및 이동로봇
US8213706B2 (en) Method and system for real-time visual odometry
Elmokadem et al. A method for autonomous collision-free navigation of a quadrotor UAV in unknown tunnel-like environments
US20130116823A1 (en) Mobile apparatus and walking robot
Ji et al. Vision-aided localization and navigation for autonomous vehicles
Noaman et al. Landmarks exploration algorithm for mobile robot indoor localization using VISION sensor
US20200125111A1 (en) Moving body control apparatus
Kozák et al. Robust visual teach and repeat navigation for unmanned aerial vehicles
Hoang et al. Combining edge and one-point ransac algorithm to estimate visual odometry
Salas et al. Collaborative object search using heterogeneous mobile robots
US20220334594A1 (en) Information processing system, information processing apparatus, and information processing program
Liang et al. Of-vo: Reliable navigation among pedestrians using commodity sensors
WO2024124421A1 (en) Robot rotation matrix estimation using manhattan world assumption
CN115930957B (zh) 一种基于多传感器融合的6自由度相对位姿估计系统
Suganthi et al. Survey on embedding economical autonomous navigation system for mobile robots and uav
US20250216851A1 (en) System and method for learning sensor measurement uncertainty
Nasir et al. Aerial robot localization using ground robot tracking–towards cooperative slam
Wang et al. Real-time visual odometry for autonomous MAV navigation using RGB-D camera

Legal Events

Date Code Title Description
STAA Information on the status of an ep patent application or granted ep patent

Free format text: STATUS: THE INTERNATIONAL PUBLICATION HAS BEEN MADE

PUAI Public reference made under article 153(3) epc to a published international application that has entered the european phase

Free format text: ORIGINAL CODE: 0009012

STAA Information on the status of an ep patent application or granted ep patent

Free format text: STATUS: REQUEST FOR EXAMINATION WAS MADE

17P Request for examination filed

Effective date: 20210209

AK Designated contracting states

Kind code of ref document: A1

Designated state(s): AL AT BE BG CH CY CZ DE DK EE ES FI FR GB GR HR HU IE IS IT LI LT LU LV MC MK MT NL NO PL PT RO RS SE SI SK SM TR

DAV Request for validation of the european patent (deleted)
DAX Request for extension of the european patent (deleted)
REG Reference to a national code

Ref country code: DE

Ipc: G06T0007246000

Ref country code: DE

Ref legal event code: R079

Ref document number: 602018088649

Country of ref document: DE

Free format text: PREVIOUS MAIN CLASS: G06T0007200000

Ipc: G06T0007246000

A4 Supplementary search report drawn up and despatched

Effective date: 20220408

RIC1 Information provided on ipc code assigned before grant

Ipc: G06T 7/70 20170101ALI20220404BHEP

Ipc: G06T 7/579 20170101ALI20220404BHEP

Ipc: G06T 7/269 20170101ALI20220404BHEP

Ipc: G06T 7/246 20170101AFI20220404BHEP

GRAP Despatch of communication of intention to grant a patent

Free format text: ORIGINAL CODE: EPIDOSNIGR1

STAA Information on the status of an ep patent application or granted ep patent

Free format text: STATUS: GRANT OF PATENT IS INTENDED

INTG Intention to grant announced

Effective date: 20250806

GRAS Grant fee paid

Free format text: ORIGINAL CODE: EPIDOSNIGR3

GRAA (expected) grant

Free format text: ORIGINAL CODE: 0009210

STAA Information on the status of an ep patent application or granted ep patent

Free format text: STATUS: THE PATENT HAS BEEN GRANTED

AK Designated contracting states

Kind code of ref document: B1

Designated state(s): AL AT BE BG CH CY CZ DE DK EE ES FI FR GB GR HR HU IE IS IT LI LT LU LV MC MK MT NL NO PL PT RO RS SE SI SK SM TR

REG Reference to a national code

Ref country code: CH

Ref legal event code: F10

Free format text: ST27 STATUS EVENT CODE: U-0-0-F10-F00 (AS PROVIDED BY THE NATIONAL OFFICE)

Effective date: 20260114

Ref country code: GB

Ref legal event code: FG4D

REG Reference to a national code

Ref country code: DE

Ref legal event code: R096

Ref document number: 602018088649

Country of ref document: DE

REG Reference to a national code

Ref country code: IE

Ref legal event code: FG4D