US20110071675A1 - Visual perception system and method for a humanoid robot - Google Patents

Visual perception system and method for a humanoid robot Download PDF

Info

Publication number
US20110071675A1
US20110071675A1 US12/564,074 US56407409A US2011071675A1 US 20110071675 A1 US20110071675 A1 US 20110071675A1 US 56407409 A US56407409 A US 56407409A US 2011071675 A1 US2011071675 A1 US 2011071675A1
Authority
US
United States
Prior art keywords
object
image
vpm
robot
range
Prior art date
Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
Granted
Application number
US12/564,074
Other versions
US8244402B2 (en
Inventor
James W. Wells
Neil David Mc Kay
Suhas E. Chelian
Douglas Martin Linn
II Charles W. Wampler
Lyndon Bridgwater
Current Assignee (The listed assignees may be inaccurate. Google has not performed a legal analysis and makes no representation or warranty as to the accuracy of the list.)
HRL Laboratories LLC
GM Global Technology Operations LLC
National Aeronautics and Space Administration (NASA)
Original Assignee
HRL Laboratories LLC
GM Global Technology Operations LLC
National Aeronautics and Space Administration (NASA)
Priority date (The priority date is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the date listed.)
Filing date
Publication date
Application filed by HRL Laboratories LLC, GM Global Technology Operations LLC, National Aeronautics and Space Administration (NASA) filed Critical HRL Laboratories LLC
Assigned to GM GLOBAL TECHNOLOGY OPERATIONS, INC. reassignment GM GLOBAL TECHNOLOGY OPERATIONS, INC. ASSIGNMENT OF ASSIGNORS INTEREST (SEE DOCUMENT FOR DETAILS). Assignors: LINN, DOUGLAS MARTIN, MCKAY, NEIL DAVID, WAMPLER, CHARLES W., II, WELLS, JAMES W.
Priority to US12/564,074 priority Critical patent/US8244402B2/en
Assigned to UAW RETIREE MEDICAL BENEFITS TRUST reassignment UAW RETIREE MEDICAL BENEFITS TRUST SECURITY AGREEMENT Assignors: GM GLOBAL TECHNOLOGY OPERATIONS, INC.
Assigned to UNITED STATES DEPARTMENT OF THE TREASURY reassignment UNITED STATES DEPARTMENT OF THE TREASURY SECURITY AGREEMENT Assignors: GM GLOBAL TECHNOLOGY OPERATIONS, INC.
Assigned to GM GLOBAL TECHNOLOGY OPERATIONS, INC. reassignment GM GLOBAL TECHNOLOGY OPERATIONS, INC. RELEASE BY SECURED PARTY (SEE DOCUMENT FOR DETAILS). Assignors: UNITED STATES DEPARTMENT OF THE TREASURY
Assigned to GM GLOBAL TECHNOLOGY OPERATIONS, INC. reassignment GM GLOBAL TECHNOLOGY OPERATIONS, INC. RELEASE BY SECURED PARTY (SEE DOCUMENT FOR DETAILS). Assignors: UAW RETIREE MEDICAL BENEFITS TRUST
Assigned to WILMINGTON TRUST COMPANY reassignment WILMINGTON TRUST COMPANY SECURITY AGREEMENT Assignors: GM GLOBAL TECHNOLOGY OPERATIONS, INC.
Assigned to GM Global Technology Operations LLC reassignment GM Global Technology Operations LLC CHANGE OF NAME (SEE DOCUMENT FOR DETAILS). Assignors: GM GLOBAL TECHNOLOGY OPERATIONS, INC.
Publication of US20110071675A1 publication Critical patent/US20110071675A1/en
Assigned to GM Global Technology Operations LLC reassignment GM Global Technology Operations LLC ASSIGNMENT OF ASSIGNORS INTEREST (SEE DOCUMENT FOR DETAILS). Assignors: CHELIAN, SUHAS E.
Publication of US8244402B2 publication Critical patent/US8244402B2/en
Application granted granted Critical
Assigned to GM Global Technology Operations LLC reassignment GM Global Technology Operations LLC RELEASE BY SECURED PARTY (SEE DOCUMENT FOR DETAILS). Assignors: WILMINGTON TRUST COMPANY
Application status is Active legal-status Critical
Adjusted expiration legal-status Critical

Links

Images

Classifications

    • GPHYSICS
    • G06COMPUTING; CALCULATING; COUNTING
    • G06KRECOGNITION OF DATA; PRESENTATION OF DATA; RECORD CARRIERS; HANDLING RECORD CARRIERS
    • G06K9/00Methods or arrangements for reading or recognising printed or written characters or for recognising patterns, e.g. fingerprints
    • G06K9/20Image acquisition
    • G06K9/32Aligning or centering of the image pick-up or image-field
    • G06K9/3216Aligning or centering of the image pick-up or image-field by locating a pattern
    • GPHYSICS
    • G06COMPUTING; CALCULATING; COUNTING
    • G06TIMAGE DATA PROCESSING OR GENERATION, IN GENERAL
    • G06T7/00Image analysis
    • G06T7/20Analysis of motion
    • G06T7/246Analysis of motion using feature-based methods, e.g. the tracking of corners or segments
    • HELECTRICITY
    • H04ELECTRIC COMMUNICATION TECHNIQUE
    • H04NPICTORIAL COMMUNICATION, e.g. TELEVISION
    • H04N13/00Stereoscopic video systems; Multi-view video systems; Details thereof
    • H04N13/20Image signal generators
    • H04N13/204Image signal generators using stereoscopic image cameras
    • HELECTRICITY
    • H04ELECTRIC COMMUNICATION TECHNIQUE
    • H04NPICTORIAL COMMUNICATION, e.g. TELEVISION
    • H04N5/00Details of television systems
    • H04N5/222Studio circuitry; Studio devices; Studio equipment ; Cameras comprising an electronic image sensor, e.g. digital cameras, video cameras, TV cameras, video cameras, camcorders, webcams, camera modules for embedding in other devices, e.g. mobile phones, computers or vehicles
    • H04N5/225Television cameras ; Cameras comprising an electronic image sensor, e.g. digital cameras, video cameras, camcorders, webcams, camera modules specially adapted for being embedded in other devices, e.g. mobile phones, computers or vehicles
    • H04N5/247Arrangements of television cameras
    • HELECTRICITY
    • H04ELECTRIC COMMUNICATION TECHNIQUE
    • H04NPICTORIAL COMMUNICATION, e.g. TELEVISION
    • H04N5/00Details of television systems
    • H04N5/30Transforming light or analogous information into electric information
    • H04N5/33Transforming infra-red radiation
    • H04N5/332Multispectral imaging comprising at least a part of the infrared region
    • GPHYSICS
    • G06COMPUTING; CALCULATING; COUNTING
    • G06TIMAGE DATA PROCESSING OR GENERATION, IN GENERAL
    • G06T2207/00Indexing scheme for image analysis or image enhancement
    • G06T2207/10Image acquisition modality
    • G06T2207/10016Video; Image sequence
    • GPHYSICS
    • G06COMPUTING; CALCULATING; COUNTING
    • G06TIMAGE DATA PROCESSING OR GENERATION, IN GENERAL
    • G06T2207/00Indexing scheme for image analysis or image enhancement
    • G06T2207/10Image acquisition modality
    • G06T2207/10028Range image; Depth image; 3D point clouds
    • GPHYSICS
    • G06COMPUTING; CALCULATING; COUNTING
    • G06TIMAGE DATA PROCESSING OR GENERATION, IN GENERAL
    • G06T2207/00Indexing scheme for image analysis or image enhancement
    • G06T2207/10Image acquisition modality
    • G06T2207/10048Infrared image
    • GPHYSICS
    • G06COMPUTING; CALCULATING; COUNTING
    • G06TIMAGE DATA PROCESSING OR GENERATION, IN GENERAL
    • G06T2207/00Indexing scheme for image analysis or image enhancement
    • G06T2207/20Special algorithmic details
    • G06T2207/20081Training; Learning

Abstract

A robotic system includes a humanoid robot with robotic joints each moveable using an actuator(s), and a distributed controller for controlling the movement of each of the robotic joints. The controller includes a visual perception module (VPM) for visually identifying and tracking an object in the field of view of the robot under threshold lighting conditions. The VPM includes optical devices for collecting an image of the object, a positional extraction device, and a host machine having an algorithm for processing the image and positional information. The algorithm visually identifies and tracks the object, and automatically adapts an exposure time of the optical devices to prevent feature data loss of the image under the threshold lighting conditions. A method of identifying and tracking the object includes collecting the image, extracting positional information of the object, and automatically adapting the exposure time to thereby prevent feature data loss of the image.

Description

    STATEMENT REGARDING FEDERALLY SPONSORED RESEARCH OR DEVELOPMENT
  • This invention was made with government support under NASA Space Act Agreement number SAA-AT-07-003. The government may have certain rights in the invention.
  • TECHNICAL FIELD
  • The present invention relates to the control of a humanoid robot, and in particular to a system and a method for automatically finding, tracking, and measuring various features and/or objects in the field of view of the humanoid robot.
  • BACKGROUND OF THE INVENTION
  • Robots are automated or autonomous devices that are able to manipulate objects using a series of rigid links, which in turn are interconnected via articulations or motor-driven robotic joints. Each joint in a typical robot represents an independent control variable, also referred to as a degree of freedom (DOF). End-effectors are the particular links used for performing a task at hand, e.g., grasping a work tool or other object. Therefore, precise motion control of a robot may be organized by the level of task specification: object level control, i.e., the ability to control the behavior of an object held in a single or cooperative grasp of a robot, end-effector control, and joint level control. Collectively, the various control levels cooperate to achieve the required robotic mobility, dexterity, and work task-related functionality.
  • Humanoid robots in particular have an approximately human structure, function, and/or appearance, whether a full body, a torso, and/or one or more appendages, with the required structural complexity of the humanoid robot being largely dependent upon the nature of the work task being performed. The use of humanoid robots may be preferred where direct interaction is required with objects, tools, or systems that are specifically intended for human use. Due to the wide spectrum of potential work tasks that may be expected of a humanoid robot, different control modes may be simultaneously required. For example, precise control must be applied within the different control spaces noted above, as well as over an applied torque, force, or resultant joint motion.
  • Conventional robotic systems may rely on task-specific end-effectors, part fixtures, and jigs to implement an assembly process. In some cases, machine vision systems may be implemented and tuned to locate a specific part, and/or to confirm completion of an assembly task or a sequential step thereof Setting up such a system may be relatively time consuming, and often requires extensive calibration, robot programming, and machine-level code compilation. State of the art machine vision systems may employ two-dimensional or three-dimensional vision systems and structured/engineered lighting techniques. The resulting assembly system may tend to be specific to just one set of like tasks and work conditions. Likewise, the set-up, commissioning, and support of the robot may be relatively challenging and time-consuming, with fixturing, tooling, and/or various material handling equipment normally required, thus resulting in added engineering, installation, and other expenses.
  • SUMMARY OF THE INVENTION
  • Accordingly, a flexible visual perception system and method are provided herein allowing a humanoid robot having multiple DOF, e.g., the over 42 DOF of a particular humanoid robot described herein, to automatically find, track, and measure a plurality of features and/or objects in the field of view of the robot, i.e., in the field of view of an array of different optical, laser, and/or infrared sensors thereof. The humanoid robot includes multiple robotic joints each being moveable through one or more DOF using at least one joint motor or other joint-level actuator(s), and may also include torque, position, and/or other sensors adapted for measuring any required control and feedback data within the robotic joints and other integrated system components.
  • A distributed controller may be adapted for controlling the various robotic joints and other integrated system components needed for executing a particular sequence, including a visual perception module (VPM) as described below, by communicating control data over multiple high-speed communication networks. The controller may include a command-level controller for transmitting a control signal commanding the performance of an automated or autonomous task or step by the robot, multiple embedded joint and system-level controllers each directly controlling the operation of a respective integrated system component, e.g., one of the robotic joints, and a joint coordination-level controller for coordinating control of the various integrated system components used for a particular sequence in response to the control signal.
  • A portion of the distributed controller, i.e., the VPM, provides the humanoid robot with precise visual perception capability via a set of sensors, e.g., optical sensors such as machine vision high-resolution cameras, general purpose and/or ambient lighting, range finding sensors such as a short range infrared range imaging device, and/or laser devices for extracting positional data of the object, and one or more image processing hardware/software devices, i.e., a respective host machine and image processing algorithm. Automation is enabled that significantly reduces the reliance of the humanoid robot on fixturing and/or other supporting mechanical structure for its basic operation, allowing tasks to be completed using tools and infrastructure similar to those used for a completely manual process.
  • The controller includes the VPM, which may include a host machine, server, or other suitable processing device, as well as an image processing algorithm embodying the method of the present invention, and a plurality of sensors configured for collecting the required image and/or range image data that is ultimately processed using the host machine. The VPM may also include an auxiliary lighting system enabling automation without the need for application-specific engineered lighting systems of the type known in the art. The VPM, by executing the algorithm as set forth herein, provides precise object/feature location and guidance in challenging threshold ambient lighting conditions, i.e., threshold low-contrast black-on-black or white-on-white feature conditions, for a range of parts that might be encountered in conducting a particular automated or autonomous task.
  • The algorithm automatically adapts the exposure time of a set of optical cameras to prevent feature data loss in threshold dark or light areas of a collected image. As noted above, the algorithm also provides black-on-black or white-on-white feature finding capability of like-contrast surfaces under the threshold ambient lighting conditions. This allows for the optimization of the sensors luminance dynamic range based on the need for the application to discern very dark or light features in the field of view. Additionally, the VPM may be used to “teach” or “show” the humanoid robot new automated task sequences, e.g., by showing the robot various data points, parts, poses, features in the environment, calibration artifacts, and/or sequential paths within a particular work area or work envelope. These and other image processing capabilities are set forth in further detail herein.
  • In particular, a robotic system is provided having a humanoid robot with a plurality of robotic joints each being moveable using at least one of a plurality of actuators. The system also includes a visual perception module (VPM). The VPM is adapted to visually identify and track an object in the field of view of the robot under threshold ambient lighting conditions, and includes a plurality of optical devices adapted for collecting an image of the object, at least one position extraction device, such as a laser device and a range imaging device adapted for extracting positional information of the object, a host machine, and an image-processing algorithm. The algorithm is executed via the host machine to automatically adapt an exposure time of the optical devices to prevent feature data loss of a captured image of the object under the threshold ambient lighting conditions, both for very dark or very light features of interest at the extremes of the luminance range of the sensor.
  • A method is also provided for identifying and tracking an object in the field of view of the humanoid robot under the same threshold ambient lighting conditions. The method includes using optical devices to collect an image of the object, using either or both of the laser and radar-based range imaging devices to extract positional information of the object, and processing the image and positional information using a host machine. The method also includes using the host machine to process the image and positional information using an image processing algorithm, to automatically adapt an exposure time of the optical devices, and to thereby prevent feature data loss of the image under the threshold ambient lighting conditions.
  • The above features and advantages and other features and advantages of the present invention are readily apparent from the following detailed description of the best modes for carrying out the invention when taken in connection with the accompanying drawings.
  • BRIEF DESCRIPTION OF THE DRAWINGS
  • FIG. 1 is a schematic illustration of a robotic system having a visual perception module (VPM) in accordance with the present invention;
  • FIG. 2 is a schematic illustration of the VPM shown in FIG. 1;
  • FIG. 3 is a graphical flow chart describing a portion of an algorithm of the VPM shown in FIG. 2;
  • FIG. 4 is a graphical flow chart describing another portion of an algorithm of the VPM shown in FIG. 2;
  • FIG. 5 is a schematic illustration of the humanoid robot of FIG. 1 viewing an object through multiple views or angles;
  • FIG. 6 is a schematic perspective view illustration of a calibration artifact or object that can be mounted to a hand of the robot shown in FIG. 1; and
  • FIG. 7 is a schematic perspective view illustration of a point/pose teaching pointer tool usable for vision-based task training of the robot of FIG. 1.
  • DESCRIPTION OF THE PREFERRED EMBODIMENT
  • With reference to the drawings, wherein like reference numbers refer to the same or similar components throughout the several views, FIG. 1 shows a robotic system that includes a dexterous humanoid robot 10, which in turn is controlled via a distributed control system or controller (C) 24. The robot 10 is adapted to perform one or more automated or autonomous tasks with multiple degrees of freedom (DOF). According to one embodiment, the robot 10 is configured with a plurality of independently and interdependently-moveable robotic joints, such as but not limited to a shoulder joint, the position of which is generally indicated by arrow A, an elbow joint (arrow B), a wrist joint (arrow C), a neck joint (arrow D), and a waist joint (arrow E), as well as the various finger joints (arrow F) positioned between the phalanges of each robotic finger 19.
  • Each robotic joint may have one or more DOF. For example, certain joints such as the shoulder joint (arrow A), elbow joint (arrow B), and wrist joint (arrow C) may have at least two DOF in the form of pitch and roll. Likewise, the neck joint (arrow D) may have at least three DOF, while the waist (arrow E) may have one or more DOF. Depending on task complexity, the robot 10 may move with over 42 DOF. Each robotic joint contains and is internally-driven by one or more actuators, e.g., joint motors, linear actuators, series elastic actuators, other rotary actuators, and the like.
  • The humanoid robot 10 may include a head 12, torso 14, waist 15, arms 16, hands 18, fingers 19, and thumbs 21, with the various joints noted above being disposed within or therebetween. The robot 10 may also include a task-suitable fixture or base (not shown) such as legs, treads, or another moveable or fixed base depending on the particular application or intended use of the robot. A power supply 13 may be integrally mounted to the robot 10, e.g., a rechargeable battery pack carried or worn on the back of the torso 14 or another suitable energy supply, or may be external to the robot and connected via electrical cables, to provide sufficient electrical energy to the controller and actuators of the various joints and other integrated system components.
  • The distributed controller 24 provides precise motion and systems control of the humanoid robot 10, i.e., over the various joints and other integrated system components such as relays, modules, internal or external circuit components, etc., including control over the fine and gross movements needed for manipulating a work piece, tool, or other object 20 that may be grasped by the fingers 19 and thumb 21 of one or more hands 18. The controller 24 is able to independently control each robotic joint in isolation from the other joints, as well as interdependently control a number of the joints to fully coordinate the actions of the multiple joints in performing a relatively complex work task.
  • The controller 24 may include multiple digital computers or data processing devices each having one or more microprocessors or central processing units (CPU), read only memory (ROM), random access memory (RAM), electrically-programmable read only memory (EPROM), a high-speed clock, analog-to-digital (A/D) circuitry, digital-to-analog (D/A) circuitry, and any required input/output (I/O) circuitry and devices, as well as signal conditioning and buffer electronics. Individual control algorithms resident in the controller 24 or readily accessible thereby may be stored in ROM and automatically executed at one or more different control levels to provide the respective control functionality.
  • The controller 24 may communicate across multiple communication nodes or connection points, and over multiple high-speed communication networks, e.g., Ethernet, multi-point low-voltage differential signaling (M-LVDS) busses, backplanes, and the like. The nodes may be embodied as the various modems, hubs, bridges, computers, data routers, workstations, servers, etc. controlling the flow of information within the robotic system.
  • Still referring to FIG. 1, the controller 24 is “distributed” in the sense that command-level control may be provided at a top or macro-level, with joint coordination control provided at a lower or intermediate-level. The intermediate-level controller in turn controls each joint involved in the control of a particular task or motion, and does so via direct communication with multiple low-level controllers.
  • Therefore, the controller 24 may include a command-level controller referred to herein as a system commander 25, and a real-time joint and system coordination controller referred to hereinafter as the brainstem 27. The controller 24 may also includes multiple embedded joint-level controllers 29 which, as the description implies, are each embedded within the structure of the robot 10, e.g., within the arms 16, in close proximity to the particular joint or other integrated system component being controlled thereby. Joint-level controllers 29 may be configured as one or more printed circuit board assemblies (PCBA) providing joint sensing, signal processing, and immediate actuator control.
  • Still referring to FIG. 1, control data and the feedback data, represented by double-arrow 11, may be exchanged between the humanoid robot 10 and the various levels of the controller 24, including a visual perception module (VPM) 30V as described below, in order to ensure precise complex motion and systems control of the robot. The controller 24 may include a user interface 22, e.g., a control panel, display screen, and/or other human-machine interface (HMI) device. User interface 22 may be configured to separately facilitate access to the humanoid robot 10 at the operational and the programming levels.
  • Within the scope of the present invention, the controller 24 may also include application modules 30. Each application module is configured to control a certain environmental or peripheral sensory application. In one embodiment, one of the command modules is configured as the VPM 30V, which provides the robot 10 with flexible visual perception capability supporting dexterous tasks. Such environmental perception/sensing capability or other related control functionality of the VPM 30V may be relayed through the various control levels of the controller 24 via the multiple high-speed networks described above.
  • Referring to FIG. 2, the VPM 30V includes hardware, software, and sensors all linked to the robot 10 of FIG. 1. The VPM 30V enables vision-based operation of the robot 10 without the need for application-specific engineered lighting systems to illuminate the work area and/or any object(s) being acted upon by the robot. The VPM 30V provides object or feature location and guidance under conditions of challenging light, low contrast surface conditions, etc., without requiring fixturing or other infrastructure in the conventional manner.
  • According to one embodiment, the VPM 30V may include a processor or host machine 32, software in the form of an image processing algorithm 100, and sensors in the form of optical sensors or cameras 34 and a positional data extraction device(s), e.g., a range-finding sensor or infrared (IR) range imager 36 and/or laser projectors 42. According to one embodiment, the cameras 34 may be configured as high-resolution cameras of at least approximately five mega-pixels, although different cameras providing other suitable resolutions may also be used to achieve a similar result.
  • The host machine 32 may be configured as a digital computer having one or more microprocessors or central processing units (CPU), read only memory (ROM), random access memory (RAM), electrically-programmable read only memory (EPROM), a high-speed clock, analog-to-digital (A/D) circuitry, digital-to-analog (D/A) circuitry, and any required input/output (I/O) circuitry and devices, as well as signal conditioning and buffer electronics. The algorithm 100 may be stored in ROM aboard the host machine 32, or in another location accessible by the host machine, and automatically executed thereby to provide the respective visual perception functionality as set forth below.
  • The host machine 32 may be linked to the cameras 34 and IR range imager 36 via a high-speed connection 37 and respective hubs 38A, 38B. In one embodiment, the hub 38A may be configured as an Ethernet hub, while the hub 38B may be configured as a universal serial bus (USB) hub, although other hub configurations may also be used without departing from the intended scope of the invention. The cameras 34, the IR range imager 36, and the laser projectors 42 may be contained or encapsulated within the head 12 of the robot 10, with a pair of the cameras being separated by a calibrated distance to provide three-dimensional (3D) stereo vision of an object 20 (see FIG. 1) positioned at a focal point 40 in the robot's field of view.
  • The VPM 30V may also include an auxiliary lighting device 44 having the functionality described below. The projectors 42 and the lighting device 44 may be mounted with respect to a frame 46 that is likewise encapsulated within the head 12 of the robot 10 shown in FIG. 1. Such a frame 46 may also be configured to support the IR range imager 36 and each of the cameras 34. In this manner, data processing is conducted by the host machine 32 while the cameras 34, IR range imager 36, lighting device 44, and laser projectors 42 are securely positioned within the head 12.
  • Still referring to FIG. 2, the VPM 30V includes a 4-axis redundant gaze control unit (GCU) 48, including an axis provided by the waist 15 of FIG. 1. The head 12 is moveable with at least two DOF, i.e., rotation and elevation. One or more actuators 50, e.g., joint motors, series elastic actuators, rotary actuators, and/or linear actuators, allow the head 12 to rotate or tilt as needed to adjust the focal point 40. Precise motion control over the gaze is provided via the controller 24 of FIG. 1 and the GCU 48 of FIG. 2, with one joint-level controller 19 being embedded in the neck joint of the robot 10 of FIG. 1 and dedicated to providing sensor and data processing for at least one DOF of the joint.
  • The VPM 30V of FIG. 2 used in conjunction with the 42+ DOF humanoid robot 10 shown in FIG. 1 provides automation capabilities that may significantly reduce the reliance on fixturing and mechanical structure, as noted above. This capability may enable automated task execution with a set of little or no more infrastructure requirements than would a completely manual process. Guiding by perception rather than fixturing may allow the mobility of the robot to have substantially relaxed requirements for repeatability and accuracy, and may exploit the task-related guidance perception capability. The flexible robot perception capability provided by the VPM 30V applies a strategy that includes highly-flexible robots/end-effectors, feature perception, and integration with human users or operators performing manual work tasks. A visual perception system that is flexible across an array of automated applications may facilitate the assignment of the robot 10 to a particular set of tasks as required by the production mix and business needs.
  • Referring to FIG. 3 in conjunction with the structure of FIGS. 1 and 2, the algorithm 100 of FIG. 2 includes a portion or algorithm 100A that attempts to acquire properly-exposed images from one or more cameras 34 for use in vision tasks. Algorithm 100A starts at step 102, wherein exposure times are obtained for use by the cameras 34 when acquiring images. In one embodiment, the obtained exposure time values may be exposure times used previously by the same algorithm 100A, as these values will produce properly-exposed images if lighting conditions have not changed.
  • The algorithm 100A then proceeds to step 104, where image exposure times are set for the cameras 34. Once set, the algorithm 100A proceeds to step 106, wherein images of the object 20, e.g., a work piece, tool, or other object being acted upon by the robot 10, are acquired by the VPM 30V using the cameras 34.
  • Timing of image acquisition may be controlled either by software or by an external hardware signal. If multiple cameras are used, the synchronization signal is sent to all cameras simultaneously. This allows accurate stereo vision calculations to be performed since the acquisition of the two images from both stereo cameras was triggered at the same instant in time. The synchronization signal may also trigger the recording of other information, such as the position of the robot 10 in either joint or Cartesian space. Alternatively, the image acquisition process may record the times when the images are taken, for example so that subsequent processing can compensate for robot motion between acquisitions of individual images.
  • The images are temporarily recorded in memory of the host machine 32. Image data preservation exposure control may be provided by automatically processing and analyzing the upper and lower intensity components of the acquired images. The algorithm 100A then proceeds to step 108.
  • At step 108, image brightness may be computed using the host machine 32. Image brightness may be calculated based on a single camera 34, or it may be based on simultaneous images from two or more cameras. In the latter case, the measured brightness may be calculated in all images individually, or it may be a mathematical combination (such as an average) of the brightness of the individual images. If the brightness is determined to be satisfactory for the task at hand, then the algorithm 100A proceeds to step 112, otherwise it proceeds to step 110, wherein exposure times are adjusted as needed.
  • Within the scope of the present invention, the nature of the image-brightness calculation noted above is task-dependent. For example, when processing dark features on a dark image, the brightness of the image may be measured by taking a histogram of the intensity of the pixels in the whole image, ignoring a fixed percentage of the darkest pixels, for example the darkest 1 percent, and then looking at the darkest pixel remaining If that pixel's intensity value is within an acceptable range, then the whole image may be deemed acceptable. Similar calculations can be performed for white-on-white processing, except that the brightness measure would be obtained by ignoring the brightest 1 percent of the pixels in the image.
  • If at step 106 it is determined that the image brightness is not acceptable, then the algorithm 100A proceeds to step 110, wherein the exposure time is adjusted, e.g., new exposure times are computed. The exposure times may be increased or decreased as needed, depending on whether the image is too dark or too bright, respectively. This computation may take into account the amount by which the images are too bright or too dark, the distribution of pixel intensities, and camera-imposed limits on the exposure times.
  • If more than one camera 34 is used, the exposure times for the cameras may be adjusted individually, based on the brightness of the images obtained by that camera, or they may be kept in a fixed relationship to one another. In the latter case, the exposure times can be based on a single camera 34 or several such cameras, as in step 108. Keeping the exposure times in fixed relationships compensates for different camera gain settings, iris openings, or other factors that influence image brightness. The relationship between camera exposure times typically may be expressed as a ratio of the exposure times, with the ratio being obtained using a calibration procedure that measures the relative brightness of the images obtained when the cameras 34 are pointed at identical scenes. If the cameras 34 are believed to be identical, then that ratio can be assumed to be equal to one, so that all cameras have identical exposure times.
  • Since lighting conditions can frequently be mostly stable over the timeframe in which several images are acquired by the cameras 34, the last converged exposure time may be fed forward to the subsequent image acquisition cycle at step 112. Exposure feed-forwarding provides an optimum initial value for the exposure time, and increases the probability that only one additional image acquisition cycle will be needed to acquire a subsequent useable image.
  • Referring to FIG. 4, the algorithm 100 of FIG. 2 may also include a portion or algorithm 100B for analyzing substantially black-on-black images for the purpose of locating certain features on monochromatic objects, e.g., a black indenture on a black work piece. This capability relies on the use of the aforementioned image data preservation exposure control, i.e., algorithm 100A of FIG. 3, to ensure that image data is not lost due to excessively short or long exposure times.
  • Algorithm 100B begins with step 114, wherein a maximum threshold area of interest is selected in a larger image, i.e., the image collected at step 102 of algorithm 100A, for further evaluation. At step 116, the selected area is automatically evaluated using a histogram to thereby determine the extent of the range of illumination of the area of interest. The algorithm 100B then proceeds to step 118.
  • At step 118, the evaluated range may be proportionally expanded and applied to the test area or area of interest to enhance contrast. A histogram of the intensity levels in the test area are evaluated based on the range and distribution of intensities. Based on the upper and lower limits of the distribution, which frequently is a subset of the available dynamic range of the image, the range of the distribution may be expanded by linearly adding intensity offsets to the initial values, such that the distribution flattens and fills the entire dynamic range of available intensity values. This technique effectively flattens and enhances contrast for the conditions of the test area where the intensity is mostly dark or mostly light.
  • The algorithm 100B then proceeds to step 120, wherein image gradients are detected such that features may now be discriminated in the image. Other image processing steps can also be performed at this step to discard selected image data that may interfere with operations in subsequent steps. These operations may include but are not limited to “thresholding” the image in a certain range of values and “blob processing” the area of interest based on the shapes of the threshold areas, as the term “blob processing” is understood in the art.
  • At step 122, this may include, but is not limited to operations that now operate on the simplified image that was the product of the previous processing in steps 118 and 120. Features of interest appearing in the simplified image may be matched based on contour models or other suitable means. In this way, the robustness and speed of the feature recognition and/or other operations are significantly facilitated by the simplifications performed in the previous steps.
  • Referring to FIG. 5, the robot 10 is shown holding an object 20, e.g., a torque sensing fastener driver or other work tool, in a hand 18. Most of the sensors of the VPM 30V (see FIG. 2) are encapsulated within the head 12 of the robot 10, as noted above. Additional cameras 34A and 34B may be respectively mounted to a forearm, palm, or chest area of the robot 10 and to an overhead structure 45, e.g., a ceiling, wall, stanchion, or other suitably stable overhead surface or location within a robotic work cell or envelope. The camera 34B may be used, for example, to determine conveyor status or global scene recognition. Overhead cameras provide a coarse localization of objects of interest or the status of the process, such as but not limited to that of any desired parts.
  • Within the scope of the invention, automatic integration is provided of different views of a task conducted with respect to the object 20 in order to provide an optimal point of view. One of the more important views to have is the point of view of a palm of hand 18 facing the object 20. This particular view allows more precise centering in the approach of the hand 18 to the grasp before contact is made with the object 20, and before any tactile senses of the robot 10 are usable. Implicit in these techniques is the co-registration of the ranges and fields of view of the various sensors.
  • As will be recognized by those of ordinary skill in the art, processing of vast amounts of array data is not only time consuming, but can also lead to task-irrelevant or false matches. Spatial attention mechanisms, on the other hand, can direct limited computational resources to areas of the search space that are task relevant and also less likely to yield false matches. As used herein, the term spatial attention mechanisms may refer to range, epipolar/stereo geometry, and/or field of view-cueing techniques.
  • Referring again to FIG. 2, range cueing utilizes relatively fast, low-resolution sensors to direct slower, higher-resolution sensors to areas of a search space which are task-relevant or that contain potential matches. In one embodiment, a fast, low-resolution sensor in the form of the IR range imager 36 may be used to provide time-of-flight (TOF) data that cues the faster, high-resolution stereo cameras, i.e., the cameras 34, which may in turn cue the placement of structured light as needed from the auxiliary area lighting device 44.
  • An example of range cueing based on task-relevancy would be to ignore areas that are outside the work envelope of the robot 10 and segment the subsequent high resolution identification of a feature as that location in the range image is translated into the field of view of the hi-res cameras 34. An example of range cueing based on potential matches would be to ignore volumes that are occluded by other objects that are at a closer range to the sensors.
  • In addition, the IR range imager 36 and the cameras 34 offer complementary features, such as the ability of the IR range imager to localize featureless surfaces. This can be further refined by structured light from the auxiliary lighting device 44, which is a task which the stereo cameras 34 cannot perform alone, and the ability of stereo cameras 34 to handle large amounts of IR illumination, e.g., conditions where the task area is illuminated with direct sunlight which overpowers the structured IR light source from the IR range imager resulting in the loss of the range image output from the sensor.
  • Epipolar cueing applies triangulation constraints to direct searches from the field of view of one camera 34 to only a handful of rows within its sister camera, i.e., the other camera 34. The width of the search rows may be further limited by knowing the approximate depth, which can be provided by the IR range imager 36, or the size of the object of interest. Field of view cueing refers to locating smaller features or objects based on their relative position to larger features or objects.
  • For example, locating the tip of a cable, or the locator hole of a component, across the entire field of view, may be computationally expensive and prone to false matches. However, searching for the same cable tip or locator after using an offset from a cable toggle or component edge may lead to faster and potentially more reliable matches. Finding a small feature or object in one camera's field of view can then direct stereo triangulation based on approximate depth, which can be provided by the IR range imager 36, or size of the object of interest. Implicit in these cueing techniques is the co-registration of sensors' ranges and fields of view.
  • Referring to FIG. 6 in conjunction with FIG. 5, eye-hand calibration may be provided by using an end-effector-mounted object 220 having a geometric pattern 60 to extract the 3D location and 6 DOF pose of a hand 18 of the robot 10. This calibration object 220, preferably an object with easily discerned high-contrast markings 62 as shown in FIG. 6, may be fixed via mounting holes 63 to the center of an end-effector's grasp, i.e., a palm of a hand 18, or reliably held in the grasp of robot 10 with known dimensional repeatability. This allows instrumentation of the location of an end-effector positioned n the visual field of view of the VPM 30V.
  • The 3D location of the position and orientation (6 DOF pose) of object 220 with respect to the head 12 may be determined by the VPM 30V, preferably using high-resolution stereoscopic imaging techniques. To perform calibration, the robot 10 may be commanded to move the calibration object 220 to multiple locations in its field of view and to fix its gaze on the calibration object. At each location, joint angles measured by the robot's internal angle sensors may be recorded, as is the 6 DOF pose of the calibration object 220 in a coordinate frame fixed in head 12.
  • A calibration algorithm (not shown) may be executed to align the visual space of the robot 10 with the geometric movement space, preferably using a least-squares fit to match the predictions of a kinematic model of the eye-hand relationship to the various measured poses of the calibration object 220. Alternately, other error measures such as the sum of absolute value of error components instead of the sum of squares could be minimized. In one embodiment, the calibrated parameters may include zero offsets of the measured joint angles in a kinematic chain leading from the robot hand 18, up the arm 16 (see FIG. 1), and along the neck of the robot 10 to the frame 46 (see FIG. 2). After measuring a sufficient number of poses with sufficient accuracy, it is possible to also make fine adjustments to the kinematic models of the links of the robot. This technique exploits the meteorology capabilities of the VPM 30V and avoids the need for other auxiliary meteorology devices to perform or check the calibration.
  • Eye-hand calibration can be extended to both hands 18 and also extended to include coordination with externally-mounted cameras, such as the camera 34B. To extend calibration to the second hand, the calibration object is moved to the second hand and another set of data comprising joint angles and the corresponding visual measurement of the object pose is collected, again at multiple hand locations. The portions of the kinematic chain in common to the two hand-to-eye chains, namely the neck joints and any intervening links, may be calibrated by applying least squares minimization to the ensemble of measurements from both arms.
  • To calibrate the rest of the robot's kinematic chain, such as waist joint 15 or other lower extremities, the calibration object 220 may be placed in a known location on any link that is more topologically-distant from the head 12, such that the kinematic chain from the head to the calibration object includes the joint to be measured. If the topologically-distant link is within reach of the robot 10, the placement of the calibration object 220 on the link may be accomplished automatically by the robot, otherwise the intervention of a human operator may be necessary.
  • Finally, to calibrate any externally-mounted camera, e.g., camera 34B, the robot 10 may show the calibration object 220 to the externally-mounted camera(s), recording both the internally measured joint angles and the location of the calibration object as measured by the external camera. Comparison of the internally computed pose of the calibration object 220, using the previously calibrated eye-hand kinematic model, with the pose as measured by the external camera, e.g., 34B, reveals the 3D position and orientation of the external camera with respect to the robot 10. Measurements of multiple such poses may be resolved using least-squares to improve the accuracy of the calibration. In this manner, subsequent measurements from that camera of other objects or features of interest may be accurately interpreted in the robot's internal reference frame, facilitating accurate motion with respect to these, such as would be required to pick up or otherwise interact with such an object.
  • Referring to FIG. 7, a 3D and 6 DOF pose-teaching pointer tool 66 may enable “teaching” or “showing” the robot 10 various system locations and/or poses in the robot's field of view. This pointer tool 66 can be scaled to any practical size necessary that is compatible with the process at hand. The pointer tool 66 may include a pointer head 64 and a handle 68 connected via a connecting portion 65, e.g., a bar, rod, or other rigid extension. A series of visual tracking fiducial marks 162 are located on the head. Using the pointer tool 66 may enable fast and easy point teaching of various points and/or poses in Cartesian space without the need to enter the immediate work envelope of the robot 10. Once an operator points the pointer tool 66 to an area of interest, i.e., a point 140, an image is captured and processed as described above, and the center location and pose of the pointer head 64 is determined in a spatially-calibrated field of view of the cameras 34.
  • This location and pose may be mathematically transposed to a tip of the pointer tool 66 by a known offset based on the physical construction of the pointer head 64. It is this location and pose of where the pointer tool 66 is pointing that is returned to the controller 24 for further use as a “taught point” that the robotic system of FIG. 1 can use in subsequent operations. Continuous updates of the pointer location at high frame rates results in the capture of a path that may also be used to follow or otherwise describe geometry saved for future use by the robotic system. This exploits the meteorology capability of the flexible perception system to significantly facilitate the set-up and configuration of the system for new tasks.
  • Referring again to FIG. 2, an integrated structured light system may be provided via the auxiliary lighting device 44. Twin intersecting planes of laser light from the laser projectors 42 may be used to extract position information from an object 20 having relatively featureless surfaces and contours where conventional stereo disparity techniques are unusable. A range function may be supported by finding the center of the intersection of the two planes of light in both a right and a left image, and then applying binocular calibration and the intersecting lines of sight to the distinct feature location of the planar light intersection in order to determine the 3D location of the surface where the reflected light is observed.
  • 3D scanning and instrumented surface model reconstruction may be accomplished by panning a laser light stripe across a surface with the 4-axis redundant GCU 48, recording stripe position with calibrated cameras and reconstructing an image model. The GCU 48 may also be used to gain multiple views of essentially the same scene. Evaluation of these different views can also provide an adaptive target gaze command for the alternative points of view. This enables active gaze control vision for the implementation of object disambiguation techniques, range calibration check vs. known camera position movement, and occlusion resolution by “try again” gaze control.
  • The location of an object 20 that a robot grasps in its end-effector, e.g., in a hand 18, is important to know if the end-effector is not capable of precisely fixturing the object in its grasp. One goal of a flexible end-effector is to at least control the object 20 without the object slipping from the initial point of grasp. This is a bit different from a fixture grasp since the precise location of the object 20 held in the grasp is not known. As objects in the form of tools are continually reused in the performance of certain work tasks, marks may be added to the visible areas of the object 20 to facilitate the reliable determination of the location or pose in the field of vision of the VPM 30V. These marks can be unique to each object 20, and can also be unique within each pose of the object to facilitate the generation of an approach path and grasp strategy for the found random location of the object within the work space.
  • Within the scope of the invention, high frame rate, high-resolution dual-camera tracking, i.e., tracking of the cameras 34 encapsulated within the head 12, is enabled by operating one of the cameras at a lower resolution and a higher frame rate than the other. The image from the first camera may be used to segment a desired search area in a field of view, and to provide a “coarse lock” onto the target's location for subsequent image updates and incremental changes of location in the field of view. The location of this target segmentation may be transferred into the image plane of the second high resolution camera for image processing by the host machine 32, including detailed analysis and location of smaller features in the same field of view.
  • This practice may allow for a more rapid and directed application of image analysis tasks at a higher resolution than would normally possible if the entire field of view were to be evaluated for the same task. The high-rate tracking can run asynchronously with respect to the high-precision image frame capture. Images may be time-stamped at the time of capture, and the results may be synchronized at the time of evaluation to enable responsive and robust object tracking with precise results. Sampling of Cartesian or joint position feedback can also be time-stamped such that the closest time-distance alignment of the image data can be attained. This technique can also be applied when one or more cameras 34 are triggered by a dedicated signal such that there is close coordination of image acquisition and the recorded time-stamp for that image and other data. This provides the most accurate synchronization of images and the robot's instantaneous position and other associated data.
  • The process of image segmentation by gray level value or by color refers to locating smaller features or objects based on the relative position of that feature or object to larger features or objects. For example, locating a tip of a screw across the entire field of view of the robot 10 can be computationally expensive and prone to false matches. However, searching for the same screw tip after using an offset from a yellow power drill/screwdriver for dark image regions, by way of example, leads to faster and potentially more reliable matches.
  • Furthermore, using the IR range imager 36 may provide a multi-spectral component that is not typically found in conventional machine vision systems, e.g., two surfaces may appear similar in the visible portion of the electromagnetic spectrum while being more clearly differentiable in the IR portion of the electromagnetic spectrum; heat and water content are two cases were IR imaging can help. Implicit in these cueing techniques is the calibration of a whitebalance point of a sensor. This calibration would change based on ambient lighting conditions, e.g., differing amounts of sunlight throughout the day, or different overhead lights used in different parts of a factory, and can be facilitated using a calibration artifact of known reflectance, e.g., a Macbeth color checker or other suitable calibration artifact.
  • Part of the flexibility of the VPM 30V shown in FIG. 2 requires that the visual perception capabilities of the VPM remain reliable under changing ambient lighting conditions. Strict conditions of structured or engineered lighting may not be practical for a robotic system providing many vision-based functions for a series of diverse complex assembly tasks. An adaptive thresholding technique is therefore provided that exploits partial knowledge of the conditions in the field of view, and then uses this information to extract the proper threshold(s) to be used in further segmenting the image for subsequent algorithm steps.
  • The VPM 30V locates areas of interest within the image for the objects to be segmented. These small areas are selected for their representative grayscale or color intensity values based on the evaluation of the dominant peak of area's the histogram. This information is then used to calculate the threshold level that will differentiate the object area with the background or other objects in the field of view.
  • The GCU 48 plays an important role in pointing the sensors and lighting devices in the direction of the desired field of view. The servo controlled “head” and “neck” components of the robot 10 can be directed under program control to specific predefined locations or to search and/or track features in the field of view as necessary for the task at hand. Auto-gaze functionality can also reduce the need to explicitly program the orientation and configuration of the head and neck axes, since interaction between features found and/or tracked in the field of view of the robot 10 can be used to direct the head and neck through the established kinematic calibration of the robot and the spatial frame of reference of VPM 30V.
  • Due to a less structured task environment of the robot 10, only an approximate location relationship may be assumed when initially searching for features or objects in the field of view. In the first segmentation of the search, an easy to find reference feature that is associated with detailed target features is located somewhere in the field of view. If not located, then a task error may be generated or the gaze commanded to change, and the search may be repeated on the new image(s). Based on the task at hand, the detailed features may be found or the gaze may be commanded such that the target features can be seen in a specific area of the field of view.
  • This may be useful in the instance when the target features are found, but are desired to be seen in the left side of the image, by way of example, such that other features can also be seen on the right side of the image such that a distance may be measured or an object can be simultaneously seen by both cameras 34 in stereo for further location or pose determination. This can be done in the progressive sequence in just a few images and gaze control commands or a high frame rate tracking of a moving target and continuous gaze tracking servo. The continuous gaze tracking servo of target features is important so that moving targets can stay in the field of view and/or structured or engineered lighting can also be reliably tracking the features of interest.
  • A truly flexible end-effector may have a plurality of jaws or fingers, e.g., the fingers 19, which may have multiple DOF as needed to perform a range of grasping tasks. These grasping components may be servo-controlled, actuated by linkage, or any combination of implementation techniques, as explained above. End-effector DOF angle or displacement relationships and position feedback devices should be properly calibrated to their actual mechanical absolute location.
  • The VPM 30V may be used to support this calibration function by observing the pose and movement of individual grasping components with respect to a non-moving reference on the end-effector structure. Alignment marks or grasping link components may be found in the image, and their geometric relationship may be determined visually while the end-effector is held in a favorable pose within the field of view.
  • In navigation and reaching alike, obstacle avoidance techniques may be used to avoid damage to the robot 10 and/or an object 20, as well as inadvertent contact with other objects or operators. Obstacle avoidance is a dynamic process because both ego-motion and external motion are possible. Whereas previous obstacle avoidance techniques focus on the immediate surroundings of the robot, the VPM 30V considers a network of interacting agents with sensors of different, possibly non-overlapping, domains. The robot 10, a static assembly cell camera, a human operator, and a parts delivery system or vehicle are examples of these agents.
  • Obstacle avoidance warnings from off-board processing could arrive at the robot 10 via the host machine 32 and cause it to reduce maximum trajectory speed, less momentum if a collision does happen, or change joint compliances, limit paths of travel, etc. Additionally, the host machine 32 could pass on present and intended paths of travel of the robot 10 in order to coordinate traffic for safety and/or efficiency reasons, e.g., by activation of an audible and/or visual signal for human safety or a queuing of part kits.
  • Underlying all machine vision-based tasks of the VPM 30V is radiometric calibration. Considerations include: removing the effect of dead pixels, removing the effect of dark currents, normalizing pixel response curves, compensation for vignetting, modeling thermal effects, etc. Previous calibration methods typically consider a sensor by itself, e.g., a single camera 34, but the present invention may also consider fusing information from complementary sensors, e.g., multiple cameras 34, 34A, 34B, the IR range imager 36, etc.
  • For example, if one camera 34 is suffering from thermal effects, pixel values could first be normalized based on a thermal model, and in the limit of extreme thermal effects, can be taken from its sister or assembly camera and processed to yield new algorithmic fault tolerance. Radiometric calibration would also yield linear response curves for the aforementioned dynamic exposure algorithm, which could greatly reduce, or possibly eliminate, the search time in hunting for a desired exposure. For example, without calibration, to get an image twice as bright might require 1.7 or 2.4 times the exposure time; after calibration, a new exposure time could be directly computed.
  • While the best modes for carrying out the invention have been described in detail, those familiar with the art to which this invention relates will recognize various alternative designs and embodiments for practicing the invention within the scope of the appended claims.

Claims (18)

1. A robotic system comprising:
a humanoid robot having a plurality of robotic joints each being moveable using at least one actuator; and
a distributed controller configured to control the movement of each of the robotic joints, the controller including a visual perception module (VPM) adapted to visually identify and track an object in the field of view of the robot under predetermined threshold ambient lighting conditions;
wherein the VPM includes a plurality of optical devices for collecting an image of the object, at least one of a laser device and a range imaging device for detecting positional information of the object, and a host machine having an algorithm for processing the image and the positional information to thereby visually identify and track the object, the VPM being configured to automatically adapt an exposure time of the optical devices to prevent feature data loss of the image under the threshold lighting conditions.
2. The robotic system of claim 1, wherein the humanoid robot includes a moveable head encapsulating the optical devices, the laser device, and the range imaging device, further comprising a gaze control unit for controlling the moveable head through at least four axes of movement.
3. The robotic system of claim 1, wherein the optical devices include at least a pair of identically-configured high-resolution cameras adapted for capturing the image with at least approximately 5 megapixels.
4. The robotic system of claim 1, wherein the range imaging device is configured as a short-range infrared (IR) time-of-flight (TOF) device.
5. The robotic system of claim 1, wherein the VPM is adapted for training the humanoid robot with respect to a new automated task by showing the robot at least one of a work sequence and a work pose within a designated work envelope.
6. The robotic system of claim 1, wherein the algorithm is adapted for discriminating either of a black-on-black or a white-on-white feature of interest on the object by automatically evaluating a small area of interest in the image, using a histogram to determine the extent of the range of illumination of the area of interest, and then processing the range of illumination using the host machine to thereby identify the features of interest.
7. The robotic system of claim 6, wherein the plurality of optical devices includes at least additional camera, wherein the VPM is configured to integrate different views of the object using information from the at least one additional camera.
8. A visual perception module (VPM) adapted to visually identify and track an object in the field of view of a humanoid robot under predetermined threshold ambient lighting conditions, the VPM comprising:
a plurality of optical devices configured to collect an image of the object;
a positional extraction device including at least one of a laser device and a range imaging device, the positional extraction device configured to extract positional data of the object;
a host machine; and
an algorithm executable via the host machine;
wherein execution of the algorithm by the host machine processes the image and positional data to thereby automatically adapt an exposure time of the optical devices, and to thereby prevent feature data loss of a captured image of an object under the threshold ambient lighting conditions.
9. The VPM of claim 8, wherein the humanoid robot includes a moveable head encapsulating the plurality of optical devices and the positional extraction device.
10. The VPM of claim 8, wherein the optical devices include at least a pair of identically-configured high-resolution cameras adapted for capturing the image with at least approximately 5 megapixels.
11. The VPM of claim 8, wherein the positional extraction device is configured as a short-range infrared (IR) time-of-flight (TOF) device.
12. The VPM of claim 8, wherein the VPM is adapted for training the humanoid robot with respect to a new automated task by showing the robot at least one of a work sequence and a work pose within a designated work envelope.
13. The VPM of claim 8, wherein the algorithm is adapted for discriminating either of a black-on-black or a white-on-white feature of interest on the object by automatically evaluating a small area of interest in the image, using a histogram to determine the extent of the range of illumination of the area of interest, and then processing the range of illumination using the host machine to thereby identify the features of interest.
14. The VPM of claim 8, wherein the plurality of optical devices includes at least additional camera, wherein the VPM is configured to integrate different views of the object as the object is located off the robot using information from the at least one additional camera, and from a perspective of at least one of a palm, a forearm, and a chest area of the robot.
15. A method of identifying and tracking an object in the field of view of a humanoid robot under threshold lighting conditions, the method comprising:
using a plurality of optical devices to collect an image of the object;
using at least one of a laser device and a range imaging device to extract positional information of the object; and
processing the image and positional information using a host machine of a visual perception module (VPM);
wherein the host machine uses an algorithm to automatically adapt an exposure time of the optical devices to thereby prevent feature data loss of the image under the threshold lighting conditions.
16. The method of claim 15, further comprising training the humanoid robot with respect to a new automated task by showing the robot at least one of a work sequence and a work pose within a designated work envelope.
17. The method of claim 15, wherein processing the image includes discriminating a black-on-black feature of interest on the object by automatically evaluating a small area of interest in the image, using a histogram to determine the extent of the range of illumination of the area of interest, and then processing the range of illumination using the host machine to thereby identify the features of interest.
18. The method of claim 15, wherein the humanoid robot includes a plurality of compliant joints each having a measurable joint angle, the method further comprising:
positioning the object in a grasp and field of view of the humanoid robot;
moving the humanoid robot and object through a series of poses;
using the host machine to track the object as it moves through the series of poses, including measuring each of the joint angles; and
recording the joint angles using the host machine; and
using the recorded joint angles to thereby calibrate eye-hand coordination of the humanoid robot.
US12/564,074 2009-09-22 2009-09-22 Visual perception system and method for a humanoid robot Active 2030-11-10 US8244402B2 (en)

Priority Applications (1)

Application Number Priority Date Filing Date Title
US12/564,074 US8244402B2 (en) 2009-09-22 2009-09-22 Visual perception system and method for a humanoid robot

Applications Claiming Priority (3)

Application Number Priority Date Filing Date Title
US12/564,074 US8244402B2 (en) 2009-09-22 2009-09-22 Visual perception system and method for a humanoid robot
DE102010045752.3A DE102010045752B4 (en) 2009-09-22 2010-09-17 Visual perception system and method for a humanoid robot
JP2010212052A JP5606241B2 (en) 2009-09-22 2010-09-22 Visual perception system and method for humanoid robot

Publications (2)

Publication Number Publication Date
US20110071675A1 true US20110071675A1 (en) 2011-03-24
US8244402B2 US8244402B2 (en) 2012-08-14

Family

ID=43757332

Family Applications (1)

Application Number Title Priority Date Filing Date
US12/564,074 Active 2030-11-10 US8244402B2 (en) 2009-09-22 2009-09-22 Visual perception system and method for a humanoid robot

Country Status (3)

Country Link
US (1) US8244402B2 (en)
JP (1) JP5606241B2 (en)
DE (1) DE102010045752B4 (en)

Cited By (65)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
US20090192524A1 (en) * 2006-06-29 2009-07-30 Intuitive Surgical, Inc. Synthetic representation of a surgical robot
US20090326553A1 (en) * 2008-06-27 2009-12-31 Intuitive Surgical, Inc. Medical robotic system providing an auxiliary view of articulatable instruments extending out of a distal end of an entry guide
US20090326556A1 (en) * 2008-06-27 2009-12-31 Intuitive Surgical, Inc. Medical robotic system providing computer generated auxiliary views of a camera instrument for controlling the positioning and orienting of its tip
US20090326318A1 (en) * 2008-06-27 2009-12-31 Intuitive Surgical, Inc. Medical robotic system providing an auxilary view including range of motion limitations for articulatable instruments extending out of a distal end of an entry guide
US20110040404A1 (en) * 2009-08-15 2011-02-17 Intuitive Surgical, Inc. Smooth control of an articulated instrument across areas with different work space conditions
US20110074923A1 (en) * 2009-09-25 2011-03-31 Samsung Electronics Co., Ltd. Image transmission system of network-based robot and method thereof
US20110202068A1 (en) * 2010-02-12 2011-08-18 Intuitive Surgical Operations, Inc. Medical robotic system providing sensory feedback indicating a difference between a commanded state and a preferred pose of an articulated instrument
US20120072022A1 (en) * 2010-09-20 2012-03-22 Samsung Electronics Co., Ltd. Robot and control method thereof
CN102514011A (en) * 2011-12-28 2012-06-27 王鹏勃 Multi-degree-of-freedom robot controlled by accurate organism sensing and achieving method thereof
US20120219176A1 (en) * 2011-02-28 2012-08-30 Al Cure Technologies, Inc. Method and Apparatus for Pattern Tracking
US20130121592A1 (en) * 2010-08-20 2013-05-16 Canon Kabushiki Kaisha Position and orientation measurement apparatus,position and orientation measurement method, and storage medium
US20130285793A1 (en) * 2006-04-28 2013-10-31 Telcred Ab Access control system and method for operating said system
US20140012415A1 (en) * 2012-07-09 2014-01-09 Technion Research & Development Foundation Limited Natural machine interface system
US20140135990A1 (en) * 2010-03-04 2014-05-15 Intouch Technologies, Inc. Remote presence system including a cart that supports a robot face and an overhead camera
US8777818B1 (en) * 2010-12-22 2014-07-15 Larry E. Tate, Jr. Training device
US8798357B2 (en) 2012-07-09 2014-08-05 Microsoft Corporation Image-based localization
US20150165620A1 (en) * 2013-12-13 2015-06-18 Canon Kabushiki Kaisha Robot apparatus, robot controlling method, program and recording medium
US9084623B2 (en) 2009-08-15 2015-07-21 Intuitive Surgical Operations, Inc. Controller assisted reconfiguration of an articulated instrument during movement into and out of an entry guide
CN104797033A (en) * 2014-01-17 2015-07-22 苏州宝时得电动工具有限公司 Automatic work equipment
US9098611B2 (en) 2012-11-26 2015-08-04 Intouch Technologies, Inc. Enhanced video interaction for a user interface of a telepresence network
US9101397B2 (en) 1999-04-07 2015-08-11 Intuitive Surgical Operations, Inc. Real-time generation of three-dimensional ultrasound image using a two-dimensional ultrasound transducer in a robotic system
US9138129B2 (en) 2007-06-13 2015-09-22 Intuitive Surgical Operations, Inc. Method and system for moving a plurality of articulated instruments in tandem back towards an entry guide
US9138891B2 (en) 2008-11-25 2015-09-22 Intouch Technologies, Inc. Server connectivity control for tele-presence robot
US9160783B2 (en) 2007-05-09 2015-10-13 Intouch Technologies, Inc. Robot system that operates through a network firewall
US9172913B1 (en) * 2010-09-24 2015-10-27 Jetprotect Corporation Automatic counter-surveillance detection camera and software
US9174342B2 (en) 2012-05-22 2015-11-03 Intouch Technologies, Inc. Social behavior rules for a medical telepresence robot
US9193065B2 (en) 2008-07-10 2015-11-24 Intouch Technologies, Inc. Docking system for a tele-presence robot
US9198728B2 (en) 2005-09-30 2015-12-01 Intouch Technologies, Inc. Multi-camera mobile teleconferencing platform
US9224181B2 (en) 2012-04-11 2015-12-29 Intouch Technologies, Inc. Systems and methods for visualizing patient and telepresence device statistics in a healthcare network
US9251313B2 (en) 2012-04-11 2016-02-02 Intouch Technologies, Inc. Systems and methods for visualizing and managing telepresence devices in healthcare networks
US9264664B2 (en) 2010-12-03 2016-02-16 Intouch Technologies, Inc. Systems and methods for dynamic bandwidth allocation
US9296107B2 (en) 2003-12-09 2016-03-29 Intouch Technologies, Inc. Protocol for a remotely controlled videoconferencing robot
US9323250B2 (en) 2011-01-28 2016-04-26 Intouch Technologies, Inc. Time-dependent navigation of telepresence robots
US9321173B2 (en) 2012-06-22 2016-04-26 Microsoft Technology Licensing, Llc Tracking and following people with a mobile robotic device
US9333042B2 (en) 2007-06-13 2016-05-10 Intuitive Surgical Operations, Inc. Medical robotic system with coupled control modes
US9345387B2 (en) 2006-06-13 2016-05-24 Intuitive Surgical Operations, Inc. Preventing instrument/tissue collisions
US9361021B2 (en) 2012-05-22 2016-06-07 Irobot Corporation Graphical user interfaces including touchpad driving interfaces for telemedicine devices
US20160171310A1 (en) * 2014-12-10 2016-06-16 Ryosuke Kasahara Image recognition system, server apparatus, and image recognition method
US9381654B2 (en) 2008-11-25 2016-07-05 Intouch Technologies, Inc. Server connectivity control for tele-presence robot
US9390346B1 (en) * 2015-06-10 2016-07-12 Robert Edwin Sheets, Jr. Object location using a mobile device
US9429934B2 (en) 2008-09-18 2016-08-30 Intouch Technologies, Inc. Mobile videoconferencing robot system with network adaptive driving
US9469034B2 (en) 2007-06-13 2016-10-18 Intuitive Surgical Operations, Inc. Method and system for switching modes of a robotic system
US9469030B2 (en) 2011-01-28 2016-10-18 Intouch Technologies Interfacing with a mobile telepresence robot
US9492927B2 (en) 2009-08-15 2016-11-15 Intuitive Surgical Operations, Inc. Application of force feedback on an input device to urge its operator to command an articulated instrument to a preferred pose
US9513113B2 (en) 2012-10-29 2016-12-06 7D Surgical, Inc. Integrated illumination and optical surface topology detection system and methods of use thereof
WO2016196512A1 (en) * 2015-05-29 2016-12-08 Abb Technology Ag Method and system for robotic adaptive production
US9602765B2 (en) 2009-08-26 2017-03-21 Intouch Technologies, Inc. Portable remote presence robot
US9616568B1 (en) * 2015-08-25 2017-04-11 X Development Llc Generating a grasp affordance for an object based on a thermal image of the object that is captured following human manipulation of the object
US9616576B2 (en) 2008-04-17 2017-04-11 Intouch Technologies, Inc. Mobile tele-presence system with a microphone system
US9715337B2 (en) 2011-11-08 2017-07-25 Intouch Technologies, Inc. Tele-presence system with a user interface that displays different communication links
US9718190B2 (en) 2006-06-29 2017-08-01 Intuitive Surgical Operations, Inc. Tool position and identification indicator displayed in a boundary area of a computer display screen
US9766624B2 (en) 2004-07-13 2017-09-19 Intouch Technologies, Inc. Mobile robot with a head-based movement mapping scheme
US9788909B2 (en) 2006-06-29 2017-10-17 Intuitive Surgical Operations, Inc Synthetic representation of a surgical instrument
WO2017200593A1 (en) * 2016-05-17 2017-11-23 Kikkeri Harshavardhana Narayana Multi -joint tracking combining embedded sensors and an external
US9842192B2 (en) 2008-07-11 2017-12-12 Intouch Technologies, Inc. Tele-presence robot system with multi-cast features
US9849593B2 (en) 2002-07-25 2017-12-26 Intouch Technologies, Inc. Medical tele-robotic system with a master remote station with an arbitrator
US9974612B2 (en) 2011-05-19 2018-05-22 Intouch Technologies, Inc. Enhanced diagnostics for a telepresence robot
US9983571B2 (en) 2009-04-17 2018-05-29 Intouch Technologies, Inc. Tele-presence robot system with software modularity, projector and laser pointer
US10008017B2 (en) 2006-06-29 2018-06-26 Intuitive Surgical Operations, Inc. Rendering tool information as graphic overlays on displayed images of tools
US10073950B2 (en) 2008-10-21 2018-09-11 Intouch Technologies, Inc. Telepresence robot with a camera boom
WO2018107182A3 (en) * 2016-12-09 2018-10-11 Lutron Electronics Co., Inc. Load control system having a visible light sensor
US10131053B1 (en) * 2016-09-14 2018-11-20 X Development Llc Real time robot collision avoidance
US10226863B2 (en) * 2012-08-09 2019-03-12 Nidec Sankyo Corporation Industrial robot
US10264651B2 (en) 2015-12-11 2019-04-16 Lutron Electronics Co., Inc. Load control system having a visible light sensor
US10334230B1 (en) * 2011-12-01 2019-06-25 Nebraska Global Investment Company, LLC Image capture system

Families Citing this family (17)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
US8941726B2 (en) * 2009-12-10 2015-01-27 Mitsubishi Electric Research Laboratories, Inc. Method and system for segmenting moving objects from images using foreground extraction
JP5573537B2 (en) * 2010-09-17 2014-08-20 株式会社デンソーウェーブ Robot teaching system of
KR101778030B1 (en) * 2010-09-27 2017-09-26 삼성전자주식회사 Robot and control method thereof
KR101761314B1 (en) * 2010-12-03 2017-07-26 삼성전자주식회사 Vital signal measurement and control method thereof
US8965563B2 (en) * 2011-04-04 2015-02-24 Palo Alto Research Incorporated High throughput parcel handling
DE102011084976A1 (en) 2011-10-21 2013-04-25 Robert Bosch Gmbh Apparatus for three-dimensional sensing of environment of motor car, has control unit that combines predetermined number of measurements to one measurement cycle and selects measurement for each pixel of measurement cycle
US9498401B2 (en) 2011-12-20 2016-11-22 Massachusetts Institute Of Technology Robotic system for simulating a wearable device and method of use
US9014857B2 (en) * 2012-01-13 2015-04-21 Toyota Motor Engineering & Manufacturing North America, Inc. Methods and computer-program products for generating grasp patterns for use by a robot
US9014850B2 (en) * 2012-01-13 2015-04-21 Toyota Motor Engineering & Manufacturing North America, Inc. Methods and computer-program products for evaluating grasp patterns, and robots incorporating the same
US9682005B2 (en) * 2012-02-24 2017-06-20 Massachusetts Institute Of Technology Elastic element exoskeleton and method of using same
JP5776716B2 (en) * 2013-03-15 2015-09-09 株式会社安川電機 Manufacturing method of a robot system and the work object
US9605950B2 (en) 2013-05-22 2017-03-28 Cognex Corporation System and method for efficient surface measurement using a laser displacement sensor
US9457475B2 (en) * 2013-10-11 2016-10-04 Rethink Robotics, Inc. Robotic placement and manipulation with enhanced accuracy
CN105339840B (en) 2013-12-06 2019-06-14 华为终端有限公司 The photographic method and bimirror head apparatus of bimirror head apparatus
EP3067746A4 (en) 2013-12-06 2016-09-21 Huawei Device Co Ltd Photographing method for dual-camera device and dual-camera device
US9555846B1 (en) 2015-03-20 2017-01-31 Google Inc. Pelvis structure for humanoid robot
US10319109B2 (en) * 2017-03-31 2019-06-11 Honda Motor Co., Ltd. Interaction with physical objects as proxy objects representing virtual objects

Citations (11)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
US5405337A (en) * 1993-02-24 1995-04-11 The Board Of Trustees Of The Leland Stanford Junior University Spatially distributed SMA actuator film providing unrestricted movement in three dimensional space
US6317652B1 (en) * 1998-09-14 2001-11-13 Honda Giken Kogyo Kabushiki Kaisha Legged mobile robot
US6587835B1 (en) * 2000-02-09 2003-07-01 G. Victor Treyz Shopping assistance with handheld computing device
US7127326B2 (en) * 2002-11-15 2006-10-24 Iguana Robotics, Inc. Certain principles of biomorphic robots including foot placement selection using non-geometric visual properties
US7206753B2 (en) * 2001-05-04 2007-04-17 Axxon Robotics, Llc Methods for facilitating a retail environment
US7269477B2 (en) * 2002-03-18 2007-09-11 Sony Corporation Image transmission device and method, transmitting device and method, receiving device and method, and robot apparatus
US7302312B2 (en) * 2001-11-28 2007-11-27 Evolution Robotics, Inc. Hardware abstraction layer (HAL) for a robot
US7428997B2 (en) * 2003-07-29 2008-09-30 Microvision, Inc. Method and apparatus for illuminating a field-of-view and capturing an image
US7438451B2 (en) * 2006-02-07 2008-10-21 Nissan Technical Center North America, Inc. Ambient light based illumination control
US7539557B2 (en) * 2005-12-30 2009-05-26 Irobot Corporation Autonomous mobile robot
US8014047B2 (en) * 2006-02-17 2011-09-06 Marvell International Technology Ltd. Sheet media stack imaging system

Family Cites Families (25)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
JPH0732994B2 (en) * 1987-05-07 1995-04-12 ファナック株式会社 Environment teaching method of the robot
JPH05126521A (en) * 1991-11-08 1993-05-21 Toshiba Corp Position-measuring device for remote-controlled manipulator
JP3021202B2 (en) * 1992-08-25 2000-03-15 三菱重工業株式会社 Induction method of the position and orientation of the robot
DE19826395A1 (en) 1998-06-12 1999-12-23 Amatec Gmbh A method for detecting and compensating cinematic variations of a robot
JP4472165B2 (en) * 1999-12-24 2010-06-02 本田技研工業株式会社 Control apparatus for a mobile robot
JP4476468B2 (en) * 2000-11-13 2010-06-09 本田技研工業株式会社 Legged mobile robot
JP2003200377A (en) * 2001-12-26 2003-07-15 Sony Corp Robot and method for measuring distance by robot
JP2004299025A (en) * 2003-04-01 2004-10-28 Honda Motor Co Ltd Mobile robot control device, mobile robot control method and mobile robot control program
JP4355341B2 (en) * 2003-05-29 2009-10-28 本田技研工業株式会社 Visual tracking using the depth data
US7336299B2 (en) * 2003-07-03 2008-02-26 Physical Optics Corporation Panoramic video system with real-time distortion-free imaging
JP2005028468A (en) * 2003-07-08 2005-02-03 National Institute Of Advanced Industrial & Technology Method for identifying position and posture of robot in visual coordinate system, and method and apparatus for coordinate transformation
CN1984756B (en) * 2004-07-13 2011-12-07 松下电器产业株式会社 Article retention system, a robot and a robot control method
JP4086024B2 (en) * 2004-09-14 2008-05-14 ソニー株式会社 The robot apparatus and behavior control method thereof
JP4198676B2 (en) * 2004-12-28 2008-12-17 株式会社東芝 Robot, movement tracking method of the robot apparatus, and a program
JP2006289508A (en) * 2005-04-05 2006-10-26 Sony Corp Robot device and its facial expression control method
JP2007160482A (en) * 2005-12-16 2007-06-28 Ixs Research:Kk Neck part joint mechanism and robot provided with the same
JP4325627B2 (en) * 2006-02-21 2009-09-02 ソニー株式会社 The image display method and image display apparatus and an imaging apparatus
JP2007319938A (en) * 2006-05-30 2007-12-13 Toyota Motor Corp Robot device and method of obtaining three-dimensional shape of object
JP4658891B2 (en) * 2006-10-02 2011-03-23 本田技研工業株式会社 Control apparatus for a robot
JP2008112349A (en) * 2006-10-31 2008-05-15 Alpine Electronics Inc Moving object detection device
JP2008148180A (en) * 2006-12-13 2008-06-26 Sony Corp Imaging apparatus, image processing device and method, and computer program
JP2009056529A (en) * 2007-08-30 2009-03-19 Honda Motor Co Ltd Legged mobile robot
JP4847420B2 (en) * 2007-08-30 2011-12-28 本田技研工業株式会社 Legged mobile robot
US8026955B2 (en) 2007-08-30 2011-09-27 Honda Motor Co., Ltd. Camera exposure controller including imaging devices for capturing an image using stereo-imaging
JP2009113190A (en) * 2007-11-09 2009-05-28 Toyota Motor Corp Autonomous working robot and method of controlling operation of autonomous working robot

Patent Citations (11)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
US5405337A (en) * 1993-02-24 1995-04-11 The Board Of Trustees Of The Leland Stanford Junior University Spatially distributed SMA actuator film providing unrestricted movement in three dimensional space
US6317652B1 (en) * 1998-09-14 2001-11-13 Honda Giken Kogyo Kabushiki Kaisha Legged mobile robot
US6587835B1 (en) * 2000-02-09 2003-07-01 G. Victor Treyz Shopping assistance with handheld computing device
US7206753B2 (en) * 2001-05-04 2007-04-17 Axxon Robotics, Llc Methods for facilitating a retail environment
US7302312B2 (en) * 2001-11-28 2007-11-27 Evolution Robotics, Inc. Hardware abstraction layer (HAL) for a robot
US7269477B2 (en) * 2002-03-18 2007-09-11 Sony Corporation Image transmission device and method, transmitting device and method, receiving device and method, and robot apparatus
US7127326B2 (en) * 2002-11-15 2006-10-24 Iguana Robotics, Inc. Certain principles of biomorphic robots including foot placement selection using non-geometric visual properties
US7428997B2 (en) * 2003-07-29 2008-09-30 Microvision, Inc. Method and apparatus for illuminating a field-of-view and capturing an image
US7539557B2 (en) * 2005-12-30 2009-05-26 Irobot Corporation Autonomous mobile robot
US7438451B2 (en) * 2006-02-07 2008-10-21 Nissan Technical Center North America, Inc. Ambient light based illumination control
US8014047B2 (en) * 2006-02-17 2011-09-06 Marvell International Technology Ltd. Sheet media stack imaging system

Non-Patent Citations (5)

* Cited by examiner, † Cited by third party
Title
Hasunuma et al., Development of Teleoperation Master System with a Kinesthetic Sensation of Presence, 1999, Internet, pg. 1-7. *
Maxwell et al., Alfred: The Robot Waiter Who Remembers You, 1999, Internet, pg., 1-12 *
Meng et al., Self-Calibration of Camera-Equipped Robot Manipulators, 2001, Internet, pg. 909-921 *
Qing et al., Active Self-Calibration of Robotic Eyes and Hand-Eye Relationships with Model Identification, 1998, IEEE, pg. 158-166 *
Zhuang et al., SIMULTANEOUS CALIBRATION OF A ROBOT AND A HAND-MOUNTED CAMERA, 1993, IEEE, pg. 149-154 *

Cited By (111)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
US9232984B2 (en) 1999-04-07 2016-01-12 Intuitive Surgical Operations, Inc. Real-time generation of three-dimensional ultrasound image using a two-dimensional ultrasound transducer in a robotic system
US10271909B2 (en) 1999-04-07 2019-04-30 Intuitive Surgical Operations, Inc. Display of computer generated image of an out-of-view portion of a medical device adjacent a real-time image of an in-view portion of the medical device
US9101397B2 (en) 1999-04-07 2015-08-11 Intuitive Surgical Operations, Inc. Real-time generation of three-dimensional ultrasound image using a two-dimensional ultrasound transducer in a robotic system
US10315312B2 (en) 2002-07-25 2019-06-11 Intouch Technologies, Inc. Medical tele-robotic system with a master remote station with an arbitrator
US9849593B2 (en) 2002-07-25 2017-12-26 Intouch Technologies, Inc. Medical tele-robotic system with a master remote station with an arbitrator
US9375843B2 (en) 2003-12-09 2016-06-28 Intouch Technologies, Inc. Protocol for a remotely controlled videoconferencing robot
US9296107B2 (en) 2003-12-09 2016-03-29 Intouch Technologies, Inc. Protocol for a remotely controlled videoconferencing robot
US9956690B2 (en) 2003-12-09 2018-05-01 Intouch Technologies, Inc. Protocol for a remotely controlled videoconferencing robot
US9766624B2 (en) 2004-07-13 2017-09-19 Intouch Technologies, Inc. Mobile robot with a head-based movement mapping scheme
US10241507B2 (en) 2004-07-13 2019-03-26 Intouch Technologies, Inc. Mobile robot with a head-based movement mapping scheme
US10259119B2 (en) 2005-09-30 2019-04-16 Intouch Technologies, Inc. Multi-camera mobile teleconferencing platform
US9198728B2 (en) 2005-09-30 2015-12-01 Intouch Technologies, Inc. Multi-camera mobile teleconferencing platform
US8723641B2 (en) * 2006-04-28 2014-05-13 Telecred AB Access control system and method for operating said system
US20130285793A1 (en) * 2006-04-28 2013-10-31 Telcred Ab Access control system and method for operating said system
US9345387B2 (en) 2006-06-13 2016-05-24 Intuitive Surgical Operations, Inc. Preventing instrument/tissue collisions
US10137575B2 (en) 2006-06-29 2018-11-27 Intuitive Surgical Operations, Inc. Synthetic representation of a surgical robot
US9801690B2 (en) 2006-06-29 2017-10-31 Intuitive Surgical Operations, Inc. Synthetic representation of a surgical instrument
US20090192524A1 (en) * 2006-06-29 2009-07-30 Intuitive Surgical, Inc. Synthetic representation of a surgical robot
US9718190B2 (en) 2006-06-29 2017-08-01 Intuitive Surgical Operations, Inc. Tool position and identification indicator displayed in a boundary area of a computer display screen
US9788909B2 (en) 2006-06-29 2017-10-17 Intuitive Surgical Operations, Inc Synthetic representation of a surgical instrument
US9789608B2 (en) 2006-06-29 2017-10-17 Intuitive Surgical Operations, Inc. Synthetic representation of a surgical robot
US10008017B2 (en) 2006-06-29 2018-06-26 Intuitive Surgical Operations, Inc. Rendering tool information as graphic overlays on displayed images of tools
US9160783B2 (en) 2007-05-09 2015-10-13 Intouch Technologies, Inc. Robot system that operates through a network firewall
US9333042B2 (en) 2007-06-13 2016-05-10 Intuitive Surgical Operations, Inc. Medical robotic system with coupled control modes
US9469034B2 (en) 2007-06-13 2016-10-18 Intuitive Surgical Operations, Inc. Method and system for switching modes of a robotic system
US10271912B2 (en) 2007-06-13 2019-04-30 Intuitive Surgical Operations, Inc. Method and system for moving a plurality of articulated instruments in tandem back towards an entry guide
US10188472B2 (en) 2007-06-13 2019-01-29 Intuitive Surgical Operations, Inc. Medical robotic system with coupled control modes
US9138129B2 (en) 2007-06-13 2015-09-22 Intuitive Surgical Operations, Inc. Method and system for moving a plurality of articulated instruments in tandem back towards an entry guide
US9629520B2 (en) 2007-06-13 2017-04-25 Intuitive Surgical Operations, Inc. Method and system for moving an articulated instrument back towards an entry guide while automatically reconfiguring the articulated instrument for retraction into the entry guide
US9901408B2 (en) 2007-06-13 2018-02-27 Intuitive Surgical Operations, Inc. Preventing instrument/tissue collisions
US9616576B2 (en) 2008-04-17 2017-04-11 Intouch Technologies, Inc. Mobile tele-presence system with a microphone system
US9089256B2 (en) 2008-06-27 2015-07-28 Intuitive Surgical Operations, Inc. Medical robotic system providing an auxiliary view including range of motion limitations for articulatable instruments extending out of a distal end of an entry guide
US10258425B2 (en) 2008-06-27 2019-04-16 Intuitive Surgical Operations, Inc. Medical robotic system providing an auxiliary view of articulatable instruments extending out of a distal end of an entry guide
US8864652B2 (en) 2008-06-27 2014-10-21 Intuitive Surgical Operations, Inc. Medical robotic system providing computer generated auxiliary views of a camera instrument for controlling the positioning and orienting of its tip
US20090326318A1 (en) * 2008-06-27 2009-12-31 Intuitive Surgical, Inc. Medical robotic system providing an auxilary view including range of motion limitations for articulatable instruments extending out of a distal end of an entry guide
US20090326556A1 (en) * 2008-06-27 2009-12-31 Intuitive Surgical, Inc. Medical robotic system providing computer generated auxiliary views of a camera instrument for controlling the positioning and orienting of its tip
US20090326553A1 (en) * 2008-06-27 2009-12-31 Intuitive Surgical, Inc. Medical robotic system providing an auxiliary view of articulatable instruments extending out of a distal end of an entry guide
US9516996B2 (en) 2008-06-27 2016-12-13 Intuitive Surgical Operations, Inc. Medical robotic system providing computer generated auxiliary views of a camera instrument for controlling the position and orienting of its tip
US9717563B2 (en) 2008-06-27 2017-08-01 Intuitive Surgical Operations, Inc. Medical robotic system providing an auxilary view including range of motion limitations for articulatable instruments extending out of a distal end of an entry guide
US9193065B2 (en) 2008-07-10 2015-11-24 Intouch Technologies, Inc. Docking system for a tele-presence robot
US9842192B2 (en) 2008-07-11 2017-12-12 Intouch Technologies, Inc. Tele-presence robot system with multi-cast features
US9429934B2 (en) 2008-09-18 2016-08-30 Intouch Technologies, Inc. Mobile videoconferencing robot system with network adaptive driving
US10073950B2 (en) 2008-10-21 2018-09-11 Intouch Technologies, Inc. Telepresence robot with a camera boom
US9381654B2 (en) 2008-11-25 2016-07-05 Intouch Technologies, Inc. Server connectivity control for tele-presence robot
US10059000B2 (en) 2008-11-25 2018-08-28 Intouch Technologies, Inc. Server connectivity control for a tele-presence robot
US9138891B2 (en) 2008-11-25 2015-09-22 Intouch Technologies, Inc. Server connectivity control for tele-presence robot
US10282881B2 (en) 2009-03-31 2019-05-07 Intuitive Surgical Operations, Inc. Rendering tool information as graphic overlays on displayed images of tools
US9983571B2 (en) 2009-04-17 2018-05-29 Intouch Technologies, Inc. Tele-presence robot system with software modularity, projector and laser pointer
US8903546B2 (en) * 2009-08-15 2014-12-02 Intuitive Surgical Operations, Inc. Smooth control of an articulated instrument across areas with different work space conditions
US10271915B2 (en) 2009-08-15 2019-04-30 Intuitive Surgical Operations, Inc. Application of force feedback on an input device to urge its operator to command an articulated instrument to a preferred pose
US20110040404A1 (en) * 2009-08-15 2011-02-17 Intuitive Surgical, Inc. Smooth control of an articulated instrument across areas with different work space conditions
US9084623B2 (en) 2009-08-15 2015-07-21 Intuitive Surgical Operations, Inc. Controller assisted reconfiguration of an articulated instrument during movement into and out of an entry guide
US9956044B2 (en) 2009-08-15 2018-05-01 Intuitive Surgical Operations, Inc. Controller assisted reconfiguration of an articulated instrument during movement into and out of an entry guide
US9492927B2 (en) 2009-08-15 2016-11-15 Intuitive Surgical Operations, Inc. Application of force feedback on an input device to urge its operator to command an articulated instrument to a preferred pose
US9602765B2 (en) 2009-08-26 2017-03-21 Intouch Technologies, Inc. Portable remote presence robot
US20110074923A1 (en) * 2009-09-25 2011-03-31 Samsung Electronics Co., Ltd. Image transmission system of network-based robot and method thereof
US9622826B2 (en) 2010-02-12 2017-04-18 Intuitive Surgical Operations, Inc. Medical robotic system providing sensory feedback indicating a difference between a commanded state and a preferred pose of an articulated instrument
US20110202068A1 (en) * 2010-02-12 2011-08-18 Intuitive Surgical Operations, Inc. Medical robotic system providing sensory feedback indicating a difference between a commanded state and a preferred pose of an articulated instrument
US8918211B2 (en) 2010-02-12 2014-12-23 Intuitive Surgical Operations, Inc. Medical robotic system providing sensory feedback indicating a difference between a commanded state and a preferred pose of an articulated instrument
US9089972B2 (en) * 2010-03-04 2015-07-28 Intouch Technologies, Inc. Remote presence system including a cart that supports a robot face and an overhead camera
US20140135990A1 (en) * 2010-03-04 2014-05-15 Intouch Technologies, Inc. Remote presence system including a cart that supports a robot face and an overhead camera
US9208395B2 (en) * 2010-08-20 2015-12-08 Canon Kabushiki Kaisha Position and orientation measurement apparatus, position and orientation measurement method, and storage medium
US20130121592A1 (en) * 2010-08-20 2013-05-16 Canon Kabushiki Kaisha Position and orientation measurement apparatus,position and orientation measurement method, and storage medium
US9193072B2 (en) * 2010-09-20 2015-11-24 Samsung Electronics Co., Ltd. Robot and control method thereof
US20120072022A1 (en) * 2010-09-20 2012-03-22 Samsung Electronics Co., Ltd. Robot and control method thereof
US9172913B1 (en) * 2010-09-24 2015-10-27 Jetprotect Corporation Automatic counter-surveillance detection camera and software
US10218748B2 (en) 2010-12-03 2019-02-26 Intouch Technologies, Inc. Systems and methods for dynamic bandwidth allocation
US9264664B2 (en) 2010-12-03 2016-02-16 Intouch Technologies, Inc. Systems and methods for dynamic bandwidth allocation
US8777818B1 (en) * 2010-12-22 2014-07-15 Larry E. Tate, Jr. Training device
US9469030B2 (en) 2011-01-28 2016-10-18 Intouch Technologies Interfacing with a mobile telepresence robot
US9785149B2 (en) 2011-01-28 2017-10-10 Intouch Technologies, Inc. Time-dependent navigation of telepresence robots
US9323250B2 (en) 2011-01-28 2016-04-26 Intouch Technologies, Inc. Time-dependent navigation of telepresence robots
US9665767B2 (en) * 2011-02-28 2017-05-30 Aic Innovations Group, Inc. Method and apparatus for pattern tracking
US20170270355A1 (en) * 2011-02-28 2017-09-21 Aic Innovations Group, Inc. Method and Apparatus for Pattern Tracking
US9892316B2 (en) * 2011-02-28 2018-02-13 Aic Innovations Group, Inc. Method and apparatus for pattern tracking
US20120219176A1 (en) * 2011-02-28 2012-08-30 Al Cure Technologies, Inc. Method and Apparatus for Pattern Tracking
US9974612B2 (en) 2011-05-19 2018-05-22 Intouch Technologies, Inc. Enhanced diagnostics for a telepresence robot
US9715337B2 (en) 2011-11-08 2017-07-25 Intouch Technologies, Inc. Tele-presence system with a user interface that displays different communication links
US10331323B2 (en) 2011-11-08 2019-06-25 Intouch Technologies, Inc. Tele-presence system with a user interface that displays different communication links
US10334230B1 (en) * 2011-12-01 2019-06-25 Nebraska Global Investment Company, LLC Image capture system
CN102514011A (en) * 2011-12-28 2012-06-27 王鹏勃 Multi-degree-of-freedom robot controlled by accurate organism sensing and achieving method thereof
US9251313B2 (en) 2012-04-11 2016-02-02 Intouch Technologies, Inc. Systems and methods for visualizing and managing telepresence devices in healthcare networks
US9224181B2 (en) 2012-04-11 2015-12-29 Intouch Technologies, Inc. Systems and methods for visualizing patient and telepresence device statistics in a healthcare network
US9361021B2 (en) 2012-05-22 2016-06-07 Irobot Corporation Graphical user interfaces including touchpad driving interfaces for telemedicine devices
US10328576B2 (en) 2012-05-22 2019-06-25 Intouch Technologies, Inc. Social behavior rules for a medical telepresence robot
US10061896B2 (en) 2012-05-22 2018-08-28 Intouch Technologies, Inc. Graphical user interfaces including touchpad driving interfaces for telemedicine devices
US9174342B2 (en) 2012-05-22 2015-11-03 Intouch Technologies, Inc. Social behavior rules for a medical telepresence robot
US9776327B2 (en) 2012-05-22 2017-10-03 Intouch Technologies, Inc. Social behavior rules for a medical telepresence robot
US9321173B2 (en) 2012-06-22 2016-04-26 Microsoft Technology Licensing, Llc Tracking and following people with a mobile robotic device
US8798357B2 (en) 2012-07-09 2014-08-05 Microsoft Corporation Image-based localization
US20140012415A1 (en) * 2012-07-09 2014-01-09 Technion Research & Development Foundation Limited Natural machine interface system
US9753453B2 (en) * 2012-07-09 2017-09-05 Deep Learning Robotics Ltd. Natural machine interface system
US10265845B2 (en) * 2012-08-09 2019-04-23 Nidec Sankyo Corporation Industrial robot
US10226863B2 (en) * 2012-08-09 2019-03-12 Nidec Sankyo Corporation Industrial robot
US9513113B2 (en) 2012-10-29 2016-12-06 7D Surgical, Inc. Integrated illumination and optical surface topology detection system and methods of use thereof
US10334205B2 (en) 2012-11-26 2019-06-25 Intouch Technologies, Inc. Enhanced video interaction for a user interface of a telepresence network
US9098611B2 (en) 2012-11-26 2015-08-04 Intouch Technologies, Inc. Enhanced video interaction for a user interface of a telepresence network
US20170036353A1 (en) * 2013-12-13 2017-02-09 Canon Kabushiki Kaisha Robot apparatus, robot controlling method, programming and recording medium
US9505133B2 (en) * 2013-12-13 2016-11-29 Canon Kabushiki Kaisha Robot apparatus, robot controlling method, program and recording medium
US20150165620A1 (en) * 2013-12-13 2015-06-18 Canon Kabushiki Kaisha Robot apparatus, robot controlling method, program and recording medium
US9902073B2 (en) * 2013-12-13 2018-02-27 Canon Kabushiki Kaisha Robot apparatus, robot controlling method, programming and recording medium
CN104797033A (en) * 2014-01-17 2015-07-22 苏州宝时得电动工具有限公司 Automatic work equipment
US20160171310A1 (en) * 2014-12-10 2016-06-16 Ryosuke Kasahara Image recognition system, server apparatus, and image recognition method
WO2016196512A1 (en) * 2015-05-29 2016-12-08 Abb Technology Ag Method and system for robotic adaptive production
US9390346B1 (en) * 2015-06-10 2016-07-12 Robert Edwin Sheets, Jr. Object location using a mobile device
US9616568B1 (en) * 2015-08-25 2017-04-11 X Development Llc Generating a grasp affordance for an object based on a thermal image of the object that is captured following human manipulation of the object
US10264651B2 (en) 2015-12-11 2019-04-16 Lutron Electronics Co., Inc. Load control system having a visible light sensor
WO2017200593A1 (en) * 2016-05-17 2017-11-23 Kikkeri Harshavardhana Narayana Multi -joint tracking combining embedded sensors and an external
US10131053B1 (en) * 2016-09-14 2018-11-20 X Development Llc Real time robot collision avoidance
US10278268B2 (en) 2016-12-09 2019-04-30 Lutron Technology Company Llc Controlling lighting loads to achieve a desired lighting pattern
WO2018107182A3 (en) * 2016-12-09 2018-10-11 Lutron Electronics Co., Inc. Load control system having a visible light sensor

Also Published As

Publication number Publication date
US8244402B2 (en) 2012-08-14
DE102010045752B4 (en) 2018-10-31
JP5606241B2 (en) 2014-10-15
DE102010045752A1 (en) 2011-06-01
JP2011067941A (en) 2011-04-07

Similar Documents

Publication Publication Date Title
Kelly et al. Stable visual servoing of camera-in-hand robotic systems
Asfour et al. Toward humanoid manipulation in human-centred environments
US6816755B2 (en) Method and apparatus for single camera 3D vision guided robotics
JP4021413B2 (en) Measuring device
US20100172733A1 (en) Intelligent interface device for grasping of an object by a manipulating robot and method of implementing this device
Wijesoma et al. Eye-to-hand coordination for vision-guided robot control applications
Taylor et al. Robust vision-based pose control
Jorg et al. Flexible robot-assembly using a multi-sensory approach
US20040172164A1 (en) Method and apparatus for single image 3D vision guided robotics
FI123306B (en) The robot tool system, as well as its control method, and computer program product
Natale et al. A sensitive approach to grasping
Lenz et al. Joint-action for humans and industrial robots for assembly tasks
WO2011143576A2 (en) System and method for robust calibration between a machine vision system and a robot
DE102010045752B4 (en) Visual perception system and method for a humanoid robot
JP5977544B2 (en) Information processing apparatus, information processing method
Corke et al. Real-time vision, tracking and control
WO2009123956A1 (en) Robot parts assembly on a workpiece moving on an assembly line
US8315455B2 (en) Robot system, robot control device and method for controlling robot
JP6180086B2 (en) Information processing apparatus and information processing method
Kofman et al. Teleoperation of a robot manipulator using a vision-based human-robot interface
US9508148B2 (en) Vision-guided alignment system and method
Xie et al. Switching control of image-based visual servoing with laser pointer in robotic manufacturing systems
CN103170973B (en) Man-machine cooperation device and method based on Kinect video camera
Bagnell et al. An integrated system for autonomous robotics manipulation
JP4265088B2 (en) The robot apparatus and control method thereof

Legal Events

Date Code Title Description
AS Assignment

Owner name: GM GLOBAL TECHNOLOGY OPERATIONS, INC., MICHIGAN

Free format text: ASSIGNMENT OF ASSIGNORS INTEREST;ASSIGNORS:WELLS, JAMES W.;MCKAY, NEIL DAVID;LINN, DOUGLAS MARTIN;AND OTHERS;REEL/FRAME:023265/0634

Effective date: 20090811

AS Assignment

Owner name: UNITED STATES DEPARTMENT OF THE TREASURY, DISTRICT

Free format text: SECURITY AGREEMENT;ASSIGNOR:GM GLOBAL TECHNOLOGY OPERATIONS, INC.;REEL/FRAME:023989/0155

Effective date: 20090710

Owner name: UAW RETIREE MEDICAL BENEFITS TRUST, MICHIGAN

Free format text: SECURITY AGREEMENT;ASSIGNOR:GM GLOBAL TECHNOLOGY OPERATIONS, INC.;REEL/FRAME:023990/0001

Effective date: 20090710

AS Assignment

Owner name: GM GLOBAL TECHNOLOGY OPERATIONS, INC., MICHIGAN

Free format text: RELEASE BY SECURED PARTY;ASSIGNOR:UNITED STATES DEPARTMENT OF THE TREASURY;REEL/FRAME:025246/0234

Effective date: 20100420

AS Assignment

Owner name: GM GLOBAL TECHNOLOGY OPERATIONS, INC., MICHIGAN

Free format text: RELEASE BY SECURED PARTY;ASSIGNOR:UAW RETIREE MEDICAL BENEFITS TRUST;REEL/FRAME:025315/0091

Effective date: 20101026

AS Assignment

Owner name: WILMINGTON TRUST COMPANY, DELAWARE

Free format text: SECURITY AGREEMENT;ASSIGNOR:GM GLOBAL TECHNOLOGY OPERATIONS, INC.;REEL/FRAME:025324/0555

Effective date: 20101027

AS Assignment

Owner name: GM GLOBAL TECHNOLOGY OPERATIONS LLC, MICHIGAN

Free format text: CHANGE OF NAME;ASSIGNOR:GM GLOBAL TECHNOLOGY OPERATIONS, INC.;REEL/FRAME:025781/0299

Effective date: 20101202

AS Assignment

Owner name: GM GLOBAL TECHNOLOGY OPERATIONS LLC, MICHIGAN

Free format text: ASSIGNMENT OF ASSIGNORS INTEREST;ASSIGNOR:CHELIAN, SUHAS E.;REEL/FRAME:028025/0975

Effective date: 20120410

STCF Information on status: patent grant

Free format text: PATENTED CASE

AS Assignment

Owner name: GM GLOBAL TECHNOLOGY OPERATIONS LLC, MICHIGAN

Free format text: RELEASE BY SECURED PARTY;ASSIGNOR:WILMINGTON TRUST COMPANY;REEL/FRAME:034192/0299

Effective date: 20141017

FPAY Fee payment

Year of fee payment: 4