EP3762190A4 - Augmented reality coordination of human-robot interaction - Google Patents

Augmented reality coordination of human-robot interaction Download PDF

Info

Publication number
EP3762190A4
EP3762190A4 EP19764482.6A EP19764482A EP3762190A4 EP 3762190 A4 EP3762190 A4 EP 3762190A4 EP 19764482 A EP19764482 A EP 19764482A EP 3762190 A4 EP3762190 A4 EP 3762190A4
Authority
EP
European Patent Office
Prior art keywords
human
augmented reality
robot interaction
coordination
reality coordination
Prior art date
Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
Pending
Application number
EP19764482.6A
Other languages
German (de)
French (fr)
Other versions
EP3762190A1 (en
Inventor
Daniel Szafir
Michael E. Walker
Hooman HEDAYATI
Current Assignee (The listed assignees may be inaccurate. Google has not performed a legal analysis and makes no representation or warranty as to the accuracy of the list.)
University of Colorado
Original Assignee
University of Colorado
Priority date (The priority date is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the date listed.)
Filing date
Publication date
Application filed by University of Colorado filed Critical University of Colorado
Publication of EP3762190A1 publication Critical patent/EP3762190A1/en
Publication of EP3762190A4 publication Critical patent/EP3762190A4/en
Pending legal-status Critical Current

Links

Classifications

    • G—PHYSICS
    • G05—CONTROLLING; REGULATING
    • G05D—SYSTEMS FOR CONTROLLING OR REGULATING NON-ELECTRIC VARIABLES
    • G05D1/00—Control of position, course, altitude or attitude of land, water, air or space vehicles, e.g. using automatic pilots
    • G05D1/0011—Control of position, course, altitude or attitude of land, water, air or space vehicles, e.g. using automatic pilots associated with a remote control arrangement
    • G05D1/0038—Control of position, course, altitude or attitude of land, water, air or space vehicles, e.g. using automatic pilots associated with a remote control arrangement by providing the operator with simple or augmented images from one or more cameras located onboard the vehicle, e.g. tele-operation
    • B—PERFORMING OPERATIONS; TRANSPORTING
    • B25—HAND TOOLS; PORTABLE POWER-DRIVEN TOOLS; MANIPULATORS
    • B25J—MANIPULATORS; CHAMBERS PROVIDED WITH MANIPULATION DEVICES
    • B25J9/00—Programme-controlled manipulators
    • B25J9/16—Programme controls
    • B25J9/1656—Programme controls characterised by programming, planning systems for manipulators
    • B25J9/1664—Programme controls characterised by programming, planning systems for manipulators characterised by motion, path, trajectory planning
    • B25J9/1666—Avoiding collision or forbidden zones
    • B—PERFORMING OPERATIONS; TRANSPORTING
    • B25—HAND TOOLS; PORTABLE POWER-DRIVEN TOOLS; MANIPULATORS
    • B25J—MANIPULATORS; CHAMBERS PROVIDED WITH MANIPULATION DEVICES
    • B25J13/00—Controls for manipulators
    • B—PERFORMING OPERATIONS; TRANSPORTING
    • B25—HAND TOOLS; PORTABLE POWER-DRIVEN TOOLS; MANIPULATORS
    • B25J—MANIPULATORS; CHAMBERS PROVIDED WITH MANIPULATION DEVICES
    • B25J13/00—Controls for manipulators
    • B25J13/06—Control stands, e.g. consoles, switchboards
    • G—PHYSICS
    • G05—CONTROLLING; REGULATING
    • G05D—SYSTEMS FOR CONTROLLING OR REGULATING NON-ELECTRIC VARIABLES
    • G05D1/00—Control of position, course, altitude or attitude of land, water, air or space vehicles, e.g. using automatic pilots
    • G05D1/0011—Control of position, course, altitude or attitude of land, water, air or space vehicles, e.g. using automatic pilots associated with a remote control arrangement
    • G05D1/0016—Control of position, course, altitude or attitude of land, water, air or space vehicles, e.g. using automatic pilots associated with a remote control arrangement characterised by the operator's input device
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06F—ELECTRIC DIGITAL DATA PROCESSING
    • G06F3/00—Input arrangements for transferring data to be processed into a form capable of being handled by the computer; Output arrangements for transferring data from processing unit to output unit, e.g. interface arrangements
    • G06F3/01—Input arrangements or combined input and output arrangements for interaction between user and computer
    • G06F3/011—Arrangements for interaction with the human body, e.g. for user immersion in virtual reality
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06F—ELECTRIC DIGITAL DATA PROCESSING
    • G06F3/00—Input arrangements for transferring data to be processed into a form capable of being handled by the computer; Output arrangements for transferring data from processing unit to output unit, e.g. interface arrangements
    • G06F3/01—Input arrangements or combined input and output arrangements for interaction between user and computer
    • G06F3/011—Arrangements for interaction with the human body, e.g. for user immersion in virtual reality
    • G06F3/012—Head tracking input arrangements
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06F—ELECTRIC DIGITAL DATA PROCESSING
    • G06F3/00—Input arrangements for transferring data to be processed into a form capable of being handled by the computer; Output arrangements for transferring data from processing unit to output unit, e.g. interface arrangements
    • G06F3/01—Input arrangements or combined input and output arrangements for interaction between user and computer
    • G06F3/011—Arrangements for interaction with the human body, e.g. for user immersion in virtual reality
    • G06F3/013—Eye tracking input arrangements
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06F—ELECTRIC DIGITAL DATA PROCESSING
    • G06F3/00—Input arrangements for transferring data to be processed into a form capable of being handled by the computer; Output arrangements for transferring data from processing unit to output unit, e.g. interface arrangements
    • G06F3/14—Digital output to display device ; Cooperation and interconnection of the display device with other functional units
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
    • G06T11/00—2D [Two Dimensional] image generation
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06V—IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING
    • G06V20/00—Scenes; Scene-specific elements
    • G06V20/10—Terrestrial scenes
    • G—PHYSICS
    • G05—CONTROLLING; REGULATING
    • G05B—CONTROL OR REGULATING SYSTEMS IN GENERAL; FUNCTIONAL ELEMENTS OF SUCH SYSTEMS; MONITORING OR TESTING ARRANGEMENTS FOR SUCH SYSTEMS OR ELEMENTS
    • G05B2219/00—Program-control systems
    • G05B2219/30—Nc systems
    • G05B2219/39—Robotics, robotics to robotics hand
    • G05B2219/39449—Pendant, pda displaying camera images overlayed with graphics, augmented reality
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
    • G06T2200/00—Indexing scheme for image data processing or generation, in general
    • G06T2200/24—Indexing scheme for image data processing or generation, in general involving graphical user interfaces [GUIs]

Landscapes

  • Engineering & Computer Science (AREA)
  • Theoretical Computer Science (AREA)
  • Physics & Mathematics (AREA)
  • General Physics & Mathematics (AREA)
  • General Engineering & Computer Science (AREA)
  • Human Computer Interaction (AREA)
  • Mechanical Engineering (AREA)
  • Robotics (AREA)
  • Radar, Positioning & Navigation (AREA)
  • Remote Sensing (AREA)
  • Aviation & Aerospace Engineering (AREA)
  • Automation & Control Theory (AREA)
  • Multimedia (AREA)
  • Computing Systems (AREA)
  • Mathematical Physics (AREA)
  • Manipulator (AREA)
  • Processing Or Creating Images (AREA)
  • User Interface Of Digital Computer (AREA)
  • Control Of Position, Course, Altitude, Or Attitude Of Moving Bodies (AREA)
EP19764482.6A 2018-03-05 2019-03-05 Augmented reality coordination of human-robot interaction Pending EP3762190A4 (en)

Applications Claiming Priority (2)

Application Number Priority Date Filing Date Title
US201862638578P 2018-03-05 2018-03-05
PCT/US2019/020831 WO2019173396A1 (en) 2018-03-05 2019-03-05 Augmented reality coordination of human-robot interaction

Publications (2)

Publication Number Publication Date
EP3762190A1 EP3762190A1 (en) 2021-01-13
EP3762190A4 true EP3762190A4 (en) 2022-01-26

Family

ID=67845769

Family Applications (1)

Application Number Title Priority Date Filing Date
EP19764482.6A Pending EP3762190A4 (en) 2018-03-05 2019-03-05 Augmented reality coordination of human-robot interaction

Country Status (4)

Country Link
US (1) US11986963B2 (en)
EP (1) EP3762190A4 (en)
JP (2) JP2021515336A (en)
WO (1) WO2019173396A1 (en)

Families Citing this family (24)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
WO2018195869A1 (en) * 2017-04-27 2018-11-01 SZ DJI Technology Co., Ltd. Systems and methods for generating real-time map using movable object
CN112313042B (en) * 2018-06-29 2024-05-10 国立研究开发法人科学技术振兴机构 Remote operation system
US11480968B1 (en) * 2018-08-29 2022-10-25 Amazon Technologies, Inc. System for dynamic positioning of an autonomous mobile device with respect to a user
KR20200098034A (en) * 2019-02-11 2020-08-20 삼성전자주식회사 Electronic device for providing augmented reality user interface and operating method thereof
US11958183B2 (en) * 2019-09-19 2024-04-16 The Research Foundation For The State University Of New York Negotiation-based human-robot collaboration via augmented reality
US11461867B2 (en) * 2019-09-19 2022-10-04 Sanctuary Cognitive Systems Corporation Visual interface and communications techniques for use with robots
US20220234612A1 (en) * 2019-09-25 2022-07-28 Hewlett-Packard Development Company, L.P. Location indicator devices
US11274936B2 (en) * 2019-11-14 2022-03-15 Nissan North America, Inc. Safety-assured remote driving for autonomous vehicles
CN111267099B (en) * 2020-02-24 2023-02-28 东南大学 Accompanying machine control system based on virtual reality
TWI747186B (en) * 2020-03-05 2021-11-21 國立臺北科技大學 Methods and systems of augmented reality processing, computer program product and computer-readable recording medium
US20210287382A1 (en) * 2020-03-13 2021-09-16 Magic Leap, Inc. Systems and methods for multi-user virtual and augmented reality
US11662737B2 (en) * 2020-12-28 2023-05-30 Irobot Corporation Systems and methods for dock placement for an autonomous mobile robot
CN116745816A (en) * 2021-01-19 2023-09-12 Abb瑞士股份有限公司 System and method for visualizing multiple mobile robots
CN112975972B (en) * 2021-03-01 2022-06-17 中电海康集团有限公司 Robot control method based on head-mounted AR equipment and operating handle
CN113741687B (en) * 2021-08-10 2023-05-23 广东工业大学 Industrial air conditioner control communication method, system and storage medium based on AR (augmented reality) glasses
CN113618723A (en) * 2021-08-18 2021-11-09 东华大学 Ship small-erection auxiliary assembly system based on augmented reality
DE102021121766A1 (en) * 2021-08-23 2023-02-23 Still Gesellschaft Mit Beschränkter Haftung Method and system for setting up a robotic and/or assisting system unit of an industrial truck
US20230087202A1 (en) * 2021-09-17 2023-03-23 Ford Global Technologies, Llc Augmented Reality And Touch-Based User Engagement Parking Assist
CN113918015B (en) * 2021-10-08 2024-04-19 三星电子（中国）研发中心 Interaction method and device for augmented reality
DE102021214584A1 (en) 2021-12-17 2023-06-22 Volkswagen Aktiengesellschaft Assigning a task to a remote expert
DE102022100840A1 (en) * 2022-01-14 2023-07-20 Sick Ag AUGMENTED REALITY SYSTEM FOR TEACHING A USER OF AN APPLICATION
DE102022206854A1 (en) 2022-07-05 2024-01-11 Robert Bosch Gesellschaft mit beschränkter Haftung Monitoring a machine with predicted visibility
CN115157261A (en) * 2022-07-27 2022-10-11 清华大学深圳国际研究生院 Flexible mechanical arm teleoperation man-machine interaction device and method based on mixed reality
US20240165812A1 (en) * 2022-11-23 2024-05-23 Dexterity, Inc. Safeguarded exit from physically constrained robotic workspace

Citations (2)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
US20160311115A1 (en) * 2015-04-27 2016-10-27 David M. Hill Enhanced configuration and control of robots
US9643314B2 (en) * 2015-03-04 2017-05-09 The Johns Hopkins University Robot control, training and collaboration in an immersive virtual reality environment

Family Cites Families (7)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
JP2008080472A (en) * 2006-09-29 2008-04-10 Toshiba Corp Robot system and its control method
US8718837B2 (en) 2011-01-28 2014-05-06 Intouch Technologies Interfacing with a mobile telepresence robot
JP5246672B2 (en) * 2011-02-17 2013-07-24 独立行政法人科学技術振興機構 Robot system
KR101978740B1 (en) 2012-02-15 2019-05-15 삼성전자주식회사 Tele-operation system and control method thereof
US9505130B2 (en) 2012-07-13 2016-11-29 General Electric Company System and method for performing remote welding operations on an apparatus
JP6440745B2 (en) * 2014-08-25 2018-12-19 エックス デベロップメント エルエルシー Method and system for augmented reality displaying a virtual representation of the action of a robotic device
JP2016107379A (en) * 2014-12-08 2016-06-20 ファナック株式会社 Robot system including augmented reality corresponding display

Patent Citations (2)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
US9643314B2 (en) * 2015-03-04 2017-05-09 The Johns Hopkins University Robot control, training and collaboration in an immersive virtual reality environment
US20160311115A1 (en) * 2015-04-27 2016-10-27 David M. Hill Enhanced configuration and control of robots

Non-Patent Citations (1)

* Cited by examiner, † Cited by third party
Title
HOOMAN HEDAYATI ET AL: "Improving Collocated Robot Teleoperation with Augmented Reality", HUMAN-ROBOT INTERACTION, ACM, 2 PENN PLAZA, SUITE 701NEW YORKNY10121-0701USA, 26 February 2018 (2018-02-26), pages 78 - 86, XP058387230, ISBN: 978-1-4503-4953-6, DOI: 10.1145/3171221.3171251 *

Also Published As

Publication number Publication date
US20210094180A1 (en) 2021-04-01
WO2019173396A1 (en) 2019-09-12
EP3762190A1 (en) 2021-01-13
JP2021515336A (en) 2021-06-17
JP2023018097A (en) 2023-02-07
US11986963B2 (en) 2024-05-21

Similar Documents

Publication Publication Date Title
EP3762190A4 (en) Augmented reality coordination of human-robot interaction
EP3629134B8 (en) Augmented reality interaction techniques
EP3625774A4 (en) Augmented reality
EP3709908A4 (en) Virtual reality wrist assembly
EP3629120B8 (en) Auto-recharging of robot
EP3676805A4 (en) Assisted augmented reality
EP3688526A4 (en) Augmented reality display
EP3412039A4 (en) Augmented reality headphone environment rendering
EP3655203A4 (en) Virtual robot base
EP3704689A4 (en) Chat-based application interface for automation
EP3555727A4 (en) Allocating virtual objects based on augmented reality
EP3526755A4 (en) Offline-service multi-user interaction based on augmented reality
EP3549004A4 (en) Augmented reality control of internet of things devices
EP3844725A4 (en) Augmented reality anthropomorphization system
EP3740809A4 (en) Interactive augmented or virtual reality devices
EP3646291A4 (en) Augmented reality assisted pickup
EP3516630A4 (en) Augmented reality spectroscopy
EP3649576A4 (en) Cloud enabled augmented reality
EP3647852A4 (en) Augmented reality glasses
EP3564731A4 (en) Virtual reality head-mounted apparatus
EP3814829A4 (en) Gaze based interface for augmented reality environment
EP3509581A4 (en) Formulations of (r
EP3268937A4 (en) Augmented reality memorial
EP3445661A4 (en) Packing robot
EP3561572A4 (en) Virtual reality head-mounted apparatus

Legal Events

Date Code Title Description
STAA Information on the status of an ep patent application or granted ep patent

Free format text: STATUS: THE INTERNATIONAL PUBLICATION HAS BEEN MADE

PUAI Public reference made under article 153(3) epc to a published international application that has entered the european phase

Free format text: ORIGINAL CODE: 0009012

STAA Information on the status of an ep patent application or granted ep patent

Free format text: STATUS: REQUEST FOR EXAMINATION WAS MADE

17P Request for examination filed

Effective date: 20201002

AK Designated contracting states

Kind code of ref document: A1

Designated state(s): AL AT BE BG CH CY CZ DE DK EE ES FI FR GB GR HR HU IE IS IT LI LT LU LV MC MK MT NL NO PL PT RO RS SE SI SK SM TR

AX Request for extension of the european patent

Extension state: BA ME

DAV Request for validation of the european patent (deleted)
DAX Request for extension of the european patent (deleted)
REG Reference to a national code

Ref country code: DE

Ref legal event code: R079

Free format text: PREVIOUS MAIN CLASS: B25J0009160000

Ipc: G05D0001000000

A4 Supplementary search report drawn up and despatched

Effective date: 20220104

RIC1 Information provided on ipc code assigned before grant

Ipc: B25J 9/00 20060101ALI20211221BHEP

Ipc: B25J 9/16 20060101ALI20211221BHEP

Ipc: G05D 1/00 20060101AFI20211221BHEP

STAA Information on the status of an ep patent application or granted ep patent

Free format text: STATUS: EXAMINATION IS IN PROGRESS

17Q First examination report despatched

Effective date: 20230712