WO2019106862A1 - 操作案内システム - Google Patents

操作案内システム Download PDF

Info

Publication number
WO2019106862A1
WO2019106862A1 PCT/JP2018/018684 JP2018018684W WO2019106862A1 WO 2019106862 A1 WO2019106862 A1 WO 2019106862A1 JP 2018018684 W JP2018018684 W JP 2018018684W WO 2019106862 A1 WO2019106862 A1 WO 2019106862A1
Authority
WO
WIPO (PCT)
Prior art keywords
image
unit
operator
movable
machine
Prior art date
Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
Ceased
Application number
PCT/JP2018/018684
Other languages
English (en)
French (fr)
Japanese (ja)
Inventor
徹 寺田
聡一 戎野
賢治 岩野
Current Assignee (The listed assignees may be inaccurate. Google has not performed a legal analysis and makes no representation or warranty as to the accuracy of the list.)
Schneider Electric Japan Holdings Ltd
Original Assignee
Schneider Electric Japan Holdings Ltd
Priority date (The priority date is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the date listed.)
Filing date
Publication date
Application filed by Schneider Electric Japan Holdings Ltd filed Critical Schneider Electric Japan Holdings Ltd
Priority to EP18882638.2A priority Critical patent/EP3718708A4/en
Priority to US16/767,382 priority patent/US11475606B2/en
Priority to CN201880076541.9A priority patent/CN111565898B/zh
Publication of WO2019106862A1 publication Critical patent/WO2019106862A1/ja
Anticipated expiration legal-status Critical
Ceased legal-status Critical Current

Links

Images

Classifications

    • G—PHYSICS
    • G06—COMPUTING OR CALCULATING; COUNTING
    • G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
    • G06T11/00—Two-dimensional [2D] image generation
    • B—PERFORMING OPERATIONS; TRANSPORTING
    • B25—HAND TOOLS; PORTABLE POWER-DRIVEN TOOLS; MANIPULATORS
    • B25J—MANIPULATORS; CHAMBERS PROVIDED WITH MANIPULATION DEVICES
    • B25J9/00—Program-controlled manipulators
    • B25J9/16—Program controls
    • B25J9/1656—Program controls characterised by programming, planning systems for manipulators
    • B25J9/1671—Program controls characterised by programming, planning systems for manipulators characterised by simulation, either to verify existing program or to create and verify new program, CAD/CAM oriented, graphic oriented programming systems
    • B—PERFORMING OPERATIONS; TRANSPORTING
    • B25—HAND TOOLS; PORTABLE POWER-DRIVEN TOOLS; MANIPULATORS
    • B25J—MANIPULATORS; CHAMBERS PROVIDED WITH MANIPULATION DEVICES
    • B25J13/00—Controls for manipulators
    • B25J13/02—Hand grip control means
    • B—PERFORMING OPERATIONS; TRANSPORTING
    • B25—HAND TOOLS; PORTABLE POWER-DRIVEN TOOLS; MANIPULATORS
    • B25J—MANIPULATORS; CHAMBERS PROVIDED WITH MANIPULATION DEVICES
    • B25J19/00—Accessories fitted to manipulators, e.g. for monitoring, for viewing; Safety devices combined with or specially adapted for use in connection with manipulators
    • B25J19/02—Sensing devices
    • B25J19/021—Optical sensing devices
    • B25J19/023—Optical sensing devices including video camera means
    • G—PHYSICS
    • G05—CONTROLLING; REGULATING
    • G05B—CONTROL OR REGULATING SYSTEMS IN GENERAL; FUNCTIONAL ELEMENTS OF SUCH SYSTEMS; MONITORING OR TESTING ARRANGEMENTS FOR SUCH SYSTEMS OR ELEMENTS
    • G05B19/00—Program-control systems
    • G05B19/02—Program-control systems electric
    • G05B19/18—Numerical control [NC], i.e. automatically operating machines, in particular machine tools, e.g. in a manufacturing environment, so as to execute positioning, movement or co-ordinated operations by means of program data in numerical form
    • G05B19/409—Numerical control [NC], i.e. automatically operating machines, in particular machine tools, e.g. in a manufacturing environment, so as to execute positioning, movement or co-ordinated operations by means of program data in numerical form characterised by using manual data input [MDI] or by using control panel, e.g. controlling functions with the panel; characterised by control panel details or by setting parameters
    • G—PHYSICS
    • G06—COMPUTING OR CALCULATING; COUNTING
    • G06F—ELECTRIC DIGITAL DATA PROCESSING
    • G06F3/00—Input arrangements for transferring data to be processed into a form capable of being handled by the computer; Output arrangements for transferring data from processing unit to output unit, e.g. interface arrangements
    • G06F3/01—Input arrangements or combined input and output arrangements for interaction between user and computer
    • G06F3/011—Arrangements for interaction with the human body, e.g. for user immersion in virtual reality
    • G—PHYSICS
    • G06—COMPUTING OR CALCULATING; COUNTING
    • G06F—ELECTRIC DIGITAL DATA PROCESSING
    • G06F3/00—Input arrangements for transferring data to be processed into a form capable of being handled by the computer; Output arrangements for transferring data from processing unit to output unit, e.g. interface arrangements
    • G06F3/14—Digital output to display device ; Cooperation and interconnection of the display device with other functional units
    • G—PHYSICS
    • G06—COMPUTING OR CALCULATING; COUNTING
    • G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
    • G06T7/00—Image analysis
    • G06T7/70—Determining position or orientation of objects or cameras
    • G06T7/73—Determining position or orientation of objects or cameras using feature-based methods
    • G—PHYSICS
    • G05—CONTROLLING; REGULATING
    • G05B—CONTROL OR REGULATING SYSTEMS IN GENERAL; FUNCTIONAL ELEMENTS OF SUCH SYSTEMS; MONITORING OR TESTING ARRANGEMENTS FOR SUCH SYSTEMS OR ELEMENTS
    • G05B2219/00—Program-control systems
    • G05B2219/30—Nc systems
    • G05B2219/36—Nc in input of data, input key till input tape
    • G05B2219/36167—Use camera of handheld device, pda, pendant, head mounted display
    • G—PHYSICS
    • G05—CONTROLLING; REGULATING
    • G05B—CONTROL OR REGULATING SYSTEMS IN GENERAL; FUNCTIONAL ELEMENTS OF SUCH SYSTEMS; MONITORING OR TESTING ARRANGEMENTS FOR SUCH SYSTEMS OR ELEMENTS
    • G05B2219/00—Program-control systems
    • G05B2219/30—Nc systems
    • G05B2219/39—Robotics, robotics to robotics hand
    • G05B2219/39451—Augmented reality for robot programming
    • G—PHYSICS
    • G06—COMPUTING OR CALCULATING; COUNTING
    • G06F—ELECTRIC DIGITAL DATA PROCESSING
    • G06F3/00—Input arrangements for transferring data to be processed into a form capable of being handled by the computer; Output arrangements for transferring data from processing unit to output unit, e.g. interface arrangements
    • G06F3/01—Input arrangements or combined input and output arrangements for interaction between user and computer
    • G06F3/03—Arrangements for converting the position or the displacement of a member into a coded form
    • G06F3/041—Digitisers, e.g. for touch screens or touch pads, characterised by the transducing means
    • G06F3/044—Digitisers, e.g. for touch screens or touch pads, characterised by the transducing means by capacitive means
    • G—PHYSICS
    • G06—COMPUTING OR CALCULATING; COUNTING
    • G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
    • G06T2207/00—Indexing scheme for image analysis or image enhancement
    • G06T2207/10—Image acquisition modality
    • G06T2207/10004—Still image; Photographic image
    • G06T2207/10012—Stereo images

Definitions

  • the present invention relates to an operation guidance system for guiding an operation of an operation target.
  • Patent Document 1 based on the acquired information on the position and the field of view of the user, a line image indicating the operation state of the robot is generated, and the line image is superimposed on the field of view of the user on the glasses type display.
  • a robot operation system is disclosed to display.
  • Patent Document 3 discloses an operation device for remotely operating a robot including a movable member.
  • this controller detects a switch operation less than a predetermined operation amount
  • a movable portion image showing a movable member that operates in response to the switch operation is displayed using the movable portion image of another movable member that is not activated and a color etc. Display in an identifiable manner.
  • the machine control device 1 is a device that controls the operation of the machine 8 (actuator 10).
  • the machine control device 1 includes an input unit 11, a control unit 12, and an output unit 13.
  • the operation terminal 2 is a device for the operator to input an operation instruction to operate the machine 8.
  • the operation terminal 2 includes a switch 21 and a touch sensor 22.
  • a photosensor, a two-stage switch, or the like can be used as a sensor that detects that a finger has touched the switch 21.
  • the operation terminal 2 may have an enable switch and an operation switch instead of the switch 21 and the touch sensor 22.
  • the guidance display model may be displayed as described later when only the operation switch is pressed, and the machine 8 may be operated when both the enable switch and the operation switch are pressed.
  • the video output device 33 is a device (such as a liquid crystal display) that displays the composite image output from the arithmetic processing unit 32 in front of the eye of the operator.
  • the positional relationship estimation unit 321 analyzes the plurality of pieces of image information acquired from each camera, and identifies the direction in which the operator is facing and the distance between the operator and the machine 8 to The positional relationship with the machine 8 is estimated.
  • the image combining unit 324 combines the three-dimensional model data and the captured image information (captured image data) so that the three-dimensional model data whose coordinates are converted is superimposed on the captured image.
  • the image combining unit 324 outputs the combined image data to the video output device 33.
  • the coordinate conversion unit 323 recognizes the distance from the position of the operator to the machine 8 and the angle at which the operator faces the machine 8 with respect to the front of the machine 8 based on the above positional relationship, and is mainly based on the angle. To recognize the direction indicated by the three-dimensional model for the operator. Then, the coordinate conversion unit 323 calculates direction specifying coordinates specifying the direction in the three-dimensional model data based on the direction, and replaces the direction specifying coordinates before conversion with the calculated direction specifying coordinates.
  • the operation terminal 2 receives an operation instruction to the movable unit 9 by the operator and detects an operation preparation state in which the operation instruction can be received.
  • the operator can easily confirm the movement direction of the movable part 9. Further, since the three-dimensional model is generated in the direction according to the direction in which the machine 8 (the movable unit 9) is viewed from the operator, when the operator changes the position, the direction of the three-dimensional model is also change.
  • the operation guidance image may include not only the operating direction of the movable portion 9 but also the name and function information of the switch 21 touched by the operator. Such information is also generated by the model generation unit 322.
  • a marker may be used as means for estimating the positional relationship between the machine 8 and the operator (image sensor 31) and detecting the position at which the guidance display model of the movable unit 9 is displayed.
  • the operation terminal 2A has a switch 21 and a touch sensor 22.
  • the operation terminal 2A further includes a communication unit 23.
  • the communication unit 23 transmits switch information and touch sensor information to the communication unit 14 of the machine control device 1A.
  • the operation guide systems 101 to 103 have a direction display unit (positional relationship estimation unit 321) that causes the display device to display the direction of the movable device from the operator when the imaging device does not image the movable device. You may provide further.

Landscapes

  • Engineering & Computer Science (AREA)
  • Theoretical Computer Science (AREA)
  • General Physics & Mathematics (AREA)
  • Physics & Mathematics (AREA)
  • Robotics (AREA)
  • Mechanical Engineering (AREA)
  • Human Computer Interaction (AREA)
  • General Engineering & Computer Science (AREA)
  • Computer Vision & Pattern Recognition (AREA)
  • Multimedia (AREA)
  • Manufacturing & Machinery (AREA)
  • Automation & Control Theory (AREA)
  • User Interface Of Digital Computer (AREA)
  • Manipulator (AREA)
  • Numerical Control (AREA)
PCT/JP2018/018684 2017-11-28 2018-05-15 操作案内システム Ceased WO2019106862A1 (ja)

Priority Applications (3)

Application Number Priority Date Filing Date Title
EP18882638.2A EP3718708A4 (en) 2017-11-28 2018-05-15 ACTION GUIDANCE SYSTEM
US16/767,382 US11475606B2 (en) 2017-11-28 2018-05-15 Operation guiding system for operation of a movable device
CN201880076541.9A CN111565898B (zh) 2017-11-28 2018-05-15 操作引导系统

Applications Claiming Priority (2)

Application Number Priority Date Filing Date Title
JP2017228079A JP6882147B2 (ja) 2017-11-28 2017-11-28 操作案内システム
JP2017-228079 2017-11-28

Publications (1)

Publication Number Publication Date
WO2019106862A1 true WO2019106862A1 (ja) 2019-06-06

Family

ID=66664763

Family Applications (1)

Application Number Title Priority Date Filing Date
PCT/JP2018/018684 Ceased WO2019106862A1 (ja) 2017-11-28 2018-05-15 操作案内システム

Country Status (5)

Country Link
US (1) US11475606B2 (https=)
EP (1) EP3718708A4 (https=)
JP (1) JP6882147B2 (https=)
CN (1) CN111565898B (https=)
WO (1) WO2019106862A1 (https=)

Families Citing this family (4)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
JP2023017440A (ja) * 2021-07-26 2023-02-07 株式会社ダイヘン 画像処理装置
JP7149440B1 (ja) 2022-04-21 2022-10-06 三菱製鋼株式会社 操作装置
WO2023248439A1 (ja) * 2022-06-23 2023-12-28 ファナック株式会社 ロボットシステム、ロボット制御装置およびロボット制御プログラム
WO2025052590A1 (ja) * 2023-09-06 2025-03-13 株式会社Fuji ロボットシステム

Citations (10)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
JP2007236008A (ja) * 2007-06-18 2007-09-13 Fujifilm Corp 画像表示器付きカメラ
JP2008070817A (ja) 2006-09-15 2008-03-27 Brother Ind Ltd ヘッドマウントディスプレイ
JP2009119579A (ja) * 2007-11-16 2009-06-04 Canon Inc 情報処理装置、情報処理方法
JP2011161586A (ja) 2010-02-12 2011-08-25 Tadano Ltd 作業用装置
JP2012218120A (ja) * 2011-04-12 2012-11-12 Seiko Epson Corp マニピュレーター動作予告装置、ロボットシステム及びマニピュレーター動作予告方法
JP2013022651A (ja) * 2011-07-15 2013-02-04 Olympus Corp マニピュレータシステム
JP2014235841A (ja) * 2013-05-31 2014-12-15 三菱電機株式会社 誘導加熱調理器
JP2015045990A (ja) * 2013-08-28 2015-03-12 Dmg森精機株式会社 表示装置
JP2016107379A (ja) * 2014-12-08 2016-06-20 ファナック株式会社 拡張現実対応ディスプレイを備えたロボットシステム
JP2017100204A (ja) 2015-11-30 2017-06-08 株式会社デンソーウェーブ ロボット操作システム

Family Cites Families (12)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
BRPI0508748B1 (pt) * 2004-03-26 2018-05-02 Takahashi Atsushi Sistema tridimensional para orientação e instruções visuais remotas, dotado de visor tridimensional com câmeras.
JP5784388B2 (ja) * 2011-06-29 2015-09-24 オリンパス株式会社 医療用マニピュレータシステム
JP6021353B2 (ja) * 2011-08-04 2016-11-09 オリンパス株式会社 手術支援装置
JP2013184257A (ja) * 2012-03-08 2013-09-19 Sony Corp ロボット装置及びロボット装置の制御方法、並びにコンピューター・プログラム
US9817396B1 (en) * 2014-06-09 2017-11-14 X Development Llc Supervisory control of an unmanned aerial vehicle
US9849981B1 (en) * 2014-08-28 2017-12-26 X Development Llc Payload-release device position tracking
US10179407B2 (en) * 2014-11-16 2019-01-15 Robologics Ltd. Dynamic multi-sensor and multi-robot interface system
US9836117B2 (en) * 2015-05-28 2017-12-05 Microsoft Technology Licensing, Llc Autonomous drones for tactile feedback in immersive virtual reality
JP6643843B2 (ja) * 2015-09-14 2020-02-12 オリンパス株式会社 撮像操作ガイド装置および撮像装置の操作ガイド方法
US20180098052A1 (en) * 2016-09-30 2018-04-05 Sony Interactive Entertainment Inc. Translation of physical object viewed by unmanned aerial vehicle into virtual world object
CN107223082B (zh) 2017-04-21 2020-05-12 深圳前海达闼云端智能科技有限公司 一种机器人控制方法、机器人装置及机器人设备
US10852847B2 (en) * 2017-07-26 2020-12-01 Google Llc Controller tracking for multiple degrees of freedom

Patent Citations (10)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
JP2008070817A (ja) 2006-09-15 2008-03-27 Brother Ind Ltd ヘッドマウントディスプレイ
JP2007236008A (ja) * 2007-06-18 2007-09-13 Fujifilm Corp 画像表示器付きカメラ
JP2009119579A (ja) * 2007-11-16 2009-06-04 Canon Inc 情報処理装置、情報処理方法
JP2011161586A (ja) 2010-02-12 2011-08-25 Tadano Ltd 作業用装置
JP2012218120A (ja) * 2011-04-12 2012-11-12 Seiko Epson Corp マニピュレーター動作予告装置、ロボットシステム及びマニピュレーター動作予告方法
JP2013022651A (ja) * 2011-07-15 2013-02-04 Olympus Corp マニピュレータシステム
JP2014235841A (ja) * 2013-05-31 2014-12-15 三菱電機株式会社 誘導加熱調理器
JP2015045990A (ja) * 2013-08-28 2015-03-12 Dmg森精機株式会社 表示装置
JP2016107379A (ja) * 2014-12-08 2016-06-20 ファナック株式会社 拡張現実対応ディスプレイを備えたロボットシステム
JP2017100204A (ja) 2015-11-30 2017-06-08 株式会社デンソーウェーブ ロボット操作システム

Non-Patent Citations (1)

* Cited by examiner, † Cited by third party
Title
See also references of EP3718708A4

Also Published As

Publication number Publication date
JP6882147B2 (ja) 2021-06-02
EP3718708A1 (en) 2020-10-07
US11475606B2 (en) 2022-10-18
CN111565898A (zh) 2020-08-21
EP3718708A4 (en) 2021-12-22
CN111565898B (zh) 2023-04-21
JP2019101476A (ja) 2019-06-24
US20200394823A1 (en) 2020-12-17

Similar Documents

Publication Publication Date Title
JP5936155B2 (ja) ３次元ユーザインタフェース装置及び３次元操作方法
US10198870B2 (en) Information processing apparatus, information processing system, and information processing method
JP5871345B2 (ja) ３次元ユーザインタフェース装置及び３次元操作方法
JP5843340B2 (ja) ３次元環境共有システム及び３次元環境共有方法
US20160357491A1 (en) Information processing apparatus, information processing method, non-transitory computer-readable storage medium, and system
JPWO2014141504A1 (ja) ３次元ユーザインタフェース装置及び３次元操作処理方法
WO2019022803A1 (en) TRACKING OF CONTROL DEVICE FOR MULTIPLE DEGREES OF FREEDOM
JP2017059062A (ja) 情報処理装置、情報処理方法およびプログラム
WO2015093130A1 (ja) 情報処理装置、情報処理方法およびプログラム
JPWO2005119591A1 (ja) 表示制御方法および装置、プログラム、並びに携帯機器
JP2012108842A (ja) 表示システム、表示処理装置、表示方法、および表示プログラム
WO2019106862A1 (ja) 操作案内システム
JP2024056455A (ja) 情報処理システム、情報処理方法、プログラム
US9760180B2 (en) User interface device and user interface method
JP7670364B2 (ja) 分離型仮想システムの空間位置特定方法、仮想システム
US20250013418A1 (en) Work assistance system
CN121753074A (zh) 信息处理设备、信息处理系统、信息处理方法和程序
JP2025162718A (ja) 情報処理装置、情報処理装置の制御方法、及びプログラム
JP2025091784A (ja) 情報処理装置、情報処理装置の制御方法、システム、およびプログラム
HK1210858B (en) Three-dimensional user-interface device, and three-dimensional operation method
HK1205311B (en) Three-dimensional user-interface device, and three-dimensional operation method

Legal Events

Date Code Title Description
121 Ep: the epo has been informed by wipo that ep was designated in this application

Ref document number: 18882638

Country of ref document: EP

Kind code of ref document: A1

NENP Non-entry into the national phase

Ref country code: DE

ENP Entry into the national phase

Ref document number: 2018882638

Country of ref document: EP

Effective date: 20200629