WO2021189224A1 - 一种面向机器人增强现实示教的注册系统和方法 - Google Patents

一种面向机器人增强现实示教的注册系统和方法 Download PDF

Info

Publication number
WO2021189224A1
WO2021189224A1 PCT/CN2020/080786 CN2020080786W WO2021189224A1 WO 2021189224 A1 WO2021189224 A1 WO 2021189224A1 CN 2020080786 W CN2020080786 W CN 2020080786W WO 2021189224 A1 WO2021189224 A1 WO 2021189224A1
Authority
WO
WIPO (PCT)
Prior art keywords
robot
physical
coordinate system
physical robot
registration
Prior art date
Application number
PCT/CN2020/080786
Other languages
English (en)
French (fr)
Inventor
陈成军
丁旭彤
潘勇
李东年
洪军
Original Assignee
青岛理工大学
Priority date (The priority date is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the date listed.)
Filing date
Publication date
Application filed by 青岛理工大学 filed Critical 青岛理工大学
Priority to US17/417,416 priority Critical patent/US11969904B2/en
Priority to CA3124850A priority patent/CA3124850A1/en
Priority to PCT/CN2020/080786 priority patent/WO2021189224A1/zh
Publication of WO2021189224A1 publication Critical patent/WO2021189224A1/zh

Links

Images

Classifications

    • B—PERFORMING OPERATIONS; TRANSPORTING
    • B25—HAND TOOLS; PORTABLE POWER-DRIVEN TOOLS; MANIPULATORS
    • B25J—MANIPULATORS; CHAMBERS PROVIDED WITH MANIPULATION DEVICES
    • B25J9/00—Programme-controlled manipulators
    • B25J9/16—Programme controls
    • B25J9/1694—Programme controls characterised by use of sensors other than normal servo-feedback from position, speed or acceleration sensors, perception control, multi-sensor controlled systems, sensor fusion
    • B25J9/1697—Vision controlled systems
    • B—PERFORMING OPERATIONS; TRANSPORTING
    • B25—HAND TOOLS; PORTABLE POWER-DRIVEN TOOLS; MANIPULATORS
    • B25J—MANIPULATORS; CHAMBERS PROVIDED WITH MANIPULATION DEVICES
    • B25J9/00—Programme-controlled manipulators
    • B25J9/02—Programme-controlled manipulators characterised by movement of the arms, e.g. cartesian coordinate type
    • B25J9/023—Cartesian coordinate type
    • B—PERFORMING OPERATIONS; TRANSPORTING
    • B25—HAND TOOLS; PORTABLE POWER-DRIVEN TOOLS; MANIPULATORS
    • B25J—MANIPULATORS; CHAMBERS PROVIDED WITH MANIPULATION DEVICES
    • B25J9/00—Programme-controlled manipulators
    • B25J9/16—Programme controls
    • B25J9/1628—Programme controls characterised by the control loop
    • B25J9/163—Programme controls characterised by the control loop learning, adaptive, model based, rule based expert control
    • B—PERFORMING OPERATIONS; TRANSPORTING
    • B25—HAND TOOLS; PORTABLE POWER-DRIVEN TOOLS; MANIPULATORS
    • B25J—MANIPULATORS; CHAMBERS PROVIDED WITH MANIPULATION DEVICES
    • B25J9/00—Programme-controlled manipulators
    • B25J9/16—Programme controls
    • B25J9/1628—Programme controls characterised by the control loop
    • B25J9/1653—Programme controls characterised by the control loop parameters identification, estimation, stiffness, accuracy, error analysis
    • B—PERFORMING OPERATIONS; TRANSPORTING
    • B25—HAND TOOLS; PORTABLE POWER-DRIVEN TOOLS; MANIPULATORS
    • B25J—MANIPULATORS; CHAMBERS PROVIDED WITH MANIPULATION DEVICES
    • B25J9/00—Programme-controlled manipulators
    • B25J9/16—Programme controls
    • B25J9/1656—Programme controls characterised by programming, planning systems for manipulators
    • B—PERFORMING OPERATIONS; TRANSPORTING
    • B25—HAND TOOLS; PORTABLE POWER-DRIVEN TOOLS; MANIPULATORS
    • B25J—MANIPULATORS; CHAMBERS PROVIDED WITH MANIPULATION DEVICES
    • B25J9/00—Programme-controlled manipulators
    • B25J9/16—Programme controls
    • B25J9/1656—Programme controls characterised by programming, planning systems for manipulators
    • B25J9/1671—Programme controls characterised by programming, planning systems for manipulators characterised by simulation, either to verify existing program or to create and verify new program, CAD/CAM oriented, graphic oriented programming systems
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
    • G06T7/00—Image analysis
    • G06T7/70—Determining position or orientation of objects or cameras
    • G—PHYSICS
    • G09—EDUCATION; CRYPTOGRAPHY; DISPLAY; ADVERTISING; SEALS
    • G09G—ARRANGEMENTS OR CIRCUITS FOR CONTROL OF INDICATING DEVICES USING STATIC MEANS TO PRESENT VARIABLE INFORMATION
    • G09G5/00—Control arrangements or circuits for visual indicators common to cathode-ray tube indicators and other visual indicators
    • G09G5/36—Control arrangements or circuits for visual indicators common to cathode-ray tube indicators and other visual indicators characterised by the display of a graphic pattern, e.g. using an all-points-addressable [APA] memory
    • G09G5/37—Details of the operation on graphic patterns
    • G09G5/377—Details of the operation on graphic patterns for mixing or overlaying two or more graphic patterns
    • G—PHYSICS
    • G05—CONTROLLING; REGULATING
    • G05B—CONTROL OR REGULATING SYSTEMS IN GENERAL; FUNCTIONAL ELEMENTS OF SUCH SYSTEMS; MONITORING OR TESTING ARRANGEMENTS FOR SUCH SYSTEMS OR ELEMENTS
    • G05B2219/00—Program-control systems
    • G05B2219/30—Nc systems
    • G05B2219/39—Robotics, robotics to robotics hand
    • G05B2219/39451—Augmented reality for robot programming

Definitions

  • the invention relates to a registration system and method for robot augmented reality teaching, belonging to the field of intelligent manufacturing and robot teaching.
  • Augmented reality registration technology accurately superimposes virtual objects and virtual information on the real world, enabling observers to see a scene that combines virtual and real, and perceives virtual objects as a component of the surrounding real world from the senses.
  • Augmented reality registration technology is one of the most important technologies in the augmented reality system, and it is of great significance to the realization of the augmented reality system.
  • the robot augmented reality teaching accurately superimposes the virtual robot model in the physical production environment. The operator uses the human-computer interaction device to drive the virtual robot model to move, and plan the robot's path in a virtual and real superimposed scene.
  • the patent "Robot Teaching System and Method Based on RGB-D Image and Teach Pendant” uses the augmented reality registration technical solution: manually place an AR registration card, and the computer uses the camera to collect AR registration The image of the card is detected by the machine vision and augmented reality registration algorithm to detect the logo on the AR registration card, and the conversion matrix between the camera and the AR registration card is calculated. The computer uses the conversion matrix to set the position of the virtual camera in the virtual world, and the virtual robot model Place it exactly on the position of the AR registration card in the image.
  • this technical solution requires the AR registration card to be placed in advance.
  • the present invention provides a registration system and method for robot augmented reality teaching. It is not necessary to install an AR registration card on a physical entity other than the robot in advance to realize augmented reality registration and realize the virtual robot model.
  • the base coordinate system is consistent with the base coordinate system of the physical robot, which is suitable for remote augmented reality teaching of industrial robots and augmented reality teaching of mobile industrial robots.
  • a registration system for robot augmented reality teaching includes a physical robot unit, a registration unit, a virtual robot generation unit, and a computer;
  • the physical robot unit includes a physical robot, a physical robot controller, and a robot point intermittent motion control program; the physical robot is provided with a physical robot base coordinate system; the physical robot controller is connected to the physical robot and the computer respectively , The physical robot controller is used to control the motion of the physical robot and obtain the motion trajectory of the physical robot; the computer sends a point-to-point intermittent motion control program to the physical robot controller, and the physical robot controller executes the robot intermittent motion control A program that controls the actuator of the physical robot to perform at least three or more broken line movements;
  • the registration unit includes a registration identifier, a camera, and a conversion calculation unit;
  • the registration identifier is set on the body of the physical robot;
  • the camera is fixedly installed in a physical environment other than the physical robot, and is aimed at the working area of the physical robot Shooting;
  • the camera is connected to the computer, and the conversion calculation unit is set in the computer for calculating the three-dimensional coordinates of the registered marker in the camera coordinate system and the three-dimensional coordinates of the registered marker in the physical robot base coordinate system The conversion relationship between;
  • the virtual robot generating unit is arranged in the computer and is used to generate a virtual robot model.
  • the registered identifier is arranged on an actuator at the end of the physical robot, and the registered identifier is an object with a characteristic shape or color.
  • the registered identifier is arranged on a joint of the physical robot, and the registered identifier is an object with a characteristic shape or color.
  • the computer is specifically configured to: at each intermittent point of the intermittent motion, the computer reads the three-dimensional coordinates of the registered marker in the physical robot base coordinate system through the robot controller; at the same time, obtains the physical working environment through the camera Using computer vision algorithms to identify registered markers and calculate the pixel coordinates of the registered markers in the pixel coordinate system.
  • the conversion calculation unit is specifically configured to use the imaging model of the camera and the internal parameter matrix of the camera, according to the pixel coordinates of the registered marker in the pixel coordinate system and the three-dimensionality of the registered marker in the physical robot base coordinate system. Coordinates construct the conversion relationship between the camera coordinate system and the base coordinate system of the physical robot by constructing the least squares problem.
  • a registration method for robot augmented reality teaching is implemented based on the registration system for robot augmented reality teaching described in Technical Solution 1. The specific steps are as follows:
  • the camera takes an image and sends it to a computer.
  • the computer calculates the pixel coordinates of the registered marker in the image.
  • the computer obtains the three-dimensional coordinates of the registered marker in the physical robot base coordinate system through the physical robot controller;
  • the conversion calculation unit reads the pixel coordinates of the registered marker in the image and the three-dimensional coordinates of the registered marker in the base coordinate system of the physical robot, and calculates the conversion relationship between the camera coordinate system and the base coordinate system of the physical robot;
  • the virtual robot generating unit generates a virtual robot model consistent with the physical robot base coordinate system according to the conversion relationship between the calculated camera coordinate system and the physical robot base coordinate system, and superimposes the virtual robot model on the real scene through the augmented reality device In, complete the registration.
  • the registered identifier is arranged on an actuator at the end of the physical robot, and the registered identifier is an object with a characteristic shape or color.
  • the registered identifier is arranged on a joint of the physical robot, and the registered identifier is an object with a characteristic shape or color.
  • the camera takes an image and sends it to a computer.
  • the computer calculates the pixel coordinates of the registered marker in the image.
  • the computer obtains the three-dimensional image of the registered marker in the physical robot base coordinate system through the physical robot controller.
  • the computer controls the camera to shoot, and at the same time reads the three-dimensional coordinates of the current registered marker in the physical robot base coordinate system.
  • the conversion calculation unit reads the pixel coordinates of the registered marker in the image and the three-dimensional coordinates of the registered marker in the base coordinate system of the physical robot, and calculates the conversion relationship between the camera coordinate system and the base coordinate system of the physical robot.
  • the specific steps are:
  • the conversion calculation unit uses the imaging model of the camera and the internal parameter matrix of the camera to construct a least squares problem based on the pixel coordinates of the registered marker in the pixel coordinate system and the three-dimensional coordinates of the registered marker in the physical robot base coordinate system. , Calculate the conversion relationship between the camera coordinate system and the physical robot base coordinate system.
  • the present invention is a registration system and method for robot augmented reality teaching. It is not necessary to install an AR registration card on a physical entity other than the robot in advance to realize augmented reality registration and realize the base coordinate system of the virtual robot model and the physical robot.
  • the base coordinate system is consistent, which is suitable for remote augmented reality teaching of industrial robots and augmented reality teaching of mobile industrial robots;
  • Figure 1 is a schematic diagram of a registration system for robot augmented reality teaching according to the present invention
  • FIG. 2 is a schematic diagram of the use of a registration system for robot augmented reality teaching according to the present invention
  • Fig. 3 is a schematic flowchart of a registration method for robot augmented reality teaching according to the present invention.
  • a registration system for robot augmented reality teaching including a physical robot unit, a registration unit, a virtual robot generation unit, and a computer;
  • the physical robot unit includes a physical robot, a physical robot controller, and a robot point intermittent motion control program; the physical robot is provided with a physical robot base coordinate system; the physical robot controller is connected to the physical robot and the computer respectively The physical robot controller is used to control the motion of the physical robot and obtain the motion trajectory of the physical robot; the robot point-to-point intermittent motion control program is installed in the computer, and the computer sends the point-to-point intermittent motion control program to the physical robot.
  • the robot controller executes the robot intermittent motion control program by the physical robot controller, controls the execution mechanism of the physical robot to perform at least three or more fold line motions, and can control the robot to pause motion at the turning point of the fold line segment;
  • the registration unit includes a registration identifier, a camera, and a conversion calculation unit;
  • the registration identifier is set on the body of the physical robot;
  • the camera is fixedly installed in a physical environment other than the physical robot, and is aimed at the working area of the physical robot Shooting;
  • the camera is connected to the computer, and the conversion calculation unit is set in the computer for calculating the three-dimensional coordinates of the registered marker in the camera coordinate system and the three-dimensional coordinates of the registered marker in the physical robot base coordinate system The conversion relationship between;
  • the virtual robot generating unit is arranged in the computer and is used to generate a virtual robot model.
  • This embodiment does not need to install an AR registration card on a physical entity other than the physical robot in advance to realize augmented reality registration, avoids the impact of the installation accuracy of the AR registration card on the accuracy of the augmented reality registration, and can realize the base of the virtual robot with high precision.
  • the coordinate system is consistent with the base coordinate system of the physical robot, which improves the accuracy of the teaching path and is suitable for remote augmented reality teaching of industrial robots and augmented reality teaching of mobile industrial robots.
  • the registered marker is set on the actuator at the end of the physical robot, and the coordinates of the registered marker in the robot coordinate system can be read in real time.
  • the registered marker has a characteristic shape. (Such as sphere, cube, etc.) or color (such as red, yellow) objects.
  • the registered marker is set on the joints of the physical robot, and the joint data of the robot can be read in real time, and the coordinates of the marker in the robot coordinate system can be obtained through the forward kinematics model of the robot, in order to facilitate the detection and calculation
  • the registered identifier is an object with a characteristic shape or color.
  • the computer is specifically configured to: at each intermittent point of the intermittent motion, that is, the endpoint of each line segment in the intermittent motion, the computer reads the registered identifier in the physical robot base coordinate system O w -through the robot controller.
  • the robot intermittent motion trajectory contains at least 3 or more polyline segments, so the data of at least 4 non-collinear feature points (such as 4 endpoints of a square) can be obtained, that is, the data of 4 Pi data.
  • the conversion calculation unit is specifically configured to calculate the conversion matrix R: assuming that the conversion matrix from the world coordinate system to the camera coordinate system is R, the following relational expression can be obtained by using the imaging model of the camera:
  • R 4*4 is the transformation matrix from the world coordinate system to the camera coordinate system as R.
  • FIG. 3 a registration method for robot augmented reality teaching.
  • the method is implemented based on the registration system for robot augmented reality teaching described in Embodiment 1.
  • the specific steps are as follows:
  • the camera takes an image and sends it to a computer.
  • the computer calculates the pixel coordinates of the registered marker in the pixel coordinate system.
  • the computer obtains the three-dimensional coordinates of the registered marker in the physical robot base coordinate system through the physical robot controller;
  • the conversion calculation unit reads the pixel coordinates of the registered marker in the image and the three-dimensional coordinates of the registered marker in the base coordinate system of the physical robot, and calculates the conversion relationship between the camera coordinate system and the base coordinate system of the physical robot;
  • the virtual robot generating unit generates a virtual robot model consistent with the physical robot base coordinate system according to the conversion relationship between the calculated camera coordinate system and the physical robot base coordinate system, and superimposes the virtual robot model on the real scene through the augmented reality device In, complete the registration.
  • This embodiment does not need to install an AR registration card on a physical entity other than the physical robot in advance to realize augmented reality registration, avoids the impact of the installation accuracy of the AR registration card on the accuracy of the augmented reality registration, and can realize the base of the virtual robot with high precision.
  • the coordinate system is consistent with the base coordinate system of the physical robot, which improves the accuracy of the teaching path and is suitable for remote augmented reality teaching of industrial robots and augmented reality teaching of mobile industrial robots.
  • the registered marker is set on the actuator at the end of the physical robot, and the coordinates of the registered marker in the robot coordinate system can be read in real time.
  • the registered marker has a characteristic shape. (Such as sphere, cube, etc.) or color (such as red, yellow) objects.
  • the registered marker is set on the joints of the physical robot, and the joint data of the robot can be read in real time, and the coordinates of the marker in the robot coordinate system can be obtained through the forward kinematics model of the robot, in order to facilitate the detection and calculation
  • the registered identifier is an object with a characteristic shape or color.
  • the camera takes an image and sends it to a computer.
  • the computer calculates the pixel coordinates of the registered marker in the image.
  • the computer obtains the three-dimensional image of the registered marker in the physical robot base coordinate system through the physical robot controller. step coordinates: in each of the intermittent movement of the intermittent point, the computer controlling the camera to shoot, and the identifier registered identification register identifier calculated pixel coordinate Z at the pixel coordinates i (u i, v i) , while reading current three-dimensional coordinates on the physical register identifier of the robot base coordinate system O w -X w Y w Z w of P i (x wi, y wi , z wi).
  • the conversion calculation unit reads the pixel coordinates of the registered marker in the image and the three-dimensional coordinates of the registered marker in the base coordinate system of the physical robot, and calculates the conversion relationship between the camera coordinate system and the base coordinate system of the physical robot.
  • the specific steps are:
  • R 4*4 is the transformation matrix from the world coordinate system to the camera coordinate system as R.

Abstract

一种面向机器人增强现实示教的注册系统：包括物理机器人单元、注册单元、虚拟机器人生成单元以及计算机；物理机器人单元包括物理机器人、物理机器人控制器和机器人点位间歇运动控制程序；所述物理机器人自设置有物理机器人基坐标系；所述物理机器人控制器分别与所述物理机器人和计算机连接；所述机器人点位间歇运动控制程序安装于所述计算机中；注册单元包括注册标识物、相机和转换计算单元；所述注册标识物设置在物理机器人本体上；所述相机固定安装在物理机器人以外的物理环境中；所述相机连接所述计算机，所述转换计算单元设置于所述计算机中；所述虚拟机器人生成单元设置于所述计算机中，用于生成虚拟机器人模型。

Description

一种面向机器人增强现实示教的注册系统和方法 技术领域
本发明涉及一种面向机器人增强现实示教的注册系统和方法，属于智能制造和机器人示教领域。
背景技术
当前，增强现实技术在制造业中的应用日益广泛。增强现实注册技术将虚拟物体和虚拟信息准确叠加在真实世界上，使观察者能够看到一个虚实结合的场景，从感官上认为虚拟物体是周围现实世界的组成部分。增强现实注册技术是增强现实系统中至关重要的技术之一，对增强现实系统的实现具有重要意义。机器人增强现实示教将虚拟的机器人模型准确叠加在物理的生产环境中，操作者使用人机交互设备驱动虚拟机器人模型运动，在一个虚实叠加的场景中规划机器人的路径。
专利《基于RGB-D图像及示教器的机器人示教系统及方法》(公开号为CN201910665326.9)使用的增强现实注册技术方案为：人工放置一张AR注册卡，计算机使用摄像头采集AR注册卡的图像，通过机器视觉和增强现实注册算法检测AR注册卡上的标识，计算摄像机与AR注册卡之间的转换矩阵，计算机以该转换矩阵设置虚拟世界中虚拟相机的方位，将虚拟机器人模型准确地放置在图像中AR注册卡的位置上。但是此技术方案需要提前放置AR注册卡，如果虚拟机器人基坐标与AR注册卡的坐标不重合，则需要提前测得两个坐标系之间坐标的转换关系。另外该方法在有物理机器人的情况下，为了使虚拟机器人的基坐标系与物理机器人的基坐标系一致，需要提前测量AR注册卡和机器人基坐标系间的转换矩阵，这种方式人机交互性差，不适合机器人远程示教和移动工 业机器人(放置在移动平台如AGV上的工业机器人)示教。
发明内容
为了解决上述技术问题，本发明提供一种面向机器人增强现实示教的注册系统和方法，不需要在机器人以外的物理实体上提前安装AR注册卡，即可实现增强现实注册，实现虚拟机器人模型的基坐标系与物理机器人的基坐标系的一致，适合工业机器人远程增强现实示教和移动工业机器人增强现实示教。
本发明所采用的技术方案如下：
技术方案一：
一种面向机器人增强现实示教的注册系统：包括物理机器人单元、注册单元、虚拟机器人生成单元以及计算机；
所述物理机器人单元包括物理机器人、物理机器人控制器和机器人点位间歇运动控制程序；所述物理机器人自设置有物理机器人基坐标系；所述物理机器人控制器分别与所述物理机器人和计算机连接，所述物理机器人控制器用于控制物理机器人运动，并获取物理机器人的运动轨迹；所述计算机向物理机器人控制器发送点位间歇运动控制程序，所述物理机器人控制器执行所述机器人间歇运动控制程序，控制所述物理机器人的执行机构进行至少三条及以上的折线运动；
所述注册单元包括注册标识物、相机和转换计算单元；所述注册标识物设置在物理机器人本体上；所述相机固定安装在物理机器人以外的物理环境中，对准所述物理机器人的工作区域拍摄；所述相机连接所述计算机，所述转换计算单元设置于所述计算机中用于计算注册标识物在相机坐标系下的三维坐标与注册标识物在物理机器人基坐标系下的三维坐标之间的转换关系；
所述虚拟机器人生成单元设置于所述计算机中，用于生成虚拟机器人模型。
进一步的，所述注册标识物设置于所述物理机器人末端的执行机构上，所述注册标识物为具有特点形状或颜色的物体。
可替代的，所述注册标识物设置于所述物理机器人的关节上，所述注册标识物为具有特点形状或颜色的物体。
进一步的，所述计算机具体用于：在间歇运动的每个间歇点，所述计算机通过机器人控制器读取注册标识物在物理机器人基坐标系下的三维坐标；同时，通过相机获取物理工作环境的图像，通过计算机视觉算法识别注册标识物并计算注册标识物在像素坐标系下的像素坐标。
进一步的，所述转换计算单元具体用于：利用相机的成像模型及相机的内参矩阵，根据所述注册标识物在像素坐标系下的像素坐标以及注册标识物在物理机器人基坐标系下的三维坐标通过构建最小二乘问题相机坐标系与物理机器人基坐标系间的转换关系。
技术方案二
一种面向机器人增强现实示教的注册方法，该方法是基于技术方案一所述的一种面向机器人增强现实示教的注册系统实现的，具体步骤如下：
将注册标识物安装在所述物理机器人上，并固定好相机；
在计算机上编写机器人点位间歇运动控制程序并发送给物理机器人控制器，所述机器人点位间歇运动控制程序至少控制所述物理机器人的执行机构进行至少三次以上的折线运动；
将编写完成的机器人点位间歇运动控制程序下载至所述物理机器人控制器，所述物理机器人控制器执行所述机器人点位间歇运动控制程序控制所述物理机 器人动作；
相机拍摄图像，并发送至计算机中，所述计算机计算注册标识物在图像中的像素坐标，同时，计算机通过物理机器人控制器获取注册标识物在物理机器人基坐标系下的三维坐标；
转换计算单元读取注册标识物在图像中的像素坐标和注册标识物在物理机器人基坐标系下的三维坐标，并计算相机坐标系与物理机器人基坐标系间的转换关系；
所述虚拟机器人生成单元根据所述计算相机坐标系与物理机器人基坐标系间的转换关系，生成与物理机器人基坐标系一致的虚拟机器人模型，并通过增强现实设备将虚拟机器人模型叠加在现实场景中，完成注册。
进一步的，所述注册标识物设置于所述物理机器人末端的执行机构上，所述注册标识物为具有特点形状或颜色的物体。
可替代的，所述注册标识物设置于所述物理机器人的关节上，所述注册标识物为具有特点形状或颜色的物体。
进一步的，所述相机拍摄图像，并发送至计算机中，所述计算机计算注册标识物在图像中的像素坐标，同时，计算机通过物理机器人控制器获取注册标识物在物理机器人基坐标系下的三维坐标的步骤中：在间歇运动的每个间歇点，计算机控制相机进行拍摄，同时读取当前注册标识物在物理机器人基坐标系下的三维坐标。
进一步的，所述转换计算单元读取注册标识物在图像中的像素坐标和注册标识物在物理机器人基坐标系下的三维坐标，并计算相机坐标系与物理机器人基坐标系间的转换关系的步骤具体为：
所述转换计算单元利用相机的成像模型及相机的内参矩阵，根据所述注册标识物在像素坐标系下的像素坐标以及注册标识物在物理机器人基坐标系下的三维坐标通过构建最小二乘问题，计算相机坐标系与物理机器人基坐标系间的转换关系。
本发明具有如下有益效果：
1、本发明面向于机器人增强现实示教的注册系统和方法，不需要在机器人以外的物理实体上提前安装AR注册卡，即可实现增强现实注册，实现虚拟机器人模型的基坐标系与物理机器人的基坐标系的一致，适合工业机器人远程增强现实示教和移动工业机器人增强现实示教；
2、由于无需在物理场景中安装AR注册卡，避免了AR注册卡安装精度对增强现实注册精度的影响，可以高精度地实现虚拟机器人的基坐标系与物理机器人的基坐标系的一致，提高示教路径的精度。
附图说明
图1为本发明一种面向机器人增强现实示教的注册系统的示意图；
图2为本发明一种面向机器人增强现实示教的注册系统的使用示意图；
图3为本发明一种面向机器人增强现实示教的注册方法的流程示意图。
具体实施方式
下面结合附图和具体实施例来对本发明进行详细的说明。
实施例一
请参阅图1和图2，一种面向机器人增强现实示教的注册系统：包括物理机器人单元、注册单元、虚拟机器人生成单元以及计算机；
所述物理机器人单元包括物理机器人、物理机器人控制器和机器人点位间 歇运动控制程序；所述物理机器人自设置有物理机器人基坐标系；所述物理机器人控制器分别与所述物理机器人和计算机连接，所述物理机器人控制器用于控制物理机器人运动，并获取物理机器人的运动轨迹；所述机器人点位间歇运动控制程序安装于所述计算机中，所述计算机将点位间歇运动控制程序发送至物理机器人控制器由所述物理机器人控制器执行所述机器人间歇运动控制程序，控制所述物理机器人的执行机构进行至少三条及以上的折线运动，可以控制机器人在折线段的转折点处暂停运动；
所述注册单元包括注册标识物、相机和转换计算单元；所述注册标识物设置在物理机器人本体上；所述相机固定安装在物理机器人以外的物理环境中，对准所述物理机器人的工作区域拍摄；所述相机连接所述计算机，所述转换计算单元设置于所述计算机中用于计算注册标识物在相机坐标系下的三维坐标与注册标识物在物理机器人基坐标系下的三维坐标之间的转换关系；
所述虚拟机器人生成单元设置于所述计算机中，用于生成虚拟机器人模型。
本实施例不需要在物理机器人以外的物理实体上提前安装AR注册卡，即可实现增强现实注册，避免了AR注册卡安装精度对增强现实注册精度的影响，可以高精度地实现虚拟机器人的基坐标系与物理机器人的基坐标系的一致，提高示教路径的精度，适合工业机器人远程增强现实示教和移动工业机器人增强现实示教。
实施例二
进一步的，所述注册标识物设置于所述物理机器人末端的执行机构上，可实时读取注册标识物在机器人坐标系下的坐标，为了方便检测计算方便，所述 注册标识物为具有特点形状(如球体、立方体等)或颜色(如红色、黄色)的物体。
可替代的，所述注册标识物设置于所述物理机器人的关节上，可以实时读取机器人关节数据，通过机器人正运动学模型，求得该标识在机器人坐标系下的坐标，为了方便检测计算方便，所述注册标识物为具有特点形状或颜色的物体。
进一步的，所述计算机具体用于：在间歇运动的每个间歇点，即间歇运动中每条线段的端点，所述计算机通过机器人控制器读取注册标识物在物理机器人基坐标系O w-X wY wZ w下的三维坐标P i(x wi,y wi,z wi)；同时，通过相机获取物理工作环境的图像，在计算机上通过计算机视觉算法识别注册标识物，并计算注册标识物在像素坐标系下的像素坐标Z i(u i,v i)。具体参见图2中的间歇运动路径，机器人间歇运动轨迹至少包含3条及以上的折线段，因此可以获得至少4个非共线特征点(如正方形4个端点)的数据，即4个Pi的数据。
进一步的，所述转换计算单元具体用于计算转换矩阵R：假设由世界坐标系到相机坐标系的转换矩阵为R，利用相机的成像模型可以得到如下关系式：
Figure PCTCN2020080786-appb-000001
其中A 3*4为相机的内参矩阵，Q i(x ci,y ci,z ci)为注册标识物在相机坐标系O c-X cY cZ c中的三维坐标。
根据所有特征点(4个及以上)的像素坐标Z i(u i,v i),和对应的三维坐标P i(x wi,y wi,z wi)，以下式为目标
Figure PCTCN2020080786-appb-000002
利用奇异值分解求最小二乘刚性转置方法求解转换矩阵R 4*4，利用矩阵R 4*4，实现由物理机器人基坐标系O w-X wY wZ w到相机坐标系O c-X cY cZ c的变换，R 4*4即由世界坐标系到相机坐标系的转换矩阵为R。
最后，以该转换矩阵R设置虚拟世界中虚拟相机位置，并将虚拟机器人模型叠加在相机拍摄的图像上物理机器人的位置，完成增强现实注册，实现虚拟机器人模型的基坐标系与物理机器人的基坐标系的一致。
实施例三
参加图3，一种面向机器人增强现实示教的注册方法，该方法是基于实施例1所述的一种面向机器人增强现实示教的注册系统实现的，具体步骤如下：
将注册标识物安装在所述物理机器人上，并固定好相机；
在计算机上编写机器人点位间歇运动控制程序并发送给物理机器人控制器，所述机器人点位间歇运动控制程序至少控制所述物理机器人的执行机构进行至少3次以上的折线运动；
将编写完成的机器人点位间歇运动控制程序下载至所述物理机器人控制器，所述物理机器人控制器执行所述机器人点位间歇运动控制程序控制所述物理机器人动作；
相机拍摄图像，并发送至计算机中，所述计算机计算注册标识物在像素坐标系下的像素坐标，同时，计算机通过物理机器人控制器获取注册标识物在物理机器人基坐标系下的三维坐标；
转换计算单元读取注册标识物在图像中的像素坐标和注册标识物在物理机 器人基坐标系下的三维坐标，并计算相机坐标系与物理机器人基坐标系间的转换关系；
所述虚拟机器人生成单元根据所述计算相机坐标系与物理机器人基坐标系间的转换关系，生成与物理机器人基坐标系一致的虚拟机器人模型，并通过增强现实设备将虚拟机器人模型叠加在现实场景中，完成注册。
本实施例不需要在物理机器人以外的物理实体上提前安装AR注册卡，即可实现增强现实注册，避免了AR注册卡安装精度对增强现实注册精度的影响，可以高精度地实现虚拟机器人的基坐标系与物理机器人的基坐标系的一致，提高示教路径的精度，适合工业机器人远程增强现实示教和移动工业机器人增强现实示教。
实施例四
进一步的，所述注册标识物设置于所述物理机器人末端的执行机构上，可实时读取注册标识物在机器人坐标系下的坐标，为了方便检测计算方便，所述注册标识物为具有特点形状(如球体、立方体等)或颜色(如红色、黄色)的物体。
可替代的，所述注册标识物设置于所述物理机器人的关节上，可以实时读取机器人关节数据，通过机器人正运动学模型，求得该标识在机器人坐标系下的坐标，为了方便检测计算方便，所述注册标识物为具有特点形状或颜色的物体。
进一步的，所述相机拍摄图像，并发送至计算机中，所述计算机计算注册标识物在图像中的像素坐标，同时，计算机通过物理机器人控制器获取注册标 识物在物理机器人基坐标系下的三维坐标的步骤中：在间歇运动的每个间歇点，计算机控制相机进行拍摄，识别注册标识物并计算注册标识物在像素坐标系下的像素坐标Z i(u i,v i)，同时读取当前注册标识物在物理机器人基坐标系O w-X wY wZ w下的三维坐标P i(x wi,y wi,z wi)。
进一步的，所述转换计算单元读取注册标识物在图像中的像素坐标和注册标识物在物理机器人基坐标系下的三维坐标，并计算相机坐标系与物理机器人基坐标系间的转换关系的步骤具体为：
假设由世界坐标系到相机坐标系的转换矩阵为R，利用相机的成像模型可以得到如下关系式：
Figure PCTCN2020080786-appb-000003
其中A 3*4为相机的内参矩阵，Q i(x ci,y ci,z ci)为注册标识物在相机坐标系O c-X cY cZ c中的三维坐标。
根据所有特征点(4个及以上)的像素坐标Z i(u i,v i),和对应的三维坐标P i(x wi,y wi,z wi)，以下式为目标
Figure PCTCN2020080786-appb-000004
利用奇异值分解求最小二乘刚性转置方法求解转换矩阵R 4*4，利用矩阵R 4*4，实现由物理机器人基坐标系O w-X wY wZ w到相机坐标系O c-X cY cZ c的变换，R 4*4即由世界坐标系到相机坐标系的转换矩阵为R。
最后，以该转换矩阵R设置虚拟世界中虚拟相机位置，并将虚拟机器人模型叠加在相机拍摄的图像上物理机器人的位置，完成增强现实注册，实现虚拟 机器人模型的基坐标系与物理机器人的基坐标系的一致。
以上所述仅为本发明的实施例，并非因此限制本发明的专利范围，凡是利用本发明说明书及附图内容所作的等效结构或等效流程变换，或直接或间接运用在其他相关的技术领域，均同理包括在本发明的专利保护范围内。

Claims (10)

  1. 一种面向机器人增强现实示教的注册系统，其特征在于：
    包括物理机器人单元、注册单元、虚拟机器人生成单元以及计算机；
    所述物理机器人单元包括物理机器人、物理机器人控制器和机器人点位间歇运动控制程序；所述物理机器人自设置有物理机器人基坐标系；所述物理机器人控制器分别与所述物理机器人和计算机连接，所述物理机器人控制器用于控制物理机器人运动，并获取物理机器人的运动轨迹；所述机器人点位间歇运动控制程序安装于所述计算机中，由所述物理机器人控制器执行所述机器人间歇运动控制程序，控制所述物理机器人的执行机构进行至少三条及以上的折线运动；
    所述注册单元包括注册标识物、相机和转换计算单元；所述注册标识物设置在物理机器人本体上；所述相机固定安装在物理机器人以外的物理环境中，对准所述物理机器人的工作区域拍摄；所述相机连接所述计算机，所述转换计算单元设置于所述计算机中用于计算注册标识物在相机坐标系下的三维坐标与注册标识物在物理机器人基坐标系下的三维坐标之间的转换关系；
    所述虚拟机器人生成单元设置于所述计算机中，用于生成虚拟机器人模型。
  2. 根据权利要求1所述的一种面向机器人增强现实示教的注册系统，其特征在于：所述注册标识物设置于所述物理机器人末端的执行机构上，所述注册标识物为具有特点形状或颜色的物体。
  3. 根据权利要求1所述的一种面向机器人增强现实示教的注册系统，其特征在于：所述注册标识物设置于所述物理机器人的关节上，所述注册标识物为具有特点形状或颜色的物体。
  4. 根据权利要求2或3所述的一种面向机器人增强现实示教的注册系统， 其特征在于，所述计算机具体用于：在间歇运动的每个间歇点，所述计算机通过机器人控制器读取注册标识物在物理机器人基坐标系下的三维坐标；同时，通过相机获取物理工作环境的图像，通过计算机视觉算法识别注册标识物并计算注册标识物在像素坐标系下的像素坐标。
  5. 根据权利要求4所述的一种面向机器人增强现实示教的注册系统，其特征在于，所述转换计算单元具体用于：利用相机的成像模型及相机的内参矩阵，根据所述注册标识物在像素坐标系下的像素坐标以及注册标识物在物理机器人基坐标系下的三维坐标通过构建最小二乘问题，计算相机坐标系与物理机器人基坐标系间的转换关系。
  6. 一种面向机器人增强现实示教的注册方法，其特征在于，该方法是基于权利要求1所述一种面向机器人增强现实示教的注册系统实现的，具体步骤如下：
    将注册标识物安装在所述物理机器人上，并固定好相机；
    在计算机上编写机器人点位间歇运动控制程序并发送给物理机器人控制器，所述机器人点位间歇运动控制程序至少控制所述物理机器人的执行机构进行至少三次以上的折线运动；
    将编写完成的机器人点位间歇运动控制程序下载至所述物理机器人控制器，所述物理机器人控制器执行所述机器人点位间歇运动控制程序控制所述物理机器人动作；
    相机拍摄图像，并发送至计算机中，所述计算机计算注册标识物在图像中的像素坐标，同时，计算机通过物理机器人控制器获取注册标识物在物理机器人基坐标系下的三维坐标；
    转换计算单元读取注册标识物在图像中的像素坐标和注册标识物在物理机器人基坐标系下的三维坐标，并计算相机坐标系与物理机器人基坐标系间的转换关系；
    所述虚拟机器人生成单元根据所述计算相机坐标系与物理机器人基坐标系间的转换关系，生成与物理机器人基坐标系一致的虚拟机器人模型，并通过增强现实设备将虚拟机器人模型叠加在现实场景中，完成注册。
  7. 根据权利要求6所述的一种面向机器人增强现实示教的注册方法，其特征在于：所述注册标识物设置于所述物理机器人末端的执行机构上，所述注册标识物为具有特点形状或颜色的物体。
  8. 根据权利要求6所述的一种面向机器人增强现实示教的注册方法，其特征在于：所述注册标识物设置于所述物理机器人的关节上，所述注册标识物为具有特点形状或颜色的物体。
  9. 根据权利要求7或8所述的一种面向机器人增强现实示教的注册方法，其特征在于，所述相机拍摄图像，并发送至计算机中，所述计算机计算注册标识物在图像中的像素坐标，同时，计算机通过物理机器人控制器获取注册标识物在物理机器人基坐标系下的三维坐标的步骤中：在间歇运动的每个间歇点，计算机控制相机进行拍摄，同时读取当前注册标识物在物理机器人基坐标系下的三维坐标。
  10. 根据权利要求9所述的一种面向机器人增强现实示教的注册方法，其特征在于，所述转换计算单元读取注册标识物在图像中的像素坐标和注册标识物在物理机器人基坐标系下的三维坐标，并计算相机坐标系与物理机器人基坐标系间的转换关系的步骤具体为：
    所述转换计算单元利用相机的成像模型及相机的内参矩阵，根据所述注册标识物在像素坐标系下的像素坐标以及注册标识物在物理机器人基坐标系下的三维坐标通过构建最小二乘问题，计算相机坐标系与物理机器人基坐标系间的转换关系。
PCT/CN2020/080786 2020-03-24 2020-03-24 一种面向机器人增强现实示教的注册系统和方法 WO2021189224A1 (zh)

Priority Applications (3)

Application Number Priority Date Filing Date Title
US17/417,416 US11969904B2 (en) 2020-03-24 Registration system and method for robot-oriented augmented reality teaching system
CA3124850A CA3124850A1 (en) 2020-03-24 2020-03-24 Registration system and method for robot-oriented augmented reality teching system
PCT/CN2020/080786 WO2021189224A1 (zh) 2020-03-24 2020-03-24 一种面向机器人增强现实示教的注册系统和方法

Applications Claiming Priority (1)

Application Number Priority Date Filing Date Title
PCT/CN2020/080786 WO2021189224A1 (zh) 2020-03-24 2020-03-24 一种面向机器人增强现实示教的注册系统和方法

Publications (1)

Publication Number Publication Date
WO2021189224A1 true WO2021189224A1 (zh) 2021-09-30

Family

ID=77857658

Family Applications (1)

Application Number Title Priority Date Filing Date
PCT/CN2020/080786 WO2021189224A1 (zh) 2020-03-24 2020-03-24 一种面向机器人增强现实示教的注册系统和方法

Country Status (2)

Country Link
CA (1) CA3124850A1 (zh)
WO (1) WO2021189224A1 (zh)

Citations (5)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN108481323A (zh) * 2018-03-14 2018-09-04 清华大学天津高端装备研究院洛阳先进制造产业研发基地 基于增强现实的机器人运动轨迹自动编程系统及方法
CN209373826U (zh) * 2018-11-15 2019-09-10 塔普翊海（上海）智能科技有限公司 智能机器人仿真教学系统及便携箱
CN110238831A (zh) * 2019-07-23 2019-09-17 青岛理工大学 基于rgb-d图像及示教器的机器人示教系统及方法
CN110815189A (zh) * 2019-11-20 2020-02-21 福州大学 基于混合现实的机器人快速示教系统及方法
WO2020055909A1 (en) * 2018-09-10 2020-03-19 Fanuc America Corporation Zero teach for robotic continuous path

Patent Citations (5)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN108481323A (zh) * 2018-03-14 2018-09-04 清华大学天津高端装备研究院洛阳先进制造产业研发基地 基于增强现实的机器人运动轨迹自动编程系统及方法
WO2020055909A1 (en) * 2018-09-10 2020-03-19 Fanuc America Corporation Zero teach for robotic continuous path
CN209373826U (zh) * 2018-11-15 2019-09-10 塔普翊海（上海）智能科技有限公司 智能机器人仿真教学系统及便携箱
CN110238831A (zh) * 2019-07-23 2019-09-17 青岛理工大学 基于rgb-d图像及示教器的机器人示教系统及方法
CN110815189A (zh) * 2019-11-20 2020-02-21 福州大学 基于混合现实的机器人快速示教系统及方法

Also Published As

Publication number Publication date
CA3124850A1 (en) 2021-09-24
US20220324117A1 (en) 2022-10-13

Similar Documents

Publication Publication Date Title
AU2020201554B2 (en) System and method for robot teaching based on RGB-D images and teach pendant
CN109859275B (zh) 一种基于s-r-s结构的康复机械臂的单目视觉手眼标定方法
US11813749B2 (en) Robot teaching by human demonstration
Gong et al. An uncalibrated visual servo method based on projective homography
CN113706621B (zh) 基于带标记图像的标志点定位及姿态获取方法和系统
WO2023279874A1 (zh) 一种手术机器人导航定位系统及测量视角多目标优化方法
Cipolla et al. Visually guided grasping in unstructured environments
CN109048893A (zh) 一种基于单目rgb相机的机械臂定位方法
Gratal et al. Visual servoing on unknown objects
CN113103230A (zh) 一种基于处置机器人遥操作的人机交互系统及方法
CN111283664B (zh) 一种面向机器人增强现实示教的注册系统和方法
Lopez-Nicolas et al. Nonholonomic epipolar visual servoing
CN113172659A (zh) 基于等效中心点识别的柔性机器人臂形测量方法及系统
CN113172632A (zh) 一种基于图像的简化机器人视觉伺服控制方法
CN114224489B (zh) 用于手术机器人的轨迹跟踪系统及利用该系统的跟踪方法
Han et al. Grasping control method of manipulator based on binocular vision combining target detection and trajectory planning
CN112958960B (zh) 一种基于光学靶标的机器人手眼标定装置
WO2021189224A1 (zh) 一种面向机器人增强现实示教的注册系统和方法
CN110722547B (zh) 模型未知动态场景下移动机器人视觉镇定
US11969904B2 (en) Registration system and method for robot-oriented augmented reality teaching system
WO2021189223A1 (zh) 一种基于标识卡运动的机器人增强现实示教的注册系统及方法
Tsoy et al. Estimation of 4-DoF manipulator optimal configuration for autonomous camera calibration of a mobile robot using on-board templates
CN113240751A (zh) 一种机器人末端相机的标定方法
Zhang et al. High-precision pose estimation method of the 3C parts by combining 2D and 3D vision for robotic grasping in assembly applications
CN112258562A (zh) 一种基于图像特征的配准系统及配准方法

Legal Events

Date Code Title Description
ENP Entry into the national phase

Ref document number: 3124850

Country of ref document: CA

Kind code of ref document: A

121 Ep: the epo has been informed by wipo that ep was designated in this application

Ref document number: 20926893

Country of ref document: EP

Kind code of ref document: A1

NENP Non-entry into the national phase

Ref country code: DE

122 Ep: pct application non-entry in european phase

Ref document number: 20926893

Country of ref document: EP

Kind code of ref document: A1