WO2021103558A1 - 基于rgb-d数据融合的机器人视觉引导方法和装置 - Google Patents

基于rgb-d数据融合的机器人视觉引导方法和装置 Download PDF

Info

Publication number
WO2021103558A1
WO2021103558A1 PCT/CN2020/101335 CN2020101335W WO2021103558A1 WO 2021103558 A1 WO2021103558 A1 WO 2021103558A1 CN 2020101335 W CN2020101335 W CN 2020101335W WO 2021103558 A1 WO2021103558 A1 WO 2021103558A1
Authority
WO
WIPO (PCT)
Prior art keywords
rgb
data
point
registration
point cloud
Prior art date
Application number
PCT/CN2020/101335
Other languages
English (en)
French (fr)
Inventor
刁世普
郑振兴
秦磊
Original Assignee
广东技术师范大学
广东汇博机器人技术有限公司
Priority date (The priority date is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the date listed.)
Filing date
Publication date
Application filed by 广东技术师范大学, 广东汇博机器人技术有限公司 filed Critical 广东技术师范大学
Publication of WO2021103558A1 publication Critical patent/WO2021103558A1/zh

Links

Images

Classifications

    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
    • G06T7/00—Image analysis
    • G06T7/30—Determination of transform parameters for the alignment of images, i.e. image registration
    • G06T7/33—Determination of transform parameters for the alignment of images, i.e. image registration using feature-based methods
    • G—PHYSICS
    • G05—CONTROLLING; REGULATING
    • G05B—CONTROL OR REGULATING SYSTEMS IN GENERAL; FUNCTIONAL ELEMENTS OF SUCH SYSTEMS; MONITORING OR TESTING ARRANGEMENTS FOR SUCH SYSTEMS OR ELEMENTS
    • G05B19/00—Programme-control systems
    • G05B19/02—Programme-control systems electric
    • G05B19/18—Numerical control [NC], i.e. automatically operating machines, in particular machine tools, e.g. in a manufacturing environment, so as to execute positioning, movement or co-ordinated operations by means of programme data in numerical form
    • G05B19/19—Numerical control [NC], i.e. automatically operating machines, in particular machine tools, e.g. in a manufacturing environment, so as to execute positioning, movement or co-ordinated operations by means of programme data in numerical form characterised by positioning or contouring control systems, e.g. to control position from one programmed point to another or to control movement along a programmed continuous path
    • G05B19/21—Numerical control [NC], i.e. automatically operating machines, in particular machine tools, e.g. in a manufacturing environment, so as to execute positioning, movement or co-ordinated operations by means of programme data in numerical form characterised by positioning or contouring control systems, e.g. to control position from one programmed point to another or to control movement along a programmed continuous path using an incremental digital measuring device
    • G05B19/25—Numerical control [NC], i.e. automatically operating machines, in particular machine tools, e.g. in a manufacturing environment, so as to execute positioning, movement or co-ordinated operations by means of programme data in numerical form characterised by positioning or contouring control systems, e.g. to control position from one programmed point to another or to control movement along a programmed continuous path using an incremental digital measuring device for continuous-path control
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
    • G06T7/00—Image analysis
    • G06T7/10—Segmentation; Edge detection
    • G06T7/11—Region-based segmentation
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
    • G06T7/00—Image analysis
    • G06T7/50—Depth or shape recovery
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
    • G06T7/00—Image analysis
    • G06T7/70—Determining position or orientation of objects or cameras
    • G06T7/73—Determining position or orientation of objects or cameras using feature-based methods
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
    • G06T2207/00—Indexing scheme for image analysis or image enhancement
    • G06T2207/10—Image acquisition modality
    • G06T2207/10028—Range image; Depth image; 3D point clouds
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
    • G06T2207/00—Indexing scheme for image analysis or image enhancement
    • G06T2207/20—Special algorithmic details
    • G06T2207/20081—Training; Learning
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
    • G06T2207/00—Indexing scheme for image analysis or image enhancement
    • G06T2207/20—Special algorithmic details
    • G06T2207/20084—Artificial neural networks [ANN]
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
    • G06T2207/00—Indexing scheme for image analysis or image enhancement
    • G06T2207/30—Subject of image; Context of image processing
    • G06T2207/30108—Industrial image inspection

Definitions

  • the invention relates to the field of robot vision, in particular to a method and device for robot vision guidance based on RGB-D data fusion.
  • intelligent automation equipment As a powerful tool for manufacturing automation equipment (robot system), it must be high-speed and intelligent.
  • An important means of intelligent automation equipment is to equip the machine with "eyes" and a "brain” that can cooperate with this eye.
  • This "eye” can be a monocular camera, a binocular camera, a multi-eye camera, a three-dimensional scanner, or an RGB-D (RGB+Depth) sensor.
  • the core work content of intelligent automation equipment includes: analyzing the image data acquired by this "eye” (such as image recognition), and then guiding the robot system to complete specific processing or assembly operations based on the analysis results. Therefore, image data analysis based on two-dimensional images, which is widely used at present, is a key basic core technology.
  • the existing point cloud analysis methods have the following shortcomings: the method of point cloud segmentation of processing targets is very time-consuming and cannot meet the needs of current high-speed production; the accuracy of the 3D processing target positioning method based purely on 3D point clouds and deep learning The resolution is too poor and the resolution accuracy is not high, which does not meet the current needs of high-precision processing.
  • the existing point cloud analysis methods are not universal and are only suitable for specific automated processing systems.
  • the present invention discloses a method and device for a robot to determine a target orientation based on RGB-D data, thereby providing a vision guidance method and device that can meet the current processing target detection accuracy requirements and is suitable for processing target workpieces .
  • the main purpose of the present invention is to provide a robot vision guidance method and device based on RGB-D data fusion, which aims to solve the problem that the existing method of point cloud segmentation processing target is very time-consuming, cannot meet the current needs of high-speed production, and is pure
  • the accuracy of the 3D processing target positioning method based on 3D point cloud and deep learning is too poor, the resolution accuracy is not high, it does not meet the current needs of high-precision processing, and the universality of the existing point cloud analysis methods is not high.
  • the purpose of the present invention is to provide a target orientation recognition method and device that can meet the current machining target detection accuracy requirements, and is suitable for machining target workpieces, which can meet the current machining target detection and processing needs, and the accuracy meets medium Precision is required, and it is suitable for the visual guidance method and device of the workpiece of the processing target.
  • the robot vision guidance method based on RGB-D data fusion includes:
  • Step 1 Obtain the RGB two-dimensional image and depth data containing the target to be processed by the RGB-D composite sensor, and combine the RGB two-dimensional image with the depth data according to the preset registration parameters of the RGB-D composite sensor registering process, thereby generating the registration corresponding to the registration of the two-dimensional image of the I RGB RGB two-dimensional image corresponding to the depth data and depth data I D;
  • Step 2 Use a pre-trained image segmentation model to segment the region S RGB corresponding to the target to be processed from the registered two-dimensional image I RGB as input, and use the pre-trained image segmentation model to segment the region S RGB corresponding to the target to be processed according to the region S RGB .
  • the KX i is a salient feature point
  • the i is the serial number corresponding to the salient feature point KX i
  • the value range of the i is [1, m]
  • the m is the total number of the salient feature points KX i
  • the S 3D-j is a local point cloud
  • the j is the local point cloud S 3D -the serial number of j
  • the value range of j is [1, n]
  • the n is the total number of the local point cloud S 3D-j;
  • Step 4 setting the order j from 1 to n, by using a local search algorithm to search and extract a plane perpendicular to the local point corresponding to the local cloud point S 3D-j bounding box cloud S 3D-j
  • the j is sequentially set from 1 to n, the local point cloud S 3D-j is fitted to the curved surface SS 3D-j by a surface fitting algorithm, and then the local point cloud S 3D-j is fitted by using the local
  • the step 2 the registration of the two-dimensional image I RGB and the corresponding depth data registration I D are combined to generate a fused data I RGB-D, by using pre-trained image segmentation model from fusion as the input data I RGB-D in the divided region S RGB-D corresponding to the object to be processed, according to the registration area S RGB-D I D extracted depth data to be processed from the feature The 3D point cloud data S 3D of the target.
  • the training samples required for the pre-training of the image segmentation model in step 2 are collected by using the RGB-D composite sensor to collect a registered two-dimensional image I RGB containing the same type of target to be processed the registration and the corresponding depth data I D generated.
  • the conversion matrix BET in the step 5 is generated by pre-calibrated Denavit-Hartenberg (D-H) parameters.
  • D-H Denavit-Hartenberg
  • the present invention further provides a robot vision guidance device based on RGB-D data fusion, including:
  • the processing target data acquisition module is used to acquire the RGB two-dimensional image and depth data containing the target to be processed through the RGB-D composite sensor, and to convert the RGB two-dimensional image according to the preset registration parameters of the RGB-D composite sensor registration process performed with the depth data, thereby generating the registration corresponding to the two-dimensional image registration RGB two-dimensional image corresponding to the I and the depth data RGB depth data I D;
  • the processing target recognition module is configured to use a pre-trained image segmentation model to segment the region S RGB corresponding to the target to be processed from the registered two-dimensional image I RGB as the input, according to the region S RGB depth data from the registration I D extracting 3D point cloud data object to be processed S 3D;
  • the value range of the i is [1, m]
  • the m is the total number of the salient feature points KX i
  • the S 3D-j is the local point cloud
  • the j is the local
  • the sequence number of the point cloud S 3D-j the value range of j is [1, n]
  • the n is the total number of the local point cloud S 3D-j;
  • the processing path point acquisition module is used to sequentially set the j from 1 to n, and search and extract the plane perpendicular to the bounding box of the local point cloud S 3D-j by using a local search algorithm and the corresponding local point
  • the target identification processing module for the registration and I RGB two-dimensional image corresponding to the registration depth data I D are combined to generate a fused data I RGB-D, by using pre-trained image segmentation model as an input from the fusion among the data I RGB-D segmented region S RGB-D corresponding to the object to be processed, according to the area S RGB-D depth data from the registration extract I D
  • the 3D point cloud data S 3D of the target to be processed are combined to generate a fused data I RGB-D, by using pre-trained image segmentation model as an input from the fusion among the data I RGB-D segmented region S RGB-D corresponding to the object to be processed, according to the area S RGB-D depth data from the registration extract I D
  • the 3D point cloud data S 3D of the target to be processed are combined to generate a fused data I RGB-D, by using pre-trained image segmentation model as an input from the fusion among the data I RGB-D segmented region S RGB-D corresponding to the object to be processed
  • the training samples required for the pre-training of the image segmentation model in the processing target recognition module are obtained by using the RGB-D composite sensor to collect registration two containing the same type of the target to be processed. I RGB-dimensional image corresponding to the registration and depth data I D generated.
  • the conversion matrix BET in the processing guide point conversion module is generated by pre-calibrated Denavit-Hartenberg (D-H) parameters.
  • D-H Denavit-Hartenberg
  • the present invention can provide the processing target detection accuracy requirements that can meet the current processing needs, and greatly reduce the amount of calculation, reduce the complexity of the calculation, accelerate the processing speed, reduce the calculation time, and meet the requirements of real-time processing. , And reduce the performance requirements of software and hardware, can save costs, reduce the difficulty of development, and meet the requirements for high-speed mass production mode.
  • FIG. 1 is a schematic flowchart of a first embodiment of a robot vision guidance method based on RGB-D data fusion according to the present invention
  • FIG. 2 is a schematic diagram of functional modules of the first embodiment of a robot vision guidance device based on RGB-D data fusion according to the present invention
  • FIG. 3 is a schematic diagram of an RGB-D composite sensor implementing the present invention.
  • FIG. 1 is a schematic flowchart of a first embodiment of a robot vision guidance method based on RGB-D data fusion according to the present invention. As shown in the embodiment shown in Fig. 1, the robot vision guidance method based on RGB-D data fusion includes the following steps:
  • the RGB two-dimensional image and depth data containing the target to be processed are acquired through the RGB-D composite sensor, and the RGB two-dimensional image and the depth data are matched according to the preset registration parameters of the RGB-D composite sensor. the registration processing, thereby generating a two-dimensional image corresponding to the I RGB registration of the RGB two-dimensional image data and depth corresponding to the depth data registration I D.
  • the RGB-D composite sensor is set at the top of the robotic arm D40, the RGB camera D20 is in the middle of the RGB-D composite vision sensor, and the color image data will be compressed before being transmitted to the computer. Ensure the speed of RGB data analysis.
  • the sensors D10 and D30 on the left and right sides of the RGB-D composite vision sensor are respectively responsible for emitting and receiving infrared rays: firstly, the infrared ray is emitted to the target O10 to be processed through the infrared ray emitter D10 on the left.
  • the light spots formed by reflection at any two different positions in the space are different, forming a three-dimensional "light code" for the environment; then the infrared receiver D30 on the right is used to collect the infrared image in the field of view; finally, the use of RGB-D
  • the parameters of the composite vision sensor perform a series of complex calculations on this infrared image, and then the depth data in the field of view can be obtained.
  • a pre-trained image segmentation model is used to segment the region S RGB corresponding to the target to be processed from the registered two-dimensional image I RGB as input, and the region S RGB is obtained from the registration depth according to the region S RGB. extracting data I D 3D point cloud data of the object to be processed S 3D.
  • the image segmentation model based on the deep learning framework realizes the neural network model of semantic segmentation of the target to be processed, and has the characteristics of high accuracy, fast processing speed, and real-time processing.
  • the KX i is a salient feature point
  • the i is the serial number corresponding to the salient feature point KX i
  • the i The value range of is [1, m]
  • the m is the total number of the salient feature points KX i
  • the S 3D-j is a local point cloud
  • the j is the local point cloud S 3D-j
  • the value range of j is [1,n]
  • the n is the total number of the local point cloud S 3D-j.
  • the KX i is a coordinate vector corresponding to a salient feature point
  • the S 3D-j includes a set of all points of the local point cloud.
  • the processing path point SX j is the position coordinate information in the coordinate system corresponding to the RGB-D composite vision sensor, so it needs to be converted to the position coordinate information in the corresponding work coordinate system.
  • the use of the above processing steps can reduce the amount of calculation, reduce the complexity of the calculation, accelerate the processing speed, reduce the calculation time, meet the requirements of real-time processing, and reduce the performance requirements of software and hardware, which can save costs and reduce The difficulty of development meets the requirements for high-speed mass production mode.
  • step S20 the registration of the two-dimensional image I RGB and the corresponding depth data registration I D are combined to generate a fused data I RGB-D, by using pre-trained from a model image segmentation fusion among the input data I RGB-D segmented region S RGB-D corresponding to the object to be processed, according to the registration area S RGB-D I D depth data from said feature extracting target to be processed
  • the 3D point cloud data S 3D The 3D point cloud data S 3D .
  • Using the fusion data I RGB-D can effectively improve the accuracy and accuracy of segmenting the region S RGB-D corresponding to the target to be processed, and greatly enhance the robustness and stability of the segmentation.
  • the training samples required for the pre-training of the image segmentation model in the step S20 are obtained by using the RGB-D composite sensor to collect the registered two-dimensional image I RGB and the same type of the target to be processed. registration of the corresponding depth data I D generated.
  • a large number of the registered two-dimensional images I RGB and the corresponding registration depth data I D of the target to be processed can be obtained, and then the registration is performed by labeling. It can be a training sample; then the image segmentation model based on the deep learning framework is trained, and the relevant parameters in the training process are fine-tuned until the accuracy of the model reaches the desired value.
  • a large amount of training sample data can be obtained very efficiently, thereby ensuring the accuracy and robustness requirements of the image segmentation model based on the deep learning framework.
  • the conversion matrix BET in the step S50 is generated by Denavit-Hartenberg (D-H) parameters that are calibrated in advance.
  • D-H Denavit-Hartenberg
  • the purpose of calibrating Denavit-Hartenberg (DH) parameters by using a laser tracker is to improve the overall accuracy of the robot vision guidance algorithm, and to ensure that the processing path point SX j is converted to the processing guidance point BX j accurately, and the method It has the characteristics of fast processing speed, maturity and reliability, and easy engineering realization.
  • the robot vision guidance method based on RGB-D data fusion in the first embodiment of the robot vision guidance method based on RGB-D data fusion of the present invention can be implemented by the first implementation of the robot vision guidance device based on RGB-D data fusion of the present invention
  • the robot vision guidance device based on RGB-D data fusion provided in the example is realized.
  • the device 1 includes:
  • the processing target data acquisition module 10 is configured to acquire the RGB two-dimensional image and depth data containing the target to be processed through the RGB-D composite sensor, and convert the RGB two-dimensional image according to the preset registration parameters of the RGB-D composite sensor. image and the depth data registering process, thereby generating a depth corresponding to the registration data registered two-dimensional images I D the I the RGB RGB two-dimensional image corresponding to the depth data.
  • the RGB-D composite sensor is set at the top of the robotic arm D40, and the RGB camera D20 is in the middle of the RGB-D composite vision sensor.
  • the color image data will be compressed before being transmitted to the computer. Ensure the speed of RGB data analysis.
  • the sensors D10 and D30 on the left and right sides of the RGB-D composite vision sensor are respectively responsible for emitting and receiving infrared rays: firstly, the infrared ray is emitted to the target O10 to be processed through the infrared ray emitter D10 on the left.
  • the light spots formed by reflection at any two different positions in the space are different, forming a three-dimensional "light code" for the environment; then the infrared receiver D30 on the right is used to collect the infrared image in the field of view; finally, the use of RGB-D
  • the parameters of the composite vision sensor perform a series of complex calculations on this infrared image, and then the depth data in the field of view can be obtained.
  • the processing target recognition module 20 is configured to use a pre-trained image segmentation model to segment the region S RGB corresponding to the target to be processed from the registered two-dimensional image I RGB as input, according to the region S extracting RGB 3D point cloud data of the object to be processed S 3D depth data from the registration I D.
  • the image segmentation model based on the deep learning framework realizes the neural network model of semantic segmentation of the target to be processed, and has the characteristics of high accuracy, fast processing speed, and real-time processing.
  • m is the total number of the salient features of the KX-point i of the S 3D-j is a partial cloud point
  • the j is the The serial number of the local point cloud S 3D-j
  • the value range of j is [1, n]
  • the n is the total number of the local point cloud S 3D-j.
  • the KX i is a coordinate vector corresponding to a salient feature point
  • the S 3D-j includes a set of all points of the local point cloud.
  • the processing path point acquisition module 40 is configured to sequentially set the j from 1 to n, and use a local search algorithm to search for and extract the plane of the bounding box perpendicular to the local point cloud S 3D-j and the plane corresponding to the local point cloud S 3D-j
  • the processing guide point conversion module 50 is configured to sequentially set the j from 1 to n, and use the conversion matrix BET to convert the processing path point SX j to the processing guide point BX j , thereby converting the processing path point
  • the processing path point SX j is the position coordinate information in the coordinate system corresponding to the RGB-D composite vision sensor, so it needs to be converted to the position coordinate information in the corresponding work coordinate system.
  • the use of the above-mentioned modules can reduce the amount of calculation, reduce the complexity of the calculation, speed up the processing speed, reduce the calculation time, meet the requirements of real-time processing, and reduce the requirements for the performance of software and hardware, which can save costs and reduce development.
  • the difficulty meets the requirements of high-speed mass production mode.
  • the processing target identification module 20, for the registration I RGB two-dimensional image corresponding to the registration and depth data I D are combined to generate a fused data I RGB-D, by using pre-trained image segmentation model as an input from the fusion among the data I RGB-D segmented region S RGB-D corresponding to the object to be processed, according to the area S RGB-D depth data from the registration extract I D
  • the 3D point cloud data S 3D of the target to be processed are combined to generate a fused data I RGB-D, by using pre-trained image segmentation model as an input from the fusion among the data I RGB-D segmented region S RGB-D corresponding to the object to be processed, according to the area S RGB-D depth data from the registration extract I D.
  • Using the fusion data I RGB-D can effectively improve the accuracy and accuracy of segmenting the region S RGB-D corresponding to the target to be processed, and greatly enhance the robustness and stability of the segmentation.
  • the training samples required for the pre-training of the image segmentation model of the processing target recognition module 20 are acquired by using the RGB-D composite sensor to collect a registered two-dimensional image containing the same type of target to be processed I RGB and depth corresponding to the registration data generated by I D.
  • a large number of the registered two-dimensional images I RGB and the corresponding registration depth data I D of the target to be processed can be obtained, and then the registration is performed by labeling. It can be a training sample; then the image segmentation model based on the deep learning framework is trained, and the relevant parameters in the training process are fine-tuned until the accuracy of the model reaches the desired value.
  • a large amount of training sample data can be obtained very efficiently, thereby ensuring the accuracy and robustness requirements of the image segmentation model based on the deep learning framework.
  • the conversion matrix BET of the processing guide point conversion module 50 is generated by Denavit-Hartenberg (D-H) parameters that are calibrated in advance.
  • the purpose of calibrating Denavit-Hartenberg (DH) parameters by using a laser tracker is to improve the overall accuracy of the robot vision guidance algorithm, and to ensure that the processing path point SX j is converted to the processing guidance point BX j accurately, and the method It has the characteristics of fast processing speed, maturity and reliability, and easy engineering realization.
  • module units or steps of the present invention can be implemented by a general computing device. Alternatively, they can be implemented by program codes executable by the computing device. They are stored in a storage device to be executed by a computing device, and in some cases, the steps shown or described can be executed in a different order than here, or they can be made into individual integrated circuit modules, or the Multiple modules or steps in them are made into a single integrated circuit module to achieve. In this way, the present invention is not limited to any specific combination of hardware and software.
  • the technical solution of the present invention essentially or the part that contributes to the existing technology can be embodied in the form of a software product, and the computer software product is stored in a storage medium (such as ROM/RAM, magnetic disk, The optical disc) includes several instructions to enable a terminal device (which can be a mobile phone, a computer, a server, an air conditioner, or a network device, etc.) to execute the method described in each embodiment of the present invention.
  • a terminal device which can be a mobile phone, a computer, a server, an air conditioner, or a network device, etc.

Abstract

本发明公开了一种基于RGB-D数据融合的机器人视觉引导方法和装置，基于RGB-D复合传感器，从加工目标数据采集开始，依次经过加工目标识别，加工目标分割，加工路径点获取，加工引导点转换步骤，最终获得加工引导点序列，从而减少了计算时间，满足实时处理的要求，并且降低了对软硬件的性能的要求，可以节约成本，降低开发的难度，符合对高速化大规模生产模式的要求。

Description

基于RGB-D数据融合的机器人视觉引导方法和装置 技术领域
本发明涉及机器人视觉领域，特别涉及一种基于RGB-D数据融合的机器人视觉引导方法和装置。
背景技术
作为制造强国利器的自动化装备(机器人系统)必须要向高速化，智能化方向迈进。自动化装备的智能化的一个重要手段是给机器装上“眼睛”和能够与这颗眼睛配合的“大脑”。这只“眼睛”可以是单目相机，双目相机，多目相机，三维扫描仪，也可以是RGB-D(RGB+Depth)传感器。而自动化装备智能化的核心工作内容包括了：通过对这只“眼睛”所获取图像数据进行分析(例如图像识别)，再根据分析结果来引导机器人系统完成特定的加工或者装配操作。因此目前所广泛采用的基于二维图像的图像数据分析是一项关键的基础核心技术。但基于二维图像数据分析的方法很容易受到光线等外部环境因素的干扰，从而造成识别准确率较低，鲁棒性较差，并且精度也达不到要求。为了应对当前对图像数据分析的高速化和高精度的需求，能够在获取传统二维图像数据的同时也能获取不容易受到光线等外部环境因素干扰的对应二维图像数据的深度信息的RGB-D传感器得到了推广应用。通过对RGB-D传感器所获取的RGB图像数据和深度数据进行处理可以获得对应目标对象的3D数据(点云数据)。由于现有的点云分析方法存在如下所述的缺点：点云分割加工目标的方法非常耗时，不能满足当前高速生产的需要；纯粹基于3D点云和深度学习的3D加工目标定位方法的准确度太差，分辨率精度不高，不符合当前高精度加工的需要。综上所述，现有的点云分析方法的普适性不高， 仅仅适用于特定的自动化加工系统。此外，由于目前还没有面向特定加工或者装配操作任务的基于RGB-D数据的关键信息引导机器人系统的研究应用。因此，本发明公开了一种基于RGB-D数据的机器人确定目标方位的方法和装置，从而提供能够满足当前加工所需要的加工目标检测精度需要，并适用于加工目标工件的视觉引导方法和装置。
发明内容
本发明的主要目的在于提供一种基于RGB-D数据融合的机器人视觉引导方法和装置，旨在解决现有的点云分割加工目标的方法非常耗时，不能满足当前高速生产的需要，并且纯粹基于3D点云和深度学习的3D加工目标定位方法的准确度太差，分辨率精度不高，不符合当前高精度加工的需要，以及现有的点云分析方法的普适性不高，仅仅适用于特定的自动化加工系统的这些技术问题。针对这些缺点，本发明的目的是提供能够满足当前加工所需要的加工目标检测精度需要，并适用于加工目标工件的目标方位识别方法和装置能够满足当前加工目标检测和加工需要的，精度满足中等精度需要，适用于加工目标的工件的视觉引导方法和装置。
为解决上述问题，本发明提供的基于RGB-D数据融合的机器人视觉引导方法，包括：
步骤1、通过RGB-D复合传感器获取包含待加工目标的RGB二维图像和深度数据，依据所述RGB-D复合传感器的预设的配准参数将所述RGB二维图像与所述深度数据进行配准处理，从而生成对应所述RGB二维图像的配准二维图像I RGB和对应所述深度数据的配准深度数据I D；
步骤2、使用通过预先训练好的图像分割模型从作为输入的所述配准二维图像I RGB之中分割出对应所述待加工目标的区域S RGB，依据所述区域S RGB从所述配准深度数据I D提取所述待加工目标的3D点云数据S 3D；
步骤3、从所述3D点云数据S 3D提取显著特征点序列{KX i} i＝1->m，以所 述显著特征点序列{KX i} i＝1->m为分界点将所述3D点云数据S 3D分割为局部点云集{S 3D-j} j＝1->n，所述KX i为显著特征点，所述i为对应所述显著特征点KX i的序号，所述i的取值范围为[1,m],所述m为所述显著特征点KX i的总数目，所述S 3D-j为局部点云，所述j为所述局部点云S 3D-j的序号，所述j的取值范围为[1,n]，所述n为所述局部点云S 3D-j的总数目；
步骤4、将所述j依次设置为从1至n，通过使用局部搜索算法搜索并提取垂直于所述局部点云S 3D-j的包围盒的平面与对应所述局部点云S 3D-j的交界处的加工路径点SX j，从而获得所述加工路径点序列{SX j} j＝1->n；
步骤5、将所述j依次设置为从1至n，使用包含通过转换矩阵BET将所述加工路径点SX j转换为加工引导点BX j，从而将所述加工路径点序列{SX j} j＝1->n转换为加工引导点序列{BX j} j＝1->n。
优选地，所述步骤4、将所述j依次设置为从1至n，通过曲面拟合算法将所述局部点云S 3D-j拟合为曲面SS 3D-j，再通过使用所述局部搜索算法搜索并提取垂直于所述局部点云S 3D-j的所述包围盒的所述平面与对应所述曲面SS 3D-j的交界处的所述加工路径点SX j，从而获得所述加工路径点序列{SX j} j＝1->n。
优选地，所述步骤2、将所述配准二维图像I RGB和对应的所述配准深度数据I D进行合并从而生成融合数据I RGB-D，使用通过预先训练好的图像分割模型从作为输入的所述融合数据I RGB-D之中分割出对应所述待加工目标的区域S RGB-D，依据所述区域S RGB-D从所述配准深度数据I D提取所述待加工目标的3D点云数据S 3D。
优选地，所述步骤2的所述图像分割模型的所述预先训练所需的训练样本是通过使用所述RGB-D复合传感器采集包含同类的所述待加工目标的配准二维图像I RGB和对应的所述配准深度数据I D所生成。
优选地，所述步骤5的所述转换矩阵BET是由通过预先校准的Denavit-Hartenberg(D-H)参数所生成。
本发明进一步提供基于RGB-D数据融合的机器人视觉引导装置，包括：
加工目标数据采集模块、用于通过RGB-D复合传感器获取包含待加工目标的RGB二维图像和深度数据，依据所述RGB-D复合传感器的预设的配准参数将所述RGB二维图像与所述深度数据进行配准处理，从而生成对应所述RGB二维图像的配准二维图像I RGB和对应所述深度数据的配准深度数据I D；
加工目标识别模块、用于使用通过预先训练好的图像分割模型从作为输入的所述配准二维图像I RGB之中分割出对应所述待加工目标的区域S RGB，依据所述区域S RGB从所述配准深度数据I D提取所述待加工目标的3D点云数据S 3D；
加工目标分割模块、用于从所述3D点云数据S 3D提取显著特征点序列{KX i} i＝1->m，以所述显著特征点序列{KX i} i＝1->m为分界点将所述3D点云数据S 3D分割为局部点云集{S 3D-j} j＝1->n，所述KX i为显著特征点，所述i为对应所述显著特征点KX i的序号，所述i的取值范围为[1,m],所述m为所述显著特征点KX i的总数目，所述S 3D-j为局部点云，所述j为所述局部点云S 3D-j的序号，所述j的取值范围为[1,n]，所述n为所述局部点云S 3D-j的总数目；
加工路径点获取模块、用于将所述j依次设置为从1至n，通过使用局部搜索算法搜索并提取垂直于所述局部点云S 3D-j的包围盒的平面与对应所述局部点云S 3D-j的交界处的加工路径点SX j，从而获得所述加工路径点序列{SX j} j＝1->n；
加工引导点转换模块、用于将所述j依次设置为从1至n，使用包含通过转换矩阵BET将所述加工路径点SX j转换为加工引导点BX j，从而将所述加工路径点序列{SX j} j＝1->n转换为加工引导点序列{BX j} j＝1->n。
优选地，所述加工路径点获取模块、用于将所述j依次设置为从1至n，通过曲面拟合算法将所述局部点云S 3D-j拟合为曲面SS 3D-j，再通过使用所述局部搜索算法搜索并提取垂直于所述局部点云S 3D-j的所述包围盒的所述平面与对应所述曲面SS 3D-j的交界处的所述加工路径点SX j，从而获得所述加工路径 点序列{SX j} j＝1->n。
优选地，所述加工目标识别模块、用于将所述配准二维图像I RGB和对应的所述配准深度数据I D进行合并从而生成融合数据I RGB-D，使用通过预先训练好的图像分割模型从作为输入的所述融合数据I RGB-D之中分割出对应所述待加工目标的区域S RGB-D，依据所述区域S RGB-D从所述配准深度数据I D提取所述待加工目标的3D点云数据S 3D。
优选地，所述加工目标识别模块之中的所述图像分割模型的所述预先训练所需的训练样本是通过使用所述RGB-D复合传感器采集包含同类的所述待加工目标的配准二维图像I RGB和对应的所述配准深度数据I D所生成。
优选地，所述加工引导点转换模块之中的所述转换矩阵BET是由通过预先校准的Denavit-Hartenberg(D-H)参数所生成。
本发明通过上述技术方案从而可以提供能够满足当前加工所需要的加工目标检测精度需要，并大大减少了计算量，降低计算的复杂度，加快了处理速度，减少了计算时间，满足实时处理的要求，并且降低了对软硬件的性能的要求，可以节约成本，降低了开发的难度，符合对高速化大规模生产模式的要求。
附图说明
图1为本发明基于RGB-D数据融合的机器人视觉引导方法第一实施例的流程示意图；
图2为本发明基于RGB-D数据融合的机器人视觉引导装置第一实施例的功能模块示意图；
图3为实现本发明的RGB-D复合传感器的示意图。
本发明目的的实现、功能特点及优点将结合实施例，参照附图做进一步 说明。
具体实施方式
应当理解，此处所描述的具体实施例仅仅用以解释本发明，并不用于限定本发明。
现在将参考附图描述实现本发明各个实施例。在后续的描述中，使用用于表示元件的诸如“模块”、“部件”或“单元”的后缀仅为了有利于本发明的说明，其本身并没有特定的意义。因此，"模块"与"部件"可以混合地使用。
参照图1，图1为本发明基于RGB-D数据融合的机器人视觉引导方法的第一实施例的流程示意图。如图1所示的实施例，所述基于RGB-D数据融合的机器人视觉引导方法包括如下步骤：
S10、加工目标数据采集。
即通过RGB-D复合传感器获取包含待加工目标的RGB二维图像和深度数据，依据所述RGB-D复合传感器的预设的配准参数将所述RGB二维图像与所述深度数据进行配准处理，从而生成对应所述RGB二维图像的配准二维图像I RGB和对应所述深度数据的配准深度数据I D。
参照图3，所述RGB-D复合传感器被设置在机械臂D40的顶端，RGB摄像头D20在所述RGB-D复合视觉传感器的中间位置，彩色图像数据在传递到计算机之前会在进行压缩，以保证对RGB数据分析时的速度。所述RGB-D复合视觉传感器左右两侧的传感器D10和D30分别负责发射和接收红外线：首先通过左侧的红外线发射器D10向待加工目标O10发射红外线，这束红外线由于具有高度随机性，其在空间中任意两个不同位置所反射形成的光斑都不相同，对环境形成立体的“光编码”；再通过右侧的红外线接收器D30来采集视野中的红外线图像；最终，利用RGB-D复合视觉传感器的参数对这副红外图像进行一系列复杂的计算，就可以得到视野中的深度数据。
S20、加工目标识别。
即使用通过预先训练好的图像分割模型从作为输入的所述配准二维图像I RGB之中分割出对应所述待加工目标的区域S RGB，依据所述区域S RGB从所述配准深度数据I D提取所述待加工目标的3D点云数据S 3D。
基于深度学习框架的所述图像分割模型实现了对待加工目标的语义分割的神经网络模型，具有精度高，处理速度快，可以实时处理的特点。
S30、加工目标分割。
即从所述3D点云数据S 3D提取显著特征点序列{KX i} i＝1->m，以所述显著特征点序列{KX i} i＝1->m为分界点将所述3D点云数据S 3D分割为局部点云集{S 3D-j} j＝1->n，所述KX i为显著特征点，所述i为对应所述显著特征点KX i的序号，所述i的取值范围为[1,m],所述m为所述显著特征点KX i的总数目，所述S 3D-j为局部点云，所述j为所述局部点云S 3D-j的序号，所述j的取值范围为[1,n]，所述n为所述局部点云S 3D-j的总数目。
所述KX i是对应显著特征点的坐标向量，所述S 3D-j包含所述局部点云的全部点的集合。
S40、加工路径点获取。
即将所述j依次设置为从1至n，通过使用局部搜索算法搜索并提取垂直于所述局部点云S 3D-j的包围盒的平面与对应所述局部点云S 3D-j的交界处的加工路径点SX j，从而获得所述加工路径点序列{SX j} j＝1->n。
所述步骤S30将所述待加工目标的所述3D点云数据S 3D以显著特征点序列{KX i} i＝1->m为分界点将其分割为局部点云集{S 3D-j} j＝1->n，在步骤S40可以简单有效的获取对应的加工路径点序列{SX j} j＝1->n。特别是对非常复杂的所述3D点云数据S 3D，更加利于对加工路径点序列{SX j} j＝1->n的提取，并且大大增强稳定性和鲁棒性。
S50、加工引导点转换。
将所述j依次设置为从1至n，使用包含通过转换矩阵BET将所述加工路径 点SX j转换为加工引导点BX j，从而将所述加工路径点序列{SX j} j＝1->n转换为加工引导点序列{BX j} j＝1->n。
加工路径点SX j为对应RGB-D复合视觉传感器的坐标系下的位置坐标信息，因此需要转换至对应作业坐标系下的位置坐标信息。最后将所述的加工引导点序列{BX j} j＝1->n发送给机器人执行相应的作业。通过对加工引导点序列{BX j} j＝1->n使用曲线插值算法可以得到加工路径，从而引导加工作业。
因此，采用上述处理步骤可以减少计算量，降低计算的复杂度，加快了处理速度，减少了计算时间，满足实时处理的要求，并且降低了对软硬件的性能的要求，可以节约成本，降低了开发的难度，符合对高速化大规模生产模式的要求。
进一步，所述步骤S40、将所述j依次设置为从1至n，通过曲面拟合算法将所述局部点云S 3D-j拟合为曲面SS 3D-j，再通过使用所述局部搜索算法搜索并提取垂直于所述局部点云S 3D-j的所述包围盒的所述平面与对应所述曲面SS 3D-j的交界处的所述加工路径点SX j，从而获得所述加工路径点序列{SX j} j＝1->n。
采用拟合的曲面SS 3D-j，可以滤除局部点云S 3D-j的数据冗余，使局部点云S 3D-j的数据均匀化，并减轻因测量系统造成的测量偏差,消除数据的波动。特别是采用NURBS曲线拟合,最后可以生成一条光滑、平顺的加工路径。
进一步，所述步骤S20、将所述配准二维图像I RGB和对应的所述配准深度数据I D进行合并从而生成融合数据I RGB-D，使用通过预先训练好的图像分割模型从作为输入的所述融合数据I RGB-D之中分割出对应所述待加工目标的区域S RGB-D，依据所述区域S RGB-D从所述配准深度数据I D提取所述待加工目标的3D点云数据S 3D。
采用所述融合数据I RGB-D能有效提高分割出对应所述待加工目标的区域S RGB-D的精度和准确度，并且大大增强分割的鲁棒性和稳定性。
进一步，所述步骤S20的所述图像分割模型的所述预先训练所需的训练样本是通过使用所述RGB-D复合传感器采集包含同类的所述待加工目标的配准二维图像I RGB和对应的所述配准深度数据I D所生成。
使用所述RGB-D复合传感器以局部栅格空间放置法，可以获得大量的待加工目标的所述配准二维图像I RGB和对应的所述配准深度数据I D，再通过打标签作可以为训练样本；然后对所述的基于深度学习框架的图像分割模型进行训练，并微调训练过程中的相关参数，直到模型的准确率达到期望值。通过上述处理步骤可以非常高效的获得大量的训练样本数据，从而确保所述基于深度学习框架的所述图像分割模型对精度以及鲁棒性的要求。
进一步，所述步骤S50的所述转换矩阵BET是由通过预先校准的Denavit-Hartenberg(D-H)参数所生成。
通过使用激光跟踪仪校准Denavit-Hartenberg(D-H)参数的目的是提高所述机器人视觉引导算法的总体精度，并能确保将所述加工路径点SX j转换为加工引导点BX j准确，并且该方法具有处理速度快，成熟可靠，易于工程实现的特点。
上述本发明基于RGB-D数据融合的机器人视觉引导方法的第一实施例中的基于RGB-D数据融合的机器人视觉引导方法可以由本发明基于RGB-D数据融合的机器人视觉引导装置的第一实施例所提供的基于RGB-D数据融合的机器人视觉引导装置来实现。
参照图2，图2为本发明基于RGB-D数据融合的机器人视觉引导装置的第一实施例所提供的一种基于RGB-D数据融合的机器人视觉引导装置1，所述装置1包括：
加工目标数据采集模块10，用于通过RGB-D复合传感器获取包含待加工 目标的RGB二维图像和深度数据，依据所述RGB-D复合传感器的预设的配准参数将所述RGB二维图像与所述深度数据进行配准处理，从而生成对应所述RGB二维图像的配准二维图像I RGB和对应所述深度数据的配准深度数据I D。
参照图3，所述RGB-D复合传感器被设置在机械臂D40的顶端，RGB摄像头D20在所述RGB-D复合视觉传感器的中间位置，彩色图像数据在传递到计算机之前会在进行压缩，以保证对RGB数据分析时的速度。所述RGB-D复合视觉传感器左右两侧的传感器D10和D30分别负责发射和接收红外线：首先通过左侧的红外线发射器D10向待加工目标O10发射红外线，这束红外线由于具有高度随机性，其在空间中任意两个不同位置所反射形成的光斑都不相同，对环境形成立体的“光编码”；再通过右侧的红外线接收器D30来采集视野中的红外线图像；最终，利用RGB-D复合视觉传感器的参数对这副红外图像进行一系列复杂的计算，就可以得到视野中的深度数据。
加工目标识别模块20、用于使用通过预先训练好的图像分割模型从作为输入的所述配准二维图像I RGB之中分割出对应所述待加工目标的区域S RGB，依据所述区域S RGB从所述配准深度数据I D提取所述待加工目标的3D点云数据S 3D。
基于深度学习框架的所述图像分割模型实现了对待加工目标的语义分割的神经网络模型，具有精度高，处理速度快，可以实时处理的特点。
加工目标分割模块30、用于从所述3D点云数据S 3D提取显著特征点序列{KX i} i＝1->m，以所述显著特征点序列{KX i} i＝1->m为分界点将所述3D点云数据S 3D分割为局部点云集{S 3D-j} j＝1->n，所述KX i为显著特征点，所述i为对应所述显著特征点KX i的序号，所述i的取值范围为[1,m],所述m为所述显著特征点KX i的总数目，所述S 3D-j为局部点云，所述j为所述局部点云S 3D-j的序号，所述j的取值范围为[1,n]，所述n为所述局部点云S 3D-j的总数目。
所述KX i是对应显著特征点的坐标向量，所述S 3D-j包含所述局部点云的 全部点的集合。
加工路径点获取模块40、用于将所述j依次设置为从1至n，通过使用局部搜索算法搜索并提取垂直于所述局部点云S 3D-j的包围盒的平面与对应所述局部点云S 3D-j的交界处的加工路径点SX j，从而获得所述加工路径点序列{SX j} j＝1->n。
所述加工目标分割模块30将所述待加工目标的所述3D点云数据S 3D以显著特征点序列{KX i} i＝1->m为分界点将其分割为局部点云集{S 3D-j} j＝1->n，在所述加工路径点获取模块40可以简单有效的获取对应的加工路径点序列{SX j} j＝1->n。特别是对非常复杂的所述3D点云数据S 3D，更加利于对加工路径点序列{SX j} j＝1->n的提取，并且大大增强稳定性和鲁棒性。
加工引导点转换模块50、用于将所述j依次设置为从1至n，使用包含通过转换矩阵BET将所述加工路径点SX j转换为加工引导点BX j，从而将所述加工路径点序列{SX j} j＝1->n转换为加工引导点序列{BX j} j＝1->n。
加工路径点SX j为对应RGB-D复合视觉传感器的坐标系下的位置坐标信息，因此需要转换至对应作业坐标系下的位置坐标信息。最后将所述的加工引导点序列{BX j} j＝1->n发送给机器人执行相应的作业。通过对加工引导点序列{BX j} j＝1->n使用曲线插值算法可以得到加工路径，从而引导加工作业。
因此，采用上述模块可以减少计算量，降低计算的复杂度，加快了处理速度，减少了计算时间，满足实时处理的要求，并且降低了对软硬件的性能的要求，可以节约成本，降低了开发的难度，符合对高速化大规模生产模式的要求。
进一步，所述加工路径点获取模块40、用于将所述j依次设置为从1至n，通过曲面拟合算法将所述局部点云S 3D-j拟合为曲面SS 3D-j，再通过使用所述局部搜索算法搜索并提取垂直于所述局部点云S 3D-j的所述包围盒的所述平面与对应所述曲面SS 3D-j的交界处的所述加工路径点SX j，从而获得所述加工路径 点序列{SX j} j＝1->n。
采用拟合的曲面SS 3D-j，可以滤除局部点云S 3D-j的数据冗余，使局部点云S 3D-j的数据均匀化，并减轻因测量系统造成的测量偏差,消除数据的波动。特别是采用NURBS曲线拟合,最后可以生成一条光滑、平顺的加工路径。
进一步，所述加工目标识别模块20、用于将所述配准二维图像I RGB和对应的所述配准深度数据I D进行合并从而生成融合数据I RGB-D，使用通过预先训练好的图像分割模型从作为输入的所述融合数据I RGB-D之中分割出对应所述待加工目标的区域S RGB-D，依据所述区域S RGB-D从所述配准深度数据I D提取所述待加工目标的3D点云数据S 3D。
采用所述融合数据I RGB-D能有效提高分割出对应所述待加工目标的区域S RGB-D的精度和准确度，并且大大增强分割的鲁棒性和稳定性。
进一步，所述加工目标识别模块20的所述图像分割模型的所述预先训练所需的训练样本是通过使用所述RGB-D复合传感器采集包含同类的所述待加工目标的配准二维图像I RGB和对应的所述配准深度数据I D所生成。
使用所述RGB-D复合传感器以局部栅格空间放置法，可以获得大量的待加工目标的所述配准二维图像I RGB和对应的所述配准深度数据I D，再通过打标签作可以为训练样本；然后对所述的基于深度学习框架的图像分割模型进行训练，并微调训练过程中的相关参数，直到模型的准确率达到期望值。通过上述处理步骤可以非常高效的获得大量的训练样本数据，从而确保所述基于深度学习框架的所述图像分割模型对精度以及鲁棒性的要求。
进一步，所述加工引导点转换模块50的所述转换矩阵BET是由通过预先校准的Denavit-Hartenberg(D-H)参数所生成。
通过使用激光跟踪仪校准Denavit-Hartenberg(D-H)参数的目的是提高所 述机器人视觉引导算法的总体精度，并能确保将所述加工路径点SX j转换为加工引导点BX j准确，并且该方法具有处理速度快，成熟可靠，易于工程实现的特点。
需要说明的是，在本文中，术语“包括”、“包含”或者其任何其他变体意在涵盖非排他性的包含，从而使得包括一系列要素的过程、方法、物品或者装置不仅包括那些要素，而且还包括没有明确列出的其他要素，或者是还包括为这种过程、方法、物品或者装置所固有的要素。在没有更多限制的情况下，由语句“包括一个……”限定的要素，并不排除在包括该要素的过程、方法、物品或者装置中还存在另外的相同要素。
上述本发明实施例序号仅仅为了描述，不代表实施例的优劣。
显然，本领域的技术人员应该明白，上述的本发明的各模块单元或各步骤可以用通用的计算装置来实现，可选地，它们可以用计算装置可执行的程序代码来实现，从而，可以将它们存储在存储装置中由计算装置来执行，并且在某些情况下，可以以不同于此处的顺序执行所示出或描述的步骤，或者将它们分别制作成各个集成电路模块，或者将它们中的多个模块或步骤制作成单个集成电路模块来实现。这样，本发明不限制于任何特定的硬件和软件结合。
通过以上的实施方式的描述，本领域的技术人员可以清楚地了解到上述实施例方法可借助软件加必需的通用硬件平台的方式来实现，当然也可以通过硬件，但很多情况下前者是更佳的实施方式。基于这样的理解，本发明的技术方案本质上或者说对现有技术做出贡献的部分可以以软件产品的形式体现出来，该计算机软件产品存储在一个存储介质(如ROM/RAM、磁碟、光 盘)中，包括若干指令用以使得一台终端设备(可以是手机，计算机，服务器，空调器，或者网络设备等)执行本发明各个实施例所述的方法。
以上仅为本发明的优选实施例，并非因此限制本发明的专利范围，凡是利用本发明说明书及附图内容所作的等效结构或等效流程变换，或直接或间接运用在其他相关的技术领域，均同理包括在本发明的专利保护范围内。

Claims (10)

  1. 基于RGB-D数据融合的机器人视觉引导方法，其特征在于包括如下步骤：
    步骤1、通过RGB-D复合传感器获取包含待加工目标的RGB二维图像和深度数据，依据所述RGB-D复合传感器的预设的配准参数将所述RGB二维图像与所述深度数据进行配准处理，从而生成对应所述RGB二维图像的配准二维图像I RGB和对应所述深度数据的配准深度数据I D；
    步骤2、使用通过预先训练好的图像分割模型从作为输入的所述配准二维图像I RGB之中分割出对应所述待加工目标的区域S RGB，依据所述区域S RGB从所述配准深度数据I D提取所述待加工目标的3D点云数据S 3D；
    步骤3、从所述3D点云数据S 3D提取显著特征点序列{KX i} i＝1->m，以所述显著特征点序列{KX i} i＝1->m为分界点将所述3D点云数据S 3D分割为局部点云集{S 3D-j} j＝1->n，所述KX i为显著特征点，所述i为对应所述显著特征点KX i的序号，所述i的取值范围为[1,m],所述m为所述显著特征点KX i的总数目，所述S 3D-j为局部点云，所述j为所述局部点云S 3D-j的序号，所述j的取值范围为[1,n]，所述n为所述局部点云S 3D-j的总数目；
    步骤4、将所述j依次设置为从1至n，通过使用局部搜索算法搜索并提取垂直于所述局部点云S 3D-j的包围盒的平面与对应所述局部点云S 3D-j的交界处的加工路径点SX j，从而获得所述加工路径点序列{SX j} j＝1->n；
    步骤5、将所述j依次设置为从1至n，使用包含通过转换矩阵BET将所述加工路径点SX j转换为加工引导点BX j，从而将所述加工路径点序列{SX j} j＝1->n转换为加工引导点序列{BX j} j＝1->n。
  2. 如权利要求1所述的基于RGB-D数据融合的机器人视觉引导方法，其特征在于，
    所述步骤4、将所述j依次设置为从1至n，通过曲面拟合算法将所述局 部点云S 3D-j拟合为曲面SS 3D-j，再通过使用所述局部搜索算法搜索并提取垂直于所述局部点云S 3D-j的所述包围盒的所述平面与对应所述曲面SS 3D-j的交界处的所述加工路径点SX j，从而获得所述加工路径点序列{SX j} j＝1->n。
  3. 如权利要求1所述的基于RGB-D数据融合的机器人视觉引导方法，其特征在于，
    所述步骤2、将所述配准二维图像I RGB和对应的所述配准深度数据I D进行合并从而生成融合数据I RGB-D，使用通过预先训练好的图像分割模型从作为输入的所述融合数据I RGB-D之中分割出对应所述待加工目标的区域S RGB-D，依据所述区域S RGB-D从所述配准深度数据I D提取所述待加工目标的3D点云数据S 3D。
  4. 如权利要求1所述的基于RGB-D数据融合的机器人视觉引导方法，其特征在于，所述步骤2的所述图像分割模型的所述预先训练所需的训练样本是通过使用所述RGB-D复合传感器采集包含同类的所述待加工目标的配准二维图像I RGB和对应的所述配准深度数据I D所生成。
  5. 如权利要求1所述的基于RGB-D数据融合的机器人视觉引导方法，其特征在于，所述步骤5的所述转换矩阵BET是由通过预先校准的Denavit-Hartenberg(D-H)参数所生成。
  6. 基于RGB-D数据融合的机器人视觉引导装置，其特征在于，包括：
    加工目标数据采集模块、用于通过RGB-D复合传感器获取包含待加工目标的RGB二维图像和深度数据，依据所述RGB-D复合传感器的预设的配准参数将所述RGB二维图像与所述深度数据进行配准处理，从而生成对应所述RGB二维图像的配准二维图像I RGB和对应所述深度数据的配准深度数 据I D；
    加工目标识别模块、用于使用通过预先训练好的图像分割模型从作为输入的所述配准二维图像I RGB之中分割出对应所述待加工目标的区域S RGB，依据所述区域S RGB从所述配准深度数据I D提取所述待加工目标的3D点云数据S 3D；
    加工目标分割模块、用于从所述3D点云数据S 3D提取显著特征点序列{KX i} i＝1->m，以所述显著特征点序列{KX i} i＝1->m为分界点将所述3D点云数据S 3D分割为局部点云集{S 3D-j} j＝1->n，所述KX i为显著特征点，所述i为对应所述显著特征点KX i的序号，所述i的取值范围为[1,m],所述m为所述显著特征点KX i的总数目，所述S 3D-j为局部点云，所述j为所述局部点云S 3D-j的序号，所述j的取值范围为[1,n]，所述n为所述局部点云S 3D-j的总数目；
    加工路径点获取模块、用于将所述j依次设置为从1至n，通过使用局部搜索算法搜索并提取垂直于所述局部点云S 3D-j的包围盒的平面与对应所述局部点云S 3D-j的交界处的加工路径点SX j，从而获得所述加工路径点序列{SX j} j＝1->n；
    加工引导点转换模块、用于将所述j依次设置为从1至n，使用包含通过转换矩阵BET将所述加工路径点SX j转换为加工引导点BX j，从而将所述加工路径点序列{SX j} j＝1->n转换为加工引导点序列{BX j} j＝1->n。
  7. 如权利要求6所述的基于RGB-D数据融合的机器人视觉引导装置，其特征在于，
    所述加工路径点获取模块、用于将所述j依次设置为从1至n，通过曲面拟合算法将所述局部点云S 3D-j拟合为曲面SS 3D-j，再通过使用所述局部搜索算法搜索并提取垂直于所述局部点云S 3D-j的所述包围盒的所述平面与对应所述曲面SS 3D-j的交界处的所述加工路径点SX j，从而获得所述加工路径点序列{SX j} j＝1->n。
  8. 如权利要求6所述的基于RGB-D数据融合的机器人视觉引导装置，其特征在于，
    所述加工目标识别模块、用于将所述配准二维图像I RGB和对应的所述配准深度数据I D进行合并从而生成融合数据I RGB-D，使用通过预先训练好的图像分割模型从作为输入的所述融合数据I RGB-D之中分割出对应所述待加工目标的区域S RGB-D，依据所述区域S RGB-D从所述配准深度数据I D提取所述待加工目标的3D点云数据S 3D。
  9. 如权利要求6所述的基于RGB-D数据融合的机器人视觉引导装置，其特征在于，所述加工目标识别模块之中的所述图像分割模型的所述预先训练所需的训练样本是通过使用所述RGB-D复合传感器采集包含同类的所述待加工目标的配准二维图像I RGB和对应的所述配准深度数据I D所生成。
  10. 如权利要求6所述的基于RGB-D数据融合的机器人视觉引导装置，其特征在于，所述加工引导点转换模块之中的所述转换矩阵BET是由通过预先校准的Denavit-Hartenberg(D-H)参数所生成。
PCT/CN2020/101335 2019-11-29 2020-07-10 基于rgb-d数据融合的机器人视觉引导方法和装置 WO2021103558A1 (zh)

Applications Claiming Priority (2)

Application Number Priority Date Filing Date Title
CN201911196215.4 2019-11-29
CN201911196215.4A CN110992410B (zh) 2019-11-29 2019-11-29 基于rgb-d数据融合的机器人视觉引导方法和装置

Publications (1)

Publication Number Publication Date
WO2021103558A1 true WO2021103558A1 (zh) 2021-06-03

Family

ID=70088010

Family Applications (1)

Application Number Title Priority Date Filing Date
PCT/CN2020/101335 WO2021103558A1 (zh) 2019-11-29 2020-07-10 基于rgb-d数据融合的机器人视觉引导方法和装置

Country Status (2)

Country Link
CN (1) CN110992410B (zh)
WO (1) WO2021103558A1 (zh)

Families Citing this family (3)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN110992410B (zh) * 2019-11-29 2020-10-30 广东技术师范大学 基于rgb-d数据融合的机器人视觉引导方法和装置
CN113103226A (zh) * 2021-03-08 2021-07-13 同济大学 一种面向陶瓷素坯加工制造的视觉引导机器人系统
CN116652951B (zh) * 2023-06-08 2024-04-05 广州鑫帅机电设备有限公司 一种非结构化大作业空间的机器人视觉定位方法及装置

Citations (5)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN104856720A (zh) * 2015-05-07 2015-08-26 东北电力大学 一种基于rgb-d传感器的机器人辅助超声扫描系统
CN106251353A (zh) * 2016-08-01 2016-12-21 上海交通大学 弱纹理工件及其三维位姿的识别检测方法及系统
US10150213B1 (en) * 2016-07-27 2018-12-11 X Development Llc Guide placement by a robotic device
CN110014426A (zh) * 2019-03-21 2019-07-16 同济大学 一种利用低精度深度相机高精度抓取形状对称工件的方法
CN110992410A (zh) * 2019-11-29 2020-04-10 广东技术师范大学 基于rgb-d数据融合的机器人视觉引导方法和装置

Family Cites Families (6)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN104063894B (zh) * 2014-06-13 2017-02-22 中国科学院深圳先进技术研究院 点云三维模型重建方法及系统
CN104324861B (zh) * 2014-08-12 2016-06-15 清华大学 一种多参数时变机器人喷涂方法
US10217277B2 (en) * 2015-12-04 2019-02-26 Autodesk, Inc. Keypoint-based point-pair-feature for scalable automatic global registration of large RGB-D scans
CN105665970B (zh) * 2016-03-01 2018-06-22 中国科学院自动化研究所 用于焊接机器人的路径点自动生成系统和方法
JP6911777B2 (ja) * 2018-01-23 2021-07-28 トヨタ自動車株式会社 動作軌道生成装置
CN109541997B (zh) * 2018-11-08 2020-06-02 东南大学 一种面向平面/近似平面工件的喷涂机器人快速智能编程方法

Patent Citations (5)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN104856720A (zh) * 2015-05-07 2015-08-26 东北电力大学 一种基于rgb-d传感器的机器人辅助超声扫描系统
US10150213B1 (en) * 2016-07-27 2018-12-11 X Development Llc Guide placement by a robotic device
CN106251353A (zh) * 2016-08-01 2016-12-21 上海交通大学 弱纹理工件及其三维位姿的识别检测方法及系统
CN110014426A (zh) * 2019-03-21 2019-07-16 同济大学 一种利用低精度深度相机高精度抓取形状对称工件的方法
CN110992410A (zh) * 2019-11-29 2020-04-10 广东技术师范大学 基于rgb-d数据融合的机器人视觉引导方法和装置

Non-Patent Citations (1)

* Cited by examiner, † Cited by third party
Title
LIU YI-DAN, WANG ZI-NIU2 , CHEN JIE , LI SHAO-BO: "Research of the NC-Simulation Based on DEXE L and t he Discrete", COMPUTER ENGINEERING AND SCIENCE, GUOFANG KEJI DAXUE JISUANJI XUEYUAN, CN, vol. 33, no. 1, 15 January 2011 (2011-01-15), CN, pages 171 - 175, XP055815507, ISSN: 1007-130X *

Also Published As

Publication number Publication date
CN110992410A (zh) 2020-04-10
CN110992410B (zh) 2020-10-30

Similar Documents

Publication Publication Date Title
WO2021109575A1 (zh) 基于融入全局视觉和局部视觉的机器人视觉引导方法和装置
WO2021103558A1 (zh) 基于rgb-d数据融合的机器人视觉引导方法和装置
EP3629057B1 (en) Method and apparatus for calibrating relative pose and medium
CN111089569B (zh) 一种基于单目视觉的大型箱体测量方法
CN110426051B (zh) 一种车道线绘制方法、装置及存储介质
CN111179324B (zh) 基于颜色和深度信息融合的物体六自由度位姿估计方法
CN110176078B (zh) 一种训练集数据的标注方法及装置
CN113450408B (zh) 一种基于深度相机的非规则物体位姿估计方法及装置
CN111340797A (zh) 一种激光雷达与双目相机数据融合检测方法及系统
US8340401B2 (en) Method and system for aligning three-dimensional surfaces
CN111476841B (zh) 一种基于点云和图像的识别定位方法及系统
WO2021103824A1 (zh) 基于标定块的机器人手眼标定中关键点位置确定方法与装置
CN107545247B (zh) 基于双目识别的立体认知方法
CN115972093B (zh) 工件表面的测量方法及装置、机翼壁板软模的打磨方法
CN110533716A (zh) 一种基于3d约束的语义slam系统及方法
US11455767B1 (en) Intelligent material completeness detection and configuration method based on digital twin and augmented reality (AR)
Zhang et al. EANet: Edge-attention 6D pose estimation network for texture-less objects
CN110568437A (zh) 一种基于雷达辅助的精确环境建模方法
CN114310883A (zh) 一种基于多知识库的机械臂自主装配方法
CN111598172B (zh) 基于异构深度网络融合的动态目标抓取姿态快速检测方法
CN106055244B (zh) 一种基于Kinect和语音的人机交互方法
Liu et al. AR-Driven Industrial Metaverse for the Auxiliary Maintenance of Machine Tools in IoT-Enabled Manufacturing Workshop
CN113191279A (zh) 数据标注方法、装置、设备、存储介质以及计算机程序产品
Gu et al. Research on static vision-based target localization for astronaut assistant robots
CN111415384A (zh) 一种基于深度学习的工业图像组件精准定位系统

Legal Events

Date Code Title Description
121 Ep: the epo has been informed by wipo that ep was designated in this application

Ref document number: 20893916

Country of ref document: EP

Kind code of ref document: A1

NENP Non-entry into the national phase

Ref country code: DE

122 Ep: pct application non-entry in european phase

Ref document number: 20893916

Country of ref document: EP

Kind code of ref document: A1