CN112045676A - A method for robot grasping transparent objects based on deep learning - Google Patents

A method for robot grasping transparent objects based on deep learning Download PDF

Info

Publication number
CN112045676A
CN112045676A CN202010755192.2A CN202010755192A CN112045676A CN 112045676 A CN112045676 A CN 112045676A CN 202010755192 A CN202010755192 A CN 202010755192A CN 112045676 A CN112045676 A CN 112045676A
Authority
CN
China
Prior art keywords
coordinate system
robot
camera
grasping
depth
Prior art date
Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
Pending
Application number
CN202010755192.2A
Other languages
Chinese (zh)
Inventor
雷渠江
徐杰
李秀昊
桂光超
王雨禾
潘艺芃
周纪民
王卫军
韩彰秀
Current Assignee (The listed assignees may be inaccurate. Google has not performed a legal analysis and makes no representation or warranty as to the accuracy of the list.)
Guangzhou Institute of Advanced Technology of CAS
Original Assignee
Guangzhou Institute of Advanced Technology of CAS
Priority date (The priority date is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the date listed.)
Filing date
Publication date
Application filed by Guangzhou Institute of Advanced Technology of CAS filed Critical Guangzhou Institute of Advanced Technology of CAS
Priority to CN202010755192.2A priority Critical patent/CN112045676A/en
Publication of CN112045676A publication Critical patent/CN112045676A/en
Pending legal-status Critical Current

Links

Images

Classifications

    • B—PERFORMING OPERATIONS; TRANSPORTING
    • B25—HAND TOOLS; PORTABLE POWER-DRIVEN TOOLS; MANIPULATORS
    • B25J—MANIPULATORS; CHAMBERS PROVIDED WITH MANIPULATION DEVICES
    • B25J9/00—Programme-controlled manipulators
    • B25J9/16—Programme controls
    • B25J9/1694—Programme controls characterised by use of sensors other than normal servo-feedback from position, speed or acceleration sensors, perception control, multi-sensor controlled systems, sensor fusion
    • B25J9/1697—Vision controlled systems
    • B—PERFORMING OPERATIONS; TRANSPORTING
    • B25—HAND TOOLS; PORTABLE POWER-DRIVEN TOOLS; MANIPULATORS
    • B25J—MANIPULATORS; CHAMBERS PROVIDED WITH MANIPULATION DEVICES
    • B25J9/00—Programme-controlled manipulators
    • B25J9/16—Programme controls
    • B25J9/1628—Programme controls characterised by the control loop
    • B25J9/163—Programme controls characterised by the control loop learning, adaptive, model based, rule based expert control

Landscapes

  • Engineering & Computer Science (AREA)
  • Robotics (AREA)
  • Mechanical Engineering (AREA)
  • Image Analysis (AREA)

Abstract

本发明公开了一种基于深度学习的机器人抓取透明物体的方法，包括以下步骤：S1：完成机器人抓取透明物体系统的硬件环境搭建；S2：完成机器人抓取透明物体系统的相机校准；S3：完成基于卷积神经网络的抓取规划模型的训练，和机器人在真实环境下的抓取。S3具体实现方法包括：利用深度相机扫描抓取透明物体的彩色图像及深度图像；对采集的图像进行滤波；使用ClearGrasp深度学习算法完成透明物体检测与分割；使用接触线寻找法对物体进行抓取位置的搜索及评分，在得到最佳抓取位置后进行正确的抓取。本发明能通过RGB‑D摄像头准确预测高透明度物体的3D数据，通过反射光点来准确推算透明物体曲面法线，提高透明物体预测准确性。

Figure 202010755192

The invention discloses a deep learning-based method for a robot to grasp a transparent object, comprising the following steps: S1: complete the hardware environment construction of the robot grasping transparent object system; S2: complete the camera calibration of the robot grasping transparent object system; S3 : Complete the training of the grasping planning model based on the convolutional neural network, and the grasping of the robot in the real environment. The specific implementation method of S3 includes: using a depth camera to scan and grab color images and depth images of transparent objects; filtering the collected images; using the ClearGrasp deep learning algorithm to complete the detection and segmentation of transparent objects; using the contact line finding method to grasp the objects Searching and scoring of locations, and crawls correctly after getting the best crawl location. The invention can accurately predict the 3D data of the high-transparency object through the RGB-D camera, accurately calculate the surface normal of the transparent object through the reflection light spot, and improve the prediction accuracy of the transparent object.

Figure 202010755192

Description

Method for grabbing transparent object by robot based on deep learning
Technical Field
The invention relates to the technical field of robot grabbing, in particular to a method for grabbing a transparent object by a robot based on deep learning.
Background
For the service robot, it is most important to be able to grasp the target object more quickly and accurately in a home environment, and only then, the service robot can help people with mobility disabilities better. The key of successful grabbing is the identification and positioning of targets, and at present, a vision sensor is generally adopted on a robot to identify objects. Among a plurality of grabbed objects, transparent objects are very common in life, and whether the transparent objects can be effectively identified and positioned plays a crucial role in the grabbing efficiency of the objects. However, when the robot recognizes the transparent object by using vision, the transparent object area is sensitive to light changes, and does not have enough texture features to extract, and has the reasons of dependence on the background environment, external influence on the strength gradient features and the like, so that the recognition of the transparent object is always a problem which is difficult to effectively solve.
At present, the commonly used transparent object detection methods include a non-visual method and transparent object detection based on an RGB two-dimensional image. Among them, the non-visual method is complicated to use, and makes the robot cost very high, is inconvenient for the service robot to use; the two-dimensional object obtained by the RGB image method has weak robustness, the detection condition is harsh, and the spatial position of the object cannot be obtained.
Disclosure of Invention
In view of the above, there is a need to provide a method for a robot to grab a transparent object based on deep learning, that is, a three-dimensional geometry of the transparent object is accurately estimated from an RGB-D image by using a deep learning method for the robot to operate, so as to solve the task of grabbing the transparent object in a home scene by the robot.
In order to realize the purpose, the invention is realized according to the following technical scheme:
a method for grabbing a transparent object by a robot based on deep learning comprises the following steps:
step S1: completing the establishment of a hardware environment of a system for grabbing the transparent object by the robot;
step S2: completing the calibration of a camera of a system for grabbing the transparent object by the robot;
step S3: and finishing the training of a grasping planning model based on the convolutional neural network and the grasping of the robot in a real environment.
Further, the hardware environment of the robot grasping transparent object system comprises a depth camera, at least one computer with ROS dynamics, at least one robot with a gripper, and at least one object to be grasped;
the depth camera is used for acquiring 3D visual data and is installed on the robot;
the computer is used for finishing the training of grabbing the network model;
the robot is used for grabbing an object to be grabbed.
Further, when the camera shoots an object, the camera captures a depth image and a color image at the same time, when the camera is calibrated, the color image and the depth image need to be calibrated, and each pixel point of the depth image corresponds to each pixel point of the color image through calibration, where the step S2 specifically includes the following steps:
step S21: determining internal parameters and external parameters of a binocular camera through camera calibration, and completing the transformation from a world coordinate system to a camera coordinate system;
step S22: and determining the relative position between the camera and the end effector through hand-eye calibration, and finishing the transformation of a camera coordinate system and a robot end effector coordinate system.
Further, the specific implementation method of step S21 includes:
the transformation of the world coordinate system into the camera coordinate system is described using the rotation matrix R and the translation vector T, as shown in equation (1):
Figure BDA0002611308060000031
in the formula (1), R1、T1Is an external reference of the Levoeye camera, R2、T2Is external reference of the right eye camera, which is obtained by camera calibration, (X)W，YW，ZW) Is the coordinate of a point in space under the world coordinate system, (X)1，Y1，Z1) Is the coordinate of a point in space under the coordinate system of the eye-lens camera (X)2，Y2，Z2) A point in space is a coordinate under a coordinate system of a right-eye camera;
taking the left eye camera coordinate system as a reference, taking the rotation matrix from the right eye camera coordinate system to the left eye camera coordinate system as R ', taking the translation vector as T', then:
Figure BDA0002611308060000032
according to formula (1) and formula (2):
Figure BDA0002611308060000033
the position of the calibration plate is kept unchanged when the binocular camera is used for shooting, the left-eye camera and the right-eye camera shoot images of the calibration plate at the same time, a plurality of groups of image pairs are collected and then led into the tool box, the tool box automatically calculates a rotation matrix and a translation vector between the two cameras, and the rotation matrix and the translation vector are used for completing the transformation from a world coordinate system to a camera coordinate system.
Further, the specific implementation method of step S22 includes:
the method comprises the following steps of solving transformation from a camera coordinate system to a robot end effector coordinate system through hand-eye calibration, wherein a hand represents an end effector, an eye represents a camera, and in the hand-eye calibration process, 4 coordinate systems are involved, namely a calibration plate coordinate system B, a camera coordinate system C, an end effector coordinate system T and a robot base coordinate system R;
using transformation matrices
Figure BDA0002611308060000041
Describing the transformation of the calibration plate coordinate system B to the robot base coordinate system R,
Figure BDA0002611308060000042
is represented as follows:
Figure BDA0002611308060000043
in the formula (4), the reaction mixture is,
Figure BDA0002611308060000044
expressing a transformation matrix from the coordinate system B of the calibration plate to the coordinate system C of the camera, namely camera external parameters, and obtaining the transformation matrix through camera calibration;
Figure BDA0002611308060000045
a transformation matrix representing the coordinate system T of the end effector to the coordinate system R of the robot base is obtained through parameters on the robot demonstrator;
Figure BDA0002611308060000046
a hand-eye matrix to be solved is obtained;
in the calibration process, the position of the calibration plate is kept unchanged, the robot is controlled to shoot images of the calibration plate from different positions, and two positions are selected for analysis, so that the following formula (5) can be obtained:
Figure BDA0002611308060000047
in the formula (5), the reaction mixture is,
Figure BDA0002611308060000048
calibration board for respectively representing position i and position i +1A transformation matrix from coordinate system B to robot base coordinate system R,
Figure BDA0002611308060000049
respectively representing transformation matrixes from a position i and a position i +1 of the end effector coordinate system T to a robot base coordinate system R,
Figure BDA00026113080600000410
respectively representing the hand-eye matrix to be solved at the position i and the position i +1,
Figure BDA00026113080600000411
respectively representing transformation matrixes from a position i and a position i +1 calibration board coordinate system B to a camera coordinate system C; because the relative position between the calibration plate and the robot base is not changed, and the relative position between the robot end effector and the camera is not changed, the method comprises the following steps
Figure BDA00026113080600000412
This is obtained simultaneously for formula (6):
Figure BDA00026113080600000413
in the formula (6), the reaction mixture is,
Figure BDA00026113080600000414
are all known quantities, and are finally solved to obtain
Figure BDA00026113080600000415
I.e. a transformation matrix from the camera coordinate system to the robot end effector coordinate system.
Further, the specific implementation method of step S3 includes:
s31: utilizing a depth camera to scan and capture a color image and a depth image of a transparent object;
s32: filtering the acquired image;
s33: completing transparent object detection and segmentation by using a ClearGrasp deep learning algorithm;
s34: and searching and scoring the grabbing position of the object by using a contact line searching method, and accurately grabbing the object after the optimal grabbing position is obtained.
Further, in step S32, a gaussian filtering algorithm with balanced speed and effect is selected to filter the acquired image, where the gaussian filtering formula is shown in equation (7):
Figure BDA0002611308060000051
in equation (7), f (x, y) represents a gaussian function value, the squares of x and y represent the distances between other pixels in the neighborhood and the center pixel in the neighborhood, respectively, and σ represents a standard deviation.
Further, the specific implementation method of step S33 includes:
predicting a surface normal, identifying a boundary and segmenting a transparent object from the filtered image by adopting a ClearGrasp deep learning method, wherein the segmented mask is used for modifying the input depth image; then, the depth of all the surfaces of the high-transparency objects in the scene is reconstructed by using a global optimization algorithm, and the edges, the occlusion and the segmentation of the 3D reconstruction are optimized by using the predicted surface normal.
Further, in step S33, the cleargraph includes 3 neural networks, and the outputs of the 3 neural networks are integrated for global optimization;
the 3 neural networks include: a transparent object segmentation network, an edge identification network and a surface normal vector estimation network;
transparent object segmentation network: inputting a single RGB picture, and outputting a pixel Mask of a transparent object in a scene, namely judging that each pixel point belongs to a transparent or non-transparent object, and removing the pixel judged as the transparent object in subsequent optimization to obtain a modified depth map;
edge identification network: for a single RGB picture, outputting information of a shielding edge and a connected edge, which helps a network to better distinguish different edges in the picture and make more accurate prediction on the edge with discontinuous depth;
surface normal vector estimation: using the RGB picture as input, and performing L2 regularization on the output;
reconstructing the three-dimensional surface of the missing depth area of the transparent object by using the global optimization algorithm, filling the removed depth area by using the normal vector of the surface of the predicted transparent object, and observing the depth discontinuity of the information displayed by the shielding edge, wherein the depth discontinuity is expressed by the following formula:
E＝λDED+λSES+λNENB (8)
in the formula (8), E represents the predicted depth, EDDistance representing predicted depth and observed original depth, ESRepresenting depth differences of adjacent points, ENDenotes the consistency of the normal vector of the predicted depth and the predicted surface, B denotes the boundary occlusion based on whether the pixel occludes the boundary, lambdaD、λS、λNRepresenting the correlation coefficient.
Further, in step S34, the direction of the best capture position is the main direction of the object image gradient, the main position extraction is performed on the depth image of the object to increase the speed of selecting the capture position, that is, gradient values are calculated on the x-axis and the y-axis, respectively, and the gradient direction of each pixel is calculated and arranged and counted through a histogram, wherein the method for calculating the object gradient and calculating the gradient direction is as follows:
using [ -1,0,1 [ ]]And [ -1,0,1 [ -1]TThe two convolution kernels perform two-dimensional convolution on the image to calculate the object gradient;
the gradient magnitude and direction are calculated as follows:
Figure BDA0002611308060000061
Figure BDA0002611308060000062
in the above formula, gxAnd gyRespectively representing gradient values in x and y directions, g representing gradient magnitude, and theta representing gradient direction;
after obtaining the gradient, a threshold value g is setThreshAt 250 f, the robot has enough depth to place the splint for effective grasping only if the gradient is greater than the threshold value, i.e. the robot has sufficient depth to place the splint for effective grasping
Figure BDA0002611308060000063
In the process of grabbing a transparent object by a robot, two contact lines exist when a clamping jaw is in contact with the object, and the conditions for selecting the proper contact lines are as follows:
the gradient directions of two contact lines are basically opposite;
the distance between the two contact lines does not exceed the maximum opening distance of the gripper;
the depth of the two contact lines is not more than 1/2 of the maximum depth in the clamping jaw;
the depth difference between the shallowest point in the area contained between the two contact lines and the shallowest point of the contact line does not exceed the internal depth of the clamping jaw;
the following formula was used to evaluate the grasping reliability of a pair of contact wires:
Figure BDA0002611308060000071
wherein G represents the grasping reliability,/1、l2Respectively showing the lengths of two contact lines of the clamping jaw and the transparent object to be grabbed, L showing the width of the clamping jaw,
Figure BDA0002611308060000072
for the purpose of evaluating the length of the contact line,
Figure BDA0002611308060000073
evaluation of the ratio of the lengths of the two contact lines,/maxIndicating the long strip in the contact line, lminWhich represents the short strip of the strip,
Figure BDA0002611308060000074
for evaluating contact line fitting degree of paw，dlRepresenting the shallowest point of the contact line, dsRepresenting the shallowest point in the rectangular frame area, and using sin theta to evaluate the error degree of two contact lines, wherein theta is an acute angle formed by a connecting line of the midpoints of the two contact lines and the contact lines;
all contact line combinations are traversed through equation (12), and the combination with the highest score is selected as the best grasping position.
The invention has the advantages and positive effects that: aiming at the problem that the transparent object is difficult to grasp, the invention provides a clearGrasp-based deep learning algorithm which is characterized in that 3D data of a high-transparency object can be accurately predicted through an RGB-D camera.
Drawings
In order to more clearly illustrate the technical solutions in the embodiments of the present invention, the drawings needed to be used in the description of the embodiments are briefly introduced below, and it is obvious that the drawings in the following description are only some embodiments of the present invention, and it is obvious for those skilled in the art to obtain other drawings based on these drawings without creative efforts.
FIG. 1 is a schematic flow chart of a method for grabbing a transparent object by a robot based on deep learning according to the present invention;
FIG. 2 is a system hardware diagram of the robot based on deep learning for grabbing transparent objects according to the present invention;
fig. 3 is a schematic diagram of a cleargrass algorithm model network structure according to the present invention.
Detailed Description
In order to make the aforementioned objects, features and advantages of the present invention comprehensible, embodiments accompanied with figures are described in detail below. It should be noted that the described embodiments are only a part of the embodiments of the present invention, and not all embodiments, and all other embodiments obtained by those skilled in the art without any inventive work based on the embodiments of the present invention belong to the protection scope of the present invention.
Examples
Fig. 1 is a schematic flow chart of a method for grabbing a transparent object by a robot based on deep learning according to the present invention, and as shown in fig. 1, the present invention provides a method for grabbing a transparent object by a robot based on deep learning, which includes the following steps:
step S1: completing the establishment of a hardware environment of a system for grabbing the transparent object by the robot;
step S2: completing the calibration of a camera of a system for grabbing the transparent object by the robot;
step S3: and finishing the training of a grasping planning model based on the convolutional neural network and the grasping of the robot in a real environment.
Specifically, the hardware environment of the system for grabbing the transparent object by the robot is shown in fig. 2, and comprises an Inter Realsense depth camera, at least one ROS dynamic Ubantu18.04 computer, at least one UR5 robot with a gripper and at least one object to be grabbed;
the Inter Realsense depth camera is used for collecting 3D visual data and is installed on the UR5 robot;
the Ubantu18.04 computer is used for finishing the training of grabbing the network model;
the UR5 robot is used to grab objects to be grabbed.
Specifically, when the depth camera shoots an object, the depth camera captures a depth image and a color image at the same time, when the camera is calibrated, the color image and the depth image need to be calibrated, and each pixel point of the depth image corresponds to each pixel point of the color image through calibration, where the step S2 specifically includes the following steps:
step S21: determining internal parameters and external parameters of a binocular camera through camera calibration, and completing the transformation from a world coordinate system to a camera coordinate system;
step S22: and determining the relative position between the camera and the end effector through hand-eye calibration, and finishing the transformation of a camera coordinate system and a robot end effector coordinate system.
Specifically, the method for implementing step S21 includes:
the transformation of the world coordinate system into the camera coordinate system is described using the rotation matrix R and the translation vector T, as shown in equation (1):
Figure BDA0002611308060000091
in the formula (1), R1、T1Is an external reference of the Levoeye camera, R2、T2Is an external reference of the right eye camera, which can be obtained by camera calibration, (X)W，YW，ZW) Is the coordinate of a point in space under the world coordinate system, (X)1，Y1，Z1) Is the coordinate of a point in space under the coordinate system of the eye-lens camera (X)2，Y2，Z2) A point in space is a coordinate under a coordinate system of a right-eye camera;
taking the left eye camera coordinate system as a reference, taking the rotation matrix from the right eye camera coordinate system to the left eye camera coordinate system as R ', taking the translation vector as T', then:
Figure BDA0002611308060000101
according to formula (1) and formula (2):
Figure BDA0002611308060000102
the position of the calibration plate is kept unchanged when the binocular camera is used for shooting, the left-eye camera and the right-eye camera shoot images of the calibration plate at the same time, a plurality of groups of image pairs are collected and then led into a tool kit of Matlab, the tool kit automatically calculates a rotation matrix and a translation vector between the two cameras, and the transformation from a world coordinate system to a camera coordinate system can be completed by using the rotation matrix and the translation vector.
Specifically, the method for implementing step S22 includes:
the method comprises the following steps of solving transformation from a camera coordinate system to a robot end effector coordinate system through hand-eye calibration, wherein a hand represents an end effector, an eye represents a camera, and in the hand-eye calibration process, 4 coordinate systems are involved, namely a calibration plate coordinate system B, a camera coordinate system C, an end effector coordinate system T and a robot base coordinate system R;
using transformation matrices
Figure BDA0002611308060000107
Describing the transformation of the calibration plate coordinate system B to the robot base coordinate system R,
Figure BDA0002611308060000108
is represented as follows:
Figure BDA0002611308060000103
in the formula (4), the reaction mixture is,
Figure BDA0002611308060000104
a transformation matrix representing the coordinate system B of the calibration plate to the coordinate system C of the camera, namely camera external parameters, can be obtained through camera calibration;
Figure BDA0002611308060000105
a transformation matrix representing the coordinate system T of the end effector to the coordinate system R of the robot base can be obtained through parameters on the robot demonstrator;
Figure BDA0002611308060000106
a hand-eye matrix to be solved is obtained;
in the calibration process, the position of the calibration plate is kept unchanged, the robot is controlled to shoot images of the calibration plate from different positions, and two positions are selected for analysis, so that the following formula (5) can be obtained:
Figure BDA0002611308060000111
in the formula (5), the reaction mixture is,
Figure BDA0002611308060000112
respectively representing transformation matrixes from a coordinate system B of the calibration board at the position i and a coordinate system R of the robot base at the position i +1,
Figure BDA0002611308060000113
respectively representing transformation matrixes from a position i and a position i +1 of the end effector coordinate system T to a robot base coordinate system R,
Figure BDA0002611308060000114
respectively representing the hand-eye matrix to be solved at the position i and the position i +1,
Figure BDA0002611308060000115
respectively representing transformation matrixes from a position i and a position i +1 calibration board coordinate system B to a camera coordinate system C; because the relative position between the calibration plate and the robot base is not changed, and the relative position between the robot end effector and the camera is not changed, the method comprises the following steps
Figure BDA0002611308060000116
This is obtained simultaneously for formula (6):
Figure BDA0002611308060000117
in the formula (6), the reaction mixture is,
Figure BDA0002611308060000118
and
Figure BDA0002611308060000119
are all known quantities, and are finally solved to obtain
Figure BDA00026113080600001110
I.e. a transformation matrix from the camera coordinate system to the robot end effector coordinate system.
Specifically, the method for implementing step S3 includes:
s31: utilizing a RealSense RGB-D camera to scan and capture a color image and a depth image of a transparent object;
s32: filtering the acquired image;
s33: completing transparent object detection and segmentation by using a ClearGrasp deep learning algorithm;
s34: and searching and scoring the grabbing position of the object by using a contact line searching method, and accurately grabbing the object after the optimal grabbing position is obtained.
Specifically, in step S32, a gaussian filtering algorithm with balanced speed and effect is selected to filter the acquired image, where the gaussian filtering formula is shown in equation (7):
Figure BDA00026113080600001111
in equation (7), f (x, y) represents a gaussian function value, the squares of x and y represent the distances between other pixels in the neighborhood and the center pixel in the neighborhood, respectively, and σ represents a standard deviation.
Specifically, a cleargraph deep learning algorithm model network structure is shown in fig. 3, and the specific implementation method of step S33 includes:
predicting a surface normal, identifying a boundary and segmenting a transparent object from the filtered image by adopting a ClearGrasp deep learning method, wherein the segmented mask is used for modifying the input depth image; then, the depth of all the surfaces of the high-transparency objects in the scene is reconstructed by using a global optimization algorithm, and the edges, the occlusion and the segmentation of the 3D reconstruction are optimized by using the predicted surface normal.
Specifically, in step S33, the cleargraph includes 3 neural networks, and the outputs of the 3 neural networks are integrated for global optimization;
the 3 neural networks include: a transparent object segmentation network, an edge identification network and a surface normal vector estimation network;
transparent object segmentation network: inputting a single RGB picture, and outputting a pixel Mask of a transparent object in a scene, namely judging that each pixel point belongs to a transparent or non-transparent object, and removing the pixel judged as the transparent object in subsequent optimization to obtain a modified depth map;
edge identification network: for a single RGB picture, outputting information of a shielding edge and a connected edge, which helps a network to better distinguish different edges in the picture and make more accurate prediction on the edge with discontinuous depth;
surface normal vector estimation: using the RGB picture as input, and performing L2 regularization on the output;
reconstructing the three-dimensional surface of the missing depth area of the transparent object by using the global optimization algorithm, filling the removed depth area by using the normal vector of the surface of the predicted transparent object, and observing the depth discontinuity of the information displayed by the shielding edge, wherein the depth discontinuity can be expressed by the following formula:
E＝λDED+λSES+λNENB (8)
in the formula (8), E represents the predicted depth, EDDistance representing predicted depth and observed original depth, ESRepresenting depth differences of adjacent points, ENDenotes the consistency of the normal vector of the predicted depth and the predicted surface, B denotes the boundary occlusion based on whether the pixel occludes the boundary, lambdaD、λS、λNRepresenting the correlation coefficient.
Specifically, in step S34, the direction of the best capture position is the main direction of the object image gradient, the main position extraction is performed on the depth image of the object to accelerate the selection speed of the capture position, that is, gradient values are calculated on the x-axis and the y-axis, respectively, and the gradient direction of each pixel is calculated, and the gradient directions are arranged and counted through a histogram, wherein the method for calculating the object gradient and calculating the gradient direction is as follows:
using [ -1,0,1 [ ]]And [ -1,0,1 [ -1]TThe two convolution kernels perform two-dimensional convolution on the image to calculate the object gradient;
the gradient magnitude and direction are calculated as follows:
Figure BDA0002611308060000131
Figure BDA0002611308060000132
in the above formula, gxAnd gyRespectively representing gradient values in x and y directions, g representing gradient magnitude, and theta representing gradient direction;
after obtaining the gradient, a threshold value g is setThreshAt 250 f, the robot has enough depth to place the splint for effective grasping only if the gradient is greater than the threshold value, i.e. the robot has sufficient depth to place the splint for effective grasping
Figure BDA0002611308060000133
In the process of grabbing a transparent object by a robot, two contact lines exist when a clamping jaw is in contact with the object, and the conditions for selecting the proper contact lines are as follows:
the gradient directions of two contact lines are basically opposite;
the distance between the two contact lines does not exceed the maximum opening distance of the gripper;
the depth of the two contact lines is not more than 1/2 of the maximum depth in the clamping jaw;
the depth difference between the shallowest point in the area contained between the two contact lines and the shallowest point of the contact line does not exceed the internal depth of the clamping jaw;
the following formula was used to evaluate the grasping reliability of a pair of contact wires:
Figure BDA0002611308060000141
wherein G represents the grasping reliability,/1、l2Respectively showing the lengths of two contact lines of the clamping jaw and the transparent object to be grabbed, L showing the width of the clamping jaw,
Figure BDA0002611308060000142
for the purpose of evaluating the length of the contact line,
Figure BDA0002611308060000143
evaluation of twoLength ratio of strip contact line, /)maxIndicating the long strip in the contact line, lminWhich represents the short strip of the strip,
Figure BDA0002611308060000144
for evaluating the contact line engaging the paw, dlRepresenting the shallowest point of the contact line, dsRepresenting the shallowest point in the rectangular frame area, and using sin theta to evaluate the error degree of two contact lines, wherein theta is an acute angle formed by a connecting line of the midpoints of the two contact lines and the contact lines;
all contact line combinations are traversed through equation (12), and the combination with the highest score is selected as the best grasping position.
The invention has the advantages and positive effects that: aiming at the problem that the transparent object is difficult to grasp, the invention provides a clearGrasp-based deep learning algorithm which is characterized in that 3D data of a high-transparency object can be accurately predicted through an RGB-D camera.
The above-mentioned embodiments only express several embodiments of the present invention, and the description thereof is more specific and detailed, but not construed as limiting the scope of the present invention. It should be noted that, for a person skilled in the art, several variations and modifications can be made without departing from the inventive concept, which falls within the scope of the present invention. Therefore, the protection scope of the present invention should be subject to the appended claims.

Claims (10)

1.一种基于深度学习的机器人抓取透明物体的方法，其特征在于，包括以下步骤：1. a method for grasping a transparent object based on a deep learning robot, is characterized in that, comprises the following steps: 步骤S1：完成机器人抓取透明物体系统的硬件环境搭建；Step S1: complete the hardware environment construction of the robot grasping transparent object system; 步骤S2：完成机器人抓取透明物体系统的相机校准；Step S2: complete the camera calibration of the robot grasping transparent object system; 步骤S3：完成基于卷积神经网络的抓取规划模型的训练，和机器人在真实环境下的抓取。Step S3: Complete the training of the grasping planning model based on the convolutional neural network, and the grasping of the robot in the real environment. 2.根据权利要求1所述的基于深度学习的机器人抓取透明物体的方法，其特征在于，所述机器人抓取透明物体系统的硬件环境包括深度相机、至少一台ROS动力学特性的计算机、至少一台带有夹持器的机器人以及至少一个待抓取物体；2. the method for robot grasping transparent object based on deep learning according to claim 1, is characterized in that, the hardware environment of described robot grasping transparent object system comprises depth camera, the computer of at least one ROS dynamic characteristic, at least one robot with a gripper and at least one object to be grasped; 所述深度相机用于采集3D视觉数据，并安装在所述机器人上；The depth camera is used to collect 3D visual data and is installed on the robot; 所述计算机用于完成抓取网络模型的训练；The computer is used to complete the training of the grabbing network model; 所述机器人用于抓取待抓取物体。The robot is used for grasping the object to be grasped. 3.根据权利要求1所述的基于深度学习的机器人抓取透明物体的方法，其特征在于，相机在拍摄物体时，会同时捕获深度图像和彩色图像，在对相机进行标定时，需要对彩色图像和深度图像都进行标定，通过标定使得深度图像的每一个像素点与彩色图像的每一个像素点相对应，所述步骤S2具体包括以下步骤：3. The method for grasping a transparent object based on a deep learning robot according to claim 1, wherein the camera captures a depth image and a color image at the same time when photographing the object, and when calibrating the camera, it is necessary to measure the color image. Both the image and the depth image are calibrated, and through the calibration, each pixel of the depth image corresponds to each pixel of the color image, and the step S2 specifically includes the following steps: 步骤S21：通过相机标定确定双目相机内参和外参，完成世界坐标系到相机坐标系的变换；Step S21: Determine the internal parameters and external parameters of the binocular camera through camera calibration, and complete the transformation from the world coordinate system to the camera coordinate system; 步骤S22：通过手眼标定确定相机与末端执行器之间的相对位置，完成相机坐标系及机器人末端执行器坐标系的变换。Step S22: Determine the relative position between the camera and the end effector through hand-eye calibration, and complete the transformation of the camera coordinate system and the robot end effector coordinate system. 4.根据权利要求3所述的基于深度学习的机器人抓取透明物体的方法，其特征在于，所述步骤S21具体实现方法包括：4. The method for robot grasping transparent objects based on deep learning according to claim 3, wherein the specific implementation method of step S21 comprises: 使用旋转矩阵R和平移向量T描述世界坐标系到相机坐标系的变换，如式(1)所示：Use the rotation matrix R and the translation vector T to describe the transformation from the world coordinate system to the camera coordinate system, as shown in formula (1):
Figure FDA0002611308050000021
Figure FDA0002611308050000021
 式(1)中，R1、T1为左目相机的外参，R2、T2为右目相机的外参，其通过相机标定获得，(XW，YW，ZW)为空间中一点在世界坐标系下的坐标，(X1，Y1，Z1)为空间中一点在左目相机坐标系下的坐标，(X2，Y2，Z2)为空间中一点在右目相机坐标系下的坐标；In formula (1), R 1 and T 1 are the external parameters of the left camera, R 2 and T 2 are the external parameters of the right camera, which are obtained by camera calibration, and (X W , Y W , Z W ) is a point in the space The coordinates in the world coordinate system, (X 1 , Y 1 , Z 1 ) are the coordinates of a point in the space in the left-eye camera coordinate system, (X 2 , Y 2 , Z 2 ) is a point in the space in the right-eye camera coordinate system the coordinates below; 以左目相机坐标系为基准，右目相机坐标系到左目相机坐标系的旋转矩阵为R'，平移向量为T'，则有：Taking the left-eye camera coordinate system as the benchmark, the rotation matrix from the right-eye camera coordinate system to the left-eye camera coordinate system is R', and the translation vector is T', there are:
Figure FDA0002611308050000022
Figure FDA0002611308050000022
 根据式(1)和式(2)可得：According to formula (1) and formula (2), we can get:
Figure FDA0002611308050000023
Figure FDA0002611308050000023
 使用双目相机拍摄时保持标定板位置不变，左目和右目相机同时拍摄标定板图像，采集若干组图像对，然后导入工具箱中，工具箱自动计算出两个相机之间的旋转矩阵和平移向量，使用旋转矩阵和平移向量即可完成世界坐标系到相机坐标系的变换。When shooting with a binocular camera, keep the position of the calibration board unchanged. The left and right cameras capture images of the calibration board at the same time, collect several sets of image pairs, and then import them into the toolbox. The toolbox automatically calculates the rotation matrix between the two cameras and Translation vector, use the rotation matrix and translation vector to complete the transformation from the world coordinate system to the camera coordinate system. 5.根据权利要求3所述的基于深度学习的机器人抓取透明物体的方法，其特征在于，所述步骤S22具体实现方法包括：5. The method for grasping a transparent object by a robot based on deep learning according to claim 3, wherein the specific implementation method of the step S22 comprises: 通过手眼标定求解相机坐标系到机器人末端执行器坐标系的变换，其中手代表末端执行器，眼代表相机，在手眼标定过程中，涉及4个坐标系，分别为标定板坐标系B、相机坐标系C、末端执行器坐标系T、机器人底座坐标系R；Solve the transformation from the camera coordinate system to the robot end-effector coordinate system through hand-eye calibration, where the hand represents the end-effector and the eye represents the camera. In the process of hand-eye calibration, four coordinate systems are involved, namely the calibration board coordinate system B and the camera coordinate. System C, end effector coordinate system T, robot base coordinate system R; 使用变换矩阵
Figure FDA0002611308050000031
描述标定板坐标系B到机器人底座坐标系R的变换，
Figure FDA0002611308050000032
表示如下：Use transformation matrices
Figure FDA0002611308050000031
Describe the transformation from the calibration plate coordinate system B to the robot base coordinate system R,
Figure FDA0002611308050000032
It is expressed as follows:
Figure FDA0002611308050000033
Figure FDA0002611308050000033
 式(4)中，
Figure FDA0002611308050000034
表示标定板坐标系B到相机坐标系C的变换矩阵，即相机外参，通过相机标定求得；
Figure FDA0002611308050000035
表示末端执行器坐标系T到机器人底座坐标系R的变换矩阵，通过机器人示教器上的参数求得；
Figure FDA0002611308050000036
为需要求解的手眼矩阵；In formula (4),
Figure FDA0002611308050000034
Represents the transformation matrix from the calibration board coordinate system B to the camera coordinate system C, that is, the camera external parameters, obtained through camera calibration;
Figure FDA0002611308050000035
Represents the transformation matrix from the end effector coordinate system T to the robot base coordinate system R, which is obtained through the parameters on the robot teach pendant;
Figure FDA0002611308050000036
is the hand-eye matrix to be solved; 在标定过程中，保持标定板的位置不变，控制机器人从不同位置拍摄标定板的图像，选择其中两个位置进行分析，可得下式(5)：During the calibration process, keep the position of the calibration board unchanged, control the robot to take images of the calibration board from different positions, and select two positions for analysis, the following formula (5) can be obtained:
Figure FDA0002611308050000037
Figure FDA0002611308050000037
 式(5)中，
Figure FDA0002611308050000038
分别表示位置i、位置i+1标定板坐标系B到机器人底座坐标系R的变换矩阵，
Figure FDA0002611308050000039
分别表示位置i、位置i+1末端执行器坐标系T到机器人底座坐标系R的变换矩阵，
Figure FDA00026113080500000310
分别表示位置i、位置i+1需要求解的手眼矩阵，
Figure FDA00026113080500000311
分别表示位置i、位置i+1标定板坐标系B到相机坐标系C的变换矩阵；因为标定板和机器人底座之间的相对位置不变，机器人末端执行器和相机之间的相对位置不变，故有
Figure FDA00026113080500000312
对式(6)联立可得：In formula (5),
Figure FDA0002611308050000038
respectively represent the transformation matrix of position i and position i+1 from the coordinate system B of the calibration board to the coordinate system R of the robot base,
Figure FDA0002611308050000039
respectively represent the transformation matrix of the position i, position i+1 end effector coordinate system T to the robot base coordinate system R,
Figure FDA00026113080500000310
Represents the hand-eye matrix to be solved for position i and position i+1, respectively,
Figure FDA00026113080500000311
Represents the transformation matrix of position i, position i+1 from the calibration board coordinate system B to the camera coordinate system C; because the relative position between the calibration board and the robot base does not change, the relative position between the robot end effector and the camera does not change. , so there is
Figure FDA00026113080500000312
Simultaneously with equation (6), we can get:
Figure FDA00026113080500000313
Figure FDA00026113080500000313
 式(6)中，
Figure FDA00026113080500000314
与
Figure FDA00026113080500000315
均为已知量，最终求解得到
Figure FDA00026113080500000316
即为相机坐标系到机器人末端执行器坐标系的变换矩阵。In formula (6),
Figure FDA00026113080500000314
and
Figure FDA00026113080500000315
are known quantities, and the final solution is
Figure FDA00026113080500000316
It is the transformation matrix from the camera coordinate system to the robot end-effector coordinate system. 6.根据权利要求1所述的基于深度学习的机器人抓取透明物体的方法，其特征在于，所述步骤S3具体实现方法包括：6. The method for robot grasping transparent objects based on deep learning according to claim 1, wherein the specific implementation method of step S3 comprises: S31：利用深度相机扫描抓取透明物体的彩色图像及深度图像；S31: Use a depth camera to scan and capture color images and depth images of transparent objects; S32：对采集的图像进行滤波；S32: filter the collected image; S33：使用ClearGrasp深度学习算法完成透明物体检测与分割；S33: Use ClearGrasp deep learning algorithm to complete transparent object detection and segmentation; S34：使用接触线寻找法对物体进行抓取位置的搜索及评分，在得到最佳抓取位置后进行正确的抓取。S34: Search and score the grasping position of the object by using the contact line finding method, and perform correct grasping after obtaining the best grasping position. 7.根据权利要求6所述的基于深度学习的机器人抓取透明物体的方法，其特征在于，在步骤S32中，选择速度与效果平衡的高斯滤波算法对采集的图像进行滤波，高斯滤波公式如式(7)所示：7. the method for grasping transparent objects based on deep learning according to claim 6, is characterized in that, in step S32, selects the Gaussian filtering algorithm of speed and effect balance to filter the image collected, and Gaussian filtering formula is as follows: Formula (7) shows:
Figure FDA0002611308050000041
Figure FDA0002611308050000041
 式(7)中，f(x,y)表示高斯函数值，x的平方和y的平方分别表示邻域内其他像素与邻域内中心像素的距离，σ表示标准差。In formula (7), f(x, y) represents the Gaussian function value, the square of x and the square of y respectively represent the distance between other pixels in the neighborhood and the central pixel in the neighborhood, and σ represents the standard deviation. 8.根据权利要求6所述的基于深度学习的机器人抓取透明物体的方法，其特征在于，所述步骤S33的具体实现方法包括：8. The method for grasping a transparent object by a robot based on deep learning according to claim 6, wherein the specific implementation method of the step S33 comprises: 采用ClearGrasp深度学习方法从滤波后的图像中预测曲面法线、识别边界、分割透明物体，分割出的掩模将用于修改输入的深度图像；接着利用全局优化算法来重建场景中所有高透明度物体表面的深度，并利用预测的曲面法线来优化3D重建的边缘、遮挡和分割。The ClearGrasp deep learning method is used to predict surface normals, identify boundaries, and segment transparent objects from the filtered image, and the segmented mask will be used to modify the input depth image; then use a global optimization algorithm to reconstruct all high-transparency objects in the scene The depth of the surface and use the predicted surface normals to optimize 3D reconstruction for edges, occlusion and segmentation. 9.根据权利要求6所述的基于深度学习的机器人抓取透明物体的方法，其特征在于，在步骤S33中，ClearGrasp包括3个神经网络，并集合了上述3个神经网络的输出做全局优化；9. the method for grasping transparent objects based on deep learning according to claim 6, is characterized in that, in step S33, ClearGrasp comprises 3 neural networks, and has assembled the output of above-mentioned 3 neural networks to do global optimization ; 3个神经网络包括：透明物体分割网络、边缘识别网络和表面法向量估计网络；3 neural networks including: transparent object segmentation network, edge recognition network and surface normal vector estimation network; 透明物体分割网络：输入单张RGB图片，输出场景中透明物体的像素Mask，即判断每个像素点是属于透明或者非透明物体，在后续优化中会去除被判定为透明物体的像素，得到修改后的深度图；Transparent Object Segmentation Network: Input a single RGB image and output the pixel Mask of the transparent object in the scene, that is, determine whether each pixel belongs to a transparent or non-transparent object, and in the subsequent optimization, the pixels determined as transparent objects will be removed and modified. the depth map after; 边缘识别网络：对于单张RGB图片，输出遮挡边缘和相连边缘信息，这帮助网络更好的分辨图片中不同的边缘，对深度不连续的边缘做出更准确的预测；Edge recognition network: For a single RGB image, output occlusion edge and connected edge information, which helps the network to better distinguish different edges in the image and make more accurate predictions for edges with discontinuous depths; 表面法向量估计：使用RGB图片作为输入，并对输出做L2正则化；Surface normal vector estimation: use an RGB image as input and perform L2 regularization on the output; 利用所述全局优化算法重构透明物体缺失深度区域的三维表面，并利用预测透明物体的表面法向量填充去除的深度区域，同时观察遮挡边缘所显示信息的深度不连续性，用下列公式表示：The three-dimensional surface of the missing depth region of the transparent object is reconstructed by using the global optimization algorithm, and the removed depth region is filled with the surface normal vector of the predicted transparent object. At the same time, the depth discontinuity of the information displayed by the occlusion edge is observed, which is expressed by the following formula: E＝λDED+λSES+λNENB (8)E=λ D E D +λ S E S +λ N E N B (8) 式(8)中，E表示预测深度，ED表示预测深度和观测原始深度的距离，ES表示邻近点的深度差，EN表示预测深度和预测表面法向量的一致性，B表示基于此像素是否遮挡边界，λD、λS、λN表示相关系数。In formula (8), E represents the predicted depth, E D represents the distance between the predicted depth and the observed original depth, ES represents the depth difference between adjacent points, E N represents the consistency between the predicted depth and the predicted surface normal vector, and B represents the Whether the pixel blocks the boundary, λ D , λ S , and λ N represent the correlation coefficient. 10.根据权利要求6所述的基于深度学习的机器人抓取透明物体的方法，其特征在于，在步骤S34中，最佳抓取位置的方向为物体图像梯度的主要方向，对物体的深度图像进行主要位置提取以加快抓取位置的选择速度，即在x轴与y轴分别计算梯度值，并计算每一个像素的梯度方向，并将梯度方向通过直方图进行排列和统计，其中，物体梯度计算及梯度方向计算的方法如下：10. The method for grasping a transparent object based on deep learning according to claim 6, wherein in step S34, the direction of the best grasping position is the main direction of the object image gradient, and the depth image of the object is Extract the main position to speed up the selection of the grabbing position, that is, calculate the gradient value on the x-axis and the y-axis respectively, and calculate the gradient direction of each pixel, and arrange and count the gradient directions through the histogram. Among them, the gradient of the object The method of calculation and gradient direction calculation is as follows: 使用[-1,0,1]和[-1,0,1]T的两个卷积核对图像进行二维卷积来进行物体梯度的计算；Use the two convolution kernels of [-1,0,1] and [-1,0,1] T to perform two-dimensional convolution on the image to calculate the gradient of the object; 对梯度幅值和方向进行计算，公式如下：The gradient magnitude and direction are calculated as follows:
Figure FDA0002611308050000061
Figure FDA0002611308050000061
Figure FDA0002611308050000062
Figure FDA0002611308050000062
 上式中，gx和gy分别表示x和y方向的梯度值，g表示梯度幅值，θ表示梯度方向；In the above formula, g x and g y represent the gradient values in the x and y directions, respectively, g represents the gradient amplitude, and θ represents the gradient direction; 得到梯度之后设定阈值gThresh＝250，只有梯度大于此阈值机器人才有足够的深度放入夹板进行有效抓取，即After obtaining the gradient, set the threshold value g Thresh = 250. Only when the gradient is greater than this threshold value can the robot be placed into the splint with sufficient depth for effective grasping, that is,
Figure FDA0002611308050000063
Figure FDA0002611308050000063
 在机器人抓取透明物体过程中，夹爪与物体接触时有两条接触线，选择合适接触线的条件如下：When the robot grabs a transparent object, there are two contact lines when the gripper is in contact with the object. The conditions for selecting the appropriate contact line are as follows: ①两条接触线的梯度方向基本相反；①The gradient directions of the two contact lines are basically opposite; ②两条接触线之间的距离不超过抓手最大张开距离；②The distance between the two contact lines does not exceed the maximum opening distance of the gripper; ③两条接触线的深度不超过夹爪内部最大深度的1/2；③ The depth of the two contact lines does not exceed 1/2 of the maximum depth inside the jaws; ④两条接触线之间包含的区域中最浅点与接触线最浅点的深度差值不超过夹爪内部深度；④ The depth difference between the shallowest point of the area contained between the two contact lines and the shallowest point of the contact line shall not exceed the internal depth of the gripper; 使用以下公式用于评估一对接触线的抓取可靠性：The following formula was used to evaluate the grasping reliability of a pair of contact lines:
Figure FDA0002611308050000064
Figure FDA0002611308050000064
 式中，G表示抓取可靠性，l1、l2分别表示夹爪和待抓取透明物体的两条接触线的长度，L表示夹爪宽度，
Figure FDA0002611308050000065
用来评估接触线长度，
Figure FDA0002611308050000066
评估两条接触线的长度比，lmax表示接触线中长的那条，lmin代表短的那条，
Figure FDA0002611308050000067
用来评估接触线贴合手爪的程度，dl表示接触线的最浅点，ds表示矩形框区域内最浅点，sinθ用来评估两接触线的错误程度，θ为两接触线中点连线与接触线所夹锐角角度；In the formula, G represents the grasping reliability, l 1 and l 2 respectively represent the lengths of the two contact lines between the gripper and the transparent object to be gripped, L represents the width of the gripper,
Figure FDA0002611308050000065
used to evaluate the contact line length,
Figure FDA0002611308050000066
Evaluate the length ratio of the two contact lines, l max represents the longer one of the contact lines, l min represents the short one,
Figure FDA0002611308050000067
It is used to evaluate the degree to which the contact line fits the paw, d l represents the shallowest point of the contact line, d s represents the shallowest point in the rectangular frame area, sinθ is used to evaluate the error degree of the two contact lines, and θ is the difference between the two contact lines. The acute angle between the point connecting line and the contact line; 通过公式(12)遍历所有的接触线组合，选择得分最高的组合作为最佳抓取位置。All contact line combinations are traversed by formula (12), and the combination with the highest score is selected as the best grasping position.
CN202010755192.2A 2020-07-31 2020-07-31 A method for robot grasping transparent objects based on deep learning Pending CN112045676A (en)

Priority Applications (1)

Application Number Priority Date Filing Date Title
CN202010755192.2A CN112045676A (en) 2020-07-31 2020-07-31 A method for robot grasping transparent objects based on deep learning

Applications Claiming Priority (1)

Application Number Priority Date Filing Date Title
CN202010755192.2A CN112045676A (en) 2020-07-31 2020-07-31 A method for robot grasping transparent objects based on deep learning

Publications (1)

Publication Number Publication Date
CN112045676A true CN112045676A (en) 2020-12-08

Family

ID=73601341

Family Applications (1)

Application Number Title Priority Date Filing Date
CN202010755192.2A Pending CN112045676A (en) 2020-07-31 2020-07-31 A method for robot grasping transparent objects based on deep learning

Country Status (1)

Country Link
CN (1) CN112045676A (en)

Cited By (7)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN112802105A (en) * 2021-02-05 2021-05-14 梅卡曼德（北京）机器人科技有限公司 Object grabbing method and device
CN112809679A (en) * 2021-01-25 2021-05-18 清华大学深圳国际研究生院 Method and device for grabbing deformable object and computer readable storage medium
CN113160313A (en) * 2021-03-03 2021-07-23 广东工业大学 Transparent object grabbing control method and device, terminal and storage medium
CN114049399A (en) * 2022-01-13 2022-02-15 上海景吾智能科技有限公司 Mirror positioning method combining RGBD image
CN114055501A (en) * 2021-11-17 2022-02-18 长春理工大学 A robot grasping system and its control method
CN114627359A (en) * 2020-12-08 2022-06-14 山东新松工业软件研究院股份有限公司 Out-of-order stacked workpiece grabbing priority evaluation method
CN114750164A (en) * 2022-05-25 2022-07-15 清华大学深圳国际研究生院 Transparent object grabbing method and system and computer readable storage medium

Citations (4)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN105014678A (en) * 2015-07-16 2015-11-04 深圳市得意自动化科技有限公司 Robot hand-eye calibration method based on laser range finding
CN108898634A (en) * 2018-07-06 2018-11-27 张显磊 Pinpoint method is carried out to embroidery machine target pinprick based on binocular camera parallax
CN110246193A (en) * 2019-06-20 2019-09-17 南京博蓝奇智能科技有限公司 Industrial robot end camera online calibration method
CN110370286A (en) * 2019-08-13 2019-10-25 西北工业大学 Dead axle motion rigid body spatial position recognition methods based on industrial robot and monocular camera

Patent Citations (4)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN105014678A (en) * 2015-07-16 2015-11-04 深圳市得意自动化科技有限公司 Robot hand-eye calibration method based on laser range finding
CN108898634A (en) * 2018-07-06 2018-11-27 张显磊 Pinpoint method is carried out to embroidery machine target pinprick based on binocular camera parallax
CN110246193A (en) * 2019-06-20 2019-09-17 南京博蓝奇智能科技有限公司 Industrial robot end camera online calibration method
CN110370286A (en) * 2019-08-13 2019-10-25 西北工业大学 Dead axle motion rigid body spatial position recognition methods based on industrial robot and monocular camera

Non-Patent Citations (3)

* Cited by examiner, † Cited by third party
Title
SHREEYAK S. SAJJAN等: "《ClearGrasp:3D Shape Estimation of Transparent Objects for Manipulation》", 《ARXIV:1910.02550V2 [CS.CV] 14 OCT 2019》 *
卢岸潇: "《基于双目立体视觉的工件定位技术研究》", 《信息科技辑》 *
王俊义: "《基于视觉伺服的机器人抓取关键技术研究》", 《信息科技辑》 *

Cited By (8)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN114627359A (en) * 2020-12-08 2022-06-14 山东新松工业软件研究院股份有限公司 Out-of-order stacked workpiece grabbing priority evaluation method
CN112809679A (en) * 2021-01-25 2021-05-18 清华大学深圳国际研究生院 Method and device for grabbing deformable object and computer readable storage medium
CN112802105A (en) * 2021-02-05 2021-05-14 梅卡曼德（北京）机器人科技有限公司 Object grabbing method and device
CN113160313A (en) * 2021-03-03 2021-07-23 广东工业大学 Transparent object grabbing control method and device, terminal and storage medium
CN114055501A (en) * 2021-11-17 2022-02-18 长春理工大学 A robot grasping system and its control method
CN114049399A (en) * 2022-01-13 2022-02-15 上海景吾智能科技有限公司 Mirror positioning method combining RGBD image
CN114049399B (en) * 2022-01-13 2022-04-12 上海景吾智能科技有限公司 Mirror positioning method combining RGBD image
CN114750164A (en) * 2022-05-25 2022-07-15 清华大学深圳国际研究生院 Transparent object grabbing method and system and computer readable storage medium

Similar Documents

Publication Publication Date Title
CN112045676A (en) A method for robot grasping transparent objects based on deep learning
CN109255813B (en) Man-machine cooperation oriented hand-held object pose real-time detection method
CN112270249B (en) Target pose estimation method integrating RGB-D visual characteristics
CN113409384B (en) Pose estimation method and system of target object and robot
CN115147723B (en) Method, system, medium, equipment and terminal for identification and ranging of inland ships
CN108955718B (en) Visual odometer and positioning method thereof, robot and storage medium
CN113850865A (en) Human body posture positioning method and system based on binocular vision and storage medium
US20150317821A1 (en) Geodesic Distance Based Primitive Segmentation and Fitting for 3D Modeling of Non-Rigid Objects from 2D Images
CN103903237B (en) Sonar image sequence assembly method is swept before one kind
CN115272271A (en) Pipeline defect detecting and positioning ranging system based on binocular stereo vision
CN111996883B (en) Method for detecting width of road surface
CN104036522A (en) Water flowing speed monitoring method based on moving target identification in videos
CN115816460B (en) Mechanical arm grabbing method based on deep learning target detection and image segmentation
WO2014044126A1 (en) Coordinate acquisition device, system and method for real-time 3d reconstruction, and stereoscopic interactive device
CN114029946A (en) Method, device and equipment for guiding robot to position and grab based on 3D grating
CN113393439A (en) Forging defect detection method based on deep learning
WO2023070312A1 (en) Image processing method
CN101819024A (en) Machine vision-based two-dimensional displacement detection method
JP2018169660A (en) Object attitude detection apparatus, control apparatus, robot and robot system
CN113701750A (en) Fusion positioning system of underground multi-sensor
CN110807799B (en) A Line Feature Visual Odometry Method Combined with Depth Map Inference
CN115035546B (en) Three-dimensional human body posture detection method and device and electronic equipment
CN108388854A (en) A kind of localization method based on improvement FAST-SURF algorithms
CN116152697A (en) Three-dimensional model measuring method and related device for concrete structure cracks
CN110514140B (en) A three-dimensional imaging method, device, equipment and storage medium

Legal Events

Date Code Title Description
PB01 Publication
PB01 Publication
SE01 Entry into force of request for substantive examination
SE01 Entry into force of request for substantive examination