CN117619769A - Multi-category stacked workpiece mechanical arm sorting method based on point cloud and deep learning - Google Patents

Abstract

The invention relates to a multi-category stacked workpiece mechanical arm sorting method based on point cloud and deep learning, which sorts multi-category stacked workpieces based on a mechanical arm sorting system and comprises the following steps: acquiring point cloud data of each type of single workpiece under different poses by using a three-dimensional depth camera, and preprocessing to obtain template point cloud and original point cloud data; training a network model of a point cloud classification network based on deep learning based on original point cloud data; acquiring, preprocessing and dividing point clouds of actual multi-category stacked workpieces by using a three-dimensional depth camera, and classifying the point clouds of each single workpiece by using a point cloud classification network; registering the point clouds of each single workpiece and the point clouds of the predicted category template, and determining the point clouds of the workpiece to be grabbed based on the total degree of coincidence corresponding to the point clouds of each single workpiece; and establishing and calibrating a local coordinate system of the point cloud of the workpiece to be grabbed, determining the space pose of the point cloud, and guiding the mechanical arm to accurately sort the point cloud. The method is beneficial to improving the accuracy of the mechanical arm in sorting the multi-category stacked workpieces.

Description

Multi-category stacked workpiece mechanical arm sorting method based on point cloud and deep learning

Technical Field

The invention relates to the field of industrial mechanical arms, in particular to a multi-category stacked workpiece mechanical arm sorting method based on point cloud and deep learning.

Background

Along with the continuous upgrading of modern industrial systems, industrial mechanical arms have been widely applied to the industrial production fields of product sorting, carrying, stacking and the like by virtue of the advantages of high production efficiency, low post production cost and the like. In order to realize batch and streamline sorting in early stage, people often adopt a traditional mechanical arm sorting method, namely workpieces to be sorted are firstly conveyed to designated positions of a production line one by one, and then the mechanical arm is directly controlled to finish preset sorting actions through identification of specific identifiers of various workpieces. However, the above-described over-procedural sorting operations are in fact difficult to address workpiece sorting problems in the multi-category workpiece stacking scenario in some specific production environments in the future.

In order to realize intelligent sorting of the mechanical arm on the multi-class workpieces in the stacking scene, not only are the workpieces in the scene accurately sorted, but also the actual pose of the workpiece which is most suitable for grabbing is determined at the same time, so that the mechanical arm is guided to sort. At present, a pure two-dimensional image visual classification method adopted by the main stream in the industrial production field can only obtain the plane contour and position information of a workpiece, but cannot obtain the depth information of the workpiece; in other words, the classification of the workpieces can be realized by simply starting from the two-dimensional image, but the spatial pose information of the workpieces is often difficult to accurately obtain. Compared with a two-dimensional image, the three-dimensional point cloud data can better and more intuitively describe the actual pose information of a space object, and can also provide more abundant object surface geometric information. In the field of industrial production, three-dimensional point clouds have been gradually used in the fields of reverse modeling, classification, accurate positioning, defect detection and the like of various industrial products; however, the three-dimensional point cloud data structure has the characteristics of disorder, unstructured and the like, so that the problem of classifying the workpiece point clouds in a multi-category workpiece stacking scene is quite challenging. Therefore, part of researches have combined the two-dimensional image recognition technology and the three-dimensional point cloud technology, firstly, the two-dimensional image is utilized to recognize the outline of the workpiece, and then the two-dimensional recognition result is matched into the three-dimensional point cloud to intercept the corresponding workpiece point cloud, but the method is easy to generate the problems of conversion errors among data with different dimensions, loss of the information of the workpiece point cloud and the like.

Disclosure of Invention

The invention aims to provide a multi-category stacked workpiece mechanical arm sorting method based on point cloud and deep learning, which is beneficial to improving the accuracy of sorting multi-category stacked workpieces by the mechanical arm.

In order to achieve the above purpose, the technical scheme adopted by the invention is as follows: the multi-category stacked workpiece mechanical arm sorting method based on point cloud and deep learning provides a mechanical arm sorting system mainly composed of a mechanical arm, a three-dimensional depth camera and a computer, wherein the computer is respectively connected with the mechanical arm and the three-dimensional depth camera and is in bidirectional communication; sorting the multi-category stacked workpieces based on the mechanical arm sorting system, comprising the following steps:

(1) Respectively acquiring multiple groups of complete point cloud data of each single workpiece to be sorted under different placement postures by using a three-dimensional depth camera, and carrying out point cloud preprocessing on the complete point cloud data to obtain template point clouds of each single workpiece and original point cloud data for subsequent point cloud classification network training;

(2) Performing network model training on the point cloud classification network based on deep learning based on the original point cloud data of each single workpiece to obtain a trained point cloud classification network;

(3) Collecting multi-category stacked workpiece point clouds in an actual scene by using a three-dimensional depth camera, and preprocessing and dividing the collected multi-category stacked workpiece point clouds to obtain a plurality of divided single workpiece point clouds;

(4) Classifying the point clouds of each segmented single workpiece by using a trained point cloud classification network to obtain corresponding class labels;

(5) On the basis of registering each single workpiece point cloud with the predicted category template point cloud, calculating the total degree of coincidence corresponding to each single workpiece point cloud based on a bidirectional octree searching method, and taking the workpiece point cloud with the highest total degree of coincidence as the workpiece point cloud to be grabbed of the mechanical arm;

(6) And establishing and calibrating a local coordinate system of the point cloud of the workpiece to be grabbed when each wheel of the mechanical arm grabs, and determining the space pose of the point cloud of the workpiece to be grabbed so as to guide the mechanical arm to accurately sort the point cloud of the workpiece to be grabbed.

Further, the implementation method of the step (1) is as follows:

respectively acquiring multiple groups of complete point cloud data of each single workpiece to be sorted under different placement postures by using a three-dimensional depth camera, wherein the acquired point cloud data of each single workpiece at least comprises (x, y, z) three-dimensional information; and then, carrying out point cloud preprocessing comprising noise reduction and outlier removal on the acquired point cloud data of each single workpiece, and determining template point clouds of each single workpiece and original point cloud data for subsequent point cloud classification network training.

Further, in order to reduce the data processing amount of the subsequent point cloud classification network, on the premise of ensuring that the original structural features of the workpiece are not lost, the acquired single workpiece point clouds are subjected to uniform downsampling processing and then input into the point cloud classification network based on deep learning for network model training.

Further, the implementation method of the step (2) is as follows:

in the point cloud classification network, firstly, the information dimension of the workpiece point cloud is properly expanded by utilizing a multi-layer perceptron, workpiece characteristic information contained in the point cloud is projected to a higher dimension step by step, and the internal weight structures of the multi-layer perceptron are the same and the parameters are shared; secondly, outputting the integral characteristic parameters of the workpiece point cloud information after the processing of the maximum pooling layer; finally, classifying the pooled integral characteristic parameters through a multi-layer perceptron, and further determining the corresponding category of the workpiece point cloud; and training a network model by utilizing the original point cloud data of each single workpiece to obtain the optimal network parameters of each layer of the point cloud classification network, thereby obtaining the neural network model for the point cloud classification of the workpiece.

Further, the implementation method of the step (3) comprises the following steps:

Acquiring multi-category stacked workpiece point clouds in an actual scene by using a three-dimensional depth camera; background filtering is carried out on the point clouds of the multi-category stacked workpieces, and then downsampling treatment is carried out on the point clouds subjected to background filtering; in order to remove useless noise points in the point cloud caused by the camera or light interference, denoising the point cloud after the downsampling process by adopting an outlier removal method based on Euclidean distance;

performing curvature calculation on the preprocessed point cloud, selecting a point with the minimum curvature as an initial segmentation point, and calculating normals of other points in a neighborhood range of the initial segmentation point; presetting a smoothing threshold, and judging that the neighborhood point and the segmentation point belong to the same segmentation body if the calculated included angle between the normal line at the neighborhood point and the normal line at the segmentation point is smaller than the set smoothing threshold; otherwise, judging that the neighborhood point and the segmentation point do not belong to the same segmentation body; then, presetting a curvature threshold value, if the curvature of the neighborhood points belonging to the same division body is smaller than the set curvature threshold value, deleting the current division points and regarding the neighborhood points as new division points to perform next growth search until the new division points cannot be found, and completing division of a single workpiece point cloud; repeating the above dividing process to obtain the point clouds of other single workpieces after dividing.

Further, because the stacked workpieces are shielded from each other, a plurality of incomplete small single workpiece point clouds exist after the split; setting a point threshold of the single workpiece point cloud, filtering small workpiece point clouds with the point smaller than the point threshold, and only reserving large single workpiece point clouds with more points.

Further, the implementation method of the step (4) is as follows:

in order to reduce the data processing amount of the point cloud classification network, on the premise of ensuring that the structural characteristics of the point cloud of the workpiece are not lost, the single workpiece point clouds screened in the step (3) are subjected to uniform downsampling, and the downsampled single workpiece point clouds are input into the trained point cloud classification network to obtain the prediction category of each single workpiece.

Further, the implementation method of the step (5) is as follows:

registering the point clouds of each single workpiece and the corresponding prediction category template point clouds: firstly, establishing a point pair relation between points in a point cloud and neighborhood points thereof by adopting a rapid point characteristic histogram method, and searching for point pairs with similar characteristic values in a template point cloud and a single workpiece point cloud to perform rough matching; secondly, setting a distance error threshold value, calculating the distance error between each monomer workpiece point cloud and the nearest point pair of the template point cloud after rough matching by adopting an iterative nearest point method, and finishing the precise matching between the template point cloud and the monomer workpiece point cloud when the distance error is smaller than the threshold value;

After the point clouds of each single workpiece are matched with the corresponding prediction type template point clouds, a bidirectional octree searching method is adopted to calculate the positive and negative double overlapping ratios, the sum of the two overlapping ratios is used as a final judgment standard, namely, the single workpiece point cloud with the highest overlapping ratio is the point cloud of the single workpiece with correct prediction type, no shielding and the most complete point cloud information, and the single workpiece point cloud is regarded as the point cloud of the workpiece to be grabbed, which is preferably grabbed by the mechanical arm.

Further, the single workpiece point clouds obtained by segmentation are sharedMethod for searching by utilizing two-way octreeCalculate the firstThe specific method for the overlap ratio of the single workpieces is as follows:

forward octree search: establishing an octree grid aiming at the gamma monomer workpiece point cloud, and setting a proper grid size; the total point number of the point cloud of the corresponding category template is recorded asInitializing the coincidence point->Traversing a point in a template point cloud +.>Inquiring whether points in the point cloud of the workpiece to be grabbed are contained in the octree grid by coordinates corresponding to the point i, if so, regarding the point i as a coincidence point, and updating the coincidence point +.>If not, the coincidence point number is not updated>Wherein (1)>Represents the number of coincidence points obtained after querying point i, and +.>After the traversal is completed, the final obtained coincidence point number is utilized >Calculating the coincidence ratio between the single workpiece point cloud and the corresponding category template point cloud:

reverse octree search: establishing octree grids aiming at corresponding category template point clouds of the gamma monomer workpieces, and setting proper grid sizes; recording the total point number of the point cloud of the gamma-th monomer workpiece asInitializing the coincidence point->Traversing points in a single workpiece point cloud +.>Inquiring whether the octree grid contains the points in the template point cloud or not by the coordinates corresponding to the points j, if so, taking the points j as coincident points, and updating the coincident points +.>If not, the coincidence point number is not updated>Wherein (1)>Represents the number of coincidence points obtained after inquiring the point j, andafter the traversal is completed, the final obtained coincidence point number is utilized>Calculating the coincidence ratio between the template point cloud and the single workpiece point cloud:

then, the total degree of coincidence corresponding to the gamma-th monomer work-piece point cloud is:

and comparing the total degree of coincidence corresponding to the point clouds of the single workpieces, and taking the point cloud with the highest total degree of coincidence as the point cloud of the workpiece to be grabbed.

Further, the implementation method of the step (6) comprises the following steps:

in order to guide the mechanical arm to grab the workpiece, the mass center of the point cloud of the workpiece to be grabbed is required to be determined, and a local coordinate system of the point cloud of the workpiece to be grabbed is established by taking the mass center as an origin; let o _w Representing the point cloud centroid of the workpiece to be grabbed, and solving the position coordinates of the point cloud centroid by the following formula:

wherein, (x) _k ,y _k ,z _k ) Representing three-dimensional coordinates of points k (k=1, 2, …, N) of the point cloud of the workpiece to be grabbed under camera coordinates; n is the total point number contained in the point cloud of the workpiece to be grabbed;

at centroid o _w Calculating the symmetry and positive covariance matrix of the workpiece point cloud:

wherein,

solving covariance matrix C _w Is a three characteristic value lambda of (2) ₁ 、λ ₂ 、λ ₃ Corresponding to three unit feature vectors And combine these three featuresThe straight lines where the vectors are located are respectively defined as x of a local coordinate system of the point cloud of the workpiece to be grabbed _w Axis, y _w Axis, z _w The shaft is used for simultaneously initially determining the directions of the three feature vectors as the positive directions of three coordinate axes of a workpiece point cloud local coordinate system;

considering that workpieces to be grabbed are different when each wheel of the mechanical arm grabs, the corresponding point cloud local coordinate system z of the workpieces to be grabbed _w The axes are also not oriented the same, if the robot arm is gripping the workpiece, the end effector coordinate system o is adopted _t -x _t y _t z _t Z of (2) _t Shaft and workpiece point cloud local coordinate system o to be grasped _w -x _w y _w z _w Z of (2) _w The operation mode of coaxially approaching to grabbing is carried out after the shaft alignment;

in order to ensure the consistency of the grabbing operation of the mechanical arm, the local coordinate system z of the point cloud of the workpiece to be grabbed during each grabbing of the mechanical arm is needed _w The forward direction of the shaft is corrected timely to unify z _w The forward direction of the shaft; for this purpose, in the camera coordinate system o _c -x _c y _c z _c Z of (2) _c Taking the axial forward direction as a reference, and taking the point cloud local coordinate system z of the workpiece to be grabbed when each wheel of the mechanical arm grabs _w Correcting the axial forward direction; is provided withFor camera coordinate system z _c Unit vector of axis, if->Indicating a local coordinate system z of point cloud of a workpiece to be grabbed _w Axis and camera coordinate system z _c The axes are oriented approximately the same, and a predetermined local coordinate system x of the point cloud of the workpiece is maintained _w Axis, y _w Axis, z _w The axial direction is unchanged; if->Then the local coordinate system z of the point cloud of the workpiece to be grabbed is indicated _w Axis and camera coordinate system z _c The axes are oriented approximately opposite, in which case the workpiece to be grasped should be held in the local coordinate system z of the point cloud _w Forward direction of shaftCorrected to z _w The axis is initially oriented in the opposite direction, and the local coordinate system y of the point cloud of the workpiece is maintained _w Under the condition that the axial forward direction is unchanged, the local coordinate system x of the point cloud of the workpiece to be grabbed is re-corrected according to the right-hand rule of the coordinate system _w The forward direction of the axis, namely the local coordinate system x of the point cloud of the workpiece to be grabbed _w The positive correction of the axis is x _w The shaft is initially oriented in the opposite direction;

after the correction is finished, a point cloud local coordinate system o of the workpiece to be grabbed is obtained _w -x _w y _w z _w Relative to camera coordinate system o _c -x _c y _c z _c Is a rotation matrix R of (2) ₁ ∈R ^3×3 And combined with a local coordinate system o _w -x _w y _w z _w In the camera coordinate system o _c -x _c y _c z _c Position vector T in (a) ₁ ∈R ^3×1 Obtaining a homogeneous transformation matrix between a local coordinate system of the workpiece to be grabbed and a camera coordinate system:

wherein 0 is _1×3 A zero vector representing 1 row and 3 columns;

by means of cone tip calibration method, the position of the standard cone on the mechanical arm workbench is adjusted for multiple times, and the coordinate system origin o of the mechanical arm end operator is controlled _t Aligned with the cone tips to obtain multiple groups of cone tips respectively in a camera coordinate system o _c -x _c y _c z _c And a mechanical arm coordinate system o _r -x _r y _r z _r The space coordinates below and further the rotation matrix R of the camera coordinate system relative to the robot arm coordinate system are obtained ₂ ∈R ^3×3 The method comprises the steps of carrying out a first treatment on the surface of the Position vector T in mechanical arm coordinate system combined with camera coordinate system ₂ ∈R ^3×1 Obtaining a homogeneous transformation matrix between a camera coordinate system and a mechanical arm coordinate system:

then, the object point cloud local coordinate system o to be grasped _w -x _w y _w z _w Relative to the arm coordinate system o _r -x _r y _r z _r The homogeneous transformation matrix of (c) is finally expressed as:

M＝M ₂ ·M ₁

and calculating the space pose of the workpiece to be grabbed by combining the point cloud information of the workpiece to be grabbed and the homogeneous transformation matrix M, and further guiding the mechanical arm to carry out sorting operation on the workpiece to be grabbed.

Compared with the prior art, the invention has the following beneficial effects: the method directly starts from the three-dimensional point cloud, effectively segments the multi-category stacked workpiece point cloud in the original scene by adopting a proper point cloud processing method, classifies each segmented single workpiece point cloud by utilizing a deep learning technology, ensures the integrity of the space point cloud information of each workpiece to the greatest extent, and avoids the problems of data conversion errors, workpiece point cloud information loss and the like possibly generated by the traditional two-dimensional image classification and three-dimensional point cloud interception method; in order to effectively filter incomplete workpiece point clouds which are caused by information deficiency and do not have grabbing conditions and workpiece point clouds with wrong class prediction, the invention provides a method for registering each single workpiece point cloud and each predicted class standard template point cloud, calculating the total combination degree corresponding to each single workpiece point cloud based on a bidirectional octree searching method according to the registered result, and taking the workpiece point cloud with the highest total combination degree as the workpiece point cloud to be grabbed, wherein the point cloud information is the most complete and the predicted class is correct. In addition, in order to ensure the consistency of the grabbing operation of the mechanical arm, the invention actively corrects the point cloud local coordinate system of the workpiece to be grabbed by each wheel of the mechanical arm, and calculates the actual pose of the workpiece to be grabbed on the basis of the point cloud local coordinate system, so as to guide the mechanical arm to accurately sort the workpiece to be grabbed.

Drawings

FIG. 1 is a schematic diagram of a robotic arm sorting system according to an embodiment of the present invention;

in the figure: 1. a three-dimensional depth camera; 2. an industrial robot; 3. stacking the workpieces in multiple categories; 4. a work table; 5. a computer;

FIG. 2 is a flow chart of a method implementation of an embodiment of the present invention;

fig. 3 is a schematic structural diagram of a point cloud classification network based on deep learning in an embodiment of the present invention;

FIG. 4 is a schematic diagram of correcting a local coordinate system of a point cloud of a workpiece to be grasped according to an embodiment of the invention; wherein, (1) is an initially determined local coordinate system of the workpiece to be grabbed, and (2) is a corrected local coordinate system of the workpiece to be grabbed;

fig. 5 is a schematic diagram of calculating the coincidence degree based on the point cloud registration and the bidirectional octree searching method in the embodiment of the invention.

Detailed Description

The invention will be further described with reference to the accompanying drawings and examples.

It should be noted that the following detailed description is exemplary and is intended to provide further explanation of the present application. Unless defined otherwise, all technical and scientific terms used herein have the same meaning as commonly understood by one of ordinary skill in the art to which this application belongs.

It is noted that the terminology used herein is for the purpose of describing particular embodiments only and is not intended to be limiting of example embodiments in accordance with the present application. As used herein, the singular is also intended to include the plural unless the context clearly indicates otherwise, and furthermore, it is to be understood that the terms "comprises" and/or "comprising" when used in this specification are taken to specify the presence of stated features, steps, operations, devices, components, and/or combinations thereof.

The embodiment provides a multi-category stacked workpiece mechanical arm sorting method based on point cloud and deep learning, which firstly provides a mechanical arm sorting system and then sorts the multi-category stacked workpieces based on the mechanical arm sorting system.

As shown in fig. 1, the mechanical arm sorting system is composed of a three-dimensional depth camera 1, an industrial mechanical arm 2, a workbench 4, a computer 5 and the like. The computer 5 is connected with the industrial mechanical arm 2 and the three-dimensional depth camera 1 respectively and communicates bidirectionally. The multi-category stacked workpieces 3 may be placed on a table 4.

As shown in fig. 2, the sorting method of the multi-category stacked workpiece mechanical arm comprises the following steps:

(1) And respectively acquiring multiple groups of complete point cloud data of each single workpiece to be sorted under different placement postures by using a three-dimensional depth camera, and carrying out point cloud preprocessing on the complete point cloud data to obtain template point clouds of each single workpiece and original point cloud data for subsequent point cloud classification network training.

(2) And training a network model of the point cloud classification network based on deep learning based on the original point cloud data of each single workpiece to obtain a trained point cloud classification network.

(3) And acquiring multi-category stacked workpiece point clouds in an actual scene by using a three-dimensional depth camera, and preprocessing and dividing the acquired multi-category stacked workpiece point clouds to obtain a plurality of divided single workpiece point clouds.

(4) And classifying the point clouds of each segmented single workpiece by using a trained point cloud classification network to obtain corresponding class labels.

(5) On the basis of registering each single workpiece point cloud with the predicted category template point cloud, calculating the total degree of coincidence corresponding to each single workpiece point cloud based on a bidirectional octree searching method, and taking the workpiece point cloud with the highest total degree of coincidence as the workpiece point cloud to be grabbed of the mechanical arm.

(6) And establishing and calibrating a local coordinate system of the point cloud of the workpiece to be grabbed when each wheel of the mechanical arm grabs, and determining the space pose of the point cloud of the workpiece to be grabbed so as to guide the mechanical arm to accurately sort the point cloud of the workpiece to be grabbed.

The relevant matters related to the method are further detailed below.

S1: in the invention, a three-dimensional depth camera is utilized to respectively collect multiple groups of complete point cloud data of each type of single workpiece under different placing postures of the system, and the collected point cloud data of each type of single workpiece at least comprises (x, y, z) three dimensional information; and then, carrying out noise reduction, outlier removal and other point cloud preprocessing on the acquired point cloud data of each single workpiece, and determining template point clouds of each single workpiece and original point cloud data for subsequent point cloud classification network training.

In order to reduce the data processing amount of the subsequent point cloud classification network, on the premise of ensuring that the original structural features of the workpiece are not lost, the acquired single workpiece point clouds are subjected to uniform downsampling processing and then input into the point cloud classification network based on deep learning as shown in fig. 3 for network model training.

In the point cloud classification network, firstly, the information dimension of the workpiece point cloud is properly expanded by utilizing a multi-layer perceptron, workpiece characteristic information contained in the point cloud is projected to a higher dimension step by step, and the internal weight structures of the multi-layer perceptron are the same and the parameters are shared; secondly, outputting the integral characteristic parameters of the workpiece point cloud information after the processing of the maximum pooling layer; and finally, classifying the pooled integral characteristic parameters through a multi-layer perceptron, and further determining the corresponding category of the workpiece point cloud. And performing network model training on the point cloud classification network by utilizing the original point cloud data of the single workpieces of each class, so that the optimal network parameters of each layer of the point cloud classification network can be obtained, and further, a neural network model for the point cloud classification of the workpieces can be obtained.

S2: acquiring multi-category stacked workpiece point clouds in an actual scene by using a three-dimensional depth camera; and carrying out background filtering on the point cloud of the multi-category stacked workpieces, and then carrying out downsampling treatment on the workpiece point cloud subjected to background filtering. In order to remove useless noise points caused by camera self or light interference and the like in the point cloud of the workpiece, an outlier removal method based on Euclidean distance is adopted to remove noise of the point cloud.

S3: performing curvature calculation on the point cloud, selecting a point with the minimum curvature as an initial segmentation point, and calculating normals of other points in a neighborhood range of the initial segmentation point; by presetting a smoothing threshold, if the calculated normal angle between the neighborhood point normal and the segmentation point normal is smaller than a given smoothing threshold, judging that the neighborhood point and the segmentation point belong to the same segmentation body; otherwise, judging that the neighborhood point and the segmentation point do not belong to the same segmentation body. And then presetting a curvature threshold value, deleting the current segmentation point and regarding the neighborhood points as new segmentation points to perform next growth search if the curvature of the neighborhood points belonging to the same segmentation body is smaller than a given curvature threshold value, and completing segmentation of a single workpiece point cloud until the new segmentation points cannot be found. Repeating the growth and segmentation process to obtain the point cloud of other segmented single workpieces.

Because the stacked workpieces are shielded from each other, a plurality of incomplete small single workpiece point clouds are likely to exist after the split; and filtering small workpiece point clouds with the points smaller than the point threshold by setting the point threshold of the single workpiece point clouds, and only retaining large single workpiece point clouds with more points.

S4: in order to reduce the data processing amount of the point cloud classification network, on the premise of ensuring that the structural characteristics of the point cloud of the workpiece are not lost, the screened single workpiece point clouds are subjected to uniform downsampling, and the downsampled single workpiece point clouds are input into the trained point cloud classification network to obtain the prediction category of each single workpiece.

S5: registering the point clouds of each single workpiece and the corresponding prediction category template point clouds. Firstly, establishing a point-to-point relation between points in a point cloud and neighborhood points thereof by adopting a Fast Point Feature Histogram (FPFH) method, and searching for point pairs with similar feature values in a template point cloud and a single workpiece point cloud for rough matching; and secondly, setting a distance error threshold value, calculating the distance error between each monomer workpiece point cloud and the nearest point pair of the template point cloud after rough matching by adopting an iterative nearest point (ICP) method, and finishing the fine matching between the template point cloud and the monomer workpiece point cloud when the distance error is smaller than the threshold value.

S6: among the plurality of single workpiece point clouds obtained by the division, there are some large residual point clouds which are actually remained due to the mutual shielding between the workpieces, and the workpieces corresponding to the residual point clouds may not actually have the grabbing condition. Therefore, it should be considered that on the basis of performing coarse and fine matching between each single workpiece point cloud and the template point cloud, the point cloud octree searching method is used to further calculate the contact ratio between each single workpiece point cloud and the template point cloud, and the single workpiece point cloud with the highest contact ratio can be regarded as the point cloud to be grabbed, which is not shielded and has the most complete point cloud information. However, considering that some similar shape features may exist between each part of the workpieces in the stacked scene, if the remaining residual point clouds just originate from the similar parts between the workpieces, when the point cloud classification network is used for classifying and predicting the workpieces, an erroneous classification result is likely to be obtained. At this time, after the single workpiece point cloud and the template point cloud with the misprediction are registered, if the single octree searching method is only adopted to calculate the contact ratio between the single workpiece point cloud and the template point cloud with the misprediction, the situation of excessively high contact ratio is likely to occur, and the judgment result is further affected. In order to solve the problem, as shown in fig. 5, after the single workpiece point clouds are matched with the corresponding prediction category template point clouds in thickness and in fineness, a bidirectional octree search method is adopted to perform forward and backward two-time overlap ratio calculation, the sum of the two-time overlap ratio is used as a final judgment standard, namely, the single workpiece point cloud with the highest overlap ratio, namely, the single workpiece point cloud with the correct prediction category, no shielding and the most complete point cloud information, is finally regarded as the point cloud to be grabbed of the workpiece to be grabbed, which is preferably grabbed by the mechanical arm.

The single workpiece point cloud obtained by the division is provided to be sharedCalculating the eighth by using the bidirectional octree searching methodThe specific process of the overlap ratio of the single workpieces is as follows:

forward octree search: establishing an octree grid aiming at the gamma monomer workpiece point cloud, and setting a proper grid size; the total point number of the point cloud of the corresponding category template is recorded asInitializing the coincidence point->Traversing a point in a template point cloud +.>Inquiring whether points in the point cloud of the workpiece to be grabbed are contained in the octree grid by coordinates corresponding to the point i, if so, regarding the point i as a coincidence point, and updating the coincidence point +.>If not, the coincidence point number is not updated>Wherein (1)>Represents the number of coincidence points obtained after querying point i, and +.>After the traversal is completed, the final obtained coincidence point number is utilized>Calculating the coincidence ratio between the single workpiece point cloud and the corresponding category template point cloud:

reverse octree search: establishing octree grids aiming at corresponding category template point clouds of the gamma monomer workpieces, and setting proper grid sizes; recording the total point number of the point cloud of the gamma-th monomer workpiece asInitializing the coincidence point->Traversing points in a single workpiece point cloud +.>And inquiring whether the octree grid is in the octree grid by the coordinates corresponding to the point j If the point in the template point cloud is not contained, the point j is regarded as a coincidence point, and the coincidence point is updated>If not, the coincidence point number is not updated>Wherein (1)>Represents the number of coincidence points obtained after inquiring the point j, andafter the traversal is completed, the final obtained coincidence point number is utilized>Calculating the coincidence ratio between the template point cloud and the single workpiece point cloud:

then, the total degree of coincidence corresponding to the gamma-th monomer work-piece point cloud is:

and comparing the total degree of coincidence corresponding to the point clouds of the single workpieces, and taking the point cloud with the highest total degree of coincidence as the point cloud of the workpiece to be grabbed.

S7: in order to guide the mechanical arm to grab the workpiece, the mass center of the point cloud of the workpiece to be grabbed is determined, and a local coordinate system of the point cloud of the workpiece to be grabbed is established by taking the mass center as an origin. Let o _w The position coordinates of the point cloud centroid representing the workpiece to be grasped can be solved by the following formula:

wherein, (x) _k ,y _k ,z _k ) Representing three-dimensional coordinates of points k (k=1, 2, …, N) of the point cloud of the workpiece to be grabbed under camera coordinates; n is the total point number contained in the point cloud of the workpiece to be grabbed.

At centroid o _w Calculating the symmetry and positive covariance matrix of the workpiece point cloud:

wherein,

solving covariance matrix C _w Is a three characteristic value lambda of (2) ₁ 、λ ₂ 、λ ₃ Corresponding to three unit feature vectors Respectively defining the straight lines of the three feature vectors as x of a local coordinate system of the point cloud of the workpiece to be grabbed _w Axis, y _w Axis, z _w And the shaft simultaneously initially determines the directions of the three feature vectors as the positive directions of three coordinate axes of the workpiece point cloud local coordinate system.

Considering that workpieces to be grabbed are different when each wheel of the mechanical arm grabs, the corresponding point cloud local coordinate system z of the workpieces to be grabbed _w The axes are also not oriented the same, if the robot arm is gripping the workpiece, the end effector coordinate system o is adopted _t -x _t y _t z _t Z of (2) _t Shaft and workpiece point cloud local coordinate system o to be grasped _w -x _w y _w z _w Z of (2) _w The operation mode of coaxially approaching to grabbing is carried out after the shaft alignment; in order to ensure the consistency of the grabbing operation of the mechanical arm, the local coordinate system z of the point cloud of the workpiece to be grabbed during each grabbing of the mechanical arm is needed _w The forward direction of the shaft is corrected timely to unify z _w The axis is oriented in the forward direction.

For this purpose, as shown in FIG. 4, in a camera coordinate system o _c -x _c y _c z _c Z of (2) _c Taking the axial forward direction as a reference, and taking the point cloud local coordinate system z of the workpiece to be grabbed when each wheel of the mechanical arm grabs _w The axis is corrected in the forward direction. Is provided withFor camera coordinate system z _c Unit vector of axis, ifIndicating a local coordinate system z of point cloud of a workpiece to be grabbed _w Axis and camera coordinate system z _c The axes are oriented approximately the same, and a predetermined local coordinate system x of the point cloud of the workpiece is maintained _w Axis, y _w Axis, z _w The axial direction is unchanged; if->Then the local coordinate system z of the point cloud of the workpiece to be grabbed is indicated _w Axis and camera coordinate system z _c The axes are oriented approximately opposite, in which case the workpiece to be grasped should be held in the local coordinate system z of the point cloud _w The positive correction of the axis is z _w The axis is initially oriented in the opposite direction, and the local coordinate system y of the point cloud of the workpiece is maintained _w Under the condition that the axial forward direction is unchanged, the local coordinate system x of the point cloud of the workpiece to be grabbed is re-corrected according to the right-hand rule of the coordinate system _w Forward direction of axis (i.e. local coordinate system x of point cloud of workpiece to be grabbed _w The positive correction of the axis is x _w The axis is initially oriented in the opposite direction).

S8: after the correction is finished, the local coordinate system o of the point cloud of the workpiece to be grabbed can be obtained _w -x _w y _w z _w Relative to camera coordinate system o _c -x _c y _c z _c Is a rotation matrix R of (2) ₁ ∈R ^3×3 And combined with a local coordinate system o _w -x _w y _w z _w In the camera coordinate system o _c -x _c y _c z _c Position vector T in (a) ₁ ∈R ^3×1 Obtaining a homogeneous transformation matrix between a local coordinate system of the workpiece to be grabbed and a camera coordinate system:

wherein 0 is _1×3 Representing a zero vector of 1 row and 3 columns.

S9: by means of cone tip calibration method, the position of the standard cone on the mechanical arm workbench is adjusted for multiple times, and the coordinate system origin o of the mechanical arm end operator is controlled _t Aligned with the cone tips to obtain multiple groups of cone tips respectively in a camera coordinate system o _c -x _c y _c z _c And a mechanical arm coordinate system o _r -x _r y _r z _r The space coordinates below and further the rotation matrix R of the camera coordinate system relative to the robot arm coordinate system are obtained ₂ ∈R ^3×3 The method comprises the steps of carrying out a first treatment on the surface of the Position vector T in mechanical arm coordinate system combined with camera coordinate system ₂ ∈R ^3×1 The homogeneous transformation matrix between the camera coordinate system and the mechanical arm coordinate system can be obtained:

s10: then, the object point cloud local coordinate system o to be grasped _w -x _w y _w z _w Relative to the arm coordinate system o _r -x _r y _r z _r The homogeneous transformation matrix of (c) can be finally expressed as:

M＝M ₂ ·M ₁

by combining the point cloud information of the workpiece to be grabbed and the homogeneous transformation matrix M, the spatial pose of the workpiece to be grabbed can be calculated, and the mechanical arm is guided to carry out sorting operation on the workpiece to be grabbed.

The embodiment also provides a multi-category stacked workpiece mechanical arm sorting system based on point cloud and deep learning, which comprises a memory, a processor and computer program instructions which are stored on the memory and can be run by the processor, wherein the method steps can be realized when the processor runs the computer program instructions.

It will be appreciated by those skilled in the art that embodiments of the present application may be provided as a method, system, or computer program product. Accordingly, the present application may take the form of an entirely hardware embodiment, an entirely software embodiment, or an embodiment combining software and hardware aspects. Furthermore, the present application may take the form of a computer program product embodied on one or more computer-usable storage media (including, but not limited to, disk storage, CD-ROM, optical storage, and the like) having computer-usable program code embodied therein.

The present application is described with reference to flowchart illustrations and/or block diagrams of methods, apparatus (systems) and computer program products according to embodiments of the application. It will be understood that each flow and/or block of the flowchart illustrations and/or block diagrams, and combinations of flows and/or blocks in the flowchart illustrations and/or block diagrams, can be implemented by computer program instructions. These computer program instructions may be provided to a processor of a general purpose computer, special purpose computer, embedded processor, or other programmable data processing apparatus to produce a machine, such that the instructions, which execute via the processor of the computer or other programmable data processing apparatus, create means for implementing the functions specified in the flowchart flow or flows and/or block diagram block or blocks.

These computer program instructions may also be stored in a computer-readable memory that can direct a computer or other programmable data processing apparatus to function in a particular manner, such that the instructions stored in the computer-readable memory produce an article of manufacture including instruction means which implement the function specified in the flowchart flow or flows and/or block diagram block or blocks.

These computer program instructions may also be loaded onto a computer or other programmable data processing apparatus to cause a series of operational steps to be performed on the computer or other programmable apparatus to produce a computer implemented process such that the instructions which execute on the computer or other programmable apparatus provide steps for implementing the functions specified in the flowchart flow or flows and/or block diagram block or blocks.

The above description is only a preferred embodiment of the present invention, and is not intended to limit the invention in any way, and any person skilled in the art may make modifications or alterations to the disclosed technical content to the equivalent embodiments. However, any simple modification, equivalent variation and variation of the above embodiments according to the technical substance of the present invention still fall within the protection scope of the technical solution of the present invention.

Claims (10)

1. A multi-category stacked workpiece mechanical arm sorting method based on point cloud and deep learning is characterized by providing a mechanical arm sorting system mainly composed of a mechanical arm, a three-dimensional depth camera and a computer, wherein the computer is respectively connected with the mechanical arm and the three-dimensional depth camera and is in bidirectional communication; sorting the multi-category stacked workpieces based on the mechanical arm sorting system, comprising the following steps:

(1) Respectively acquiring multiple groups of complete point cloud data of each single workpiece to be sorted under different placement postures by using a three-dimensional depth camera, and carrying out point cloud preprocessing on the complete point cloud data to obtain template point clouds of each single workpiece and original point cloud data for subsequent point cloud classification network training;

(2) Performing network model training on the point cloud classification network based on deep learning based on the original point cloud data of each single workpiece to obtain a trained point cloud classification network;

(3) Collecting multi-category stacked workpiece point clouds in an actual scene by using a three-dimensional depth camera, and preprocessing and dividing the collected multi-category stacked workpiece point clouds to obtain a plurality of divided single workpiece point clouds;

(4) Classifying the point clouds of each segmented single workpiece by using a trained point cloud classification network to obtain corresponding class labels;

(5) On the basis of registering each single workpiece point cloud with the predicted category template point cloud, calculating the total degree of coincidence corresponding to each single workpiece point cloud based on a bidirectional octree searching method, and taking the workpiece point cloud with the highest total degree of coincidence as the workpiece point cloud to be grabbed of the mechanical arm;

(6) And establishing and calibrating a local coordinate system of the point cloud of the workpiece to be grabbed when each wheel of the mechanical arm grabs, and determining the space pose of the point cloud of the workpiece to be grabbed so as to guide the mechanical arm to accurately sort the point cloud of the workpiece to be grabbed.

2. The method for sorting the multi-category stacked workpiece mechanical arm based on the point cloud and the deep learning according to claim 1, wherein the implementation method of the step (1) is as follows:

respectively acquiring multiple groups of complete point cloud data of each single workpiece to be sorted under different placement postures by using a three-dimensional depth camera, wherein the acquired point cloud data of each single workpiece at least comprises (x, y, z) three-dimensional information; and then, carrying out point cloud preprocessing comprising noise reduction and outlier removal on the acquired point cloud data of each single workpiece, and determining template point clouds of each single workpiece and original point cloud data for subsequent point cloud classification network training.

3. The multi-category stacked workpiece mechanical arm sorting method based on point cloud and deep learning according to claim 2 is characterized in that in order to reduce the data processing amount of a subsequent point cloud sorting network, on the premise of ensuring that original structural features of the workpiece are not lost, the acquired single workpiece point clouds are subjected to uniform downsampling processing and then input into the point cloud sorting network based on the deep learning for network model training.

4. The method for sorting the multi-category stacked workpiece mechanical arm based on the point cloud and the deep learning according to claim 1, wherein the implementation method of the step (2) is as follows:

In the point cloud classification network, firstly, the information dimension of the workpiece point cloud is properly expanded by utilizing a multi-layer perceptron, workpiece characteristic information contained in the point cloud is projected to a higher dimension step by step, and the internal weight structures of the multi-layer perceptron are the same and the parameters are shared; secondly, outputting the integral characteristic parameters of the workpiece point cloud information after the processing of the maximum pooling layer; finally, classifying the pooled integral characteristic parameters through a multi-layer perceptron, and further determining the corresponding category of the workpiece point cloud; and training a network model by utilizing the original point cloud data of each single workpiece to obtain the optimal network parameters of each layer of the point cloud classification network, thereby obtaining the neural network model for the point cloud classification of the workpiece.

5. The multi-category stacked workpiece mechanical arm sorting method based on point cloud and deep learning as claimed in claim 1, wherein the implementation method of the step (3) is as follows:

acquiring multi-category stacked workpiece point clouds in an actual scene by using a three-dimensional depth camera; background filtering is carried out on the point clouds of the multi-category stacked workpieces, and then downsampling treatment is carried out on the point clouds subjected to background filtering; in order to remove useless noise points in the point cloud caused by the camera or light interference, denoising the point cloud after the downsampling process by adopting an outlier removal method based on Euclidean distance;

Performing curvature calculation on the preprocessed point cloud, selecting a point with the minimum curvature as an initial segmentation point, and calculating normals of other points in a neighborhood range of the initial segmentation point; presetting a smoothing threshold, and judging that the neighborhood point and the segmentation point belong to the same segmentation body if the calculated included angle between the normal line at the neighborhood point and the normal line at the segmentation point is smaller than the set smoothing threshold; otherwise, judging that the neighborhood point and the segmentation point do not belong to the same segmentation body; then, presetting a curvature threshold value, if the curvature of the neighborhood points belonging to the same division body is smaller than the set curvature threshold value, deleting the current division points and regarding the neighborhood points as new division points to perform next growth search until the new division points cannot be found, and completing division of a single workpiece point cloud; repeating the above dividing process to obtain the point clouds of other single workpieces after dividing.

6. The multi-category stacked workpiece mechanical arm sorting method based on point cloud and deep learning as claimed in claim 5, wherein due to mutual shielding among stacked workpieces, a few incomplete small single workpiece point clouds exist after division; setting a point threshold of the single workpiece point cloud, filtering small workpiece point clouds with the point smaller than the point threshold, and only reserving large single workpiece point clouds with more points.

7. The multi-category stacked workpiece mechanical arm sorting method based on point cloud and deep learning as claimed in claim 1, wherein the implementation method of the step (4) is as follows:

in order to reduce the data processing amount of the point cloud classification network, on the premise of ensuring that the structural characteristics of the point cloud of the workpiece are not lost, the single workpiece point clouds screened in the step (3) are subjected to uniform downsampling, and the downsampled single workpiece point clouds are input into the trained point cloud classification network to obtain the prediction category of each single workpiece.

8. The multi-category stacked workpiece mechanical arm sorting method based on point cloud and deep learning as claimed in claim 1, wherein the implementation method of the step (5) is as follows:

registering the point clouds of each single workpiece and the corresponding prediction category template point clouds: firstly, establishing a point pair relation between points in a point cloud and neighborhood points thereof by adopting a rapid point characteristic histogram method, and searching for point pairs with similar characteristic values in a template point cloud and a single workpiece point cloud to perform rough matching; secondly, setting a distance error threshold value, calculating the distance error between each monomer workpiece point cloud and the nearest point pair of the template point cloud after rough matching by adopting an iterative nearest point method, and finishing the precise matching between the template point cloud and the monomer workpiece point cloud when the distance error is smaller than the threshold value;

After the point clouds of each single workpiece are matched with the corresponding prediction type template point clouds, a bidirectional octree searching method is adopted to calculate the positive and negative double overlapping ratios, the sum of the two overlapping ratios is used as a final judgment standard, namely, the single workpiece point cloud with the highest overlapping ratio is the point cloud of the single workpiece with correct prediction type, no shielding and the most complete point cloud information, and the single workpiece point cloud is regarded as the point cloud of the workpiece to be grabbed, which is preferably grabbed by the mechanical arm.

9. The method for sorting multi-category stacked workpiece mechanical arms based on point cloud and deep learning as claimed in claim 8, wherein the single workpiece point cloud obtained by segmentation is provided to be sharedCalculating the eighth by using the bidirectional octree searching methodThe specific method for the overlap ratio of the single workpieces is as follows:

forward octree search: establishing an octree grid aiming at the gamma monomer workpiece point cloud, and setting a proper grid size; the total point number of the point cloud of the corresponding category template is recorded asInitializing the coincidence point->Traversing points in a template point cloudInquiring whether points in the point cloud of the workpiece to be grabbed are contained in the octree grid by coordinates corresponding to the point i, if so, regarding the point i as a coincidence point, and updating the coincidence point +.>If not, the coincidence point number is not updated >Wherein (1)>Represents the number of coincidence points obtained after querying point i, and +.>After the traversal is completed, the final obtained coincidence point number is utilized>Calculating the coincidence ratio between the single workpiece point cloud and the corresponding category template point cloud:

reverse octree search: establishing octree grids aiming at corresponding category template point clouds of the gamma monomer workpieces, and setting proper grid sizes; recording the total point number of the point cloud of the gamma-th monomer workpiece asInitializing the coincidence point->Traversing points in a single workpiece point cloud +.>Inquiring whether the octree grid contains the points in the template point cloud or not by the coordinates corresponding to the points j, if so, taking the points j as coincident points, and updating the coincident points +.>If not, the coincidence point number is not updated>Wherein (1)>Represents the number of coincidence points obtained after inquiring the point j, andafter the traversal is completed, the final obtained coincidence point number is utilized>Calculating the coincidence ratio between the template point cloud and the single workpiece point cloud:

then, the total degree of coincidence corresponding to the gamma-th monomer work-piece point cloud is:

and comparing the total degree of coincidence corresponding to the point clouds of the single workpieces, and taking the point cloud with the highest total degree of coincidence as the point cloud of the workpiece to be grabbed.

10. The multi-category stacked workpiece mechanical arm sorting method based on point cloud and deep learning as claimed in claim 1, wherein the implementation method of the step (6) is as follows:

In order to guide the mechanical arm to grab the workpiece, the mass center of the point cloud of the workpiece to be grabbed is required to be determined, and a local coordinate system of the point cloud of the workpiece to be grabbed is established by taking the mass center as an origin; let o _w Representing the point cloud centroid of the workpiece to be grabbed, and solving the position coordinates of the point cloud centroid by the following formula:

wherein, (x) _k ,y _k ,z _k ) Representing three-dimensional coordinates of points k (k=1, 2, …, N) of the point cloud of the workpiece to be grabbed under camera coordinates; n is the total point number contained in the point cloud of the workpiece to be grabbed;

at centroid o _w On the meterCalculating the symmetry and positive definite covariance matrix of the point cloud of the workpiece:

wherein,

solving covariance matrix C _w Is a three characteristic value lambda of (2) ₁ 、λ ₂ 、λ ₃ Corresponding to three unit feature vectors Respectively defining the straight lines of the three feature vectors as x of a local coordinate system of the point cloud of the workpiece to be grabbed _w Axis, y _w Axis, z _w The shaft is used for simultaneously initially determining the directions of the three feature vectors as the positive directions of three coordinate axes of a workpiece point cloud local coordinate system;

considering that workpieces to be grabbed are different when each wheel of the mechanical arm grabs, the corresponding point cloud local coordinate system z of the workpieces to be grabbed _w The axes are also not oriented the same, if the robot arm is gripping the workpiece, the end effector coordinate system o is adopted _t -x _t y _t z _t Z of (2) _t Shaft and workpiece point cloud local coordinate system o to be grasped _w -x _w y _w z _w Z of (2) _w The operation mode of coaxially approaching to grabbing is carried out after the shaft alignment;

In order to ensure the consistency of the grabbing operation of the mechanical arm, the local coordinate system z of the point cloud of the workpiece to be grabbed during each grabbing of the mechanical arm is needed _w The forward direction of the shaft is corrected timely to unify z _w The forward direction of the shaft; for this purpose, in the camera coordinate system o _c -x _c y _c z _c Z of (2) _c Taking the axial forward direction as a reference, and carrying out point cloud office on a workpiece to be grabbed when each wheel of the mechanical arm grabsPart coordinate system z _w Correcting the axial forward direction; is provided withFor camera coordinate system z _c Unit vector of axis, if->Indicating a local coordinate system z of point cloud of a workpiece to be grabbed _w Axis and camera coordinate system z _c The axes are oriented approximately the same, and a predetermined local coordinate system x of the point cloud of the workpiece is maintained _w Axis, y _w Axis, z _w The axial direction is unchanged; if it isThen the local coordinate system z of the point cloud of the workpiece to be grabbed is indicated _w Axis and camera coordinate system z _c The axes are oriented approximately opposite, in which case the workpiece to be grasped should be held in the local coordinate system z of the point cloud _w The positive correction of the axis is z _w The axis is initially oriented in the opposite direction, and the local coordinate system y of the point cloud of the workpiece is maintained _w Under the condition that the axial forward direction is unchanged, the local coordinate system x of the point cloud of the workpiece to be grabbed is re-corrected according to the right-hand rule of the coordinate system _w The forward direction of the axis, namely the local coordinate system x of the point cloud of the workpiece to be grabbed _w The positive correction of the axis is x _w The shaft is initially oriented in the opposite direction;

after the correction is finished, a point cloud local coordinate system o of the workpiece to be grabbed is obtained _w -x _w y _w z _w Relative to camera coordinate system o _c -x _c y _c z _c Is a rotation matrix R of (2) ₁ ∈R ^3×3 And combined with a local coordinate system o _w -x _w y _w z _w In the camera coordinate system o _c -x _c y _c z _c Position vector T in (a) ₁ ∈R ^3×1 Obtaining a homogeneous transformation matrix between a local coordinate system of the workpiece to be grabbed and a camera coordinate system:

wherein 0 is _1×3 A zero vector representing 1 row and 3 columns;

by means of cone tip calibration method, the position of the standard cone on the mechanical arm workbench is adjusted for multiple times, and the coordinate system origin o of the mechanical arm end operator is controlled _t Aligned with the cone tips to obtain multiple groups of cone tips respectively in a camera coordinate system o _c -x _c y _c z _c And a mechanical arm coordinate system o _r -x _r y _r z _r The space coordinates below and further the rotation matrix R of the camera coordinate system relative to the robot arm coordinate system are obtained ₂ ∈R ^3×3 The method comprises the steps of carrying out a first treatment on the surface of the Position vector T in mechanical arm coordinate system combined with camera coordinate system ₂ ∈R ^3×1 Obtaining a homogeneous transformation matrix between a camera coordinate system and a mechanical arm coordinate system:

then, the object point cloud local coordinate system o to be grasped _w -x _w y _w z _w Relative to the arm coordinate system o _r -x _r y _r z _r The homogeneous transformation matrix of (c) is finally expressed as:

M＝M ₂ ·M ₁

and calculating the space pose of the workpiece to be grabbed by combining the point cloud information of the workpiece to be grabbed and the homogeneous transformation matrix M, and further guiding the mechanical arm to carry out sorting operation on the workpiece to be grabbed.