CN114972495A - Grabbing method and device for object with pure plane structure and computing equipment - Google Patents

Abstract

The invention discloses a method, device and computing device for grasping objects with pure plane structure, wherein the method includes: acquiring point clouds corresponding to multiple objects in a current scene; for each object, corresponding to the object Extract the edge of the point cloud, get the edge point cloud corresponding to the object, and calculate the tangent vector of each 3D point in the edge point cloud; for any two 3D points in the edge point cloud, construct a point containing two 3D points Yes, and according to the tangent vector of the two 3D points, the point-to-point feature vector of the point pair is generated; according to the point-to-point feature vector of each point pair of the object, the edge point cloud corresponding to the object is matched with the preset template point cloud , get the pose information of the object. The scheme generates point-to-point feature vectors based on the tangent vector of 3D points in the edge point cloud, and matches the edge point cloud with the preset template point cloud according to the point-to-point feature vector, which realizes the rapid detection of the pose information of the object. , Accurate identification.

Description

Grabbing method, device and computing device for objects with pure plane structure

technical field

The present invention relates to the technical field of computers, and in particular to a method, device and computing device for grasping objects with a purely planar structure.

Background technique

With the development of industrial intelligence, the operation of objects (such as industrial parts, boxes, etc.) by robots instead of humans is becoming more and more popular. When the robot is operating, it is generally necessary to grab the object, move the object from one position and place it in another position, such as grabbing the object from a conveyor belt and placing it on a pallet or a cage, or grabbing from a pallet Objects, placed on conveyor belts or other pallets, etc. as required. However, in the prior art, the identification of the pose information of the object to be grasped is not accurate enough, and the identification efficiency is low, so it is difficult to meet the needs of high-speed industrial automation.

SUMMARY OF THE INVENTION

In view of the above problems, the present invention is proposed to provide a method, device and computing device for grasping objects with a purely planar structure that overcome the above problems or at least partially solve the above problems.

According to an aspect of the present invention, there is provided a method for grasping objects with a purely planar structure, the method comprising:

Obtain the point clouds corresponding to multiple objects in the current scene; wherein, multiple objects have a purely planar structure;

For each object, perform edge extraction on the point cloud corresponding to the object, obtain the edge point cloud corresponding to the object, and calculate the tangent vector of each 3D point in the edge point cloud;

For any two 3D points in the edge point cloud, construct a point pair containing two 3D points, and generate the point-to-point feature vector of the point pair according to the tangent vector of the two 3D points;

According to the point-to-point feature vector of each point pair of the object, the edge point cloud corresponding to the object is matched with the preset template point cloud, and the pose information of the object is obtained, so that the robot can perform grasping according to the pose information of the object. fetch operation.

According to another aspect of the present invention, there is provided a grasping device for objects with a purely planar structure, the device comprising:

a first acquisition module, adapted to acquire point clouds corresponding to multiple objects in the current scene, wherein the multiple objects have a pure plane structure;

The edge extraction module is suitable for performing edge extraction on the point cloud corresponding to the object for each object, obtaining the edge point cloud corresponding to the object, and calculating the tangent vector of each 3D point in the edge point cloud;

The point pair building module is suitable for constructing a point pair containing two 3D points for any two 3D points in the edge point cloud, and according to the tangent vector of the two 3D points, the point pair feature vector of the point pair is generated;

The matching module is adapted to match the edge point cloud corresponding to the object with the preset template point cloud according to the point-to-point feature vector of each point pair of the object, and obtain the pose information of the object for the robot to use according to the object's The pose information performs the grasping operation.

According to another aspect of the present invention, a computing device is provided, including: a processor, a memory, a communication interface, and a communication bus, and the processor, the memory, and the communication interface communicate with each other through the communication bus;

The memory is used for storing at least one executable instruction, and the executable instruction enables the processor to perform the operations corresponding to the above-mentioned method for grasping objects with a purely planar structure.

According to yet another aspect of the present invention, a computer storage medium is provided, the storage medium stores at least one executable instruction, and the executable instruction causes the processor to perform the operations corresponding to the above-mentioned method for grasping objects of pure plane structure.

According to the technical solution provided by the present invention, the pure plane structure feature is fully studied, the edge point cloud is extracted from the point cloud corresponding to the object, and the point-to-point feature vector of the point pair is generated according to the tangent vector of the 3D point in the edge point cloud, The point-to-point feature vector is used to accurately reflect the shape and structure characteristics of the object; the edge point cloud is matched with the preset template point cloud according to the point-to-point feature vector, and the accurate identification of the pose information of the object is realized, which is helpful for The robot can accurately and firmly perform the grasping operation according to the pose information of the object, so as to avoid grasping errors such as the robot failing to grasp the object successfully or the object falling after being grasped; and, in this solution, only the corresponding object is extracted. The edge point cloud of 1 participates in the matching process, and the non-edge point cloud does not participate in the matching process, which effectively reduces the matching workload and improves the recognition efficiency of the pose information of the object.

The above description is only an overview of the technical solutions of the present invention, in order to be able to understand the technical means of the present invention more clearly, it can be implemented according to the content of the description, and in order to make the above and other purposes, features and advantages of the present invention more obvious and easy to understand , the following specific embodiments of the present invention are given.

Description of drawings

Various other advantages and benefits will become apparent to those of ordinary skill in the art upon reading the following detailed description of the preferred embodiments. The drawings are for the purpose of illustrating preferred embodiments only and are not to be considered limiting of the invention. Also, the same components are denoted by the same reference numerals throughout the drawings. In the attached image:

FIG. 1 shows a schematic flowchart of a method for grasping objects with a purely planar structure according to an embodiment of the present invention;

FIG. 2 shows a schematic flowchart of a method for grasping objects with a purely planar structure according to another embodiment of the present invention;

3 shows a structural block diagram of a device for grasping objects with a purely planar structure according to an embodiment of the present invention;

FIG. 4 shows a schematic structural diagram of a computing device according to an embodiment of the present invention.

Detailed ways

Exemplary embodiments of the present disclosure will be described in more detail below with reference to the accompanying drawings. While exemplary embodiments of the present disclosure are shown in the drawings, it should be understood that the present disclosure may be embodied in various forms and should not be limited by the embodiments set forth herein. Rather, these embodiments are provided so that the present disclosure will be more thoroughly understood, and will fully convey the scope of the present disclosure to those skilled in the art.

FIG. 1 shows a schematic flowchart of a method for grasping objects with a purely planar structure according to an embodiment of the present invention. As shown in FIG. 1 , the method includes the following steps:

Step S101, acquiring point clouds corresponding to multiple objects in the current scene.

The current scene contains multiple objects, and the multiple objects have a purely planar structure. For example, the objects can be boxes in the shape of a cuboid. In step S101, point clouds corresponding to multiple objects in the current scene obtained by preprocessing may be acquired. The point cloud includes the pose information of each 3D point, and the pose information of each 3D point may specifically include the coordinate value of each 3D point in the XYZ three-axis in space, and the XYZ three-axis direction of each 3D point itself. After the point clouds corresponding to the multiple objects are acquired, each object in the current scene is processed sequentially in the manner of steps S102 to S104.

Step S102, for each object, perform edge extraction on the point cloud corresponding to the object, obtain the edge point cloud corresponding to the object, and calculate the tangent vector of each 3D point in the edge point cloud.

Wherein, for each object, the edge point cloud corresponding to the object can be obtained by extracting from the point cloud corresponding to the object based on 3D or 2D projection. In the process of invention, the inventor has carefully analyzed the edge point clouds corresponding to objects with a purely planar structure, and found that when objects of different shapes are in a flat state, the normal vector characteristics of each 3D point in the edge point cloud are the same. , the normal direction is vertically upward, but its tangent vector is different, then the tangent vector can be used to reflect the shape and structure characteristics of the object. After the edge point cloud corresponding to the object is extracted, the tangent vector of each 3D point in the edge point cloud can be calculated by using the vector calculation algorithm used to calculate the tangent vector in the three-dimensional space in the prior art, so as to generate the tangent vector according to the tangent vector. Point-to-point eigenvectors for point-to-point pairs. Those skilled in the art can select a vector calculation algorithm according to actual needs, which is not specifically limited here.

Step S103, for any two 3D points in the edge point cloud, construct a point pair including the two 3D points, and generate a point pair feature vector of the point pair according to the tangent vector of the two 3D points.

The edge point cloud corresponding to the object includes the pose information of each 3D point. For any two 3D points in the edge point cloud, a point pair containing two 3D points is constructed, and according to the pose information of the two 3D points and The tangent vector of these two 3D points generates the point-to-point feature vector of the point pair, so that the object corresponds to multiple point pairs. Specifically, the point-to-feature vector may include the Euclidean distance of the line vector between the two 3D points, the angle between the tangent vector of each of the two 3D points and the line vector, and the two 3D points. The angle between the tangent vectors of the 3D points. Those skilled in the art can also set point-to-feature vectors including other contents according to actual needs, which are not limited here.

Step S104, according to the point-to-point feature vector of each point pair of the object, the edge point cloud corresponding to the object is matched with the preset template point cloud, and the pose information of the object is obtained for the robot according to the pose of the object. information to perform a fetch operation.

In order to easily and accurately identify the pose information of each object in the scene image, a template library containing multiple preset template point clouds is pre-built. The preset template point clouds are pre-determined known objects that serve as matching benchmarks. the corresponding point cloud. For each object in the current scene, according to the point-to-point feature vector of each point pair of the object, the edge point cloud corresponding to the object is matched with the preset template point cloud, so as to obtain the pose information of the object, wherein, The pose information of the object may specifically include information such as the coordinate value of the center of the object in the XYZ three-axis of space, and the XYZ three-axis direction of the object itself. After the pose information of the object is obtained, the pose information of the object can be transmitted to the robot, so that the robot can perform a grasping operation on the object according to the pose information of the object.

According to the grasping method for objects of pure plane structure provided in this embodiment, the characteristics of pure plane structure are fully studied, edge point clouds are extracted from the point cloud corresponding to the object, and the tangent vector of 3D points in the edge point cloud is used to extract the edge point cloud. The point-to-point feature vector of the point-to-point pair is generated, and the precise reflection of the shape and structure characteristics of the object is realized through the point-to-point feature vector; the edge point cloud is matched with the preset template point cloud according to the point-to-point feature vector, and the position of the object is realized. Accurate recognition of posture information helps the robot to accurately and firmly perform grasping operations according to the posture information of the object, avoiding grasping errors such as the robot failing to grasp the object successfully or the object falling after being grasped; In this scheme, only the edge point cloud corresponding to the object is extracted to participate in the matching process, and the non-edge point cloud does not participate in the matching process, thereby effectively reducing the matching workload and improving the recognition efficiency of the pose information of the object.

FIG. 2 shows a schematic flowchart of a method for grasping objects with a purely planar structure according to another embodiment of the present invention. As shown in FIG. 2 , the method includes the following steps:

Step S201: Obtain a scene image of the current scene and a point cloud corresponding to the scene image, input the scene image into a trained deep learning segmentation model for instance segmentation processing, and obtain segmentation results of each object in the scene image.

Among them, the current scene contains a plurality of objects with pure plane structure. The scene image and depth image of the current scene can be collected by a camera set at an upper position, wherein the camera can be a 3D camera, and the 3D camera can be set at an upper position, such as directly above or obliquely above, for simultaneous acquisition. The information of the current scene within the camera's view angle can be used to obtain scene images and depth images. Specifically, the 3D camera may include components such as laser detectors, visible light detectors such as LEDs, infrared detectors, and/or radar detectors. The current scene is probed to get a depth image. The scene image may specifically be an RGB image, and the pixel points of the scene image and the depth image correspond one-to-one. By processing the scene image and the depth image, the point cloud corresponding to the scene image can be easily obtained. In step S201, a scene image of the current scene collected by the camera and a point cloud corresponding to the scene image obtained by processing the scene image and the depth image may be acquired.

In order to easily and accurately segment each object contained in the scene image, sample scene images can be collected in advance, a training sample set can be constructed, and each sample scene image in the training sample set can be trained by using a deep learning algorithm. In the deep learning segmentation model, after obtaining the scene image of the current scene, the scene image can be input into the trained deep learning segmentation model, and a series of model calculations can be performed by using the trained deep learning segmentation model. Each object included is subjected to instance segmentation processing, so as to obtain the segmentation result of each object in the scene image.

Step S202, according to the point cloud corresponding to the scene image and the segmentation result of each object, determine the point cloud corresponding to each object.

Wherein, the segmentation result of each object may include a binarized segmentation image of each object, and for each object, the binary segmentation image of the object may include the object area where the object is located and the non-object area other than the object area. The object area, the object area can be represented by a white area, and the non-object area can be represented by a black area.

Then for each object, the point cloud corresponding to the scene image can be projected into the binarized segmentation image of the object, and the 3D point in the object region of the object region of the binary segmentation image can be projected from the point cloud corresponding to the scene image as the The 3D point corresponding to the object is obtained, and the point cloud corresponding to the object is obtained. Specifically, all 3D points in the point cloud corresponding to the scene image are projected. If a 3D point in the point cloud corresponding to the scene image falls into the white object area after being projected, then the 3D point is considered to belong to the object. That is, the 3D point is the 3D point corresponding to the object, and all the 3D points corresponding to the object are summarized to obtain the point cloud corresponding to the object. Through this processing method, the accurate determination of the point cloud corresponding to the object is achieved.

Step S203, acquiring point clouds corresponding to multiple objects in the current scene.

Step S204, for each object, perform edge extraction on the point cloud corresponding to the object, obtain the edge point cloud corresponding to the object, and calculate the tangent vector of each 3D point in the edge point cloud.

In an optional implementation manner, the edge point cloud can be extracted based on a 3D manner. Specifically, for each 3D point in the point cloud corresponding to each object, search for the adjacent 3D point located in the preset neighborhood area of the 3D point in the point cloud, and obtain the 3D point and the adjacent 3D point. A point collection of points, for example, a neighborhood radius (such as 1cm or 0.3cm, etc.) can be set, and the area with the 3D point as the center and the neighborhood radius as the radius is configured as the preset neighborhood area of the 3D point, and then in the Find the 3D point in the preset neighborhood area of the 3D point in the point cloud corresponding to the object, call the found 3D point as the adjacent 3D point, and summarize the 3D point and all adjacent 3D points of the 3D point Get a set of points. After the point set is obtained, the normal vector calculation method in the prior art can be used to obtain the normal vector according to the points in the point set, and a plane where the normal vector is located is set for the normal vector.

Next, construct the connection line between the 3D point and each adjacent 3D point in the point set, and calculate the first angle between the projection line of each connection line in the plane where the normal vector is located and the specified reference direction axis. Each connection line is projected onto the plane where the normal vector is located, and the projection line corresponding to each connection line is obtained. The connection line and the projection line are in one-to-one correspondence, and the relationship between each projection line and the specified reference direction axis (such as the X axis, etc.) angle between. In this embodiment, in order to facilitate the angle between the angle and the tangent vector and the connection vector of each of the two 3D points in the edge point cloud below, and the angle between any two of the edge point clouds The angle between the tangent vectors of the 3D points is distinguished. The angle between the projection line and the specified reference direction axis in the plane where the normal vector is located is called the first angle, and the angle between the tangent vector of the 3D point and the connection vector The included angle is called the second included angle, and the included angle between the tangent vectors of two 3D points is called the third included angle.

After the first included angles corresponding to the respective projection lines are obtained, the respective projection lines can be sorted according to the first included angles, for example, in a clockwise direction, a counterclockwise direction, the first included angle in descending order, or the first included angle Sort each projection line in the order of the angles from small to large, and then calculate the angle difference between the first angles corresponding to the two adjacent projection lines after sorting, select the maximum value of the angle difference, and judge the difference between the angles. Whether the maximum value is greater than the preset angle difference threshold; if yes, it means that there is at least one area around the 3D point without other 3D points, then the 3D point is determined as an edge point; if not, it means that there are other 3D points around the 3D point 3D point, the 3D point is determined as a non-edge point. The above processing is performed for each 3D point in the point cloud corresponding to the object, by comparing the maximum value of the angle difference with the preset angle difference threshold to determine whether it is an edge point, and then summarizing all the edge points of the object , get the edge point cloud corresponding to the object. Those skilled in the art can set the preset angle difference threshold according to actual needs, which is not specifically limited here.

In another optional implementation manner, edge point clouds can also be extracted based on 2D projection. Specifically, each 3D point in the point cloud corresponding to the object can be projected onto a preset plane (such as the plane where the camera is located) to obtain a projected image. Areas where 3D point projections exist are marked in black. Then, the contour pixels in the projected image can be identified, the 3D points corresponding to the contour pixels are determined as edge points, and all the edge points of the object are summarized to obtain the edge point cloud corresponding to the object. Among them, an image recognition algorithm can be used to identify the contour boundary line formed in the projected image, determine the pixel point corresponding to the contour boundary line as the contour pixel point, and project the point cloud corresponding to the object to the 3D point on the contour pixel point. It is determined as the 3D point corresponding to the contour pixel, that is to say, if a 3D point in the point cloud corresponding to the object falls into the position of the contour pixel after being projected, then the 3D point is considered to have a correspondence with the contour pixel. relationship, the 3D point is the 3D point corresponding to the contour pixel point, and the 3D point is the edge point.

Through the above-mentioned 3D and 2D projection methods, the precise extraction of the edge point cloud corresponding to the object can be easily achieved. After the edge point cloud corresponding to the object is extracted, the vector calculation algorithm for calculating the tangent vector in the three-dimensional space in the prior art can be used to calculate the tangent vector of each 3D point in the edge point cloud, so that according to the tangent vector Generate point-to-point feature vectors for point pairs.

Step S205 , for any two 3D points in the edge point cloud, construct a point pair including the two 3D points, and generate a point pair feature vector of the point pair according to the tangent vector of the two 3D points.

The edge point cloud corresponding to the object includes the pose information of each 3D point, and the pose information includes information such as the coordinate values of the 3D point in the XYZ three axes of the space. In step S205, a line vector between the two 3D points can be constructed according to the coordinate values of these arbitrary two 3D points in the XYZ three axes of space, and the Euclidean distance of the line vector is calculated; then the two 3D points are calculated The second included angle between the tangent vector of each 3D point and the connecting line vector and the third included angle between the tangent vectors of the two 3D points in Triangulate, generate point-to-point feature vectors for point-to-point pairs.

Suppose any two 3D points in the edge point cloud are denoted as m ₁ and m ₂ respectively, the tangent vector of m ₁ is denoted as t ₁ , the tangent vector of m ₂ is denoted as t ₂ , the connection vector between m ₁ and m ₂ Denoted as d, where d=m ₂ -m ₁ , then the point-to-point feature vector containing the point pair of m ₁ and m ₂ can be expressed as (||d|| ₂ ,∠(t ₁ ,d),∠ (t ₂ ,d),∠(t ₁ ,t ₂ )), where ||d|| ₂ represents the Euclidean distance of the connection vector d, ∠(t ₁ ,d) represents the tangent vector t _{1 of m 1 and} The second angle between the line vectors d, ∠(t ₂ , d) represents the second angle between the tangent vector t ₂ of m ₂ and the line vector d, ∠(t ₁ , t ₂ ) represents m The third angle between the tangent vector _{t1 of 1 and} the tangent vector _t2 of _m2 .

Step S206, the point-to-point feature vector of each point pair in the edge point cloud corresponding to the object is matched with the point-to-point feature vector of each point pair in the edge point cloud of the preset template point cloud for the first time to obtain the first time match results.

Among them, the preset template point cloud is a predetermined point cloud corresponding to a known object as a matching reference, and the edge point cloud of the preset template point cloud is the edge point cloud corresponding to the known object, and the edge point cloud includes each 3D point. For any two 3D points in the edge point cloud of the preset template point cloud, a point pair containing two 3D points is constructed, and according to the pose information of the two 3D points and the two 3D points The point-to-point feature vector of the point-to-point pair is generated by the tangent vector of , so that the preset template point cloud also corresponds to multiple point-to-point pairs.

After the point-to-point feature vector of each point pair in the edge point cloud corresponding to the object is obtained by processing in step S205, the point-to-point feature vector of each point pair in the edge point cloud corresponding to the object and the edge point cloud of the preset template point cloud The first matching is performed on the feature vector of each point pair in , and the first matching result is obtained. The pose information of each 3D point is pre-defined in each preset template point cloud. In the first matching process, each preset template point cloud needs to be transformed into the current scene, so that the preset template point after pose transformation The edge point cloud of the cloud and the edge point cloud corresponding to the object in the current scene overlap as much as possible, so as to obtain the first matching result, wherein the first matching result may include multiple pose transformations of the matching preset template point cloud relation.

In step S207, the pose information of each 3D point in the edge point cloud corresponding to the object is matched with a plurality of pose transformation relationships of the matching preset template point cloud for the second time, according to the matching score in the second matching result. The pose transformation relationship of the matching preset template point cloud with the highest value determines the pose information of the object.

Since the first matching result may include multiple pose transformation relationships of the matched preset template point cloud, in order to more accurately determine the pose information of the object, in this embodiment, the edge corresponding to the object is The pose information of each 3D point in the point cloud is matched with multiple pose transformation relationships of the matched preset template point cloud for the second time. Specifically, a preset evaluation algorithm can be used to calculate the matching score between the pose information of each 3D point in the edge point cloud corresponding to the object and the multiple pose transformation relationships of the matching preset template point cloud, to obtain Second match result. Those skilled in the art can select a preset evaluation algorithm according to actual needs, which is not limited here. For example, the preset evaluation algorithm may be an ICP (Iterative Closest Point, iterative closest point) algorithm, a GMM (Gaussian Mixed Model, Gaussian Mixed Model) algorithm, and the like. The second matching is the further optimization and correction of the first matching result, and the pose transformation relationship of the matching preset template point cloud with the highest matching score in the second matching result is used as the final matching object. The pose transformation relationship of the matched preset template point cloud with the highest matching score in the secondary matching results determines the pose information of the object.

Each object in the current scene is processed in the manner from step S204 to step S207 to obtain pose information of each object. Since the pose information of the object is determined in the camera coordinate system, in order to facilitate the robot to locate the object, it is necessary to use a preset transformation algorithm to convert the pose information of the object into the robot coordinate system, and then convert the transformed pose of the object. The information is transmitted to the robot for the robot to perform grasping operations on the object according to the transformed pose information of the object.

According to the method for grasping objects with a purely planar structure provided in this embodiment, the scene image is segmented by using a deep learning segmentation model, so as to realize the precise segmentation of each object in the scene image; The edge point cloud is extracted from the point cloud, which realizes the precise extraction of the edge point cloud, and generates the point-to-point feature vector according to the tangent vector of the 3D point in the edge point cloud. The accurate reflection of the shape and structure features of the object; according to the point-to-feature vector, the point cloud corresponding to each object is matched with the preset template point cloud twice, which realizes the accurate identification of the pose information of the object, and helps the robot to identify the object according to the object. The pose information can accurately and firmly perform the grasping operation; and, in this scheme, only the edge point cloud corresponding to the object is extracted to participate in the matching process, and the non-edge point cloud does not participate in the matching process, thus effectively reducing the matching process. The workload is improved, and the recognition efficiency of the pose information of the object is improved.

Fig. 3 shows a structural block diagram of a device for grasping objects with a purely planar structure according to an embodiment of the present invention. As shown in Fig. 3, the device includes: a first acquisition module 301, an edge extraction module 302, a point pair construction module 303 and matching module 304.

The first acquisition module 301 is adapted to: acquire point clouds corresponding to multiple objects in the current scene; wherein the multiple objects have a pure plane structure.

The edge extraction module 302 is adapted to: for each object, perform edge extraction on the point cloud corresponding to the object, obtain the edge point cloud corresponding to the object, and calculate the tangent vector of each 3D point in the edge point cloud.

The point pair construction module 303 is adapted to: for any two 3D points in the edge point cloud, construct a point pair including the two 3D points, and generate a point pair feature vector of the point pair according to the tangent vector of the two 3D points.

The matching module 304 is adapted to: according to the point-to-point feature vector of each point pair of the object, match the edge point cloud corresponding to the object with the preset template point cloud, and obtain the pose information of the object for the robot to use according to the object. The pose information to perform the grasping operation.

Optionally, the apparatus may further include: a second acquisition module 305 , an instance segmentation module 306 and an object point cloud determination module 307 . The second obtaining module 305 is adapted to: obtain a scene image of the current scene and a point cloud corresponding to the scene image. The instance segmentation module 306 is adapted to: input the scene image into the trained deep learning segmentation model to perform instance segmentation processing, and obtain segmentation results of each object in the scene image. The object point cloud determination module 307 is adapted to: determine the point cloud corresponding to each object according to the point cloud corresponding to the scene image and the segmentation result of each object.

Optionally, the edge extraction module 302 is further adapted to: for each 3D point in the point cloud, search for the adjacent 3D point located in the preset neighborhood area of the 3D point in the point cloud, and obtain the 3D point containing the 3D point. and the point set of adjacent 3D points, and calculate the normal vector according to the point set; construct the connection between the 3D point and each adjacent 3D point in the point set, and calculate the projection of each connection on the plane where the normal vector is located The first angle between the line and the specified reference direction axis; sort each projection line according to the first angle, and calculate the angle difference between the first angles corresponding to the two adjacent projection lines after sorting; Whether the maximum value of the angle difference is greater than the preset angle difference threshold; if so, determine the 3D point as an edge point; if not, determine the 3D point as a non-edge point; summarize all edge points to get the edge corresponding to the object point cloud.

Optionally, the edge extraction module 302 is further adapted to: project each connection line on the plane where the normal vector is located to obtain the corresponding projection line of each connection line; calculate the distance between each projection line and the specified reference direction axis in the plane where the normal vector is located first angle.

Optionally, the edge extraction module 302 is further adapted to: project each 3D point in the point cloud corresponding to the object onto a preset plane to obtain a projected image; identify the contour pixels in the projected image, and convert the 3D points corresponding to the contour pixels to the projected image. The point is determined as an edge point; all edge points are aggregated to obtain the edge point cloud corresponding to the object.

Optionally, the edge extraction module 302 is further adapted to: identify the outline boundary line formed in the projected image, determine the pixel point corresponding to the outline boundary line as the outline pixel point, and project the point cloud corresponding to the object to the outline pixel point. The 3D point of is determined as the 3D point corresponding to the contour pixel point.

Optionally, the point pair construction module 303 is further adapted to: construct a line vector between two 3D points, and calculate the Euclidean distance of the line vector; calculate the tangent vector and the connection line of each of the two 3D points. The second included angle between vectors and the third included angle between the tangent vectors of two 3D points; the point-to-point feature vector of the point pair is generated using the Euclidean distance, second included angle, and third included angle of the connection vector .

Optionally, the matching module 304 is further adapted to: perform the first step between the point-to-point feature vector of each point pair in the edge point cloud corresponding to the object and the point-to-point feature vector of each point pair in the edge point cloud of the preset template point cloud. One matching, the first matching result is obtained; wherein, the first matching result includes multiple pose transformation relationships of the matched preset template point cloud; the pose of each 3D point in the edge point cloud corresponding to the object The information is matched with multiple pose transformation relationships of the matched preset template point cloud for the second time, and the pose transformation relationship of the matching preset template point cloud with the highest matching score in the second matching result is determined. The pose information of the object.

Optionally, the matching module 304 is further adapted to: use a preset evaluation algorithm to calculate the relationship between the pose information of each 3D point in the edge point cloud corresponding to the object and the multiple pose transformation relationships of the matching preset template point cloud. matching score between.

According to the grasping device for objects of pure plane structure provided in this embodiment, the scene image is segmented by using the deep learning segmentation model, and the precise segmentation of each object in the scene image is realized; The edge point cloud is extracted from the point cloud, which realizes the precise extraction of the edge point cloud, and generates the point-to-point feature vector according to the tangent vector of the 3D point in the edge point cloud. The accurate reflection of the shape and structure features of the object; according to the point-to-feature vector, the point cloud corresponding to each object is matched with the preset template point cloud twice, which realizes the accurate identification of the pose information of the object, and helps the robot to identify the object according to the object. The pose information can accurately and firmly perform the grasping operation; and, in this scheme, only the edge point cloud corresponding to the object is extracted to participate in the matching process, and the non-edge point cloud does not participate in the matching process, thus effectively reducing the matching process. The workload is improved, and the recognition efficiency of the pose information of the object is improved.

The present invention also provides a non-volatile computer storage medium, where the computer storage medium stores at least one executable instruction, and the executable instruction can execute any of the above method embodiments for grabbing a purely planar object.

FIG. 4 shows a schematic structural diagram of a computing device according to an embodiment of the present invention. The specific embodiment of the present invention does not limit the specific implementation of the computing device.

As shown in FIG. 4 , the computing device may include: a processor (processor) 402 , a communications interface (Communications Interface) 404 , a memory (memory) 406 , and a communication bus 408 .

in:

The processor 402 , the communication interface 404 , and the memory 406 communicate with each other through the communication bus 408 .

The communication interface 404 is used for communicating with network elements of other devices such as clients or other servers.

The processor 402 is configured to execute the program 410, and specifically, may execute the relevant steps in the above embodiments of the method for grasping objects with a purely planar structure.

Specifically, the program 410 may include program code including computer operation instructions.

The processor 402 may be a central processing unit (CPU), or an application specific integrated circuit (ASIC), or one or more integrated circuits configured to implement embodiments of the present invention. The one or more processors included in the computing device may be the same type of processors, such as one or more CPUs; or may be different types of processors, such as one or more CPUs and one or more ASICs.

The memory 406 is used to store the program 410 . Memory 406 may include high-speed RAM memory, and may also include non-volatile memory, such as at least one disk memory.

The program 410 can specifically be used to cause the processor 402 to execute the method for grasping objects with a purely planar structure in any of the foregoing method embodiments. For the specific implementation of the steps in the program 410, reference may be made to the corresponding descriptions in the corresponding steps and units in the above embodiment for grasping objects with a purely planar structure, which will not be repeated here. Those skilled in the art can clearly understand that, for the convenience and brevity of description, for the specific working process of the above-described devices and modules, reference may be made to the corresponding process descriptions in the foregoing method embodiments, which will not be repeated here.

The algorithms and displays provided herein are not inherently related to any particular computer, virtual system, or other device. Various general-purpose systems can also be used with teaching based on this. The structure required to construct such a system is apparent from the above description. Furthermore, the present invention is not directed to any particular programming language. It is to be understood that various programming languages may be used to implement the inventions described herein, and that the descriptions of specific languages above are intended to disclose the best mode for carrying out the invention.

In the description provided herein, numerous specific details are set forth. It will be understood, however, that embodiments of the invention may be practiced without these specific details. In some instances, well-known methods, structures and techniques have not been shown in detail in order not to obscure an understanding of this description.

Similarly, it is to be understood that in the above description of exemplary embodiments of the invention, various features of the invention are sometimes grouped together into a single embodiment, figure, or its description. This disclosure, however, should not be construed as reflecting an intention that the invention as claimed requires more features than are expressly recited in each claim. Rather, as the following claims reflect, inventive aspects lie in less than all features of a single foregoing disclosed embodiment. Thus, the claims following the Detailed Description are hereby expressly incorporated into this Detailed Description, with each claim standing on its own as a separate embodiment of this invention.

Those skilled in the art will understand that the modules in the device in the embodiment can be adaptively changed and arranged in one or more devices different from the embodiment. The modules or units or components in the embodiments may be combined into one module or unit or component, and further they may be divided into multiple sub-modules or sub-units or sub-assemblies. All features disclosed in this specification (including accompanying claims, abstract and drawings) and any method so disclosed may be employed in any combination, unless at least some of such features and/or procedures or elements are mutually exclusive. All processes or units of equipment are combined. Each feature disclosed in this specification (including accompanying claims, abstract and drawings) may be replaced by alternative features serving the same, equivalent or similar purpose, unless expressly stated otherwise.

Furthermore, those skilled in the art will appreciate that although some of the embodiments described herein include certain features, but not others, included in other embodiments, that combinations of features of different embodiments are intended to be within the scope of the invention within and form different embodiments. For example, in the claims, any of the claimed embodiments may be used in any combination.

Various component embodiments of the present invention may be implemented in hardware, or in software modules running on one or more processors, or in a combination thereof. Those skilled in the art should understand that a microprocessor or a digital signal processor (DSP) may be used in practice to implement some or all of the functions of some or all of the components according to the embodiments of the present invention. The present invention can also be implemented as apparatus or apparatus programs (eg, computer programs and computer program products) for performing part or all of the methods described herein. Such a program implementing the present invention may be stored on a computer-readable medium, or may be in the form of one or more signals. Such signals may be downloaded from Internet sites, or provided on carrier signals, or in any other form.

It should be noted that the above-described embodiments illustrate rather than limit the invention, and that alternative embodiments may be devised by those skilled in the art without departing from the scope of the appended claims. In the claims, any reference signs placed between parentheses shall not be construed as limiting the claim. The word "comprising" does not exclude the presence of elements or steps not listed in a claim. The word "a" or "an" preceding an element does not exclude the presence of a plurality of such elements. The invention can be implemented by means of hardware comprising several different elements and by means of a suitably programmed computer. In a unit claim enumerating several means, several of these means may be embodied by one and the same item of hardware. The use of the words first, second, and third, etc. do not denote any order. These words can be interpreted as names.

Claims (20)

1. A method for grasping objects of pure plane structure, the method comprising: Acquiring point clouds corresponding to multiple objects in the current scene; wherein the multiple objects have a purely planar structure; For each object, perform edge extraction on the point cloud corresponding to the object, obtain the edge point cloud corresponding to the object, and calculate the tangent vector of each 3D point in the edge point cloud; For any two 3D points in the edge point cloud, construct a point pair including the two 3D points, and generate a point-to-point feature vector of the point pair according to the tangent vector of the two 3D points; According to the point-to-point feature vector of each point pair of the object, the edge point cloud corresponding to the object is matched with the preset template point cloud, and the pose information of the object is obtained, so that the robot can perform grasping according to the pose information of the object. fetch operation. 2. The method according to claim 1, wherein before the acquiring the point clouds corresponding to the multiple objects in the current scene, the method further comprises: Obtain a scene image of the current scene and a point cloud corresponding to the scene image, input the scene image into a trained deep learning segmentation model to perform instance segmentation processing, and obtain segmentation results of each object in the scene image; According to the point cloud corresponding to the scene image and the segmentation result of each object, the point cloud corresponding to each object is determined. 3. The method according to claim 1, wherein, for each object, performing edge extraction on the point cloud corresponding to the object, and obtaining the edge point cloud corresponding to the object further comprises: For each 3D point in the point cloud, search the point cloud for adjacent 3D points located in the preset neighborhood area of the 3D point, and obtain a point set containing the 3D point and the adjacent 3D points , and calculate the normal vector according to the set of points; Construct the connection line between the 3D point and each adjacent 3D point in the point set, and calculate the first included angle between the projection line of each connection line in the plane where the normal vector is located and the specified reference direction axis ; Sort each projection line according to the first included angle, and calculate the included angle difference between the first included angles corresponding to the sorted adjacent two projection lines; Determine whether the maximum value of the included angle difference is greater than a preset angle difference threshold; if so, determine the 3D point as an edge point; if not, determine the 3D point as a non-edge point; Summarize all edge points to get the edge point cloud corresponding to the object. 4. The method according to claim 3, wherein the calculating the first included angle between the projection line of each connecting line in the plane where the normal vector is located and the specified reference direction axis further comprises: Projecting each connection line onto the plane where the normal vector is located to obtain a projection line corresponding to each connection line; Calculate the first included angle between each projection line and the specified reference direction axis in the plane where the normal vector is located. 5. The method according to claim 1, wherein, for each object, performing edge extraction on the point cloud corresponding to the object, and obtaining the edge point cloud corresponding to the object further comprises: Project each 3D point in the point cloud corresponding to the object onto a preset plane to obtain a projected image; Identifying contour pixels in the projected image, and determining the 3D points corresponding to the contour pixels as edge points; Summarize all edge points to get the edge point cloud corresponding to the object. 6. The method of claim 5, wherein the identifying contour pixels in the projected image further comprises: Identify the contour boundary line formed in the projection image, determine the pixel point corresponding to the contour boundary line as the contour pixel point, and determine the 3D point projected to the contour pixel point in the point cloud corresponding to the object as the contour pixel point. The 3D point corresponding to the contour pixel point. 7. The method according to any one of claims 1-6, wherein the generating the point-to-point feature vector of the point pair according to the tangent vector of the two 3D points further comprises: constructing a line vector between the two 3D points, and calculating the Euclidean distance of the line vector; calculating the second included angle between the tangent vector of each of the two 3D points and the connecting line vector and the third included angle between the tangent vectors of the two 3D points; Using the Euclidean distance of the connection vector, the second included angle, and the third included angle, a point-to-point feature vector of the point pair is generated. 8. The method according to any one of claims 1-7, wherein, according to the point-to-point feature vector of each point pair of the object, the edge point cloud corresponding to the object is matched with the preset template point cloud, Obtaining the pose information of the object further includes: The point-to-point feature vector of each point pair in the edge point cloud corresponding to the object is matched with the point-to-point feature vector of each point pair in the edge point cloud of the preset template point cloud for the first time, and the first matching result is obtained; Wherein, the first matching result includes a plurality of pose transformation relationships of the matched preset template point cloud; Perform a second match between the pose information of each 3D point in the edge point cloud corresponding to the object and the multiple pose transformation relationships of the matching preset template point cloud, and according to the second matching result, the one with the highest matching score. The pose transformation relationship of the matched preset template point cloud determines the pose information of the object. 9 . The method according to claim 8 , wherein the pose information of each 3D point in the edge point cloud corresponding to the object and a plurality of pose transformation relationships of the matching preset template point cloud are carried out secondly. 10 . Secondary matches further include: Using a preset evaluation algorithm, the matching score between the pose information of each 3D point in the edge point cloud corresponding to the object and the multiple pose transformation relationships of the matching preset template point cloud is calculated. 10. A grasping device for objects with a purely planar structure, the device comprising: a first acquisition module, adapted to acquire point clouds corresponding to multiple objects in the current scene; wherein the multiple objects have a pure plane structure; The edge extraction module is adapted to perform edge extraction on the point cloud corresponding to the object for each object, obtain the edge point cloud corresponding to the object, and calculate the tangent vector of each 3D point in the edge point cloud; A point pair construction module, adapted to construct a point pair including the two 3D points for any two 3D points in the edge point cloud, and generate the point pair according to the tangent vector of the two 3D points The point-to-feature vector of ; The matching module is adapted to match the edge point cloud corresponding to the object with the preset template point cloud according to the point-to-point feature vector of each point pair of the object, and obtain the pose information of the object for the robot to use according to the object's The pose information performs the grasping operation. 11. The apparatus of claim 10, wherein the apparatus further comprises: a second acquisition module, adapted to acquire a scene image of the current scene and a point cloud corresponding to the scene image; an instance segmentation module, adapted to input the scene image into a trained deep learning segmentation model for instance segmentation processing, to obtain segmentation results of each object in the scene image; The object point cloud determination module is adapted to determine the point cloud corresponding to each object according to the point cloud corresponding to the scene image and the segmentation result of each object. 12. The apparatus of claim 10, wherein the edge extraction module is further adapted to: For each 3D point in the point cloud, search the point cloud for adjacent 3D points located in the preset neighborhood area of the 3D point, and obtain a point set containing the 3D point and the adjacent 3D points , and calculate the normal vector according to the set of points; Construct the connection line between the 3D point and each adjacent 3D point in the point set, and calculate the first included angle between the projection line of each connection line in the plane where the normal vector is located and the specified reference direction axis ; Sort each projection line according to the first included angle, and calculate the included angle difference between the first included angles corresponding to the sorted adjacent two projection lines; Determine whether the maximum value of the included angle difference is greater than a preset angle difference threshold; if so, determine the 3D point as an edge point; if not, determine the 3D point as a non-edge point; Summarize all edge points to get the edge point cloud corresponding to the object. 13. The apparatus of claim 12, wherein the edge extraction module is further adapted to: Projecting each connection line onto the plane where the normal vector is located to obtain a projection line corresponding to each connection line; Calculate the first included angle between each projection line and the specified reference direction axis in the plane where the normal vector is located. 14. The apparatus of claim 10, wherein the edge extraction module is further adapted to: Project each 3D point in the point cloud corresponding to the object onto a preset plane to obtain a projected image; Identifying contour pixels in the projected image, and determining the 3D points corresponding to the contour pixels as edge points; Summarize all edge points to get the edge point cloud corresponding to the object. 15. The apparatus of claim 14, wherein the edge extraction module is further adapted to: Identify the contour boundary line formed in the projection image, determine the pixel point corresponding to the contour boundary line as the contour pixel point, and determine the 3D point projected to the contour pixel point in the point cloud corresponding to the object as the contour pixel point. The 3D point corresponding to the contour pixel point. 16. The apparatus of any of claims 10-15, wherein the peer-to-peer building block is further adapted to: constructing a line vector between the two 3D points, and calculating the Euclidean distance of the line vector; calculating the second included angle between the tangent vector of each of the two 3D points and the connecting line vector and the third included angle between the tangent vectors of the two 3D points; Using the Euclidean distance of the connection vector, the second included angle, and the third included angle, a point-to-point feature vector of the point pair is generated. 17. The apparatus of any of claims 10-16, wherein the matching module is further adapted to: The point-to-point feature vector of each point pair in the edge point cloud corresponding to the object is matched with the point-to-point feature vector of each point pair in the edge point cloud of the preset template point cloud for the first time, and the first matching result is obtained; Wherein, the first matching result includes a plurality of pose transformation relationships of the matched preset template point cloud; Perform a second match between the pose information of each 3D point in the edge point cloud corresponding to the object and the multiple pose transformation relationships of the matching preset template point cloud, and according to the second matching result, the one with the highest matching score. The pose transformation relationship of the matched preset template point cloud determines the pose information of the object. 18. The apparatus of claim 17, wherein the matching module is further adapted to: Using a preset evaluation algorithm, the matching score between the pose information of each 3D point in the edge point cloud corresponding to the object and the multiple pose transformation relationships of the matching preset template point cloud is calculated. 19. A computing device, comprising: a processor, a memory, a communication interface and a communication bus, the processor, the memory and the communication interface communicate with each other through the communication bus; The memory is used for storing at least one executable instruction, and the executable instruction causes the processor to perform an operation corresponding to the method for grabbing an object with a purely planar structure according to any one of claims 1-9. 20. A computer storage medium in which at least one executable instruction is stored, the executable instruction causing a processor to execute the method for a purely planar object according to any one of claims 1-9. The operation corresponding to the grab method.

Images