CN115937043B - A method of touch-assisted point cloud completion - Google Patents

Abstract

The invention belongs to the field of three-dimensional point cloud completion, and particularly relates to a three-dimensional point cloud completion method based on haptic assistance. The problem of partial detail deficiency in the process of generating the complement point cloud by the missing point cloud under a single view angle is solved, and the point cloud complement effect is improved by utilizing the partial tactile information. Mainly comprises the following steps: step 1, initializing a Pybullet simulation environment, and acquiring a touch picture by using a mechanical arm connected with an electric clamping jaw and a touch sensor; step 2, converting the touch picture into a touch point cloud, and splicing the touch point cloud with the missing point cloud; step 3, training the PoinTr network by using the data set; and 4, inputting the spliced point cloud into a PoinTr network to obtain a completed point cloud.

Description

A method of touch-assisted point cloud completion

technical field

The invention belongs to the field of three-dimensional point cloud completion, and in particular relates to a tactile-assisted point cloud completion method.

Background technique

Understanding 3D space is critical for humans and machines to understand how to interact with objects around them. Point clouds, as an easily accessible 3D structural data, have facilitated extensive research in computer vision for understanding 3D scenes and objects. However, the raw point clouds routinely captured by lidar scanners or RGB-D cameras are inevitably sparse and incomplete due to the limitations of sensor resolution, object occlusion, and object surface materials. Benefiting from large-scale point cloud datasets, deep learning-based point cloud completion methods have attracted increasing research interest. Recently, advances in 3D point cloud-based processing techniques have facilitated point cloud completion research. The seminal work PointNet proposes to apply MLP independently at each point, aggregating features through pooling operation to achieve permutation invariance. PointCleanNet is the first learning-based architecture, which proposes an encoder-decoder framework, and uses FoldingNet to map two-dimensional points to three-dimensional surfaces by simulating a two-dimensional plane, and introduces a completion strategy from coarse to fine. Parts gradually recover details. SeedFormer introduces a new shape representation method, Patch Seeds, which can capture the overall structure from partial inputs and preserve local region information, while introducing an upsampling Transformer module in the completion task. PoinTr defines point cloud completion as a set-to-set transformation problem, representing point clouds as a set of unordered point groups with positional embeddings, converting missing point clouds into a set of point proxies, embedding special geometry-aware blocks, using Generate a complete point cloud based on Transformer's encoder-decoder architecture. Due to the discreteness of the point cloud and the unstructured local area predicted by the point cloud, it is difficult for the above method to maintain a good point distribution structure in the local area, and thus cannot capture the geometric details and structure of the local area well, such as a smooth area. , sharp edges and corners.

The sense of touch is another way of perceiving the 3D shape of objects. For robotics, most tactile sensors measure the geometry of the contact surface. The robot can reconstruct the shape of an object through multiple touches, combined with the position and pose of the sensor at each touch, without being affected by ambiguity caused by the color or material of the object's surface. However, tactile information is limited by the size and scale of the sensor: since each touch can only obtain information on a local area, multiple touches and a long time are required to reconstruct the complete shape of the object, so a single point cloud based only on tactile Reconstruction is difficult to apply in practice.

Therefore, in the field of 3D point cloud completion, it is necessary to explore a point cloud completion method that can not only use the learning ability of neural networks, but also reasonably use the local touch information of objects, so as to improve the efficiency and complementarity of point cloud completion. full effect.

Contents of the invention

Existing 3D point cloud completion methods have certain limitations. For example, point cloud completion methods based on neural networks use missing point clouds as network input, and it is often difficult to capture local geometric details in the completion task of point clouds in missing regions. structure, resulting in poor complementary reconstruction of complex objects; while object reconstruction based solely on tactile information will be limited by the size and scale of the sensor at each touch: since each touch can only obtain information on a local area, the reconstruction efficiency Often inefficient. In the present invention, a set of 3D point cloud completion method based on DIGIT simulation tactile point cloud assistance is proposed. In the Pybullet simulation engine, the robotic arm connected with the electric gripper and the DIGIT tactile sensor provided by Facebook are used to obtain controllable tactile information on the surface of the object and generate a tactile point cloud corresponding to the touch position. At the same time, using 3D deep learning and a large-scale 3D shape repository, the implicit shape prior of the object is obtained during the machine learning process. The tactile point cloud contains the local geometric information of the object, integrates the local tactile point cloud into the incomplete point cloud, and uses machine learning to constrain the overall situation and optimize the shape, making up for the shortcomings of ordinary machine learning networks that cannot capture local details.

The invention provides a method for tactile-assisted point cloud completion, comprising the following steps:

Step 1, initialize the Pybullet simulation environment, use the robotic arm connected with the electric gripper and the DIGIT tactile sensor to touch the missing area of the object point cloud, and obtain the tactile image and pose information of the touched area;

Step 2, convert the tactile image into a preliminary tactile point cloud, and then transform the preliminary tactile point cloud from the world coordinate system to the target feature coordinate system, and stitch it with the object missing point cloud;

Step 3, use the dataset to train the PointTr network;

Step 4, input the point cloud spliced in step 2 into the trained PoinTr network to obtain the complete point cloud after completion.

Further, in step 1, obtain the tactile image of the missing area of the point cloud of the object, specifically: move the robotic arm and control the closing of the gripper, touch the surface of the missing area of the object point cloud, and generate a tactile image of the touched area;

Record the surface depth information H∈R ^n×m of the object captured on the DIGIT tactile sensor at this time, the space coordinate x∈R ³ and the rotation vector r∈R ³ of the center of the DIGIT tactile sensor, to determine the touch area in the world coordinate system pose.

Further, in step 2, the tactile image is converted into a preliminary tactile point cloud, and then the preliminary tactile point cloud is transformed from the world coordinate system to the target feature coordinate system, and spliced with the object missing point cloud, specifically including the following steps:

Step 2.1, take the center point of the DIGIT tactile sensor as the center, and construct an initialization plane point cloud on the tangent plane of the center point;

In step 2.2, the preliminary haptic point cloud P _c is obtained by superimposing the depth information H of the touch area on the initialization plane point cloud. Then, the rotation vector r of the DIGIT tactile sensor recorded during touch is transformed into a rotation matrix R, and combined with the space coordinate x, the tactile point cloud P _t in the world coordinate system is obtained:

P _t =P _c *R+x

In step 2.3, the tactile point cloud P _t in the world coordinate system is spliced with the missing point cloud of the object, and input into the PointTr network to obtain the complete point cloud after completion.

Further, select different touch areas to add touch to the missing point cloud, repeat steps 1-4, and select the appropriate touch position to help reconstruct the details of the point cloud.

Beneficial effects: the point cloud completion method using the local touch information of the object integrates the local tactile point cloud into the incomplete point cloud, thereby improving the efficiency and effect of point cloud completion.

Description of drawings

Figure 1 is a flow chart of the method of the present invention.

Figure 2 is a visualization of touching objects and tactile pictures in the Pybullet simulation environment.

Figure 3 is the overall framework diagram of the point cloud completion network.

Figure 4 is the effect diagram of point cloud completion with one touch added.

Figure 5 is the effect diagram of point cloud completion with different touch times.

Figure 6 is an effect diagram of point cloud completion at different touch positions.

Detailed ways

In order to deepen the understanding of the present invention, the present invention will be further described below in conjunction with the examples, which are only used to explain the present invention, and do not constitute a limitation to the protection scope of the present invention.

The method for point cloud completion based on tactile sense of the present invention, as shown in Figure 1, comprises the following steps:

Step 1, initialize the Pybullet simulation environment, use the robotic arm connected with the electric gripper and the DIGIT tactile sensor to touch the missing area of the object point cloud, and obtain the tactile image and pose information of the touched area;

In the Pybullet simulation engine, a mechanical arm connected with a two-finger electric gripper is built, and each finger of the gripper is equipped with a DIGIT tactile sensor provided by Facebook to obtain tactile images. Place the object at a fixed position in front of the robotic arm. Initially, the electric gripper is directly above the target object. Subsequently, move the robotic arm and control the closing of the jaws to make it touch the surface of the target object, as shown in Figure 2(a), until the DIGIT tactile sensor feedback image appears obvious contact image and then stops moving, and the object touches the area The tactile image comes from the depth information of the contact surface provided by the DIGIT tactile sensor;

As shown in Figure 2(b); record the depth information H∈R ^n×m obtained on the DIGIT tactile sensor at this time, the space coordinate x∈R ³ and the rotation vector r∈R ³ of the center of the DIGIT tactile sensor to determine the The pose of the touch area in the world coordinate system.

Step 2, convert the tactile image into a preliminary tactile point cloud, and then convert the preliminary tactile point cloud from the world coordinate system to the object coordinate system, and stitch it with the object missing point cloud. The specific steps are as follows:

Step 2.1, take the center point of the DIGIT tactile sensor as the center, and construct an initialization plane point cloud on the tangent plane of the center point;

Step 2.2, by superimposing the depth information H of the touch area on the initialization point cloud plane to obtain the preliminary tactile point cloud as P _c ; then transform the rotation vector r of the DIGIT tactile sensor recorded during the touch into a rotation matrix R, and then combine the space coordinates x, get the tactile point cloud P _t in the world coordinate system:

P _t =P _c *R+x

Denoise the preliminary tactile point cloud _Pt by a given threshold;

Step 2.3, use the space coordinate x and the rotation vector r of the DIGIT tactile sensor to transform the preliminary tactile point cloud P _t to the corresponding position in the object coordinate system, and splicing with the missing point cloud of the target object. The specific instructions are as follows:

The missing point cloud of the target object is located in the object coordinate system, and the tactile point cloud P _t is located in the world coordinate system; align the object coordinate system, the robot coordinate system and the world coordinate system to ensure that the missing object point cloud and the tactile point cloud P _t are in the same coordinates Department;

Assuming that the world coordinate system and the robot coordinate system coincide, you only need to align the object coordinate system and the robot coordinate system. Take the three non-overlapping points w1, w2, w3 in the robot coordinate system, and record the positions r1, r2, r3 of the three points in the target object coordinate system, and obtain the alignment between the robot coordinate system and the object coordinate system by solving the linear equation The required rotation matrix R _r and translation vector T _r :

X _r ＝R _r *X _w +T _r

X _r =[r1,r2,r3]

X _w = [w1,w2,w3]

In order to simplify the calculation process, it is assumed that the target object coordinate system and the robot coordinate system rotation matrix R _r are unit matrices, so only the translation vector T _r between the two coordinate systems is required. Place the target object at a fixed position w1=(m,0,0) ^T in the x-axis direction of the robot coordinate system, and a point p _i in the target object coordinate system corresponds to

The p _j in the robot coordinate system are:

p _j =p _i +w1

Since the obtained tactile point cloud is too dense, it will affect the performance of the subsequent PoinTr network local attention module. In order to prevent the model from paying too much attention to local information and ignoring long-range information, it is necessary to use the farthest point sampling method to downsample the tactile point cloud to 100 points. . Through the conversion between the coordinate systems, the final tactile point cloud and the missing point cloud of the target object can be stitched together.

Step 3, use the dataset to train the PointTr network;

The PoinTr network is based on Transformer encoder-decoder structure, including feature extraction module, geometry-aware encoder module, geometry-aware decoder module and upsampling module.

First, the feature extraction module extracts the feature vector from the point cloud obtained after splicing, and then inputs the feature vector to the geometric perception encoder module, and establishes the geometric relationship between the point clouds through the geometric perception encoder module; the geometry perception decoder module Query the geometric relationship between point clouds to generate predicted point proxies and proxy features for missing regions.

Finally, the predicted point proxy and proxy features are input to the upsampling module, which is used to restore the detailed local shape centered on the predicted point proxy, and output the completed point cloud.

The training dataset is ShapeNet-55, which contains 41,952 object models from 55 categories. For each object model, 8192 points are randomly sampled from the surface of the object as a complete point cloud. Considering the uncertainty of the viewing angle of the missing point cloud in the test, a point of view is randomly selected first, and then the 4096 points farthest from the point of view are removed to obtain the training point cloud. Partially missing point clouds. The overall framework of the network is shown in Figure 3.

Step 4: Input the spliced point cloud into the trained Pointr network to obtain the complete point cloud after completion. If you need to add touch points or secondary points, return to step 1;

For the verification of the effect of DIGIT tactile-assisted point cloud completion, a number of experiments were carried out, including the impact of adding one touch on point cloud completion, the impact of different touch times on point cloud completion, and the effect of different touch positions on point cloud completion. Influence. The object model used in the experimental phase is selected from the ShapeNet-55 training dataset. This method uses the PointTr network to achieve the reconstruction of missing point clouds. The PoinTr network is based on the Transfromer encoder and decoder structure, and introduces a local attention module in the encoder and decoder, which can effectively complete the point cloud completion task. The following experimental results are all comparisons between the method of the present invention and the PoinTr network benchmark.

Table 1 shows the experimental results of the effect of adding a touch on point cloud completion. The evaluation standard of the data in Table 1 is the chamfer distance (CD2) between the complementary reconstruction point cloud and the real point cloud. The smaller the value, the better the complementary reconstruction effect.

Table 1 Comparison of point cloud completion results with one touch added

Figure 4 shows the completion effects of spheres, cubes, guitars, and buckets as examples. It can be seen from the figure that all objects can be reconstructed, and most objects can be reconstructed better after adding tactile information. For more complex objects, such as buckets and guitars, the details of the shape cannot be well reconstructed without adding tactile point clouds, and the actual objects can only be obtained after using tactile assistance. For relatively simple objects, such as spheres and cubes, adding tactile information may lead to poor reconstruction results. One of the reasons is that the model itself is too regular and simple, and good reconstruction results can be achieved only by locally missing point clouds; , when the model is relatively small, the DIGIT haptic simulator will have lens distortion at the edge of its depth map imaging range. However, considering that most of the practical applications are for the reconstruction of more complex objects, the method is feasible in most scenarios.

The experimental results of adding multiple touches to the point cloud completion and reconstruction are compared as shown in Figure 5 and Table 2. Most objects can be reconstructed better after adding more touches.

Table 2 is a comparison of point cloud completion results for different touch times

Figure 5 shows the reconstruction results of the bucket, guitar and basket. The data of each type of object in the figure is represented as follows: in Figure 5 (a) the real point cloud; (b) the missing point cloud with three tactile additions; (c) the completion result without tactile addition; (d) the complementary tactile addition (e) the completion result of adding the second haptic; (f) the completion result of adding the third haptic. The other side of these three models is missing local details without adding tactile sensation, such as the bucket and basket missing the handle on the other side, and the guitar does not reconstruct the missing shape very well, but adding tactile point cloud Afterwards, the local details on the other side of them are better recovered, while the originally divergent point clouds become more clustered. In short, for the three models, as the number of added tactile point clouds increases, the local details are more in line with reality. In the complementary point cloud output by the network when no touch is added, the point cloud of the original missing area is restored relatively sparsely. and divergence, which can be restored to be more dense and clustered after adding multiple haptic point clouds.

The impact of different touch positions on point cloud completion and reconstruction is shown in Figure 6. Taking buckets, chairs, and guitars as examples, selecting an appropriate touch position can better reconstruct the details of the point cloud. Each column of data in the figure is represented as follows: in Figure 6 (a) real point cloud; (b) add a touch at position 1; (c) corresponding to the completion result of (b); (d) add a touch at position 2; (e) corresponds to the completion result of (d). The position 1 and position 2 are to distinguish the two added touch areas, and the exact position is not limited; the experimental results show that if the touch is added to the area where the completion effect is not good, the completion result will be greatly improved. As shown in the bucket in the picture, when the touch position is in the middle and upper part of the bucket, the reconstruction result cannot obtain the corresponding local details. When the touch position is at the handle of the bucket, the details of the point cloud are well reconstructed; for chairs and guitars, when The touch area has recovered well before the touch is added, and the tactile point cloud cannot play its role well during reconstruction, resulting in the final reconstruction result not reaching the ideal goal, which is basically the same as the conclusion obtained by adding touch to the above-mentioned simple shape object unanimous.

It should be emphasized that the simulation environment applicable to this method takes the DIGIT tactile sensor provided by Facebook as an example, but it is not limited to this, and it is also applicable to many other types of tactile sensors; this method is based on the background of point cloud completion Expanded, the simulated environment construction and experimental process shown in the method, including the construction of the simulated environment, the acquisition of simulated tactile point clouds, the splicing of different modal point cloud data, etc., are also applicable to fusion of tactile and visual point clouds. Multimodal point cloud processing tasks.

Claims (8)

1. A method for tactile-assisted point cloud completion, comprising the steps of: Step 1, initialize the Pybullet simulation environment, use the robotic arm connected with the electric gripper and the DIGIT tactile sensor to touch the missing area of the object point cloud, and obtain the tactile image and pose information of the touched area; Among them, to obtain the tactile picture of the missing area of the point cloud of the object, specifically: move the robotic arm and control the closing of the gripper, touch the surface of the missing area of the point cloud of the object, and generate a tactile picture of the touched area; record the DIGIT tactile sensor at this time The depth information H∈R ^n×m of the surface of the object captured above and the space coordinate x∈R ³ and the rotation vector r∈R ³ of the center of the DIGIT tactile sensor to determine the pose of the touch area in the world coordinate system; Step 2, convert the tactile image into a preliminary tactile point cloud, and then transform the preliminary tactile point cloud from the world coordinate system to the target feature coordinate system, and stitch it with the object missing point cloud, specifically including the following steps: Step 2.1, take the center point of the DIGIT tactile sensor as the center, and construct an initialization plane point cloud on the tangent plane of the center point; Step 2.2, by superimposing the depth information H of the touch area on the initialization plane point cloud to obtain the preliminary tactile point cloud P _c ; then transform the rotation vector r of the DIGIT tactile sensor recorded during the touch into a rotation matrix R, and then combine the space coordinate x , to get the tactile point cloud P _t in the world coordinate system: P _t =P _c *R+x Step 2.3, splicing the tactile point cloud P _t in the world coordinate system and the missing point cloud of the object, and inputting it into the PointTr network to obtain the complete point cloud after completion; Step 3, use the dataset to train the PointTr network; Step 4, input the point cloud spliced in step 2 into the trained PoinTr network to obtain the complete point cloud after completion. 2. A method for tactile-assisted point cloud completion according to claim 1, characterized in that, selecting different touch areas to add touches to the missing point cloud respectively, and repeating steps 1-4, is conducive to reconstructing the details of the point cloud. 3. according to the described method of a kind of tactile sense auxiliary point cloud complementing of claim 1, it is characterized in that, the specific content of step 3 is as follows: target object missing point cloud is positioned at object coordinate system, and tactile sense point cloud _P is positioned at world coordinate system; Align the object coordinate system, the robot coordinate system and the world coordinate system to ensure that the object missing point cloud and the tactile point cloud P _t are in the same coordinate system. 4. A method for tactile-assisted point cloud completion according to claim 3, characterized in that, assuming that the world coordinate system and the robot coordinate system coincide, it is only necessary to align the object coordinate system and the robot coordinate system, and take the robot coordinate system The three non-overlapping points w1, w2, w3, and record the positions r1, r2, r3 of the three points in the coordinate system of the target object, and obtain the rotation required to align the coordinate system of the robot with the coordinate system of the object by solving the linear equation Matrix R _r and translation vector T _r : X _r ＝R _r *X _w +T _r X _r = [r1, r2, r3] _Xw = [w1, w2, w3]. 5. according to the described method of a kind of tactile-assisted point cloud completion of claim 4, it is characterized in that, assuming that the target object coordinate system and the robot coordinate system rotation matrix R _r are identity matrix, only the translation between the two coordinate systems is required Vector T _r ; place the target object at a fixed position w1=(m,0,0) ^T in the x-axis direction of the robot coordinate system, and a point p _i in the target object coordinate system corresponds to the robot coordinate system The p _j have: p _j =p _i +w1. 6. A method for tactile-assisted point cloud completion according to claim 1, characterized in that the tactile point cloud is down-sampled to 100 points by sampling the farthest point. 7. The method for a kind of tactile-assisted point cloud completion according to claim 1, wherein the PointTr network is based on a Transformer encoder-decoder structure. 8. a kind of tactile-assisted point cloud complementing method according to claim 1, is characterized in that, described PoinTr network comprises feature extraction module, geometry perception coder module, geometry perception decoder module and upsampling module; First, the feature extraction module extracts the feature vector from the point cloud obtained after splicing, and then inputs the feature vector to the geometric perception encoder module, and establishes the geometric relationship between the point clouds through the geometric perception encoder module; the geometry perception decoder module Query the geometric relationship between point clouds to generate the predicted point proxy and proxy features of the missing area; finally, input the predicted point proxy and proxy features into the upsampling module, use the upsampling module to restore the detailed local shape centered on the predicted point proxy, and output Complete point cloud after completion.

Images