CN114821263A - Weak texture target pose estimation method based on feature fusion - Google Patents

Abstract

The invention discloses a weak texture target pose estimation method based on feature fusion, which comprises the following steps of firstly, performing semantic segmentation on an RGB image where a target is located to obtain a target pixel mask and a minimum bounding box corresponding to the target; secondly, cutting the RGB image by adopting a minimum bounding box; thirdly, extracting color features by adopting a convolutional neural network; fourthly, acquiring a depth image; fifthly, segmenting the depth image and converting the depth image into point cloud data; sixthly, acquiring point features, local geometric features and global geometric features in the point cloud data; fusing the color features with the point features, the local geometric features and the global geometric features to obtain target fusion features; and eighthly, inputting the target fusion characteristics into a pose estimation network and outputting a pose estimation result. The method can be effectively applied to the pose estimation of the weak texture target, the problem that the pose estimation is insufficient in local feature consideration during feature fusion is solved, the accuracy of the pose estimation is improved, and the method is convenient to popularize and use.

Description

A Pose Estimation Method for Weak Texture Targets Based on Feature Fusion

technical field

The invention belongs to the technical field of target pose estimation, in particular to a weak texture target pose estimation method based on feature fusion.

Background technique

Target pose estimation can load virtual objects generated by the computer into real image sequences, obtain the pose of the object, and help the robotic arm to grip the object.

In the prior art, feature descriptors are used to extract target features to achieve pose estimation, for example, "A Method for Pose Estimation Combining SURF Descriptors and Autoencoders" (CN114037742A), the invention extracts SURF feature points of color images, and then The features extracted by the convolutional self-encoder and the corresponding pose information in the rendering data constitute an offline feature template, and the K feature vectors with the smallest distance in the feature template are selected to vote to obtain the 6D pose of the target. The invention reduces manual labeling, reduces the complexity of the environment, and reduces the amount of calculation. However, the use of feature descriptors such as SURF requires the target to have rich texture patterns. Therefore, this method is difficult to extract features for weak texture targets, and there is a problem that weak texture targets have poor effect. "A Pose Estimation Method for Weak Textured Objects" (CN113223181A), optimized for weak texture targets, the invention obtains color embedded features and geometric embedded features through color images and point clouds respectively, and then uses self-attention mechanism to extract positions Relying on the feature map, the pose estimation is performed after pixel-by-pixel fusion. The invention can enrich the information of each pixel feature, and adaptively adjust the weights of different features to improve the recognition accuracy of each pixel. However, this invention uses a dense fusion method, ignoring the influence of local features existing between point clouds on pose estimation, thus limiting the accuracy of pose estimation.

SUMMARY OF THE INVENTION

The technical problem to be solved by the present invention is to provide a weak texture target pose estimation method based on feature fusion in view of the above-mentioned deficiencies in the prior art. The method has simple steps, reasonable design, convenient implementation, and can be effectively applied to weak texture targets. In pose estimation, the problem that pose estimation does not consider local features in feature fusion is solved, the accuracy of pose estimation is improved, the model is more adaptable, the use effect is good, and it is easy to popularize and use. .

In order to solve the above-mentioned technical problems, the technical solution adopted in the present invention is: a method for estimating the pose of a weak texture target based on feature fusion, comprising the following steps:

Step 1. Semantically segment the RGB image where the target is located to obtain the target pixel mask and the minimum bounding box corresponding to the target;

Step 2, using the minimum bounding box to crop the RGB image to obtain the cropped RGB image;

Step 3, using a convolutional neural network to extract the color feature of the target in the cropped RGB image;

Step 4: Obtain the depth image of the cropped RGB image;

Step 5, using the mask to segment the depth image and convert it into point cloud data;

Step 6, obtaining point features, local geometric features and global geometric features in the point cloud data;

Step 7, fuse the color feature with the point feature, the local geometric feature and the global geometric feature to obtain the target fusion feature;

Step 8: Input the target fusion feature into the pose estimation network, and output the pose estimation result.

In the above-mentioned method for estimating the pose of a weak texture target based on feature fusion, the specific process of using a convolutional neural network to extract the color feature of the target in the cropped RGB image described in step 3 includes:

Step 301, using 18 convolutional layers to downsample the cropped RGB image features to obtain features with a dimension of 512;

Step 302 , up-sampling the features using four up-sampling layers to obtain 32-dimensional color features.

In the above-mentioned method for estimating the pose of a weak texture target based on feature fusion, the specific process of obtaining point features, local geometric features and global geometric features in the point cloud data described in step 6 includes:

Step 601, using PointNet network to extract point features in point cloud data;

Step 602, randomly select 256 position points, and fuse the features of multiple position points to reduce the influence of target occlusion and segmentation noise;

Step 603: Using the farthest point sampling method, find 128 points that are evenly distributed in the space;

Step 604, taking each evenly distributed point as the center point, dividing the sphere of fixed radius into a local area;

Step 605: In each local area, use PointNet to extract features for point clouds in three spatial scales of 0.05cm, 0.1cm and 0.2cm, and perform connection aggregation to form local geometric features;

Step 606, using the farthest point sampling method to find 64 points evenly distributed in the space;

Step 607, using each evenly distributed point as the center point, divide the sphere of fixed radius into a local area;

Step 608 , in each local area, perform feature extraction and aggregation on point clouds in two spatial scales of 0.2 cm and 0.3 cm;

Step 609 , using MLP to extract the global geometric features of the target on the basis of the local geometric features.

In the above-mentioned method for estimating the pose of a weak texture target based on feature fusion, as described in step 7, the color feature is fused with the point feature, the local geometric feature and the global geometric feature, and the specific process of obtaining the target fusion feature includes:

Step 701: Perform a 1d convolution operation on the color feature, and then fuse with the point feature to form a point fusion feature;

Step 702: Perform a 1d convolution operation on the point fusion feature again, and then fuse with the local geometric feature to form a local fusion feature;

Step 703 , fuse the global geometric feature in step 609 , the point fusion feature in step 701 and the local fusion feature in step 702 to form the final target fusion feature.

In the above-mentioned method for estimating the pose of a weak texture target based on feature fusion, the target fusion feature is input into the pose estimation network as described in step 8, and the specific process of outputting the pose estimation result includes:

Step 801, the target fusion feature is used as a training set to train a pose estimation network;

Step 802, the pose estimation network predicts the rotation, translation and the confidence of the pose prediction of the target;

Step 803, use the pose prediction made by the position point with the highest confidence as the initial pose;

Step 804 , using a four-layer fully connected network to optimize the initial pose to obtain a final pose estimation result.

In the above-mentioned method for estimating the pose of a weak texture target based on feature fusion, the pose estimation network in step 8 includes a loss function, and the loss function weights the pose loss through confidence, and the loss function Loss is:

表示N个位置点中第i点的位姿损失，c_i表示第i点预测位姿的置信度，ω代表平衡参数。Among them, i represents the ith point of N position points,

Represents the pose loss of the ith point in the N position points, ci represents the confidence of the predicted pose of the _ith point, and ω represents the balance parameter.

Compared with the prior art, the present invention has the following advantages:

1. The method of the present invention has simple steps, reasonable design and convenient implementation.

2. The present invention obtains higher-quality target features by paying attention to the fine-grained geometric features existing between parts of the point cloud data.

3. The present invention solves the problem of insufficient consideration of local features during feature fusion in pose estimation, and improves the accuracy of pose estimation.

4. The present invention can be effectively applied in the pose estimation of weak texture targets, has high precision, stronger model adaptability, good use effect, and is easy to popularize and use.

To sum up, the method of the present invention has simple steps, reasonable design and convenient implementation, and can be effectively applied in the pose estimation of weak texture targets, solves the problem of insufficient consideration of local features during feature fusion in pose estimation, and improves the pose The accuracy of the estimation, the model adaptability is stronger, the use effect is good, and it is easy to popularize and use.

The technical solutions of the present invention will be further described in detail below through the accompanying drawings and embodiments.

Description of drawings

Fig. 1 is the method flow chart of the present invention;

Fig. 2 is the network structure diagram of the present invention;

FIG. 3 is a visualization effect diagram of each target pose estimation result of the present invention.

Detailed ways

As shown in Figure 1 and Figure 2, the feature fusion-based weak texture target pose estimation method of the present invention includes the following steps:

Step 1. Semantically segment the RGB image where the target is located to obtain the target pixel mask and the minimum bounding box corresponding to the target;

Step 2, using the minimum bounding box to crop the RGB image to obtain the cropped RGB image;

Step 3, using a convolutional neural network to extract the color feature of the target in the cropped RGB image;

Step 4: Obtain the depth image of the cropped RGB image;

Step 5, using the mask to segment the depth image and convert it into point cloud data;

Step 6, obtaining point features, local geometric features and global geometric features in the point cloud data;

Step 7, fuse the color feature with the point feature, the local geometric feature and the global geometric feature to obtain the target fusion feature;

Step 8: Input the target fusion feature into the pose estimation network, and output the pose estimation result.

In this embodiment, the specific process of using the convolutional neural network to extract the color feature of the target in the cropped RGB image described in step 3 includes:

Step 301, using 18 convolutional layers to downsample the cropped RGB image features to obtain features with a dimension of 512;

Step 302 , up-sampling the features using four up-sampling layers to obtain 32-dimensional color features.

In this embodiment, the specific process of acquiring point features, local geometric features and global geometric features in the point cloud data described in step 6 includes:

Step 601, using PointNet network to extract point features in point cloud data;

Step 602, randomly select 256 position points, and fuse the features of multiple position points to reduce the influence of target occlusion and segmentation noise;

Step 603: Using the farthest point sampling method, find 128 points that are evenly distributed in the space;

Step 604, taking each evenly distributed point as the center point, dividing the sphere of fixed radius into a local area;

Step 605: In each local area, use PointNet to extract features for point clouds in three spatial scales of 0.05cm, 0.1cm and 0.2cm, and perform connection aggregation to form local geometric features;

Step 606, using the farthest point sampling method to find 64 points evenly distributed in the space;

Step 607, using each evenly distributed point as the center point, divide the sphere of fixed radius into a local area;

Step 608 , in each local area, perform feature extraction and aggregation on point clouds in two spatial scales of 0.2 cm and 0.3 cm;

Step 609 , using MLP to extract the global geometric features of the target on the basis of the local geometric features.

In this embodiment, as described in step 7, the color feature is fused with the point feature, the local geometric feature and the global geometric feature, and the specific process of obtaining the target fusion feature includes:

Step 701: Perform a 1d convolution operation on the color feature, and then fuse with the point feature to form a point fusion feature;

Step 702: Perform a 1d convolution operation on the point fusion feature again, and then fuse with the local geometric feature to form a local fusion feature;

Step 703 , fuse the global geometric feature in step 609 , the point fusion feature in step 701 and the local fusion feature in step 702 to form the final target fusion feature.

In this embodiment, the target fusion feature is input into the pose estimation network described in step 8, and the specific process of outputting the pose estimation result includes:

Step 801, the target fusion feature is used as a training set to train a pose estimation network;

Step 802, the pose estimation network predicts the rotation, translation and the confidence of the pose prediction of the target;

Step 803, use the pose prediction made by the position point with the highest confidence as the initial pose;

Step 804 , using a four-layer fully connected network to optimize the initial pose to obtain a final pose estimation result.

In this embodiment, the pose estimation network in step 8 includes a loss function, and the loss function weights the pose loss through the confidence, and the loss function Loss is:

表示N个位置点中第i点的位姿损失，c_i表示第i点预测位姿的置信度，ω代表平衡参数。Among them, i represents the ith point of N position points,

Represents the pose loss of the ith point in the N position points, ci represents the confidence of the predicted pose of the _ith point, and ω represents the balance parameter.

为目标3D模型采样点的坐标，分别通过Ground Truth位姿矩阵[R|t]和估计位姿矩阵

变换之后，对应点坐标之间的平均距离。

计算公式为：When implemented,

The coordinates of the sampling points of the target 3D model are passed through the Ground Truth pose matrix [R|t] and the estimated pose matrix respectively.

After transformation, the average distance between corresponding point coordinates.

The calculation formula is:

表示通过第点估计位姿变换后的坐标。Among them, M represents the set of sampling points of the target 3D model, i represents the ith point of the N position points, p is used as a marker, which means the pose loss here, x _j represents the jth point of the set of sampling points, (Rx _j + t) represents the coordinates of the sampling point transformed by the Ground Truth pose,

Represents the coordinates transformed by the estimated pose of the point.

变换后的坐标，与通过Ground Truth位姿矩阵[R|t]变换之后，最近点坐标之间的平均距离，计算公式为：For targets with rotational symmetry, the pose loss is defined as: the coordinates of the sample points of the target 3D model, by estimating the pose matrix

The average distance between the transformed coordinates and the coordinates of the closest point after transformation by the Ground Truth pose matrix [R|t], the calculation formula is:

Among them, x _k represents the point closest to x _j .

The invention is based on Intel(R) Xeon(R) CPU E5-2678 v3 2.50GHz processor, NVIDIA GeForceRTX 2080 graphics card, under Ubuntu16.04 system, using Pytorch0.4.1 and Python3.6 for development, using CUDA9.0 and cuDNN7 .6.4 Perform network acceleration training.

Table 1 shows the initial pose results and pose optimization results of our method in the LineMOD dataset. The dataset contains 13 weakly textured object sequences against complex backgrounds, and each sequence contains 1100-1300 RGB-D images. They are Ape, Benchvise, Camera, Can, Cat, Driller, Duck, Eggbox, Glue, Holepuncher, Iron, Lamp and Phone. Among them, Eggbox and Glue have rotational symmetry, and the size of each target is different. In the experiment, 15% of the RGB-D images of each target are divided into training sets and the rest are test sets. The training set contains a total of 2372 RGB-D images for 13 targets, and the test set contains a total of 13406 RGB-D images for 13 targets.

The comparison index adopts ADD accuracy, and the ADD calculation formula is as follows:

Among them, Num _pre represents the number of correct pose estimates, and Num _GT represents the number of all true poses. If the ADD is less than 10% of the maximum diameter of the target, the pose estimation is considered correct. The ADD calculation formula is:

The higher the accuracy, the better the pose estimation method.

Table 1 Pose optimization results

When extracting the local geometric features of the point cloud, the method of the invention extracts the point cloud features respectively from the point clouds of different radius spaces in the local area. Since different targets are trained at the same time, the selected multi-scale radii are the same. As a result, the extraction of local geometric features is not precise enough for small targets, which affects the experimental accuracy. For objects of large and medium size, the pose estimation accuracy is better.

In order to further verify the effect of the method of the present invention, the pose estimation result of the target is visualized, and the result is shown in Figure 3.

The above are only preferred embodiments of the present invention and do not limit the present invention. Any simple modifications, changes and equivalent structural changes made to the above embodiments according to the technical essence of the present invention still belong to the technology of the present invention. within the scope of the program.

Claims (6)

1. A weak texture target pose estimation method based on feature fusion is characterized by comprising the following steps:

firstly, performing semantic segmentation on an RGB image where a target is located to obtain a target pixel mask and a minimum bounding box corresponding to the target;

step two, cutting the RGB image by adopting the minimum bounding box to obtain the cut RGB image;

extracting color features of the target in the cut RGB image by adopting a convolutional neural network;

step four, obtaining a depth image of the cut RGB image;

fifthly, dividing the depth image by adopting the mask and converting the depth image into point cloud data;

sixthly, acquiring point features, local geometric features and global geometric features in the point cloud data;

step seven, fusing the color features with the point features, the local geometric features and the global geometric features to obtain target fusion features;

and step eight, inputting the target fusion characteristics into a pose estimation network, and outputting a pose estimation result.

2. The feature fusion-based weak texture object pose estimation method according to claim 1, wherein the specific process of extracting the color features of the objects in the clipped RGB image by using the convolutional neural network in the third step comprises:

step 301, adopting 18 convolution layers to carry out down-sampling on the cut RGB image characteristics to obtain characteristics with dimension of 512;

and 302, performing up-sampling on the features by adopting four up-sampling layers to obtain 32-dimensional color features.

3. The weak texture target pose estimation method based on feature fusion as claimed in claim 1, wherein the specific process of acquiring the point features, the local geometric features and the global geometric features in the point cloud data in the sixth step comprises:

601, extracting point characteristics in point cloud data by using a PointNet network;

step 602, randomly selecting 256 position points, and fusing the characteristics of the position points to reduce the influence of noise on target shielding and segmentation;

step 603, finding 128 points which are uniformly distributed in the space by adopting a farthest point sampling mode;

step 604, dividing the sphere with the fixed radius into a local area by taking each uniformly distributed point as a central point;

605, extracting features of point clouds in three spatial scales of 0.05cm, 0.1cm and 0.2cm in each local area by PointNet, and performing connection aggregation to form local geometric features;

step 606, adopting a farthest point sampling mode to find 64 points which are uniformly distributed in the space;

step 607, dividing the sphere with fixed radius into a local area by taking each evenly distributed point as a central point;

step 608, in each local area, performing feature extraction and aggregation on point clouds in two spatial scales of 0.2cm and 0.3 cm;

and step 609, extracting the global geometric features of the target on the basis of the local geometric features by adopting MLP.

4. The weak texture target pose estimation method based on feature fusion as claimed in claim 3, wherein the specific process of fusing the color feature with the point feature, the local geometric feature and the global geometric feature to obtain the target fusion feature in the seventh step includes:

701, performing 1d convolution operation on the color features, and fusing the color features and the point features to form point fusion features;

step 702, performing 1d convolution operation on the point fusion features again, and fusing the point fusion features with the local geometric features to form local fusion features;

and step 703, fusing the global geometric feature in step 609, the point fusion feature in step 701 and the local fusion feature in step 702 to form a final target fusion feature.

5. The weak texture target pose estimation method based on feature fusion as claimed in claim 4, wherein the specific process of inputting the target fusion features into the pose estimation network and outputting the pose estimation result in step eight includes:

step 801, taking the target fusion characteristics as a training set, and training a pose estimation network;

step 802, the pose estimation network predicts the rotation and translation of the target and the confidence of pose prediction;

step 803, using the pose prediction made by the position point with the highest confidence coefficient as the initial pose;

and 804, optimizing the initial pose by adopting a four-layer fully-connected network to obtain a final pose estimation result.

6. The weak texture target pose estimation method based on feature fusion of claim 5, wherein the pose estimation network in the eighth step comprises a Loss function, the Loss function weights pose Loss through confidence, and the Loss function Loss is:

wherein i represents the ith point of the N position points,

representing the pose loss of the ith point in the N position points, c _i And representing the confidence coefficient of the predicted pose of the ith point, wherein omega represents a balance parameter.

Images