KR20220065234A - Apparatus and method for estimating of 6d pose - Google Patents

Abstract

The present invention relates to a 6D pose estimation apparatus capable of efficiently estimating the pose of an object. According to the present invention, an apparatus for estimating an object pose by using unsupervised learning comprises: a feature extraction unit including an external feature extraction unit inputting RGB data of an image into a first deep learning model to extract external features and a geometric feature extraction unit inputting depth data of the image to a second deep learning model to extract geometric features; a combination unit combining extracted external features and geometric features to generate a feature map for the image; and an object pose estimation unit estimating a 6D pose of an object corresponding to the image on the basis of the generated feature map. The second deep learning model performs transfer learning on the basis of weights of a network learned by a user-defined problem (Pretext task).

Description

6D pose estimation apparatus and method {APPARATUS AND METHOD FOR ESTIMATING OF 6D POSE}

The present invention relates to a 6D pose estimation apparatus and method.

In general, the pose estimation of an object uses a deep learning model trained using training data including labels. That is, the pose of the object can be estimated by inputting the 3D data of the object into the learned deep learning model.

Good quality data is very important for training deep learning models for 6D (x, y, z, roll, pitch, yaw) pose estimation. In this case, since random data must be generated in order to create a large data set, performance in a real environment may be degraded. In particular, it is very difficult to prepare a data set for handling objects with six degrees of freedom (6DOF) using 3D data, and it takes much more time and cost compared to 2D data.

Unsupervised learning is a type of machine learning that belongs to the category of problems of figuring out how data is structured. Unlike supervised learning or reinforcement learning, training data with only input values is It is a learning method to find regularity of inputs using

Among unsupervised learning, self-supervised learning is a type of unsupervised learning, and it is a problem to actually solve a network that has learned a user-defined problem (pretext task) using unlabeled data. It is a learning method that uses transfer learning as a downstream task.

Korean Patent Publication No. 1994316 (Registered on June 24, 2019)

The present invention is to solve the problems of the prior art described above, and a 6D pose estimation apparatus capable of solving the problem of performance degradation that occurs when estimating the pose of an object using a deep learning model learned based on arbitrary data. would like to provide

Another object of the present invention is to provide a 6D pose estimation apparatus capable of estimating an object pose efficiently using only unlabeled data or data containing a small amount of labels.

However, the technical problems to be achieved by the present embodiment are not limited to the technical problems described above, and other technical problems may exist.

As a means for achieving the above-described technical problem, an embodiment of the present invention provides an apparatus for estimating an object pose using unsupervised learning, by applying RGB data for an image to a first deep learning model. a feature extracting unit comprising: an appearance feature extracting unit for inputting and extracting external features; and a geometric feature extracting unit for extracting geometric features by inputting depth data for the image into a second deep learning model; a combining unit for generating a feature map for the image by combining the extracted external features and geometric features; and an object pose estimator for estimating a 6D pose for the object corresponding to the image based on the generated feature map, wherein the second deep learning model is a network learned by a user-defined problem (Pretext task). It is possible to provide an apparatus for estimating a 6D pose, which is transfer-learned based on weights.

Another embodiment of the present invention provides a method for estimating an object pose using unsupervised learning, the method comprising: inputting RGB data for an image into a first deep learning model to extract appearance features; A feature extraction step comprising extracting geometric features by inputting depth data for an image into a second deep learning model; generating a feature map for the image by combining the extracted external features and geometric features; and estimating a 6D pose for the object corresponding to the image based on the generated feature map, wherein the second deep learning model is based on the weight of the network learned by a user-defined problem (Pretext task). It is possible to provide a 6D pose estimation method, which is transfer learned based on the

The above-described problem solving means are merely exemplary, and should not be construed as limiting the present invention. In addition to the exemplary embodiments described above, there may be additional embodiments described in the drawings and detailed description.

According to any one of the above-described problem solving means of the present invention, it is possible to provide a 6D pose estimation apparatus capable of estimating the pose of an object efficiently with only data without labels or data including a small amount of labels.

In addition, using the 6D information of the estimated object, it is possible to perform complex tasks including assembly as well as pick & place tasks of the robot.

1 is a block diagram of a 6D pose estimation apparatus according to an embodiment of the present invention.
2 is an exemplary diagram for explaining the configuration of a 6D pose estimation apparatus according to an embodiment of the present invention.
3 is an exemplary view for explaining a tray model according to an embodiment of the present invention.
4 is an exemplary diagram for explaining a noise unit of a user-defined problem (Pretext task) performing unit according to an embodiment of the present invention.
5 is a flowchart of a 6D pose estimation method according to an embodiment of the present invention.

DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS Hereinafter, embodiments of the present invention will be described in detail with reference to the accompanying drawings so that those of ordinary skill in the art can easily implement them. However, the present invention may be embodied in several different forms and is not limited to the embodiments described herein. And in order to clearly explain the present invention in the drawings, parts irrelevant to the description are omitted, and similar reference numerals are attached to similar parts throughout the specification.

Throughout the specification, when a part is "connected" with another part, this includes not only the case of being "directly connected" but also the case of being "electrically connected" with another element interposed therebetween. . In addition, when a part "includes" a certain component, it means that other components may be further included, rather than excluding other components, unless otherwise stated, and one or more other features However, it is to be understood that the existence or addition of numbers, steps, operations, components, parts, or combinations thereof is not precluded in advance.

In this specification, a "part" includes a unit realized by hardware, a unit realized by software, and a unit realized using both. In addition, one unit may be implemented using two or more hardware, and two or more units may be implemented by one hardware.

Some of the operations or functions described as being performed by the terminal or device in this specification may be instead performed by an apparatus connected to the terminal or device. Likewise, some of the operations or functions described as being performed by the apparatus may also be performed in a terminal or device connected to the apparatus.

Hereinafter, an embodiment of the present invention will be described in detail with reference to the accompanying drawings.

1 is a block diagram of a 6D pose estimation apparatus according to an embodiment of the present invention. Referring to FIG. 1 , the 6D pose estimation apparatus 100 may include a feature extraction unit 110 , a combiner 120 , an object pose estimation unit 130 , and a user-defined problem (Pretext task) performing unit 140 . can

The feature extractor 110 may include an external feature extractor 111 and a geometric feature extractor 112 , and the object pose estimator 130 includes a semantic segmentation module 131 and a feature point detection module. (Keypoint detection module, 132) and may include a center voting module (Center voting module, 133), the user-defined problem (Pretext task) performing unit 140 is a modeling unit 141, a noise unit 142 and learning It may include a portion 143 . However, the above components 110 to 140 are merely illustrative of components that can be controlled by the 6D pose estimation apparatus 100 .

The apparatus 100 for estimating a 6D pose according to an embodiment of the present invention can efficiently estimate the pose of an object using only data without labels or data including a small amount of labels.

Also, the apparatus 100 for estimating a 6D pose according to an embodiment of the present invention may perform a complex operation including assembly as well as a pick and place operation of the robot using 6D information of the estimated object.

2 is an exemplary diagram for explaining the configuration of a 6D pose estimation apparatus according to an embodiment of the present invention. Referring to FIG. 2 , the feature extracting unit 110 may include an external feature extracting unit 111 and a geometrical feature extracting unit 112 .

The appearance feature extraction unit 111 according to an embodiment of the present invention may extract the appearance features by inputting the RGB data 111a for the image into the first deep learning model 111b. Here, the first deep learning model 111b may be a Convolutional Neural Network (CNN). For example, the appearance feature extraction unit 111 may receive the RGB data 111a and extract the appearance features of the image through the CNN 111b.

The geometric feature extraction unit 112 may extract the geometric features by inputting the depth data 112a of the image into the second deep learning model 112b. Here, the second deep learning model 112b may be a PointNet model. For example, the geometric feature extraction unit 112 may receive the depth data 112a and extract the geometric features of the image through the second deep learning model 112b.

The geometric feature extraction unit 112 according to an embodiment of the present invention may transfer-learning the weight of the second deep learning model 112b to the weight of a separate network 143a. Here, the network 143a may be learned using unlabeled data through self-supervised learning, which is a type of unsupervised learning. For example, the network 143a may be trained by a user-defined task (Pretext task). As described above, if a user-defined problem (pretext task) is used as self-supervised learning, understanding of 3D data itself can be increased, so that performance for a real environment can be improved when estimating a 6D pose.

The geometrical feature extraction unit 112 may extract geometrical features of the image using the second deep learning model 112b transferred by transfer-learning the network learned by the user-defined problem (Pretext task).

The user-defined task (pretext task) performing unit 140 according to an embodiment of the present invention may perform a user-defined task (pretext task) based on the network 143a. For example, the user-defined task (Pretext task) performing unit 140 learns the network 143a using unlabeled data, that is, a user-defined user-defined user-defined task (Pretext task, input) to obtain the network 143a. Iteratively learns to understand the data and extract meaningful features (output) from the data.

The modeling unit 141 of the user-defined problem (Pretext task) performing unit 140 models a tray model including a plurality of holes and an object model insertable into at least one of the plurality of holes. can do.

3 is an exemplary view for explaining a tray model according to an embodiment of the present invention. Referring to FIG. 3 , the tray model 300 may include a plurality of holes 310 . The tray model 300 may include a plurality of holes 310 into which, for example, an electric drill, a nipper, a stripper, a screwdriver, a wrench, and a spanner can be inserted as an object model.

The modeling unit 141 may model the tray model 300 including a plurality of holes 310 into which various object models can be inserted. Also, the modeling unit 141 may model a plurality of object models that can be inserted into at least one of the plurality of holes 310 .

When the user-defined problem (Pretext task) performing unit 140 learns the network 143a using the tray model and the object model without any noise in the virtual environment, the object pose estimator 130 may You may not understand the object properly in the image.

To solve this problem, the noise unit 142 may add noise to the point cloud of the modeled tray model and the object model. For example, the noise unit 142 adds noise to the point clouds of the modeled tray model and the object model, so that the object pose estimator 130 estimates a 6D pose for the object corresponding to the real image. can improve comprehension.

For example, the noise unit 142 may randomly change the value of the point cloud based on the depth quality specification of the RGB-D camera used. For example, the noise unit 142 may add or subtract a random value from the original data within a depth standard deviation range, and data corresponding to a non-fill rate ratio is NaN. It can be treated as (not a number).

4 is an exemplary diagram for explaining a noise unit of a user-defined problem (Pretext task) performing unit according to an embodiment of the present invention. Referring to FIG. 4 , the depth quality is taken using the RealSense ^TM D400 series camera as an example.

The learning unit 143 may train the network 143a to determine a hole into which the object model is inserted by inputting it into the network 143a while rotating the object model to which the noise is added.

For example, the learning unit 143 may input a noise-added and rotated object model to the network 143a. When a point cloud of an object is input to the network 143a, the learning unit 143 may output a correct answer probability for each object model. The learning unit 143 may match the object model with the highest probability among the output correct probabilities to the tray model.

Also, the learning unit 143 may calculate an overlapping area and an empty area between the object model and the tray model as a loss (LOSS). For example, when the learning unit 143 matches the object model with the highest probability of correct answer to the tray model, the tray model and the tray model as well as the area where the tray model and the object model overlap by using the object model before adding noise and rotating The blank area between object models can be calculated as loss.

Referring back to FIG. 2 , the feature extracting unit 110 combines the external features of the image extracted through the external feature extracting unit 111 and the geometrical features of the image extracted through the geometrical feature extracting unit 112 into the combining unit 120 . ) can be passed to

The combining unit 120 according to an embodiment of the present invention may generate a feature map for an image by combining the extracted external features and geometric features. For example, the combining unit 120 combines the external features extracted by the external feature extracting unit 111 and the geometric features extracted by the geometric feature extracting unit 112 to each coordinate (point) of the object recognized in the image. It is possible to create a feature map having the combined features for each.

The object pose estimator 130 according to an embodiment of the present invention may estimate a 6D pose of an object corresponding to an image based on the generated feature map. For example, the object pose estimator 130 may estimate ( 136 ) a 6D pose of the object based on a feature map having a combined feature with respect to each coordinate of the object recognized in the image.

The object pose estimator 130 may calculate a 6D pose for the object based on the semantic segmentation module 131 , the feature point detection module 132 , and the center voting module 133 .

For example, the object pose estimator 130 may receive the feature map as input, and firstly classify one or more objects included in the image through the semantic segmentation module 131, respectively, and then, the feature point detection module ( It is possible to detect a 3D keypoint for each object surface divided through 132, and then, by detecting the center point of the object through the center voting module 133, a 6D pose for the object is estimated (136) )can do.

The semantic segmentation module 131 according to an embodiment of the present invention may classify each object when there are a plurality of objects on the image based on the feature map.

The feature point detection module 132 may detect a 3D feature point on the surface of the divided object. For example, the feature point detection module 132 detects the 3D feature point on the surface of the divided object, and predicts the Euclidean transformation offset from the visible point to the target keypoint for each detected point, and , it is possible to detect 3D feature points on the surface of the divided object through the process of voting and clustering the target feature points again.

The center voting module 133 may detect a center point of the object. For example, the center voting module 133 may extend the center point of the object from 2D to 3D.

Instance Segmentation (134) is an object detected by the semantic segmentation module 131, the feature point detection module 132, and the center voting module 133. Based on the 3D feature point and the center point of the object, global features and Local features can be extracted. The instance segmentation 134 may distinguish objects having similar shapes but different sizes from the learned size information in order to predict an offset with respect to a feature point.

The object pose estimator 130 is configured to fit the least-squares method based on the features extracted through the semantic segmentation module 131 , the feature point detection module 132 , the center voting module 133 , and the instance segmentation 134 . Fitting, 135) can be performed. For example, the object pose estimator 130 calculates pose parameters (R, t) for an object model that minimizes a square loss with M feature points detected in the camera coordinate system and points in the object coordinate system corresponding thereto. Estimate (136) a 6D pose for .

Meanwhile, when the data is a 3D point cloud, the object pose estimator 130 may perform voxelization on the data. In this case, the user-defined problem (Pretext task) performing unit 140 also performs input data voxelization may be additionally performed.

The 6D pose estimation apparatus 100 according to the present invention improves the understanding of data including an object through the user-defined problem (pretext task) performing unit 140 using self-supervised learning, thereby improving the object pose estimation unit 130 ), it is possible to more accurately estimate the 6D pose for the object included in the image.

In addition, the 6D pose estimation apparatus 100 may use 6D pose information of the estimated object to enable not only a pick and place operation of the robot, but also a complex and various operation including assembly. For example, the 6D pose estimation apparatus 100 may determine the position and angle of the object using 6D information of the object.

5 is a flowchart of a 6D pose estimation method according to an embodiment of the present invention. The apparatus 100 for estimating a 6D pose shown in FIG. 1 includes steps that are time-series processed according to the embodiments shown in FIGS. 1 to 4 . Therefore, even if omitted below, it is also applied to the method of estimating the object pose in the 6D pose estimation apparatus 100 according to the embodiment shown in FIGS. 1 to 4 .

In step S510 , the 6D pose estimation apparatus 100 may input RGB data for the image into the first deep learning model and extract the external features.

In operation S520, the 6D pose estimation apparatus 100 may extract geometric features by inputting depth data of the image to the second deep learning model.

In step S530, the 6D pose estimation apparatus 100 may transfer-learning the second deep learning model based on the weight of the network learned by the user-defined problem (Pretext task).

In operation S540, the 6D pose estimation apparatus 100 may generate a feature map for the image by combining the extracted external features and geometric features.

In operation S550 , the 6D pose estimation apparatus 100 may estimate a 6D pose of an object corresponding to an image based on the generated feature map.

In the above description, steps S510 to S550 may be further divided into additional steps or combined into fewer steps according to an embodiment of the present invention. In addition, some steps may be omitted as needed, and the order between the steps may be switched.

The description of the present invention described above is for illustration, and those of ordinary skill in the art to which the present invention pertains can understand that it can be easily modified into other specific forms without changing the technical spirit or essential features of the present invention. will be. Therefore, it should be understood that the embodiments described above are illustrative in all respects and not restrictive. For example, each component described as a single type may be implemented in a dispersed form, and likewise components described as distributed may be implemented in a combined form.

The scope of the present invention is indicated by the following claims rather than the above detailed description, and all changes or modifications derived from the meaning and scope of the claims and their equivalent concepts should be interpreted as being included in the scope of the present invention. do.

100: 6D pose estimation device
110: feature extraction unit
120: coupling part
130: object pose estimation unit
140: User-defined problem (Pretext task) execution unit

Claims (20)

An apparatus for estimating an object pose using unsupervised learning, the apparatus comprising:
An appearance feature extraction unit for extracting external features by inputting RGB data for an image into a first deep learning model, and a geometric feature for extracting geometric features by inputting depth data for the image into a second deep learning model a feature extraction unit including an extraction unit;
a combining unit for generating a feature map for the image by combining the extracted external features and geometric features; and,
An object pose estimator for estimating a 6D pose of the object corresponding to the image based on the generated feature map
including,
The second deep learning model is transfer-trained based on the weight of the network learned by a user-defined problem (Pretext task), 6D pose estimation apparatus.
The method of claim 1,
The first deep learning model is a CNN (Convolutional Neural Network), and the second deep learning model is a point net (PointNet) model, 6D pose estimation apparatus.
The method of claim 1,
A user-defined problem (Pretext task) performing unit that performs the user-defined problem (Pretext task) based on the network
Further comprising a, 6D pose estimation apparatus.
4. The method of claim 3,
The user-defined problem (Pretext task) performing unit,
a modeling unit for modeling a tray model including a plurality of holes and an object model insertable into at least one of the plurality of holes;
a noise unit for adding noise to the point cloud of the modeled tray model and the object model; and
A learning unit that trains the network to determine a hole into which the object model is inserted by inputting it into the network while rotating the object model to which the noise is added
That comprising a, 6D pose estimation device.
5. The method of claim 4,
6D pose estimation apparatus, wherein the learning unit calculates an overlapping area and an empty area between the object model and the tray model as a loss.
The method of claim 1,
6D pose estimation apparatus, wherein the geometric feature extraction unit transfer learning the weight of the second deep learning model to the weight of the network.
The method of claim 1,
The object pose estimation unit,
6D pose estimation apparatus, which calculates a pose for the object based on a semantic segmentation module, a keypoint detection module, and a center voting module.
8. The method of claim 7,
The semantic segmentation module is
If there are a plurality of objects on the image based on the feature map, the 6D pose estimation apparatus will classify each object.
9. The method of claim 8,
The feature point detection module is
The 6D pose estimation apparatus that detects a 3D feature point (Keypoint) on the surface of the divided object.
10. The method of claim 9,
The center voting module,
6D pose estimation apparatus, which detects a center point of the object.
In a method for estimating an object pose using unsupervised learning,
extracting appearance features by inputting RGB data for the image into the first deep learning model;
a feature extraction step comprising extracting geometric features by inputting depth data for the image into a second deep learning model;
generating a feature map for the image by combining the extracted external features and geometric features; and,
estimating a 6D pose for the object corresponding to the image based on the generated feature map
including,
The second deep learning model is a 6D pose estimation method that is transfer-learned based on the weight of the network learned by a user-defined problem (Pretext task).
12. The method of claim 11,
The first deep learning model is a CNN (Convolutional Neural Network), and the second deep learning model is a point net (PointNet) model, 6D pose estimation method.
12. The method of claim 11,
6D pose estimation method, further comprising the step of performing the user-defined problem (Pretext task) based on the network.
14. The method of claim 13,
The step of performing the user-defined problem (Pretext task) is,
modeling a tray model including a plurality of holes and an object model insertable into at least one of the plurality of holes;
adding noise to the point cloud of the modeled tray model and the object model; and
training the network to determine a hole into which the object model is inserted by inputting it into the network while rotating the object model to which the noise has been added;
A method for estimating 6D poses, comprising:
15. The method of claim 14,
In the training of the network, an overlapping area and an empty area between the object model and the tray model are calculated as a loss.
12. The method of claim 11,
The step of extracting the geometrical features is to transfer the weight of the second deep learning model to the weight of the network, 6D pose estimation method.
12. The method of claim 11,
The step of estimating the 6D pose for the object comprises:
calculating a pose for the object based on a semantic segmentation module, a Keypoint detection module and a Center voting module.
18. The method of claim 17,
The semantic segmentation module is
When there are a plurality of objects on the image based on the feature map, the 6D pose estimation method of classifying each object.
19. The method of claim 18,
The feature point detection module is
The 6D pose estimation method of detecting a 3D feature point (Keypoint) for the surface of the divided object.
20. The method of claim 19,
The center voting module,
6D pose estimation method of detecting a center point of the object.

Images