CN114882113A - Five-finger mechanical dexterous hand grabbing and transferring method based on shape correspondence of similar objects - Google Patents
Five-finger mechanical dexterous hand grabbing and transferring method based on shape correspondence of similar objects Download PDFInfo
- Publication number
- CN114882113A CN114882113A CN202210560549.0A CN202210560549A CN114882113A CN 114882113 A CN114882113 A CN 114882113A CN 202210560549 A CN202210560549 A CN 202210560549A CN 114882113 A CN114882113 A CN 114882113A
- Authority
- CN
- China
- Prior art keywords
- shape
- grabbing
- point
- dexterous hand
- module
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Pending
Links
Images
Classifications
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
- G06T7/00—Image analysis
- G06T7/70—Determining position or orientation of objects or cameras
- G06T7/73—Determining position or orientation of objects or cameras using feature-based methods
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06N—COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS
- G06N3/00—Computing arrangements based on biological models
- G06N3/02—Neural networks
- G06N3/08—Learning methods
- G06N3/084—Backpropagation, e.g. using gradient descent
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
- G06T17/00—Three dimensional [3D] modelling, e.g. data description of 3D objects
- G06T17/20—Finite element generation, e.g. wire-frame surface description, tesselation
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
- G06T2207/00—Indexing scheme for image analysis or image enhancement
- G06T2207/10—Image acquisition modality
- G06T2207/10028—Range image; Depth image; 3D point clouds
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
- G06T2207/00—Indexing scheme for image analysis or image enhancement
- G06T2207/20—Special algorithmic details
- G06T2207/20081—Training; Learning
Abstract
The invention discloses a five-finger mechanical dexterous hand grabbing and transferring method based on shape correspondence of similar objects. On one hand, the method can realize the estimation of the grabbing pose of the same type of object only by marking one example of the same type of object, and greatly reduces the difficulty of grabbing and marking. On the other hand, the grabbing and marking of the single-instance object in the class can be completed quickly by manpower, and therefore the grabbing posture after the migration is ensured to accord with the grabbing operation habit of human beings.
Description
Technical Field
The invention relates to a method for estimating the grabbing pose of a five-finger mechanical dexterous hand, in particular to a method for grabbing and transferring the five-finger mechanical dexterous hand based on the shape correspondence of similar objects by utilizing a deep learning technology, which estimates the correspondence relation of the shapes of the similar objects and further realizes the grabbing pose transferring method of the five-finger mechanical dexterous hand aiming at different examples of the similar objects.
Background
With the development of robotics, intelligent service robots have been gradually applied to various aspects such as catering, medical services, and living care. However, due to the complexity and variety of human activity scenes, the variety of articles, and the different target sizes, shapes and materials, the existing two-finger manipulator with only a clamping function is difficult to complete complex service tasks. In recent years, humanoid dexterous hands represented by Shadow Hand have flexibility of human hands, can adapt to diversity of operated objects, and provide hardware guarantee for service robots to realize humanoid operation. However, for a multi-finger dexterous manipulator, due to the flexibility of the high-dimensional structure and the diversity of the shapes of the objects in the scene, it is very difficult to directly estimate the grabbing posture of the dexterous manipulator to the object.
In order to solve the problem, many researches implement simulation learning of a dexterous hand grabbing method by tracking information such as positions, angles, speeds, accelerations and the like of joints in the process of grabbing an object by a human hand and directly mapping the information to a mechanical dexterous hand. However, due to the diversity of the shapes of the objects and the complexity of the grasping mode, the method is difficult to expand to unknown environments and articles, and the autonomous operation of the robot is difficult to realize. In order to realize the autonomous grasping of the dexterous manipulator, many researches utilize the force closing criterion of the dexterous manipulator at a target contact point to optimize and solve grasping positions and grasping gestures, however, the methods only can ensure that the dexterous manipulator can stably grasp an object, but cannot ensure that the grasping gestures meet the grasping habits of human beings. Furthermore, such methods require the knowledge of accurate 3D models, which is difficult to obtain in practical capture scenarios. In recent years, the development of neural networks and deep learning techniques provides a new idea for the grabbing posture estimation of the smart manipulator. Due to the powerful feature extraction capability of deep learning techniques, some methods directly utilize deep neural networks to fit the correspondence between the shape of an input object and the output grasp pose. The method depends on labeling of a large number of grabbing postures, for a dexterous manipulator with high degree of freedom, manually labeling the grabbing postures of a large number of mechanical dexterous hands on a large number of objects is a huge and tedious project, the grabbing postures generated by the assistance of simulation software only use stable grabbing as a criterion, and a deep neural network is trained by using the data set, so that the estimated grabbing postures can not be ensured to accord with the grabbing operation habit of human beings.
Disclosure of Invention
Although the shapes of the objects are different in an actual scene, the contact positions and the grabbing modes for grabbing the objects in the same class are similar due to the similar functional structures and geometrical structures. Based on the method, the invention provides a five-finger mechanical dexterous hand grabbing pose migration method for different examples of similar objects, and the grabbing mode of the mechanical dexterous hand is migrated from one object (source object) to other objects (target objects) of the same type by taking the corresponding relation of the shapes of the similar objects as a bridge. In addition, the invention considers the grabbing migration as an optimization problem, designs a contact point migration objective function and an anti-collision objective function, and gradually optimizes the unreasonable grabbing pose through a micro-dexterous hand forward kinematics module.
In order to achieve the purpose, the technical scheme of the invention is as follows:
the five-finger mechanical dexterous hand grabbing and transferring method based on the shape correspondence of the similar objects comprises the following steps:
step 1, preprocessing a data set;
the data set consists of 3D model data for a plurality of object classes, each class containing objects of different shapes and stored in the form of a triangular mesh. The collected 3D model data set is divided into a training set and a test set for network training and testing, respectively. For each 3D model, the preprocessing process includes: rendering the 3D model into point clouds under different viewing angles; a directed Distance Field (SDF) is generated covering the object spatial sample point.
Step 2, building an intra-class shape correspondence estimation network;
the intra-class shape correspondence estimation network comprises three sub-modules: the device comprises a shape coding module, a shape deformation module and a shape reconstruction module. And the shape coding module codes the input single view point cloud into a shape implicit vector. The shape deformation module deforms the object space sampling points of the input model to the corresponding points of the similar inner shape template space according to the shape implicit vectors, and based on the deformation, the shape dense correspondence between the similar different-shape objects can be established through the similar inner shape template. The shape decoding module decodes the deformed sampling points into the SDF values of the points, and the complete three-dimensional shape of the object can be reconstructed by using the covering object space sampling points and the SDF values thereof. The three sub-modules are all fully connected networks.
the training process comprises two steps: firstly, a shape deformation module and a shape reconstruction module are trained in a combined mode, the shape implicit vector and an object space sampling point are input into a network, the deformation amount from the sampling point to a corresponding point in a template space is output in the middle, then the deformed sampling point is input into the shape reconstruction module, and finally an SDF value corresponding to an original sampling point is output. The shape implicit vector is initialized randomly and updated along with the network parameters. And secondly, training a shape coding module, wherein the input of the module is a single view point cloud of the 3D model, and the output is a shape implicit vector of the 3D model obtained by training in the first step.
Step 4, marking the grabbing pose of the dexterous hand on the source object;
for each category in the 3D model data set, selecting an object from the training set as a source object, and manually marking the grabbing poses of a plurality of mechanical dexterous hands on the source object, wherein the grabbing poses comprise six-degree-of-freedom poses of wrists and rotation angles of finger joints. For each grabbing pose g marked on the source object i Firstly, acquiring the position g of a dexterous hand through forward kinematics i Calculating the SDF value of the point cloud relative to the source object by utilizing the in-class shape correspondence estimation network in the step 2, wherein the point mark with the SDF absolute value smaller than a set threshold valueSet of points marked as contact with an object on a dexterous hand
Points with SDF values greater than a set threshold are marked as a set of points on the dexterous hand that do not contact the object
Marking a set of points contacted by a dexterous hand on a source object at the same time
Point set
And
each point in (a) is one-to-one. Followed by
And
and migrating the grabbing pose used in the step 5.
Step 5, transferring the grabbing pose marked on the source object to the target object;
and the grabbing pose migration is to migrate the grabbing marks on the source object in the step 4 to other objects of the same type and different shapes, and the grabbing marks are input into the single-view point cloud of the other objects of the same type, which is acquired by the depth sensor. Firstly, the method reconstructs the object by the input single-view-point cloud through the in-class shape correspondence estimation network in the step 2, establishes the correspondence relation between the reconstructed object and the source object, and migrates the point set contacted by the dexterous hand on the source object to the reconstructed target object on the basis. And then, realizing grabbing pose migration by using a micro dexterous hand forward kinematics module and two objective functions in an iterative optimization method.
The micro dexterous hand forward kinematics module and the two objective functions are specifically as follows:
(1) the micro positive kinematics module of the dexterous hand obtains the position of each part of the dexterous hand under a wrist coordinate system by inputting the grabbing pose of the dexterous hand. The module is mainly used for carrying out back propagation on the gradient of the objective function when the grabbing pose is migrated so as to gradually optimize the less-reasonable grabbing pose. This module can be described as follows:
wherein, P H Representing a clever hand point cloud in an initial state,
is shown in pose g i And (5) the following clever hand point clouds.
(2) The contact point migration objective function L transfer The definition is as follows:
wherein the content of the first and second substances,
the step 4 is performed by the shape dense correspondence of the source object and the target object
Migration to the target object, wherein the shape dense correspondence is established by the shape morphing module described in step 2.
Is the set of points on the dexterous hand contacted with the object in step 4. Point set
And
all of which comprise N points, and the number of the points,
and
respectively represent
And
point j in (d). The contact point migration target function can adjust the rotation angle of the finger joints of the dexterous hand, so that the dexterous hand is close to the set of the contact points on the surface of the target object, the shape of the dexterous hand attached to the target object is made, and the grabbing success rate is improved.
(3) The anti-collision objective function L collision The definition is as follows:
wherein the content of the first and second substances,
is the point set which is not contacted with the object on the dexterous hand in the step 4
Position after passing through the forward kinematics module. Point set
The method comprises the steps of containing M points,
indicating the jth point therein. sdf (-) means use of step 2The intra-class shape correspondence estimation network solves the SDF value of the input point for the target object. The anti-collision target function can adjust the translation component of the wrist of the dexterous hand and the rotation angle of the finger joint, and effectively avoids collision or permeation between the dexterous hand and a target object.
In summary, the capture pose migration can be defined as the following optimization problem:
wherein λ is transfer 、λ collision Is the weight of the objective function.
In the optimization process, in order to fully play the respective functions of the contact point migration objective function and the anti-collision objective function, the invention designs the following optimization strategies: firstly, the anti-collision objective function is independently used for adjusting the translation component of the wrist of the dexterous hand so as to ensure that the whole dexterous hand is at a reasonable grabbing position. And then adjusting the rotation angle of the finger joint of the dexterous hand by using the two objective functions together to enable the finger to be attached to the surface of the target object.
The invention has the beneficial effects that:
the invention designs a five-finger mechanical dexterous hand grabbing and transferring method based on shape correspondence of similar objects. On one hand, the method can realize the estimation of the grabbing pose of the same type of object only by marking one example of the same type of object, and greatly reduces the difficulty of grabbing and marking. On the other hand, the grabbing and marking of the single-instance object in the class can be completed quickly by manpower, and therefore the grabbing posture after the migration is ensured to accord with the grabbing operation habit of human beings.
Drawings
FIG. 1 is a flow chart of the method steps of the present invention.
Fig. 2 is a diagram of an intra-class correspondence estimation network architecture of the present invention.
FIG. 3 is a general diagram of the grab migration method of the present invention.
Fig. 4 (a) and (b) are a capture label display diagram and a capture migration result display diagram, respectively, in an embodiment of the present invention.
Detailed Description
The following further describes a specific embodiment of the present invention by taking a mug and a bottle category as an example, in combination with the attached drawings and technical solutions.
As shown in fig. 1, the method for grabbing and transferring a five-finger mechanical dexterous hand based on the shape correspondence of the same kind of objects comprises the following steps:
step 1, preprocessing a data set. The object 3D model dataset employs the mug and bottle classes of the sharenetcore dataset, each class containing differently shaped object 3D models, the same class models being placed in the same orientation and normalized to unit space. The model is stored in the form of a triangular mesh. The collected 3D model data set is divided into training and testing sets at a ratio of 6:1 for network training and testing, respectively. For each 3D model, the preprocessing process includes:
(1) rendering point clouds of the 3D model under 100 different visual angles;
(2) a directed Distance Field (SDF) covering 50000 sample points in object space is generated, each sample point corresponds to an SDF value, the SDF represents the closest Distance of the sample point to the surface of the object, the SDF value is positive when the sample point is outside the object, the SDF value is negative when the sample point is inside the object, and the SDF value is zero when the sample point is on the surface of the object.
And 2, building an intra-class shape correspondence estimation network. As shown in fig. 2, the intra-class shape correspondence estimation network includes three sub-modules: the device comprises a shape coding module, a shape deformation module and a shape reconstruction module. And the shape coding module codes the input single view point cloud into a shape implicit vector. Specifically, for a single-view point cloud with n points, firstly, the single-view point cloud is sent to a Multi-Layer perceptron (MLP) sharing weight, characteristics of each point are extracted, then the characteristics extracted from the n points are subjected to characteristic fusion through a maximum pooling Layer, and finally, a 128-dimensional shape implicit vector is output after the fused characteristics are subjected to another MLP. And the shape deformation module deforms the object space sampling points of the input model to corresponding points of the similar inner shape template space according to the shape implicit vectors, and based on the deformation, the dense correspondence between objects with different shapes in the class can be established through the shape template. Specifically, m points are sampled in a normalized space, and then the m points and a 128-dimensional shape implicit vector output by a shape coding module are sent into an MLP together to obtain the deformation from a sampling point to a corresponding point of a template, so that the corresponding point of the sampling point in the template space is found. And the shape decoding module decodes the deformed sampling point into an SDF value of the sampling point through another MLP, and then obtains the complete three-dimensional shape of the reconstructed object by using a Marching cube algorithm.
And 3, training the intra-class shape correspondence estimation network in the step 2. The training process comprises two steps: firstly, training a shape deformation module and a shape reconstruction module in a combined manner, inputting a shape implicit vector and sampling points obtained through preprocessing in the step 1 into a network, outputting the deformation amount of the sampling points to corresponding points of a template space in the middle, inputting the deformed sampling points into the shape reconstruction module, and finally outputting the deformed sampling points as SDF values corresponding to original sampling points. The shape implicit vector is randomly initialized using the normal distribution N (0,0.01) and updated along with the network parameters. The loss functions used include: SDF value constraint L sdf Normal constraint L normal Shape implicit vector regularization term L code Normal consistency constraint on deformation L deform_normal And smoothness constraint L deform_smooth The concrete formula is as follows.
Where Ω represents the 3D space in which the object is located, S represents the object shape surface, and S pred And s gt Respectively representing the predicted value and the true value of the SDF, wherein lambda is a penalty coefficient, and lambda > 1; n is pred Representing the normal direction of the object surface, n, derived from the coordinates of the sampling points by means of the SDF predictor gt A true value representing the normal direction; alpha represents a shape implicit vector;
expressing the normal direction obtained by utilizing the SDF prediction value to derive the coordinate of the deformed sampling point; delta pred The deformation amount corresponding to the sampling point p is represented by { Δ x, Δ y, Δ z }.
And secondly, a training shape coding module inputs the single view point cloud obtained by preprocessing in the step 1 and outputs a shape implicit vector of the 3D model obtained by training in the first step. The loss function is a shape implicit vector loss L α The formula is as follows. Wherein alpha is pred And alpha gt Respectively representing the predicted value and the true value of the shape implicit vector.
L α ＝|α pred -α gt |
The two steps of training are completed on an NVIDIA GTX 1080 video card, an Adam optimizer is used in the training process, and the initial learning rates are all 1 e-4. After the first-step training is converged, evaluating a training result by calculating chamfer angles of the reconstructed three-dimensional shape and a true value of the three-dimensional shape, and taking a network parameter with the minimum chamfer angle as a final model of a shape deformation module and a shape reconstruction module; and after the second step of training is converged, evaluating a training result by calculating the predicted value of the shape implicit variable and the L1 distance of the true value, and taking the network parameter with the minimum L1 distance as a final model of the shape coding module.
And 4, marking the grabbing pose of the dexterous hand on the source object. For each category in the 3D model dataset, selecting an object from the training set as a source object, manually labeling a plurality of grabbing poses on the source object by using a Shadow Hand dexterous Hand, wherein the grabbing poses comprise six-degree-of-freedom poses [ R, t ] of a wrist]And a rotation angle theta of 22 finger joints. Then, the grabbing marks are guaranteed to be successfully grabbed under the simulation environment. Capture of note as shown in fig. 4 (a), the simulation environment used in this example is Isaac Gym. In addition, for each grabbing pose g marked on the source object i Firstly, acquiring the position g of a dexterous hand through forward kinematics i And (3) calculating the SDF value of the point cloud to the source object by utilizing the network in the step 2 (the shape implicit vector of the source object is known), wherein the points with the SDF absolute value smaller than the threshold value (5e-3) are marked as a point set contacted with the object on the dexterous hand
Points with an SDF value greater than a threshold value (1e-2) are marked as a set of points on a dexterous hand that do not contact an object
Marking a set of points contacted by a dexterous hand on a source object at the same time
Point set
And
each point in (a) is one-to-one. Followed by
And
and migrating the grabbing pose used in the step 5.
And 5, transferring the grabbing pose mark of the source object to the target object.
And the grabbing attitude migration is to migrate the grabbing marks on the source object in the step 4 to other objects of the same type and different shapes, and the grabbing marks are input into the single-view point cloud of the other objects of the same type, which is acquired by the depth sensor. Firstly, the input single-view-point cloud is subjected to reconstruction of the object through the network in the step 2, the corresponding relation between the reconstructed object and the source object is established, and on the basis, the contact point set on the source object is transferred to the reconstructed target object. Then, grabbing pose migration is realized by an iterative optimization method by utilizing a micro dexterous hand forward kinematics module and two objective functions, as shown in fig. 3. The micro dexterous hand forward kinematics module and the two objective functions are specifically as follows:
(1) the micro positive kinematics module of the dexterous hand obtains the position of each part of the dexterous hand under a wrist coordinate system by inputting the grabbing pose of the dexterous hand. The module is mainly used for carrying out back propagation on the gradient of the objective function when the grabbing pose is migrated so as to gradually optimize the less-reasonable grabbing pose. This module can be described as follows:
wherein, P H Representing a clever hand point cloud in an initial state,
show the grabbing pose g i The following clever hand point cloud, as shown in fig. 3. The forward kinematics module is implemented using a Pytorch frame.
(2) The contact point migration objective function L transfer The definition is as follows:
wherein the content of the first and second substances,
the step 4 is performed by the shape dense correspondence of the source object and the target object
Migration to the target object, wherein the shape dense correspondence is established by the shape morphing module described in step 2.
Is the set of points on the dexterous hand contacting the object described in step 4. Point set
And
all of which comprise N points, and the number of the points,
and
respectively represent
And
point j in (d). The contact point migration target function can adjust the rotation angle of the finger joints of the dexterous hand, so that the dexterous hand is close to the set of the contact points on the surface of the target object, the shape of the dexterous hand attached to the target object is made, and the grabbing success rate is improved.
(3) The anti-collision objective function L collision The definition is as follows:
wherein the content of the first and second substances,
is the non-contact point set of the dexterous hand point cloud in the step 4
Position after passing through the forward kinematics module. Point set
The method comprises the steps of containing M points,
indicating the jth point therein. SDF (-) refers to solving the SDF value of the input point for the target object using the network described in step 2. The anti-collision target function can adjust the translation component of the wrist of the dexterous hand and the rotation angle of the finger joint, and effectively avoids collision or permeation between the dexterous hand and a target object.
In summary, the capture pose migration can be defined as the following optimization problem:
wherein λ is transfer 、λ collision Is the weight of the objective function. In the optimization process, in order to fully play the respective functions of the contact point migration objective function and the anti-collision objective function, the invention designs an optimization strategy which comprises the following steps: the anti-collision objective function is used alone to adjust the translation component t of the wrist first to ensure that the whole body of the Shadow Hand dexterous Hand is in a reasonable gripping position. And then, adjusting the rotation angle theta of the finger joint in the grabbing pose by using the two objective functions together to enable the finger to be attached to the surface of the target object. The optimization process uses an Adam optimizer, the learning rate is set to be 1e-3, and the iterative optimization times are 200.
And 6, testing the capturing success rate of the captured pose after the migration in the simulation environment.
In the embodiment, Isaac Gym simulation environment is adopted for test grabbing, and the environment can be used for testing a plurality of objects in parallel. The experimental setup was as follows: the method comprises the following steps that in the first stage, a simulation environment is set to be zero gravity, a 3D model of a target object is placed in the air, then a dexterous hand is arranged according to a grabbing pose, penetration detection can be carried out at the moment, when the dexterous hand and the object are greatly penetrated, the object can be bounced off by the dexterous hand, and grabbing fails; and 3 seconds later, the second stage is carried out, gravity is recovered, the grabbing stability is detected, and the object can fall off when the grabbing stability is weaker. The number of successful snatchs can be counted after 10 seconds of adding gravity. The invention judges whether the grabbing is successful according to the variation of the pose of the object before and after the object is grabbed. The amount of change in position is less than Δ t only before and after the object is grasped max While the angle variation is smaller than Delta theta max The grabbing is regarded as successful. Amount of positional change Δ t of object obj And the amount of change in angle Δ θ obj The specific calculation formula of (2) is as follows.
Wherein the content of the first and second substances,
a rotation matrix and a translation vector representing the initial pose of the object,
and the rotation matrix and the translation vector of the final pose of the object are represented. tr (-) denotes the trace of the matrix.
Table 1 shows the quantitative evaluation results of two objective functions, the contact point migration objective function and the anti-collision objective function. The invention provides the maximum deviation delta t of the position of the target object max Is 3 cm and the maximum angle deviation Delta theta max The grabbing success rates are respectively 5 degrees, 15 degrees and 25 degrees.The strategy of not adopting any objective function means that the grabbing marked on the source object is directly copied to the target object. Compared with direct copy grabbing, the method has the advantages that the grabbing success rate is effectively improved when two objective functions are independently used for optimization; in addition, the performance is obviously improved under the condition of the contact point migration objective function. When two objective functions are simultaneously used for optimization, the capturing success rate reaches the optimal performance. Fig. 4 (b) shows the successful capturing result of the part of the object with the posture offset smaller than 5 cm 3.
TABLE 1
Claims (1)
1. The five-finger mechanical dexterous hand grabbing and transferring method based on the shape correspondence of similar objects is characterized by comprising the following steps of:
step 1, preprocessing a data set;
the data set is composed of 3D model data of a plurality of object categories, each category contains objects of different shapes and is stored in a triangular mesh form; dividing the collected 3D model data set into a training set and a testing set, and respectively using the training set and the testing set for network training and testing; for each 3D model, the preprocessing process includes: rendering the 3D model into point clouds under different viewing angles; generating a directed distance field SDF covering a spatial sample point of an object;
step 2, building an intra-class shape correspondence estimation network;
the intra-class shape correspondence estimation network comprises three sub-modules: the shape reconstruction module is used for reconstructing the shape of the object; the shape coding module codes the input single view point cloud into a shape implicit vector; the shape deformation module deforms the object space sampling points of the input model to the corresponding points of the similar inner shape template space according to the shape implicit vectors, and based on the deformation, the shape dense correspondence between objects of the same type with different shapes is established through the similar inner shape template; the shape decoding module decodes the deformed sampling point into the SDF value of the point, and reconstructs the complete three-dimensional shape of the object by using the spatial sampling point of the covered object and the SDF value thereof; the three sub-modules are all fully connected networks;
step 3, training the intra-class shape correspondence estimation network in the step 2;
the training process comprises two steps: firstly, training a shape deformation module and a shape reconstruction module in a combined manner, wherein the input of a network is a shape implicit vector and an object space sampling point, the deformation from the sampling point to a corresponding point in a template space is output in the middle, then the deformed sampling point is input into the shape reconstruction module, and finally an SDF value corresponding to an original sampling point is output; the shape implicit vector is initialized randomly and is updated along with the network parameters; secondly, training a shape coding module, wherein the input of the module is a single view point cloud of the 3D model, and the output is a shape implicit vector of the 3D model obtained by training in the first step;
step 4, marking the grabbing pose of the dexterous hand on the source object;
for each category in the 3D model data set, selecting an object from the training set as a source object, and manually marking the grabbing poses of a plurality of mechanical dexterous hands on the source object, wherein the grabbing poses comprise six-degree-of-freedom poses of wrists and rotation angles of finger joints; for each grabbing pose g marked on the source object i Firstly, acquiring the position g of a dexterous hand through forward kinematics i Calculating the SDF value of the point cloud relative to the source object by utilizing the in-class shape correspondence estimation network in the step 2, wherein the points with the SDF absolute value smaller than a set threshold are marked as a point set P contacted with the object on the dexterous hand i C Points with SDF values greater than a set threshold are marked as a set of points on the dexterous hand that do not contact the object
While marking the set of points touched by the dexterous hand on the source object as P i O Set of points P i C And P i O Each point in (a) is in one-to-one correspondence;
step 5, transferring the grabbing pose marked on the source object to the target object;
the grabbing pose migration is to migrate the grabbing marks on the source object in the step 4 to other objects of the same type and different shapes, and the grabbing marks are input into a single viewpoint point cloud of the other objects of the same type, which is acquired by a depth sensor; firstly, reconstructing an object by an input single-view-point cloud through the in-class shape correspondence estimation network in the step 2, establishing a correspondence relation between the reconstructed object and a source object, and transferring a point set contacted by a dexterous hand on the source object to a reconstructed target object on the basis; then, realizing grabbing pose migration by using a micro dexterous hand forward kinematics module and two objective functions in an iterative optimization method;
the micro dexterous hand forward kinematics module and the two objective functions are specifically as follows:
(1) the micro positive kinematics module of the dexterous hand obtains the position of each part of the dexterous hand under a wrist coordinate system by inputting the grabbing pose of the dexterous hand; the module is used for propagating the gradient of the objective function in the reverse direction during the capture pose migration so as to gradually optimize the less-reasonable capture pose, and is described as follows:
P i H ＝FK(g i ,P H )
wherein, P H Representing a clever hand point cloud in the initial state, P i H Is shown in pose g i The next clever hand point cloud;
(2) contact point migration objective function L transfer The definition is as follows:
wherein Q is i O P in step 4 is determined by the dense correspondence of the shapes of the source object and the target object i O Obtained by migration to a target object, wherein the shape is denseThe correspondence is established by the shape deformation module of step 2; p i C Is the set of points on the dexterous hand contacted with the object in the step 4; point set
And P i C All of which comprise N points, and the number of the points,
and
respectively represent
And P i C J-th point in (1);
(3) collision avoidance objective function L collision The definition is as follows:
wherein the content of the first and second substances,
is the point set which is not contacted with the object on the dexterous hand in the step 4
Position after passing through the forward kinematics module; point set
The method comprises the steps of containing M points,
represents the jth point therein; SDF (-) refers to solving the SDF value of the input point for the target object using the intra-class shape correspondence estimation network described in step 2;
in summary, the capture pose migration is defined as the following optimization problem:
wherein λ is transfer 、λ collision Is the weight of the objective function;
in the optimization process, firstly, the anti-collision objective function is independently used for adjusting the translation component of the wrist of the dexterous hand so as to ensure that the whole dexterous hand is at a reasonable grabbing position; and then adjusting the rotation angle of the finger joint of the dexterous hand by using the two objective functions together to enable the finger to be attached to the surface of the target object.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202210560549.0A CN114882113A (en) | 2022-05-23 | 2022-05-23 | Five-finger mechanical dexterous hand grabbing and transferring method based on shape correspondence of similar objects |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202210560549.0A CN114882113A (en) | 2022-05-23 | 2022-05-23 | Five-finger mechanical dexterous hand grabbing and transferring method based on shape correspondence of similar objects |
Publications (1)
| Publication Number | Publication Date |
|---|---|
| CN114882113A true CN114882113A (en) | 2022-08-09 |
Family
ID=82677711
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN202210560549.0A Pending CN114882113A (en) | 2022-05-23 | 2022-05-23 | Five-finger mechanical dexterous hand grabbing and transferring method based on shape correspondence of similar objects |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN114882113A (en) |
Cited By (1)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN116330290A (en) * | 2023-04-10 | 2023-06-27 | 大连理工大学 | Multi-agent deep reinforcement learning-based five-finger smart robot control method |
-
2022
- 2022-05-23 CN CN202210560549.0A patent/CN114882113A/en active Pending
Cited By (2)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN116330290A (en) * | 2023-04-10 | 2023-06-27 | 大连理工大学 | Multi-agent deep reinforcement learning-based five-finger smart robot control method |
| CN116330290B (en) * | 2023-04-10 | 2023-08-18 | 大连理工大学 | Multi-agent deep reinforcement learning-based five-finger smart robot control method |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| Newbury et al. | Deep learning approaches to grasp synthesis: A review | |
| CN108972494B (en) | Humanoid manipulator grabbing control system and data processing method thereof | |
| Schmidt et al. | Grasping of unknown objects using deep convolutional neural networks based on depth images | |
| Watanabe et al. | Survey of robotic manipulation studies intending practical applications in real environments-object recognition, soft robot hand, and challenge program and benchmarking | |
| Sahbani et al. | An overview of 3D object grasp synthesis algorithms | |
| WO2018221614A1 (en) | Learning device, learning method, learning model, estimation device, and grip system | |
| Morales et al. | Integrated grasp planning and visual object localization for a humanoid robot with five-fingered hands | |
| CN106346485B (en) | The Non-contact control method of bionic mechanical hand based on the study of human hand movement posture | |
| Alonso et al. | Current research trends in robot grasping and bin picking | |
| JP2019508273A (en) | Deep-layer machine learning method and apparatus for grasping a robot | |
| Chella et al. | A cognitive framework for imitation learning | |
| Lévesque et al. | A model-based scooping grasp for the autonomous picking of unknown objects with a two-fingered gripper | |
| Bohg et al. | Task-based grasp adaptation on a humanoid robot | |
| Valarezo Anazco et al. | Natural object manipulation using anthropomorphic robotic hand through deep reinforcement learning and deep grasping probability network | |
| Huang et al. | Grasping novel objects with a dexterous robotic hand through neuroevolution | |
| Aktaş et al. | Deep dexterous grasping of novel objects from a single view | |
| Khargonkar et al. | Neuralgrasps: Learning implicit representations for grasps of multiple robotic hands | |
| CN114882113A (en) | Five-finger mechanical dexterous hand grabbing and transferring method based on shape correspondence of similar objects | |
| Vinayavekhin et al. | Towards an automatic robot regrasping movement based on human demonstration using tangle topology | |
| El-Khoury et al. | Learning the natural grasping component of an unknown object | |
| Song et al. | Learning optimal grasping posture of multi-fingered dexterous hands for unknown objects | |
| JP7051751B2 (en) | Learning device, learning method, learning model, detection device and gripping system | |
| Aleotti et al. | Grasp programming by demonstration in virtual reality with automatic environment reconstruction | |
| CN114700949B (en) | Mechanical arm smart grabbing planning method based on voxel grabbing network | |
| CN115446835A (en) | Rigid-soft humanoid-hand autonomous grabbing method based on deep learning |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination |