CN115810219A - Three-dimensional gesture tracking method based on RGB camera - Google Patents

Abstract

The present invention proposes a three-dimensional gesture tracking method based on an RGB camera. The method includes the following steps: taking the RGB camera as input, standardizing the picture, and sending it to the three-dimensional joint point detection module, extracting image features, and predicting the preliminary three-dimensional joint points and calculate the length of the hand bones; use the particle swarm optimization algorithm to iteratively fit the length of the hand bones and the MAMO hand model to find the optimal hand shape and obtain a new 3D joint point; send the new 3D joint point to Enter the reverse analytical kinematics module to solve the posture parameters required by the MAMO model; use the MANO model to combine the optimal hand shape and posture parameters θ to obtain the final joint points and grid vertices; reconstruct and render the obtained grid vertices and joint points Output to get the final 3D real-time tracking pose of the hand. The method of the invention not only has high prediction accuracy and high precision of three-dimensional gesture tracking effect, but also the tracking process and results are easy to realize.

Description

A 3D Gesture Tracking Method Based on RGB Camera

technical field

The invention belongs to the technical field of gesture recognition, in particular to a three-dimensional gesture tracking method based on an RGB camera.

Background technique

The hand is one of the most frequently used parts of the human body in real life, and it is also one of the most dynamic parts. Hands can express a variety of postures, thereby conveying a wealth of information. Capturing the motion state of the hand is very important for various applications in the fields of virtual reality, augmented reality, human-computer interaction and so on. Therefore, in recent years, many researchers have started to carry out research in this area, and have made some progress.

Due to the wide application of depth cameras, many early researchers estimated the pose of the hand by fitting the generated model to the depth image based on the depth image. Tompson et al. combined CNN with random decision forests and inverse kinematics to estimate hand pose in real time from a single depth image. Wan et al. utilized unlabeled depth images for self-supervised fine-tuning, while Mueller et al. constructed a photorealistic dataset for better robustness. There are also some researchers who separate point clouds and 3D voxels from depth images for research.

Due to the high price of depth sensors, high power consumption, and strict requirements on the experimental environment, more and more people have begun to study 3D hand pose estimation based on monocular RGB images. Zimmermann and Brox train a CNN-based model that estimates 3D joint coordinates directly from RGB images. Iqbal et al. used a 2.5D heatmap formulation that encodes 2D joint locations together with depth information, which greatly improves accuracy. Many researchers exploit deep image datasets to amplify the diversity seen during training. Mueller et al. propose a large-scale rendering dataset post-processed by CycleGAN to bridge the domain gap. However, they only focus on joint position estimation without performing joint rotation recovery. Ge et al. used GraphCNN to directly regress hand meshes, but required a special dataset with real hand meshes. This model-free approach works moderately for challenging scenarios. This approach works well for fully utilizing existing datasets from different modalities, including image data and non-image data. Using image data with 2D or 3D annotations, and 3D animation images without corresponding image data, Zhou et al. proposed a 3D hand joint detection module and an inverse kinematics module, which not only regressed the 3D joint positions, but also Focusing on the joint rotation problem, this method has certain prospects in the field of computer vision and graphics applications. However, Zhou's method does not take into account the optimal matching of 3D joint points and MANO's hand model, and the algorithm complexity of the inverse kinematics module is slightly high, which is not easy to implement.

Among the above methods, some use depth images and some use RGB color images, but on the one hand, they do not make full use of the image features of 2D image information and 3D image information, and on the other hand, because the network model is not advanced enough or the method is too complicated, resulting in hand The prediction accuracy of 3D coordinates is not very high or the effect of 3D gesture tracking is average. At the same time, there are problems such as hand occlusion and poor real-time gesture tracking.

Contents of the invention

The main purpose of the present invention is to design a three-dimensional gesture tracking method to improve prediction accuracy and obtain high-precision three-dimensional gesture tracking effect, and at the same time, the tracking process and results are easy to realize.

To achieve the above object, the present invention provides a three-dimensional gesture tracking method based on an RGB camera, comprising the following steps:

Step 1. Use the RGB camera as input, standardize the image, send the processed image to the 3D joint point detection module, extract image features, use the convolutional neural network to predict the initial 3D joint point and calculate the length of the hand bones;

Step 2. Use the particle swarm optimization algorithm to iteratively fit the hand bone length and the MAMO hand model to find the optimal hand shape and obtain a new 3D joint point;

Step 3. Send the new 3D joint points to the inverse analytical kinematics module to solve the posture parameter θ required by the MAMO model;

Step 4, use the MANO model to combine the optimal hand shape and posture parameters θ to obtain the final joint points and mesh vertices; and

Step 5: Reconstruct and render the obtained mesh vertices and joint points to obtain the final 3D real-time tracking pose of the hand.

A further improvement of the present invention is that the three-dimensional joint point detection module uses a neural network model based on ResNet50, including a feature extractor, a 2D detector and a 3D detector, and uses ResNet50 adding an attention mechanism as a feature extractor, and the input is a resolution The image rate is 128×128, and the output feature volume F is 32×32×256.

A further improvement of the present invention is that the 2D detector is a two-layer CNN, which acquires the feature body F and outputs the heat map H of 21 joints, and the heat map H is used for 2D pose estimation; the 3D detector first uses 2 layers CNN estimates delta map D from heatmap H and feature volume F, heatmap H, feature volume F, and delta map D are concatenated and fed into another 2-layer CNN to obtain the final location map L, with location The form of Figure L estimates 3D hand joint positions.

A further improvement of the present invention is that the MANO model in step 4 is a 3D parametric model, which forms a complete hand chain based on 16 joint points and 5 fingertip points obtained from vertices.

The further improvement of the present invention is that in the fifth step, open3D is used to reconstruct the vertices of the hand mesh.

是一个手掌摊平的姿势，通过模板

形状函数B_s(β)和姿势函数B_p(θ)，可以得到手部形变模板T，然后结合姿势参数θ，蒙皮权重ω，关节位置J(θ)，对其进行蒙皮操作，数学表达式如下：The further improvement of the present invention is that, in the MANO model

is a palm-flat pose, through the template

The shape function B _s (β) and the posture function B _p (θ) can obtain the hand deformation template T, and then combine the posture parameters θ, skin weight ω, and joint position J(θ) to perform skinning operations on it. Mathematics The expression is as follows:

M(θ,β)=W(T(θ,β),θ,ω,J(θ)).

The further improvement of the present invention is that the particle swarm optimization algorithm first initializes a group of random particles, and then finds the optimal solution through multiple iterations. In each iteration, the particles update themselves by tracking two extreme values. After the optimal value, the particle updates its speed and position by the following formula:

是粒子i在第k次迭代中第d维的速度向量，

是粒子i在第k次迭代中第d维的位置向量，

是粒子i在第k次迭代中第d维的历史最优位置，既在第k次迭代之后，第i个粒子搜索得到的最优解，

是群体在第k次迭代中第d维的历史最优位置，即在第k次迭代后，整个粒子群体中的最优解。Among them, i=1, 2,..., N, N is the particle swarm scale, d is the particle dimension sequence number, k is the number of iterations, ω is the inertia weight, c ₁ is the individual learning factor, c ₂ is the group learning factor, r ₁ and r ₂ are random numbers in the interval [0, 1], increasing the randomness of the search,

is the d-dimensional velocity vector of particle i in the k-th iteration,

is the d-th dimension position vector of particle i in the k-th iteration,

is the historical optimal position of particle i in the d-th dimension in the k-th iteration, that is, the optimal solution obtained by the i-th particle search after the k-th iteration,

is the historical optimal position of the swarm in the d-th dimension in the k-th iteration, that is, the optimal solution in the entire particle swarm after the k-th iteration.

Beneficial effects of the present invention: use the combination of particle swarm optimization algorithm and MANO model to restore the shape of the hand, improve the accuracy of gesture tracking, and have better tracking effects for self-occlusion of hand movement and object occlusion. Analytical inverse kinematics is used to solve the posture parameters, and the algorithm structure is optimized. The complexity of the whole method is controlled, and it has better real-time performance. Finally, high-precision real-time 3D gesture tracking is achieved.

Description of drawings

FIG. 1 is an overall framework diagram of the 3D gesture tracking method based on an RGB camera in the present invention.

Fig. 2 is a network model diagram of the three-dimensional joint point detection module in the present invention.

Fig. 3 is an experimental effect diagram of the present invention.

Detailed ways

In order to make the object, technical solution and advantages of the present invention clearer, the present invention will be described in detail below in conjunction with the accompanying drawings and specific embodiments.

It should be emphasized that in the process of describing the present invention, various formulas and constraints are distinguished by using consistent labels, but it does not exclude the use of different labels to mark the same formulas and/or constraints. The purpose of such setting is In order to illustrate the features of the present invention more clearly.

The present invention proposes a gesture tracking method based on a convolutional neural network, adding a particle swarm optimization algorithm, and combining reverse analytical kinematics. The method of the present invention uses 2D image data and 3D image data, which can better utilize image features to train the model, preprocess the original data and send it to the convolutional neural network that adds the attention mechanism, and the obtained hand 3D The accuracy of the joint points is high, combined with the particle swarm optimization algorithm to iteratively fit the MANO model to find the optimal shape of the hand, and then use analytic inverse kinematics to deduce the hand posture parameters to obtain the hand required by the MANO model parameter. This method combined with inverse analytical kinematics can make better use of hand motion characteristics, and at the same time perform PSO iterative fitting on bone length and MANO hand model parameter files to find the optimal hand shape. On the premise of controlling the complexity of the algorithm, the high-precision three-dimensional gesture tracking effect is finally realized.

The three-dimensional gesture tracking method based on the RGB camera of the present invention mainly comprises the following steps:

Step 1. Obtain the video stream, process the picture, send the processed picture to the 3D joint point detection module, extract image features, use the convolutional neural network to predict the preliminary 3D joint point and calculate the length of the hand bones;

Step 2. Use the particle swarm optimization algorithm to iteratively fit the hand bone length and the MAMO hand model to find the optimal hand shape and obtain a new 3D joint point;

Step 3. Send the new 3D joint points to the inverse analytical kinematics module to solve the posture parameter θ required by the MAMO model;

Step 4, use the MANO model to combine the optimal hand shape and posture parameters θ to obtain the final joint points and mesh vertices; and

Step 5: Reconstruct and render the obtained mesh vertices and joint points to obtain the final 3D real-time tracking pose of the hand.

The present invention will be described in detail below in conjunction with the accompanying drawings.

Step 1. Get the video stream, process the picture, send the processed picture to the 3D joint point detection module, extract the image features, use the convolutional neural network to predict the preliminary 3D joint point and calculate the length of the hand bones.

In step 1, the RGB three-channel camera is used as input, the image is cropped and normalized, the processed image is sent to the 3D joint point detection module, image features are extracted, and the convolutional neural network is used to predict the preliminary hand 3D joint points and calculate bone length. Among them, the hardware device uses an ordinary RGB three-channel camera, takes the form of video stream as input, processes the video stream frame by frame, first uniformly cuts it into a size of 128*128, and uses the mean value and standard deviation of each channel to compare the three Channels are standardized.

As shown in Figure 2, the 3D joint point detection module consists of a feature extractor, a 2D detector and a 3D detector. Using ResNet50 with attention mechanism as the feature extractor, the input is an image with a resolution of 128×128, and the output size is a feature body F of 32×32×256; the 2D detector is a two-layer CNN, which obtains the feature body F and Output the heat map H of 21 joints, the heat map H is used for 2D pose estimation, and the 2D coordinates of the 21 joint points of the hand:

P _2d =[[x ₁ , y ₁ ], [x ₂ , y ₂ ], . . . , [x ₁ , y ₁ ]] ^T .

Use a two-dimensional Gaussian distribution to generate a heat map for the 2D image coordinates of each joint. The heat map generation formula is as follows:

Among them, σ determines the size of the hot soil radius. f(x, y) represents the probability value of the joint point i whose image coordinates are [4x, 4y]. The position with the largest response in the i-th heat map corresponds to the 2D image coordinates of the i-th joint.

则表示骨骼的方向。对于整只手，共有20个骨向量，因此可将整只手的骨骼长度表示为矩阵B_L，B_L∈R^(J-1)×1。The 3D detector first uses a 2-layer CNN to estimate the delta map D from the heatmap H and the feature volume F, the heatmap H, the feature volume F and the delta map D are concatenated and fed into another 2-layer CNN to obtain the final position map L, and estimate the 3D hand joint positions in the form of the position map L. The 3D _hand joint _position represents _the 3D coordinates P3 _{d of} the 21 joint points of the hand relative to the _spatial coordinate system of the wrist node. ..，[x ₁ ,y ₁ ,z ₁ ]] ^T . Among them, the existence of hand bones limits the size, range of motion and distance between joint points of the hand. Any bone in the skeleton can be expressed as a vector b _ij between the i-th and j-th joint points of the hand, the length of the bone vector b _ij |b _ij | corresponds to the length of the bone, and the direction of the bone vector b _ij

indicates the direction of the bone. For the whole hand, there are 20 bone vectors in total, so the bone length of the whole hand can be expressed as a matrix B _L , B _L ∈ R ^(J-1)×1 .

Step 2: Use the particle swarm optimization algorithm to iteratively fit the hand bone length and the MAMO hand model to find the optimal hand shape and obtain a new 3D joint point.

The particle swarm optimization algorithm is a population-based stochastic optimization technique. The algorithm first initializes a group of random particles (random solutions), and then finds the optimal solution through multiple iterations. In each iteration, the particle updates itself by tracking the two extrema. After finding these two optimal values, the particle updates its speed and position by the following formula:

是粒子i在第k次迭代中第d维的速度向量，

是粒子i在第k次迭代中第d维的位置向量，

是粒子i在第k次迭代中第d维的历史最优位置，既在第k次迭代之后，第i个粒子搜索得到的最优解，

是群体在第k次迭代中第d维的历史最优位置，即在第k次迭代后，整个粒子群体中的最优解。Among them, i=1, 2,..., N, N is the particle swarm scale, d is the particle dimension sequence number, k is the number of iterations, ω is the inertia weight, c ₁ is the individual learning factor, c ₂ is the group learning factor, r ₁ and r ₂ are random numbers in the interval [0, 1], increasing the randomness of the search,

is the d-dimensional velocity vector of particle i in the k-th iteration,

is the d-th dimension position vector of particle i in the k-th iteration,

is the historical optimal position of particle i in the d-th dimension in the k-th iteration, that is, the optimal solution obtained by the i-th particle search after the k-th iteration,

is the historical optimal position of the swarm in the d-th dimension in the k-th iteration, that is, the optimal solution in the entire particle swarm after the k-th iteration.

The iteration termination condition of the particle swarm optimization algorithm is different according to the specific problem. Generally, it is selected as the maximum number of iterations or meets the accuracy requirements. In the present invention, after many comparison experiments, the maximum number of iterations is finally set to 150 times. In each iteration, the particle updates itself by tracking the two extrema.

为初始MANO网络模板，用来表示静止状态下标准的MANO模型表面的初始网格顶点位置。通过MANO初始模板

形状函数B_s(β)和姿势函数B_p(θ)，我们可以得到手部形变模板T，然后结合姿势参数θ，蒙皮权重ω，关节位置J(θ)，对其进行蒙皮操作，数学表达式如下：The MANO model is a mainstream hand pose estimation parametric model. The model forms a complete hand chain based on 16 joint points and 5 fingertip points obtained from the vertices. Combined with the pose parameters, the MANO model can be Restoring the shape of the hand.

It is the initial MANO network template, which is used to represent the initial grid vertex position of the standard MANO model surface in the static state. Via MANO initial template

Shape function B _s (β) and pose function B _p (θ), we can get the hand deformation template T, and then combine the pose parameters θ, skin weight ω, and joint position J(θ) to perform skinning operations on it, The mathematical expression is as follows:

M(θ,β)=W(T(θ,β),θ,ω,J(θ))

Step 3. Send the new 3D joint points to the inverse analytical kinematics module to solve the posture parameter θ required by the MAMO model.

The 3D joint point coordinates can explain the hand pose to a certain extent, but it is not enough to represent the 3D hand model, so we need to derive the joint rotation from the joint point coordinates. The analytical inverse kinematics is to solve the posture parameter θ by decomposing the rotation into twisting and swinging, and the posture parameter is finally used in the MANO model to restore the shape and posture of the hand.

Step 4: Use the MANO model to combine the optimal hand shape and posture parameters θ to obtain the final joint points and mesh vertices.

According to the overall process of the above model generation, when the posture parameter θ and the optimal hand shape are given, the corresponding mesh shape can be generated through the MANO model.

Step 5: Reconstruct and render the obtained mesh vertices and joint points to obtain the final 3D real-time tracking pose of the hand.

In step five, use open3D to reconstruct the vertices of the hand mesh.

The design of the present invention has completed two experiments altogether.

The first experiment is to test whether the method can improve the accuracy of 3D joint point detection. Our model is trained on Nvidia DGX GPU Tesla P100-SXM2. The training set of the model uses CMU HandDB, Rendered Handpose Dataset, GANerated Hands Dataset Three datasets. The test set uses four datasets: Rendered Handpose Dataset, EgoDexter Dataset, STB Dataset, and DexterObject Dataset.

The evaluation indicators are the proportion of correct key points (PCK) and the area under the curve (AUC). Calculating PCK requires artificially setting a 3D joint point error threshold c. When the 3D joint point error is less than c, the joint point is considered to be detected correctly. The proportion of correct joint points to all related nodes is the value of PCK. Under the same threshold, the higher the PCK value, the better the performance of the method. Different PCK can be obtained by setting different threshold c. With the threshold c as the horizontal axis and the PCK value as the vertical axis, the curve of PCK changing with the threshold can be obtained, and the area between the curve and the horizontal axis can be calculated to obtain the AUC. The higher the AUC value, the more accurate the pose estimation is.

The experimental results are shown in Table 1. It can be seen that the method of the present invention can estimate the coordinates of the joint points of the hand more accurately.

Table 1 Experimental results

The second experiment aims to test the real-time hand tracking effect of the method and the hand occlusion recovery problem. The experimental environment is Intel(R) Xeon(R) CPU E5-2620. We open the palm, hold the pen, and hold the cup respectively. And other actions, the experimental results are shown in Figure 3. It can be found that this method has a good three-dimensional gesture tracking ability, and there is no obvious deformation and self-occlusion of the hand. For the occlusion of objects, this method can also restore the hand shape in real time.

To sum up, the 3D gesture tracking performed by this method has better real-time performance, and has better real-time effects on self-occlusion of hands and object occlusion.

The present invention also provides a three-dimensional gesture tracking device based on an RGB camera. The device includes a picture acquisition and preprocessing module, a three-dimensional joint point detection module, a pose parameter calculation module, a joint point acquisition module and a rendering output module.

The image acquisition and preprocessing module and the 3D joint point detection module are used to standardize the picture with the RGB camera as input, send the processed picture to the 3D joint point detection module, extract image features, and use the convolutional neural network to predict Preliminary 3D joint points of the hand and calculation of bone length;

The posture parameter calculation module is used to use the particle swarm optimization algorithm to iteratively fit the hand bone length and the MANO model, find out the optimal hand shape and obtain a set of new 3D joint points; and send the new 3D joint points into The reverse analysis kinematics module solves the posture parameter θ required by the MANO model;

The joint point acquisition module is used to combine the optimal hand shape and posture parameters θ with the MANO model to obtain the final joint points and mesh vertices; and

The rendering output module is used to reconstruct and render the obtained mesh vertices and joint points to obtain the final 3D real-time tracking posture of the hand.

In the method of the present invention, the hand bone length is deduced from the 3D coordinates of the hand predicted by the convolutional neural network model, and the hand bone length and the MANO hand model parameter file are iteratively fitted for 150 times using the particle swarm optimization algorithm to match the optimal The shape of the hand, and then use inverse analytical kinematics to infer the pose parameters from the joint positions, and finally combine the pose parameters with the MANO model to get the final hand tracking pose. This algorithm has better real-time performance in real use scenarios, and has better robustness to self-occlusion and object occlusion of hands.

The above embodiments are only used to illustrate the technical solutions of the present invention without limitation. Although the present invention has been described in detail with reference to preferred embodiments, those of ordinary skill in the art should understand that the technical solutions of the present invention can be modified or equivalently replaced. Without departing from the spirit and scope of the technical solution of the present invention.

Claims (7)

1. A three-dimensional gesture tracking method based on RGB camera, is characterized in that, comprises the following steps: Step 1. Use the RGB camera as input, standardize the image, send the processed image to the 3D joint point detection module, extract image features, use the convolutional neural network to predict the initial 3D joint point and calculate the length of the hand bones; Step 2. Use the particle swarm optimization algorithm to iteratively fit the hand bone length and the MAMO hand model to find the optimal hand shape and obtain a new 3D joint point; Step 3. Send the new 3D joint points to the inverse analytical kinematics module to solve the posture parameter θ required by the MAMO model; Step 4, use the MANO model to combine the optimal hand shape and posture parameters θ to obtain the final joint points and mesh vertices; and Step 5: Reconstruct and render the obtained mesh vertices and joint points to obtain the final 3D real-time tracking pose of the hand. 2. the three-dimensional gesture tracking method based on RGB camera according to claim 1, is characterized in that: described three-dimensional joint point detection module uses the neural network model based on ResNet50, comprises feature extractor, 2D detector and 3D detector, Using ResNet50 with attention mechanism as the feature extractor, the input is an image with a resolution of 128×128, and the output size is a feature volume F of 32×32×256. 3. the three-dimensional gesture tracking method based on RGB camera according to claim 2, is characterized in that: described 2D detector is a two-layer CNN, obtains feature body F and outputs heat map H of 21 articulation points, heat map H is used for 2D pose estimation; the 3D detector first uses a 2-layer CNN to estimate a delta map D from heatmap H and feature volume F, heatmap H, feature volume F and delta map D are concatenated and fed to another 2-layer CNN to obtain the final position map L, and estimate the 3D hand joint position in the form of position map L. 4. the three-dimensional gesture tracking method based on RGB cameras according to claim 3, characterized in that: in step 4, the MANO model is a 3D parametric model, and the model is obtained according to 16 joint points and 5 points from the vertices. Fingertips to form a complete hand chain. 5. The three-dimensional gesture tracking method based on an RGB camera according to claim 3, wherein in step five, open3D is used to reconstruct the vertices of the hand mesh. 6. the three-dimensional gesture tracking method based on RGB camera according to claim 3, characterized in that: in the MANO model

is a palm-flat pose, through the template

The shape function B _s (β) and the posture function B _p (θ) can obtain the hand deformation template T, and then combine the posture parameters θ, skin weight ω, and joint position J(θ) to perform skinning operations on it. Mathematics The expression is as follows:

M(θ,β)=W(T(θ,β),θ,ω,J(θ)). 7. The three-dimensional gesture tracking method based on RGB camera according to claim 6, characterized in that: the particle swarm optimization algorithm first initializes a group of random particles, and then finds the optimal solution through multiple iterations, and in each iteration, Particles update themselves by tracking two extreme values. After finding these two optimal values, particles update their speed and position through the following formula:

Among them, i=1,2,...,N,N is the size of particle swarm, d is the number of particle dimension, k is the number of iterations, ω is the inertia weight, c ₁ is the individual learning factor, c ₂ is the group learning factor, r ₁ , r ₂ is a random number in the interval [0,1], increasing the randomness of the search,

is the d-dimensional velocity vector of particle i in the k-th iteration,

is the d-th dimension position vector of particle i in the k-th iteration,

is the historical optimal position of particle i in the d-th dimension in the k-th iteration, that is, the optimal solution obtained by the i-th particle search after the k-th iteration,

is the historical optimal position of the swarm in the d-th dimension in the k-th iteration, that is, the optimal solution in the entire particle swarm after the k-th iteration.

Images