CN115546888A - A Convolutional Pose Estimation Method Based on Body Part Grouping with Symmetrical Semantic Graphs - Google Patents

Abstract

The invention discloses a symmetric semantic graph convolution pose estimation method based on body part grouping, comprising the following steps: S1, inputting two-dimensional human joint points and their connection relations, constructing a symmetric semantic graph convolution layer of the joint point graph structure and Non-local layer; S2, according to the body trunk, group the body parts, obtain the local and non-local features of each torso and the local and non-local features of the whole body, and perform fusion calculation on the obtained features; S3, based on the symmetrical semantic map volume Laminate layer, non-local layer and body part grouping, build the symmetric semantic graph convolution pose estimation network model of body part grouping; S4, use Human3.6M data set to train described symmetric semantic graph convolution pose estimation network model, will The 2D human joint points to be estimated are input into the trained symmetric semantic graph convolution pose estimation network model, and the estimated 3D human joint points are output. The invention can be applied to the fields of movie animation, virtual reality, sports action analysis and the like, and the method has better effect and improved generalization ability.

Description

A Convolutional Pose Estimation Method Based on Body Part Grouping with Symmetrical Semantic Graphs

technical field

The invention relates to the technical field of computer vision, in particular to a body part grouping-based symmetric semantic graph convolution pose estimation method.

Background technique

Human pose estimation has been widely used in many computer vision tasks, such as virtual reality, human-computer interaction and action recognition. Thanks to the rapid development of deep learning, the performance of estimating 3D human pose from images has been significantly improved, and has become a current research hotspot.

There are two types of existing 3D attitude estimation methods, one is to directly predict the 3D attitude from the image, and the other is to first predict the 2D attitude and then regress to the 3D attitude. The first type of method can get a lot of information from images, but the model is greatly affected by factors such as image background and human clothing, and the learning features required by the model are complex. The second type of method reduces the overall work complexity, and the network model is easier to learn the mapping from 2D to 3D space. At the same time, thanks to the maturity of 2D pose estimation research, this type of model is more mainstream.

"A 3D Human Pose Estimation Method Based on Graph Convolutional Network" (CN112712019A) provides a 3D human pose estimation method based on graph convolutional network, which has the advantages of improving the performance of 3D human pose regression and reducing the use of network parameters. Ability to be improved. In the existing research, the human body pose estimation algorithm under the background of deep learning is easily affected by self-occlusion, environmental occlusion, etc., and the human body poses are diverse, and the generalization ability of the current model is not good. Therefore, it is urgent to explore a more reasonable and more universal network model to improve the effect of attitude estimation.

Contents of the invention

The purpose of the present invention is to solve the above-mentioned problems in the prior art, and provide a method for symmetric semantic map convolution pose estimation based on body part grouping.

The purpose of the present invention can be achieved by taking the following technical solutions:

A kind of symmetric semantic graph convolution pose estimation method based on body parts grouping, as shown in Figure 1, described symmetric semantic graph convolution pose estimation method comprises the steps:

S1. Input the two-dimensional human joint points and their connection relationship in movie animation, virtual reality or sports action, and construct the symmetrical semantic graph convolution layer and non-local layer of the joint point graph structure;

S2. Carry out grouping of body parts according to the trunk of the body, respectively obtain the local and non-local features of each trunk and the local and non-local features of the whole body, and perform fusion calculation on the obtained features;

S3. Based on the symmetric semantic graph convolution layer, non-local layer and body part grouping, construct a symmetric semantic graph convolution pose estimation network model for body part grouping;

S4. Using the H _u man _3.6M data set to train the symmetric semantic graph convolution pose estimation network model, and input the two-dimensional human body joint points to be estimated into the trained symmetric semantic graph convolution pose estimation network model, Output estimated 3D human joint points.

Further, in the step S1, using the two-dimensional human body joints and their connections, the process of constructing the symmetrical semantic graph convolution layer and the non-local layer of the joint point graph structure is as follows:

Let X ^(l) and X ^(l+1) represent the characteristics of the nodes in the graph structure before and after the l-th layer of convolution respectively, then the form of the symmetric graph convolution is:

X ^(l+1) = σ(WX ^(l) A ^sym ) (1)

Among them, σ() represents an activation function, W represents a learnable weight parameter, and A ^sym is a matrix obtained after symmetrically normalizing the adjacency matrix A of the graph, expressed as follows:

Among them, A is the adjacency matrix of the graph, and D is the degree matrix. Symmetric normalization can better aggregate the information of neighboring nodes to obtain balanced node characteristics;

By adding a learnable weighting matrix M on the basis of symmetric graph convolution, a symmetric semantic graph convolution layer is constructed. The calculation formula of the symmetric semantic graph convolution layer is expressed as follows:

X ^(l+1) = σ(WX ^(l) ρ _i (M⊙A ^sym )) (3)

Among them, ρ _i () is a Softmax nonlinear function, which is used to normalize the matrix of node i, and ⊙ represents the multiplication operation of elements corresponding to the matrix;

In order to capture the global features between nodes in the graph, the concept of non-local layer is introduced, and the operation of non-local layer is defined as:

分别表示节点i，j的输入特征；

表示节点i的输出特征；f(，)是可学习的二元函数，用于计算两个输入特征的相似度；g()是可学习的一元函数，对输入特征进行变换。Among them, W ^x represents the normalization factor of the learnable weight parameter W, K represents the number of nodes, i represents the index of the target node to be calculated, and j represents the index of nodes other than i;

represent the input features of nodes i and j respectively;

Represents the output feature of node i; f(,) is a learnable binary function used to calculate the similarity between two input features; g() is a learnable unary function that transforms the input features.

Further, in the step S2, for the grouping of body parts, the joint points of the human body are decomposed into the left limb group, the right limb group, and the whole body group. Each joint point in the group has a stronger correlation, and each group is performed through an independent sub-network. Feature extraction to enhance local relationships.

As shown in Figure 4, the feature fusion adopts the late fusion method, first learn the features in each group, and then fuse the features in each group, the feature fusion is defined as:

f ^fuse ＝Concat(f ^left ，f ^right ，f ^all ) (5)

Among them, Concat(,,) means to connect the features, f ^left is the feature of the left limb group, f ^right is the feature of the right limb group, ^{fall is the feature of the whole body group, and f fuse is} the feature obtained after fusion.

The implementation of body part grouping, which learns local joint consistency while maintaining global pose consistency, generalizes better to symmetric poses in the training data, as well as rare, occluded poses.

Further, in the step S3, multiple symmetric semantic graph convolution modules are constructed based on the symmetric semantic graph convolution layer and the non-local layer, all symmetric semantic graph convolution modules have the same structure, and each symmetric semantic graph convolution module It consists of two symmetric semantic graph convolutional layers and a non-local layer connected sequentially, and alternates the symmetric semantic graph convolutional layer and the non-local layer to obtain local and global semantic relationships between nodes;

In the symmetric semantic graph convolution network, as shown in Figure 3, first use a symmetric semantic graph convolution layer and use a non-local layer to map the input to the potential space; then through four sequentially connected symmetric semantic graph volumes Product module to obtain the encoded features, all symmetric semantic graph convolution layers in the symmetric semantic graph convolutional network are followed by batch normalization and R _e LU nonlinear activation;

The symmetric semantic graph convolution pose estimation network model of the body parts grouping includes a first branch, a second branch, and a third branch, as shown in Figure 2, wherein, the first branch, the second branch, and the third branch all use symmetric Semantic graph convolutional network for feature extraction: the left limb group is input to the first branch, and the feature f ^left of the left limb is extracted through the symmetrical semantic graph convolutional network; the right limb group is input to the second branch, and the symmetrical semantic graph convolutional network is used to extract The feature f ^right of the right limb; the whole body group is input to the third branch, and the feature f ^all of the whole body is extracted through the symmetric semantic graph convolution network; the fused feature f ^fuse is calculated according to formula (5), and then a symmetric semantic graph convolution is used layer, which projects the encoded features to the output space.

Further, in the step S4, the loss function L _smoothhl1 () defined by the formula (6) is used to train on the Human3.6M data set, the formula is as follows:

Among them, X represents the difference between the true value and the predicted value, |·| represents the absolute value of the difference between the true value and the predicted value, J′ _i represents the predicted 3D joint coordinates of node i, and J _i corresponds to the true value of node i in the data set. The L _smoothl1 (J) loss function is not sensitive to outlier nodes and outliers, and can control the magnitude of the gradient to make the training converge reasonably.

Further, the evaluation index usually used for attitude estimation is MPJPE (Mean Per Joint Position Error), and the formula is defined as (7):

The E _MPJPE () index represents the mean value of the L2 distance between the predicted value and the true value of each joint, and ||·|| ₂ represents the L2 distance between the predicted value and the true value. When the evaluation index MPJPE is smaller, it is considered that the 3D human body pose estimation result is better.

Further, during training, the initial learning rate is 0.001, and a batch size of 64 is used. The initial learning rate directly affects the convergence state of the model, and the batch size affects the generalization ability of the model. The initial learning rate of 0.001 is conducive to model convergence, and the batch size of 64 is conducive to model generalization.

Compared with the prior art, the present invention has the following advantages and effects:

The symmetric semantic graph convolution pose estimation network based on body part grouping proposed by the present invention introduces symmetric semantic graph convolution, which can better aggregate the information of neighbor nodes and obtain balanced node features; design body part grouping, body Segmented by parts into left/right torso, these body part groups are learned through independent sub-networks to enhance local features. Compared with other methods on the H _u man _3.6M data set, the method is generally more effective and the generalization ability has been improved.

Description of drawings

The accompanying drawings described here are used to provide a further understanding of the present invention and constitute a part of the application. The schematic embodiments of the present invention and their descriptions are used to explain the present invention and do not constitute improper limitations to the present invention. In the attached picture:

Fig. 1 is a flow chart of a three-dimensional human body posture transfer method based on posture estimation driven by the present invention;

Fig. 2 is a symmetrical semantic graph convolutional network model diagram based on body parts grouping in an embodiment of the present invention;

Fig. 3 is a schematic diagram of a symmetrical semantic graph convolution module in an embodiment of the present invention;

Fig. 4 is a schematic diagram of a body part grouping feature fusion module in an embodiment of the present invention.

detailed description

In order to make the object, technical solution and advantages of the present invention more clear and definite, the present invention will be further described in detail below with reference to the accompanying drawings and examples. It should be understood that the specific embodiments described here are only used to explain the present invention, not to limit the present invention.

Example 1

A symmetric semantic graph convolution pose estimation method based on body parts grouping, as shown in Figure 1, the method includes the following steps:

S1. Input the two-dimensional human joint points and their connection relationship in movie animation, virtual reality or sports action, and construct the symmetrical semantic graph convolution layer and non-local layer of the joint point graph structure;

In step S1, using two-dimensional human body joints and their connection relations, the process of constructing a symmetrical semantic graph convolution layer and a non-local layer of the joint point graph structure is as follows:

Let X ^(l) and X ^(l+1) represent the characteristics of the nodes in the graph structure before and after the l-th layer of convolution respectively, then the form of the symmetric graph convolution is:

X ^(l+1) = σ(WX ^(l) A ^sym ) (1)

Among them, σ() represents an activation function, W represents a learnable weight parameter, and A ^sym is a matrix obtained after symmetrically normalizing the adjacency matrix A of the graph, expressed as follows:

Among them, A is the adjacency matrix of the graph, and D is the degree matrix;

By adding a learnable weighting matrix M on the basis of symmetric graph convolution, a symmetric semantic graph convolution layer is constructed. The calculation formula of the symmetric semantic graph convolution layer is expressed as follows:

X ^(l+1) ＝σ(WX ^(l) ρ _i (M⊙A ^sym) ) (3)

Among them, ρ _i () is a Softmax nonlinear function, which is used to normalize the matrix of node i, and ⊙ represents the multiplication operation of elements corresponding to the matrix;

In order to capture the global features between nodes in the graph, the concept of non-local layer is introduced, and the operation of non-local layer is defined as:

分别表示节点i，j的输入特征；

表示节点i的输出特征；f(，)是可学习的二元函数，用于计算两个输入特征的相似度；g()是可学习的一元函数，对输入特征进行变换。Among them, W _x represents the normalization factor of the learnable weight parameter W, K represents the number of nodes, i represents the index of the target node to be calculated, and j represents the index of nodes other than i;

represent the input features of nodes i and j respectively;

Represents the output feature of node i; f(,) is a learnable binary function used to calculate the similarity between two input features; g() is a learnable unary function that transforms the input features.

S2. According to the torso data in the movie animation, virtual reality or sports action, the body parts are grouped, and the local and non-local features of each torso and the local and non-local features of the whole body are respectively obtained, and the obtained features are fused and calculated;

In step S2, for the grouping of body parts, the joint points of the human body in movie animation, virtual reality or sports actions are decomposed into left limb group, right limb group, and whole body group. Each joint point in the group has a stronger correlation. Independent sub-networks are used for feature extraction to enhance local relations.

As shown in Figure 4, the feature fusion adopts the late fusion method, first learn the features in each group, and then perform fusion calculation on the features in each group, the feature fusion is defined as:

f ^fuse ＝Concat(f ^left ，f ^right ，f ^all ) (5)

Among them, Concat(,,) means to connect the features, f ^left is the feature of the left limb group, f ^right is the feature of the right limb group, ^{fall is the feature of the whole body group, and f fuse is} the feature obtained after fusion.

S3. Based on the symmetric semantic graph convolution layer, non-local layer and body part grouping, construct a symmetric semantic graph convolution pose estimation network model for body part grouping;

In step S3, multiple symmetric semantic graph convolution modules are constructed based on the symmetric semantic graph convolution layer and the non-local layer. All symmetric semantic graph convolution modules have the same structure, and each symmetric semantic graph convolution module consists of two symmetric semantic graph convolution modules. The graph convolutional layer and a non-local layer are connected sequentially;

In the symmetric semantic graph convolution network, as shown in Figure 3, first use a symmetric semantic graph convolution layer and use a non-local layer to map the input to the potential space; then through four sequentially connected symmetric semantic graph volumes Product module to get the encoded features, all symmetric semantic graph convolution layers in the symmetric semantic graph convolutional network are followed by batch normalization and ReLU nonlinear activation;

The symmetric semantic graph convolution pose estimation network model of body parts grouping includes the first branch, the second branch, and the third branch, as shown in Figure 2, where the first branch, the second branch, and the third branch all use symmetric semantic graphs Convolutional network for feature extraction: the left limb group is input to the first branch, and the feature f ^left of the left limb is extracted through the symmetric semantic graph convolution network; the right limb group is input to the second branch, and the right limb is extracted through the symmetric semantic graph convolution network The feature f ^right of the whole body group is input to the third branch, and the feature f ^all of the whole body is extracted through the symmetric semantic graph convolution network; the fused feature f ^fuse is calculated according to formula (5), and then a symmetric semantic graph convolution layer is used, Project the encoded features to the output space.

S4. Use the Human3.6M data set to train the symmetric semantic graph convolution pose estimation network model, and input the two-dimensional human body joint points in the movie animation, virtual reality or sports action to be estimated into the trained symmetric semantic graph convolution The pose estimation network model outputs the estimated three-dimensional human joint points in movie animation, virtual reality or sports actions.

In step S4, the loss function L _smoothhl1 () defined by the formula (6) is used to train on the Human3.6M data set, and the formula is as follows:

Among them, X represents the difference between the real value and the predicted value of movie animation, virtual reality or sports action data, |·| represents the absolute value of the difference between the real value and the predicted value, J′ _i represents the predicted 3D joint coordinates of node i, J _i Corresponds to the true value of the i-node in the data set.

Among them, the evaluation index usually used for attitude estimation is MPJPE (Mean Per Joint Position Error), and the formula is defined as (7):

The E _MPJPE () index represents the mean value of the L2 distance between the predicted value and the real value of each joint in movie animation, virtual reality or sports action, and ||·|| ₂ represents the L2 distance between the predicted value and the real value. When the evaluation index MPJPE is smaller, it is considered that the 3D human body pose estimation result is better.

During training, the initial learning rate is 0.001 and the batch size is 64.

Example 2

This embodiment is based on a symmetric semantic graph convolution pose estimation method based on body part grouping disclosed in Embodiment 1. In order to verify the effectiveness of the present invention, experiments are carried out on the Human3.6M data set, and the technology of the present invention is combined with the experimental results. The effect is explained.

Human3.6M is one of the most widely used datasets for 3D pose estimation, covering 3.6 million images, collected in an indoor controllable environment, with a total of 11 experimenters, using a marked motion capture device to capture the daily activity scenes of the experimenters body posture, including 15 actions.

Experimental configuration: Hardware environment: GPU RTX 2080Ti Video memory: 11GB, CPU 4-core Intel(R) Xeon(R) Silver 4110 CPU@2.10GHz Memory: 16GB. Software environment: Python v2.7, Pytorch v1.1.0, CUDA10.2. Operating system: Ubuntu18.04.

Ablation studies were performed on the method proposed by the present invention. With the above configuration, the pose estimation network proposed by the present invention contains two main modules: a symmetric semantic graph convolution module and a body part grouping. To verify their effectiveness, ablation experiments are set up as follows: the first experiment uses semantic graph convolution only, the second experiment uses a symmetric semantic graph convolution module, the second experiment uses body part grouping, and the third experiment uses symmetric Semantic graph convolution module and body part grouping. The experimental results are shown in Table 1:

Table 1. Ablation experiment results of the convolution pose estimation method based on body parts grouping

Symmetric Semantic Graph Convolution Module Body Part Grouping MPJPE 41.47mm √ 40.68mm √ 40.53mm √ √ 39.93mm

Table 2 shows the experimental results of the method of the present invention compared with the baseline method and the semantic graph convolution method under the MPJPE evaluation index according to the classification of human actions. The best method for each action is highlighted in bold.

Table 2. Comparison of experimental results of symmetric semantic graph convolution pose estimation methods based on body part grouping

It can be seen that the symmetric semantic graph convolution pose estimation network based on body part grouping proposed by the present invention achieves better performance, which shows that the model in this paper can effectively utilize the relationship between different joint groups in the graph.

The above-mentioned embodiment is a preferred embodiment of the present invention, but the embodiment of the present invention is not limited by the above-mentioned embodiment, and any other changes, modifications, substitutions, combinations, Simplifications should be equivalent replacement methods, and all are included in the protection scope of the present invention.

Claims (8)

1. a symmetric semantic graph convolution pose estimation method based on body parts grouping, is characterized in that, described symmetric semantic graph convolution pose estimation method comprises the steps: S1. Input the two-dimensional human joint points and their connection relationship in movie animation, virtual reality or sports action, and construct the symmetrical semantic graph convolution layer and non-local layer of the joint point graph structure; S2. Carry out grouping of body parts according to the trunk of the body, respectively obtain the local and non-local features of each trunk and the local and non-local features of the whole body, and perform fusion calculation on the obtained features; S3. Based on the symmetric semantic graph convolution layer, non-local layer and body part grouping, construct a symmetric semantic graph convolution pose estimation network model for body part grouping; S4. Use the Human3.6M data set to train the symmetric semantic graph convolution pose estimation network model, input the two-dimensional human joint points to be estimated into the trained symmetric semantic graph convolution pose estimation network model, and output the estimated three-dimensional Human joints. 2. A method for symmetric semantic graph convolution pose estimation based on body parts grouping according to claim 1, characterized in that, in said step S1, two-dimensional human body joint points and their connection relationships are used to construct a joint point graph structure The symmetric semantic graph convolution layer process is as follows: Let X ^(l) and X ^(l+1) represent the characteristics of the nodes in the graph structure before and after the l-th layer of convolution respectively, then the form of the symmetric graph convolution is: X ^(l+1) = σ(WX ^(l) A ^sym ) (1) Among them, σ() represents an activation function, W represents a learnable weight parameter, and A ^sym is a matrix obtained after symmetrically normalizing the adjacency matrix A of the graph, expressed as follows:

Among them, A is the adjacency matrix of the graph, and D is the degree matrix; By adding a learnable weighting matrix M on the basis of symmetric graph convolution, a symmetric semantic graph convolution layer is constructed. The calculation formula of the symmetric semantic graph convolution layer is expressed as follows: X ^(l+1) = σ(WX ^(l) ρ _i (M⊙A ^sym )) (3) Among them, ρ _i () is a Softmax nonlinear function, which is used to normalize the matrix of node i, and ⊙ represents the multiplication operation of elements corresponding to the matrix. 3. A method for symmetric semantic graph convolution pose estimation based on body parts grouping according to claim 2, characterized in that, in said step S1, two-dimensional human body joint points and their connection relationships are used to construct a joint point graph structure The process of the non-local layer is as follows: Define the operation of the non-local layer as:

Among them, W _x represents the normalization factor of the learnable weight parameter W, K represents the number of nodes, i represents the index of the target node to be calculated, and j represents the index of nodes other than i;

represent the input features of nodes i and j respectively;

Indicates the output feature of node i; f(,) is a learnable binary function used to calculate the similarity between two input features; g() is a learnable unary function used to transform the input features. 4. A method of symmetric semantic graph convolution pose estimation based on body parts grouping according to claim 3, characterized in that, in the step S2, human body joints are decomposed into left limb group, right limb group, whole body Each group enhances the local relationship through an independent sub-network, and then adopts the feature fusion method of late fusion, first learns the features in each group, and then fuses the features in each group. The feature fusion is defined as: f ^fuse ＝Concat(f ^left ,f ^right ,f ^all ) (5) Among them, Concat(,,) means to connect the features, f ^left is the feature of the left limb group, f ^right is the feature of the right limb group, fall ^all is the feature of the whole body group, and f ^fuse is the feature obtained after fusion. 5. A method for symmetric semantic graph convolution pose estimation based on body part grouping according to claim 4, characterized in that, in the step S3, multiple symmetric semantic graph convolution layers and non-local layers are constructed based on the symmetric semantic graph convolution layer Semantic graph convolution module, all symmetric semantic graph convolution modules have the same structure, and each symmetric semantic graph convolution module is composed of two symmetric semantic graph convolution layers and a non-local layer connected sequentially; In the symmetric semantic graph convolutional network, a symmetric semantic graph convolution layer and a non-local layer are used to map the input to the potential space; then four sequentially connected symmetric semantic graph convolution modules are used to obtain the encoded Features, all symmetric semantic graph convolution layers in the symmetric semantic graph convolutional network are followed by batch normalization and ReLU nonlinear activation; The symmetric semantic graph convolution pose estimation network model of the body parts grouping includes a first branch, a second branch, and a third branch, wherein, the first branch, the second branch, and the third branch are all performed using a symmetric semantic graph convolution network. Feature extraction: the left limb group is input to the first branch, and the feature f ^left of the left limb is extracted through the symmetric semantic graph convolution network; the right limb group is input to the second branch, and the feature f ^right of the right limb is extracted through the symmetric semantic graph convolution network ;The whole body group is input to the third branch, through the symmetric semantic graph convolutional network, to extract the full body feature f ^all ; according to the formula (5), the fused feature f ^fuse is obtained, and then a symmetric semantic graph convolution layer is used to convert the encoded feature Projected to the output space. 6. a kind of symmetric semantic graph convolution attitude estimation method based on body parts grouping according to claim 5, is characterized in that, adopts the loss function L _smoothhl1 () defined in formula (6) in described step S4, in Human3 .6M data set for training, the formula is as follows:

Among them, X represents the difference between the true value and the predicted value, |·| represents the absolute value of the difference between the true value and the predicted value, J′ _i represents the predicted 3D joint coordinates of node i, and J _i corresponds to the true value of node i in the data set. 7. a kind of symmetric semantic graph convolution posture estimation method based on body parts grouping according to claim 6, is characterized in that, the evaluation index that described posture estimation adopts is MPJPE, and formula is defined as follows:

The E _MPJPE () index represents the mean value of the L2 distance between the predicted value and the true value of each joint, and ‖·‖ ₂ represents the L2 distance between the predicted value and the true value. 8. A method for symmetric semantic graph convolution pose estimation based on body part grouping according to claim 6, characterized in that, in the training process, the initial learning rate is 0.001, and a batch size of 64 is used.

Images