CN113158459A - Human body posture estimation method based on visual and inertial information fusion - Google Patents

Abstract

A human body posture estimation method based on visual and inertial information fusion aims at the defect that a human body posture estimation method based on a 3D visual sensor cannot provide three-degree-of-freedom rotation information, the visual information, the inertial information and human body posture prior information are fused in a self-adaptive mode by utilizing the complementarity of the visual information and the inertial information and adopting a nonlinear optimization method, the rotation angle of a human body skeleton node and the global position of a root skeleton node at each moment are obtained, and real-time human body posture estimation is completed. The invention effectively improves the accuracy and robustness of human body posture estimation, and makes up the defects that a visual sensor is easy to be shielded and inertial data accumulate errors along with time.

Description

A Human Pose Estimation Method Based on Fusion of Visual and Inertial Information

technical field

The invention belongs to the field of human body posture estimation, in particular to a human body posture estimation method based on fusion of visual and inertial information.

Background technique

Human pose estimation technology has important application value. With the development of visual sensors, inertial measurement units and artificial intelligence technologies, human pose estimation is gradually applied in the fields of human-machine collaboration, video surveillance, film and television production, and industrial and agricultural production. For example, It is used to ensure the safety of workers in the process of human-machine collaboration, and to record and analyze the behavior of people in the monitoring screen.

3D human pose estimation technology has become mature. With the development of behavior recognition, auxiliary training and human-machine collaboration, people need 6D human pose estimation information to develop applications. For example, in dance assistant training, 6D human pose estimation includes With the joint rotation information, the captured dance movement details are more abundant, and the training effect of the students is better. In daily production and life, the human body pose estimation method based on 3D vision is the most common and practical method. This method can accurately extract the joint points of the human skeleton and obtain 3D human body pose information. However, when the human body is self-occluded or the camera is partially occluded , the reliability of the data is reduced. The inertial measurement unit can provide spatial rotation information, and its output is stable, but the error of the rotation information will accumulate over time. The complementarity of visual and inertial information can be used to estimate the 6D posture of the human body, and the three-degree-of-freedom displacement information of the visual output and the three-degree-of-freedom rotation information of the inertial output can be simply combined. At this time, the obtained human body posture estimation system has poor robustness. Accuracy is not high. Currently, there is no technology that can robustly and real-time fuse visual and inertial information to solve the problem of 6D human pose estimation.

SUMMARY OF THE INVENTION

In order to overcome the disadvantage that the human body pose estimation method based on 3D vision sensor cannot provide three-degree-of-freedom rotation information, the present invention provides a human body pose estimation method based on the fusion of visual and inertial information. Non-linear fusion of inertial and human pose prior knowledge to obtain the rotation angle of the human skeleton node and the global position of the root skeleton node at each moment, complete real-time human pose estimation, and effectively improve the accuracy and robustness of human pose estimation .

The technical scheme adopted in the present invention comprises the following steps:

A human pose estimation method based on fusion of visual and inertial information, comprising the following steps:

惯性坐标系n与全局坐标系g之间的旋转矩阵

惯性传感器i与对应骨骼坐标系b_i之间的位移矩阵

和旋转矩阵

Step 1) Establish the kinematic model of each skeleton node of the human body, determine the optimization variable θ, and determine the homogeneous transformation matrix between the camera coordinate system c and the global coordinate system g

Rotation matrix between inertial coordinate system n and global coordinate system g

Displacement matrix between inertial sensor i and corresponding bone coordinate system b _i

and rotation matrix

Step 2) Set the visual and inertial output frequencies to be consistent, and use the inertial sensor rotation constraint _ER (θ), acceleration constraint EA (θ), visual sensor position constraint _EP (θ) and human body pose prior constraint _{E prior} ( θ) Construct an optimization problem for the optimization item, and set the weight of each optimization item;

以及惯性传感器的旋转测量值R_i和加速度测量值a_i，计算各优化项在统一坐标系后的传感器测量值

与估计值

Step 3) Read the position measurement of the vision sensor at each moment

and the rotation measurement value R _i and acceleration measurement value a _i of the inertial sensor, calculate the sensor measurement value of each optimization item after the unified coordinate system

with estimated value

Step 4) Solve the nonlinear least squares optimization problem, the optimal solution θ at each moment is the optimal rotation angle of each skeleton node of the human body at the current moment and the optimal global position of the root skeleton node n ₁ , and then according to the established human skeleton Node kinematics model to get the human pose estimation at the current moment;

Repeat steps 3) and 4) to complete the state estimation of each joint point of the human body at each moment, and obtain a real-time human body pose estimation based on the fusion of visual and inertial information.

Further, in the step 1), the camera coordinate system c represents the coordinate system of the depth camera, the inertial coordinate system n represents the unified coordinate system after calibration of all inertial sensors, the global coordinate system g and the initial coordinates of the skeleton node system aligned.

In the step 2), the rotation constraint E _R (θ) is established by the difference between the measured value and the estimated value of each IMU rotation matrix; the acceleration constraint E _A (θ) is established by minimizing each IMU rotation matrix. The difference between the measured value and the estimated value of the acceleration of each IMU is established; the position constraint _EP (θ) is established by minimizing the difference between the measured value and the estimated value of the global position of each skeletal node; the human body pose The prior constraint E _prior (θ) is established from the existing human pose estimation dataset.

是从视觉传感器中读取的人体各关节点位置信息，R_i是从惯性的陀螺仪中读取的旋转信息，a_i是从惯性的加速度计中读取的加速度信息。In the step 3), the

is the position information of each joint point of the human body read from the vision sensor, R _i is the rotation information read from the inertial gyroscope, and a _i is the acceleration information read from the inertial accelerometer.

In the step 4), the root bone node n ₁ is located at the joint point of the human pelvis.

The beneficial effects of the present invention are mainly manifested in: providing a human body attitude estimation method based on the fusion of visual and inertial information, aiming at the shortcoming that the human body attitude estimation method based on 3D visual sensor lacks the output of three-degree-of-freedom rotation information, a nonlinear optimization method is adopted. Adaptive fusion of visual information, inertial information, and human pose prior information to obtain 6D human pose estimation, which improves the accuracy and robustness of human pose estimation, and makes up for the shortcomings of visual sensors that are easily occluded and inertial data accumulated over time.

Description of drawings

Figure 1 is a flowchart of a method for human pose estimation based on fusion of visual and inertial information.

Figure 2 is a schematic diagram of the upper body skeleton nodes and the wearing position of the IMU.

Figure 3 is a schematic diagram of the placement position of the vision sensor.

Figure 4 is a flowchart of a human pose estimation algorithm based on fusion of visual and inertial information.

Detailed ways

In order to make the technical solutions and design ideas of the present invention clearer, the upper body of the human body is selected as the object of attitude estimation, and two visual sensors and five inertial sensors are used to further describe the present invention with reference to the accompanying drawings.

Referring to Figure 1, Figure 2, Figure 3 and Figure 4, a method for estimating human body pose based on visual and inertial information fusion, the method comprises the following steps:

惯性坐标系n与全局坐标系g之间的旋转矩阵

惯性传感器i与对应骨骼坐标系b_i之间的位移矩阵

和旋转矩阵

过程如下：Step 1) Establish the kinematic model of each skeleton node of the human body, determine the optimization variable θ, and determine the homogeneous transformation matrix between the camera coordinate system c and the global coordinate system g

Rotation matrix between inertial coordinate system n and global coordinate system g

Displacement matrix between inertial sensor i and corresponding bone coordinate system b _i

and rotation matrix

The process is as follows:

1.1) Human bones are defined as rigid bodies connected to each other. The initial coordinate system B of the skeleton node is aligned with the global coordinate system g, and the number of bones in the upper body is defined as n _b = 13. As shown in Figure 2, the skeleton nodes are the left hand, right hand, and left forearm respectively. , right forearm, left upper arm, right upper arm, left shoulder, right shoulder, spine 1-4, and pelvis, where the pelvis is regarded as the root bone node n ₁ , and the child bone node n _b (b≥2) is relative to its parent node The rotation matrix R _b and the relatively invariant displacement t _b , each bone has 3 rotational degrees of freedom, the root node has a global displacement (x ₁ , y ₁ , z ₁ ), with 42 (d=3+3 ×n _b = 42) degrees of freedom to describe the motion of the entire upper body of the human body, record the 42 variables as a 42-dimensional vector θ, the vector θ is used as the optimization variable of the optimization problem, and each rigidity is derived from the forward kinematics formula. Homogeneous transformation matrix of bones in global coordinate system

where P(b) is the set of all bones;

和旋转矩阵

进而确定相机坐标系c和全局坐标系g的齐次变换矩阵

1.2) As shown in Figure 3, two visual sensors are placed in front of the tester respectively, the visual sensor is L=2 meters away from the tester, and the "Zhang Zhengyou camera calibration method" is used to obtain the translation matrix of the two cameras for the global coordinate system g

and rotation matrix

Then determine the homogeneous transformation matrix of the camera coordinate system c and the global coordinate system g

重复上述操作，得到第i(i＝1,2,3,4,5)个惯性传感器坐标系n_i与全局坐标系g之间的旋转矩阵

1.3) Place the inertial sensor IMU at the global coordinate system g, so that the inertial sensor coordinate system n is aligned with the global coordinate system g, and the output value of the inertial sensor at this time is obtained, which is the difference between the inertial sensor coordinate system n and the global coordinate system g. rotation matrix

Repeat the above operations to obtain the rotation matrix between the ith (i=1, 2, 3, 4, 5) inertial sensor coordinate system n _i and the global coordinate system g

在初始时刻测试者做“T-pose”校准姿态，此时定义IMU_i的测量值为R_{i_initial}，则IMU_i与对应骨骼坐标系b_i之间的旋转矩阵

表示为：1.4) As shown in Figure 2, the IMU is worn at the corresponding bone points of the left hand, right hand, left forearm, right forearm, and pelvis, and there is no displacement between IMU _i and the corresponding bone coordinate system b _i , that is,

At the initial moment, the tester performs the "T-pose" calibration posture. At this time, the measured value of IMU _i is defined as R _{i_initial} , then the rotation matrix between IMU _i and the corresponding bone coordinate system b _i

Expressed as:

Step 2) Set the output frequency of vision and inertia to 30HZ, and use inertial sensor rotation constraint E _R (θ), acceleration constraint E _A (θ), visual sensor position constraint E _P (θ) and human body pose prior constraint E _prior (θ) Build an optimization problem for the optimization items, and set the weight of each optimization item; the process is as follows:

2.1) The difference between the measured value and the estimated value of the rotation matrix of the skeleton node corresponding to IMU _i in the global coordinate system is used as the rotation term constraint of the IMU. The rotation matrix measurements of the corresponding skeleton nodes are expressed as:

where R _i is the rotation measurement of IMU _i . The estimated value of the rotation matrix of the corresponding skeleton node is expressed as:

Among them, P( _bi ) is the set of all parent bones of bone _bi .

In summary, the energy function of the rotation term is defined as:

Among them, ψ(·) extracts the vector part of the rotation matrix quaternion representation method, λ _R is the weight of the rotation term energy function, and ρ _R (·) represents a loss function.

表示为：2.2) Minimize the error between the IMU _i acceleration measurement value a _i and the estimated value as the acceleration constraint term of the IMU. acceleration estimate

Expressed as:

等式(6)左边(t-1)表示当前时刻使用上一时刻的加速度约束。全局坐标系下的加速度测量值

通过前一帧的旋转信息和加速度测量值计算得出。加速度测量值

表示为：in,

The left side (t-1) of equation (6) indicates that the current moment uses the acceleration constraint of the previous moment. Acceleration measurements in the global coordinate system

Calculated from rotation information and acceleration measurements from the previous frame. acceleration measurement

Expressed as:

where a _g is the acceleration of gravity.

In summary, the energy function of the acceleration term is defined as:

Among them, λ _A is the weight of the acceleration term energy function, and ρ _A (·) represents a loss function.

相机数量为n_c。骨骼节点位置的估计值

表示为：2.3) Obtain the global coordinates (x, y, z) of the human skeleton node from the depth camera of the vision sensor, and add a constraint item that minimizes the minimum between the measured value and the estimated value of the global position of the skeleton node. Define the number of skeletal nodes used for position constraints as n _p , and the position of the skeletal nodes in the camera c coordinate system is

The number of cameras is n _c . Estimates of skeletal node positions

Expressed as:

In summary, the energy function of the position constraint is defined as:

Among them, λ _P is the weight of the energy function of the position constraint term, and ρ _P (·) represents a loss function.

2.4) Considering that there are limitations in the freedom of movement of the actual bones, the pose prior term E _prior (θ) is used to limit the unreasonable movement of the joints. E _prior (θ) is established by the existing human pose estimation dataset "TotalCapture (2017)", which contains 126,000 frames of human motion pose data.

First, perform k-means clustering on all the data in the dataset, and select the clustering type k=126000/100=1260. Then take the mean of all the cluster centers to get the mean value μ of the pose. Finally, statistical analysis is performed on the original data to obtain the standard deviation σ of the pose and the upper and lower limits θ _max and θ _min of the degrees of freedom of each skeletal node. From this, the pose prior is defined as:

的维度为36，不对根节点的位移和旋转作限制，λ_prior为姿态先验项能量函数的权重，ρ_prior(·)表示一种损失函数。in,

The dimension of is 36, and the displacement and rotation of the root node are not restricted. λ _prior is the weight of the energy function of the attitude prior item, and ρ _prior ( ) represents a loss function.

2.5) To sum up, construct the optimization problem:

Among them, the loss function in E _A , E _P , and E _prior is set to ρ(x)=log(1+x), and the influence of outliers is limited by increasing the penalty for outliers in a set proportion. The weights of each optimization term are set as λ _R =0.1, λ _P =10, λ _A =0.005, λ _prior =0.0001.

以及惯性传感器的旋转测量值R_i和加速度测量值a_i，计算各优化项在统一坐标系后的传感器测量值

与估计值

Step 3) Read the position measurement of the vision sensor at each moment

and the rotation measurement value R _i and acceleration measurement value a _i of the inertial sensor, calculate the sensor measurement value of each optimization item after the unified coordinate system

with estimated value

Step 4) Solve the nonlinear least squares optimization problem, the optimal solution θ at each moment is the optimal rotation angle of each skeleton node of the human body at the current moment and the optimal global position of the root skeleton node n ₁ , and then according to the established human skeleton Node kinematics model to get the human pose estimation at the current moment;

As shown in Figure 1, steps 3) and 4) are repeatedly performed to complete the optimal estimation of the position and rotation of human joint points at each moment, and a real-time human pose estimation based on the fusion of visual and inertial information is obtained.

The content described in the embodiments of the present specification is merely an enumeration of the implementation forms of the inventive concept, and is only used for illustration purposes. The protection scope of the present invention should not be construed as being limited to the specific forms stated in this embodiment, and the protection scope of the present invention also extends to equivalent technical means that those of ordinary skill in the art can think of according to the inventive concept.

Claims (7)

1. a human body pose estimation method based on vision and inertial information fusion, is characterized in that, described method comprises the following steps: Step 1) Establish the kinematic model of each skeleton node of the human body, determine the optimization variable θ, and determine the homogeneous transformation matrix between the camera coordinate system c and the global coordinate system g

Rotation matrix between inertial coordinate system n and global coordinate system g

Displacement matrix between inertial sensor i and corresponding bone coordinate system b _i

and rotation matrix

Step 2) Set the visual and inertial output frequencies to be consistent, and use the inertial sensor rotation constraint _ER (θ), acceleration constraint EA (θ), visual sensor position constraint _EP (θ) and human body pose prior constraint _{E prior} ( θ) Construct an optimization problem for the optimization item, and set the weight of each optimization item; Step 3) Read the position measurement of the vision sensor at each moment

and the rotation measurement value R _i and acceleration measurement value a _i of the inertial sensor, calculate the sensor measurement value of each optimization item after the unified coordinate system

with estimated value

Step 4) Solve the nonlinear least squares optimization problem, and obtain the optimal solution θ at each moment, that is, the optimal rotation angle of each skeleton node of the human body and the optimal global position of the root skeleton node n ₁ at the current moment. Skeletal node kinematics model to obtain the human body pose estimation at the current moment; Step 5) Repeat steps 3) and 4) to complete the state estimation of each joint point of the human body at each moment, and obtain a real-time human body pose estimation based on the fusion of visual and inertial information. 2. the human body attitude estimation method based on vision and inertial information fusion as claimed in claim 1, is characterized in that, in described step 1), described camera coordinate system c represents the coordinate system of depth camera, inertial coordinate system n represents the unified coordinate system of all inertial sensors after calibration, and the global coordinate system g is aligned with the initial coordinate system of the skeleton node. 3. the human body attitude estimation method based on vision and inertial information fusion as claimed in claim 1 or 2, is characterized in that, in described step 2), described rotation constraint E _R (θ) rotates by each IMU The difference between the measured value and the estimated value of the matrix is established; the described acceleration constraint E _A (θ) is established by minimizing the difference between the measured value and estimated value of each IMU acceleration; the described position constraint E _P ( θ) is established by minimizing the difference between the measured value and estimated value of the global position of each skeleton node; the human pose prior constraint E _prior (θ) is established through the existing human pose estimation data set. 4. the human body posture estimation method based on vision and inertial information fusion as described in claim 1 or 2, is characterized in that, in described step 3), described

is the position information of each joint point of the human body read from the vision sensor, R _i is the rotation information read from the inertial gyroscope, and a _i is the acceleration information read from the inertial accelerometer. 5. the human body pose estimation method based on vision and inertial information fusion as claimed in claim 1 or 2, is characterized in that, in described step 4), described root skeleton node n ₁ is located at human pelvic joint point . 6. the human body posture estimation method based on vision and inertial information fusion as described in claim 1 or 2, is characterized in that, the process of described step 1) is: 1.1) Human bones are defined as interconnected rigid bodies. The initial coordinate system B of the skeleton node is aligned with the global coordinate system g, and the number of upper body bones is defined as n _b = 13. The skeleton nodes are the left hand, right hand, left forearm, right forearm, and left upper arm. , right upper arm, left shoulder, right shoulder, spine 1-4, and pelvis, where the pelvis is regarded as the root bone node n ₁ , and the child bone node n _b (b≥2) has a relative rotation matrix R _b and its parent node. Relatively constant displacement t _b , each bone has 3 rotational degrees of freedom, the root node has a global displacement (x ₁ , y ₁ , z ₁ ), with 42 (d=3+3×n _b =42) A degree of freedom is used to describe the motion of the entire upper body of the human body. The 42 variables are recorded as a 42-dimensional vector θ, and the vector θ is used as the optimization variable of the optimization problem. The forward kinematics formula deduces that each rigid bone is in the global coordinate system. The homogeneous transformation matrix of

where P(b) is the set of all bones; 1.2) Place the two visual sensors in front of the tester respectively, the visual sensor is L=2 meters away from the tester, and use the "Zhang Zhengyou camera calibration method" to obtain the translation matrix of the two cameras for the global coordinate system g

and rotation matrix

Then determine the homogeneous transformation matrix of the camera coordinate system c and the global coordinate system g

1.3) Place the inertial sensor IMU at the global coordinate system g, so that the inertial sensor coordinate system n is aligned with the global coordinate system g, and the output value of the inertial sensor at this time is obtained, which is the difference between the inertial sensor coordinate system n and the global coordinate system g. rotation matrix

Repeat the above operations to obtain the rotation matrix between the ith (i=1, 2, 3, 4, 5) inertial sensor coordinate system n _i and the global coordinate system g

1.4) The IMU is worn at the corresponding bone points of the left hand, right hand, left forearm, right forearm, and pelvis, and there is no displacement between IMU _i and the corresponding bone coordinate system b _i , that is,

At the initial moment, the tester performs the "T-pose" calibration posture. At this time, the measured value of IMU _i is defined as R _{i_initial} , then the rotation matrix between IMU _i and the corresponding bone coordinate system b _i

Expressed as:

7. the human body posture estimation method based on vision and inertial information fusion as claimed in claim 6, is characterized in that, the process of described step 2) is: 2.1) The difference between the measured value and the estimated value of the rotation matrix of the skeleton node corresponding to IMU _i in the global coordinate system is used as the rotation item constraint of the IMU, and the measured value of the rotation matrix of the corresponding skeleton node is expressed as:

Among them, R _i is the rotation measurement value of IMU _i , and the estimated value of the rotation matrix of the corresponding skeleton node is expressed as:

Among them, P( _bi ) is the set of all parent bones of bone _bi ; In summary, the energy function of the rotation term is defined as:

Among them, ψ(·) extracts the vector part of the rotation matrix quaternion representation method, λ _R is the weight of the rotation term energy function, and ρ _R (·) represents a loss function; 2.2) Minimize the error between the IMU _i acceleration measurement value a _i and the estimated value as the acceleration constraint term of the IMU, and the acceleration estimated value

Expressed as:

in,

The left side (t-1) of equation (6) represents the acceleration measurement value in the global coordinate system using the acceleration constraint of the previous moment at the current moment

Calculated from the rotation information and acceleration measurement value of the previous frame, the acceleration measurement value

Expressed as:

Among them, a _g is the acceleration of gravity; In summary, the energy function of the acceleration term is defined as:

Among them, λ _A is the weight of the acceleration term energy function, and ρ _A (·) represents a loss function; 2.3) Obtain the global coordinates (x, y, z) of the human skeleton node from the depth camera of the vision sensor, add a constraint item that minimizes the minimum between the measured value and the estimated value of the global position of the skeleton node, and define the position constraint The number of bone nodes of the item is n _p , and the position of the bone nodes in the camera c coordinate system is

The number of cameras is n _c , the estimated value of the position of the skeleton node

Expressed as:

In summary, the energy function of the position constraint is defined as:

Among them, λ _P is the weight of the energy function of the position constraint term, and ρ _P (·) represents a loss function; 2.4) Considering the limitations of the freedom of movement of the actual bones, the pose prior term E _prior (θ) is used to limit the unreasonable motion of the joints. E _prior (θ) uses the existing human pose estimation dataset "TotalCapture (2017)" to build, which contains 126,000 frames of human motion pose data; First, perform k-means clustering on all the data in the dataset, select the clustering type k=126000/100=1260, then take the mean value of all cluster centers to obtain the mean value μ of the pose, and finally perform statistical analysis on the original data to obtain The standard deviation σ of the pose and the upper and lower limits θ _max and θ _min of the degrees of freedom of each skeletal node, thus, the pose prior term is defined as:

in,

The dimension of is 36, and the displacement and rotation of the root node are not restricted, λ _prior is the weight of the energy function of the attitude prior item, and ρ _prior ( ) represents a loss function; 2.5) To sum up, construct the optimization problem:

Among them, the loss function in E _A , E _P , and E _prior is set to ρ(x)=log(1+x), and the influence of outliers is limited by increasing the penalty for outliers according to the set ratio. The weights of the terms are set to λ _R =0.1, λ _P =10, λ _A =0.005, λ _prior =0.0001.

Images