CN118015288B - Hand key point denoising method and device, electronic equipment and storage medium - Google Patents
Hand key point denoising method and device, electronic equipment and storage medium Download PDFInfo
- Publication number
- CN118015288B CN118015288B CN202410045087.8A CN202410045087A CN118015288B CN 118015288 B CN118015288 B CN 118015288B CN 202410045087 A CN202410045087 A CN 202410045087A CN 118015288 B CN118015288 B CN 118015288B
- Authority
- CN
- China
- Prior art keywords
- coordinate system
- finger
- joint
- key points
- coordinates
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Active
Links
- 238000000034 method Methods 0.000 title claims abstract description 49
- 238000001914 filtration Methods 0.000 claims abstract description 25
- 238000009499 grossing Methods 0.000 claims abstract description 10
- 210000003811 finger Anatomy 0.000 claims description 179
- 210000000811 metacarpophalangeal joint Anatomy 0.000 claims description 112
- 210000001503 joint Anatomy 0.000 claims description 72
- 210000000707 wrist Anatomy 0.000 claims description 47
- 210000003857 wrist joint Anatomy 0.000 claims description 34
- 239000011159 matrix material Substances 0.000 claims description 29
- 239000013598 vector Substances 0.000 claims description 28
- 230000005057 finger movement Effects 0.000 claims description 21
- 238000004590 computer program Methods 0.000 claims description 20
- 238000013519 translation Methods 0.000 claims description 16
- 210000004932 little finger Anatomy 0.000 claims description 13
- 238000000354 decomposition reaction Methods 0.000 claims description 6
- 210000000236 metacarpal bone Anatomy 0.000 claims description 5
- 238000001514 detection method Methods 0.000 abstract description 10
- 230000008569 process Effects 0.000 description 10
- 210000003813 thumb Anatomy 0.000 description 10
- 238000004364 calculation method Methods 0.000 description 6
- 238000010586 diagram Methods 0.000 description 6
- 230000000694 effects Effects 0.000 description 6
- 238000004422 calculation algorithm Methods 0.000 description 3
- 238000006243 chemical reaction Methods 0.000 description 3
- 230000009286 beneficial effect Effects 0.000 description 2
- 238000004891 communication Methods 0.000 description 2
- 230000006870 function Effects 0.000 description 2
- 210000002411 hand bone Anatomy 0.000 description 2
- 230000003993 interaction Effects 0.000 description 2
- 238000012986 modification Methods 0.000 description 2
- 230000004048 modification Effects 0.000 description 2
- 238000012545 processing Methods 0.000 description 2
- 230000001360 synchronised effect Effects 0.000 description 2
- 238000013459 approach Methods 0.000 description 1
- 238000005452 bending Methods 0.000 description 1
- 230000001413 cellular effect Effects 0.000 description 1
- 238000013461 design Methods 0.000 description 1
- 230000007613 environmental effect Effects 0.000 description 1
- 230000003287 optical effect Effects 0.000 description 1
- 230000009467 reduction Effects 0.000 description 1
- 230000004044 response Effects 0.000 description 1
- 230000009466 transformation Effects 0.000 description 1
Classifications
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06V—IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING
- G06V10/00—Arrangements for image or video recognition or understanding
- G06V10/20—Image preprocessing
- G06V10/30—Noise filtering
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06V—IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING
- G06V10/00—Arrangements for image or video recognition or understanding
- G06V10/20—Image preprocessing
- G06V10/22—Image preprocessing by selection of a specific region containing or referencing a pattern; Locating or processing of specific regions to guide the detection or recognition
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06V—IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING
- G06V40/00—Recognition of biometric, human-related or animal-related patterns in image or video data
- G06V40/10—Human or animal bodies, e.g. vehicle occupants or pedestrians; Body parts, e.g. hands
- G06V40/107—Static hand or arm
Landscapes
- Engineering & Computer Science (AREA)
- Physics & Mathematics (AREA)
- General Physics & Mathematics (AREA)
- Multimedia (AREA)
- Theoretical Computer Science (AREA)
- Human Computer Interaction (AREA)
- Collating Specific Patterns (AREA)
- Image Analysis (AREA)
Abstract
The invention discloses a hand key point denoising method, a device, electronic equipment and a storage medium, wherein when a hand key point is denoised, coordinates of the hand key point before denoising under a first coordinate system are parameterized to obtain a joint rotation angle of a current hand and a scale factor for representing the scaling of the current hand relative to a standard hand, and then Kalman filtering is performed on the joint rotation angle to obtain a denoised joint rotation angle; performing exponential smoothing on the scale factors to obtain denoised scale factors; and finally, inversely parameterizing the denoised joint rotation angle and the denoised scale factor to obtain the coordinates of the denoised hand key points under the first coordinate system, thereby finishing the denoising of the hand key points. By implementing the method and the device, noise in the key points of the hand can be effectively removed, and the accuracy and the stability of the key point detection are improved.
Description
Technical Field
The present invention relates to the field of computer vision, and in particular, to a hand key point denoising method, device, electronic apparatus, and storage medium.
Background
In the field of computer vision, hand keypoint detection is the basis for many applications, such as gesture recognition, human-computer interaction, etc. However, due to the image acquisition equipment and environmental factors, hand keypoint detection is often subject to noise, resulting in inaccurate or severe keypoint jitter. Therefore, a method capable of removing noise of the key points of the hand is required to improve reliability and stability of the key point detection.
Disclosure of Invention
The embodiment of the invention provides a hand key point denoising method, a device, electronic equipment and a storage medium, which can effectively remove noise in hand key points and improve the accuracy and stability of key point detection.
The embodiment of the invention provides a hand key point denoising method, which comprises the following steps:
acquiring coordinates of hand key points of the current hand before denoising under a first coordinate system;
Parameterizing coordinates of key points of the hand before denoising under a first coordinate system to obtain a joint rotation angle of the current hand and a scale factor for representing the scaling of the current hand relative to a standard hand;
carrying out Kalman filtering on the joint rotation angle to obtain a denoised joint rotation angle; performing exponential smoothing on the scale factors to obtain denoised scale factors;
And carrying out inverse parameterization on the denoised joint rotation angle and the denoised scale factor to obtain coordinates of the denoised hand key points under a first coordinate system.
Further, the first coordinate system is a world coordinate system; the hand key points comprise: wrist joint key points and finger key points corresponding to all fingers; the finger key points of each finger include: metacarpophalangeal joint key points, proximal interphalangeal joint key points, distal interphalangeal joint key points and fingertip key points; the wrist joint key points and the metacarpophalangeal joint key points of all fingers form metacarpal bone key points;
the determining of the scale factor includes:
Based on the coordinates of the hand key points in the world coordinate system before denoising and the coordinates of the preset average metacarpal key points in the wrist coordinate system, the reciprocal of the scale factor is calculated by the following formula:
U，S，VT＝svd(H)；
R01＝(U*VT)T；
μ-1＝∑((T1)T*(R01*(P0)T))/∑(P0*P0)；
calculating the scale factor by the reciprocal of the scale factor;
Wherein, Is the coordinate of the metacarpal key point under the world coordinate system; t1 is the coordinate of the average metacarpal key point under the wrist coordinate system; svd (H) is represented as singular value decomposition of matrix H; u is a left singular vector; s is a diagonal matrix composed of singular values of H; v is a group of orthogonal bases which are formed by right singular vectors and describe a wrist coordinate system; v T is the transposed matrix of V; r 01 is a rotation matrix corresponding to the world coordinate system when the world coordinate system is converted into the wrist coordinate system; p 0 is the coordinates of the key points of the hand in the world coordinate system; mu -1 is the inverse of the scale factor.
Further, the determining of the joint rotation angle includes:
the translation vector corresponding to the world coordinate system when it is converted to the wrist coordinate system is calculated by the following formula:
t01＝-R01*mean(P0)T+mean(T1)T；
According to the reciprocal of the scale factor, the rotation moment and the translation vector corresponding to the world coordinate system when the world coordinate system is converted into the wrist coordinate system, converting the coordinate of the hand key point before denoising under the first coordinate system into the wrist coordinate system, and replacing the coordinate of the average metacarpal key point under the wrist coordinate system with the coordinate of the metacarpal key point converted corresponding to the current hand to obtain the final coordinate of the hand key point under the wrist coordinate system;
For each finger, converting the final coordinates of the finger key points under the wrist coordinate system to corresponding metacarpophalangeal joint coordinate systems to obtain the coordinates of the finger key points under the corresponding metacarpophalangeal joint coordinate systems; according to the coordinates of the key points of the finger parts under the corresponding metacarpophalangeal joint coordinate system, calculating to obtain the transverse swing angle of the metacarpophalangeal joint;
converting coordinates of the finger key points under the metacarpophalangeal joint coordinate system into corresponding finger movable plane coordinate systems to obtain coordinates of the finger key points under the finger movable plane coordinate systems; calculating pitch angles of metacarpophalangeal joints, proximal interphalangeal joints and distal interphalangeal joints according to coordinates of the key points of the finger parts under the finger movement plane coordinate system;
Obtaining the joint rotation angle of the current hand according to the transverse swing angle of the metacarpophalangeal joint, the pitch angle of the proximal interphalangeal joint and the pitch angle of the distal interphalangeal joint of each finger;
the coordinate origin of each finger under the metacarpophalangeal joint coordinate system is a corresponding metacarpophalangeal joint key point, the Y-axis direction is the wrist joint key point to the metacarpophalangeal joint key point, the X-axis direction is the cross of the Y-axis and the Z-axis direction, the Z-axis direction is satisfied when the finger is not a little finger (P i,1-P0)×(Pi+1,1-P0), the Z-axis direction is parallel to the Z-axis of the ring finger when the finger is a little finger, P i,1 is the metacarpophalangeal key point of the current finger, and P i+1,1 is the metacarpophalangeal key point of the next finger adjacent to the current finger;
the finger movable plane coordinate system corresponding to each finger is obtained by rotating around the Z axis according to the transverse swing angle of the metacarpophalangeal joint by the corresponding metacarpophalangeal joint coordinate system.
Further, the calculating to obtain the lateral swing angle of the metacarpophalangeal joint according to the coordinates of the finger key points under the corresponding metacarpophalangeal joint coordinate system includes:
the lateral swing angle of the metacarpophalangeal joint is calculated by the following formula:
wherein, theta i,1 is the transverse swing angle of the metacarpophalangeal joint of the ith finger; Is the X coordinate of the key point of the proximal interphalangeal joint of the ith finger under the metacarpophalangeal joint coordinate system; /(I) Is the Y coordinate of the proximal interphalangeal joint key point of the ith finger in the metacarpophalangeal joint coordinate system.
Further, according to the coordinates of the key points of the finger in the finger movement plane coordinate system, calculating the pitch angle of the metacarpophalangeal joint, the pitch angle of the proximal interphalangeal joint and the pitch angle of the distal interphalangeal joint, including:
the pitch angle of the metacarpophalangeal joint is calculated by the following formula:
the pitch angle of the proximal interphalangeal joint is calculated by the following formula:
The pitch angle of the distal interphalangeal joint is calculated by the following formula:
Wherein OY 3 is a unit vector (0, 1) in the Y direction of the finger movement plane coordinate system; θ i,2 is the pitch angle of the metacarpophalangeal joint; θ i,3 is the pitch angle of the proximal interphalangeal joint; θ i,4 is the pitch angle of the distal interphalangeal joint; The key point of the metacarpophalangeal joint of the ith finger is the coordinate of the zoy plane of the corresponding finger movable plane coordinate system; /(I) As the key point of the proximal interphalangeal joint of the ith finger, the coordinates of the zoy plane of the corresponding finger movement plane coordinate system; /(I)Coordinates of a zoy plane of a corresponding finger movement plane coordinate system are distal interphalangeal joint key points of the ith finger; /(I)And the coordinates of the key point of the fingertip of the ith finger on the zoy plane of the corresponding finger moving plane coordinate system.
Further, the performing kalman filtering on the joint rotation angle includes: determining a corresponding filter coefficient according to the joint rotation angle; and carrying out Kalman filtering on the joint rotation angle according to the filter coefficient.
Further, performing inverse parameterization on the denoised joint rotation angle and the denoised scale factor to obtain coordinates of denoised hand key points under a first coordinate system, where the coordinates include:
For each finger, calculating the coordinates of key points at the back of the filter finger under a finger movement plane coordinate system according to the filtered joint rotation angle and the preset average phalange length;
Calculating the coordinates of the key points of the back of the filter finger under the metacarpophalangeal joint coordinate system according to the coordinates of the key points of the back of the filter finger under the finger movable plane coordinate system and the corresponding rotation matrix when the metacarpophalangeal joint coordinate system is converted into the finger movable plane coordinate system;
Calculating the coordinates of the key points at the back of the filter finger under the coordinate system of the wrist joint according to the coordinates of the key points at the back of the filter finger under the coordinate system of the metacarpophalangeal joint, the corresponding rotation matrix and translation vector when the coordinate system of the wrist joint is converted into the coordinate system of the metacarpophalangeal joint;
And according to the corresponding rotation moment and translation vector when the world coordinate system is converted into the wrist coordinate system, restoring the coordinates of the filtered finger key points under the wrist joint coordinate system and the coordinates of the filtered wrist joints under the wrist joint coordinate system to the world coordinate system to obtain the coordinates of the denoised hand key points under the first coordinate system.
On the basis of the method item embodiment, the invention correspondingly provides a device item embodiment:
an embodiment of the present invention provides a hand key point denoising apparatus, including: the system comprises an initial key point coordinate acquisition module, a parameterization module, a denoising module and an inverse parameterization module;
The initial key point coordinate acquisition module is used for acquiring coordinates of hand key points of the current hand under a first coordinate system before denoising;
the parameterization module is used for parameterizing coordinates of key points of the denoising front hand under a first coordinate system to obtain a joint rotation angle of the current hand and a scale factor used for representing the scaling of the current hand relative to a standard hand;
the denoising module is used for carrying out Kalman filtering on the joint rotation angle to obtain a denoised joint rotation angle; performing exponential smoothing on the scale factors to obtain denoised scale factors;
the inverse parameterization module is used for carrying out inverse parameterization on the denoised joint rotation angle and the denoised scale factor to obtain coordinates of the denoised hand key points under the first coordinate system.
On the basis of the method item embodiments, the invention correspondingly provides the electronic equipment item embodiments;
an embodiment of the present invention provides an electronic device, including a processor, a memory, and a computer program stored in the memory and configured to be executed by the processor, where the processor implements the hand key point denoising method according to any one of the present invention when executing the computer program.
On the basis of the method item embodiment, the invention correspondingly provides a storage medium item embodiment;
An embodiment of the present invention provides a storage medium, where the storage medium includes a stored computer program, and when the computer program runs, controls a device where the storage medium is located to execute the hand key point denoising method according to any one of the present invention.
The embodiment of the invention has the following beneficial effects:
The embodiment of the invention provides a hand key point denoising method, a device, electronic equipment and a storage medium, wherein when a hand key point is denoised, the method parameterizes coordinates of the hand key point before denoising under a first coordinate system to obtain a joint rotation angle of a current hand and a scale factor for representing the scaling of the current hand relative to a standard hand, and then Kalman filtering is performed on the joint rotation angle to obtain a denoised joint rotation angle; performing exponential smoothing on the scale factors to obtain denoised scale factors; and finally, inversely parameterizing the denoised joint rotation angle and the denoised scale factor to obtain the coordinates of the denoised hand key points under the first coordinate system, thereby finishing the denoising of the hand key points. The Kalman filtering and exponential smoothing method is a filtering algorithm commonly used for key point denoising scenes, and compared with the method for directly denoising three-dimensional space coordinates of key points, the method firstly carries out parameterization treatment on the coordinates to obtain joint rotation angles which are more in line with linear dynamic characteristics, and noise is more prone to Gaussian distribution. Therefore, the Kalman filtering is carried out on the joint rotation angle, so that the filtering effect with quick response and strong noise resistance can be obtained more easily, and the denoising effect is improved.
Drawings
Fig. 1 is a flowchart of a hand key point denoising method according to an embodiment of the present invention.
Fig. 2 is a schematic diagram of a hand key point according to an embodiment of the present invention.
Fig. 3 is a schematic view of a lateral swing angle of a metacarpophalangeal joint according to an embodiment of the present invention.
Fig. 4 is a schematic view of pitch angles of metacarpophalangeal joints, proximal interphalangeal joints, and distal interphalangeal joints as provided by an embodiment of the present invention.
Fig. 5 is a schematic diagram of coordinate systems according to an embodiment of the present invention.
Fig. 6 is a further schematic view of pitch angles of metacarpophalangeal joints, proximal interphalangeal joints, and distal interphalangeal joints as provided in accordance with an embodiment of the present invention.
FIG. 7 is a schematic diagram of a frame of hand image according to an embodiment of the present invention;
fig. 8 is a schematic diagram of a denoising front hand key point according to an embodiment of the present invention.
Fig. 9 is a schematic diagram of a denoised hand key point according to an embodiment of the present invention.
Fig. 10 is a schematic structural diagram of a hand key point denoising apparatus according to an embodiment of the present invention.
Detailed Description
The following description of the embodiments of the present invention will be made clearly and completely with reference to the accompanying drawings, in which it is apparent that the embodiments described are only some embodiments of the present invention, but not all embodiments. All other embodiments, which can be made by those skilled in the art based on the embodiments of the invention without making any inventive effort, are intended to be within the scope of the invention.
Unless defined otherwise, all technical and scientific terms used herein have the same meaning as commonly understood by one of ordinary skill in the art to which this application belongs; the terminology used herein is for the purpose of describing particular embodiments only and is not intended to be limiting of the application; the terms "comprising" and "having" and any variations thereof in the description of the application and the claims and the description of the drawings above are intended to cover a non-exclusive inclusion.
In the description of embodiments of the present application, the technical terms "first," "second," and the like are used merely to distinguish between different objects and are not to be construed as indicating or implying a relative importance or implicitly indicating the number of technical features indicated, a particular order or a primary or secondary relationship. In the description of the embodiments of the present application, the meaning of "plurality" is two or more unless explicitly defined otherwise.
Reference herein to "an embodiment" means that a particular feature, structure, or characteristic described in connection with the embodiment may be included in at least one embodiment of the application. The appearances of such phrases in various places in the specification are not necessarily all referring to the same embodiment, nor are separate or alternative embodiments mutually exclusive of other embodiments. Those of skill in the art will explicitly and implicitly appreciate that the embodiments described herein may be combined with other embodiments.
In the description of the embodiments of the present application, the term "and/or" is merely an association relationship describing an association object, and indicates that three relationships may exist, for example, a and/or B may indicate: a exists alone, A and B exist together, and B exists alone. In addition, the character "/" herein generally indicates that the front and rear associated objects are an "or" relationship.
In the description of the embodiments of the present application, the term "plurality" means two or more (including two), and similarly, "plural sets" means two or more (including two), and "plural sheets" means two or more (including two).
In the description of the embodiments of the present application, unless explicitly specified and limited otherwise, the terms "mounted," "connected," "secured" and the like should be construed broadly and may be, for example, fixedly connected, detachably connected, or integrally formed; or may be mechanically or electrically connected; can be directly connected or indirectly connected through an intermediate medium, and can be communicated with the inside of two elements or the interaction relationship of the two elements. The specific meaning of the above terms in the embodiments of the present application will be understood by those of ordinary skill in the art according to specific circumstances.
As shown in fig. 1, an embodiment of the present invention provides a hand key point denoising method, which at least includes:
Step S1: and acquiring coordinates of the hand key points of the current hand before denoising under a first coordinate system.
Illustratively, as shown in fig. 2, there are 21 hand key points P 0-P54 in total in the present invention; wherein, P 0 is the wrist joint key point, P 11-P14 is the thumb finger key point, P 11 is the thumb metacarpophalangeal joint key point, P 12 is the thumb proximal interphalangeal joint key point, P 13 is the thumb distal interphalangeal joint key point, P 14 is the thumb fingertip key point, P 21-P24 is the index finger key point, P 21 is the index finger metacarpophalangeal joint key point, P 22 is the index finger proximal interphalangeal joint key point, p 23 is the distal interphalangeal joint key point of the index finger, P 24 is the fingertip key point of the index finger, P 31-P34 is the interphalangeal joint key point of the middle finger, P 31 is the metacarpophalangeal joint key point of the middle finger, P 32 is the proximal interphalangeal joint key point of the middle finger, P 33 is the distal interphalangeal joint key point of the middle finger, P 34 is the fingertip key point of the middle finger, P 41-P44 is the interphalangeal joint key point of the ring finger, P 41 is the metacarpophalangeal joint key point of the ring finger, P 42 is the proximal interphalangeal joint key point of the ring finger, p 43 is distal interphalangeal joint keypoints of the ring finger, P 44 is fingertip keypoints of the ring finger, P 51-P54 is metacarpophalangeal joint keypoints of the little finger, P 51 is proximal interphalangeal joint keypoints of the little finger, P 53 is distal interphalangeal joint keypoints of the little finger, and P 54 is fingertip keypoints of the little finger.
The wrist joint key point P 0, the metacarpophalangeal joint key point P 11 of the thumb, the metacarpophalangeal joint key point P 21 of the index finger, the metacarpophalangeal joint key point P 31 of the middle finger, the metacarpophalangeal joint key point P 41 of the ring finger and the metacarpophalangeal joint key point P 51 of the little finger form metacarpal bone key points which are described below, and are denoted by P palm in the following, and the { P 0,P11,P21,P31,P41,P51 } is P palm.
Preferably, the first coordinate system is a world coordinate system, which is denoted by S 0 in the following.
The current hand is shot by the structured light depth camera, the shot image is detected by the key point detection model, the hand key points of the current hand can be detected, and the coordinates of the hand key points of the current hand under the world coordinate system can be obtained through conversion of the camera internal parameters and the depth query algorithm.
Step S2: and parameterizing coordinates of key points of the denoising front hand under a first coordinate system to obtain a joint rotation angle of the current hand and a scale factor for representing the scaling of the current hand relative to the standard hand.
Specifically, the parameterized model of the present invention will be described first:
In the present invention, a parameterized model M (θ 5*4,μ,T6*3,L5*3);θ5*4 is a joint rotation angle, μ is a scale factor, T 6*3 is a preset average metacarpal key point (i.e., the average metacarpal key point T 1),L5*3 is a preset average phalange length;
Considering that the motion and size of the hand are the most important features of the hand, the present invention uses only θ 5*4 and μ as control parameters of the model. This approach ignores differences in hand shape and assumes that palm shape is unchanged, experimental data shows that this approximation and assumption is beneficial for denoising and saves computation. While T 6*3 and L 5*3 are taken as fixed built-in parameters. And acquiring hand data of a plurality of persons in advance to obtain the average value of key points of a rigid structure of the metacarpal bones and the average value of the phalangeal lengths, thereby obtaining the average metacarpal key points and the average phalangeal lengths.
Defining a joint rotation angle theta i*j; wherein i=1, 2,3,4,5; j=1, 2,3,4; for each element of the joint rotation angle θ 5*4,θ5*4, the euler angles of a certain knuckle are represented, each euler angle corresponding to one degree of freedom of the knuckle. i=1, 2,3,4,5, in turn, indicates thumb, index finger, middle finger, ring finger, little finger; j=1, the lateral swing angle of the metacarpophalangeal joint (as indicated by the yellow arrow in fig. 3); j=2, 3,4 indicates the pitch angle of the metacarpophalangeal joint, the pitch angle of the proximal interphalangeal joint, and the pitch angle of the distal interphalangeal joint, respectively (as indicated by the green arrow in fig. 4, for example, the index finger). The above design has two degrees of freedom based on anatomical knowledge, the interphalangeal joint has only one degree of freedom, so that the motion of each finger requires a total of 4 angles to represent.
For the scale factor μ, the larger the value of μ, which represents the scaling of the current hand and the average hand (i.e., standard hand described above), the larger the current hand size. The size of the hand bones is affected by age, height, but hand bones of different sizes are approximately in a linearly scaled geometry, so that the hand size is represented using only one scale factor μ.
The parameterization process is described below:
in a preferred embodiment, the determination of the scale factor comprises:
Based on the coordinates of the hand key points in the world coordinate system before denoising and the coordinates of the preset average metacarpal key points in the wrist coordinate system, the reciprocal of the scale factor is calculated by the following formula:
U，S，VT＝svd(H)；
R01＝(U*VT)T；
μ-1＝∑((T1)T*(R01*(P0)T))/∑(P0*P0)；
calculating the scale factor by the reciprocal of the scale factor;
Wherein, Is the coordinate of the metacarpal key point under the world coordinate system; t 1 is the coordinate of the average metacarpal key point under the wrist coordinate system; svd (H) is represented as singular value decomposition of matrix H; u is a left singular vector; s is a diagonal matrix composed of singular values of H; v is a group of orthogonal bases which are formed by right singular vectors and describe a wrist coordinate system; v T is the transposed matrix of V; r 01 is a rotation matrix corresponding to the world coordinate system when the world coordinate system is converted into the wrist coordinate system; p 0 is the coordinates of the key points of the hand in the world coordinate system; mu -1 is the inverse of the scale factor.
In a preferred embodiment, the determination of the joint rotation angle comprises:
the translation vector corresponding to the world coordinate system when it is converted to the wrist coordinate system is calculated by the following formula:
t01＝-R01*mean(P0)T+mean(T1)T；
According to the reciprocal of the scale factor, the rotation moment and the translation vector corresponding to the world coordinate system when the world coordinate system is converted into the wrist coordinate system, converting the coordinate of the hand key point before denoising under the first coordinate system into the wrist coordinate system, and replacing the coordinate of the average metacarpal key point under the wrist coordinate system with the coordinate of the metacarpal key point converted corresponding to the current hand to obtain the final coordinate of the hand key point under the wrist coordinate system;
For each finger, converting the final coordinates of the finger key points under the wrist coordinate system to corresponding metacarpophalangeal joint coordinate systems to obtain the coordinates of the finger key points under the corresponding metacarpophalangeal joint coordinate systems; according to the coordinates of the key points of the finger parts under the corresponding metacarpophalangeal joint coordinate system, calculating to obtain the transverse swing angle of the metacarpophalangeal joint;
converting coordinates of the finger key points under the metacarpophalangeal joint coordinate system into corresponding finger movable plane coordinate systems to obtain coordinates of the finger key points under the finger movable plane coordinate systems; calculating pitch angles of metacarpophalangeal joints, proximal interphalangeal joints and distal interphalangeal joints according to coordinates of the key points of the finger parts under the finger movement plane coordinate system;
Obtaining the joint rotation angle of the current hand according to the transverse swing angle of the metacarpophalangeal joint, the pitch angle of the proximal interphalangeal joint and the pitch angle of the distal interphalangeal joint of each finger;
the coordinate origin of each finger under the metacarpophalangeal joint coordinate system is a corresponding metacarpophalangeal joint key point, the Y-axis direction is the wrist joint key point to the metacarpophalangeal joint key point, the X-axis direction is the cross of the Y-axis and the Z-axis direction, the Z-axis direction is satisfied when the finger is not a little finger (P i,1-P0)×(Pi+1,1-P0), the Z-axis direction is parallel to the Z-axis of the ring finger when the finger is a little finger, P i,1 is the metacarpophalangeal key point of the current finger, and P i+1,1 is the metacarpophalangeal key point of the next finger adjacent to the current finger;
the finger movable plane coordinate system corresponding to each finger is obtained by rotating around the Z axis according to the transverse swing angle of the metacarpophalangeal joint by the corresponding metacarpophalangeal joint coordinate system.
In a preferred implementation, the calculating the lateral swing angle of the metacarpophalangeal joint according to the coordinates of the finger key points under the corresponding metacarpophalangeal joint coordinate system includes:
the lateral swing angle of the metacarpophalangeal joint is calculated by the following formula:
wherein, theta i,1 is the transverse swing angle of the metacarpophalangeal joint of the ith finger; Is the X coordinate of the key point of the proximal interphalangeal joint of the ith finger under the metacarpophalangeal joint coordinate system; /(I) Is the Y coordinate of the proximal interphalangeal joint key point of the ith finger in the metacarpophalangeal joint coordinate system.
In a preferred embodiment, the calculating the pitch angle of metacarpophalangeal joints, the pitch angle of proximal interphalangeal joints and the pitch angle of distal interphalangeal joints according to the coordinates of the finger key points in the coordinate system of the finger movement plane includes:
the pitch angle of the metacarpophalangeal joint is calculated by the following formula:
the pitch angle of the proximal interphalangeal joint is calculated by the following formula:
The pitch angle of the distal interphalangeal joint is calculated by the following formula:
Wherein OY 3 is a unit vector (0, 1) in the Y direction of the finger movement plane coordinate system; θ i,2 is the pitch angle of the metacarpophalangeal joint; θ i,3 is the pitch angle of the proximal interphalangeal joint; θ i,4 is the pitch angle of the distal interphalangeal joint; The key point of the metacarpophalangeal joint of the ith finger is the coordinate of the zoy plane of the corresponding finger movable plane coordinate system; /(I) As the key point of the proximal interphalangeal joint of the ith finger, the coordinates of the zoy plane of the corresponding finger movement plane coordinate system; /(I)Coordinates of a zoy plane of a corresponding finger movement plane coordinate system are distal interphalangeal joint key points of the ith finger; /(I)And the coordinates of the key point of the fingertip of the ith finger on the zoy plane of the corresponding finger moving plane coordinate system.
The above parameterization is described in detail below: for convenience of description, P i,j or P ij is used below to represent the j-th key point of the finger i, i=1, 2,3,4,5 represents the thumb, index finger, middle finger, ring finger, and little finger in order; j=1, 2,3,4 sequentially represent metacarpophalangeal joints, proximal interphalangeal joints, distal interphalangeal joints, and fingertips, and particularly, P 0 is used to represent wrist joint key points separately, and a specific arrangement can be seen in fig. 2;
For calculating θ 5*4, except the world coordinate system S 0, a three-level coordinate system is defined, which is: wrist coordinate system S 1, metacarpophalangeal joint coordinate system S 2, finger movement plane coordinate system S 3. The wrist coordinate system S 1, the metacarpophalangeal joint coordinate system S 2 and the finger movement plane coordinate system S 3 are set up in reference to fig. 5, and red arrows indicate that S 3 is obtained by rotating S 2 around the z axis, and specific definitions of the coordinate systems are described below. Accordingly, P 0 (comprising: ) P 1 (containing: /(I) ) P 2 (containing: /(I) ) P 3 (containing: /(I)) Representing the coordinates of key points of the current hand in a coordinate system S 0、S1、S2、S3;
The conversion from S 0 to S 1 means that the current hand is aligned with the built-in average metacarpal key point T 1 before the parameterization calculation of the finger portion can be performed. P 1 is derived from the inverse of the scale factor μ of the wrist node multiplied by the rotation matrix R 01 and the translation vector t 01:
Namely: p 1T＝μ-1R01P0T+t01 T;
Then (I.e., P 0,P11,P21,P31,P41,P51 in the S 1 coordinate system) at this point in time aligned(I.e., the above average metacarpal key point T 1), since the shape of the metacarpal is constant in motion, the average metacarpal key point/>, is used in the wrist coordinate system S1Replace the metacarpal key point of current hand, play preliminary denoising's effect:
For R 01、t01、μ-1, the invention uses singular value decomposition to obtain the R 01、t01、μ-1 through the corresponding formula; r 01、t01、μ-1 represents the current wrist position and orientation, which needs to be inverted for inverse parameterization, so R 01、t01、μ-1 is saved as an intermediate variable.
The conversion of S 1 to S 2 is performed by the 6 key points of the palmAnd (5) uniquely determining. Five fingers correspond to five S 2 coordinate systems, which are sequentially marked asThe coordinate system is defined as follows:
The origin of coordinates is P i,1, and the y-axis points to P 0 to P i,1; the z-axis direction (P i,1-P0)×(Pi+1,1-P0) (applicable to i < 5, set when i=5 Is ANDMay be parallel) and the x-axis is the y-axis multiplied by the z-axis.
The unit vectors of x, y and z axes of the S 2 coordinate system and the origin of coordinates are set, so that corresponding R 12,i (a rotation matrix corresponding to the wrist joint coordinate system when the wrist joint coordinate system is converted to the metacarpophalangeal joint coordinate system) and t 12,i (a translation vector corresponding to the wrist joint coordinate system when the wrist joint coordinate system is converted to the metacarpophalangeal joint coordinate system) can be obtained for each finger i according to a coordinate transformation formula, and the coordinate system of the finger i is obtainedCoordinates of (c):
Then at this time, the calculation of θ i,1 is in the metacarpophalangeal joint coordinate system Is completed by the following steps:
Next, θ i,1 can determine a rotation matrix R 23,i (i.e., a rotation matrix corresponding to the metacarpophalangeal joint coordinate system converted to the finger movable plane coordinate system) of "counterclockwise rotation about z axis θ i,1", i.e. Movable plane coordinate system to finger iIs transformed by:
calculation of θ i,2、θi,3、θi,4 is shown in As shown in fig. 6:
is the two-dimensional plane of finger i activity of the parameterized model, will/> Projection ontoZoy plane of (projected from blue polyline to green polyline), to obtain Pitch angle of metacarpophalangeal joints: /(I)
Pitch angle of proximal interphalangeal joint:
pitch angle of distal interphalangeal joint:
finally, according to the pitch angle of the metacarpophalangeal joints of each finger, the pitch angle of the proximal interphalangeal joints, the pitch angle of the distal interphalangeal joints and the transverse swing angle of the metacarpophalangeal joints, the joint rotation angle theta 5*4 of the whole hand can be obtained;
So far, all control parameters are solved, and the parameterization process is finished.
Step S3: carrying out Kalman filtering on the joint rotation angle to obtain a denoised joint rotation angle; and carrying out exponential smoothing on the scale factors to obtain the denoised scale factors.
Specifically, θ '5*4 and μ' are obtained by filtering the above θ 5*4 and μ;
In a preferred embodiment, said kalman filtering of said joint rotation angle comprises:
Determining a corresponding filter coefficient according to the joint rotation angle; and carrying out Kalman filtering on the joint rotation angle according to the filter coefficient.
In the embodiment of the invention, the filter coefficient corresponding to each joint rotation angle is preset and stored in the database, when filtering is performed, the corresponding filter coefficient is matched in the database according to the joint rotation angle calculated currently, and then Kalman filtering is performed on the joint rotation angle according to the matched filter coefficient, so that differential denoising is realized, and the denoising effect is further improved.
Step S4: and carrying out inverse parameterization on the denoised joint rotation angle and the denoised scale factor to obtain coordinates of the denoised hand key points under a first coordinate system.
In a preferred embodiment, the inverse parameterizing the denoised joint rotation angle and the denoised scale factor to obtain the coordinates of the denoised hand key point in the first coordinate system includes:
For each finger, calculating the coordinates of key points at the back of the filter finger under a finger movement plane coordinate system according to the filtered joint rotation angle and the preset average phalange length;
Calculating the coordinates of the key points of the back of the filter finger under the metacarpophalangeal joint coordinate system according to the coordinates of the key points of the back of the filter finger under the finger movable plane coordinate system and the corresponding rotation matrix when the metacarpophalangeal joint coordinate system is converted into the finger movable plane coordinate system;
Calculating the coordinates of the key points at the back of the filter finger under the coordinate system of the wrist joint according to the coordinates of the key points at the back of the filter finger under the coordinate system of the metacarpophalangeal joint, the corresponding rotation matrix and translation vector when the coordinate system of the wrist joint is converted into the coordinate system of the metacarpophalangeal joint;
And according to the corresponding rotation moment and translation vector when the world coordinate system is converted into the wrist coordinate system, restoring the coordinates of the filtered finger key points under the wrist joint coordinate system and the coordinates of the filtered wrist joints under the wrist joint coordinate system to the world coordinate system to obtain the coordinates of the denoised hand key points under the first coordinate system.
In particular, in a two-dimensional plane in which the finger i is active, i.e.From θ i,2、θi,3、θi,4 and phalange length L i,1、Li,2、Li,3 (i.e., the average phalange length of finger i in average phalange length L 5*3 of the entire hand), the filtered finger i is locatedKey point coordinates
Immediately after calculation of the filtered finger i is locatedKey point coordinates of (c):
And further calculating the key point coordinates of the finger i in the wrist joint coordinate system S 1 after filtering:
note that the wrist joint key point is constant at S 1/>
After the above operation is carried out on all the fingers i, the coordinates of the wrist joint key points after the filtering are carried outAnd restoring to a world coordinate system S 0:
And finally, sorting according to the definition of the original key points to obtain a final denoising result.
It should be noted that, in the above formula of the present invention, R 23 is a rotation matrix when the metacarpophalangeal joint coordinate system S 2 is converted to the finger movement plane coordinate system S 3, R 12 is a rotation matrix when the wrist coordinate system S 1 is converted to the metacarpophalangeal joint coordinate system S 2, T 12 is a translation vector when the wrist coordinate system S 1 is converted to the metacarpophalangeal joint coordinate system S 2, R 01 is a rotation matrix when the world coordinate system S 0 is converted to the wrist coordinate system S 1, T 01 is a translation vector when the world coordinate system S 0 is converted to the wrist coordinate system S 1, and in addition, the superscript T in all the formulas of the present invention represents a transpose of the corresponding element.
To further illustrate the denoising process, the following is exemplified by an actual scenario:
the example is a sequence of hand keypoints in a motion, obtained from a video captured by a structured light depth camera and a keypoint detection model, and has been converted to a world coordinate system in millimeters by camera internal parameters and depth query algorithms. The key point J 21×3 of one frame (as shown in figures 8 and 9) is selected to show the calculation process,
J21×3＝
[[-261.6630,127.8280,679.1311],
[-232.2920,101.4740,678.6361],
[-203.7620,84.3669,666.7366],
[-183.0360,65.3114,651.6150],
[-169.9880,48.2716,636.2446],
[-173.9680,118.6460,648.3915],
[-148.2780,93.4884,623.8501],
[-140.0730,73.6308,608.9761],
[-136.3830,57.3253,598.3170],
[-184.1720,123.8400,635.0055],
[-167.5560,82.6320,610.4641],
[-172.8390,57.5322,600.3000],
[-178.3420,41.2008,595.0936],
[-199.9620,122.8140,624.3466],
[-188.5440,82.0424,601.5391],
[-195.5090,57.6348,597.8206],
[-201.6520,43.3185,599.3086],
[-217.5930,117.7900,612.9420],
[-208.2350,83.0272,596.3340],
[-210.1310,64.2066,595.8375],
[-213.4830,51.6226,598.3170]]；
Then
By acquiring multiple hand data in advance, the average metacarpal key point T 6×3 = of the built-in parameters
[[0 0 0],
[26.9591 -24.1867 -19.2479],
[48.4238 -43.4442 -34.5729],
[65.5292 -58.7903 -46.7856],
[78.8207 -70.7149 -56.2753],
[94.1914 -16.3095 -5.0408]]；
Average phalangeal length L 5×3 =
[[30.1293 27.4869 35.7133],
[34.0802 23.2788 24.6805],
[33.4508 24.566 26.2397],
[30.3353 26.3507 25.3738]，
[21.7728 19.6307 20.2736]]；
Subsequently, R 01、t01、μ-1 is found using singular value decomposition:
R01＝
[[0.9444 -0.1518 -0.2917]，
[-0.1175 0.6728 -0.7304]，
[0.3071 0.7241 0.6176]]；
t01＝
[[466.43804747]，
[368.91997943]，
[-411.96710567]]；
μ-1＝1.1015；
Then
Replacement of the palm portion with T 6×3
Thus is provided with
Next, a subsequent parameterization process needs to be performed for each finger i, where only i=1, i.e. the parameterization process of the thumb, is shown in detail, the calculation steps of the other fingers being identical.
R12,1＝
[[0.174 0.982 0.073],
[0.984 -0.17 -0.053],
[-0.039 0.081 -0.996]]；
t12,1＝[-0 -95.726 -0]；
Obtaining the thumb in the coordinate systemCoordinates of (c): /(I)
Order theIs 0, i.e. projected ontoZoy plane of (c).
Then: θ 1,2 =0.087pi
θ1,3＝0.262π
θ1,4＝0.035π
Repeating the above procedure for the other fingers, resulting in values of all control parameters (θ 5×4, μ) and intermediate variables (R 01,t01,R12,t12,R23),
Wherein θ 5×4 =
[[0.399 0.262 -0.035 0.000]，
[0.313 0.087 0.262 0.035]，
[0.145 0.244 0.419 0.122]，
[0.074 0.908 0.524 0.052]，
[-0.191 1.134 0.244 0.070]]；
μ＝1.1015；
Then, the Kalman filtering is carried out on the theta 5×4 to obtain
θ5*4＝
[[0.441 0.304 0.001 -0.04]，
[0.33 0.006 0.216 0.015]，
[0.14 0.233 0.434 0.207]，
[0.122 0.915 0.49 0.102]]
[-0.192 1.16 0.253 0.044]]；
μ′＝1.1042
Then, inverse parameterization is performed fromThe reduction starts to S 0 and the wrist is calculated from the fingertip. Still taking finger i=1 as an example:
Finally
and
The final results are shown in FIG. 9. From the spatial distribution, compared with the original key points, the finger bending degree is closer to a true value, the skeleton length is more reasonable, and the overall effect is better. From the time variation, the key point sequence is smoother and more stable due to the Kalman filtering and the exponential smoothing to remove Gaussian noise in the acquisition process and the detection process.
On the basis of the method item embodiments, the invention correspondingly provides device item embodiments;
As shown in fig. 10, an embodiment of the present invention provides a hand key point denoising apparatus, including: the system comprises an initial key point coordinate acquisition module, a parameterization module, a denoising module and an inverse parameterization module;
The initial key point coordinate acquisition module is used for acquiring coordinates of hand key points of the current hand under a first coordinate system before denoising;
the parameterization module is used for parameterizing coordinates of key points of the denoising front hand under a first coordinate system to obtain a joint rotation angle of the current hand and a scale factor used for representing the scaling of the current hand relative to a standard hand;
the denoising module is used for carrying out Kalman filtering on the joint rotation angle to obtain a denoised joint rotation angle; performing exponential smoothing on the scale factors to obtain denoised scale factors;
the inverse parameterization module is used for carrying out inverse parameterization on the denoised joint rotation angle and the denoised scale factor to obtain coordinates of the denoised hand key points under the first coordinate system.
It should be noted that the above-described apparatus embodiments are merely illustrative, and the units described as separate units may or may not be physically separate, and units shown as units may or may not be physical units, may be located in one place, or may be distributed over a plurality of network units. Some or all of the modules may be selected according to actual needs to achieve the purpose of the solution of this embodiment. In addition, in the drawings of the embodiment of the synchronous head detection device provided by the invention, the connection relation between the modules indicates that the modules are in communication connection, and the synchronous head detection device can be specifically realized as one or more communication buses or signal lines. Those of ordinary skill in the art will understand and implement the present invention without undue burden.
It will be clearly understood by those skilled in the art that, for convenience and brevity, the specific working process of the apparatus described above may refer to the corresponding process in the foregoing method embodiment, which is not repeated herein.
On the basis of the embodiment of the method item, the invention correspondingly provides electronic equipment, which comprises a processor, a memory and a computer program stored in the memory and configured to be executed by the processor, wherein the processor realizes the hand key point denoising method.
The Processor may be a central processing unit (Central Processing Unit, CPU), other general purpose Processor, digital signal Processor (DIGITAL SIGNAL Processor, DSP), application SPECIFIC INTEGRATED Circuit (ASIC), off-the-shelf Programmable gate array (Field-Programmable GATE ARRAY, FPGA) or other Programmable logic device, discrete gate or transistor logic device, discrete hardware components, or the like. A general purpose processor may be a microprocessor or the processor may be any conventional processor or the like that is a control center of the electronic device, connecting various parts of the overall electronic device using various interfaces and lines.
The memory may be used to store the computer program, and the processor may implement various functions of the electronic device by running or executing the computer program stored in the memory, and invoking data stored in the memory. The memory may mainly include a storage program area and a storage data area, wherein the storage program area may store an operating system, an application program required for at least one function, and the like; the storage data area may store data created according to the use of the cellular phone, etc. In addition, the memory may include high-speed random access memory, and may also include non-volatile memory, such as a hard disk, memory, plug-in hard disk, smart memory card (SMART MEDIA CARD, SMC), secure Digital (SD) card, flash memory card (FLASH CARD), at least one disk storage device, flash memory device, or other volatile solid-state storage device.
On the basis of the embodiment of the method, the invention correspondingly provides a storage medium, which comprises a stored computer program, wherein the computer program controls equipment where the storage medium is located to execute the hand key point denoising method according to any one of the method.
The storage medium referred to herein is a computer-readable storage medium. The modules/units integrated with the hand key point denoising method may be stored in a computer readable storage medium if implemented in the form of software functional units and sold or used as independent products. Based on such understanding, the present invention may implement all or part of the flow of the method of the above embodiment, or may be implemented by a computer program to instruct related hardware, where the computer program may be stored in a computer readable storage medium, and when the computer program is executed by a processor, the computer program may implement the steps of each of the method embodiments described above. Wherein the computer program comprises computer program code which may be in source code form, object code form, executable file or some intermediate form etc. The computer readable medium may include: any entity or device capable of carrying the computer program code, a recording medium, a U disk, a removable hard disk, a magnetic disk, an optical disk, a computer Memory, a Read-Only Memory (ROM), a random access Memory (RAM, random Access Memory), an electrical carrier signal, a telecommunications signal, a software distribution medium, and so forth.
While the foregoing is directed to the preferred embodiments of the present invention, it will be appreciated by those skilled in the art that changes and modifications may be made without departing from the principles of the invention, such changes and modifications are also intended to be within the scope of the invention.
Claims (9)
1. The hand key point denoising method is characterized by comprising the following steps of:
acquiring coordinates of hand key points of the current hand before denoising under a first coordinate system;
Parameterizing coordinates of key points of the hand before denoising under a first coordinate system to obtain a joint rotation angle of the current hand and a scale factor for representing the scaling of the current hand relative to a standard hand;
carrying out Kalman filtering on the joint rotation angle to obtain a denoised joint rotation angle; performing exponential smoothing on the scale factors to obtain denoised scale factors;
inversely parameterizing the denoised joint rotation angle and the denoised scale factor to obtain coordinates of denoised hand key points under a first coordinate system;
Wherein the first coordinate system is a world coordinate system; the hand key points comprise: wrist joint key points and finger key points corresponding to all fingers; the finger key points of each finger include: metacarpophalangeal joint key points, proximal interphalangeal joint key points, distal interphalangeal joint key points and fingertip key points; the wrist joint key points and the metacarpophalangeal joint key points of all fingers form metacarpal bone key points;
the determining of the scale factor includes:
Based on the coordinates of the hand key points in the world coordinate system before denoising and the coordinates of the preset average metacarpal key points in the wrist coordinate system, the reciprocal of the scale factor is calculated by the following formula:
*(-());
=svd(H)；
=；
;
calculating the scale factor by the reciprocal of the scale factor;
Wherein, Is the coordinate of the metacarpal key point under the world coordinate system; /(I)Is the coordinate of the average metacarpal key point under the wrist coordinate system; svd (H) is represented as singular value decomposition of matrix H; u is left singular vector, S is diagonal matrix composed of singular values of H; v is a group of orthogonal bases which are formed by right singular vectors and describe a wrist coordinate system; /(I)Is the transposed matrix of V; -The rotation matrix corresponds to the world coordinate system when being converted into the wrist coordinate system; /(I)Coordinates of key points of the hand in a world coordinate system; /(I)Is the inverse of the scale factor.
2. The hand keypoint denoising method of claim 1, wherein the determination of the joint rotation angle comprises:
the translation vector corresponding to the world coordinate system when it is converted to the wrist coordinate system is calculated by the following formula: =-*+;
According to the reciprocal of the scale factor, the rotation moment and the translation vector corresponding to the world coordinate system when the world coordinate system is converted into the wrist coordinate system, converting the coordinate of the hand key point before denoising under the first coordinate system into the wrist coordinate system, and replacing the coordinate of the average metacarpal key point under the wrist coordinate system with the coordinate of the metacarpal key point converted corresponding to the current hand to obtain the final coordinate of the hand key point under the wrist coordinate system;
For each finger, converting the final coordinates of the finger key points under the wrist coordinate system to corresponding metacarpophalangeal joint coordinate systems to obtain the coordinates of the finger key points under the corresponding metacarpophalangeal joint coordinate systems; according to the coordinates of the key points of the finger parts under the corresponding metacarpophalangeal joint coordinate system, calculating to obtain the transverse swing angle of the metacarpophalangeal joint;
converting coordinates of the finger key points under the metacarpophalangeal joint coordinate system into corresponding finger movable plane coordinate systems to obtain coordinates of the finger key points under the finger movable plane coordinate systems; calculating pitch angles of metacarpophalangeal joints, proximal interphalangeal joints and distal interphalangeal joints according to coordinates of the key points of the finger parts under the finger movement plane coordinate system;
Obtaining the joint rotation angle of the current hand according to the transverse swing angle of the metacarpophalangeal joint, the pitch angle of the proximal interphalangeal joint and the pitch angle of the distal interphalangeal joint of each finger;
Wherein the origin of coordinates under the metacarpophalangeal joint coordinate system corresponding to each finger is a corresponding metacarpophalangeal joint key point, the Y-axis direction is the wrist joint key point to the metacarpophalangeal joint key point, the X-axis is the cross of the Y-axis and the Z-axis, and the Z-axis direction satisfies the conditions when the finger is not a little finger ）×(–) When the finger is a little finger, the Z axis is parallel to the Z axis of the ring finger,Is the palm-finger key point of the current finger,Is the metacarpophalangeal key point of the next finger adjacent to the current finger;
the finger movable plane coordinate system corresponding to each finger is obtained by rotating around the Z axis according to the transverse swing angle of the metacarpophalangeal joint by the corresponding metacarpophalangeal joint coordinate system.
3. The method for denoising the finger key points according to claim 2, wherein the calculating the lateral swing angle of the metacarpophalangeal joint according to the coordinates of the finger key points under the corresponding metacarpophalangeal joint coordinate system comprises:
the lateral swing angle of the metacarpophalangeal joint is calculated by the following formula:
；
Wherein, Is the transverse swing angle of the metacarpophalangeal joint of the ith finger; /(I)Is the X coordinate of the key point of the proximal interphalangeal joint of the ith finger under the metacarpophalangeal joint coordinate system; /(I)Is the Y coordinate of the proximal interphalangeal joint key point of the ith finger in the metacarpophalangeal joint coordinate system.
4. The hand keypoint denoising method as claimed in claim 3, wherein the calculating the pitch angle of metacarpophalangeal joints, the pitch angle of proximal interphalangeal joints and the pitch angle of distal interphalangeal joints according to the coordinates of the finger keypoints under the finger movement plane coordinate system comprises:
the pitch angle of the metacarpophalangeal joint is calculated by the following formula:
=arccos(()*()÷÷)；
the pitch angle of the proximal interphalangeal joint is calculated by the following formula:
=arccos(()*()÷÷)；
The pitch angle of the distal interphalangeal joint is calculated by the following formula:
=arccos(()*()÷÷)；
Wherein, Is a unit vector (0, 1) in the Y direction of the finger movement plane coordinate system; /(I)Pitch angle of metacarpophalangeal joints; /(I)Pitch angle for proximal interphalangeal joint; /(I)；The key point of the metacarpophalangeal joint of the ith finger is the coordinate of the zoy plane of the corresponding finger movable plane coordinate system; /(I)As the key point of the proximal interphalangeal joint of the ith finger, the coordinates of the zoy plane of the corresponding finger movement plane coordinate system; /(I)Coordinates of a zoy plane of a corresponding finger movement plane coordinate system are distal interphalangeal joint key points of the ith finger; /(I)And the coordinates of the key point of the fingertip of the ith finger on the zoy plane of the corresponding finger moving plane coordinate system.
5. The hand keypoint denoising method of claim 4, wherein the kalman filtering the joint rotation angle comprises:
Determining a corresponding filter coefficient according to the joint rotation angle;
And carrying out Kalman filtering on the joint rotation angle according to the filter coefficient.
6. The method for denoising a hand key point according to claim 5, wherein inverse parameterizing the denoised joint rotation angle and the denoised scale factor to obtain coordinates of the denoised hand key point in a first coordinate system comprises:
For each finger, calculating the coordinates of key points at the back of the filter finger under a finger movement plane coordinate system according to the filtered joint rotation angle and the preset average phalange length;
Calculating the coordinates of the key points of the back of the filter finger under the metacarpophalangeal joint coordinate system according to the coordinates of the key points of the back of the filter finger under the finger movable plane coordinate system and the corresponding rotation matrix when the metacarpophalangeal joint coordinate system is converted into the finger movable plane coordinate system;
Calculating the coordinates of the key points at the back of the filter finger under the coordinate system of the wrist joint according to the coordinates of the key points at the back of the filter finger under the coordinate system of the metacarpophalangeal joint, the corresponding rotation matrix and translation vector when the coordinate system of the wrist joint is converted into the coordinate system of the metacarpophalangeal joint;
And according to the corresponding rotation moment and translation vector when the world coordinate system is converted into the wrist coordinate system, restoring the coordinates of the filtered finger key points under the wrist joint coordinate system and the coordinates of the filtered wrist joints under the wrist joint coordinate system to the world coordinate system to obtain the coordinates of the denoised hand key points under the first coordinate system.
7. The utility model provides a hand key point denoising device which characterized in that includes: the system comprises an initial key point coordinate acquisition module, a parameterization module, a denoising module and an inverse parameterization module;
The initial key point coordinate acquisition module is used for acquiring coordinates of hand key points of the current hand under a first coordinate system before denoising;
the parameterization module is used for parameterizing coordinates of key points of the denoising front hand under a first coordinate system to obtain a joint rotation angle of the current hand and a scale factor used for representing the scaling of the current hand relative to a standard hand;
the denoising module is used for carrying out Kalman filtering on the joint rotation angle to obtain a denoised joint rotation angle; performing exponential smoothing on the scale factors to obtain denoised scale factors;
The inverse parameterization module is used for carrying out inverse parameterization on the denoised joint rotation angle and the denoised scale factor to obtain coordinates of the denoised hand key points under a first coordinate system;
Wherein the first coordinate system is a world coordinate system; the hand key points comprise: wrist joint key points and finger key points corresponding to all fingers; the finger key points of each finger include: metacarpophalangeal joint key points, proximal interphalangeal joint key points, distal interphalangeal joint key points and fingertip key points; the wrist joint key points and the metacarpophalangeal joint key points of all fingers form metacarpal bone key points;
the determining of the scale factor includes:
Based on the coordinates of the hand key points in the world coordinate system before denoising and the coordinates of the preset average metacarpal key points in the wrist coordinate system, the reciprocal of the scale factor is calculated by the following formula:
*(-());
=svd(H)；
=；
;
calculating the scale factor by the reciprocal of the scale factor;
Wherein, Is the coordinate of the metacarpal key point under the world coordinate system; /(I)Is the coordinate of the average metacarpal key point under the wrist coordinate system; svd (H) is represented as singular value decomposition of matrix H; u is left singular vector, S is diagonal matrix composed of singular values of H; v is a group of orthogonal bases which are formed by right singular vectors and describe a wrist coordinate system; /(I)Is the transposed matrix of V; -The rotation matrix corresponds to the world coordinate system when being converted into the wrist coordinate system; /(I)Coordinates of key points of the hand in a world coordinate system; /(I)Is the inverse of the scale factor.
8. An electronic device comprising a processor, a memory, and a computer program stored in the memory and configured to be executed by the processor, the processor implementing the hand keypoint denoising method according to any one of claims 1-6 when executing the computer program.
9. A storage medium comprising a stored computer program, wherein the computer program, when run, controls a device in which the storage medium is located to perform the hand keypoint denoising method as claimed in any one of claims 1-6.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202410045087.8A CN118015288B (en) | 2024-01-12 | 2024-01-12 | Hand key point denoising method and device, electronic equipment and storage medium |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202410045087.8A CN118015288B (en) | 2024-01-12 | 2024-01-12 | Hand key point denoising method and device, electronic equipment and storage medium |
Publications (2)
| Publication Number | Publication Date |
|---|---|
| CN118015288A CN118015288A (en) | 2024-05-10 |
| CN118015288B true CN118015288B (en) | 2024-06-14 |
Family
ID=90959830
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN202410045087.8A Active CN118015288B (en) | 2024-01-12 | 2024-01-12 | Hand key point denoising method and device, electronic equipment and storage medium |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN118015288B (en) |
Family Cites Families (5)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN110991319B (en) * | 2019-11-29 | 2021-10-19 | 广州市百果园信息技术有限公司 | Hand key point detection method, gesture recognition method and related device |
| AU2022247290A1 (en) * | 2021-03-30 | 2023-11-16 | Martin FISCH | Systems and methods for computer recognition of 3d gesture movements |
| WO2023000119A1 (en) * | 2021-07-17 | 2023-01-26 | 华为技术有限公司 | Gesture recognition method and apparatus, system, and vehicle |
| CN115798030A (en) * | 2021-09-09 | 2023-03-14 | 广州视源电子科技股份有限公司 | Method and device for gesture recognition based on rotation, electronic equipment and storage medium |
| CN115359098A (en) * | 2022-08-17 | 2022-11-18 | 佛山科学技术学院 | Multi-target motion capturing skeleton key point tracking method |
-
2024
- 2024-01-12 CN CN202410045087.8A patent/CN118015288B/en active Active
Non-Patent Citations (2)
| Title |
|---|
| Body Surface Context: A New Robust Feature for Action Recognition From Depth Videos;Y. Song, J. Tang, F. Liu and S. Yan;《IEEE Transactions on Circuits and Systems for Video Technology》;20140124;全文 * |
| 王婧瑶 ; 王红军. 基于Mask R-CNN与SG滤波的手势识别关键点特征提取方法.《电子测量与仪器学报》.2021,全文. * |
Also Published As
| Publication number | Publication date |
|---|---|
| CN118015288A (en) | 2024-05-10 |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| WO2021169839A1 (en) | Action restoration method and device based on skeleton key points | |
| CN100559398C (en) | Automatic deepness image registration method | |
| Mokhtarian et al. | Multi-scale free-form 3D object recognition using 3D models | |
| CN110363817B (en) | Target pose estimation method, electronic device, and medium | |
| Bhuyan et al. | Hand pose identification from monocular image for sign language recognition | |
| US20220262093A1 (en) | Object detection method and system, and non-transitory computer-readable medium | |
| CN112336342A (en) | Hand key point detection method and device and terminal equipment | |
| CN109241844A (en) | Attitude estimation method, apparatus, equipment and the storage medium of three-dimension object | |
| CN113191243B (en) | Human hand three-dimensional attitude estimation model establishment method based on camera distance and application thereof | |
| CN112101073B (en) | Face image processing method, device, equipment and computer storage medium | |
| JP2016014954A (en) | Method for detecting finger shape, program thereof, storage medium of program thereof, and system for detecting finger shape | |
| CN111709270A (en) | Three-dimensional shape recovery and attitude estimation method and device based on depth image | |
| CN113298870A (en) | Object posture tracking method and device, terminal equipment and storage medium | |
| CN111433783A (en) | Hand model generation method and device, terminal device and hand motion capture method | |
| CN111460910A (en) | Face type classification method and device, terminal equipment and storage medium | |
| CN118015288B (en) | Hand key point denoising method and device, electronic equipment and storage medium | |
| CN113674395A (en) | 3D hand lightweight real-time capturing and reconstructing system based on monocular RGB camera | |
| CN115147433A (en) | Point cloud registration method | |
| JP7385747B2 (en) | Method and system for determining the three-dimensional position, local reference frame, and grasping pattern of a grasping point on an object | |
| Heap et al. | Automated pivot location for the cartesian-polar hybrid point distribution model | |
| Laskov et al. | Curvature-based algorithms for nonrigid motion and correspondence estimation | |
| CN116503940A (en) | Multi-view multi-person three-dimensional human body posture estimation and tracking method, system and equipment | |
| CN110647925A (en) | Rigid object identification method and device based on improved LINE-MOD template matching | |
| CN109840490B (en) | Human motion representation processing method and system, electronic device and storage medium | |
| Fu et al. | Taylor Neural Network for Unseen Object Instance Segmentation in Hierarchical Grasping |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination | ||
| GR01 | Patent grant | ||
| GR01 | Patent grant |