CN113781654B - Method for extracting three-dimensional grid model skeleton of hand by using spherical expansion side writing - Google Patents
Method for extracting three-dimensional grid model skeleton of hand by using spherical expansion side writing Download PDFInfo
- Publication number
- CN113781654B CN113781654B CN202111160093.0A CN202111160093A CN113781654B CN 113781654 B CN113781654 B CN 113781654B CN 202111160093 A CN202111160093 A CN 202111160093A CN 113781654 B CN113781654 B CN 113781654B
- Authority
- CN
- China
- Prior art keywords
- hand
- side writing
- spherical
- model
- coordinates
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Active
Links
- 238000000034 method Methods 0.000 title claims abstract description 52
- 210000002478 hand joint Anatomy 0.000 claims abstract description 15
- 230000009466 transformation Effects 0.000 claims abstract description 10
- 230000000295 complement effect Effects 0.000 claims abstract description 6
- 238000000605 extraction Methods 0.000 claims description 15
- 230000008569 process Effects 0.000 claims description 14
- 210000000707 wrist Anatomy 0.000 claims description 11
- 210000004247 hand Anatomy 0.000 claims description 10
- 238000010586 diagram Methods 0.000 claims description 7
- 208000037170 Delayed Emergence from Anesthesia Diseases 0.000 claims description 3
- 238000013507 mapping Methods 0.000 claims description 3
- 230000004044 response Effects 0.000 claims description 3
- 238000005070 sampling Methods 0.000 claims description 3
- 230000002708 enhancing effect Effects 0.000 claims 1
- 230000001131 transforming effect Effects 0.000 abstract 1
- 230000036544 posture Effects 0.000 description 5
- 238000013528 artificial neural network Methods 0.000 description 3
- 230000000694 effects Effects 0.000 description 3
- XLYOFNOQVPJJNP-UHFFFAOYSA-N water Substances O XLYOFNOQVPJJNP-UHFFFAOYSA-N 0.000 description 3
- 238000013135 deep learning Methods 0.000 description 2
- 238000005516 engineering process Methods 0.000 description 2
- 230000006870 function Effects 0.000 description 2
- 238000007476 Maximum Likelihood Methods 0.000 description 1
- 230000009286 beneficial effect Effects 0.000 description 1
- 238000010276 construction Methods 0.000 description 1
- 230000003993 interaction Effects 0.000 description 1
Classifications
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
- G06T17/00—Three dimensional [3D] modelling, e.g. data description of 3D objects
- G06T17/20—Finite element generation, e.g. wire-frame surface description, tesselation
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06N—COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS
- G06N3/00—Computing arrangements based on biological models
- G06N3/02—Neural networks
- G06N3/04—Architecture, e.g. interconnection topology
- G06N3/045—Combinations of networks
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06N—COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS
- G06N3/00—Computing arrangements based on biological models
- G06N3/02—Neural networks
- G06N3/08—Learning methods
Landscapes
- Engineering & Computer Science (AREA)
- Physics & Mathematics (AREA)
- Theoretical Computer Science (AREA)
- Software Systems (AREA)
- General Physics & Mathematics (AREA)
- General Health & Medical Sciences (AREA)
- Computing Systems (AREA)
- Computational Linguistics (AREA)
- Data Mining & Analysis (AREA)
- Evolutionary Computation (AREA)
- Biomedical Technology (AREA)
- Molecular Biology (AREA)
- Biophysics (AREA)
- General Engineering & Computer Science (AREA)
- Artificial Intelligence (AREA)
- Mathematical Physics (AREA)
- Life Sciences & Earth Sciences (AREA)
- Health & Medical Sciences (AREA)
- Computer Graphics (AREA)
- Geometry (AREA)
- Image Processing (AREA)
Abstract
The invention discloses a method for extracting a three-dimensional hand grid model skeleton by using spherical expansion side writing, which comprises the steps of firstly converting a three-dimensional hand grid model into spherical expansion side writing, wherein the spherical expansion side writing is formed by transforming a Cartesian coordinate system into a spherical coordinate system, projecting a curved surface corresponding to a three-dimensional grid onto three complementary side writing images which are indexed by spherical coordinates, and sequentially named as ray side writing, warp side writing and weft side writing; then, the obtained spherical expansion side writing result is sent into a heat map regression network to output a hand joint heat map; calculating the space coordinates of the hand node ball according to the output result of the heat map regression network; and finally, converting the spherical coordinates of the joint points into Cartesian coordinates by utilizing the inverse transformation from the spherical coordinates to the Cartesian coordinate system. The method can extract the hand skeleton from the hand grid model with arbitrary grid topology, hand gesture and watertight degree.
Description
Technical Field
The invention relates to the fields of computer vision and computer graphics, in particular to a method for extracting a three-dimensional grid model skeleton of a hand by utilizing spherical expansion side writing.
Background
The three-dimensional vision technology based on the deep learning enables the acquisition of the personalized hand scanning model reconstruction of the user from the image to be more convenient. Driving these scan models is an important step in a range of AR/MR/VR technology applications, such as remote interaction, remote control. In order to accomplish hand pose estimation, hand motion redirection, and hand model driving, it is important to extract consistent skeletons from diverse models. However, the existing grid model skeleton extraction method takes a human cartoon-like character and a human scanning grid model as research objects, estimates the skeleton of a human body (generally without fingers) under a standard posture (T-phase), and has strict requirements on the water tightness of the grid model. In contrast, skeletal extraction of hands faces more challenges:
1) Hand models do not have a uniform standard pose in the industry. This means that the hand skeleton extraction method needs to have the capability of extracting consistent hand skeletons from hand grid models with various postures;
2) Chirality is unique to the hand model. The hand skeleton extraction method of the hand grid model needs to have mirror symmetry transformation of robustness on the left hand and the right hand.
3) Many hand models are not watertight. Since the hands are only part of the human body, many existing hand scan data and cartoon models do not topologically require occlusion.
Disclosure of Invention
In order to solve the problems, the invention discloses a method for extracting a hand three-dimensional grid model skeleton by utilizing spherical expansion side writing, which can extract the hand skeleton from a hand grid model with arbitrary grid topology, hand gesture and watertight degree.
The above purpose is achieved by the following technical scheme:
the method for extracting the three-dimensional grid model skeleton of the hand by utilizing the spherical expansion side writing comprises the following steps:
step 1, converting a hand three-dimensional grid model into a spherical expansion side writing, wherein the size of a corresponding tensor of an input image is (3, H) a ,W a ) Wherein H is a To input the height of the image, W a For inputting the width of an image, the expansion side writing of the spherical surface is to project a curved surface corresponding to a three-dimensional grid onto three complementary side writing images which are indexed by spherical coordinates by utilizing the transformation from a Cartesian coordinate system to a spherical coordinate system, and the three complementary side writing images are sequentially named as ray side writing, meridian side writing and latitude side writing;
step 2, sending the spherical expansion side writing result obtained in the step 1 into a heat map regression network to output hand joint heat map sizes of (3 x J, H) b ,W b ) Wherein J is the number of hand joints, H b To output the height of the hand joint heat map, W b The width of the thermal map of the hand joint is output;
step 3, calculating the space coordinates of the hand node ball according to the output result of the heat map regression network;
and 4, converting the spherical coordinates of the joint points into Cartesian coordinates by utilizing inverse transformation from the spherical coordinates to the Cartesian coordinate system.
Further, the specific method for converting the hand three-dimensional grid model into the spherical expansion side writing in the step 1 is as follows:
step 1.1. Normalizing the given hand mesh model: specifically, the mean value of the vertexes of the grid model is positionedSetting the distance from the farthest vertex to the mean point as a scaling factor to each vertex of the model, wherein the whole model is limited in the Cartesian space contained in the unit sphere; grid model surfaceAny point (x, y, z) above can be converted into a spherical space coordinate representation by the following formula +.>
Since the range of the value range of the "atan2" function is (-pi, pi), for facilitating the subsequent image coordinate transformation, for negative valuesIncreasing 2π, wherein ρ represents the distance from the surface point to the origin, θ represents the angle between the surface point and the origin line and the Z axis; />Representing the included angle between the projection of the surface point and the origin on the XoY plane and the X axis, it is obvious that for any normalized surface point, there is ρ E [0,1 ]],θ∈[0,π],/>
Step 1.21 generating a ray side writing image P according to the normalized hand model r : the generation mode of ray side writing is equivalent to that a laser radar is placed at the origin of the normalized coordinates, the whole model is subjected to ball scanning, and each laser radar is used for scanning in the scanning processDefined ray +.>The furthest point intersecting the surface is recorded in the result map, the result image size (W a ,H a ) The mathematical expression of the generation process is as follows:
step 1.22 generating a warp side writing image P according to the normalized hand model s : the warp side writing is equivalently generated by using a series of balls with radius from 0 to 1 at the normalized origin of coordinates, and recording each of the ballsDefined warp threadThe point where the (semicircular arc) intersects the curved surface is concentrated at the nearest point to the XoY plane, and the resulting image size (W a ,H a ) The mathematical expression of the generation process is as follows:
step 1.23 generating a weft-side writing image P according to the normalized hand model c : the weft-side writing was generated in the equivalent of using a series of balls with radii from 0 to 1 at the normalized origin of coordinates, recording each of the weft circles ∈ C The closest point of the point set to the negative half plane of XoZ, which intersects the curved surface, is set to the resulting image size (W a ,H a ) The mathematical expression of the generation process is as follows:
step 1.3, the three side writing gray maps generated in the steps 1.21-1.23 are connected in series and are respectively stored in three channels of a color image, and the obtained result is spherical expansion side writing.
Further, the heat map regression network is cascaded by a residual error module and comprises an encoder and a decoder, wherein each convolution layer of the encoder adopts a mode of expansion convolution and annular complementation and is used for improving distortion caused by sphere-to-plane mapping on sphere expansion side writing; the encoder and decoder are designed as symmetrical structures, and residual blocks with the same size of the characteristic diagram are connected by a U-Net-like structure.
Further, in the hand joint heat map output in step 2, the joint heat maps of the first J channels correspond to joint points returned by ray side writingProbability distribution map; joint heat map of middle J channels corresponds to joint points regressed by meridian side writing ++>Probability distribution map; the joint heat map of the last J channels corresponds to the joint point (ρ, θ) probability distribution map regressed by weft-side writing.
Further, the specific method of step 3 is that the output size of the regression network for the heat map is (3×j, h) b ,W b ) In the output result of (1), the sequence number is i E [1, J]The joint point coordinates of (2) are related to the channel serial number I, (i+J) and (i+2J), and under the redundant coding standard, the final coordinate values are obtained by adopting a 'winner general eating' paradigm: when determining ρ, taking out the heat maps of the (i+J) and (i+2J), comparing the relative magnitudes of the maximum responsivity between the two, and taking the ρ coordinate corresponding to the heat with high response probability as the amplitude ρ of the articulation point i i And the like, finally obtaining the spherical space coordinates of J joints of the hand.
Further, the data set of the three-dimensional hand grid model adopts a data enhancement method, a linear mixed skin model of multiple hands is used, gesture parameters are randomly transformed to obtain a hand model with various gestures and corresponding joint positions, and in the random sampling process, the gesture parameters of each dimension obey normal distribution.
Further, the method of data enhancement further includes adding a variable wrist length offset to a linear hybrid skin (LBS) model of the various hands.
Further, the data enhancement method further includes adding variable wrist length offsets to the linear hybrid skin model of the plurality of hands, and adding random offsets to each vertex.
Further, the data enhancement method further comprises the step of randomly eliminating grid vertices and face sheets on the grid model of the hand.
The beneficial effects of the invention are as follows:
because the hand grid models with different fine degrees of topology are converted into unified image representation (side-writing diagram), the skeleton extraction method based on deep learning can use richer and more diverse data to intensively learn skeleton extraction, and the universality of the hand curved surface representation method is improved; the complementary side writing construction scheme and the redundant coding joint heat map scheme are adopted, so that the information source diversity and reliability of the neural network in the reasoning process are increased; because the methods of expansion convolution, annular completion, residual error module cascade and the like are used in the convolution neural network framework, the characteristic distribution non-uniformity of the side writing representation method is overcome; the adoption of the diversified hand grid data enhancement method effectively avoids the overfitting of the skeleton extraction grids, and also makes the whole method more robust to noise in a model and the differences of hand shapes and wrist lengths of different users.
Drawings
FIG. 1 is a flow chart of a method for converting a triangular mesh hand model into a spherical expansion side writing according to an embodiment of the invention;
FIG. 2 is a flow chart of estimating a hand skeleton from a triangular mesh hand model according to a second embodiment of the present invention;
FIG. 3 is a network structure diagram of hand joint point heat map regression in the present invention;
FIG. 4 is a schematic representation of training data enhancement in the present invention; wherein (a) is a strictly watertight hand grid model example; (b) is an example of a hand mesh model that grows the wrist; (c) a hand grid example of a mirror transformation; (d) An example of a hand mesh model for removing the watertight surface of the wrist; (e) An example of a hand mesh model with incomplete watertightness; (f) Adding Gaussian noise to the vertex to form a hand grid model example;
fig. 5 is an effect diagram of extracting a skeleton from a hand model downloaded from the internet by applying the present invention. For ease of viewing, each embodiment is viewed from two perspectives. Wherein (a) - (d) correspond to CAD hand model skeleton extraction results, (e) - (h) correspond to cartoon hand model skeleton extraction results, and (i) - (l) correspond to kotah data set hand model skeleton extraction results;
FIG. 6 is an effect diagram of the extraction of skeleton from a scanned hand model using the present invention. For ease of viewing, each embodiment is viewed from two perspectives. Wherein (a) - (d) correspond to the hand-held scanner scan model skeleton extraction results, (e) - (h) correspond to the noisy multi-view reconstruction model skeleton extraction results, and (i) - (l) correspond to the wrist multi-view reconstruction model skeleton extraction results.
Detailed Description
The following will describe embodiments of the present invention in detail with reference to the drawings and examples, thereby solving the technical problems by applying technical means to the present invention, and realizing the technical effects can be fully understood and implemented accordingly. It should be noted that, as long as no conflict is formed, each embodiment of the present invention and each feature of each embodiment may be combined with each other, and the formed technical solutions are all within the protection scope of the present invention. .
Fig. 1 is a flowchart of converting a triangular mesh hand model into a sphere expansion side writing according to an embodiment of the present invention, and each step is described in detail below with reference to fig. 1.
Step S110, input is a hand triangular mesh model without limitation in chirality, appearance, posture, mesh patch topology, water tightness and wrist length. This model is normalized into unit sphere space by scaling the vertex mean to the origin and the length furthest from the origin to unit 1. And the intersection relationship is expressed using the spherical coordinates in the subsequent step.
Specifically, the mean position of the vertices of the mesh model is taken as the origin of Cartesian space and the distance of the furthest vertex to the mean point is used as a scaling factor to be applied to each vertex of the model. Through this step, the whole model is defined in the Cartesian space contained by the unit sphere; grid model surfaceAny point (x, y, z) above can be converted into a spherical space coordinate representation by the following formula +.>
Since the range of the value range of the "atan2" function is (-pi, pi), for facilitating the subsequent image coordinate transformation, for negative valuesIncreasing 2pi, wherein ρ represents the distance from the surface point to the origin, and θ represents the included angle between the connecting line of the surface point and the origin and the Z axis; />Representing the angle between the projection of the surface point and origin line on XoY plane and the X axis. Obviously, for any normalized surface point, there is ρ ε [0,1 ]],θ∈[0,π],/>
Step S120, generating three-channel spherical expansion side writing based on the normalized grid model. The first is called ray side writing, and the specific generation mode is to calculate the intersection point of the ray and the grid surface; the second mode is called warp side writing, and the specific generation mode is that firstly, the intersection line of the half plane and the grid is calculated, and then, the intersection point of the intersection line and the corresponding warp is calculated; and thirdly, called weft side writing, wherein the specific generation mode is to calculate the intersection line of the spherical surface and the grid, and then calculate the intersection point of the intersection line and the corresponding weft. Since the intersection of the half-plane, sphere and mesh corresponding surface is in fact a non-parametric curve, here a continuous straight line segment is used to approximate the two intersection curves mentioned above. The specific method comprises the following steps:
step S1201 generates a ray side writing image P according to the normalized hand model r . The generation mode of ray side writing can be understood as placing a laser radar at the normalized origin of coordinates, and performing ball scanning on the whole model. During the scanning process, each strip is composed ofDefined ray +.>The furthest point of intersection with the surface is recorded in the result map. Setting the resulting image size (W a ,H a ) The mathematical expression of the generation process can be written as:
step S1202 generates a warp side writing image P according to the normalized hand model s . The generation of the warp side writing can be understood as using a series of spheres with radii from 0 to 1 at the normalized origin of coordinates, each sphere being recordedDefined warp thread->The point at which the (semi-circular arc) intersects the curved surface is centered at the closest point to the plane XoY. Setting the resulting image size (W a ,H a ) The mathematical expression of the generation process can be written as:
step S1203 generates a weft-side writing image P according to the normalized hand model c . The weft-side writing can be understood as being generated by using a series of balls with radii from 0 to 1 at the normalized origin of coordinates, recording the weft circles ∈ C The point of intersection with the curved surface is centered at the closest point to the XoZ negative half-plane. Setting the resulting image size (W a ,H a ) The mathematical expression of the generation process can be written as:
and step S130, storing the three side writes generated by the same model into three channels of the color map respectively as spherical expansion side write output.
Fig. 2 is a flow chart for estimating a hand skeleton from a triangular mesh hand model. The steps are described in detail below with reference to fig. 2.
Step 210, corresponding to the given hand mesh model, converting the hand mesh model into a corresponding image representation of the sphere expansion side writing in the manner of the previous embodiment, wherein the corresponding tensor of the image has a size of (3, H a ,W a ) Wherein H is a To input the height of the image, W a Is the width of the input image.
And 220, writing the expanded side of the spherical surface into a heat map regression network to obtain a hand joint point heat map. The number of channels of the heat map is three times of the number J of joints, namely, each side writing channel corresponds to the adjacent J joints; the heat map regression network is formed by cascade connection of residual modules and comprises two parts of an encoder and a decoder. Each convolution layer of the encoder adopts a mode of expansion convolution (dilated convolution) and annular padding (circular padding) to improve distortion caused by sphere-to-plane mapping on sphere expansion side writing; the encoder and decoder are designed as symmetrical structures, and residual blocks with the same size of the characteristic diagram are connected by a U-Net-like structure. The final output hand joint heat map size of the network is (3 x J, H) b ,W b ) Wherein J is the number of hand joints, H b To output handsHeight of thermal map of part joint, W b To output the width of the hand joint heat map. Wherein the joint heat map of the first J channels corresponds to joint points regressed by ray side writingProbability distribution map; joint heat map of middle J channels corresponds to joint points regressed by meridian side writing ++>Probability distribution map; the joint heat map of the last J channels corresponds to the joint point (ρ, θ) probability distribution map regressed by weft-side writing.
Step 230, analyzing three heat maps corresponding to the same joint point, and obtaining the spherical coordinates of each joint point relative to the hand model by sequentially using the principle of maximum likelihood estimation and 'winning person general eating'; specifically, since the above size is (3×j, h b ,W b ) In the output result of (1), the sequence number is i E [1, J]The joint coordinates of (2) are related to three heat maps with channel number I (i+J) and (i+2J). Under such redundant coding specifications, a final coordinate value is obtained using a paradigm of "winner general eat": when determining ρ, taking out the heat maps of the (i+J) and (i+2J), comparing the relative magnitudes of the maximum responsivity between the two, and taking the ρ coordinate corresponding to the heat with high response probability as the amplitude ρ of the articulation point i i . According to a similar method, the spherical space coordinates of J joints of the hand are finally obtained.
Step 240, converting the spherical space coordinates of the joint point into Cartesian space coordinates of the joint point, which is used as the output of the present embodiment.
In order to train and obtain the neural network meeting the requirements, the invention further provides a series of data enhancement methods based on the existing hand data set:
1) A linear mixed skin (LBS) model of various hands is used, and the hand model with various postures and corresponding joint points are obtained through randomly changing posture parameters. In the random sampling process, the attitude parameters of each dimension obey normal distribution;
2) Adding variable wrist length offsets to the model to accommodate the diversity of wrist lengths in the scan model;
3) Adding random offset on each vertex of the grid model obtained on the model so as to adapt to noise in a scanning model, resolution difference of scanning hardware, diversity of hand palmprints and the like;
4) And (5) randomly removing grid vertexes and face sheets from the grid model of the hand to adapt to the water tightness difference of the grids.
It will be appreciated by those skilled in the art that the modules or steps of the invention described above may be implemented in a general purpose computing device, they may be centralized on a single computing device, or distributed over a network of computing devices, or they may alternatively be implemented in program code executable by computing devices, such that they may be stored in a memory device and executed by computing devices, or they may be separately fabricated as individual integrated circuit modules, or multiple modules or steps within them may be fabricated as a single integrated circuit module. Thus, the present invention is not limited to any specific combination of hardware and software.
Claims (8)
1. A method for extracting a three-dimensional grid model skeleton of a hand by utilizing spherical expansion side writing is characterized by comprising the following steps:
step 1, converting a hand three-dimensional grid model into a spherical expansion side writing, wherein the size of a corresponding tensor of an input image is (3, H) a ,W a ) Wherein H is a To input the height of the image, W a For inputting the width of an image, the expansion side writing of the spherical surface is to project a curved surface corresponding to a three-dimensional grid onto three complementary side writing images which are indexed by spherical coordinates by utilizing the transformation from a Cartesian coordinate system to a spherical coordinate system, and the three complementary side writing images are sequentially named as ray side writing, meridian side writing and latitude side writing;
step 2, sending the spherical expansion side writing result obtained in the step 1 into a heat map regression network to output hand joint heat map sizes of (3 x J, H) b ,W b ) Wherein J is the number of hand joints, H b To output the heat map of the hand jointHeight, W of b The width of the thermal map of the hand joint is output;
step 3, calculating the space coordinates of the hand node ball according to the output result of the heat map regression network;
step 4, converting spherical coordinates of the joint points into Cartesian coordinates by utilizing inverse transformation from the spherical coordinates to the Cartesian coordinate system;
the specific method for converting the hand three-dimensional grid model into the spherical expansion side writing in the step 1 is as follows:
step 1.1. Normalizing the given hand mesh model: specifically, taking the average value position of the vertexes of the grid model as the origin of the Cartesian space, and applying the distance from the farthest vertexes to the average value points to each vertex of the model as a scaling factor, wherein the whole model is limited in the Cartesian space contained in the unit sphere through the step; grid model surfaceAny point (x, y, z) above can be converted into a spherical space coordinate representation by the following formula +.>
Since the range of the value range of the "atan2" function is (-pi, pi), for facilitating the subsequent image coordinate transformation, for negative valuesIncreasing 2π, wherein ρ represents the distance from the surface point to the origin, θ represents the angle between the surface point and the origin line and the Z axis; />Representing the included angle between the projection of the surface point and the origin on the XoY plane and the X axis, obviously, for any surface point after normalizationHas ρ ε [0,1 ]],θ∈[0,π],/>
Step 1.21 generating a ray side writing image P according to the normalized hand model r : the generation mode of ray side writing is equivalent to that a laser radar is placed at the origin of the normalized coordinates, the whole model is subjected to ball scanning, and each laser radar is used for scanning in the scanning processDefined ray +.>The furthest point intersecting the surface is recorded in the result map, the result image size (W a ,H a ) The mathematical expression of the generation process is as follows:
step 1.22 generating a warp side writing image P according to the normalized hand model s : the warp side writing is equivalently generated by using a series of balls with radius from 0 to 1 at the normalized origin of coordinates, and recording each of the ballsDefined warp threadThe point where the (semicircular arc) intersects the curved surface is concentrated at the nearest point to the XoY plane, and the resulting image size (W a ,H a ) The mathematical expression of the generation process is as follows:
step 1.23 generating a weft-side writing image P according to the normalized hand model c : the weft-side writing was generated in the equivalent of using a series of balls with radii from 0 to 1 at the normalized origin of coordinates, recording each of the weft circles ∈ C The closest point of the point set to the negative half plane of XoZ, which intersects the curved surface, is set to the resulting image size (W a ,H a ) The mathematical expression of the generation process is as follows:
step 1.3, the three side writing gray maps generated in the steps 1.21-1.23 are connected in series and are respectively stored in three channels of a color image, and the obtained result is spherical expansion side writing.
2. The method for extracting the three-dimensional grid model skeleton of the hand by utilizing the spherical expansion side writing according to claim 1, wherein the heat map regression network is formed by cascading residual modules and comprises two parts of an encoder and a decoder, wherein each convolution layer of the encoder adopts a mode of expansion convolution and annular complementation and is used for improving distortion caused by spherical to plane mapping on the spherical expansion side writing; the encoder and decoder are designed as symmetrical structures, and residual blocks with the same size of the characteristic diagram are connected by a U-Net-like structure.
3. The method for extracting three-dimensional grid model skeleton of hand by spherical expansion side writing according to claim 1, wherein in the hand joint heat map outputted in step 2, joint heat maps of the first J channels correspond to joint points regressed by ray side writingProbability distribution map; joint heat map of middle J channels corresponds to joint points regressed by meridian side writingProbability distribution map; the joint heat map of the last J channels corresponds to the joint point (ρ, θ) probability distribution map regressed by weft-side writing.
4. The method for extracting three-dimensional grid model skeleton from hand by spherical expansion side writing according to claim 1, wherein the specific method in step 3 is that the size of the regression network output of the heat map is (3×j, h b ,W b ) In the output result of (1), the sequence number is i E [1, J]The joint point coordinates of (2) are related to the channel serial number I, (i+J) and (i+2J), and under the redundant coding standard, the final coordinate values are obtained by adopting a 'winner general eating' paradigm: when determining ρ, taking out the heat maps of the (i+J) and (i+2J), comparing the relative magnitudes of the maximum responsivity between the two, and taking the ρ coordinate corresponding to the heat with high response probability as the amplitude ρ of the articulation point i i And the like, finally obtaining the spherical space coordinates of J joints of the hand.
5. The method for extracting the three-dimensional grid model skeleton of the hand by utilizing the spherical expansion side writing according to claim 1, wherein a data enhancement method is adopted in a data set of the three-dimensional grid model of the hand, a linear mixed skin model of multiple hands is used, gesture parameters are randomly transformed to obtain a hand model with various gestures and corresponding joint positions, and in the random sampling process, the gesture parameters of each dimension obey normal distribution.
6. The method of claim 5, wherein the method of data enhancement further comprises adding a variable wrist length offset to a linear hybrid skin model of multiple hands.
7. The method for three-dimensional mesh model skeleton extraction of hands using sphere expansion sidewriting according to claim 6, wherein the method for data enhancement further comprises adding variable wrist length offsets to the linear hybrid skin model of multiple hands, and adding random offsets to each vertex.
8. The method for extracting three-dimensional mesh model skeleton of hand by using spherical expansion sidewriting according to claim 7, wherein the method for enhancing data further comprises randomly eliminating mesh vertices and face sheets on the mesh model of hand.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202111160093.0A CN113781654B (en) | 2021-09-30 | 2021-09-30 | Method for extracting three-dimensional grid model skeleton of hand by using spherical expansion side writing |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202111160093.0A CN113781654B (en) | 2021-09-30 | 2021-09-30 | Method for extracting three-dimensional grid model skeleton of hand by using spherical expansion side writing |
Publications (2)
| Publication Number | Publication Date |
|---|---|
| CN113781654A CN113781654A (en) | 2021-12-10 |
| CN113781654B true CN113781654B (en) | 2023-12-19 |
Family
ID=78854705
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN202111160093.0A Active CN113781654B (en) | 2021-09-30 | 2021-09-30 | Method for extracting three-dimensional grid model skeleton of hand by using spherical expansion side writing |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN113781654B (en) |
Citations (2)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN113095262A (en) * | 2021-04-21 | 2021-07-09 | 大连理工大学 | Three-dimensional voxel gesture attitude estimation method based on multitask information complementation |
| CN113362452A (en) * | 2021-06-07 | 2021-09-07 | 中南大学 | Hand gesture three-dimensional reconstruction method and device and storage medium |
-
2021
- 2021-09-30 CN CN202111160093.0A patent/CN113781654B/en active Active
Patent Citations (2)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN113095262A (en) * | 2021-04-21 | 2021-07-09 | 大连理工大学 | Three-dimensional voxel gesture attitude estimation method based on multitask information complementation |
| CN113362452A (en) * | 2021-06-07 | 2021-09-07 | 中南大学 | Hand gesture three-dimensional reconstruction method and device and storage medium |
Also Published As
| Publication number | Publication date |
|---|---|
| CN113781654A (en) | 2021-12-10 |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| US11514642B2 (en) | Method and apparatus for generating two-dimensional image data describing a three-dimensional image | |
| CN112396703A (en) | Single-image three-dimensional point cloud model reconstruction method | |
| CN110070595B (en) | Single image 3D object reconstruction method based on deep learning | |
| Zhou et al. | Extending superquadrics with exponent functions: modeling and reconstruction | |
| CN111079565B (en) | Construction method and identification method of view two-dimensional attitude template and positioning grabbing system | |
| JPH07509081A (en) | Method and device for computer graphic processing using memory | |
| CN108932536A (en) | Human face posture method for reconstructing based on deep neural network | |
| CN113421328B (en) | Three-dimensional human body virtual reconstruction method and device | |
| CN104574432A (en) | Three-dimensional face reconstruction method and three-dimensional face reconstruction system for automatic multi-view-angle face auto-shooting image | |
| CN111382618B (en) | Illumination detection method, device, equipment and storage medium for face image | |
| CN114842136A (en) | Single-image three-dimensional face reconstruction method based on differentiable renderer | |
| Todd et al. | The many facets of shape | |
| CN110223382A (en) | Single-frame images free view-point method for reconstructing three-dimensional model based on deep learning | |
| DE102022113244A1 (en) | Joint shape and appearance optimization through topology scanning | |
| CN112785712A (en) | Three-dimensional model generation method and device and electronic equipment | |
| CN111292415A (en) | Single-view three-dimensional face reconstruction method based on spherical coordinate position map | |
| CN116740290B (en) | Three-dimensional interaction double-hand reconstruction method and system based on deformable attention | |
| CN113781654B (en) | Method for extracting three-dimensional grid model skeleton of hand by using spherical expansion side writing | |
| CN116385667B (en) | Reconstruction method of three-dimensional model, training method and device of texture reconstruction model | |
| CN116863044A (en) | Face model generation method and device, electronic equipment and readable storage medium | |
| CN116612513A (en) | Head posture estimation method and system | |
| CN115731275A (en) | Non-rigid three-dimensional point cloud registration method and system based on attention mechanism | |
| KR102461111B1 (en) | Texture mesh reconstruction system based on single image and method thereof | |
| CN113989444A (en) | Method for carrying out three-dimensional reconstruction on human face based on side face photo | |
| CN110349268B (en) | Three-dimensional human body posture, expression and gesture reconstruction method |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination | ||
| GR01 | Patent grant | ||
| GR01 | Patent grant |