CN115994993A - Stylized face three-dimensional shape modeling method, system, equipment and storage medium - Google Patents
Stylized face three-dimensional shape modeling method, system, equipment and storage medium Download PDFInfo
- Publication number
- CN115994993A CN115994993A CN202310111635.8A CN202310111635A CN115994993A CN 115994993 A CN115994993 A CN 115994993A CN 202310111635 A CN202310111635 A CN 202310111635A CN 115994993 A CN115994993 A CN 115994993A
- Authority
- CN
- China
- Prior art keywords
- face
- stylized
- face image
- mesh
- triangle mesh
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Pending
Links
Images
Classifications
-
- Y—GENERAL TAGGING OF NEW TECHNOLOGICAL DEVELOPMENTS; GENERAL TAGGING OF CROSS-SECTIONAL TECHNOLOGIES SPANNING OVER SEVERAL SECTIONS OF THE IPC; TECHNICAL SUBJECTS COVERED BY FORMER USPC CROSS-REFERENCE ART COLLECTIONS [XRACs] AND DIGESTS
- Y02—TECHNOLOGIES OR APPLICATIONS FOR MITIGATION OR ADAPTATION AGAINST CLIMATE CHANGE
- Y02T—CLIMATE CHANGE MITIGATION TECHNOLOGIES RELATED TO TRANSPORTATION
- Y02T10/00—Road transport of goods or passengers
- Y02T10/10—Internal combustion engine [ICE] based vehicles
- Y02T10/40—Engine management systems
Landscapes
- Image Analysis (AREA)
Abstract
The invention discloses a method, a system, equipment and a storage medium for modeling a three-dimensional shape of a stylized face, which belong to the technical field of vision and comprise the steps of training a 3DMM fitting model by using a real face image; inputting the stylized face image into a trained 3DMM fitting model to obtain an initial triangle mesh matched with the face shape; and carrying out Laplacian deformation on the triangular meshes by taking the key points of the faces in the stylized face images as targets to obtain final triangular meshes matched with the shape of the stylized faces. The invention carries out Laplace deformation on the 3DMM of the real face, and can automatically and efficiently reconstruct the three-dimensional shape of the face from the stylized face image.
Description
Technical Field
The invention belongs to the technical field of vision, and particularly relates to a method, a system, equipment and a storage medium for modeling a three-dimensional shape of a stylized face.
Background
The three-dimensional model of the face is the basis for making a lot of digital contents such as films, animations, etc., and modeling of the three-dimensional shape of the face is one of the key steps. In addition, in many application scenes in real life, a stylized three-dimensional shape model of a face is required, for example, when creating a figure of a Disney animation character with a cartoon big eye style, a three-dimensional shape model of a face with an exaggerated eye size is required to be created. However, modeling the three-dimensional shape of such stylized faces is currently largely dependent on the manual fabrication of professional modelers, and lacks an automatic and efficient method.
Disclosure of Invention
The invention provides a method, a system, equipment and a storage medium for modeling a three-dimensional shape of a stylized face, wherein the method comprises the following steps:
training the 3DMM fitting model by using the real face image;
inputting the stylized face image into the trained 3DMM fitting model to obtain an initial triangle mesh matched with the face shape;
and carrying out Laplacian deformation on the initial triangular grid by taking the key points of the face in the stylized face image as targets to obtain a final triangular grid which is accurately matched with the face shape.
The step of carrying out Laplacian deformation on the initial triangle mesh by taking the key points of the face in the stylized face image as targets, and the step of obtaining the final triangle mesh which is accurately matched with the face shape comprises the following steps:
the objective function formula of the final triangle mesh is as follows:
wherein:the deviation between Laplacian coordinates of the vertexes of the triangle meshes before and after deformation;is the stylized face image of the key points on the deformed triangle meshDeviation between the projection position and the corresponding key point position detected on the stylized face image; alpha is a parameter.
Wherein: l represents the number of key points,two-dimensional image coordinates representing key points detected on a stylized face image, +.>Representing three-dimensional space coordinates of the associated triangular mesh keypoints; p is a three-dimensional space coordinate set of the deformed triangle mesh vertexes; pi (II) κ Is a perspective projection operation based on the camera reference matrix k.
wherein:is the vertex v of the deformed triangle mesh i Laplace coordinates of>Is the vertex v of the triangle mesh before deformation i Is a laplace coordinate of (c); e is the edge set of the triangle mesh.
V={v 1 ,v 2 ,…,v n };
E={e 1 ,e 2 ,…,e k },e i ∈V×V;
F={f 1 ,f 2 ,…,f m },t i ∈V×V×V;
wherein: v i ∈V、e i E and f i E F is the vertex, the edge and the triangle in the triangle meshIndex in, n, k and m are the number of vertices, edges and triangles, respectively, in the triangle mesh, +.>Is triangle mesh vertex v i Is a three-dimensional space coordinate of (c).
Preferably, the stylized face image is input into the trained 3DMM fitting model to obtain an initial triangle mesh, and a camera internal reference matrix and an external reference matrix are also obtained.
Preferably, the initial triangle mesh is optimized for laplace deformation, and the optimizing step includes:
detecting face key point coordinates in the stylized face image;
optimizing the vertex coordinate set of the initial triangle mesh, wherein an optimization formula is as follows:
and iterating the optimization step until the iteration times reach a set upper limit.
The embodiment of the invention provides a system for modeling a three-dimensional shape of a stylized face, which comprises the following components:
the training module is used for training the 3DMM fitting model by using the real face image;
the output module is used for inputting the stylized face image into the trained 3DMM fitting model to obtain an initial triangle mesh matched with the face shape;
and the deformation module is used for carrying out Laplacian deformation on the initial triangular mesh by taking the face key points in the stylized face image as targets to obtain a final triangular mesh which is accurately matched with the face shape.
The device provided by the embodiment of the invention comprises at least one processing unit and at least one storage unit, wherein the storage unit stores a program, and when the program is executed by the processing unit, the processing unit is caused to execute the method.
A computer readable storage medium storing a computer program executable by a device, which when run on the device causes the device to perform the method described above.
Compared with the prior art, the invention has the beneficial effects that:
the invention carries out Laplace deformation on the 3DMM of the real face, and can automatically and efficiently reconstruct the three-dimensional shape of the face from the stylized face image.
Drawings
Fig. 1 is a schematic flow chart of a method for modeling a three-dimensional shape of a stylized face according to an embodiment of the present invention.
Detailed Description
For the purpose of making the objects, technical solutions and advantages of the embodiments of the present invention more apparent, the technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings in the embodiments of the present invention, and it is apparent that the described embodiments are some embodiments of the present invention, but not all embodiments of the present invention. All other embodiments, which can be made by those skilled in the art based on the embodiments of the invention without making any inventive effort, are intended to be within the scope of the invention.
In the description of the present invention, it should be noted that the directions or positional relationships indicated by the terms "center", "upper", "lower", "left", "right", "vertical", "horizontal", "inner", "outer", etc. are based on the directions or positional relationships shown in the drawings, are merely for convenience of describing the present invention and simplifying the description, and do not indicate or imply that the devices or elements referred to must have a specific orientation, be configured and operated in a specific orientation, and thus should not be construed as limiting the present invention. Furthermore, the terms "first," "second," and "third" are used for descriptive purposes only and are not to be construed as indicating or implying relative importance.
In the description of the present invention, it should also be noted that, unless explicitly specified and limited otherwise, the terms "mounted," "connected," and "connected" are to be construed broadly, and may be either fixedly connected, detachably connected, or integrally connected, for example; can be mechanically or electrically connected; can be directly connected or indirectly connected through an intermediate medium, and can be communication between two elements. The specific meaning of the above terms in the present invention will be understood in specific cases by those of ordinary skill in the art.
Referring to fig. 1, a method, a system, a device and a storage medium for modeling a three-dimensional shape of a stylized face, the method includes:
training the 3DMM fitting model by using the real face image;
specifically, the 3DMM fitting model is trained as prior art, and reference may be made to the paper "Accurate 3DFace Reconstruction with Weakly-Supervised Learning: from Single Image to Image Set". After training, a new real face image is input, so that triangular grids matched with the shape of the face in the image can be obtainedCamera intrinsic matrix->And the external reference matrix->(/>Representing a rotation matrix +.>Representing a translation vector).
Inputting the stylized face image into a trained 3DMM fitting model to obtain an initial triangle mesh matched with the face shape;
specifically, the stylized face image is input into a trained 3DMM fitting model to be predicted to obtain an initial triangle meshCamera reference matrix kappa 0 And an extrinsic matrix [ R ] 0 t 0 ](the superscript of a variable in a formula indicates the corresponding state, where "0" is the initial state). Since the 3DMM fitting model is trained on real face image data, its prediction on stylized face images is not accurate. However, through experimental verification, the matching of the triangular meshes predicted by the 3DMM fitting model and the faces in the stylized face images reaches a good degree on the overall face shape. Therefore, the 3DMM fitting model prediction result is taken as an initial stylized face three-dimensional shape model, and further optimization is developed on the basis.
And carrying out Laplacian deformation on the initial triangular grid by taking the key points of the face in the stylized face image as the target to obtain a final triangular grid which is accurately matched with the face shape.
Specifically, an initial triangular meshThe face image has better matching degree with the face in the stylized face image on the whole face shape, but the faces still have a certain gap in the aspects of the shape of the five sense organs and the like. Therefore, a group of sparse face key points in the fitting stylized face image is taken as a target to enable the triangle mesh to be subjected to Laplacian deformation, so that the deformed triangle mesh is more matched with the face shape in the stylized face image while the local detail characteristics are reserved. This is a typical optimization problem, the variables of which are the three-dimensional set of coordinates P of the vertices of the triangular mesh, the objective function comprising +.>And->Two energy terms.
The deviation between the projection position of the key points on the triangle mesh on the stylized face image and the corresponding key point positions detected on the stylized face image is measured, and the formula is as follows:
wherein, l represents the number of key points,two-dimensional image coordinates representing key points detected on a stylized face image, +.>Three-dimensional space coordinates representing associated triangular mesh keypoints, pi κ A perspective projection operation based on the camera reference matrix k is represented. />Is a nonlinear function of the triangle mesh vertex coordinate set P and is relatively difficult to optimize. For this purpose, will +.>Further approximately expressed as a linear function of P>
The deviation between Laplacian coordinates of the triangle mesh vertexes before and after deformation is calculated to measure the retention degree of the triangle mesh local detail characteristics after deformation, and the formula is as follows: />
Wherein, the liquid crystal display device comprises a liquid crystal display device,is the vertex v of the deformed triangle mesh i Laplace coordinates of>Is the vertex v of the triangle mesh before deformation i Is a laplace coordinate of (c).
The objective function of the final triangle mesh is:
wherein, the liquid crystal display device comprises a liquid crystal display device,for regulating->And->Two energy terms are in the objective function +.>Is a weight of (a).
Further, triangular meshRepresenting the three-dimensional shape of the face. Triangle mesh->Is formed by connecting triangles in a set of three-dimensional spaces through shared edges or vertices. Mathematically, it is composed of a triad of vertices (V, E, F) representing the mesh topology, composed of a set of vertices (V), a set of edges (E), and a set of triangles (F), combined with a set of three-dimensional spatial coordinates (P) of vertices, triangular meshComprising the following steps:
V={v 1 ,v 2 ,…,v n };
E={e 1 ,e 2 ,…,e k },e i ∈V×V;
F={f 1 ,f 2 ,…,f m },t i ∈V×V×V;
wherein: v i ∈V、e i E and f i E F is the vertex, the edge and the triangle in the triangle meshIndex in, n, k and m are the number of vertices, edges and triangles, respectively, in the triangle mesh, +.>Is triangle mesh vertex v i Is a three-dimensional space coordinate of (c).
Still further, due to the initial camera intrinsic matrix κ 0 And an extrinsic matrix [ R ] 0 t 0 ]Not sufficiently accurate, so that the two matrices need to be optimized before optimizing the initial triangle mesh. The step of optimizing the initial triangular mesh comprises:
detecting face key point coordinates in a stylized face image, wherein the key point detection technology can refer to paper Robust Face Alignment via Deep Progressive Reinitialization and Adaptive Error-drive Learning;
optimizing the extrinsic matrix [ R, t ]]The optimization formula is as follows:techniques for optimizing the reference matrix and the reference matrix can be found in the paper adam: A Method for Stochastic Optimization;
optimizing the vertex coordinate set of the initial triangle mesh, wherein the optimization formula is as follows:which is solved by Cholesky decomposition;
and iterating the optimization step until the iteration times reach a set upper limit.
The embodiment of the invention provides a system for modeling a three-dimensional shape of a stylized face, which comprises the following components:
the training module is used for training the 3DMM fitting model by using the real face image;
the output module is used for inputting the stylized face image into the trained 3DMM fitting model to obtain an initial triangle mesh matched with the face shape;
and the deformation module is used for carrying out Laplacian deformation on the initial triangular grid by taking the key points of the face in the stylized face image as the target to obtain a final triangular grid which is accurately matched with the shape of the face.
The device provided by the embodiment of the invention comprises at least one processing unit and at least one storage unit, wherein the storage unit stores a program, and when the program is executed by the processing unit, the processing unit is caused to execute the method.
The embodiment of the invention provides a computer readable storage medium storing a computer program executable by a device, which when run on the device causes the device to perform the method described above.
The above is only a preferred embodiment of the present invention, and is not intended to limit the present invention, but various modifications and variations can be made to the present invention by those skilled in the art. Any modification, equivalent replacement, improvement, etc. made within the spirit and principle of the present invention should be included in the protection scope of the present invention.
Claims (10)
1. A method for modeling a three-dimensional shape of a stylized face, comprising:
training the 3DMM fitting model by using the real face image;
inputting the stylized face image into the trained 3DMM fitting model to obtain an initial triangle mesh matched with the face shape;
and carrying out Laplacian deformation on the initial triangular grid by taking the key points of the face in the stylized face image as targets to obtain a final triangular grid which is accurately matched with the face shape.
2. The method of claim 1, wherein the step of performing laplace deformation on the initial triangular mesh with respect to face key points in the stylized face image to obtain a final triangular mesh that exactly matches the face shape comprises:
the objective function formula of the final triangle mesh is as follows:
wherein:the deviation between Laplacian coordinates of the vertexes of the triangle meshes before and after deformation; />The method is characterized in that the deviation between the projection position of the key points on the deformed triangle mesh on the stylized face image and the corresponding key point positions detected on the stylized face image is obtained; alpha is a parameter.
3. A method of three-dimensional shape modeling of a stylized face according to claim 2,the formula is:
Wherein: l represents the number of key points,two-dimensional image coordinates representing key points detected on a stylized face image, +.>Representing three-dimensional space coordinates of the associated triangular mesh keypoints; p is a three-dimensional space coordinate set of the deformed triangle mesh vertexes; pi (II) k Is a perspective projection operation based on the camera reference matrix k.
4. A method of three-dimensional shape modeling of a stylized face according to claim 2,the formula is:
5. A method of three-dimensional shape modeling of a stylized face as defined in claim 4, wherein said triangular meshComprising the following steps:
V={v 1 ,v 2 ,…,v n };
E={e 1 ,e 2 ,…,e k },e i ∈V×V;
F={f 1 ,f 2 ,…,f m },t i ∈V×V×V;
6. The method for three-dimensional shape modeling of a stylized face of claim 5, wherein the stylized face image is input into the trained 3DMM fitted model to obtain an initial triangular mesh, and further to obtain a camera internal reference matrix and an external reference matrix.
7. The method of stylized face three-dimensional shape modeling of claim 6, wherein the optimization of the initial triangle mesh to laplace deformation comprises:
detecting face key point coordinates in the stylized face image;
optimizing the vertex coordinate set of the initial triangle mesh, wherein an optimization formula is as follows:
and iterating the optimization step until the iteration times reach a set upper limit.
8. A system for stylized face three-dimensional shape modeling, comprising:
the training module is used for training the 3DMM fitting model by using the real face image;
the output module is used for inputting the stylized face image into the trained 3DMM fitting model to obtain an initial triangle mesh matched with the face shape;
and the deformation module is used for carrying out Laplacian deformation on the initial triangular mesh by taking the face key points in the stylized face image as targets to obtain a final triangular mesh which is accurately matched with the face shape.
9. An apparatus comprising at least one processing unit and at least one storage unit, wherein the storage unit stores a computer program which, when executed by the processing unit, causes the processing unit to perform the method of any of claims 1 to 7.
10. A storage medium storing a computer program executable by a device, which when run on the device causes the device to perform the method of any one of claims 1 to 7.
Priority Applications (1)
Application Number | Priority Date | Filing Date | Title |
---|---|---|---|
CN202310111635.8A CN115994993A (en) | 2023-01-30 | 2023-01-30 | Stylized face three-dimensional shape modeling method, system, equipment and storage medium |
Applications Claiming Priority (1)
Application Number | Priority Date | Filing Date | Title |
---|---|---|---|
CN202310111635.8A CN115994993A (en) | 2023-01-30 | 2023-01-30 | Stylized face three-dimensional shape modeling method, system, equipment and storage medium |
Publications (1)
Publication Number | Publication Date |
---|---|
CN115994993A true CN115994993A (en) | 2023-04-21 |
Family
ID=85991921
Family Applications (1)
Application Number | Title | Priority Date | Filing Date |
---|---|---|---|
CN202310111635.8A Pending CN115994993A (en) | 2023-01-30 | 2023-01-30 | Stylized face three-dimensional shape modeling method, system, equipment and storage medium |
Country Status (1)
Country | Link |
---|---|
CN (1) | CN115994993A (en) |
Cited By (1)
Publication number | Priority date | Publication date | Assignee | Title |
---|---|---|---|---|
CN117808943A (en) * | 2024-02-29 | 2024-04-02 | 天度(厦门)科技股份有限公司 | Three-dimensional cartoon face reconstruction method, device, equipment and storage medium |
-
2023
- 2023-01-30 CN CN202310111635.8A patent/CN115994993A/en active Pending
Cited By (1)
Publication number | Priority date | Publication date | Assignee | Title |
---|---|---|---|---|
CN117808943A (en) * | 2024-02-29 | 2024-04-02 | 天度(厦门)科技股份有限公司 | Three-dimensional cartoon face reconstruction method, device, equipment and storage medium |
Similar Documents
Publication | Publication Date | Title |
---|---|---|
CN109145366B (en) | Web 3D-based lightweight visualization method for building information model | |
US8331615B2 (en) | Match, expand, and filter technique for multi-view stereopsis | |
US8711143B2 (en) | System and method for interactive image-based modeling of curved surfaces using single-view and multi-view feature curves | |
CN112927354B (en) | Three-dimensional reconstruction method, system, storage medium and terminal based on example segmentation | |
CN116310076A (en) | Three-dimensional reconstruction method, device, equipment and storage medium based on nerve radiation field | |
CN114708375B (en) | Texture mapping method, system, computer and readable storage medium | |
CN113393577B (en) | Oblique photography terrain reconstruction method | |
CN115994993A (en) | Stylized face three-dimensional shape modeling method, system, equipment and storage medium | |
CN107358645A (en) | Product method for reconstructing three-dimensional model and its system | |
CN115439607A (en) | Three-dimensional reconstruction method and device, electronic equipment and storage medium | |
Daniels et al. | Semi‐regular quadrilateral‐only remeshing from simplified base domains | |
CN111754431B (en) | Image area replacement method, device, equipment and storage medium | |
CN114022542A (en) | Three-dimensional reconstruction-based 3D database manufacturing method | |
CN114202632A (en) | Grid linear structure recovery method and device, electronic equipment and storage medium | |
CN114723884A (en) | Three-dimensional face reconstruction method and device, computer equipment and storage medium | |
CN115587987A (en) | Storage battery defect detection method and device, storage medium and electronic equipment | |
CN110335275B (en) | Fluid surface space-time vectorization method based on three-variable double harmonic and B spline | |
CN114782417A (en) | Real-time detection method for digital twin characteristics of fan based on edge enhanced image segmentation | |
US20220375163A1 (en) | Computationally-Efficient Generation of Simulations of Cloth-Like Materials Using Bilinear Element Models | |
US20080111814A1 (en) | Geometric tagging | |
CN110751026B (en) | Video processing method and related device | |
CN116758219A (en) | Region-aware multi-view stereo matching three-dimensional reconstruction method based on neural network | |
Liu et al. | Ghost on the Shell: An Expressive Representation of General 3D Shapes | |
CN115761116A (en) | Monocular camera-based three-dimensional face reconstruction method under perspective projection | |
CN114998551B (en) | Grid reconstruction quality optimization method, system, computer and readable storage medium |
Legal Events
Date | Code | Title | Description |
---|---|---|---|
PB01 | Publication | ||
PB01 | Publication | ||
SE01 | Entry into force of request for substantive examination | ||
SE01 | Entry into force of request for substantive examination |