CN111768477B - Three-dimensional facial expression base establishment method and device, storage medium and electronic equipment - Google Patents
Three-dimensional facial expression base establishment method and device, storage medium and electronic equipment Download PDFInfo
- Publication number
- CN111768477B CN111768477B CN202010642367.9A CN202010642367A CN111768477B CN 111768477 B CN111768477 B CN 111768477B CN 202010642367 A CN202010642367 A CN 202010642367A CN 111768477 B CN111768477 B CN 111768477B
- Authority
- CN
- China
- Prior art keywords
- target
- face model
- dimensional
- key points
- model
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Active
Links
- 230000008921 facial expression Effects 0.000 title claims abstract description 50
- 238000000034 method Methods 0.000 title claims abstract description 47
- 238000003860 storage Methods 0.000 title claims abstract description 20
- 239000011159 matrix material Substances 0.000 claims abstract description 68
- 230000014509 gene expression Effects 0.000 claims abstract description 67
- 230000001815 facial effect Effects 0.000 claims abstract description 37
- 238000012545 processing Methods 0.000 claims abstract description 21
- 230000008859 change Effects 0.000 claims description 32
- 238000004364 calculation method Methods 0.000 claims description 8
- 238000004590 computer program Methods 0.000 claims description 2
- 230000000694 effects Effects 0.000 abstract description 5
- 238000005516 engineering process Methods 0.000 abstract description 2
- 238000010586 diagram Methods 0.000 description 9
- 238000000513 principal component analysis Methods 0.000 description 8
- 230000008569 process Effects 0.000 description 4
- 239000013598 vector Substances 0.000 description 4
- 210000004709 eyebrow Anatomy 0.000 description 3
- 230000003287 optical effect Effects 0.000 description 3
- 238000005457 optimization Methods 0.000 description 3
- 230000009466 transformation Effects 0.000 description 3
- 238000013519 translation Methods 0.000 description 3
- PXFBZOLANLWPMH-UHFFFAOYSA-N 16-Epiaffinine Natural products C1C(C2=CC=CC=C2N2)=C2C(=O)CC2C(=CC)CN(C)C1C2CO PXFBZOLANLWPMH-UHFFFAOYSA-N 0.000 description 2
- 230000006978 adaptation Effects 0.000 description 2
- 238000004422 calculation algorithm Methods 0.000 description 2
- 238000000605 extraction Methods 0.000 description 2
- 210000000887 face Anatomy 0.000 description 2
- 238000004519 manufacturing process Methods 0.000 description 2
- 239000013307 optical fiber Substances 0.000 description 2
- 230000000644 propagated effect Effects 0.000 description 2
- 230000009471 action Effects 0.000 description 1
- 238000003491 array Methods 0.000 description 1
- 230000006399 behavior Effects 0.000 description 1
- 230000009286 beneficial effect Effects 0.000 description 1
- 238000004891 communication Methods 0.000 description 1
- 238000013461 design Methods 0.000 description 1
- 238000011161 development Methods 0.000 description 1
- 230000008451 emotion Effects 0.000 description 1
- 230000002708 enhancing effect Effects 0.000 description 1
- 230000006870 function Effects 0.000 description 1
- 230000003993 interaction Effects 0.000 description 1
- 230000005012 migration Effects 0.000 description 1
- 238000013508 migration Methods 0.000 description 1
- 238000012986 modification Methods 0.000 description 1
- 230000004048 modification Effects 0.000 description 1
- 230000002093 peripheral effect Effects 0.000 description 1
- 230000003252 repetitive effect Effects 0.000 description 1
- 239000004065 semiconductor Substances 0.000 description 1
- 238000007619 statistical method Methods 0.000 description 1
- 230000002123 temporal effect Effects 0.000 description 1
- 230000000007 visual effect Effects 0.000 description 1
Classifications
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
- G06T13/00—Animation
- G06T13/20—3D [Three Dimensional] animation
- G06T13/40—3D [Three Dimensional] animation of characters, e.g. humans, animals or virtual beings
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
- G06T15/00—3D [Three Dimensional] image rendering
- G06T15/10—Geometric effects
Landscapes
- Physics & Mathematics (AREA)
- Engineering & Computer Science (AREA)
- General Physics & Mathematics (AREA)
- Theoretical Computer Science (AREA)
- Geometry (AREA)
- Computer Graphics (AREA)
- Processing Or Creating Images (AREA)
Abstract
The present disclosure relates to the field of image processing technologies, and in particular, to a method and an apparatus for creating a three-dimensional facial expression group, a computer readable storage medium, and an electronic device, where the method includes: extracting target key points of a target face image in input data, determining a camera projection matrix according to the target key points and initial key points corresponding to the target key points in an average face model, wherein the average face model is a reference three-dimensional face model obtained by processing according to a face model data set; calculating a target three-dimensional face model corresponding to the target face image according to the target key points, the average face model reference three-dimensional face model and the initial key points by using a camera projection matrix; and generating a target three-dimensional facial expression base corresponding to the target facial image by using the target three-dimensional facial model. The technical scheme of the embodiment of the disclosure can solve the problems of poor effect and large calculated amount of the expression base model obtained in the prior art.
Description
Technical Field
The disclosure relates to the technical field of image processing, in particular to a three-dimensional facial expression base establishment method and device, a computer readable storage medium and electronic equipment.
Background
Along with the development of the Internet, the technology for establishing the three-dimensional facial expression base is widely applied to the aspects of animation production, game production, man-machine interaction, video advertisements and the like.
However, the three-dimensional facial expression base building method in the prior art has low automation degree, inconsistent standard vertebrae, poor effect of the obtained expression base model and large calculated amount.
Therefore, it is necessary to design a new three-dimensional facial expression base establishment method.
It should be noted that the information disclosed in the above background section is only for enhancing understanding of the background of the present disclosure and thus may include information that does not constitute prior art known to those of ordinary skill in the art.
Disclosure of Invention
The invention aims to provide a three-dimensional facial expression base establishment method and device, a computer-readable storage medium and electronic equipment, and further at least to a certain extent solve the problems of low automation degree, inconsistent standard vertebrae, poor effect of an obtained expression base model and large calculated amount of a three-dimensional facial expression base establishment cube method in the prior art.
Other features and advantages of the present disclosure will be apparent from the following detailed description, or may be learned in part by the practice of the disclosure.
According to a first aspect of the present disclosure, there is provided a three-dimensional facial expression base establishment method, including:
Extracting target key points of a target face image in input data, and determining a camera projection matrix according to the target key points and initial key points corresponding to the target key points in an average face model, wherein the average face model is a reference three-dimensional face model obtained by processing according to a face model data set;
Calculating a target three-dimensional face model corresponding to the target face image according to the target key points, the average face model reference three-dimensional face model and the initial key points by using the camera projection matrix;
And generating a target three-dimensional facial expression base corresponding to the target facial image by using the target three-dimensional facial model.
In an exemplary embodiment of the present disclosure, the average face model is a reference three-dimensional face model obtained by processing according to a face model dataset, and includes:
The face model data set is statistically analyzed, and a PCA face parameterized model is obtained;
And calculating an average face model of the PCA face parameterized model as the reference three-dimensional face model.
In an exemplary embodiment of the present disclosure, determining a camera projection matrix according to the target keypoints and initial keypoints corresponding to the target keypoints in an average face model includes:
determining mathematical relationships among the target keypoints, the initial keypoints and the camera projection matrix;
estimating the camera projection matrix by minimizing a re-projection error according to the mathematical relationship.
In an exemplary embodiment of the present disclosure, calculating, by using the camera projection matrix, a target three-dimensional face model corresponding to the target face image according to the target key point, the average face model reference three-dimensional face model, and the initial key point includes:
establishing a parameter relation between the target three-dimensional face model and the reference three-dimensional face model;
adjusting the parameter relation to enable the initial key point to coincide with the target key point after being projected by a camera projection matrix;
And calculating the target three-dimensional face model by utilizing the modified parameter relation and the reference three-dimensional face model.
In one exemplary embodiment of the present disclosure, before calculating the target three-dimensional face model using the modified parametric relationship and the reference three-dimensional face model, the method further comprises:
And optimizing the camera projection matrix and the parameter relation through multiple iterations by utilizing the target key points and the initial key points.
In one exemplary embodiment of the present disclosure, generating a target three-dimensional facial expression base corresponding to the target face image using the target three-dimensional facial model includes:
acquiring a template expression base, and determining a reference expression change triangular face piece according to the reference three-dimensional face model and the template expression base;
selecting a target expression change triangular patch from the target three-dimensional face model according to the reference expression change triangular patch;
and adjusting the target expression change triangular surface patch according to the reference expression change triangular surface patch.
In one exemplary embodiment of the present disclosure, adjusting the target expression change triangular patch according to the reference expression change triangular patch includes:
And adjusting the expression in the target expression change triangular facial mask in the same proportion according to the size and the expression of the reference expression change triangular facial mask.
In an exemplary embodiment of the present disclosure, the input data includes a single face image, a plurality of face images, or a plurality of frames of RGB-D face data.
According to an aspect of the present disclosure, there is provided a three-dimensional facial expression base establishment apparatus including:
The estimating module is used for extracting target key points of a target face image in input data, and determining a camera projection matrix according to the target key points and initial key points corresponding to the target key points in an average face model, wherein the average face model is a reference three-dimensional face model obtained by processing according to a face model data set;
The calculation module is used for calculating a target three-dimensional face model corresponding to the target face image according to the target key points, the average face model reference three-dimensional face model and the initial key points by using the camera projection matrix;
And the generating module is used for generating a target three-dimensional facial expression base corresponding to the target facial image by utilizing the target three-dimensional facial model.
According to an aspect of the present disclosure, there is provided a computer-readable storage medium having stored thereon a computer program which, when executed by a processor, implements the three-dimensional facial expression base establishment method as set forth in any one of the above.
According to one aspect of the present disclosure, there is provided an electronic device including:
A processor; and
A memory for storing one or more programs that, when executed by the one or more processors, cause the one or more processors to implement the three-dimensional facial expression base establishment method as recited in any one of the above.
The technical scheme provided by the embodiment of the disclosure can comprise the following beneficial effects:
In the three-dimensional facial expression base establishing method provided by the embodiment of the disclosure, a target key point of a target facial image in input data is extracted, a camera projection matrix is determined according to the target key point and an initial key point corresponding to the target key point in an average facial model, wherein the average facial model is a reference three-dimensional facial model obtained by processing a facial model data set; calculating a target three-dimensional face model corresponding to the target face image according to the target key points, the average face model reference three-dimensional face model and the initial key points by using a camera projection matrix; and generating a target three-dimensional facial expression base corresponding to the target facial image by using the target three-dimensional facial model. Compared with the prior art, the method has the advantages that the projection matrix is calculated by utilizing the target key points and the initial key points, the target three-dimensional face model is calculated, the calculated amount can be reduced, the calculation pressure of a server is reduced, the reference three-dimensional face model is obtained by utilizing data in the data set, then the target three-dimensional face model is calculated according to the target key points and the initial key points through the camera projection matrix, and the obtained model effect of the three-dimensional face expression base is better through the calculation of the camera projection matrix.
It is to be understood that both the foregoing general description and the following detailed description are exemplary and explanatory only and are not restrictive of the disclosure.
Drawings
The accompanying drawings, which are incorporated in and constitute a part of this specification, illustrate embodiments consistent with the disclosure and together with the description, serve to explain the principles of the disclosure. It will be apparent to those of ordinary skill in the art that the drawings in the following description are merely examples of the disclosure and that other drawings may be derived from them without undue effort. In the drawings:
fig. 1 schematically illustrates a flowchart of a three-dimensional facial expression base establishment method in an exemplary embodiment of the present disclosure;
FIG. 2 schematically illustrates a camera model schematic in an exemplary embodiment of the present disclosure;
fig. 3 schematically illustrates a face key point extraction schematic diagram when input data is an RGB-D image in an exemplary embodiment of the present disclosure;
FIG. 4 schematically illustrates a schematic diagram of obtaining a target reference model from a reference three-dimensional face model through parametric deformation in an exemplary embodiment of the present disclosure;
FIG. 5 schematically illustrates a schematic diagram of template expression group acquisition in an exemplary embodiment of the present disclosure;
FIG. 6 schematically illustrates a table base migration schematic in an exemplary embodiment of the present disclosure;
FIG. 7 schematically illustrates an explanatory diagram of an emotion base mask in an exemplary embodiment of the present disclosure;
FIG. 8 schematically illustrates a schematic diagram of a template expression group in an exemplary embodiment of the present disclosure;
Fig. 9 schematically illustrates a composition diagram of a three-dimensional facial expression base establishment apparatus in an exemplary embodiment of the present disclosure;
FIG. 10 schematically illustrates a structural schematic diagram of a computer system suitable for use in implementing the electronic device of the exemplary embodiments of the present disclosure;
Fig. 11 schematically illustrates a schematic diagram of a computer-readable storage medium according to some embodiments of the present disclosure.
Detailed Description
Example embodiments will now be described more fully with reference to the accompanying drawings. However, the exemplary embodiments may be embodied in many forms and should not be construed as limited to the examples set forth herein; rather, these embodiments are provided so that this disclosure will be thorough and complete, and will fully convey the concept of the example embodiments to those skilled in the art. The described features, structures, or characteristics may be combined in any suitable manner in one or more embodiments.
Furthermore, the drawings are merely schematic illustrations of the present disclosure and are not necessarily drawn to scale. The same reference numerals in the drawings denote the same or similar parts, and thus a repetitive description thereof will be omitted. Some of the block diagrams shown in the figures are functional entities and do not necessarily correspond to physically or logically separate entities. These functional entities may be implemented in software or in one or more hardware modules or integrated circuits or in different networks and/or processor devices and/or microcontroller devices.
In the present exemplary embodiment, there is provided a three-dimensional facial expression base establishment method first, referring to fig. 1, the three-dimensional facial expression base establishment method may include the steps of:
s110, extracting target key points of a target face image in input data, and determining a camera projection matrix according to the target key points and initial key points corresponding to the target key points in an average face model, wherein the average face model is a reference three-dimensional face model obtained by processing according to a face model data set;
s120, calculating a target three-dimensional face model corresponding to the target face image according to the target key points, the average face model reference three-dimensional face model and the initial key points by using the camera projection matrix;
S130, generating a target three-dimensional facial expression base corresponding to the target facial image by using the target three-dimensional facial model.
According to the three-dimensional facial expression base establishment method provided by the embodiment, compared with the prior art, the projection matrix is calculated by utilizing the target key points and the initial key points, and the target three-dimensional facial model is calculated, so that the calculation amount of a server can be reduced, the calculation pressure of the server is reduced, the reference three-dimensional facial model is obtained by utilizing the data in the data set, then the target three-dimensional facial model is calculated according to the target key points and the initial key points through the camera projection matrix, and the obtained three-dimensional facial expression base model effect is better through the calculation of the camera projection matrix.
Hereinafter, each step of the three-dimensional facial expression base creation method in the present exemplary embodiment will be described in more detail with reference to the drawings and embodiments.
Step S110, extracting target key points of a target face image in input data, and determining a camera projection matrix according to the target key points and initial key points corresponding to the target key points in an average face model, wherein the average face model is a reference three-dimensional face model obtained by processing according to a face model data set.
In one example embodiment of the present disclosure, the server may first obtain a face dataset from a database, and then perform a statistical analysis on the face dataset to obtain an obtained PCA (PRINCIPAL COMPONENT ANALYSIS ) face parameterized model.
Specifically, the PCA face parameterization model is a parameterization model based on a 3D face data set, each three-dimensional face model in the data set is set to be expressed as S= (X 1,Y1,Z1,X2,……,Yn,Zn)T∈R3n, which comprises n P= [ X, Y, Z ] coordinates), PCA operation is carried out on m face data, corresponding characteristic values and characteristic vectors are decomposed, main components are built, the first k main components are obtained, and the PCA face parameterization model is generated, wherein m, n and k are all positive integers.
In this example embodiment, an average face model of the PCA face parameterized model may then be calculated as a reference three-dimensional face model, specifically, the average face model is obtained by summing averages of the 3D face data sets.
In this example embodiment, the initial key points of the calibration reference three-dimensional face model may be extracted according to a preset rule, where the extraction calibration of the key points may include the face contour points, eyebrows, eyes, nose, mouth, and other parts, and may be customized according to the requirements, and the number of the initial key points may be 68 points, 106 points, and the like, which is not specifically limited in this example embodiment.
In this example embodiment, the input data may include a single face image, a plurality of face images, or a plurality of frames of RGB-D face data, and when the input data is a single face image, the target key points in the image may be according to the above preset rule, where the target key points may include positions such as a face contour point, eyebrows, eyes, nose, mouth, and the like. When the input data is a plurality of face images, the face contour points, eyebrows, eyes, nose, mouth and other parts of each face image can be respectively extracted as target key points. The above-mentioned ways of confirming the initial key points and confirming the target key points are the same, that is, the number of the obtained initial key points and the number of the target key points are the same, and the positions on the face may be the same, may be 68 points, 106 points, etc., which are not specifically defined in the present exemplary embodiment, and at the same time, the above-mentioned preset rules of the specific determination mode area may be customized according to the needs, which are not specifically defined in the present exemplary embodiment,
In this exemplary embodiment, when the input data is multi-frame RGB-D face data, key points on each frame of RGB face image are extracted, and according to the depth information, the scanned three-dimensional face model is fused, the key points of the face image are back projected onto the three-dimensional face model through the camera projection matrix, and the three-dimensional face key points are extracted as follows. As shown in fig. 2, a point p= [ x, y, z ] T in the three-dimensional space is projected onto the two-dimensional image as P' = [ u, v ] T, and the formula is as follows:
P′＝sK[RP+T]
Where R is the rotation matrix, T is the translation vector, K is the camera reference matrix, and s is the scaling factor. R and T can be estimated for each frame of three-dimensional point cloud through an ICP algorithm, as shown in FIG. 3, the key points of multiple frames of faces are 'observed' to the same three-dimensional point, and the three-dimensional coordinates of each key point can be estimated through the constraint.
In this example real-time manner, the camera projection matrix may be estimated according to the target keypoints and the initial keypoints, specifically, in an example embodiment of the present disclosure, when the input data is a single face image or multiple face images, the camera projection matrix needs to be estimated, and the camera projection matrix is a matrix that projects a three-dimensional space into a two-dimensional image space, and referring to fig. 2, the matrix generally includes a camera reference matrix K, a rotation matrix R, a translation vector T, and a point in the three-dimensional space is represented as homogeneous coordinates p= [ x, y, z,1] T, and an upper point P' = [ u, v,1] T on the image:
P′＝sK[RT]P
and (3) finishing:
P′＝MP
Wherein M ε R 3×4 is the camera projection matrix, which is degenerated to affine transformation, and the last behavior [0, 1], the camera projection matrix is estimated by minimizing the reprojection error, namely: the camera projection matrix is estimated by the following formula:
In this exemplary embodiment, when the input data is a plurality of face images with different perspectives, a projection matrix of the current frame needs to be estimated for each view, where i is a positive integer and indicates the order of points.
In the present exemplary embodiment, when the input data is multi-frame RGB-D face data, it is necessary to estimate a outlier matrix [ RT ] between two sets of initial keypoints and target keypoints, the matrix containing a rotation matrix R, a translation vector T, which can be estimated by ICP algorithm. Specifically, an existing set of matching points [ P, Q ] is set, satisfying:
Q＝RP+T
the target equation is:
min‖RP+T-Q‖
the camera projection matrix between the initial keypoints and the target keypoints is estimated by minimizing the distance between the matching points.
In step S120, a target three-dimensional face model corresponding to the target face image is obtained by calculating the target key point, the average face model reference three-dimensional face model, and the initial key point by using the camera projection matrix.
In this exemplary embodiment, the parameter relationship between the target three-dimensional face model and the reference three-dimensional face model may be first determined, and the target three-dimensional face model includes the reference three-dimensional face model, that is, the average face modelAnd k principal components v= (V 1,V2,……,Vk)∈R3n×k, as shown in fig. 4, the model parameterization can be used to represent any face model:
The four three-dimensional face models are represented by different parameters a i in the figure.
In the above formula, S may represent a target three-dimensional face model,Representing the reference three-dimensional face model, a i may represent a parameter relationship, k is a positive integer, a specific numerical value may be customized according to the requirement, and in this example embodiment, no specific limitation is made.
In the present exemplary embodiment, referring to fig. 4, the first model 420, the second model 430, the third model 440, and the fourth model 450 may be calculated by using the above formulas with reference to the three-dimensional face model 410 through different parameters a 0、a1、a2、a3, respectively.
In this example embodiment, the parameter relationship may then be adjusted such that the initial keypoints coincide with the target keypoints after projection through the camera projection matrix. Specifically, when the input data is a single face image, the minimized re-projection error is obtained according to the camera projection matrix me R 3×4 obtained in S120, that is, the parameter relationship is continuously adjusted, so that the re-projection error is minimized, and a specific formula may be:
Wherein, P Si is a key point selected from the target three-dimensional face model S, minimizing the reprojection error, obtaining a target parameter a i between the target three-dimensional face model and the reference three-dimensional face model, and calculating the target three-dimensional face model.
In this exemplary embodiment, when the input data is a plurality of face images, the reprojection errors of a plurality of frames at different angles of view are constrained together, and the objective equation is as follows:
Wherein P Si is a key point selected from the target three-dimensional face model S, ω j is a weight corresponding to different frames, and the face orientation of each face image can be obtained from the extrinsic matrix estimated in step S120, and the weight of each frame can be assigned according to the angle of orientation. M j is a camera projection matrix under different visual angles of a plurality of frames, and a target three-dimensional face model is estimated through the reprojection error of key points of the plurality of frames of faces. And continuously adjusting the parameter relation to minimize the re-projection error, selecting the parameter relation with the minimum re-projection error as the modified parameter relation, and calculating to obtain the target three-dimensional face model.
When multi-frame RGB-D face data is input, three-dimensional coordinates of target key points of the scanned face are obtained, an external parameter matrix between the target key points and the initial key points is estimated through S120, and the distance between the key points is used as an optimization target, as follows:
And (3) estimating the parameter relation between the target three-dimensional face model and the reference three-dimensional face model by combining a model parameterized expression formula. And continuously adjusting the parameter relation to minimize the re-projection error, selecting the parameter relation with the minimum re-projection error as the modified parameter relation, and calculating to obtain the target three-dimensional face model.
In an example embodiment of the present disclosure, the target keypoints and the initial keypoints may be utilized to perform iterative optimization on the camera projection matrix and the parameter relationship for a plurality of times, specifically, iterative optimization on the parameter relationship and the external parameter matrix may be performed for a plurality of times, that is, step S110 and step S120 may be performed for a plurality of times, and the specific times may be customized according to requirements, which is not specifically limited in this example embodiment.
In step S130, a target three-dimensional facial expression base corresponding to the target face image is generated using the target three-dimensional facial model.
In an example embodiment of the present disclosure, a template expression base may be first acquired, and a reference expression change triangular patch may be determined according to a reference three-dimensional face model and the template expression base, that is, a triangular patch in which a change occurs in the template expression base and the reference three-dimensional face model is determined, and then a target expression change triangular patch in the target three-dimensional face model corresponding to the reference expression change triangular patch is determined according to a parameter relationship between the reference three-dimensional face model and the target three-dimensional face model. And then the expression in the target expression change triangular face piece can be adjusted in the same proportion according to the size and the expression of the reference expression change triangular face piece.
Specifically, affine transformation a for each patch of expression base is:
Where V＝[v2-v1,v3-v1,v4-v1],v1,v2,v3 is the three vertices of the triangular patch and v 4 is a point in the direction of the triangular patch.
In this exemplary embodiment, referring to fig. 5 and 6, each expression group of the template expression group is obtained by stretching and retracting the relevant vertices and the patches on the basis of the reference three-dimensional face model, and the obtained topology of all the template expression group models is consistent, so that the corresponding relationship of each patch is not required to be calculated. The transformation of each triangular face piece in each template expression base relative to the triangular face piece corresponding to the reference three-dimensional face model is as followsTransformation of each triangular surface patch in each three-dimensional facial expression base relative to corresponding triangular surface patches of the target three-dimensional facial model is/>The vertex coordinates of the target expression base are estimated by making their transforms uniform, namely:
Wherein, Wherein/>Can be used for representing the parameters of the triangular face piece corresponding to the reference three-dimensional face model,/>Can be used to represent parameters of the template expression base corresponding to the triangular patches. /(I)Wherein/>Can be used for representing the parameters of triangular patches in the three-dimensional facial expression base,/>Can be used to represent parameters of the corresponding triangular patches of the target three-dimensional face model. The three-dimensional facial expression base is obtained by adjusting a target three-dimensional facial model.
Namely, the coordinates of the fixed point of the target expression change triangular surface patch and one point in the triangular surface patch direction in the target three-dimensional face model are adjusted, and the adjustment of the expression in the target expression change triangular surface patch is completed by selecting the coordinates of the fixed point of the target expression change triangular surface patch and one point in the triangular surface patch direction when the error is minimum, so that the establishment of the three-dimensional face expression base is completed.
Wherein α i is the weight of each triangular face, as shown in fig. 7, the template expression base is obtained by changing part of vertices on the basis of the reference three-dimensional face model, so that the vertex difference between each template expression model B i and the reference three-dimensional face model B o forms a mask model, and the mask model can be used as the basis for generating the weight α i; the generated expression base model coordinate is consistent.
In the present exemplary embodiment, referring to fig. 8, the template expression group may include expressions of a plurality of three-dimensional models, and specifically may include a plurality of expressions such as a dead expression, a closed eye, a mouth opening, a beep mouth, a skim mouth, etc., which is not particularly limited in the present exemplary embodiment.
The following describes an embodiment of an apparatus of the present disclosure, which may be used to perform the above-described three-dimensional facial expression base establishment method of the present disclosure. In addition, in an exemplary embodiment of the present disclosure, a three-dimensional facial expression base establishment apparatus is also provided. Referring to fig. 9, the three-dimensional facial expression base establishment apparatus 900 includes: an estimation module 910, a calculation module 920, and a generation module 930.
The estimation module 910 may be configured to extract a target key point of a target face image in input data, determine a camera projection matrix according to the target key point and an initial key point corresponding to the target key point in an average face model, where the average face model is a reference three-dimensional face model obtained by processing according to a face model dataset; the calculating module 920 may be configured to calculate, using the camera projection matrix, a target three-dimensional face model corresponding to the target face image according to the target key point, the average face model reference three-dimensional face model, and the initial key point; the generating module 930 may be configured to generate a target three-dimensional facial expression base corresponding to the target face image using the target three-dimensional facial model.
Since each functional module of the three-dimensional facial expression base establishment device of the exemplary embodiment of the present disclosure corresponds to a step of the foregoing exemplary embodiment of the three-dimensional facial expression base establishment method, for details not disclosed in the embodiments of the present disclosure, please refer to the foregoing embodiment of the three-dimensional facial expression base establishment method of the present disclosure.
It should be noted that although in the above detailed description several modules or units of a device for action execution are mentioned, such a division is not mandatory. Indeed, the features and functionality of two or more modules or units described above may be embodied in one module or unit in accordance with embodiments of the present disclosure. Conversely, the features and functions of one module or unit described above may be further divided into a plurality of modules or units to be embodied.
In addition, in the exemplary embodiment of the disclosure, an electronic device capable of implementing the above three-dimensional facial expression base establishment is also provided.
Those skilled in the art will appreciate that the various aspects of the present disclosure may be implemented as a system, method, or program product. Accordingly, various aspects of the disclosure may be embodied in the following forms, namely: an entirely hardware embodiment, an entirely software embodiment (including firmware, micro-code, etc.) or an embodiment combining hardware and software aspects may be referred to herein as a "circuit," module "or" system.
An electronic device 1000 according to such an embodiment of the present disclosure is described below with reference to fig. 10. The electronic device 1000 shown in fig. 10 is merely an example and should not be construed as limiting the functionality and scope of use of the disclosed embodiments.
As shown in fig. 10, the electronic device 1000 is embodied in the form of a general purpose computing device. Components of electronic device 1000 may include, but are not limited to: the at least one processing unit 1010, the at least one memory unit 1020, a bus 1030 connecting the various system components (including the memory unit 1020 and the processing unit 1010), and a display unit 1040.
Wherein the storage unit stores program code that is executable by the processing unit 1010 such that the processing unit 1010 performs steps according to various exemplary embodiments of the present disclosure described in the above-described "exemplary methods" section of the present specification. For example, the processing unit 1010 may perform step S110 as shown in fig. 1: extracting target key points of a target face image in input data, and determining a camera projection matrix according to the target key points and initial key points corresponding to the target key points in an average face model, wherein the average face model is a reference three-dimensional face model obtained by processing according to a face model data set; s120: calculating a target three-dimensional face model corresponding to the target face image according to the target key points, the average face model reference three-dimensional face model and the initial key points by using the camera projection matrix; s130: and generating a target three-dimensional facial expression base corresponding to the target facial image by using the target three-dimensional facial model.
As another example, the electronic device may implement the steps shown in fig. 1-8.
The memory unit 1020 may include readable media in the form of volatile memory units such as Random Access Memory (RAM) 1021 and/or cache memory unit 1022, and may further include Read Only Memory (ROM) 1023.
Storage unit 1020 may also include a program/utility 1024 having a set (at least one) of program modules 1025, such program modules 1025 including, but not limited to: an operating system, one or more application programs, other program modules, and program data, each or some combination of which may include an implementation of a network environment.
Bus 1030 may be representing one or more of several types of bus structures including a memory unit bus or memory unit controller, a peripheral bus, an accelerated graphics port, a processing unit, or a local bus using any of a variety of bus architectures.
The electronic device 1000 can also communicate with one or more external devices 1070 (e.g., keyboard, pointing device, bluetooth device, etc.), with one or more devices that enable a user to interact with the electronic device 1000, and/or with any device (e.g., router, modem, etc.) that enables the electronic device 1000 to communicate with one or more other computing devices. Such communication may occur through an input/output (I/O) interface 1050. Also, electronic device 1000 can communicate with one or more networks such as a Local Area Network (LAN), a Wide Area Network (WAN), and/or a public network, such as the Internet, through network adapter 1060. As shown, the network adapter 1060 communicates with other modules of the electronic device 1000 over the bus 1030. It should be appreciated that although not shown, other hardware and/or software modules may be used in connection with the electronic device 1000, including, but not limited to: microcode, device drivers, redundant processing units, external disk drive arrays, RAID systems, tape drives, data backup storage systems, and the like.
From the above description of embodiments, those skilled in the art will readily appreciate that the example embodiments described herein may be implemented in software, or in combination with the necessary hardware. Thus, the technical solution according to the embodiments of the present disclosure may be embodied in the form of a software product, which may be stored in a non-volatile storage medium (may be a CD-ROM, a U-disk, a mobile hard disk, etc.) or on a network, and includes several instructions to cause a computing device (may be a personal computer, a server, a terminal device, or a network device, etc.) to perform the method according to the embodiments of the present disclosure.
In an exemplary embodiment of the present disclosure, a computer-readable storage medium having stored thereon a program product capable of implementing the method described above in the present specification is also provided. In some possible embodiments, the various aspects of the present disclosure may also be implemented in the form of a program product comprising program code for causing a terminal device to carry out the steps according to the various exemplary embodiments of the disclosure as described in the "exemplary methods" section of this specification, when the program product is run on the terminal device.
Referring to fig. 11, a program product 1100 for implementing the above-described method according to an embodiment of the present disclosure is described, which may employ a portable compact disc read-only memory (CD-ROM) and include program code, and may be run on a terminal device, such as a personal computer. However, the program product of the present disclosure is not limited thereto, and in this document, a readable storage medium may be any tangible medium that can contain, or store a program for use by or in connection with an instruction execution system, apparatus, or device.
The program product may employ any combination of one or more readable media. The readable medium may be a readable signal medium or a readable storage medium. The readable storage medium can be, for example, but is not limited to, an electronic, magnetic, optical, electromagnetic, infrared, or semiconductor system, apparatus, or device, or a combination of any of the foregoing. More specific examples (a non-exhaustive list) of the readable storage medium would include the following: an electrical connection having one or more wires, a portable disk, a hard disk, random Access Memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), optical fiber, portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the foregoing.
The computer readable signal medium may include a data signal propagated in baseband or as part of a carrier wave with readable program code embodied therein. Such a propagated data signal may take any of a variety of forms, including, but not limited to, electro-magnetic, optical, or any suitable combination of the foregoing. A readable signal medium may also be any readable medium that is not a readable storage medium and that can communicate, propagate, or transport a program for use by or in connection with an instruction execution system, apparatus, or device.
Program code embodied on a readable medium may be transmitted using any appropriate medium, including but not limited to wireless, wireline, optical fiber cable, RF, etc., or any suitable combination of the foregoing.
Program code for carrying out operations of the present disclosure may be written in any combination of one or more programming languages, including an object oriented programming language such as Java, C++ or the like and conventional procedural programming languages, such as the "C" programming language or similar programming languages. The program code may execute entirely on the user's computing device, partly on the user's device, as a stand-alone software package, partly on the user's computing device, partly on a remote computing device, or entirely on the remote computing device or server. In the case of remote computing devices, the remote computing device may be connected to the user computing device through any kind of network, including a Local Area Network (LAN) or a Wide Area Network (WAN), or may be connected to an external computing device (e.g., connected via the Internet using an Internet service provider).
Furthermore, the above-described figures are only schematic illustrations of processes included in the method according to the exemplary embodiments of the present disclosure, and are not intended to be limiting. It will be readily appreciated that the processes shown in the above figures do not indicate or limit the temporal order of these processes. In addition, it is also readily understood that these processes may be performed synchronously or asynchronously, for example, among a plurality of modules.
Other embodiments of the disclosure will be apparent to those skilled in the art from consideration of the specification and practice of the disclosure disclosed herein. This disclosure is intended to cover any adaptations, uses, or adaptations of the disclosure following the general principles of the disclosure and including such departures from the present disclosure as come within known or customary practice within the art to which the disclosure pertains. It is intended that the specification and examples be considered as exemplary only, with a true scope and spirit of the disclosure being indicated by the following claims.
It is to be understood that the present disclosure is not limited to the precise arrangements and instrumentalities shown in the drawings, and that various modifications and changes may be effected without departing from the scope thereof. The scope of the present disclosure is limited only by the appended claims.
Claims (8)
1. The method for establishing the three-dimensional facial expression base is characterized by comprising the following steps of:
Extracting target key points of a target face image in input data, and determining a camera projection matrix according to the target key points and initial key points corresponding to the target key points in an average face model, wherein the average face model is a reference three-dimensional face model obtained by processing according to a face model data set;
Calculating a target three-dimensional face model corresponding to the target face image according to the target key points, the average face model reference three-dimensional face model and the initial key points by using the camera projection matrix;
Generating a target three-dimensional facial expression base corresponding to the target facial image by utilizing the target three-dimensional facial model;
The method for determining the camera projection matrix according to the target key points and the initial key points corresponding to the target key points in the average face model comprises the following steps:
determining mathematical relationships among the target keypoints, the initial keypoints and the camera projection matrix;
estimating the camera projection matrix by minimizing a re-projection error according to the mathematical relationship;
Calculating a target three-dimensional face model corresponding to the target face image according to the target key points, the average face model reference three-dimensional face model and the initial key points by using the camera projection matrix, wherein the method comprises the following steps of:
establishing a parameter relation between the target three-dimensional face model and the reference three-dimensional face model;
adjusting the parameter relation to enable the initial key point to coincide with the target key point after being projected by a camera projection matrix;
Calculating the target three-dimensional face model by utilizing the modified parameter relation and the reference three-dimensional face model;
generating a target three-dimensional facial expression base corresponding to the target facial image by using the target three-dimensional facial model, comprising:
acquiring a template expression base, and determining a reference expression change triangular face piece according to the reference three-dimensional face model and the template expression base;
selecting a target expression change triangular patch from the target three-dimensional face model according to the reference expression change triangular patch;
and adjusting the target expression change triangular surface patch according to the reference expression change triangular surface patch.
2. The method according to claim 1, wherein the average face model is a reference three-dimensional face model obtained by processing a face model data set, and the method comprises:
The face model data set is statistically analyzed, and a PCA face parameterized model is obtained;
And calculating an average face model of the PCA face parameterized model as the reference three-dimensional face model.
3. The method of claim 1, wherein prior to calculating the target three-dimensional face model using the modified parametric relationship and the reference three-dimensional face model, the method further comprises:
And optimizing the camera projection matrix and the parameter relation through multiple iterations by utilizing the target key points and the initial key points.
4. The method of claim 1, wherein adjusting the target expression change triangular patch according to the reference expression change triangular patch comprises:
And adjusting the expression in the target expression change triangular facial mask in the same proportion according to the size and the expression of the reference expression change triangular facial mask.
5. The method of claim 1, wherein the input data comprises a single face image, a plurality of face images, or multiple frames of RGB-D face data.
6. A three-dimensional facial expression base establishment apparatus, comprising:
The estimating module is used for extracting target key points of a target face image in input data, and determining a camera projection matrix according to the target key points and initial key points corresponding to the target key points in an average face model, wherein the average face model is a reference three-dimensional face model obtained by processing according to a face model data set;
The calculation module is used for calculating a target three-dimensional face model corresponding to the target face image according to the target key points, the average face model reference three-dimensional face model and the initial key points by using the camera projection matrix;
The generation module is used for generating a target three-dimensional facial expression base corresponding to the target facial image by utilizing the target three-dimensional facial model;
The method for determining the camera projection matrix according to the target key points and the initial key points corresponding to the target key points in the average face model comprises the following steps:
determining mathematical relationships among the target keypoints, the initial keypoints and the camera projection matrix;
estimating the camera projection matrix by minimizing a re-projection error according to the mathematical relationship;
Calculating a target three-dimensional face model corresponding to the target face image according to the target key points, the average face model reference three-dimensional face model and the initial key points by using the camera projection matrix, wherein the method comprises the following steps of:
establishing a parameter relation between the target three-dimensional face model and the reference three-dimensional face model;
adjusting the parameter relation to enable the initial key point to coincide with the target key point after being projected by a camera projection matrix;
Calculating the target three-dimensional face model by utilizing the modified parameter relation and the reference three-dimensional face model;
generating a target three-dimensional facial expression base corresponding to the target facial image by using the target three-dimensional facial model, comprising:
acquiring a template expression base, and determining a reference expression change triangular face piece according to the reference three-dimensional face model and the template expression base;
selecting a target expression change triangular patch from the target three-dimensional face model according to the reference expression change triangular patch;
and adjusting the target expression change triangular surface patch according to the reference expression change triangular surface patch.
7. A computer-readable storage medium, on which a computer program is stored, characterized in that the program, when executed by a processor, implements the three-dimensional facial expression base establishment method according to any one of claims 1 to 5.
8. An electronic device, comprising:
A processor; and
A memory for storing one or more programs that, when executed by the one or more processors, cause the one or more processors to implement the three-dimensional facial expression base establishment method of any one of claims 1 to 5.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202010642367.9A CN111768477B (en) | 2020-07-06 | 2020-07-06 | Three-dimensional facial expression base establishment method and device, storage medium and electronic equipment |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202010642367.9A CN111768477B (en) | 2020-07-06 | 2020-07-06 | Three-dimensional facial expression base establishment method and device, storage medium and electronic equipment |
Publications (2)
| Publication Number | Publication Date |
|---|---|
| CN111768477A CN111768477A (en) | 2020-10-13 |
| CN111768477B true CN111768477B (en) | 2024-05-28 |
Family
ID=72723862
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN202010642367.9A Active CN111768477B (en) | 2020-07-06 | 2020-07-06 | Three-dimensional facial expression base establishment method and device, storage medium and electronic equipment |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN111768477B (en) |
Families Citing this family (10)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN112164143A (en) * | 2020-10-23 | 2021-01-01 | 广州小马慧行科技有限公司 | Three-dimensional model construction method and device, processor and electronic equipment |
| CN112257657B (en) * | 2020-11-11 | 2024-02-27 | 网易（杭州）网络有限公司 | Face image fusion method and device, storage medium and electronic equipment |
| CN112614213B (en) * | 2020-12-14 | 2024-01-23 | 杭州网易云音乐科技有限公司 | Facial expression determining method, expression parameter determining model, medium and equipment |
| CN112541477B (en) * | 2020-12-24 | 2024-05-31 | 北京百度网讯科技有限公司 | Expression pack generation method and device, electronic equipment and storage medium |
| CN112927328B (en) * | 2020-12-28 | 2023-09-01 | 北京百度网讯科技有限公司 | Expression migration method and device, electronic equipment and storage medium |
| CN112906494B (en) * | 2021-01-27 | 2022-03-08 | 浙江大学 | Face capturing method and device, electronic equipment and storage medium |
| CN113095134B (en) * | 2021-03-08 | 2024-03-29 | 北京达佳互联信息技术有限公司 | Facial expression extraction model generation method and device and facial image generation method and device |
| CN113920282B (en) * | 2021-11-15 | 2022-11-04 | 广州博冠信息科技有限公司 | Image processing method and device, computer readable storage medium, and electronic device |
| CN114170379A (en) * | 2021-11-30 | 2022-03-11 | 聚好看科技股份有限公司 | Three-dimensional model reconstruction method, device and equipment |
| WO2024113290A1 (en) * | 2022-12-01 | 2024-06-06 | 京东方科技集团股份有限公司 | Image processing method and apparatus, interactive device, electronic device and storage medium |
Citations (4)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN107369174A (en) * | 2017-07-26 | 2017-11-21 | 厦门美图之家科技有限公司 | The processing method and computing device of a kind of facial image |
| CN109389682A (en) * | 2017-08-09 | 2019-02-26 | 上海影子智能科技有限公司 | A kind of three-dimensional face model automatic adjusting method |
| CN111127668A (en) * | 2019-12-26 | 2020-05-08 | 网易（杭州）网络有限公司 | Role model generation method and device, electronic equipment and storage medium |
| CN111274944A (en) * | 2020-01-19 | 2020-06-12 | 中北大学 | Three-dimensional face reconstruction method based on single image |
-
2020
- 2020-07-06 CN CN202010642367.9A patent/CN111768477B/en active Active
Patent Citations (4)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN107369174A (en) * | 2017-07-26 | 2017-11-21 | 厦门美图之家科技有限公司 | The processing method and computing device of a kind of facial image |
| CN109389682A (en) * | 2017-08-09 | 2019-02-26 | 上海影子智能科技有限公司 | A kind of three-dimensional face model automatic adjusting method |
| CN111127668A (en) * | 2019-12-26 | 2020-05-08 | 网易（杭州）网络有限公司 | Role model generation method and device, electronic equipment and storage medium |
| CN111274944A (en) * | 2020-01-19 | 2020-06-12 | 中北大学 | Three-dimensional face reconstruction method based on single image |
Non-Patent Citations (3)
| Title |
|---|
| 《3dmm近期论文综述》;网友;《https://www.jianshu.com/p/5d933f81d5dd》;全文 * |
| 《Bundle Adjustment---即最小化重投影误差（高翔 slam---第七讲）》;网友;《https://www.cnblogs.com/Jessica-jie/p/7739775.html》;全文 * |
| 《基于形变模型的人脸三维重构与表情传递的研究》;王娟;《中国优秀硕士学位论文全文数据库》;第21-25页 * |
Also Published As
| Publication number | Publication date |
|---|---|
| CN111768477A (en) | 2020-10-13 |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| CN111768477B (en) | Three-dimensional facial expression base establishment method and device, storage medium and electronic equipment | |
| WO2020108610A1 (en) | Image processing method, apparatus, computer readable medium and electronic device | |
| CN112257657B (en) | Face image fusion method and device, storage medium and electronic equipment | |
| CN113327278B (en) | Three-dimensional face reconstruction method, device, equipment and storage medium | |
| CN114820905B (en) | Virtual image generation method and device, electronic equipment and readable storage medium | |
| CN115409933B (en) | Multi-style texture mapping generation method and device | |
| JP2024004444A (en) | Three-dimensional face reconstruction model training, three-dimensional face image generation method, and device | |
| CN115147265B (en) | Avatar generation method, apparatus, electronic device, and storage medium | |
| CN113313832B (en) | Semantic generation method and device of three-dimensional model, storage medium and electronic equipment | |
| CN113409454B (en) | Face image processing method and device, electronic equipment and storage medium | |
| CN111754431B (en) | Image area replacement method, device, equipment and storage medium | |
| CN114708374A (en) | Virtual image generation method and device, electronic equipment and storage medium | |
| CN116563493A (en) | Model training method based on three-dimensional reconstruction, three-dimensional reconstruction method and device | |
| KR20230028253A (en) | Face image processing method, face image processing model training method, device, device, storage medium and program product | |
| CN113140034A (en) | Room layout-based panoramic new view generation method, device, equipment and medium | |
| CN113658035A (en) | Face transformation method, device, equipment, storage medium and product | |
| CN110751026B (en) | Video processing method and related device | |
| CN110378948B (en) | 3D model reconstruction method and device and electronic equipment | |
| CN116229548A (en) | Model generation method and device, electronic equipment and storage medium | |
| CN115775300A (en) | Reconstruction method of human body model, training method and device of human body reconstruction model | |
| CN113570634B (en) | Object three-dimensional reconstruction method, device, electronic equipment and storage medium | |
| CN113781653A (en) | Object model generation method and device, electronic equipment and storage medium | |
| CN116030150B (en) | Avatar generation method, device, electronic equipment and medium | |
| CN114820908B (en) | Virtual image generation method and device, electronic equipment and storage medium | |
| CN113610992B (en) | Bone driving coefficient determining method and device, electronic equipment and readable storage medium |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination | ||
| GR01 | Patent grant | ||
| GR01 | Patent grant |