CN116433812A - Method and device for generating virtual character by using 2D face picture - Google Patents
Method and device for generating virtual character by using 2D face picture Download PDFInfo
- Publication number
- CN116433812A CN116433812A CN202310671947.4A CN202310671947A CN116433812A CN 116433812 A CN116433812 A CN 116433812A CN 202310671947 A CN202310671947 A CN 202310671947A CN 116433812 A CN116433812 A CN 116433812A
- Authority
- CN
- China
- Prior art keywords
- face
- model
- head
- dimensional model
- loss
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Granted
Links
- 238000000034 method Methods 0.000 title claims abstract description 59
- 238000013527 convolutional neural network Methods 0.000 claims abstract description 36
- 238000012545 processing Methods 0.000 claims abstract description 17
- 238000003860 storage Methods 0.000 claims abstract description 17
- 238000006243 chemical reaction Methods 0.000 claims abstract description 12
- 238000005286 illumination Methods 0.000 claims abstract description 12
- 230000006870 function Effects 0.000 claims description 31
- 230000001815 facial effect Effects 0.000 claims description 25
- 230000005489 elastic deformation Effects 0.000 claims description 20
- 238000004422 calculation algorithm Methods 0.000 claims description 19
- 238000012549 training Methods 0.000 claims description 19
- 238000013507 mapping Methods 0.000 claims description 17
- 230000008447 perception Effects 0.000 claims description 10
- 238000004590 computer program Methods 0.000 claims description 6
- 238000004891 communication Methods 0.000 claims description 5
- 238000009499 grossing Methods 0.000 claims description 5
- 238000004148 unit process Methods 0.000 claims description 2
- 230000000694 effects Effects 0.000 abstract description 14
- 239000013598 vector Substances 0.000 description 18
- 238000004898 kneading Methods 0.000 description 11
- 230000009466 transformation Effects 0.000 description 11
- PXFBZOLANLWPMH-UHFFFAOYSA-N 16-Epiaffinine Natural products C1C(C2=CC=CC=C2N2)=C2C(=O)CC2C(=CC)CN(C)C1C2CO PXFBZOLANLWPMH-UHFFFAOYSA-N 0.000 description 8
- 230000008569 process Effects 0.000 description 7
- 238000010586 diagram Methods 0.000 description 6
- 230000015654 memory Effects 0.000 description 6
- 230000014509 gene expression Effects 0.000 description 5
- 230000008878 coupling Effects 0.000 description 3
- 238000010168 coupling process Methods 0.000 description 3
- 238000005859 coupling reaction Methods 0.000 description 3
- 238000013461 design Methods 0.000 description 3
- 238000004519 manufacturing process Methods 0.000 description 3
- 239000004973 liquid crystal related substance Substances 0.000 description 2
- 239000011159 matrix material Substances 0.000 description 2
- 238000005457 optimization Methods 0.000 description 2
- 238000000513 principal component analysis Methods 0.000 description 2
- 239000000047 product Substances 0.000 description 2
- 238000006467 substitution reaction Methods 0.000 description 2
- 238000012546 transfer Methods 0.000 description 2
- 238000004458 analytical method Methods 0.000 description 1
- 238000013459 approach Methods 0.000 description 1
- 238000004364 calculation method Methods 0.000 description 1
- 230000015556 catabolic process Effects 0.000 description 1
- 239000003086 colorant Substances 0.000 description 1
- 238000006731 degradation reaction Methods 0.000 description 1
- 238000009795 derivation Methods 0.000 description 1
- 210000002569 neuron Anatomy 0.000 description 1
- 230000003287 optical effect Effects 0.000 description 1
- 230000001373 regressive effect Effects 0.000 description 1
- 239000013589 supplement Substances 0.000 description 1
- 238000013519 translation Methods 0.000 description 1
Images
Classifications
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
- G06T13/00—Animation
- G06T13/20—3D [Three Dimensional] animation
- G06T13/40—3D [Three Dimensional] animation of characters, e.g. humans, animals or virtual beings
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06N—COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS
- G06N3/00—Computing arrangements based on biological models
- G06N3/02—Neural networks
- G06N3/04—Architecture, e.g. interconnection topology
- G06N3/0464—Convolutional networks [CNN, ConvNet]
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
- G06T3/00—Geometric image transformations in the plane of the image
- G06T3/08—Projecting images onto non-planar surfaces, e.g. geodetic screens
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
- G06T7/00—Image analysis
- G06T7/30—Determination of transform parameters for the alignment of images, i.e. image registration
- G06T7/33—Determination of transform parameters for the alignment of images, i.e. image registration using feature-based methods
- G06T7/344—Determination of transform parameters for the alignment of images, i.e. image registration using feature-based methods involving models
-
- Y—GENERAL TAGGING OF NEW TECHNOLOGICAL DEVELOPMENTS; GENERAL TAGGING OF CROSS-SECTIONAL TECHNOLOGIES SPANNING OVER SEVERAL SECTIONS OF THE IPC; TECHNICAL SUBJECTS COVERED BY FORMER USPC CROSS-REFERENCE ART COLLECTIONS [XRACs] AND DIGESTS
- Y02—TECHNOLOGIES OR APPLICATIONS FOR MITIGATION OR ADAPTATION AGAINST CLIMATE CHANGE
- Y02D—CLIMATE CHANGE MITIGATION TECHNOLOGIES IN INFORMATION AND COMMUNICATION TECHNOLOGIES [ICT], I.E. INFORMATION AND COMMUNICATION TECHNOLOGIES AIMING AT THE REDUCTION OF THEIR OWN ENERGY USE
- Y02D10/00—Energy efficient computing, e.g. low power processors, power management or thermal management
Landscapes
- Engineering & Computer Science (AREA)
- Physics & Mathematics (AREA)
- Theoretical Computer Science (AREA)
- General Physics & Mathematics (AREA)
- Computational Linguistics (AREA)
- Data Mining & Analysis (AREA)
- Life Sciences & Earth Sciences (AREA)
- Artificial Intelligence (AREA)
- Biomedical Technology (AREA)
- Biophysics (AREA)
- Computer Vision & Pattern Recognition (AREA)
- Health & Medical Sciences (AREA)
- Evolutionary Computation (AREA)
- General Health & Medical Sciences (AREA)
- Molecular Biology (AREA)
- Computing Systems (AREA)
- General Engineering & Computer Science (AREA)
- Mathematical Physics (AREA)
- Software Systems (AREA)
- Processing Or Creating Images (AREA)
Abstract
The application provides a method and a device for generating virtual characters by using 2D face pictures, electronic equipment and a storage medium, wherein the method comprises the following steps: acquiring a 2D face picture to be processed, and processing the 2D face picture by using a trained convolutional neural network to obtain a 3DMM face model, wherein the convolutional neural network comprises a face model, an illumination model and a camera model; the 3DMM face model is mapped to a three-dimensional model of a human body pinching face head of the virtual character through a shape conversion method to obtain the virtual character, and the head model of the virtual character which is highly similar to the face in the single 2D face picture can be generated according to the single 2D face picture, so that the efficiency and the effect of virtual character creation can be improved.
Description
Technical Field
The present invention relates to the field of 3D modeling, and in particular, to a method and apparatus for generating a virtual character using 2D face images, an electronic device, and a storage medium.
Background
In recent years, there is an increasing demand for digital virtualization of real objects, figures, and the like, and in particular, it is a focus problem how to quickly construct virtualized figures. Traditional methods use 3d modeling software, such as maya, etc., requiring professional artwork to take weeks to create a virtual character.
One of the main current methods at the present stage is to provide a basic character prime body model, and on the basis, the prime body model is adjusted by using adjustable facial and body parameters, so that quick modeling is realized, and a virtual character is created. Compared with the traditional modeling method, the method can save a large amount of time and cost, but in order to obtain a good virtual character effect, not only a large amount of pinching face and body type adjustable parameters are required to be provided, but also the ideal effect can be achieved by fine adjustment of all parameters by an art staff, so that a large amount of time and cost are still required to be input, and the virtual character can not be rapidly produced in batches. As avatars develop, the need for general users to customize the avatar increases. How to enable a common user without an art foundation to quickly create an ideal virtual character becomes a technical problem to be solved.
Disclosure of Invention
In view of this, the embodiments of the present application provide a method and apparatus for generating a virtual character using 2D face pictures, an electronic device, and a storage medium, which can generate a head model of the virtual character that is highly similar to a face in a single 2D face picture from the single 2D face picture, so as to improve efficiency and effect of virtual character creation.
In a first aspect, an embodiment of the present application provides a method for generating a virtual character by using a 2D face picture, including:
acquiring a 2D face picture to be processed, and processing the 2D face picture by using a trained convolutional neural network to obtain a 3DMM face model, wherein the convolutional neural network comprises a face model, an illumination model and a camera model;
and mapping the 3DMM face model to a three-dimensional model of the pixel pinching face head of the virtual character through a shape conversion method to obtain the virtual character.
In a second aspect, an embodiment of the present application further provides an apparatus for generating a virtual character using a 2D face picture, including:
the processing unit is used for acquiring a 2D face picture to be processed, and processing the 2D face picture by using a trained convolutional neural network to obtain a 3DMM face model, wherein the convolutional neural network comprises a face model, an illumination model and a camera model;
and the mapping unit is used for mapping the 3DMM face model to a three-dimensional model of the pixel pinching face head of the virtual character through a shape conversion method to obtain the virtual character.
In a third aspect, embodiments of the present application further provide a computer readable storage medium having stored thereon a computer program which, when executed by a processor, performs the steps of the method of generating a virtual character using 2D face pictures as described in the first aspect.
In a fourth aspect, embodiments of the present application further provide an electronic device, including: a processor, a storage medium and a bus, the storage medium storing machine-readable instructions executable by the processor, the processor in communication with the storage medium via the bus when the electronic device is running, the processor executing the machine-readable instructions to perform the steps of the method of generating virtual characters using 2D face pictures as described in the first aspect.
In summary, the method, the device, the electronic equipment and the storage medium for generating the virtual character by using the 2D face picture provided by the embodiment of the application use the given 2D face picture, the 3D face reconstruction can be realized by processing the 2D face picture by using the convolutional neural network, and the face model is applied to the pinching face character prime body model after the reconstruction is finished. On the other hand, any virtual character can be easily customized, the art design, original drawing and the like of the character do not need to be performed from the beginning, even if hundreds of parameters exist, the ideal creation effect can be achieved without manually adjusting each parameter, and the time for manufacturing a large number of virtual characters can be saved, so that quick batch virtual character generation can be performed, namely, the head model of the virtual character which is similar to the face in the single 2D face picture in height can be generated according to the single 2D face picture, and the efficiency and the effect of virtual character creation can be improved.
Drawings
Fig. 1 is a flowchart of a method for generating a virtual character by using a 2D face picture according to an embodiment of the present application;
fig. 2 is a schematic structural diagram of an apparatus for generating a virtual character by using 2D face pictures according to an embodiment of the present application;
fig. 3 is a schematic structural diagram of an electronic device according to an embodiment of the present application.
Detailed Description
For the purpose of making the objects, technical solutions and advantages of the embodiments of the present application more clear, the technical solutions of the embodiments of the present application will be clearly and completely described below with reference to the accompanying drawings in the embodiments of the present application, and it should be understood that the accompanying drawings in the present application are only for the purpose of illustration and description, and are not intended to limit the protection scope of the present application. In addition, it should be understood that the schematic drawings are not drawn to scale. A flowchart, as used in this application, illustrates operations implemented according to some embodiments of the present application. It should be understood that the operations of the flow diagrams may be implemented out of order and that steps without logical context may be performed in reverse order or concurrently. Moreover, one or more other operations may be added to the flow diagrams and one or more operations may be removed from the flow diagrams as directed by those skilled in the art.
In addition, the described embodiments are only some, but not all, of the embodiments of the present application. The components of the embodiments of the present application, which are generally described and illustrated in the figures herein, may be arranged and designed in a wide variety of different configurations. Thus, the following detailed description of the embodiments of the present application, as provided in the accompanying drawings, is not intended to limit the scope of the application, as claimed, but is merely representative of selected embodiments of the application. All other embodiments, which can be made by those skilled in the art based on the embodiments of the present application without making any inventive effort, are intended to be within the scope of the present application.
It should be noted that the term "comprising" will be used in the embodiments of the present application to indicate the presence of the features stated hereinafter, but not to exclude the addition of other features.
Referring to fig. 1, a method for generating a virtual character by using a 2D face picture according to an embodiment of the present application includes:
s10, acquiring a 2D face picture to be processed, and processing the 2D face picture by using a trained convolutional neural network to obtain a 3DMM face model, wherein the convolutional neural network comprises a face model, an illumination model and a camera model;
in this embodiment, it should be noted that, a core idea of the 3DMM (3D Morphable models, three-dimensional deformation model) face model is that faces can be matched one by one in a three-dimensional space, and can be obtained by weighting and linear addition of many other faces in an orthogonal basis. Each three-dimensional face can be represented in a base vector space formed by all faces in a database, namely, coefficients of all the base vectors are solved, so that a model of any three-dimensional face can be obtained. Basic attributes of faces include shapes and textures, each face may be represented as a linear superposition of a shape vector S (which is a vector composed of x-axis coordinates, y-axis coordinates, and z-axis coordinates of each vertex of a model) and a texture vector T (which is a vector composed of R-value, G-value, and B-value of each vertex of the model, RGB being a color space), where the shape vector S may be determined by feature parameters, the texture vector T may be determined by texture parameters, and any three-dimensional face model may be acquired by modifying the feature parameters and the texture parameters. In order to achieve a better 3D Face reconstruction effect, the method adds expression parameters, gesture parameters and illumination parameters as expansion parameters on the basis of the 3DMM basic parameters of feature parameters and texture parameters to optimize the generation effect, specifically, s=s0+bid×α+bexp× β, t=t0+bt×δ, wherein S0 and T0 respectively represent the mean value of the shape vectors and the mean value of the texture vectors of all three-dimensional Face models in a database, bid, bexp and Bt respectively represent PCA (Principal Component Analysis, principal component transformation) cardinality of identities, expressions and textures, and all undergo standardized processing, and a BFM2009 Face model can be used to represent S0, bid, T0 and Bt, and the expression cardinality established from a Face-Warehouse dataset is used to represent Bexp. Alpha, beta and delta are corresponding coefficient vectors for generating a three-dimensional Face and are parameters of a Face model, wherein alpha and delta are characteristic parameters and texture parameters of the Face respectively, a BFM2009 model can be referred to, first 80-dimensional data (original dimension is 199) can be taken, beta is an expression parameter, and first 64-dimensional data given by a Face Warehouse data set can be taken.
The illumination model may select Spherical Harmonics, and the corresponding illumination parameter γ contains 9 parameters.
The three-dimensional model is projected onto a two-dimensional plane and subjected to MVP transformation (namely model transformation), and faces with different angles can be shot at different positions of the camera, so that the camera can use a perspective camera model for 3D-2D projection geometry. Correspondingly, there are 6 gesture parameters η, including a rotation parameter and a translation parameter (3 each).
To this end, the convolutional neural network (R-Net network or C-Net network may be used, for example, resNet-50 network) has 80+80+64+9+6=239 parameters (including 80 feature parameters, 80 texture parameters, 64 expression parameters, 9 illumination parameters, and 6 pose parameters) to be regressed.
In summary, the unknown parameters to be predicted may be represented by vectors (α, β, δ, γ, η). The method comprises the steps of using a ResNet-50 network, modifying the last full-connection layer into 239 neurons, returning the coefficients, using an unsupervised/weakly supervised mode, not needing to give 239 parameters as label labels, calculating a reconstructed picture I' of a 3D face projected onto a phase plane in a computer graphics mode through face model parameters, illumination model parameters and camera model parameters in the network, finally, jointly constraining a training network through various losses, and finally, calculating and obtaining a 3D face reconstruction model.
And S11, mapping the 3DMM face model to a three-dimensional model of the pixel pinching face head of the virtual character through a shape conversion method to obtain the virtual character.
The three-dimensional model of the human face of the virtual character is processed, the other parts are not processed, the 3DMM human face model is mapped to the three-dimensional model of the human face of the virtual character by a shape conversion method, and matching the three-dimensional model of the element pinching face head with the 3DMM face model, wherein the three-dimensional model of the element pinching face head is the head model of the virtual character.
According to the method for generating the virtual character by using the 2D face picture, the given 2D face picture is used, the 2D face picture is processed by using the convolutional neural network, 3D face reconstruction can be achieved, and the face model is applied to the pinching face character body model after the reconstruction is finished. On the other hand, any virtual character can be easily customized, the art design, original drawing and the like of the character do not need to be performed from the beginning, even if hundreds of parameters exist, the ideal creation effect can be achieved without manually adjusting each parameter, and the time for manufacturing a large number of virtual characters can be saved, so that quick batch virtual character generation can be performed, namely, the head model of the virtual character which is similar to the face in the single 2D face picture in height can be generated according to the single 2D face picture, and the efficiency and the effect of virtual character creation can be improved.
On the basis of the foregoing method embodiment, before the processing the 2D face image by using the trained convolutional neural network to obtain the 3DMM face model, the method may further include:
training the convolutional neural network, wherein a sample used for training the convolutional neural network comprises a plurality of 2D face pictures and projection reconstruction pictures corresponding to the 2D face pictures, a used loss function comprises picture layer loss, perception layer loss, regular loss and texture loss, the picture layer loss comprises pixel loss and face recognition point loss, and the loss function is a weighted summation result of the pixel loss, the face recognition point loss, the perception layer loss, the regular loss and the texture loss.
In this embodiment, it should be noted that the training of the convolutional neural network may use a given training RGB image I, and use regression coefficient vectors in the network architecture to generate a reconstructed image I' through some simple and microminiatable mathematical derivation analysis. The convolutional neural network is trained without any real label, and the training is finally completed by evaluating the loss of I', so that the 3D face reconstruction is realized.
It was found in the reconstruction that the low level information using pixel level colors may be affected by local minima problems, in which case low errors will lead to unsatisfactory facial shape results. On the other hand, using only perceptual loss also leads to sub-optimal results, as it ignores the consistency of the pixel level with the original image signal. Whereby if a mixed loss function is used, the function integrates both, so that more accurate results can be obtained. Thus the hybrid loss of convolutional neural networks includes two major parts: loss of image level loss and loss of perception level loss.
Wherein the image level loss includes a pixel loss Lphoto and a face recognition point loss Llan,
where i is a pixel index, M is a face region (the position of the face skin region can be obtained by using a model of the human face segment), A i For training image coefficient coverage mask based on skin color, using skin coverage coefficient, only the skin position of the face will be subjected to loss calculation, and robustness to face shielding and appearance changes (such as beard and make-up) can be improved. I i Is the RGB value of the pixel point with index I in the face area of the original image, I i ' RGB values of pixel points with index i in the face area of the reconstructed picture (the picture generated by projecting the original picture onto the original picture after the training network processing),
representing a 2-norm;
n1 is the number of face feature points, n1=68, q n Is the coordinates of the nth feature point in the original image, q n ' coordinates, ω of nth feature point in reconstructed picture n To correspond to the weight of the nth feature point, the weights of the other feature points are 1 except for the feature points at the corners of the nose and mouth, which are 20.
Using low-level information to measure image differences can produce good results, but using only the above-described loss at the image level alone can lead to local optimization problems for 3D face reconstruction based on convolutional neural networks CNN. To solve the above problem, loss of perceptual layer is introduced to further supplement training. Specifically, the original image I and the reconstructed image I 'are respectively input into a face recognition network, and two feature vectors f (I) and f (I') are obtained, wherein the more similar the two feature vectors are, the smaller loss Lper of the perception layer is, and the specific definition is as follows:
wherein, the liquid crystal display device comprises a liquid crystal display device,<f(I)，f(I′)>represents the vector inner product of the feature vectors f (I) and f (I'),
representing vector modulo.
To prevent facial shape and texture degradation, over-fitting occurs, a regular constraint is used on the regressive 3DMM coefficients, and the regular loss Lcoef is specifically defined as follows:
ωα、ω β and omega δ To balance the weights, ωα=1, ω can be empirically set β =0.8，
ω δ =1.7×10 -3 。
Finally, adding all of the above losses together, a mixed loss of ResNet-50 can be obtained for the final training.
In summary, the ResNet-50 loss function consists of two image level losses Lphoto and Llan, one perceptual loss Lper and one canonical loss Lcoef, weighted wphoto, wlan, wper and wcoef, respectively. The wpoto=1.9 and wlan=1.6x10 can be set as required -3 ，wper=0.2，wcoef=3×10 -4 。
According to the scheme, the CNN-based single-image face reconstruction method can be realized, the mixed-level image information is utilized to perform weak supervision learning, real 3D face data is not needed to be used as a training label, and the 3D face reconstruction effect can be improved.
On the basis of the foregoing method embodiment, the mapping the 3DMM face model to a three-dimensional model of a pinching face head of a virtual character through a shape conversion method to obtain the virtual character may include:
extracting characteristic points of the 3DMM face model, carrying out elastic deformation on the three-dimensional model of the prime body pinching face head of the virtual character by utilizing the characteristic points based on a radial basis function, and registering the 3DMM face model and the three-dimensional model of the prime body pinching face head after elastic deformation by utilizing a non-rigid ICP registration algorithm to obtain the virtual character.
On the basis of the foregoing method embodiment, the extracting the feature points of the 3DMM face model, based on a radial basis function, and using the feature points to elastically deform the three-dimensional model of the element pinching face head of the virtual character, and using a non-rigid ICP registration algorithm to register the 3DMM face model and the elastically deformed three-dimensional model of the element pinching face head, to obtain the virtual character may include:
extracting facial feature points of the 3DMM face model, carrying out elastic deformation on a three-dimensional model of a pixel pinching face head of the virtual character by utilizing the facial feature points based on a radial basis function to obtain a first three-dimensional model of the head;
using the first head three-dimensional model as a face template, and registering the 3DMM face model and the first head three-dimensional model through a non-rigid ICP registration algorithm to obtain a second head three-dimensional model;
extracting facial contour feature points of the 3DMM face model, and carrying out elastic deformation on the second head three-dimensional model by utilizing the facial contour feature points based on a radial basis function to obtain a third head three-dimensional model;
the third head three-dimensional model is used as a face template, and the 3DMM face model and the third head three-dimensional model are registered through a non-rigid ICP registration algorithm to obtain a fourth head three-dimensional model;
and carrying out smoothing treatment on the fourth head three-dimensional model to obtain the virtual character.
In this embodiment, the shape conversion method mapping includes the steps of:
(1) The face feature points are utilized to elastically deform the element body based on the radial basis function to pinch the three-dimensional model of the face head, so that the element body is more similar to the 3DMM reconstructed face model, and then three-dimensional registration is carried out, so that the searching process of the corresponding points is accelerated. The elastic deformation formula is as follows:
Pi=（x i ，y i ，z i ) The coordinates of the ith vertex on the three-dimensional model of the pixel body kneading head, pin is the coordinates of the ith vertex on the three-dimensional model of the pixel body kneading head after elastic deformation, F () represents elastic deformation,
n2 is the number of vertexes, eta of the three-dimensional model of the element kneading face head j For deformation parameters a, b and c are affine transformation parameters, +.>
And l is a positive number as a radial basis function. The corresponding relation between the characteristic point set of the three-dimensional model of the element pinching face head and the characteristic point set of the 3DMM reconstruction face can be utilized for carrying out parameter solving, and the requirements are that:
k is the number of three-dimensional facial feature points, fjs is the coordinate of the jth facial feature point on the three-dimensional model of the pixel body kneading head, fjt is the coordinate of the jth facial feature point on the 3DMM reconstruction face, and therefore the above-mentioned elastic deformation pixel body kneading head three-dimensional model can be utilized. And carrying out non-rigid ICP registration by taking the deformed three-dimensional model of the element pinching face head as a face template.
(2) And matching the three-dimensional model of the face head of the plain body kneading and the face model reconstructed by the 3DMM by using a non-rigid ICP registration algorithm.
The non-rigid ICP registration deforms the three-dimensional model face of the pixel body pinching face head by carrying out different affine transformation on each vertex pi on the three-dimensional model face V of the pixel body pinching face head, so that V is deformed into a three-dimensional model face V ' of the pixel body pinching face head, the closer and the better the V ' is to the shape of the target 3DMM reconstructed face P, namely the closest point of each vertex P on the V ' in P is. The process objective function is as follows:
wherein: e (E) d For data item errors, E s To smooth term errors, E f For characteristic point registration errors, sigma, ζ and v are weighting parameters, E d The similarity degree between the three-dimensional model face of the head of the currently deformed plain kneading face and the corresponding vertex of the target 3DMM reconstructed face is represented, and the similarity degree is defined as follows:
wherein: d (D) i X is the nearest point on P to pi i An affine transformation matrix for vertex pi, θ i Error weight for the ith vertex: if pi cannot find the corresponding point on P (the distance between pi and the nearest point exceeds a certain threshold value), then theta is calculated i Set to 0, otherwise, will θ i And (3) setting 1. In order to accelerate the nearest point searching process, the characteristic points on the face can be reconstructed by using the target 3DMM to divide the characteristic points into different areas, and then the nearest point is searched in the corresponding area.
E s The consistency of affine transformation parameter matrixes of adjacent vertexes in the deformation process is characterized, and the affine transformation parameter matrixes are defined as follows:
wherein: (pi, pj) is a segment connecting pi and pj, X j The edge (V) is a set of all sides in the three-dimensional model face V of the pixel pinching face head and is an affine transformation matrix of the vertex pj.
E f And (3) representing the distance between the three-dimensional model face of the deformed element pinching face head and the corresponding characteristic point of the target 3DMM reconstructed face, and guiding the deformation process in the initial iteration stage. The labels of the characteristic points (including the eye point, nose wing point, nose point, mouth point and ear characteristic point) are marked as K l （l=1,2，…，17），E f The definition is as follows:
the three-dimensional face registration algorithm based on the non-rigid ICP comprises the following steps:
1) Initializing X i 0 （i=1,2，…，N2），X i 0 X represents i T=0 (t represents a round, the initial value is 0, the superscript t of the following parameter represents a round, and the description will not be repeated later);
2) For a fixed set of optimization parameters sigma, ζ and v,
(1) for V t (t is calculated from 1) each vertex pi on t Find its closest point on P as its corresponding point D i t ；
(2) Calculate X i t Minimizing error E;
(3) updating the shape of the template face:
(4) repeating (1) - (3) until
3) Increase sigma and xi, reduce v, repeat step 2).
(3) Extracting facial contour points on the 3DMM reconstructed three-dimensional face, and obtaining facial contour points on the three-dimensional model of the plain pinching face head by using the registration result. Repeating the steps (1) and (2), and changing the facial contour points into the element body kneading face head three-dimensional model based on the radial basis function again.
(4) And deforming the three-dimensional model of the head of the plain kneading face to be matched with the 3DMM reconstructed face by using a non-rigid ICP registration algorithm.
(5) And removing joints from the three-dimensional model of the head of the plain kneading face by using a three-dimensional smoothing algorithm.
In summary, the 3DMM reconstructed face model is mapped to the three-dimensional model of the pixel pinching face head through shape conversion shape transfer, so that the three-dimensional model of the pixel pinching face head is matched with the 3DMM reconstructed face model. And then, defining different affine transformation matrixes for each vertex through non-rigid ICP three-dimensional face data registration to deform the face of the three-dimensional model of the face head of the element body kneading. The algorithm iteratively optimizes all affine transformation matrixes, so that the three-dimensional model face of the pixel body pinching face head gradually approaches to the 3DMM reconstruction face, and the deformed three-dimensional model face of the pixel body pinching face head has higher similarity with the 3DMM reconstruction face. Finally, the shape transfer mapping from the 3D reconstructed face model to the three-dimensional model of the plain pinching face head is realized.
Referring to fig. 2, a schematic structural diagram of an apparatus for generating a virtual character by using a 2D face picture according to an embodiment of the present application is shown, where the apparatus includes:
the processing unit 20 is configured to obtain a 2D face picture to be processed, and process the 2D face picture by using a trained convolutional neural network to obtain a 3DMM face model, where the convolutional neural network includes a face model, an illumination model, and a camera model;
and a mapping unit 21, configured to obtain the virtual character by mapping the 3DMM face model to a three-dimensional model of a pixel pinching face head of the virtual character through a shape conversion method.
According to the device for generating the virtual character by using the 2D face picture, the given 2D face picture is used, the 2D face picture is processed by using the convolutional neural network, so that 3D face reconstruction can be achieved, and the face model is applied to the pinching face character body model after the reconstruction is finished. On the other hand, any virtual character can be easily customized, the art design, original drawing and the like of the character do not need to be performed from the beginning, even if hundreds of parameters exist, the ideal creation effect can be achieved without manually adjusting each parameter, and the time for manufacturing a large number of virtual characters can be saved, so that quick batch virtual character generation can be performed, namely, the head model of the virtual character which is similar to the face in the single 2D face picture in height can be generated according to the single 2D face picture, and the efficiency and the effect of virtual character creation can be improved.
On the basis of the foregoing apparatus embodiment, the apparatus may further include:
the training unit is used for training the convolutional neural network before the processing unit processes the 2D face pictures by using the trained convolutional neural network to obtain a 3DMM face model, wherein a sample used for training the convolutional neural network comprises a plurality of 2D face pictures and projection reconstruction pictures corresponding to the 2D face pictures, a used loss function comprises picture layer loss, perception layer loss, regular loss and texture loss, the picture layer loss comprises pixel loss and face recognition point loss, and the loss function is a weighted summation result of the pixel loss, the face recognition point loss, the perception layer loss, the regular loss and the texture loss.
On the basis of the foregoing apparatus embodiment, the mapping unit may be configured to:
extracting characteristic points of the 3DMM face model, carrying out elastic deformation on the three-dimensional model of the prime body pinching face head of the virtual character by utilizing the characteristic points based on a radial basis function, and registering the 3DMM face model and the three-dimensional model of the prime body pinching face head after elastic deformation by utilizing a non-rigid ICP registration algorithm to obtain the virtual character.
On the basis of the foregoing apparatus embodiment, the mapping unit may be configured to:
extracting facial feature points of the 3DMM face model, carrying out elastic deformation on a three-dimensional model of a pixel pinching face head of the virtual character by utilizing the facial feature points based on a radial basis function to obtain a first three-dimensional model of the head;
using the first head three-dimensional model as a face template, and registering the 3DMM face model and the first head three-dimensional model through a non-rigid ICP registration algorithm to obtain a second head three-dimensional model;
extracting facial contour feature points of the 3DMM face model, and carrying out elastic deformation on the second head three-dimensional model by utilizing the facial contour feature points based on a radial basis function to obtain a third head three-dimensional model;
the third head three-dimensional model is used as a face template, and the 3DMM face model and the third head three-dimensional model are registered through a non-rigid ICP registration algorithm to obtain a fourth head three-dimensional model;
and carrying out smoothing treatment on the fourth head three-dimensional model to obtain the virtual character.
The implementation process of the device for generating the virtual character by using the 2D face picture provided by the embodiment of the application is consistent with the method for generating the virtual character by using the 2D face picture provided by the embodiment of the application, and the effect achieved by the device is the same as the method for generating the virtual character by using the 2D face picture provided by the embodiment of the application, and is not repeated here.
In summary, the scheme can truly restore characters in the pictures by reconstructing the 3D face of a single face picture, map the reconstructed face model to a three-dimensional model of a plain pinching face head, and rapidly generate a pinching face model.
As shown in fig. 3, an electronic device provided in an embodiment of the present application includes: the system comprises a processor 30, a memory 31 and a bus 32, wherein the memory 31 stores machine-readable instructions executable by the processor 30, the processor 30 and the memory 31 communicate through the bus 32 when the electronic device is running, and the processor 30 executes the machine-readable instructions to perform the steps of the method for generating virtual figures using 2D face pictures as described above.
Specifically, the above-described memory 31 and processor 30 can be general-purpose memories and processors, and are not particularly limited herein, and the above-described method of generating virtual characters using 2D face pictures can be performed when the processor 30 runs a computer program stored in the memory 31.
Corresponding to the method for generating the virtual character by using the 2D face picture, the embodiment of the application also provides a computer readable storage medium, wherein a computer program is stored on the computer readable storage medium, and the computer program is executed by a processor to execute the steps of the method for generating the virtual character by using the 2D face picture.
It will be clearly understood by those skilled in the art that, for convenience and brevity of description, specific working procedures of the above-described system and apparatus may refer to corresponding procedures in the method embodiments, which are not described in detail in this application. In the several embodiments provided in this application, it should be understood that the disclosed systems, devices, and methods may be implemented in other manners. The above-described apparatus embodiments are merely illustrative, and the division of the modules is merely a logical function division, and there may be additional divisions when actually implemented, and for example, multiple modules or components may be combined or integrated into another system, or some features may be omitted or not performed. Alternatively, the coupling or direct coupling or communication connection shown or discussed with each other may be through some communication interface, indirect coupling or communication connection of devices or modules, electrical, mechanical, or other form.
The modules described as separate components may or may not be physically separate, and components shown as modules may or may not be physical units, may be located in one place, or may be distributed over multiple network units. Some or all of the units may be selected according to actual needs to achieve the purpose of the solution of this embodiment.
In addition, each functional unit in each embodiment of the present application may be integrated in one processing unit, or each unit may exist alone physically, or two or more units may be integrated in one unit.
The functions, if implemented in the form of software functional units and sold or used as a stand-alone product, may be stored in a non-volatile computer readable storage medium executable by a processor. Based on such understanding, the technical solution of the present application may be embodied essentially or in a part contributing to the prior art or in a part of the technical solution, in the form of a software product stored in a storage medium, including several instructions for causing a computer device (which may be a personal computer, a server, or a network device, etc.) to perform all or part of the steps of the methods described in the embodiments of the present application. And the aforementioned storage medium includes: a usb disk, a removable hard disk, a ROM, a RAM, a magnetic disk, or an optical disk, etc.
The foregoing is merely a specific embodiment of the present application, but the protection scope of the present application is not limited thereto, and any person skilled in the art can easily think about changes or substitutions within the technical scope of the present application, and the changes or substitutions are covered in the protection scope of the present application. Therefore, the protection scope of the present application shall be subject to the protection scope of the claims.
Claims (10)
1. A method of generating a virtual character using 2D face pictures, comprising:
acquiring a 2D face picture to be processed, and processing the 2D face picture by using a trained convolutional neural network to obtain a 3DMM face model, wherein the convolutional neural network comprises a face model, an illumination model and a camera model;
and mapping the 3DMM face model to a three-dimensional model of the pixel pinching face head of the virtual character through a shape conversion method to obtain the virtual character.
2. The method of claim 1, further comprising, prior to said processing the 2D face image using the trained convolutional neural network to obtain a 3DMM face model:
training the convolutional neural network, wherein a sample used for training the convolutional neural network comprises a plurality of 2D face pictures and projection reconstruction pictures corresponding to the 2D face pictures, a used loss function comprises picture layer loss, perception layer loss, regular loss and texture loss, the picture layer loss comprises pixel loss and face recognition point loss, and the loss function is a weighted summation result of the pixel loss, the face recognition point loss, the perception layer loss, the regular loss and the texture loss.
3. The method according to claim 1 or 2, wherein the obtaining the virtual character by mapping the 3DMM face model to a three-dimensional model of a pixel pinching face head of the virtual character through a shape conversion method includes:
extracting characteristic points of the 3DMM face model, carrying out elastic deformation on the three-dimensional model of the prime body pinching face head of the virtual character by utilizing the characteristic points based on a radial basis function, and registering the 3DMM face model and the three-dimensional model of the prime body pinching face head after elastic deformation by utilizing a non-rigid ICP registration algorithm to obtain the virtual character.
4. The method of claim 3, wherein the extracting feature points of the 3DMM face model, based on a radial basis function, and using the feature points to elastically deform the three-dimensional model of the human body pinching face head of the virtual character, and registering the 3DMM face model and the elastically deformed three-dimensional model of the human body pinching face head by using a non-rigid ICP registration algorithm, to obtain the virtual character, includes:
extracting facial feature points of the 3DMM face model, carrying out elastic deformation on a three-dimensional model of a pixel pinching face head of the virtual character by utilizing the facial feature points based on a radial basis function to obtain a first three-dimensional model of the head;
using the first head three-dimensional model as a face template, and registering the 3DMM face model and the first head three-dimensional model through a non-rigid ICP registration algorithm to obtain a second head three-dimensional model;
extracting facial contour feature points of the 3DMM face model, and carrying out elastic deformation on the second head three-dimensional model by utilizing the facial contour feature points based on a radial basis function to obtain a third head three-dimensional model;
the third head three-dimensional model is used as a face template, and the 3DMM face model and the third head three-dimensional model are registered through a non-rigid ICP registration algorithm to obtain a fourth head three-dimensional model;
and carrying out smoothing treatment on the fourth head three-dimensional model to obtain the virtual character.
5. An apparatus for generating a virtual character using 2D face pictures, comprising:
the processing unit is used for acquiring a 2D face picture to be processed, and processing the 2D face picture by using a trained convolutional neural network to obtain a 3DMM face model, wherein the convolutional neural network comprises a face model, an illumination model and a camera model;
and the mapping unit is used for mapping the 3DMM face model to a three-dimensional model of the pixel pinching face head of the virtual character through a shape conversion method to obtain the virtual character.
6. The apparatus as recited in claim 5, further comprising:
the training unit is used for training the convolutional neural network before the processing unit processes the 2D face pictures by using the trained convolutional neural network to obtain a 3DMM face model, wherein a sample used for training the convolutional neural network comprises a plurality of 2D face pictures and projection reconstruction pictures corresponding to the 2D face pictures, a used loss function comprises picture layer loss, perception layer loss, regular loss and texture loss, the picture layer loss comprises pixel loss and face recognition point loss, and the loss function is a weighted summation result of the pixel loss, the face recognition point loss, the perception layer loss, the regular loss and the texture loss.
7. The apparatus of claim 5 or 6, wherein the mapping unit is configured to:
extracting characteristic points of the 3DMM face model, carrying out elastic deformation on the three-dimensional model of the prime body pinching face head of the virtual character by utilizing the characteristic points based on a radial basis function, and registering the 3DMM face model and the three-dimensional model of the prime body pinching face head after elastic deformation by utilizing a non-rigid ICP registration algorithm to obtain the virtual character.
8. The apparatus of claim 7, wherein the mapping unit is to:
extracting facial feature points of the 3DMM face model, carrying out elastic deformation on a three-dimensional model of a pixel pinching face head of the virtual character by utilizing the facial feature points based on a radial basis function to obtain a first three-dimensional model of the head;
using the first head three-dimensional model as a face template, and registering the 3DMM face model and the first head three-dimensional model through a non-rigid ICP registration algorithm to obtain a second head three-dimensional model;
extracting facial contour feature points of the 3DMM face model, and carrying out elastic deformation on the second head three-dimensional model by utilizing the facial contour feature points based on a radial basis function to obtain a third head three-dimensional model;
the third head three-dimensional model is used as a face template, and the 3DMM face model and the third head three-dimensional model are registered through a non-rigid ICP registration algorithm to obtain a fourth head three-dimensional model;
and carrying out smoothing treatment on the fourth head three-dimensional model to obtain the virtual character.
9. A computer-readable storage medium, on which a computer program is stored, which computer program, when being executed by a processor, performs the steps of the method of generating a virtual character using 2D face pictures as claimed in any one of claims 1 to 4.
10. An electronic device, comprising: a processor, a storage medium and a bus, the storage medium storing machine-readable instructions executable by the processor, the processor in communication with the storage medium via the bus when the electronic device is running, the processor executing the machine-readable instructions to perform the steps of the method of generating virtual figures using 2D face pictures as claimed in any one of claims 1 to 4.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202310671947.4A CN116433812B (en) | 2023-06-08 | 2023-06-08 | Method and device for generating virtual character by using 2D face picture |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202310671947.4A CN116433812B (en) | 2023-06-08 | 2023-06-08 | Method and device for generating virtual character by using 2D face picture |
Publications (2)
| Publication Number | Publication Date |
|---|---|
| CN116433812A true CN116433812A (en) | 2023-07-14 |
| CN116433812B CN116433812B (en) | 2023-08-25 |
Family
ID=87085754
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN202310671947.4A Active CN116433812B (en) | 2023-06-08 | 2023-06-08 | Method and device for generating virtual character by using 2D face picture |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN116433812B (en) |
Cited By (1)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN116665284A (en) * | 2023-08-02 | 2023-08-29 | 深圳宇石科技有限公司 | Face modeling and mask model partition matching method, device, terminal and medium |
Citations (7)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN107067429A (en) * | 2017-03-17 | 2017-08-18 | 徐迪 | Video editing system and method that face three-dimensional reconstruction and face based on deep learning are replaced |
| CN107967463A (en) * | 2017-12-12 | 2018-04-27 | 武汉科技大学 | A kind of conjecture face recognition methods based on composograph and deep learning |
| CN109671023A (en) * | 2019-01-24 | 2019-04-23 | 江苏大学 | A kind of secondary method for reconstructing of face image super-resolution |
| CN109993838A (en) * | 2019-03-25 | 2019-07-09 | 河海大学常州校区 | Method and system is sent out in virtual examination based on WebGL and human face rebuilding |
| CN112541963A (en) * | 2020-11-09 | 2021-03-23 | 北京百度网讯科技有限公司 | Three-dimensional virtual image generation method and device, electronic equipment and storage medium |
| CN113269862A (en) * | 2021-05-31 | 2021-08-17 | 中国科学院自动化研究所 | Scene-adaptive fine three-dimensional face reconstruction method, system and electronic equipment |
| CN114092643A (en) * | 2021-11-18 | 2022-02-25 | 深圳大学 | Soft tissue self-adaptive deformation method based on mixed reality and 3DGAN |
-
2023
- 2023-06-08 CN CN202310671947.4A patent/CN116433812B/en active Active
Patent Citations (7)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN107067429A (en) * | 2017-03-17 | 2017-08-18 | 徐迪 | Video editing system and method that face three-dimensional reconstruction and face based on deep learning are replaced |
| CN107967463A (en) * | 2017-12-12 | 2018-04-27 | 武汉科技大学 | A kind of conjecture face recognition methods based on composograph and deep learning |
| CN109671023A (en) * | 2019-01-24 | 2019-04-23 | 江苏大学 | A kind of secondary method for reconstructing of face image super-resolution |
| CN109993838A (en) * | 2019-03-25 | 2019-07-09 | 河海大学常州校区 | Method and system is sent out in virtual examination based on WebGL and human face rebuilding |
| CN112541963A (en) * | 2020-11-09 | 2021-03-23 | 北京百度网讯科技有限公司 | Three-dimensional virtual image generation method and device, electronic equipment and storage medium |
| CN113269862A (en) * | 2021-05-31 | 2021-08-17 | 中国科学院自动化研究所 | Scene-adaptive fine three-dimensional face reconstruction method, system and electronic equipment |
| CN114092643A (en) * | 2021-11-18 | 2022-02-25 | 深圳大学 | Soft tissue self-adaptive deformation method based on mixed reality and 3DGAN |
Cited By (2)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN116665284A (en) * | 2023-08-02 | 2023-08-29 | 深圳宇石科技有限公司 | Face modeling and mask model partition matching method, device, terminal and medium |
| CN116665284B (en) * | 2023-08-02 | 2023-11-28 | 深圳宇石科技有限公司 | Face modeling and mask model partition matching method, device, terminal and medium |
Also Published As
| Publication number | Publication date |
|---|---|
| CN116433812B (en) | 2023-08-25 |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| US11302064B2 (en) | Method and apparatus for reconstructing three-dimensional model of human body, and storage medium | |
| Sharma et al. | 3d face reconstruction in deep learning era: A survey | |
| CN103208133B (en) | The method of adjustment that in a kind of image, face is fat or thin | |
| EP3992919B1 (en) | Three-dimensional facial model generation method and apparatus, device, and medium | |
| US8411081B2 (en) | Systems and methods for enhancing symmetry in 2D and 3D objects | |
| JP2021535499A (en) | Face Posture Estimate / 3D Face Reconstruction Methods, Devices, and Electronic Devices | |
| WO2020108304A1 (en) | Method for reconstructing face mesh model, device, apparatus and storage medium | |
| JP2006520054A (en) | Image matching from invariant viewpoints and generation of 3D models from 2D images | |
| CN111833236B (en) | Method and device for generating three-dimensional face model for simulating user | |
| CN112862807B (en) | Hair image-based data processing method and device | |
| CN113570684A (en) | Image processing method, image processing device, computer equipment and storage medium | |
| CN116433812B (en) | Method and device for generating virtual character by using 2D face picture | |
| CN112102480B (en) | Image data processing method, apparatus, device and medium | |
| WO2011162352A1 (en) | Three-dimensional data generating apparatus, three-dimensional data generating method, and three-dimensional data generating program | |
| JP2024506170A (en) | Methods, electronic devices, and programs for forming personalized 3D head and face models | |
| Zhang et al. | 3D statistical head modeling for face/head-related product design: a state-of-the-art review | |
| CN114429518A (en) | Face model reconstruction method, device, equipment and storage medium | |
| US11769310B2 (en) | Combining three-dimensional morphable models | |
| CN111275610B (en) | Face aging image processing method and system | |
| CN116863044A (en) | Face model generation method and device, electronic equipment and readable storage medium | |
| CN116030181A (en) | 3D virtual image generation method and device | |
| CN113076918A (en) | Video-based facial expression cloning method | |
| Bouafif et al. | Monocular 3D head reconstruction via prediction and integration of normal vector field | |
| CN116912433B (en) | Three-dimensional model skeleton binding method, device, equipment and storage medium | |
| CN117808943B (en) | Three-dimensional cartoon face reconstruction method, device, equipment and storage medium |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination | ||
| GR01 | Patent grant | ||
| GR01 | Patent grant | ||
| TR01 | Transfer of patent right |
Effective date of registration: 20231214 Address after: 230031 Room 672, 6/F, Building A3A4, Zhong'an Chuanggu Science Park, No. 900, Wangjiang West Road, High-tech Zone, Hefei, Anhui Patentee after: Anhui Haima Cloud Technology Co.,Ltd. Address before: 301700 room 2d25, Building 29, No.89 Heyuan Road, Jingjin science and Technology Valley Industrial Park, Wuqing District, Tianjin Patentee before: HAIMAYUN (TIANJIN) INFORMATION TECHNOLOGY CO.,LTD. |
|
| TR01 | Transfer of patent right |