CN114898020A - 3D character real-time face driving method and device, electronic equipment and storage medium - Google Patents
3D character real-time face driving method and device, electronic equipment and storage medium Download PDFInfo
- Publication number
- CN114898020A CN114898020A CN202210589964.9A CN202210589964A CN114898020A CN 114898020 A CN114898020 A CN 114898020A CN 202210589964 A CN202210589964 A CN 202210589964A CN 114898020 A CN114898020 A CN 114898020A
- Authority
- CN
- China
- Prior art keywords
- actor
- role
- character
- picture set
- facial
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Pending
Links
- 238000000034 method Methods 0.000 title claims abstract description 24
- 238000003860 storage Methods 0.000 title claims abstract description 9
- 230000001815 facial effect Effects 0.000 claims abstract description 42
- 230000008921 facial expression Effects 0.000 claims abstract description 16
- 238000009877 rendering Methods 0.000 claims abstract description 9
- 230000006870 function Effects 0.000 claims description 10
- 230000014509 gene expression Effects 0.000 claims description 7
- 238000004590 computer program Methods 0.000 claims description 6
- 238000005457 optimization Methods 0.000 claims description 5
- 238000009826 distribution Methods 0.000 claims description 3
- 238000000605 extraction Methods 0.000 claims description 3
- 230000000694 effects Effects 0.000 description 3
- 238000010586 diagram Methods 0.000 description 2
- 230000004048 modification Effects 0.000 description 2
- 238000012986 modification Methods 0.000 description 2
- 230000008569 process Effects 0.000 description 2
- 238000005096 rolling process Methods 0.000 description 2
- 239000000126 substance Substances 0.000 description 2
- 230000009286 beneficial effect Effects 0.000 description 1
- 230000008859 change Effects 0.000 description 1
- 238000013527 convolutional neural network Methods 0.000 description 1
- 238000005520 cutting process Methods 0.000 description 1
- 238000005286 illumination Methods 0.000 description 1
- 230000006872 improvement Effects 0.000 description 1
Images
Classifications
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
- G06T13/00—Animation
- G06T13/20—3D [Three Dimensional] animation
- G06T13/40—3D [Three Dimensional] animation of characters, e.g. humans, animals or virtual beings
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06N—COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS
- G06N3/00—Computing arrangements based on biological models
- G06N3/02—Neural networks
- G06N3/04—Architecture, e.g. interconnection topology
- G06N3/045—Combinations of networks
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06N—COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS
- G06N3/00—Computing arrangements based on biological models
- G06N3/02—Neural networks
- G06N3/08—Learning methods
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
- G06T9/00—Image coding
- G06T9/002—Image coding using neural networks
Landscapes
- Engineering & Computer Science (AREA)
- Theoretical Computer Science (AREA)
- Physics & Mathematics (AREA)
- General Physics & Mathematics (AREA)
- Evolutionary Computation (AREA)
- Artificial Intelligence (AREA)
- General Health & Medical Sciences (AREA)
- General Engineering & Computer Science (AREA)
- Data Mining & Analysis (AREA)
- Biophysics (AREA)
- Biomedical Technology (AREA)
- Molecular Biology (AREA)
- Computing Systems (AREA)
- Computational Linguistics (AREA)
- Life Sciences & Earth Sciences (AREA)
- Mathematical Physics (AREA)
- Software Systems (AREA)
- Health & Medical Sciences (AREA)
- Multimedia (AREA)
- Processing Or Creating Images (AREA)
Abstract
The invention discloses a real-time face driving method and device for a 3D role, electronic equipment and a storage medium, wherein the method comprises the steps of obtaining a role face animation file and a facial expression video file corresponding to an actor; acquiring a controller value corresponding to each frame of animation based on the character facial animation file; rendering a role video file from the role facial animation file, and extracting a corresponding role picture set and an actor facial picture set from the role video file and an actor facial expression video file; constructing a VAE model; training the VAE model based on the role picture set, the actor facial picture set and the controller value corresponding to the role picture set; after the training is finished, inputting the actor face picture into a trained VAE model to obtain the coefficient of a controller; and transmitting the coefficient of the controller to rendering software, and driving the 3D virtual image by the rendering software in real time to obtain high-quality facial animation. The invention can obtain high-precision facial animation through the VAE model.
Description
Technical Field
The invention relates to the technical field of 3D image processing, in particular to a method and a device for driving a 3D role real-time face, electronic equipment and a storage medium.
Background
With the advent of the meta-universe era, a great deal of requirements for real-time face driving of high-precision 3D face models (3-4 thousands of model vertices) are generated. Current video face driving schemes: the problem of the mobile phone terminal represented by iphoneX is that the expressiveness is not enough, such as the change of the mouth shape of a speaking person, and the mobile phone terminal cannot cross over the uncany valley (terrorist effect) when being used for driving a high-precision real virtual person; the surface capture helmet generally has a complicated 3D role, has a higher use threshold in the calibration stage of an actor, and needs to be recalibrated each time the actor is changed.
Disclosure of Invention
In order to solve the above problems in the prior art, the present invention provides a method and an apparatus for driving a 3D character real-time face, an electronic device, and a storage medium.
The invention discloses a real-time face driving method for a 3D role, which comprises the following steps:
acquiring a 3D character facial animation file and a facial expression video file of a corresponding actor;
acquiring a controller value corresponding to each frame of animation based on the 3D character facial animation file;
rendering the 3D character facial animation file into a 3D character video file, and extracting a corresponding 3D character picture set and an actor facial picture set from the 3D character video file and the actor facial expression video file;
constructing a VAE model;
training the VAE model based on the 3D role picture set, the actor facial picture set and controller values corresponding to the 3D role picture set;
after the training is finished, inputting the actor face picture into the VAE model after the training is finished, and acquiring the coefficient of the controller;
and transmitting the coefficient of the controller to rendering software, wherein the rendering software drives the 3D virtual image in real time to obtain high-quality facial animation.
Preferably, the VAE model includes an encoder and two decoders;
the encoder encodes the input pictures in the 3D character picture set and the actor face picture set;
the decoder performs decoding optimization training on the pictures in the 3D role picture set and the actor face picture set after being encoded to obtain the optimal weight values of the encoder and the decoder;
and the other decoder performs decoding training on the encoded 3D role picture set and the controller value corresponding to the 3D role picture set to obtain the controller coefficient.
Preferably, the encoder encoding function is:
Enc(x)＝(f z (x)，f id (x))；
in the formula: x is an input picture; f. of z (x) Encoding for a VAE; z is the code of expression information in a hidden space; f. of id (x) Encoding for AE; id is the code of the identity information in the hidden space.
Preferably, a loss function of the decoding optimization training is:
L＝L vae +L rec +L cycle ；
in the formula: x is the number of 1 The character is a 3D character picture; x is the number of 2 An actor face picture;
is the output of the decoder;
is f z (x)；p θ (x | z) is the reconstruction of a photograph by z; p (z) is the prior distribution of z; l is dssim Is x and
structural similarity error of (2).
Preferably, the error function of the other decoding training is:
in the formula: y is the controller value vector corresponding to a frame of animation,
is a vector of predicted controller values.
Preferably, the 3D character face animation file is an FBX file, and the controller value corresponding to each frame of animation is read out through FBX sdk of the autodesk.
The present invention further provides a device for driving the real-time face of the 3D character, including:
the acquiring module is used for acquiring a 3D role facial animation file and a facial expression video file corresponding to an actor;
the reading module is used for acquiring a controller value corresponding to each frame of animation based on the 3D role facial animation file;
the extraction module is used for rendering the 3D role facial animation file into a 3D role video file and extracting a corresponding 3D role picture set and an actor facial picture set from the 3D role video file and the actor facial expression video file;
the building module is used for building the VAE model;
a training module for training the VAE model based on the 3D character picture set, the actor face picture set, and a controller value corresponding to the 3D character picture set;
the computing module is used for inputting the actor face picture into the trained VAE model after the training is finished and acquiring the coefficient of the controller;
and the driving module is used for transmitting the coefficient of the controller to rendering software, and the rendering software drives the 3D virtual image in real time to obtain high-quality facial animation.
The invention also provides an electronic device comprising at least one processing unit and at least one memory unit, wherein the memory unit stores a computer program which, when executed by the processing unit, causes the processing unit to perform the above-mentioned method.
The invention also provides a storage medium storing a computer program executable by an electronic device, which when run on the electronic device causes the electronic device to perform the above-mentioned method.
Compared with the prior art, the invention has the beneficial effects that:
according to the invention, high-precision facial animation can be obtained through the VAE model, and the threshold is lower than that of a facial helmet.
Drawings
FIG. 1 is a schematic flow structure diagram of a 3D role real-time face driving method according to the present invention;
FIG. 2 is a schematic structural diagram of a model in the 3D role real-time face driving method of the present invention.
Detailed Description
In order to make the objects, technical solutions and advantages of the embodiments of the present invention clearer, the technical solutions in the embodiments of the present invention will be clearly and completely described below with reference to the drawings in the embodiments of the present invention, and it is obvious that the described embodiments are some, but not all, embodiments of the present invention. All other embodiments, which can be obtained by a person skilled in the art without any inventive step based on the embodiments of the present invention, are within the scope of the present invention.
The invention is described in further detail below with reference to the attached drawing figures:
referring to fig. 1, the invention discloses a 3D character real-time face driving method, including:
acquiring a 3D character facial animation file and a facial expression video file of a corresponding actor;
acquiring a controller value corresponding to each frame of animation based on the 3D character facial animation file;
specifically, the 3D character face animation file is an FBX file, and the controller name used by the 3D character model and the controller value corresponding to each frame of animation are read from the FBX sdk of the autodesk.
Rendering the 3D role facial animation file into a 3D role video file, and extracting a corresponding 3D role picture set and an actor facial picture set from the 3D role video file and the actor facial expression video file;
specifically, frames are extracted from the 3D character video file and the facial expression video file of the actor according to a certain frame rate to form a picture set, and the picture of each 3D character animation frame corresponds to the value of the controller.
Constructing a VAE model, as shown in FIG. 2;
specifically, the model comprises an encoder and two decoders, wherein the encoder and the decoders are all multilayer convolutional neural networks, and gradient descent is adopted to update the model weight during training. The VAE model comprises an encoder and two decoders; the encoder encodes the input 3D role picture set and pictures in the actor face picture set; a decoder 1 carries out decoding optimization training on the pictures in the 3D role picture set and the actor face picture set after being coded to obtain the optimal weight values of the encoder and the decoder; and fixing the parameters of the encoder, discarding the decoder 1 and the id part, and performing decoding training on the encoded 3D role picture set and the controller value corresponding to the 3D role picture set by the other decoder 2 to obtain a controller coefficient. An encoder encodes an input picture into a hidden space (spatial space), enc (x) ═ f z (x)，f id (x) X represents an input picture, and the entire Enc represents the encoder, where f z (x) Z is the typical VAE encoding; f. of id (x) Id is the code of a typical AE. f. of z (x) And f id (x) Sharing the structure of the previous layers of the encoder except for the last layer andand (4) weighting. Wherein z is the code of the expression information in the hidden space, and id is the code of the identity information in the hidden space.
Training the VAE model based on the 3D role picture set, the actor face picture set and the controller value corresponding to the 3D role picture set;
specifically, during training, the 3D character picture and the actor face picture do not need to be in one-to-one correspondence with each frame, the training is divided into two stages, the first stage is self-supervision learning, and the optimal weight values of the encoder and the decoder 1 are learned by reconstructing the input photos. The loss function of the first stage model includes:
L＝L vae +L rec +L cycle ；
L vae ELBO loss for VAE:
wherein the content of the first and second substances,
i.e. f z (x) Representing an input picture x, and acquiring a VAE code z; p is a radical of θ (x | z) denotes the reconstruction of a picture by z, corresponding to decoder 1; p (z) is the prior distribution of z. Where the first term represents the expected value of the x-likelihood and the second term, called KL divergence, is equivalent to a regularization term, such that
Can approximate p (z) as closely as possible.
L rec Error for reconstructed picture and original picture:
wherein the content of the first and second substances,
is the output of the decoder, L dssim Is x and
the first term of the error focuses on the consistency of each pixel, and the second term focuses on the similarity of the structures of the two photographs.
L cycle For cycle consistency error:
wherein x is 1 The character is a 3D character picture; x is the number of 2 An actor face picture; the first term represents x 1 The hidden space expression code f of the picture z (x 1 ) And x 2 Identity code f id (x 2 ) Pictures obtained after splicing through the decoder Dec should be summed f after passing through the encoder z (x 1 ) The distance is as close as possible; the second equality is simply from x 2 Obtaining expression code f z (x 2 )，x 1 Obtaining an identity code f id (x 1 )。
The second stage of training is a supervised learning, and the data is 3D character pictures and corresponding controller values. Inputting the 3D role picture into an encoder, inputting the output implicit code z into a decoder 2 to obtain a controller value of model prediction, wherein an error function is
Wherein y is a controller value vector corresponding to a frame of animation,
is a vector of predicted controller values.
Further, in order to ensure the robustness of the model, the training data, i.e., the 3D character picture and the actor's face picture, are processed, including but not limited to rotation, flipping, cropping, adding noise, changing brightness, chrominance, contrast, saturation, simulated illumination, distortion, etc., so as to improve the robustness of the model.
Namely, training the VAE in the first stage, inputting the actor facial picture and the 3D role picture into the model, and obtaining the optimal weight values of the encoder and the decoder through the optimal loss function. In order not to spill the expression information encoded in z into id, during training, the id obtained for the same actor photo in a batch is randomized before entering the encoder, and the id obtained for the character photo in a batch is randomized before entering the encoder. And the z and the id are spliced by a re-parameterization skill during training.
And the second stage trains the controller decoding module. And fixing parameters of an encoder, discarding a decoder 1 and an id part, training an output z of the encoder connected with a decoder 2, namely performing supervision training on the values of an input 3D role picture and a corresponding controller, wherein the controller parameters are used as output label values. By optimizing L reg A penalty function to obtain optimal weight values for the controller decoder. After the training is completed, the VAE model is used only by the encoder and decoder 2.
After the training is finished, inputting the actor face picture into a trained VAE model to obtain the coefficient of a controller;
and transmitting the coefficient of the controller to rendering software, and driving the 3D virtual image by the rendering software in real time to obtain high-quality facial animation.
Specifically, when the system is used, actor face video stream is obtained from a camera, each frame of picture of the actor face is extracted and input to a 3D role model, the obtained result of a decoder 2 is input to a 3D rendering engine as a controller coefficient, animation is rendered, the performance of the whole process is high, and under the condition that a GPU display card is accelerated, if the frame rate of the camera is high enough, the whole process can reach a frame rate more than 60. Rendering software includes, but is not limited to, UE, maya.
In this embodiment, the facial expression video files of the actors may be the same actor or may be multiple actors.
Examples
The well-pinched realistic faces on Metahuman creator are downloaded from qualel Bridge and imported into Maya and UE as 3D characters.
A performance script is prepared, which includes common expressions, Chinese pronunciations, and Chinese sentences. And the actor A performs according to the performance script, records a front face video with the frame rate of 30 frames, and takes the front face video as a reference template for the performance of the subsequent actor.
Sequence frame animation was performed from actor a performance video K using Metahuman's controller inside Maya for a total of about 8 minutes; exporting the animation into an FBX file for storage, and rendering the animation into a video file in Maya, wherein the video file comprises videos of front faces and side face angles, and the frame rate is 30 frames;
performing by the actor A or other actors B according to the performance script, recording front face and side face videos, wherein the frame rate is 30 frames;
reading the controller coefficient of each frame in the FBX animation file by FBX sdk, dimension 169; extracting photos from the 3D role video and the actor A or other actor B videos, aligning the face of each photo, and cutting out 256x256 pictures;
building a VAE model, wherein an encoder comprises a plurality of layers of rolling blocks, a decoder 1 comprises a plurality of layers of reverse rolling blocks, and a decoder 2 is a plurality of layers of MLPs;
training the VAE model by using the obtained picture, and obtaining the coefficient of the controller after the training is finished;
starting a model client, enabling the actor A or other actors B to face a camera, reading video streams of the camera, inputting each frame of actor picture into a model, and transmitting the coefficient of a controller to UE (user equipment) through levellink; therefore, the Metahuman 3D character in the UE is driven to carry out facial animation in real time, and the effect exceeds the driving effect of IphoneX.
The present invention further provides a device for driving the real-time face of the 3D character, including:
the acquiring module is used for acquiring a 3D role facial animation file and a facial expression video file corresponding to an actor;
the reading module is used for acquiring a controller value corresponding to each frame of animation based on the 3D role facial animation file;
the extraction module is used for rendering the 3D role facial animation file into a 3D role video file and extracting a corresponding 3D role picture set and an actor facial picture set from the 3D role video file and the actor facial expression video file;
the building module is used for building the VAE model;
the training module is used for training the VAE model based on the 3D role picture set, the actor face picture set and the controller value corresponding to the 3D role picture set;
the computing module is used for inputting the actor face picture into the trained VAE model after the training is finished and acquiring the coefficient of the controller;
and the driving module is used for transmitting the coefficient of the controller to rendering software, and the rendering software drives the 3D virtual image in real time to obtain high-quality facial animation.
The invention also provides an electronic device comprising at least one processing unit and at least one memory unit, wherein the memory unit stores a computer program which, when executed by the processing unit, causes the processing unit to perform the above-mentioned method.
The present invention also provides a storage medium storing a computer program executable by an electronic device, which when run on the electronic device causes the electronic device to perform the above-mentioned method.
The above is only a preferred embodiment of the present invention, and is not intended to limit the present invention, and various modifications and changes will occur to those skilled in the art. Any modification, equivalent replacement, or improvement made within the spirit and principle of the present invention should be included in the protection scope of the present invention.
Claims (9)
1. A real-time face driving method for a 3D character is characterized by comprising the following steps:
acquiring a 3D character facial animation file and a facial expression video file of a corresponding actor;
acquiring a controller value corresponding to each frame of animation based on the 3D character facial animation file;
rendering the 3D character facial animation file into a 3D character video file, and extracting a corresponding 3D character picture set and an actor facial picture set from the 3D character video file and the actor facial expression video file;
constructing a VAE model;
training the VAE model based on the 3D role picture set, the actor facial picture set and a controller value corresponding to the 3D role picture set;
after the training is finished, inputting the actor face picture into the VAE model after the training is finished, and acquiring the coefficient of the controller;
and transmitting the coefficient of the controller to rendering software, wherein the rendering software drives the 3D virtual image in real time to obtain high-quality facial animation.
2. The 3D character real-time face driving method according to claim 1, wherein the VAE model includes an encoder and two decoders;
the encoder encodes the input pictures in the 3D character picture set and the actor face picture set;
the decoder performs decoding optimization training on the pictures in the 3D role picture set and the actor face picture set after being encoded to obtain the optimal weight values of the encoder and the decoder;
and the other decoder carries out decoding training on the encoded 3D role picture set and the controller value corresponding to the 3D role picture set to obtain the controller coefficient.
3. The method of real-time face driving of a 3D character according to claim 2, wherein the encoder encoding function is:
Enc(x)＝(f z (x),f id (x))；
in the formula: x is an input picture; f. of z (x) Encoding for a VAE; z is the code of expression information in a hidden space; f. of id (x) Encoding for AE; id is the code of the identity information in the hidden space.
4. The 3D character real-time face driving method according to claim 3, wherein the loss function of the decoding optimization training is:
L＝L vae +L rec +L cycle ；
in the formula: x is the number of 1 The character is a 3D character picture; x is the number of 2 An actor face picture;
is the output of the decoder;
is f z (x)；p θ (x | z) is the reconstruction of a photograph by z; p (z) is the prior distribution of z; l is dssim Is x and
structural similarity error of (2).
5. The method of claim 4, wherein the error function of the decoding training is:
in the formula: y is the controller value vector corresponding to a frame of animation,
is a vector of predicted controller values.
6. The method of claim 1, wherein the 3D character face animation file is an FBX file, and the controller value corresponding to each frame of animation is read out through FBX sdk of autodesk.
7. A 3D character real-time face-driving apparatus, comprising:
the acquiring module is used for acquiring a 3D role facial animation file and a facial expression video file corresponding to an actor;
the reading module is used for acquiring a controller value corresponding to each frame of animation based on the 3D role facial animation file;
the extraction module is used for rendering the 3D role facial animation file into a 3D role video file and extracting a corresponding 3D role picture set and an actor facial picture set from the 3D role video file and the actor facial expression video file;
the building module is used for building the VAE model;
a training module for training the VAE model based on the 3D character picture set, the actor face picture set, and a controller value corresponding to the 3D character picture set;
the computing module is used for inputting the actor face picture into the trained VAE model after the training is finished and acquiring the coefficient of the controller;
and the driving module is used for transmitting the coefficient of the controller to rendering software, and the rendering software drives the 3D virtual image in real time to obtain high-quality facial animation.
8. An electronic device, comprising at least one processing unit and at least one memory unit, wherein the memory unit stores a computer program that, when executed by the processing unit, causes the processing unit to perform the method of any of claims 1 to 6.
9. A storage medium storing a computer program executable by an electronic device, the program, when run on the electronic device, causing the electronic device to perform the method of any one of claims 1 to 6.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202210589964.9A CN114898020A (en) | 2022-05-26 | 2022-05-26 | 3D character real-time face driving method and device, electronic equipment and storage medium |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202210589964.9A CN114898020A (en) | 2022-05-26 | 2022-05-26 | 3D character real-time face driving method and device, electronic equipment and storage medium |
Publications (1)
| Publication Number | Publication Date |
|---|---|
| CN114898020A true CN114898020A (en) | 2022-08-12 |
Family
ID=82725991
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN202210589964.9A Pending CN114898020A (en) | 2022-05-26 | 2022-05-26 | 3D character real-time face driving method and device, electronic equipment and storage medium |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN114898020A (en) |
Citations (21)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| US20100141663A1 (en) * | 2008-12-04 | 2010-06-10 | Total Immersion Software, Inc. | System and methods for dynamically injecting expression information into an animated facial mesh |
| US9104908B1 (en) * | 2012-05-22 | 2015-08-11 | Image Metrics Limited | Building systems for adaptive tracking of facial features across individuals and groups |
| CN106600667A (en) * | 2016-12-12 | 2017-04-26 | 南京大学 | Method for driving face animation with video based on convolution neural network |
| CN109934767A (en) * | 2019-03-06 | 2019-06-25 | 中南大学 | A kind of human face expression conversion method of identity-based and expressive features conversion |
| CN111598979A (en) * | 2020-04-30 | 2020-08-28 | 腾讯科技（深圳）有限公司 | Method, device and equipment for generating facial animation of virtual character and storage medium |
| US20200294294A1 (en) * | 2019-03-15 | 2020-09-17 | NeoCortext Inc. | Face-swapping apparatus and method |
| CN112200894A (en) * | 2020-12-07 | 2021-01-08 | 江苏原力数字科技股份有限公司 | Automatic digital human facial expression animation migration method based on deep learning framework |
| CN112541445A (en) * | 2020-12-16 | 2021-03-23 | 中国联合网络通信集团有限公司 | Facial expression migration method and device, electronic equipment and storage medium |
| CN112541958A (en) * | 2020-12-21 | 2021-03-23 | 清华大学 | Parametric modeling method and device for three-dimensional face |
| CN112700524A (en) * | 2021-03-25 | 2021-04-23 | 江苏原力数字科技股份有限公司 | 3D character facial expression animation real-time generation method based on deep learning |
| CN112700523A (en) * | 2020-12-31 | 2021-04-23 | 魔珐(上海)信息科技有限公司 | Virtual object face animation generation method and device, storage medium and terminal |
| CN113255457A (en) * | 2021-04-28 | 2021-08-13 | 上海交通大学 | Animation character facial expression generation method and system based on facial expression recognition |
| US20210327038A1 (en) * | 2020-04-16 | 2021-10-21 | Disney Enterprises, Inc. | Tunable models for changing faces in images |
| CN113633983A (en) * | 2021-08-16 | 2021-11-12 | 上海交通大学 | Method, device, electronic equipment and medium for controlling expression of virtual character |
| WO2021232876A1 (en) * | 2020-05-18 | 2021-11-25 | 北京搜狗科技发展有限公司 | Method and apparatus for driving virtual human in real time, and electronic device and medium |
| CN113781616A (en) * | 2021-11-08 | 2021-12-10 | 江苏原力数字科技股份有限公司 | Facial animation binding acceleration method based on neural network |
| CN113807265A (en) * | 2021-09-18 | 2021-12-17 | 山东财经大学 | Diversified human face image synthesis method and system |
| CN113822437A (en) * | 2020-06-18 | 2021-12-21 | 辉达公司 | Deep layered variational automatic encoder |
| US20220005248A1 (en) * | 2020-07-02 | 2022-01-06 | Weta Digital Limited | Generating an Animation Rig for Use in Animating a Computer-Generated Character Based on Facial Scans of an Actor and a Muscle Model |
| CN114494542A (en) * | 2022-01-24 | 2022-05-13 | 广州喳喳科技有限公司 | Character driving animation method and system based on convolutional neural network |
| CN114531561A (en) * | 2022-01-25 | 2022-05-24 | 阿里巴巴（中国）有限公司 | Face video coding method, decoding method and device |
-
2022
- 2022-05-26 CN CN202210589964.9A patent/CN114898020A/en active Pending
Patent Citations (21)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| US20100141663A1 (en) * | 2008-12-04 | 2010-06-10 | Total Immersion Software, Inc. | System and methods for dynamically injecting expression information into an animated facial mesh |
| US9104908B1 (en) * | 2012-05-22 | 2015-08-11 | Image Metrics Limited | Building systems for adaptive tracking of facial features across individuals and groups |
| CN106600667A (en) * | 2016-12-12 | 2017-04-26 | 南京大学 | Method for driving face animation with video based on convolution neural network |
| CN109934767A (en) * | 2019-03-06 | 2019-06-25 | 中南大学 | A kind of human face expression conversion method of identity-based and expressive features conversion |
| US20200294294A1 (en) * | 2019-03-15 | 2020-09-17 | NeoCortext Inc. | Face-swapping apparatus and method |
| US20210327038A1 (en) * | 2020-04-16 | 2021-10-21 | Disney Enterprises, Inc. | Tunable models for changing faces in images |
| CN111598979A (en) * | 2020-04-30 | 2020-08-28 | 腾讯科技（深圳）有限公司 | Method, device and equipment for generating facial animation of virtual character and storage medium |
| WO2021232876A1 (en) * | 2020-05-18 | 2021-11-25 | 北京搜狗科技发展有限公司 | Method and apparatus for driving virtual human in real time, and electronic device and medium |
| CN113822437A (en) * | 2020-06-18 | 2021-12-21 | 辉达公司 | Deep layered variational automatic encoder |
| US20220005248A1 (en) * | 2020-07-02 | 2022-01-06 | Weta Digital Limited | Generating an Animation Rig for Use in Animating a Computer-Generated Character Based on Facial Scans of an Actor and a Muscle Model |
| CN112200894A (en) * | 2020-12-07 | 2021-01-08 | 江苏原力数字科技股份有限公司 | Automatic digital human facial expression animation migration method based on deep learning framework |
| CN112541445A (en) * | 2020-12-16 | 2021-03-23 | 中国联合网络通信集团有限公司 | Facial expression migration method and device, electronic equipment and storage medium |
| CN112541958A (en) * | 2020-12-21 | 2021-03-23 | 清华大学 | Parametric modeling method and device for three-dimensional face |
| CN112700523A (en) * | 2020-12-31 | 2021-04-23 | 魔珐(上海)信息科技有限公司 | Virtual object face animation generation method and device, storage medium and terminal |
| CN112700524A (en) * | 2021-03-25 | 2021-04-23 | 江苏原力数字科技股份有限公司 | 3D character facial expression animation real-time generation method based on deep learning |
| CN113255457A (en) * | 2021-04-28 | 2021-08-13 | 上海交通大学 | Animation character facial expression generation method and system based on facial expression recognition |
| CN113633983A (en) * | 2021-08-16 | 2021-11-12 | 上海交通大学 | Method, device, electronic equipment and medium for controlling expression of virtual character |
| CN113807265A (en) * | 2021-09-18 | 2021-12-17 | 山东财经大学 | Diversified human face image synthesis method and system |
| CN113781616A (en) * | 2021-11-08 | 2021-12-10 | 江苏原力数字科技股份有限公司 | Facial animation binding acceleration method based on neural network |
| CN114494542A (en) * | 2022-01-24 | 2022-05-13 | 广州喳喳科技有限公司 | Character driving animation method and system based on convolutional neural network |
| CN114531561A (en) * | 2022-01-25 | 2022-05-24 | 阿里巴巴（中国）有限公司 | Face video coding method, decoding method and device |
Non-Patent Citations (7)
| Title |
|---|
| JIANG Z H, WU Q, CHEN K, ET AL.: "Disentangled representation learning for 3d face shape", CVF CONFERENCE ON COMPUTER VISION AND PATTERN RECOGNITION, 31 December 2019 (2019-12-31), pages 11957 - 11966 * |
| 李嘉豪: "基于深度学习的三维人脸变形研究与应用", 中国优秀硕士学位论文全文数据库, 信息科技辑, no. 2021, 15 December 2021 (2021-12-15), pages 3 * |
| 王浩童: "基于隐向量控制的人脸卡通漫画风格迁移研究", 中国优秀硕士学位论文全文数据库, 信息科技辑, no. 2022, 15 January 2022 (2022-01-15), pages 1 - 78 * |
| 蔡倩倩: "卡通漫画头像的个性化风格迁移算法研究", 中国优秀硕士学位论文全文数据库, 信息科技辑, no. 2021, 15 December 2021 (2021-12-15), pages 1 - 56 * |
| 费建伟, 夏志华, 余佩鹏等: "人脸合成技术综述", 计算机科学与探索, vol. 15, no. 11, 15 July 2021 (2021-07-15), pages 2025 - 2047 * |
| 陈松, 袁训明: "动态人脸表情合成的模型特征驱动算法综述", 计算机与现代化, no. 2019, 4 July 2019 (2019-07-04), pages 47 - 54 * |
| 陈柯宇: "面向数字人应用的人脸表示与动画驱动", 中国优秀硕士学位论文全文数据库, 信息科技辑, no. 2021, 15 August 2021 (2021-08-15), pages 1 - 83 * |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| CN108596024B (en) | Portrait generation method based on face structure information | |
| WO2022267641A1 (en) | Image defogging method and system based on cyclic generative adversarial network | |
| CN110751649B (en) | Video quality evaluation method and device, electronic equipment and storage medium | |
| CN113261013A (en) | System and method for realistic head rotation and facial animation synthesis on mobile devices | |
| CN113901894A (en) | Video generation method, device, server and storage medium | |
| CN115914505B (en) | Video generation method and system based on voice-driven digital human model | |
| WO2023185395A1 (en) | Facial expression capturing method and apparatus, computer device, and storage medium | |
| US11640687B2 (en) | Volumetric capture and mesh-tracking based machine learning 4D face/body deformation training | |
| US20220414838A1 (en) | Image dehazing method and system based on cyclegan | |
| WO2024098685A1 (en) | Face driving method and apparatus for virtual character, and terminal device and readable storage medium | |
| CN114820341A (en) | Image blind denoising method and system based on enhanced transform | |
| Elgharib et al. | Egocentric videoconferencing | |
| CN114202615A (en) | Facial expression reconstruction method, device, equipment and storage medium | |
| CN114187165A (en) | Image processing method and device | |
| CN115082300A (en) | Training method of image generation model, image generation method and device | |
| Zhang et al. | Adaptive affine transformation: A simple and effective operation for spatial misaligned image generation | |
| CN116091955A (en) | Segmentation method, segmentation device, segmentation equipment and computer readable storage medium | |
| US20230206955A1 (en) | Re-Timing Objects in Video Via Layered Neural Rendering | |
| CN114898020A (en) | 3D character real-time face driving method and device, electronic equipment and storage medium | |
| CN117119123A (en) | Method and system for generating digital human video based on video material | |
| CN115578298A (en) | Depth portrait video synthesis method based on content perception | |
| CN112950501B (en) | Noise field-based image noise reduction method, device, equipment and storage medium | |
| CN115496843A (en) | Local realistic-writing cartoon style migration system and method based on GAN | |
| Oshiba et al. | Face image generation of anime characters using an advanced first order motion model with facial landmarks | |
| Wang | Cartoon‐Style Image Rendering Transfer Based on Neural Networks |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination |