CN112200894B - Automatic digital human facial expression animation migration method based on deep learning framework - Google Patents
Automatic digital human facial expression animation migration method based on deep learning framework Download PDFInfo
- Publication number
- CN112200894B CN112200894B CN202011413230.2A CN202011413230A CN112200894B CN 112200894 B CN112200894 B CN 112200894B CN 202011413230 A CN202011413230 A CN 202011413230A CN 112200894 B CN112200894 B CN 112200894B
- Authority
- CN
- China
- Prior art keywords
- model
- controller
- digital
- animation
- expression
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Active
Links
Images
Classifications
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
- G06T13/00—Animation
- G06T13/20—3D [Three Dimensional] animation
- G06T13/40—3D [Three Dimensional] animation of characters, e.g. humans, animals or virtual beings
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06N—COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS
- G06N3/00—Computing arrangements based on biological models
- G06N3/02—Neural networks
- G06N3/04—Architecture, e.g. interconnection topology
- G06N3/045—Combinations of networks
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06N—COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS
- G06N3/00—Computing arrangements based on biological models
- G06N3/02—Neural networks
- G06N3/08—Learning methods
Abstract
The invention provides a digital human facial expression animation automatic migration method based on a deep learning framework, which comprises the following steps: s1: generating training data, making the same expression models of multiple frames of two different digital persons, extracting the parameter values of animation controller channels corresponding to the expression models, and generating a vector of the parameter dimensions of the animation controller; s2: building a model of the neural network, and building a four-layer neural network; s3: training a network model, and establishing the relation between the parameter values of the controllers under the same expression of different digital persons; s4: and (3) performing expression migration by using the trained model, inputting the controller parameter value of one digital person into the network model to obtain the controller parameter value corresponding to the same expression of the other digital person, and applying the generated controller parameter value to the digital person model so as to drive the network 3D space vertex position on the digital person model. The invention greatly improves the production efficiency of the virtual character animation by replacing manual operation with automatic migration.
Description
Technical Field
The invention belongs to the technical field of animation production, and particularly relates to a digital human facial expression animation automatic migration method based on a deep learning framework.
Background
In the existing animation production process, a virtual digital human facial expression animation is produced by an animator by adjusting the numerical value of a facial model controller and manually setting a key frame. Animation of a virtual character completely depends on manual operation, so that the time consumption is long, the efficiency is low, and the labor cost is high. When the same animation needs to be done for a second different virtual digital human character, the animator needs to do the same operation again, and an automatic process which can be replaced does not exist, so that the animation production efficiency of the high-precision human facial expression is greatly hindered.
Disclosure of Invention
The invention aims to provide a digital human facial expression animation automatic migration method based on a deep learning framework, which greatly improves the production efficiency of virtual character animation by replacing manual operation with automatic migration;
the invention provides the following technical scheme:
the digital human facial expression animation automatic migration method based on the deep learning framework comprises the following steps:
s1: generating training data, making the same expression models of multiple frames of two different digital persons, extracting the parameter values of animation controller channels corresponding to the expression models, and generating a vector of the parameter dimensions of the animation controller;
s2: building a model of the neural network, building four layers of neural networks, wherein input and output are controller parameter vectors, and the middle two layers of hidden layers are respectively provided with a plurality of neurons;
s3: training a network model, establishing a relation between controller parameter values under the same expression of different digital persons, taking the controller parameter value of one digital person as input, taking the controller parameter value of the other digital person as a label, establishing a regression task, adopting a mean square error as a cost function, and storing a model parameter with the lowest loss value in the training process;
s4: the method comprises the steps that a trained model is used for expression migration, a controller parameter value of one digital person is input into a network model to obtain a controller parameter value corresponding to the same expression of the other digital person, and the generated controller parameter value is applied to the digital person model, so that the network 3D space vertex position on the digital person model is driven;
preferably, the extraction process in S1 includes the following steps: extracting an animation controller channel attribute value on the model from each key frame, writing the animation controller channel attribute value into a text file, wherein each line of the text file represents one frame, each column represents one controller channel, and numerical values record the controller channel attribute value;
preferably, in S1, a key frame is selected to extract a parameter value of an animation controller channel corresponding to the expression model, where the key frame is a representative expression;
preferably, there are 500 neurons in each hidden layer in S2;
preferably, the network model in S4 is trained and then used as a tool in Maya software in real time;
the invention has the beneficial effects that: the invention learns the mapping relation between the channel parameters of the animation controller among different models by using the neural network, thereby realizing the effect of driving the animation of another model by using the animation of the known model, and the whole manufacturing process can be generated automatically without manual operation.
Drawings
The accompanying drawings, which are included to provide a further understanding of the invention and are incorporated in and constitute a part of this specification, illustrate embodiments of the invention and together with the description serve to explain the principles of the invention and not to limit the invention. In the drawings:
fig. 1 is a diagram of a network architecture used by the present invention.
Detailed Description
The digital human facial expression animation automatic migration method based on the deep learning framework comprises the following steps:
s1: generating training data, making the same expression models of multiple frames of two different digital persons, extracting the parameter values of animation controller channels corresponding to the expression models, and generating a vector of the parameter dimensions of the animation controller;
s2: building a model of the neural network, building four layers of neural networks, wherein input and output are controller parameter vectors, and the middle two layers of hidden layers are respectively provided with a plurality of neurons;
s3: training a network model, establishing a relation between controller parameter values under the same expression of different digital persons, taking the controller parameter value of one digital person as input, taking the controller parameter value of the other digital person as a label, establishing a regression task, adopting a mean square error as a cost function, and storing a model parameter with the lowest loss value in the training process;
s4: the method comprises the steps that a trained model is used for expression migration, a controller parameter value of one digital person is input into a network model to obtain a controller parameter value corresponding to the same expression of the other digital person, and the generated controller parameter value is applied to the digital person model, so that the network 3D space vertex position on the digital person model is driven;
the method comprises the following specific implementation steps:
step S1: training data generation
101: the artist makes multiple frames of the same expression of two different digital people A and B;
102: compiling plug-in units to extract parameter values of animation controller channels corresponding to the models, wherein the parameter values are between 0 and 1, and generating a vector of the animation controller parameter dimension;
103: selecting a plurality of key frames (standard representative expressions such as laughing, smiling, crying and the like);
c + + plug-ins are written in maya, the plug-ins can extract the animation controller channel attribute values on the model from each key frame, and a text file is written in the animation controller channel attribute values. Each line of the text file represents a frame, each column represents a controller channel, and numerical values are recorded to be controller channel attribute values;
step S2: neural network construction
As shown in fig. 1, a 4-layer neural network is established, the input and output are controller parameter vectors, and the middle two hidden layers respectively have 500 neurons, because the control channel parameter is small relative to the data volume (about 127 channels), the network is too complex and is easy to be over-fitted;
step S3: training network
The method aims to establish the relation between controller parameter values under the same expression of a digital person A and a digital person B, establish a regression task by taking the controller parameter value of the digital person A as input and the controller parameter value of the digital person B as a label, and store a model parameter with the lowest loss value in the training process by adopting a mean square error as a cost function;
specifically, establishing a regression task is a training process: assuming that the parameter values required for digital person B to achieve the same expression are vector y, the net prediction result is ŷ. The network training process is to compare ŷ with the actual reference value of y, and continuously adjust the weight and deviation of the network until the prediction error is minimized, and the cost function is the loss function and the mean square error can be selected. When the obtained error is smaller than a satisfactory threshold value, the training is stopped, and the state of the network (i.e., the weight of each neuron) at that time is recorded. The model recording the network weight is a trained network model, and the trained network model can be used for the next step;
step S4: expression migration using trained models
401: inputting the controller parameter value of the digital person A into the network model to obtain the controller parameter value corresponding to the same expression of the digital person B;
402: writing a plug-in to apply the generated controller parameter value to the model B, so as to drive the network 3D space vertex position on the model B;
403: after being trained, the network model can be used as a tool in maya software in real time;
according to the automatic migration method of the digital human facial expression animation based on the deep learning framework, the facial expressions between digital human models are automatically converted through deep learning without manual operation, so that the known animation of the model A is directly applied to the model B, and natural and vivid animation effects are generated;
although the present invention has been described in detail with reference to the foregoing embodiments, it will be apparent to those skilled in the art that changes may be made in the embodiments and/or equivalents thereof without departing from the spirit and scope of the invention. Any modification, equivalent replacement, or improvement made within the spirit and principle of the present invention should be included in the protection scope of the present invention.
Claims (4)
1. The digital human facial expression animation automatic migration method based on the deep learning framework is characterized by comprising the following steps of:
s1: generating training data, making the same expression models of multiple frames of two different digital persons, extracting the parameter values of animation controller channels corresponding to the expression models, and generating a vector of the parameter dimensions of the animation controller;
the extraction process in S1 includes the following steps: extracting an animation controller channel attribute value on the model from each key frame, writing the animation controller channel attribute value into a text file, wherein each line of the text file represents one frame, each column represents one controller channel, and numerical values record the controller channel attribute value;
s2: building a model of the neural network, building four layers of neural networks, wherein input and output are controller parameter vectors, and the middle two layers of hidden layers are respectively provided with a plurality of neurons;
s3: training a network model, establishing a relation between controller parameter values under the same expression of different digital persons, taking the controller parameter value of one digital person as input, taking the controller parameter value of the other digital person as a label, establishing a regression task, adopting a mean square error as a cost function, and storing a model parameter with the lowest loss value in the training process;
s4: and (3) performing expression migration by using the trained model, inputting the controller parameter value of one digital person into the network model to obtain the controller parameter value corresponding to the same expression of the other digital person, and applying the generated controller parameter value to the digital person model so as to drive the network 3D space vertex position on the digital person model.
2. The method according to claim 1, wherein in step S1, a key frame is selected to extract parameter values of an animation controller channel corresponding to the expression model, and the key frame is a representative expression.
3. The deep learning framework-based automatic migration method for digital human facial expression animations according to claim 1, wherein the hidden layers in S2 have 500 neurons each.
4. The deep learning framework-based automatic migration method for digital human facial expression animations according to claim 1, wherein the network model in S4 is trained and used as a tool in Maya software in real time.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202011413230.2A CN112200894B (en) | 2020-12-07 | 2020-12-07 | Automatic digital human facial expression animation migration method based on deep learning framework |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202011413230.2A CN112200894B (en) | 2020-12-07 | 2020-12-07 | Automatic digital human facial expression animation migration method based on deep learning framework |
Publications (2)
| Publication Number | Publication Date |
|---|---|
| CN112200894A CN112200894A (en) | 2021-01-08 |
| CN112200894B true CN112200894B (en) | 2021-03-09 |
Family
ID=74033806
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN202011413230.2A Active CN112200894B (en) | 2020-12-07 | 2020-12-07 | Automatic digital human facial expression animation migration method based on deep learning framework |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN112200894B (en) |
Families Citing this family (6)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN112700524B (en) * | 2021-03-25 | 2021-07-02 | 江苏原力数字科技股份有限公司 | 3D character facial expression animation real-time generation method based on deep learning |
| CN113724367A (en) * | 2021-07-13 | 2021-11-30 | 北京理工大学 | Robot expression driving method and device |
| CN113781616B (en) * | 2021-11-08 | 2022-02-08 | 江苏原力数字科技股份有限公司 | Facial animation binding acceleration method based on neural network |
| CN113763519B (en) * | 2021-11-09 | 2022-02-08 | 江苏原力数字科技股份有限公司 | Voice-driven 3D character facial expression method based on deep learning |
| CN114898020A (en) * | 2022-05-26 | 2022-08-12 | 唯物(杭州)科技有限公司 | 3D character real-time face driving method and device, electronic equipment and storage medium |
| CN116485959A (en) * | 2023-04-17 | 2023-07-25 | 北京优酷科技有限公司 | Control method of animation model, and adding method and device of expression |
Citations (2)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN109376692A (en) * | 2018-11-22 | 2019-02-22 | 河海大学常州校区 | Migration convolution neural network method towards facial expression recognition |
| CN111652121A (en) * | 2020-06-01 | 2020-09-11 | 腾讯科技（深圳）有限公司 | Training method of expression migration model, and expression migration method and device |
-
2020
- 2020-12-07 CN CN202011413230.2A patent/CN112200894B/en active Active
Patent Citations (2)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN109376692A (en) * | 2018-11-22 | 2019-02-22 | 河海大学常州校区 | Migration convolution neural network method towards facial expression recognition |
| CN111652121A (en) * | 2020-06-01 | 2020-09-11 | 腾讯科技（深圳）有限公司 | Training method of expression migration model, and expression migration method and device |
Non-Patent Citations (3)
| Title |
|---|
| Deep Multi-task Learning for Facial Expression Recognition and Synthesis Based on Selective Feature Sharing;Rui Zhao等;《arXiv》;20200709;第1-8页 * |
| 基于深度学习的人脸表情迁移方法;刘剑等;《计算机科学》;20190630;第46卷(第6A期);第250-253页 * |
| 面向人脸表情识别的迁移卷积神经网络研究;翟懿奎等;《信号处理》;20180630;第34卷(第6期);第729-738页 * |
Also Published As
| Publication number | Publication date |
|---|---|
| CN112200894A (en) | 2021-01-08 |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| CN112200894B (en) | Automatic digital human facial expression animation migration method based on deep learning framework | |
| CN112454390B (en) | Humanoid robot facial expression simulation method based on deep reinforcement learning | |
| CN110599573B (en) | Method for realizing real-time human face interactive animation based on monocular camera | |
| CN109543838B (en) | Image increment learning method based on variational self-encoder | |
| CN113469356A (en) | Improved VGG16 network pig identity recognition method based on transfer learning | |
| CN113269872A (en) | Synthetic video generation method based on three-dimensional face reconstruction and video key frame optimization | |
| CN107491726A (en) | A kind of real-time expression recognition method based on multi-channel parallel convolutional neural networks | |
| CN107357838A (en) | Dialog strategy canbe used on line method based on multi-task learning | |
| CN107979764A (en) | Video caption generation method based on semantic segmentation and multilayer notice frame | |
| CN106503654A (en) | A kind of face emotion identification method based on the sparse autoencoder network of depth | |
| CN108921284A (en) | Interpersonal interactive body language automatic generation method and system based on deep learning | |
| CN111598979B (en) | Method, device and equipment for generating facial animation of virtual character and storage medium | |
| CN107102727A (en) | Dynamic gesture study and recognition methods based on ELM neutral nets | |
| CN112651360B (en) | Skeleton action recognition method under small sample | |
| CN113781616B (en) | Facial animation binding acceleration method based on neural network | |
| KR20150075909A (en) | Method and apparatus for editing 3D character motion | |
| CN110992455B (en) | Real-time expression capture system | |
| CN112700524A (en) | 3D character facial expression animation real-time generation method based on deep learning | |
| CN108453737A (en) | A kind of robot motion track acquisition system and method based on neural network | |
| CN117095071A (en) | Picture or video generation method, system and storage medium based on main body model | |
| CN116957866A (en) | Individualized teaching device of digital man teacher | |
| CN113763519B (en) | Voice-driven 3D character facial expression method based on deep learning | |
| CN109859537A (en) | A kind of picture-weaving in silk tutoring system and method, information data processing terminal | |
| CN115972197A (en) | Robot action demonstration learning method based on joint angle trajectory coding | |
| CN106097373B (en) | A kind of smiling face's synthetic method based on branch's formula sparse component analysis model |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination | ||
| GR01 | Patent grant | ||
| GR01 | Patent grant |