CN110599573A - Method for realizing real-time human face interactive animation based on monocular camera - Google Patents

Abstract

The invention relates to a three-dimensional character animation technology, and discloses a method for realizing real-time human face interactive animation based on a monocular camera. The method can be summarized as follows: capturing a face video image and voice input information, and extracting a face expression animation parameter and a voice emotion animation parameter; learning a training sequence consisting of skeleton motion and corresponding skin deformation through an action state space model, establishing a virtual character skeleton skin model based on an auxiliary bone controller, driving the virtual character skeleton skin model through the extracted facial expression animation parameters and voice emotion animation parameters, and generating real-time interactive animation.

Description

A realization method of face real-time interaction animation based on monocular camera

technical field

The invention relates to a three-dimensional character animation technology, in particular to a method for realizing real-time interactive animation of human faces based on a monocular camera.

Background technique

In recent years, with the continuous development of computer software and hardware equipment (such as: Apple's newly released augmented reality application development toolkit ARKit, Google's ARCore 1.0 and a series of supporting tools, etc.), multimedia technology has been brought into a heyday. At the same time, because people have higher and higher requirements for the visualization of human-computer interaction interface, face modeling and animation technology play an increasingly important role in human-computer interaction. The application fields of 3D facial expression animation technology are very extensive, such as game entertainment, film production, human-computer interaction, advertisement production, etc., which has important application value and theoretical significance.

Since the pioneering work of Parke ^[1] and others using computer-generated face animation in 1972, more and more researchers in the world have discovered the research value of 3D face modeling and its animation technology, and made many important contribution. As shown in Figure 1, these tasks mainly include how to use an effective model to represent the shape change of the face, how to accurately and quickly capture facial expressions, how to build a real-time and fine 3D face reconstruction model, and how to construct a digital face substitute, and Drive it to generate realistic face models.

Cao ^[2] et al. proposed a real-time face tracking and animation method based on 3D shape regression in 2013. This method uses a monocular video camera as a face image acquisition device, and is mainly divided into two steps: preprocessing and real-time operation. In the preprocessing stage, a monocular camera is used to collect user-specific gestures and expressions, including a series of facial expressions and head rotations, and then the facial feature point marking algorithm is used to semi-automatically mark the user's face image. Based on the face image after calibrated feature points, Cao ^[3] et al. built a 3D facial expression library FaceWarehouse for visual computing applications in 2014. In this database, a bilinear face model (Bilinear Face Model) containing two attributes of individual and expression is proposed, which is used to fit and generate a user-specific expression fusion model. A three-dimensional face shape vector composed of three-dimensional positions of feature points in each image of the user collected by the camera is calculated through the expression fusion model. The regression algorithm of this method uses a two-level cascaded regression (Two-level Boosted Regression) algorithm of shape-related features, and all images and their corresponding 3D face shape vectors are used as input to train the user-specific 3D shape regressor. . In the real-time running stage, the user-specific 3D shape regressor regresses the 3D shape parameters and face motion parameters based on the face motion parameters obtained in the current frame and the previous frame, including the rigid transformation parameters of the head and the non-rigidity of facial expressions. Facial motion parameters, and then transfer and map these parameters to the virtual character to drive it to generate expression animation corresponding to the facial motion.

However, the above method has certain limitations. For each new user, a user-specific expression fusion model and a 3D face shape regressor need to be generated, which requires about 45 minutes of preprocessing. In 2014, Cao ^[4] et al. proposed a real-time face tracking algorithm based on offset dynamic expression regression. The processing operation realizes the real-time facial expression tracking and capturing algorithm of any user.

The real-time face tracking and animation method based on 3D shape regression proposed by Cao et al. in 2013 and the real-time face tracking algorithm based on offset dynamic expression regression proposed by Cao et al ^. in 2014 are all focused on How to accurately, efficiently, and robustly track large-scale movements of human faces in videos, such as large-scale expressions such as frowning, laughing, and mouth opening, as well as adding rigid movements such as head rotation and translation. But both ignore the detailed information on the human face, such as the forehead lines on the face when the eyebrows are raised, the secondary movement of the facial skin caused by exercise, etc., and these details are precisely to help people understand expressions and make people feel better. Important features for a more expressive face.

references:

[1]Parke F I.Computer generated animation of faces[C]//ACMConference.ACM,1972:451-457.

[2] Cao C, Weng Y, Lin S, et al. 3D shape regression for real-time facial animation [J]. ACM Transactions on Graphics, 2013, 32(4): 1.

[3] Cao C, Weng Y, Zhou S, et al. FaceWarehouse: A 3D Facial Expression Database for Visual Computing [J]. IEEE Transactions on Visualization & Computer Graphics, 2014, 20(3): 413-425.

[4]Cao C,Hou Q,Zhou K.Displaced dynamic expression regression for real-time facial tracking and animation[J].Acm Transactions on Graphics,2014,33(4):1-10.

[5] Ekman P, Friesen W V. Facial Action Coding System: Manual [J]. Agriculture, 1978.

[6] Duffy N, Helmbold D. Boosting Methods for Regression [J]. Machine Learning, 2002, 47(2-3): 153-200.

Contents of the invention

The technical problem to be solved by the present invention is: to propose a method for realizing real-time interactive animation of human face based on a monocular camera, by integrating facial expression capture and voice emotion recognition technology, animation parameters are generated, and real-time animation parameters are generated through bone-based technology Synthetic and visualized dynamic skin deformation animation makes the generated real-time animation expression more rich, natural, realistic and has its own characteristics.

The technical solution adopted by the present invention to solve the problems of the technologies described above is:

A method for realizing real-time interactive animation of human faces based on a monocular camera, comprising the following steps:

S1. Capture face video images through a monocular camera to obtain a sequence of face images; at the same time capture voice input information through a voice sensor;

S2. Mark the feature points of the face in the face image sequence, and extract the facial expression animation parameters;

S3. Extracting speech features from the captured speech input information, and extracting speech emotion animation parameters;

S4. Learn the training sequence consisting of skeletal motion and corresponding skin deformation through the action state space model, establish a virtual character skeletal skin model based on the auxiliary bone controller, and use the extracted facial expression animation parameters and voice emotion animation parameters to drive the The virtual character bone skin model is used to generate real-time interactive animation.

As a further optimization, in step S2, a double-layer cascaded regression model is used to mark facial feature points, and the Candide-3 facial model based on the facial activity coding system is used as a parameter carrier to extract facial expression animation parameters.

As a further optimization, the two-layer cascaded regression model adopts a two-layer regression structure, and the first layer adopts an enhanced regression model combined by T weak regressors in a superimposed manner; the second layer is composed of A weak regressor is formed by superimposing a strong regressor formed by cascading K regression models.

As a further optimization, step S3 specifically includes:

S31. Analyzing and extracting the features of the voice emotion information in the voice input information;

S32. Perform emotion recognition on the extracted speech emotion features to complete emotion judgment;

S33. Corresponding the voice emotion result to the facial activity coding system based on the AU unit, extracting the corresponding AU parameters, and obtaining voice emotion animation parameters.

As a further optimization, in step S4, the action state space model is composed of three key elements: (S, A, {P})

S represents the facial expression state set of each frame of the virtual character;

A represents a set of actions, and the parameters obtained through facial expression recognition and voice emotion recognition are used as a set of action vectors to drive the virtual character to change state in the next frame;

P is the state transition probability, which represents the probability distribution of the virtual character's expression state s _t ∈ S in the current frame t, after performing an action a _t ∈ A to other states.

As a further optimization, in step S4, the method for establishing a virtual character skeleton skin model based on the auxiliary bone controller includes:

a. Use the bone skin model of the virtual character that has already been made without auxiliary bones as the original model;

b. Optimize the skin weight of the bone skin model;

c. Gradually insert the auxiliary bone into the area where the maximum approximation error occurs between the original model and the face of the target model;

d. Using the block coordinate descent algorithm to solve the two sub-problems of skin weight optimization and auxiliary bone position transformation optimization;

e. Build an auxiliary bone controller: the skin transformation q based on the auxiliary bone controller is represented by a connection between a static component x and a dynamic component y, q=x+y; wherein, the static component X is calculated according to the main skeleton pose in the original model; dynamic Component y is controlled using an action state space model.

The beneficial effects of the present invention are:

1. Facial expressions are the expression of people's emotions, but in some special cases, facial expressions cannot fully express the inner emotions of characters. If only capturing and tracking facial expression feature points are used as parameters for face point-to-point driving, obviously the generated face animation is not vivid enough. For example, when a character smiles and wryly smiles, the facial expressions of the two are similar, but they will emit different tone words. Therefore, the addition of voice emotion recognition technology can better capture the current emotional state changes of the character from the perspective of voice. The invention combines the facial expression capture technology with the voice emotion recognition technology, which can greatly improve the richness, naturalness and realism of the virtual character's facial expression animation.

2. Since the movement of bones and muscles jointly drives the change of skin expression, in order to better simulate the movement of the skin, the present invention adopts the bone skin model, automatically adds auxiliary bones through the bone-based skin decomposition algorithm, and simulates the movement of the head bones The main bone and the auxiliary bone that simulates muscle movement jointly drive the virtual character for animation.

Description of drawings

Figure 1 is the research status of 3D facial animation;

Fig. 2 is the realization schematic diagram of the face real-time interactive animation of the present invention;

Figure 3 is a schematic diagram of the enhanced regression structure;

Figure 4 is a schematic diagram of a two-layer cascade regression structure;

Fig. 5 is a schematic diagram of the state transition process of ASSM.

Detailed ways

The present invention aims to propose a method for realizing real-time interactive animation of human face based on a monocular camera. By integrating facial expression capture and voice emotion recognition technology, animation parameters are generated, and a visualized dynamic skin is synthesized in real time through bone-based technology. The deformed animation makes the expression of the generated real-time animation more rich, natural, realistic and has its own characteristics. In order to achieve this purpose, the scheme among the present invention mainly realizes from the following aspects:

1. Face motion capture:

Face motion capture consists of two parts: non-rigid capture of facial expressions and rigid transformation capture of the head. According to the unique muscle movement characteristics of facial expressions, the facial features are coordinated as a unified whole to show each facial expression. Through the Facial Activity Coding System (FACS) based on the Facial Motion Unit (AU), the coding corresponding to the facial expression is used as the semantic attribute of the facial expression. Using this invariant intermediate description method is a reliable method for facial expression recognition Feature representation to make up for the lack of low-level features in facial expression recognition.

2. Voice emotion recognition: capture the current emotional state of the performer through the voice input of the performer, and generate voice emotion animation parameters corresponding to the current emotional state of the performer through steps such as voice feature extraction, dimension reduction, and classification.

3. In terms of target digital stand-in expression: use the bone-based dynamic stand-in expression method, which learns the training sequence composed of bone motion and corresponding skin deformation sequence to obtain the optimal transmission of nonlinear complex deformation including soft tissue. Through the user's expression semantics extracted from facial motion capture, the motion of the character's head bones is driven to programmatically control the auxiliary bones to simulate the dynamic deformation of the facial skin.

In terms of specific implementation, the principle of the implementation method of the real-time interactive animation of the face based on the monocular camera in the present invention is shown in Figure 2, which includes the following means:

(1) Capture the face video image through a monocular camera to obtain a sequence of face images; at the same time, capture voice input information through a voice sensor;

(2) Face capture and tracking: Mark facial feature points from captured face images, and extract facial expression animation parameters;

Face feature point location is a key link in face recognition, face tracking, face animation and 3D face modeling. Due to factors such as face diversity and illumination, it is still a difficult challenge to locate facial feature points in natural environments. The specific definition of face feature points is: for a face shape S=[x ₁ ,y ₁ ,...,x _N ,y _N ] containing N face feature points, for an input face picture, The goal of face feature point location is to estimate the shape S of a face feature point, so that S and the real shape of face feature points The difference between S and The minimum alignment difference between can be defined as L ₂ -normal form Use this formula to guide the training of the face feature point locator or to evaluate the performance of the face feature point localization algorithm.

The present invention intends to use an algorithm framework based on a regression model to perform face detection and tracking in real time and efficiently.

a) Boosted Regression

Use enhanced regression to combine T weak regressors (R ¹ ,...R ^t ,...R ^T ) in a superimposed manner. For a given face sample I and initial shape S ⁰ , each regressor calculates a shape increment according to the sample features And update the current shape in a cascading fashion:

S ^t ＝S ^t+1 +R ^t (I,S ^t-1 ),t=1,...,T (1)

R ^t (I,S ^t-1 ) represents the shape increment calculated by the regressor R ^t using the input sample image I and the previous shape S ^t-1 , and R ^t is calculated by the input sample image I and the previous shape S ^t-1 decided to use the shape index feature to learn R ^t , as shown in Figure 3;

Given N training samples Represents the true shape of I _i of the i-th sample, and train (R ¹ ,...R ^t ,...R ^T ) in a loop until the training error no longer increases. Each R ^t is calculated by minimizing the alignment error, i.e.:

Represents the previous shape estimation value of the i-th image, and the output of R ^t is a shape increment.

b) Two-level Boosted Regression

The enhanced regression algorithm is to regress the entire shape, and the large appearance difference in the input image and the rough initial face shape make the single-layer weak regressor no longer applicable. A single regressor is too weak, the convergence is slow during training, and the results are poor during testing. In order to converge faster and more stably during training, the present invention adopts a two-layer cascade structure, as shown in FIG. 4 .

The first layer employs the augmented regression model described above. For each regressor R ^t in the first layer, K regression models are used to learn, that is, R ^t = (r ¹ ,...r ^k ,...r ^K ), here r is called the primary regressor, A strong regressor is formed by cascading K primary regressors. The difference between the first layer and the second layer is that the input S ^t-1 of each regressor R ^t in the first layer is different, while the input of each regressor r ^k in the second layer is the same. For example, all regressor inputs in the second layer of R ^t are S ^t-1 .

In the generation of facial expression animation parameters, the present invention utilizes the facial activity coding system FACS based on the AU unit proposed by Ekman et al ^. Control of a certain part or mass of muscle. Specifically, the Candide-3 face model based on the facial activity coding system can be used as a parameter carrier to extract the AU parameters E corresponding to facial expressions.

The Candide-3 face model is represented as follows:

In the formula, Indicates the basic shape of the model, S is the static deformation matrix, A is the dynamic deformation matrix, σ is the static deformation parameter, α is the dynamic deformation parameter, R and t represent the head rigid rotation matrix and head translation matrix, respectively. g is a column vector of model vertex coordinates, which is used to represent various specific facial expression shapes. Model g is determined by four parameters R, t, α, σ.

(3) Extract speech features in the captured speech input information, and extract speech emotion animation parameters;

Analyze and extract the voice emotion information features in the voice input information; perform emotion recognition on the extracted voice emotion features to complete the emotion judgment; correspond the voice emotion results to the facial activity coding system based on the AU unit, and extract the corresponding AU parameter, to obtain the voice emotion animation parameter V.

(4) Learn the training sequence consisting of bone motion and corresponding skin deformation through the action state space model, establish a virtual character bone skin model based on the auxiliary bone controller, and drive it through the extracted facial expression animation parameters and voice emotion animation parameters The skeleton skin model of the virtual character generates real-time interactive animation.

(a) Action State Space Model (ASSM):

The action state space model consists of three key elements (S,A,{P}), where:

S: Indicates the state set, the facial expression state of the virtual character (such as happy, sad, etc.);

A: Represents a set of actions. The parameters obtained through facial expression recognition and voice emotion recognition are used as a set of action vectors to drive the virtual character to change state in the next frame;

P: state transition probability, which represents the probability distribution of the virtual character's expression state s _t ∈ S in the current frame t, and then transitions to other states after performing an action a _t ∈ A.

The dynamic process of ASSM is as follows: the virtual character is in the state s ₀ , driven by the performer’s action vector a ₀ ∈ A, and then moves to the next frame state s ₁ according to the probability P, and then performs the action a ₁ , ... and so on, we can get Figure 5 shows the process.

(b) Auxiliary bone frame:

Auxiliary bone localization process: Given a set of primary bone index sets P, it is used to calculate the global transformation matrix G _p∈P of the primary bones. make and Indicates the position of the i-th vertex on the main bone matrix and the static skin in the original pose. Indicates the skin transformation matrix corresponding to the main bone. The index set of the secondary bone called auxiliary bone is denoted by H, and the corresponding skin formula is as follows:

v _i represents the position of the vertices of the deformed skin, _Sh represents the skin matrix corresponding to the h-th auxiliary bone. The first term of the above equation corresponds to skin deformation driven by the main bone, and the second term provides additional control for deformation using auxiliary bones. The number of auxiliary bones is given by the designer to balance deformation quality and computational cost.

Skin Decomposition: The skin decomposition is divided into two sub-problems for description. The first subproblem estimates all optimal skin weights and skin matrix The best approximation to the training data at each frame t ∈ T. The second subproblem approximates the discrete transformation by an auxiliary bone control model based on the original skeleton

Given the main skeletal skinning matrix The training sequence of and the corresponding vertex animation Here, the skinning optimization decomposition problem is formulated as a least-squares-constrained problem that minimizes the sum of squared shape differences between the original and target models over the entire training dataset.

in,

In the above formula, |·| _n represents the l _n paradigm, and V represents the subscript of the vertex set. The constant k represents the maximum number of skin mesh vertices to be affected by bones, to adjust the balance between computational cost and accuracy.

Auxiliary bone controller: Assuming that the auxiliary bone is driven by the original skeleton with only spherical joints, the pose of the auxiliary bone is uniquely determined by the main bone r _p ∈ SO(3) of all rotating components. Represented by a column vector as:

u:＝Δt ₀ ||Δr ₀ ||r ₁ ||r ₂ ||…||r _|p| (9)

In this formula, _u∈R ³ _| ^p|+6 , || represents the connection operator of vector value, |P| ) represents the direction change of the root node.

Each auxiliary bone is attached to the main bone as a child bone of the main bone, for example, Φ(h) is considered as the main bone corresponding to the hth auxiliary bone, S(h) is the skin matrix corresponding to the hth auxiliary bone, make It is composed of the local transformation L _h and the global transformation. The local transformation L _h consists of a translation component t _h and a rotation component r _h .

The model assumes that skin deformation is modeled as a concatenation of static and dynamic deformations, the former determined from the pose of the main skeleton, and the latter depending on the skeleton motion and changes in skin deformation over past time steps. Therefore, the skin transformation of the auxiliary bone q is represented by the connection of a static component x and a dynamic component y, q=x+y. The static transformation x is computed from the skeleton pose, and the dynamic transformation y is controlled using a state-space model that takes into account accumulated information from previous skeleton poses and auxiliary skeleton transformations.

Claims (6)

1. A method for realizing real-time human face interactive animation based on a monocular camera is characterized by comprising the following steps:

s1, capturing a face video image through a monocular camera to obtain a face image sequence; simultaneously capturing voice input information through a voice sensor;

s2, marking human face characteristic points in the human face image sequence, and extracting human face expression animation parameters;

s3, extracting voice features from the captured voice input information and extracting voice emotion animation parameters;

s4, learning a training sequence consisting of skeleton motion and corresponding skin deformation through the action state space model, establishing a virtual character skeleton skin model based on the auxiliary bone controller, driving the virtual character skeleton skin model through the extracted facial expression animation parameters and voice emotion animation parameters, and generating real-time interactive animation.

2. The method for realizing real-time interactive animation of human faces based on a monocular camera as recited in claim 1,

in step S2, a double-layer cascade regression model is used to mark the facial feature points, and a Candide-3 facial model based on a facial activity coding system is used as a parameter carrier to extract facial expression animation parameters.

3. The method for realizing real-time interactive animation of human faces based on a monocular camera as recited in claim 2,

the double-layer cascade regression model adopts a two-layer regression structure, wherein the first layer adopts an enhanced regression model formed by combining T weak regressors in a superposition mode; the second layer is formed by superposing strong regressors which are formed by cascading K regression models aiming at each weak regressor in the first layer.

4. The method for realizing real-time interactive animation of human faces based on a monocular camera as recited in claim 1,

step S3 specifically includes:

s31, analyzing and extracting the speech emotion information features in the speech input information;

s32, performing emotion recognition on the extracted voice emotion characteristics to finish emotion judgment;

and S33, corresponding the voice emotion result to an AU unit-based face activity coding system, extracting corresponding AU parameters and obtaining voice emotion animation parameters.

5. The method for realizing real-time interactive animation of human faces based on a monocular camera as recited in claim 1,

in step S4, the motion state space model is composed of three key elements: (S, A, { P })

S represents a facial expression state set of each frame of the virtual character;

a represents a group of action sets, parameters obtained through facial expression recognition and speech emotion recognition are used as a group of action vectors, and the change state of the next frame of virtual characters is driven;

p is state transition probability and represents the expression state s of the virtual character in the current frame t_tE.g. S, by performing action a_te.A and then to other states.

6. The method for realizing real-time interactive animation of human faces based on a monocular camera as recited in claim 1,

in step S4, the method for creating a virtual character skeleton skin model based on an auxiliary bone controller includes:

a. taking a bone covering model of the manufactured virtual character without the auxiliary bone as an original model;

b. carrying out covering weight optimization on the skeleton covering model;

c. gradually inserting the auxiliary bone into a region where the maximum approximate error is generated between the original model and the target model face;

d. solving two sub-problems of skin weight optimization and auxiliary bone position conversion optimization by adopting a block coordinate descent algorithm;

e. constructing an auxiliary bone controller, wherein a skin transformation q based on the auxiliary bone controller is represented by connecting a static component x and a dynamic component y, and q is x + y; wherein, the static component X is calculated according to the main skeleton posture in the original model; the dynamic component y is controlled using a motion state space model.