CN108363973B - Unconstrained 3D expression migration method - Google Patents
Unconstrained 3D expression migration method Download PDFInfo
- Publication number
- CN108363973B CN108363973B CN201810124168.1A CN201810124168A CN108363973B CN 108363973 B CN108363973 B CN 108363973B CN 201810124168 A CN201810124168 A CN 201810124168A CN 108363973 B CN108363973 B CN 108363973B
- Authority
- CN
- China
- Prior art keywords
- face
- expression
- model
- shape
- image
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Active
Links
- 230000014509 gene expression Effects 0.000 title claims abstract description 85
- 238000000034 method Methods 0.000 title claims abstract description 36
- 230000005012 migration Effects 0.000 title claims abstract description 21
- 238000013508 migration Methods 0.000 title claims abstract description 21
- 239000013598 vector Substances 0.000 claims abstract description 30
- 230000004044 response Effects 0.000 claims description 21
- 230000008859 change Effects 0.000 claims description 19
- 238000012549 training Methods 0.000 claims description 16
- 238000000513 principal component analysis Methods 0.000 claims description 13
- 238000001514 detection method Methods 0.000 claims description 12
- 239000011159 matrix material Substances 0.000 claims description 12
- 238000010586 diagram Methods 0.000 claims description 10
- 230000001815 facial effect Effects 0.000 claims description 9
- 238000000605 extraction Methods 0.000 claims description 6
- 238000012887 quadratic function Methods 0.000 claims description 6
- 238000012706 support-vector machine Methods 0.000 claims description 6
- 230000009471 action Effects 0.000 claims description 4
- 230000006870 function Effects 0.000 claims description 4
- 238000010606 normalization Methods 0.000 claims description 3
- 230000008921 facial expression Effects 0.000 abstract description 16
- 230000002194 synthesizing effect Effects 0.000 abstract description 4
- 238000005516 engineering process Methods 0.000 description 6
- 230000003993 interaction Effects 0.000 description 4
- 230000008569 process Effects 0.000 description 3
- 238000004891 communication Methods 0.000 description 2
- 230000001419 dependent effect Effects 0.000 description 2
- 230000008451 emotion Effects 0.000 description 2
- 238000010801 machine learning Methods 0.000 description 2
- 238000013507 mapping Methods 0.000 description 2
- 210000003205 muscle Anatomy 0.000 description 2
- 238000001308 synthesis method Methods 0.000 description 2
- 241000282887 Suidae Species 0.000 description 1
- 230000009286 beneficial effect Effects 0.000 description 1
- 230000015572 biosynthetic process Effects 0.000 description 1
- 210000000988 bone and bone Anatomy 0.000 description 1
- 230000007547 defect Effects 0.000 description 1
- 230000000694 effects Effects 0.000 description 1
- 230000004927 fusion Effects 0.000 description 1
- 238000005286 illumination Methods 0.000 description 1
- 239000000203 mixture Substances 0.000 description 1
- 238000010422 painting Methods 0.000 description 1
- 238000012545 processing Methods 0.000 description 1
- 230000009467 reduction Effects 0.000 description 1
- 238000011895 specific detection Methods 0.000 description 1
- 238000003786 synthesis reaction Methods 0.000 description 1
Images
Classifications
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06V—IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING
- G06V40/00—Recognition of biometric, human-related or animal-related patterns in image or video data
- G06V40/10—Human or animal bodies, e.g. vehicle occupants or pedestrians; Body parts, e.g. hands
- G06V40/16—Human faces, e.g. facial parts, sketches or expressions
- G06V40/174—Facial expression recognition
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F18/00—Pattern recognition
- G06F18/20—Analysing
- G06F18/21—Design or setup of recognition systems or techniques; Extraction of features in feature space; Blind source separation
- G06F18/213—Feature extraction, e.g. by transforming the feature space; Summarisation; Mappings, e.g. subspace methods
- G06F18/2135—Feature extraction, e.g. by transforming the feature space; Summarisation; Mappings, e.g. subspace methods based on approximation criteria, e.g. principal component analysis
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06V—IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING
- G06V40/00—Recognition of biometric, human-related or animal-related patterns in image or video data
- G06V40/10—Human or animal bodies, e.g. vehicle occupants or pedestrians; Body parts, e.g. hands
- G06V40/16—Human faces, e.g. facial parts, sketches or expressions
- G06V40/168—Feature extraction; Face representation
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06V—IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING
- G06V40/00—Recognition of biometric, human-related or animal-related patterns in image or video data
- G06V40/10—Human or animal bodies, e.g. vehicle occupants or pedestrians; Body parts, e.g. hands
- G06V40/16—Human faces, e.g. facial parts, sketches or expressions
- G06V40/172—Classification, e.g. identification
Landscapes
- Engineering & Computer Science (AREA)
- Theoretical Computer Science (AREA)
- General Physics & Mathematics (AREA)
- Oral & Maxillofacial Surgery (AREA)
- Health & Medical Sciences (AREA)
- Physics & Mathematics (AREA)
- Computer Vision & Pattern Recognition (AREA)
- Multimedia (AREA)
- Human Computer Interaction (AREA)
- General Health & Medical Sciences (AREA)
- Data Mining & Analysis (AREA)
- Bioinformatics & Cheminformatics (AREA)
- General Engineering & Computer Science (AREA)
- Evolutionary Computation (AREA)
- Evolutionary Biology (AREA)
- Bioinformatics & Computational Biology (AREA)
- Artificial Intelligence (AREA)
- Life Sciences & Earth Sciences (AREA)
- Processing Or Creating Images (AREA)
- Image Analysis (AREA)
Abstract
The invention discloses an unconstrained 3D expression migration method, which is realized by using a computer vision and probability statistics based method; firstly, detecting a face region by using the adaboost + harr characteristics, then extracting face geometric characteristics in the face region according to a Constrained Local Model (CLM) method, then extracting face expression parameters by using Support Vector Regression (SVR), finally inputting the expression parameters to control the face BlendShape of the 3D Model, synthesizing expression animation, and realizing the unconstrained 3D expression migration method.
Description
Technical Field
The invention belongs to the technical field of image processing, and particularly relates to an unconstrained 3D expression migration method.
Background
The facial expression has an important role in communication between people, and the facial expression has the advantages of being more intuitive and accurate in expressing human emotion compared with media such as character voice and the like. The emotion interaction mode of people is used in the fields of virtual reality, digital entertainment, communication and video conference, man-machine interaction and the like, and has the advantages of strong expressive force, more natural interaction and the like compared with the traditional interaction modes such as voice, control panels and the like. The expression migration method roughly comprises the following three aspects: capturing the facial expression, extracting facial expression parameters, and synthesizing parameterized target facial animation.
At present, facial expression capturing technologies can be divided into an expression capturing technology without mark points and an expression capturing technology with mark points, the expression capturing technology based on the mark points can capture accurate expression details, but complex hardware equipment is often needed for assistance, a painting brush and the like need to be used for marking the face of a user in the capturing process, and the facial expression capturing technology has certain invasiveness. The unmarked expression capturing technology has less constraint on hardware and a user, and facial expression information is extracted through a 2D facial image.
Most of the expression feature parameterization methods use a machine learning algorithm, a model is trained through a large number of data sets, the mapping relation between captured expression information and expression control parameters is learned, however, the method is limited by individual differences of users, the expression result of a classifier is dependent on training data and the detection capability of the facial expression in a natural state to a great extent, and for the problem, a learner learns the expression features of the current user through an additional initialization step, so that the influence caused by personalized differences is effectively weakened, but the algorithm execution complexity is increased.
The animation synthesis method can be divided into 2D face animation and 3D model face animation according to the type of the target face, the 2D face animation is based on images, and high-reality 2D face animation can be obtained, but in the synthesized animation, the illumination condition of the face and the posture of the face are difficult to change, and seamless splicing into a 3D scene is difficult to achieve. The muscle model driving mode in the 3D model face animation is difficult to obtain control parameters through a computer vision algorithm, the manual control and the manual consumption for establishing the 3D model are very high, the face animation mode of a mixed sample needs to preset an expression library of each basic expression, the expression library needs to meet the orthogonality and the comprehensiveness, and the manual cost for establishing the expression library is very high.
The invention uses a label-free expression capture algorithm, uses a machine learning algorithm to realize automatic parameterized expression feature extraction by using comprehensive training data according to the universality feature of the facial expression, and finally uses an orthogonalized mixed sample to synthesize the facial animation.
Disclosure of Invention
The invention aims to overcome the defects of the prior art and provides an unconstrained 3D expression migration method, which comprises the steps of extracting user expression parameters according to the facial image characteristics of a user, mapping the user expression parameters to a virtual 3D model, synthesizing facial animation and realizing unconstrained 3D expression migration.
In order to achieve the above object, the present invention provides an unconstrained 3D expression migration method, comprising:
(1) off-line training of the face shape model and obtaining relevant parameters in the model
(1.1) downloading a face image with characteristic point labels from a face database to serve as a face image set;
(1.2) constructing a human face shape model:
wherein,
representing the average face shape, P is a matrix of principal components of the face shape variation, P ═ P1,P2...Pk]B is the weight vector of the face shape change, B ═ B1,b2.,..,bk]T；
(1.3) calculating relevant parameters of the face shape model by using the face image set
Setting M face images in the face image set, wherein each face image has N feature points, and the coordinate of the ith feature point is recorded as (x)i,yi)，i＝1,2,…,N；
The feature point vector composed by jth face image uses x(j)＝[x1 y1 x2 y2...xN yN]TIndicate, average face shape
Comprises the following steps:
subtracting the average face shape from the feature point vector formed by each face image to obtain a shape change matrix with the average value of 0
Then, PCA extraction matrix is used for principal component analysis
Characteristic vector P ofcAnd a corresponding characteristic value lambdacC is 1,2, …, min (M, N), and then the first k eigenvectors are selected to form a matrix P consisting of main components of the face shape change in a column discharge mode;
(1.4) respectively modeling in a neighborhood range of m multiplied by m by taking the coordinate position of each feature point as a center, then training a scoring model by using a Support Vector Machine (SVM), scoring m multiplied by m points in the neighborhood range by using the trained scoring model to obtain m multiplied by m scoring results, and forming a scoring response graph of each feature point by using the m multiplied by m scoring results;
fitting each scored response graph to a quadratic function r (x, y) ═ a (x-x)0)2+b(y-y0)2+ c, wherein (x)0,y0) The position coordinates of the center point of the response diagram, namely the coordinates of each characteristic point, a, b and c are fitting parameters, and (x, y) the position coordinates of the rest points except the center point in the response diagram;
the position coordinates of the other points are respectively substituted into the quadratic function to obtain the maximum value R (x, y) in the response diagram, and then sigma (R (x, y) -R (x, y))2Solving the values of a, b and c;
(2) acquiring an image containing the face of a user in real time, and marking the image as a source image;
(3) detecting a face region of a user in the source image by using a haar feature + adaboost face detection algorithm;
(4) solved using a training phase
P,λjA, b, c, detecting the characteristic points of the current face area
(4.1) according to the human face shape model:
and parameters
P, using a shape-changing weight vector B ═ B1,b2.,..,bk]TAcquiring all initial feature points in the face region;
(4.2) obtaining a scoring response graph of each initial characteristic point according to the method in the step (1.4);
(4.3) by
As an objective function, wherein (x)p,yp) The position coordinate of the p-th point in the scoring response graph is represented, and beta refers to the weight value of the shape constraint; enabling f (x) to converge to the maximum value through iteration, and substituting the weight vector of the corresponding shape change into the face shape model at the moment to acquire all feature points in the face region;
(4.4) taking the updated shape change weight vector B as a new shape model parameter, repeating the steps (4.2) - (4.3) until the updated shape change weight vector B is kept unchanged, and outputting the position of a face feature point corresponding to the final face shape model;
(5) parameterizing the expression characteristics of the user by adopting an Action Unit (AU) detection algorithm with normalized personal characteristics according to the positions of the facial feature points
(5.1) extracting apparent features of the human face
Using a face image set with AU parameter labels in a plurality of databases, scaling the face region scale of each face image to a fixed size, then extracting a gradient histogram of each face image, extracting gradient features in the gradient histogram by using a Principal Component Analysis (PCA), and selecting the first k features as face apparent features;
(5.2) extracting geometric features of human face
Extracting human face feature points from the step (4) as human face geometric features;
(5.3) extracting the expression in the natural state
Calculating the average value of the apparent characteristics and the geometric characteristics in the volume image through the face image in the sequence natural state in the SEMAINE data set to be used as a natural expression descriptor, and then carrying out normalization operation on the natural expression descriptor to generate natural expression characteristics;
(5.4) according to the apparent features, the geometric features and the natural expression features extracted from the face image sets in the multiple databases, training an AU detection model by using a Support Vector Regression (SVR) method of a linear kernel;
(5.5) extracting apparent characteristics, geometric characteristics and natural expression characteristics of the user face area in the source image, inputting the extracted apparent characteristics, geometric characteristics and natural expression characteristics into an AU detection model, and acquiring AU expression parameters of the user face in the current source image;
(6) inputting the AU expression parameters into a 3D model containing a shape difference model (BlendShape), and driving the model expression by using a Unity3D engine to realize the unconstrained 3D model expression migration of the user.
The invention aims to realize the following steps:
the invention relates to an unconstrained 3D expression migration method, which is realized by using a computer vision and probability statistics based method; firstly, detecting a face region by using the adaboost + harr characteristics, then extracting face geometric characteristics in the face region according to a Constrained Local Model (CLM) method, then extracting face expression parameters by using Support Vector Regression (SVR), finally inputting the expression parameters to control the face BlendShape of the 3D Model, synthesizing expression animation, and realizing the unconstrained 3D expression migration method.
Meanwhile, the unconstrained 3D expression migration method provided by the invention also has the following beneficial effects:
(1) the expression migration method applied to 3D movie making at present needs to utilize additional hardware auxiliary equipment to capture expression information, so that the problems of high system complexity, difficulty in popularization and the like are caused, and the method only adopts a common network camera and has good migration performance;
(2) and when the expression parameters of the user are extracted by a plurality of methods, the specific position of the face of the user needs to be marked, which is invasive to the user. (ii) a
(3) The invention has the advantages of low hardware cost, good real-time performance, high corresponding speed, no restriction to users and the like;
drawings
FIG. 1 is a flow chart of an unconstrained 3D expression migration method of the present invention;
FIG. 2 is a flow chart of detecting facial region features;
FIG. 3 is a flow chart of expression parameter extraction;
fig. 4 is a diagram illustrating the expression state of the prefabricated 3D model.
Detailed Description
The following description of the embodiments of the present invention is provided in order to better understand the present invention for those skilled in the art with reference to the accompanying drawings. It is to be expressly noted that in the following description, a detailed description of known functions and designs will be omitted when it may obscure the subject matter of the present invention.
Examples
FIG. 1 is a flow chart of an unconstrained 3D expression migration method according to the present invention.
In this embodiment, as shown in fig. 1, an unconstrained 3D expression migration method according to the present invention includes the following steps:
s1, training the face shape model off line and obtaining relevant parameters in the model
S1.1, training face images of which relevant parameters of a face shape model need to be manually calibrated at key point coordinates, wherein an AFLW database is used for downloading the face images with characteristic point labels from the AFLW database to serve as a face image set;
s1.2, constructing a human face shape model:
wherein,
representing the average face shape, P is a matrix of principal components of the face shape variation, P ═ P1,P2...Pk]B is the weight vector of the face shape change, B ═ B1,b2.,..,bk]T；
S1.3, using the face image set as input, comparing the true value of the characteristic points marked in the image set with the output result of the model, and iteratively updating the value of the face model parameter to be close to the true value:
setting M face images in the face image set, wherein each face image has N feature points, and the coordinate of the ith feature point is recorded as (x)i,yi)，i＝1,2,…,N；
The feature point vector composed by jth face image uses x(j)＝[x1 y1 x2 y2...xN yN]TIndicate, average face shape
Comprises the following steps:
subtracting the average face shape from the feature point vector formed by each face image to obtain a shape change matrix with the average value of 0
Then, PCA extraction matrix is used for principal component analysis
Characteristic vector P ofcAnd a corresponding characteristic value lambdacC is 1,2, …, min (M, N), and then the first k eigenvectors are selected to form a matrix P consisting of main components of the face shape change in a column discharge mode;
s1.4, respectively modeling in a neighborhood range of m multiplied by m by taking the coordinate position of each feature point as a center, then training a scoring model by using a Support Vector Machine (SVM), scoring m multiplied by m points in the neighborhood range by using the trained scoring model to obtain m multiplied by m scoring results, and forming a scoring response graph of each feature point by using the m multiplied by m scoring results;
fitting each scored response graph to a quadratic function r (x, y) ═ a (x-x)0)2+b(y-y0)2+ c, wherein (x)0,y0) The position coordinates of the center point of the response diagram, namely the coordinates of each characteristic point, a, b and c are fitting parameters, and (x, y) the position coordinates of the rest points except the center point in the response diagram;
the position coordinates of the other points are respectively substituted into the quadratic function to obtain the maximum value R (x, y) in the response diagram, and then sigma (R (x, y) -R (x, y))2Solving the values of a, b and c;
s2, acquiring an image containing a user face in real time by a camera, inputting the image as a data source, and marking the image as a source image;
s3, detecting the face area of the user in the source image by using a haar feature and adaboost face detection algorithm;
s4, using the shape model correlation parameters solved in the training phase:
P，λjand a, b and c, inputting the initialized key point position into a model by combining with the image information of the face area, and outputting the model so as to detect the feature point of the current face area, wherein the specific detection process is described below by combining with the figure 2.
According to the constructed human face shape model, a human face shape model can be initialized on a detected human face area, and each point is enabled to search for an optimal matching point in a neighborhood range according to the position of a human face characteristic point calculated by the human face shape model, and the specific steps are as follows:
s4.1, according to the human face shape model:
and parameters
P, using a shape-changing weight vector B ═ B1,b2.,..,bk]TAcquiring all initial feature points in the face region;
s4.2, obtaining a scoring response graph of each initial characteristic point according to the method in the step S1.4;
s4.3, by
As an objective function, wherein (x)p,yp) The position coordinate of the p-th point in the scoring response graph is represented, and beta refers to the weight value of the shape constraint; converging f (x) to a maximum value through iteration, wherein a point corresponding to the maximum value is an optimal matching point searched in a neighborhood range, and substituting a weight vector of the corresponding shape change into the face shape model at the moment to acquire all feature points in the face region;
and S4.4, taking the updated shape change weight vector B as a new shape model parameter, repeating the steps S4.2-S4.3 until the updated shape change weight vector B is basically unchanged, and outputting the position of the face feature point corresponding to the final face shape model.
S5, parameterizing the expression features of the user by using an Action Unit (AU) detection algorithm with normalized personal characteristics according to the positions of the facial feature points, and the specific flowchart is shown in fig. 3.
After all the feature points in the face area of the user are acquired, the feature points need to be further extracted as parametric information under a fixed rule, and then expression animation synthesis can be further performed. The invention uses the commonly used human face expression motion Unit (AU) rule to parameterize the human face expression. The actual effect of AU classifiers is very dependent on the training data due to the large variability between individual users. Aiming at the problem, the invention combines the geometric characteristics and the apparent characteristics of the facial expression of the user and trains an AU classifier under a cross-database, and the specific process is as follows:
s5.1, extracting apparent features of human faces
Using a face image set with AU parameter labels in a plurality of databases, scaling the face region of each face image to 112 × 112, and then extracting a gradient histogram of each face image, specifically, using 2 × 2 unit blocks, each unit block being 8 × 8 pixels, to form 4464-dimensional vector features describing the face. In order to reduce the quantity of HOGs (human body features), extracting gradient feature principal components in a gradient histogram by using a Principal Component Analysis (PCA) method on the basis, namely selecting the first k features as apparent features of a human face; in the embodiment, the data of a plurality of databases are used for extracting apparent features with stronger generalization capability when the PCA performs data dimension reduction;
s5.2, extracting geometric features of human face
Extracting human face feature points from the step S4 to be used as human face geometric features;
s5.3, extracting expression in natural state
Because of individual differences, for example, some faces can present different conditions like smiling or frowning in a relaxed state, and it is very difficult to acquire certain facial expressions under the condition that the natural expressions of the individual are unknown, after the apparent features and the geometric features are acquired, the expressions in the natural state need to be extracted. The method relies on that the human face in the sequence image is basically in a natural state, so that in the embodiment, the average value of the apparent characteristic and the geometric characteristic in the volume image is calculated through the human face image in the sequence natural state in the SEMAINE data set to be used as a natural expression descriptor, and then the natural expression descriptor is subjected to normalization operation to generate a natural expression characteristic;
s5.4, training an AU detection model by using a Support Vector Regression (SVR) method of a linear kernel according to the apparent characteristics, the geometric characteristics and the natural expression characteristics extracted from the face image set in the multiple databases;
s5.5, respectively extracting the apparent characteristics, the geometric characteristics and the natural expression characteristics of the user face area of the source image according to S5.1-S5.3, inputting the extracted apparent characteristics, the geometric characteristics and the natural expression characteristics into an AU detection model, and acquiring AU expression parameters of the user face of the current source image;
s6, the animation synthesis method comprises a muscle model driving mode, a shape difference model driving mode and the like, and as the facial expression is higher in deformation complexity relative to limb actions, a plurality of skeleton nodes need to be arranged in the bone driving mode, the driving links are quite complex, the shape difference model is simple and effective, one shape difference model is shown in figure 4, only partial expression states need to be preset, and the final expression animation is synthesized in a linear fusion mode. Therefore, the method uses AU expression parameters to be input into a 3D model containing a shape difference model (blend shape), and uses a Unity3D engine to drive the model expression, so that the unconstrained 3D model expression migration of the user is realized. .
Although illustrative embodiments of the present invention have been described above to facilitate the understanding of the present invention by those skilled in the art, it should be understood that the present invention is not limited to the scope of the embodiments, and various changes may be made apparent to those skilled in the art as long as they are within the spirit and scope of the present invention as defined and defined by the appended claims, and all matters of the invention which utilize the inventive concepts are protected.
Claims (1)
1. An unconstrained 3D expression migration method is characterized by comprising the following steps:
(1) off-line training a face shape model and obtaining relevant parameters in the model;
(1.1) downloading a face image with characteristic point labels from a face database to serve as a face image set;
(1.2) constructing a human face shape model:
wherein,
representing the average face shape, P is a matrix of principal components of the face shape variation, P ═ P1,P2...Pk]B is a weight vector of the face shape change,
(1.3) calculating relevant parameters of the face shape model by using the face image set;
setting M face images in the face image set, wherein each face image has N feature points, and the coordinate of the ith feature point is recorded as (x)i,yi)，i＝1,2,…,N；
The feature point vector composed by jth face image uses x(j)＝[x1 y1 x2 y2...xN yN]TIndicate, average face shape
Comprises the following steps:
subtracting the average face shape from the feature point vector formed by each face image to obtain a shape change matrix with the average value of 0
Then, PCA extraction matrix is used for principal component analysis
Characteristic vector P ofcAnd a corresponding characteristic value lambdacC is 1,2, …, min (M, N), and then the first k eigenvectors are selected to form a matrix P consisting of main components of the face shape change in a column discharge mode;
respectively modeling in a neighborhood range of m multiplied by m by taking the coordinate position of each feature point as a center, then training a scoring model by using a Support Vector Machine (SVM), scoring m multiplied by m points in the neighborhood range by using the trained scoring model to obtain m multiplied by m scoring results, and forming a scoring response graph of each feature point by using the m multiplied by m scoring results;
fitting each scored response graph to a quadratic function r (x, y) ═ a (x-x)0)2+b(y-y0)2+ c, wherein (x)0,y0) The position coordinates of the center point of the response diagram, namely the coordinates of each characteristic point, a, b and c are fitting parameters, and (x, y) the position coordinates of the rest points except the center point in the response diagram;
the position coordinates of the other points are respectively substituted into the quadratic function to obtain the maximum value R (x, y) in the response diagram, and then sigma (R (x, y) -R (x, y))2Solving the values of a, b and c;
(2) acquiring an image containing the face of a user in real time, and marking the image as a source image;
(3) detecting a face region of a user in the source image by using a haar feature + adaboost face detection algorithm;
(4) solved using a training phase
P,λcA, b and c, detecting the feature points of the current face area;
(4.1) according to the human face shape model:
and parameters
P, using weight vectors of shape changes
Acquiring all initial feature points in the face region;
(4.2) obtaining a scoring response graph of each initial characteristic point according to the method in the step (1.3);
(4.3) by
As an objective function, wherein (x)p,yp) The position coordinate of the p-th point in the scoring response graph is represented, and beta refers to the weight value of the shape constraint; enabling f (x) to converge to the maximum value through iteration, and substituting the weight vector of the corresponding shape change into the face shape model at the moment to acquire all feature points in the face region;
(4.4) taking the updated shape change weight vector B as a new shape model parameter, repeating the steps (4.2) - (4.3) until the updated shape change weight vector B is kept unchanged, and outputting the position of a face feature point corresponding to the final face shape model;
(5) adopting an expression parameter extraction algorithm based on an Action Unit (AU) to parameterize the expression characteristics of the user according to the positions of the facial feature points;
(5.1) extracting the apparent features of the human face;
using a face image set with AU parameter labels in a plurality of databases, scaling the face region scale of each face image to a fixed size, then extracting a gradient histogram of each face image, extracting gradient features in the gradient histogram by using a Principal Component Analysis (PCA), and selecting the first k features as face apparent features;
(5.2) extracting geometric features of the human face;
extracting human face feature points from the step (4) as human face geometric features;
(5.3) extracting the expression in a natural state;
calculating the average value of the apparent characteristics and the geometric characteristics in the volume image through the face image in the sequence natural state in the SEMAINE data set to be used as a natural expression descriptor, and then carrying out normalization operation on the natural expression descriptor to generate natural expression characteristics;
(5.4) according to the apparent features, the geometric features and the natural expression features extracted from the face image sets in the multiple databases, training an AU detection model by using a Support Vector Regression (SVR) method of a linear kernel;
(5.5) extracting apparent characteristics, geometric characteristics and natural expression characteristics of the user face area in the source image, inputting the extracted apparent characteristics, geometric characteristics and natural expression characteristics into an AU detection model, and acquiring AU expression parameters of the user face in the current source image;
(6) inputting the AU expression parameters into a 3D model containing a shape difference model (BlendShape), and driving the model expression by using a Unity3D engine to realize the unconstrained 3D model expression migration of the user.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201810124168.1A CN108363973B (en) | 2018-02-07 | 2018-02-07 | Unconstrained 3D expression migration method |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201810124168.1A CN108363973B (en) | 2018-02-07 | 2018-02-07 | Unconstrained 3D expression migration method |
Publications (2)
| Publication Number | Publication Date |
|---|---|
| CN108363973A CN108363973A (en) | 2018-08-03 |
| CN108363973B true CN108363973B (en) | 2022-03-25 |
Family
ID=63005138
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN201810124168.1A Active CN108363973B (en) | 2018-02-07 | 2018-02-07 | Unconstrained 3D expression migration method |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN108363973B (en) |
Families Citing this family (10)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN109493403A (en) * | 2018-11-13 | 2019-03-19 | 北京中科嘉宁科技有限公司 | A method of human face animation is realized based on moving cell Expression Mapping |
| CN110197167B (en) * | 2019-06-05 | 2021-03-26 | 清华大学深圳研究生院 | Video motion migration method |
| CN110647636B (en) * | 2019-09-05 | 2021-03-19 | 深圳追一科技有限公司 | Interaction method, interaction device, terminal equipment and storage medium |
| CN110728193B (en) * | 2019-09-16 | 2022-10-04 | 连尚（新昌）网络科技有限公司 | Method and device for detecting richness characteristics of face image |
| CN110889894A (en) * | 2019-10-25 | 2020-03-17 | 中国科学院深圳先进技术研究院 | Three-dimensional face reconstruction method and device and terminal equipment |
| CN111460945A (en) * | 2020-03-25 | 2020-07-28 | 亿匀智行(深圳)科技有限公司 | Algorithm for acquiring 3D expression in RGB video based on artificial intelligence |
| CN112489689B (en) * | 2020-11-30 | 2024-04-30 | 东南大学 | Cross-database voice emotion recognition method and device based on multi-scale difference countermeasure |
| CN112541445B (en) * | 2020-12-16 | 2023-07-18 | 中国联合网络通信集团有限公司 | Facial expression migration method and device, electronic equipment and storage medium |
| CN114677739A (en) * | 2022-03-30 | 2022-06-28 | 北京字跳网络技术有限公司 | Facial expression capturing method and device, computer equipment and storage medium |
| CN115908655B (en) * | 2022-11-10 | 2023-07-14 | 北京鲜衣怒马文化传媒有限公司 | Virtual character facial expression processing method and device |
Citations (2)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN104951743A (en) * | 2015-03-04 | 2015-09-30 | 苏州大学 | Active-shape-model-algorithm-based method for analyzing face expression |
| CN106096557A (en) * | 2016-06-15 | 2016-11-09 | 浙江大学 | A kind of semi-supervised learning facial expression recognizing method based on fuzzy training sample |
-
2018
- 2018-02-07 CN CN201810124168.1A patent/CN108363973B/en active Active
Patent Citations (2)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN104951743A (en) * | 2015-03-04 | 2015-09-30 | 苏州大学 | Active-shape-model-algorithm-based method for analyzing face expression |
| CN106096557A (en) * | 2016-06-15 | 2016-11-09 | 浙江大学 | A kind of semi-supervised learning facial expression recognizing method based on fuzzy training sample |
Non-Patent Citations (4)
| Title |
|---|
| 《Markerless 3D Facial Motion Capture System》;Hwang, Y；et al;《ENGINEERING REALITY OF VIRTUAL REALITY 2012》;20121231;第8289卷;全文 * |
| 《Spontaneous facial micro-expression detection based on deep learning》;XiaoHong 等;《IEEE 13th International Conference on Signal Processing (ICSP)》;20161231;第1131-1133页第三节 * |
| 《人脸特征点跟踪方法的研究》;周帆;《中国优秀硕士学位论文全文数据库 信息科技辑》;20170315(第3期);第18页第2.6小节，第21-27页第3.2小节 * |
| 《真实感3D人脸表情合成技术研究进展》;万贤美，等;《计算机辅助设计与图形学学报》;20140228;第26卷(第2期);全文 * |
Also Published As
| Publication number | Publication date |
|---|---|
| CN108363973A (en) | 2018-08-03 |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| CN108363973B (en) | Unconstrained 3D expression migration method | |
| Lim et al. | Isolated sign language recognition using convolutional neural network hand modelling and hand energy image | |
| Deng et al. | MVF-Net: A multi-view fusion network for event-based object classification | |
| Cao et al. | Expressive speech-driven facial animation | |
| CN109472198B (en) | Gesture robust video smiling face recognition method | |
| Murtaza et al. | Analysis of face recognition under varying facial expression: a survey. | |
| CN111028319B (en) | Three-dimensional non-photorealistic expression generation method based on facial motion unit | |
| CN112329525A (en) | Gesture recognition method and device based on space-time diagram convolutional neural network | |
| CN111680550B (en) | Emotion information identification method and device, storage medium and computer equipment | |
| Liang et al. | Resolving ambiguous hand pose predictions by exploiting part correlations | |
| CN113255457A (en) | Animation character facial expression generation method and system based on facial expression recognition | |
| Jia et al. | Two-stream temporal convolutional networks for skeleton-based human action recognition | |
| Mohanty et al. | Rasabodha: Understanding Indian classical dance by recognizing emotions using deep learning | |
| Tur et al. | Evaluation of hidden markov models using deep cnn features in isolated sign recognition | |
| Wang et al. | A comprehensive survey of rgb-based and skeleton-based human action recognition | |
| Xu et al. | A high resolution grammatical model for face representation and sketching | |
| Tur et al. | Isolated sign recognition with a siamese neural network of RGB and depth streams | |
| Liang et al. | Egocentric hand pose estimation and distance recovery in a single RGB image | |
| Zhang et al. | Linearly augmented real-time 4D expressional face capture | |
| Kwolek et al. | Recognition of JSL fingerspelling using deep convolutional neural networks | |
| Aiman et al. | Angle based hand gesture recognition using graph convolutional network | |
| RU2755396C1 (en) | Neural network transfer of the facial expression and position of the head using hidden position descriptors | |
| Al Farid et al. | Single Shot Detector CNN and Deep Dilated Masks for Vision-Based Hand Gesture Recognition From Video Sequences | |
| Rao et al. | Sign Language Recognition using LSTM and Media Pipe | |
| Angelopoulou et al. | Evaluation of different chrominance models in the detection and reconstruction of faces and hands using the growing neural gas network |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination | ||
| GR01 | Patent grant | ||
| GR01 | Patent grant |