CN104408767A - Method for building sparse consistent three-dimensional human face mesh deformation model - Google Patents

Abstract

The invention discloses a method for building a sparse consistent three-dimensional human face mesh deformation model. The method for building the sparse consistent three-dimensional human face mesh deformation model includes steps that (1) training human face data pre-processing; (2) dictionary learning; (3) test human face deformation; (4) consistence alignment. The method for building the sparse consistent three-dimensional human face mesh deformation model is a full-automatic sparse deformation model method which uses geometrical relation constraints and corresponding relation constraints to carry out deformation and sparse coding on three-dimensional human faces to build the point-to-point dense correspondence between three-dimensional human faces. By means of the method for building the sparse consistent three-dimensional human face mesh deformation model, the correspondence between dense vertexes on three-dimensional human face data is very good.

Description

A kind of method setting up the sparse distorted pattern of consistance three-dimensional face grid

Technical field

The present invention relates to human face data computer processing method, particularly relate to a kind of with geometric relationship constraint with corresponding relation constraint is out of shape three-dimensional face and sparse coding is core technology, sets up the method for the full automatic sparse distorted pattern of point-to-point dense corresponding relation between three-dimensional face.

Background technology

In recent years, along with the develop rapidly of three-dimensional data acquiring technology, obtain accurately and in real time three-dimensional data and be tending towards practical.Representation due to three-dimensional data normally discrete, sparse, three-dimensional data be difficult to utilizes usual way to process easily.The reason of this situation is caused to be that different faces contains the summit of varying number and inconsistent border condition usually because the data that three-dimensional human face scanning obtains are irregularities.With regard to two three-dimensional faces, even if summit quantity is the same with border condition, their respective summits are also at random without chapter, and its topological structure of three-dimensional face curved surface obtained through Triangulation Algorithm is completely different.

Through development in recent years, the existing method can setting up corresponding relation between summit and summit between three-dimensional face automatically mainly can be summed up, be summarized as following a few class:

1) based on the method for closest approach.For given two three-dimensional datas, the method based on closest approach is found corresponding point by the thought of alignment and is alignd, and the corresponding relation between summit is decided by the space length between the two.The corresponding relation of this type of method establishment is quite coarse, time especially deviation ratio is larger between the model subjects related to, substantially cannot keep the dense alignment had on physiological significance between its corresponding point.

2) based on the method for Isometric Maps.Method based on Isometric Maps supposes that the geodesic line distance of on a model 2 should be similar to or be equal to the geodesic distance of corresponding point on another one model.Based on this hypothesis, corresponding relation can be calculated by manifold learning.For three-dimensional face data, the cavity of face mouth region often allows the hypothesis of these class methods be false, and produces the corresponding result of mistake thus.

3) elastic deformation model.Universal model to be deformed is deformed to a target triangle grid model by these methods under normal circumstances, after this distortion, corresponding relation between universal model and target triangle grid model naturally to set up, and has identical topological structure simultaneously.The defect of these class methods needs to go manual some unique points of demarcation artificially, therefore cannot fully automatically go to set up the dense corresponding relation of three-dimensional face model, and it is still too smooth to such an extent as to or can not represent the details of target faceform well to cross the faceform after the method distortion.

Summary of the invention

The invention provides a kind of method setting up the sparse distorted pattern of consistance three-dimensional face grid, the method, based on sparse Facial metamorphosis model, can fully automatically set up corresponding relation between summit dense between three-dimensional face.

Set up a method for the sparse distorted pattern of consistance three-dimensional face grid, its step is as follows:

(1), face data prediction is trained: comprise the demarcation of training face data characteristics point, general face's template data piecemeal and training human face data non-rigid alignment three part:

(2), sparse dictionary study: based on the pre-processed results of step (1), utilize geometric constraints and corresponding relation constraint to learn a sparse dictionary to training human face data;

(3), test Facial metamorphosis: obtain dictionary based on step (2), to given input test face, utilize geometric constraints and corresponding relation constraint, by the test Facial metamorphosis of general face's template to input;

(4), consistance alignment: obtain based on step (3) the test face that the face after being out of shape substitutes input, namely complete consistance alignment task.

By the inventive method, the effect obtained be fully automatically set up that summit between three-dimensional face is dense, the one-to-one relationship had in anthropometry meaning, it is by obtaining Generic face model to target face model deformation.

Described training of human face characteristic point scaling method, is characterized in carrying out manual demarcation to all training human face data and general face's template, and covers the critical area of face.

Described general face's template data, is each unique point of feature point pairs marked based on step (1), the center of a ball is fixed on a little, the human face region of ball inside forms a block; The point that random selecting is not covered to extracts the block made new advances by same way, until everyone point on the face is at least in a block.

The method of the described non-rigid alignment of training human face data, to the deformation of training face template by general face's template, substitute original human face data with deformation results and carry out follow-up dictionary learning, deformation process is based on smooth error term, data error item and unique point error term.This deformation process is namely by minimization of energy function E ₁carry out the non-rigid alignment distortion of face:

E ₁=λ ₁e _c+ λ ₂e _e+ λ ₃e _q(3) wherein, E _cunique point error term, E _esmooth error, E _qdata error item, λ ₁, λ ₂, λ ₃be the given parameter of experience, generally obtained by the method for training data being carried out to cross validation.

The method of geometric constraints is by minimizing smooth error function between Generic face model and target faceform and data error function obtains, what adopt in data error function is the mixed principle that closest approach rule and normal ray rule combine, geometrical correspondence φ (S (M '), S (M)) is defined as

φ(S(M′),D(M))＝λ ₁E _e+λ ₂E _q(4)

Wherein E _esmooth error, E _qdata error item, λ ₁and λ ₂be coefficient, be the given parameter of experience, generally obtained by the method for training data being carried out to cross validation.

Because Generic face model is that piecemeal is good, and the face of training data carries out deformation by general face's template to obtain, so the face in training data is nature divided block.For each block on face, corresponding relation constraint carries out formalization by the process solving following sparse coding:

$\min {\left|\right|C^{k}y-D^k{x}^k\left|\right|}_2,s.t.{\left|\right|x^k\left|\right|}_0\≤T,\∀k.---\left(5\right)$

Wherein C ^kbeing partitioned matrix, being determined by piecemeal result, is known, and y is known human face data, D ^kand x ^kthe dictionary to be asked of kth block and corresponding coding.

Preferably, adopt the method for adaptive sparse degree threshold value to improve reconstruction precision in dictionary learning process, Specific Principles is: in former step iterative process, and we adopt less degree of rarefication threshold value, allow larger reconstruction error; In iterative process below, arrange larger degree of rarefication threshold value, duplicate removal builds the shape of some local.

Described is out of shape test face, refers to input one test face, retrains, by solving following optimization problem based on geometric constraints and corresponding relation its distortion:

Min||S (M ')-Dx|| ₂+ μ || x|| ₀+ φ (S (M '), S (M)). (6) are min||S (M ')-Dx|| wherein ₂+ μ || x|| ₀represent corresponding relation constraint, φ (S (M '), S (M)) represents geometric constraints, and M ' is the face after the distortion asked, and x is the coding of its correspondence, and M is the test face of input.

The problem automatically setting up corresponding relation between summit dense between three-dimensional face based on sparse Facial metamorphosis model can be defined as follows, input a three-dimensional face, basic thinking is the brand-new three-dimensional face of matching one, and this new face curved surface must meet three conditions: 1) and general three-dimensional face model there is identical topological structure; 2) and its geometric configuration is similar with the geometric configuration height of the face of input; 3) on the basis of above 2, this brand-new face and general three-dimensional face have dense and have the corresponding relation of the summit opposite vertexes on physiological significance.

Sparse Facial metamorphosis model proposed by the invention uses two constraints: geometric constraints and corresponding relation constraint meet three above conditions:

1) method that geometric configuration bundle is out of shape by three-dimensional face realizes, the present invention designs, uses mixed-blood pattern in the process of Facial metamorphosis, merge based on the method for normal ray mutually with the method for closest approach, maximize favourable factors and minimize unfavourable ones, while Fast Convergent, reduce the distortion of high curvature portions in surf deform process as far as possible.

2) foundation of corresponding relation constraint is then based on a hypothesis, namely " suppose two people on the face certain, two summits be mutually corresponding, there is identical physiologic meaning, the signal of geometry information that so these two summits and neighborhood thereof comprise is distributed in same stream shape space." we can use the several sparse item in certain dictionary trained to represent; and this dictionary can be obtained by study; and the distributed intelligence of local surface around these corresponding vertexs can be comprised, use sparse coding and this sparse dictionary then can generate the corresponding three-dimensional face being positioned at same flow shape space.

The present invention proposes a kind of sparse representation algorithm based on piecemeal and solves corresponding relation restricted problem.By continuous iteration, be progressively out of shape the position on each summit on face, until finally restrain.In initial iterative step, we select less degree of rarefication threshold value, although less sparse threshold value this may bring larger sign error to distortion face, but can set up the corresponding relation between the summit between the better overall situation.After iteration progressively, relatively large degree of rarefication threshold value is adopted to describe the local shape of face better.By above step, the summit in newly-built three-dimensional face can gradually be moved until restrain toward correct position, reaches overall corresponding relation and sets up.

Sparse Facial metamorphosis model proposed by the invention automatically can set up the corresponding relation between three-dimensional face summit, needs the artificial part participated in only to have training part.We have carried out corresponding experiment on a disclosed three-dimensional face data set, experimental result shows, between summit dense in the three-dimensional face data obtained by the method setting up the sparse Facial metamorphosis model of consistance three-dimensional face grid of the present invention, corresponding relation is very good.

Accompanying drawing explanation

Fig. 1 is the processing procedure schematic diagram of the inventive method.

Fig. 2 is overall face Dividing Curve Surface schematic diagram in the inventive method.

Fig. 3 is the piecemeal schematic diagram of the point of face periphery in face Dividing Curve Surface in the inventive method.

Embodiment:

Fig. 1 describes the processing procedure framework of the inventive method, and include the non-rigid alignment of face of training part, the Facial metamorphosis part of dictionary learning and part of detecting, is specially:

Training part

1, unique point is demarcated and Generic face model piecemeal

Human face data pre-service comprises unique point and demarcates and Generic face model piecemeal two parts.

First we carry out manual unique point to all training human face data and general face's template and demarcate, the unique point between every two faces to have in anthropometry meaning one to one, these Vertex cover critical area of face.

As shown in Figure 2, we first put a radius is the ball of r to the method for partition of face Dividing Curve Surface, to each general face template M _ron unique point, we are fixed on the center of ball a little, and the human face region of ball inside forms a block.The point of the black in random selecting figure on face extracts the block made new advances by same way, until everyone point on the face is at least in a block.In addition, we are the point of face periphery, and the point namely near face is considered separately, as shown in Figure 3, and some formation one the independently block near face.Notice the overlap existed between adjacent block and block a little.In our experiment, the value of r is empirically set to 9, and we finally obtain 206 blocks.

2, non-rigid alignment

In order to learn one can be set up corresponding relation between face dictionary with sparse representation, we need face grid corresponding in a series of anthropometry meaning to be used as training.We are by being out of shape general face's template to training data, and the new three-dimensional face obtained replaces original three-dimensional face to carry out follow-up training.After completing the process of this non-rigid alignment, the three-dimensional face of all training datas just completes alignment automatically.In addition, because in upper step, we carry out face piecemeal to general face's template, and in follow-up human face data pre-service, we be with general face's template deformation after face carry out the face of alternative training data, this means that for all faces in training data be all a point good block.

First we define series of sign, and these symbols run through the present invention all the time, hereafter repeat no more.Definition M _r/ M ' is general face's template/Initial Face template, and M is the target face in tranining database, v _i=(x _i, y _i, z _i) ^t, i=1,2,3 ..., n represents M _rpoint on/M ', v _{c (i)}, i=1,2,3 ..., l represents its unique point.P _i, i=1,2,3 ..., m represents the point on M, u _i, i=1,2,3 ..., l is the unique point on M, and v _{c (i)}and u _ione to one, o _i, i=1,2,3 ..., n is side-play amount to be asked.We carry out the pre-service distortion of face by the formula minimized below

E ₁＝λ ₁E _c+λ ₂E _e+λ ₃E _q(7)

λ ₁, λ ₂, λ ₃be the given parameter of experience, generally obtained by the method for training data being carried out to cross validation.Wherein, unique point error term E _cbe defined as

$E_c=\underset{i=1}{\overset{l}{\mathrm{\Σ}}}{\left|\right|u_i-(v_{c\left(i\right)}+o_{c\left(i\right)})\left|\right|}^2---\left(8\right)$

Smooth error term E _ebe defined as

$E_e=\underset{i=1}{\overset{n}{\mathrm{\Σ}}}\underset{j\∈N_i}{\mathrm{\Σ}}{\left|\right|o_i-o_j\left|\right|}^2---\left(9\right)$

Wherein N _irepresent some v _ithe indexed set of neighbours.Data error item E _qthe mixing principle adopting closest approach rule and normal ray rule to combine, is defined as

$E_q=\underset{i=1}{\overset{l}{\mathrm{\Σ}}}{\left|\right|{\mathrm{\η}}_1{p}_{\mathrm{vi}}+{\mathrm{\η}}_2{g}_{\mathrm{vi}}-(v_i+o_i)\left|\right|}^2---\left(10\right)$

P _virepresent distance M on M _rv on/M ' _iclosest approach, g _virepresent the point found by normal ray method, η ₁and η ₂be weight, and have η ₁+ η ₂=1.

3, dictionary learning

By the process of above-mentioned non-rigid alignment, the corresponding relation between the three-dimensional face that training data is concentrated just has established.Based on this corresponding relation, dictionary learning just can carry out.

Y is a matrix, its i row y _irepresent the three-dimensional face signal in a training set, for each piece of P _k, the dictionary D of its correspondence ^klearn by solving following problem

$\begin{array}{cc}\underset{D^k,X^k}{\min }{\left|\right|C^{k}Y-D^k{X}^k\left|\right|}_F^2& s.t.{\left|\right|x_i^k\left|\right|}_0\≤T,\∀i\end{array}---\left(11\right)$

Wherein C ^kmatrix is used for from face grid vector y _imiddle extraction block P _ksignal form a vector, be one with the relevant scalar matrix of piecemeal result, X ^ki-th row of matrix that i-th face is at block P _kon code, this optimization problem can solve D by KSVD method ^kand X ^k.

Part of detecting

1, degree of rarefication threshold value

First we define degree of rarefication threshold value.Because sparse face deformation model adopts the way of iteration, by separating the sparse representation of face characteristic, face deformation model one step is utilized to obtain correspondence results final accurately.In former step iterative process, we adopt less degree of rarefication threshold value T; In iterative process below, T is set to larger value, and duplicate removal builds the shape of some local.The upper bound T of given degree of rarefication threshold value _ewith iterations S, we decide the value of T with two functions, i.e. linear function

And exponential function

Wherein s represents current iteration number of times, represent lower floor operation, due to T _ethe degree of rarefication represented, it is set to be not more than 1% of dictionary columns usually.

2, face test

Input a test face, the method for its distortion retrained, by solving following optimization problem based on geometric constraints and corresponding relation:

Min||S (M ')-Dx|| ₂+ μ || x|| ₀+ φ (S (M '), S (M)). (14) notice that we are the dictionary D corresponding to each block face block in fact ^kcarry out solving x ^k, namely calculate each block human face region block, for the purpose of statement, represent with D and x here and solve all people's face block, is also hereafter like this.Wherein min||S (M ')-Dx|| ₂+ μ || x|| ₀represent corresponding relation constraint, φ (S (M '), S (M)) represents geometric constraints (providing definition later), and M ' is the face after the distortion asked, x is the coding of its correspondence, and M is the test face of input.

Formula (14) is exactly the core solving sparse Facial metamorphosis model herein.Dictionary D is obtained by training, only has x and M unknown in formula (14).Namely we use the thought of iteration to go to solve corresponding relation constraint by alternately solving geometric constraints and corresponding relation constraint instead of coming together to solve them.This iterative process is as follows:

Initial face is general face template M _r, in each iteration t, M ' ^t-1obtained by last iterative computation

Step 1: the face that we obtain by previous step, as input, uses above-mentioned face deformation model method, solves geometric constraints φ (S (M '), S (M)), can obtain an interim face M ' ^*.

Step 2: then go to separate corresponding relation constraint min||S (M ')-Dx|| as input with this interim face ₂+ μ || x|| ₀, utilize formula (12) or formula (13), choose a suitable degree of rarefication threshold value, x is calculated by OMP algorithm, M ' ^tthen calculated by Dx, and as entering the input face of next iterative loop.

Here geometric constraints φ (S (M '), S (M)) is defined as

φ(S(M′),S(M))＝λ ₁E _e+λ ₂E _q(15)

λ ₁, λ ₂be the given parameter of experience, generally obtained by the method for training data being carried out to cross validation.Wherein E _eand E _qbe defined at preceding formula (9) and formula (10).

The present invention uses the data in 3DBU-FED database to test.We define the distance of three kinds of RMSE to assess our corresponding relation precision:

$R_n=\sqrt{\frac{{\mathrm{\Σ}}_{i=1}^n{\left|\right|v_i-p_{\mathrm{vi}}\left|\right|}^2}{n}}---\left(16\right)$

$R_{\mathrm{lf}}=\sqrt{\frac{{\mathrm{\Σ}}_{i=1}^n{\left|\right|v_i-{v_i}^{\mathrm{lf}}\left|\right|}^2}{n}}---\left(17\right)$

$R_l=\sqrt{\frac{{\mathrm{\Σ}}_{i=1}^k{\left|\right|u_i-{u_i}^{\′}\left|\right|}^2}{k}}---\left(18\right)$

U _i, i=1,2,3 ..., k represents the unique point of demarcation, the number of k representation feature point.M _lfrepresent a real human face in test human face data, the deformation method being guiding with the unique point of demarcating obtains from M distortion.V _i, _vias defined above.

R _ncalculate institute on M ' a little with M on corresponding to the mean distance of distance of closest approach, this reflects reconstruction precision.R _lfcalculate the mean distance of corresponding point between M ' and M, this reflects the precision of corresponding relation.R _lcalculate the mean distance between character pair point between M ' and M, which reflects the precision of Feature point correspondence relation.In addition, we with the distance between every two unique points and average D _lthe precision of the model after distortion is described, namely as another one distance

$D_l=\underset{i=1}{\overset{k}{\mathrm{\Σ}}}\underset{j=1}{\overset{k}{\mathrm{\Σ}}}\sqrt{{\left(\right||u_i-u_j||-||{u_i}^{\′}-{u_j}^{\′}|\left|\right)}^2/k^2}---\left(19\right)$

U _i, i=1,2,3 ..., k represents the unique point of demarcation, the number of k representation feature point.

Table 1, based on the sparse Facial metamorphosis model experiment results of Different Strategies, illustrates under different parameters, R _n, R _lf, R _l, D _lmean value and their standard deviation (SD), the method for the degree of rarefication threshold value of " Linear " corresponding formula (12), the method for the degree of rarefication threshold value of " Exp " corresponding formula (13).

Claims (8)

1. set up a method for the sparse distorted pattern of consistance three-dimensional face grid, its step is as follows:

(1), face data prediction is trained: comprise the demarcation of training face data characteristics point, general face's template data piecemeal and training human face data non-rigid alignment three part;

(2), dictionary learning: based on the pre-processed results of step (1), utilize geometric constraints and corresponding relation constraint to learn a sparse dictionary to training human face data;

(3), test Facial metamorphosis: obtain sparse dictionary based on step (2), to given input test face, utilize geometric constraints and corresponding relation constraint, by the test Facial metamorphosis of general face's template to input;

(4), consistance alignment: obtain based on step (3) the test face that the face after being out of shape substitutes input, namely complete consistance alignment task.

2. the method setting up the sparse distorted pattern of consistance three-dimensional face grid according to claim 1, it is characterized in that, training of human face characteristic point in step (1) is demarcated, be that manual demarcation is carried out to all training human face data and general face's template, and cover the critical area of face.

3. the method setting up the sparse distorted pattern of consistance three-dimensional face grid according to claim 1, it is characterized in that, general face's template data piecemeal in step (1), to each unique point on three-dimensional face, the center of a ball is fixed on a little, and the human face region of ball inside forms a block.Point on the face that random selecting is not covered to uses the same method and extracts a block making new advances, until everyone point on the face is at least in a block.

4. the method setting up the sparse distorted pattern of consistance three-dimensional face grid according to claim 1, it is characterized in that, the non-rigid alignment of training human face data in step (1), to the deformation of training face template by general face's template, substitute original human face data with deformation results and carry out follow-up dictionary learning, deformation process is based on smooth error term, data error item and unique point error term.

5. the method setting up the sparse distorted pattern of consistance three-dimensional face grid according to claim 1, it is characterized in that, geometric constraints in step (2), that what wherein adopt in data error function is the mixed principle that closest approach rule and normal ray rule combine by minimizing smooth error function between Generic face model and target faceform and data error function obtains.

6. the method setting up the sparse distorted pattern of consistance three-dimensional face grid according to claim 1, is characterized in that, the corresponding relation constraint in step (2), being for each block on face, carrying out formalization by solving a sparse dictionary

$\min \left|\right|C^{k}y-D^k{x}^k{\left|\right|}_2,s.t.\left|\right|x^k{\left|\right|}_0\≤T,\∀k.---\left(1\right)$

Wherein C ^kbeing partitioned matrix, being determined by piecemeal result, is known, and y is known human face data, D ^kand x ^kthe dictionary to be asked of kth block and corresponding coding; And adopt the method for adaptive sparse degree threshold value to improve reconstruction precision in dictionary learning process.

7. the method setting up the sparse distorted pattern of consistance three-dimensional face grid according to claim 6, it is characterized in that, the method of described adaptive sparse degree threshold value is: in former step iterative process, adopts less degree of rarefication threshold value, allows larger reconstruction error; In iterative process below, if larger degree of rarefication threshold value, duplicate removal builds the shape of some local.

8. the method setting up the sparse distorted pattern of consistance three-dimensional face grid according to claim 1, it is characterized in that, test Facial metamorphosis in step (3), refer to input one test face, its distortion is retrained, by solving following optimization problem based on geometric constraints and corresponding relation:

min||S(M′)-Dx|| ₂+μ||x|| ₀+φ(S(M′)，S(M)). (2)

Wherein min||S (M ')-Dx|| ₂+ μ || x|| ₀represent corresponding relation constraint, φ (S (M '), S (M)) represents geometric constraints, and M ' is the face after the distortion asked, and x is the coding of its correspondence, and M is the test face of input.