CN100483462C - Establishing method of human face 3D model by fusing multiple-visual angle and multiple-thread 2D information - Google Patents
Establishing method of human face 3D model by fusing multiple-visual angle and multiple-thread 2D information Download PDFInfo
- Publication number
- CN100483462C CN100483462C CNB02146278XA CN02146278A CN100483462C CN 100483462 C CN100483462 C CN 100483462C CN B02146278X A CNB02146278X A CN B02146278XA CN 02146278 A CN02146278 A CN 02146278A CN 100483462 C CN100483462 C CN 100483462C
- Authority
- CN
- China
- Prior art keywords
- face
- point
- dimensional
- tstart
- prime
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Expired - Fee Related
Links
- 238000000034 method Methods 0.000 title claims description 73
- 238000004422 calculation algorithm Methods 0.000 claims abstract description 48
- 238000012937 correction Methods 0.000 claims description 45
- 210000001508 eye Anatomy 0.000 claims description 32
- 101000767160 Saccharomyces cerevisiae (strain ATCC 204508 / S288c) Intracellular protein transport protein USO1 Proteins 0.000 claims description 30
- 230000008859 change Effects 0.000 claims description 30
- 230000008878 coupling Effects 0.000 claims description 27
- 238000010168 coupling process Methods 0.000 claims description 27
- 238000005859 coupling reaction Methods 0.000 claims description 27
- 238000001514 detection method Methods 0.000 claims description 24
- 239000011159 matrix material Substances 0.000 claims description 24
- 238000013519 translation Methods 0.000 claims description 24
- 230000001815 facial effect Effects 0.000 claims description 21
- 230000000007 visual effect Effects 0.000 claims description 18
- 238000012545 processing Methods 0.000 claims description 14
- 101150081243 STA1 gene Proteins 0.000 claims description 12
- 238000013507 mapping Methods 0.000 claims description 11
- 238000006243 chemical reaction Methods 0.000 claims description 5
- 239000004744 fabric Substances 0.000 claims description 5
- 238000012897 Levenberg–Marquardt algorithm Methods 0.000 claims description 4
- 239000000203 mixture Substances 0.000 claims description 4
- 206010053238 Amimia Diseases 0.000 claims description 3
- 230000004927 fusion Effects 0.000 claims description 3
- 230000002427 irreversible effect Effects 0.000 claims description 3
- 230000008676 import Effects 0.000 claims description 2
- 238000005516 engineering process Methods 0.000 abstract description 4
- 238000010276 construction Methods 0.000 abstract 1
- 230000000977 initiatory effect Effects 0.000 abstract 1
- 230000006870 function Effects 0.000 description 25
- 230000014616 translation Effects 0.000 description 18
- 101100446506 Mus musculus Fgf3 gene Proteins 0.000 description 14
- 230000008569 process Effects 0.000 description 12
- 238000004364 calculation method Methods 0.000 description 8
- 238000010586 diagram Methods 0.000 description 4
- 150000001875 compounds Chemical class 0.000 description 3
- 210000001061 forehead Anatomy 0.000 description 3
- 230000007935 neutral effect Effects 0.000 description 3
- 238000002474 experimental method Methods 0.000 description 2
- 239000000284 extract Substances 0.000 description 2
- 210000001202 rhombencephalon Anatomy 0.000 description 2
- 238000013459 approach Methods 0.000 description 1
- 210000005252 bulbus oculi Anatomy 0.000 description 1
- 210000003467 cheek Anatomy 0.000 description 1
- 230000003247 decreasing effect Effects 0.000 description 1
- 230000007547 defect Effects 0.000 description 1
- 230000007812 deficiency Effects 0.000 description 1
- 238000006073 displacement reaction Methods 0.000 description 1
- 238000000605 extraction Methods 0.000 description 1
- 238000007667 floating Methods 0.000 description 1
- 230000005484 gravity Effects 0.000 description 1
- 238000010376 human cloning Methods 0.000 description 1
- 238000012986 modification Methods 0.000 description 1
- 230000004048 modification Effects 0.000 description 1
- 230000003287 optical effect Effects 0.000 description 1
- 210000000056 organ Anatomy 0.000 description 1
- 238000002203 pretreatment Methods 0.000 description 1
- 238000011084 recovery Methods 0.000 description 1
- 238000009418 renovation Methods 0.000 description 1
- 238000013341 scale-up Methods 0.000 description 1
- 238000000926 separation method Methods 0.000 description 1
- 238000012360 testing method Methods 0.000 description 1
- 230000009466 transformation Effects 0.000 description 1
Images
Landscapes
- Image Analysis (AREA)
- Image Processing (AREA)
Abstract
The present invention belongs to the field of computer-aided human face image re-construction technology. The present invention features that human face as model establishing object is first photographed with two, one upper and one lower, stere ovideo cameras to obtain video sequence from the front to the side of the face with unchanged expression; the precise posture parameters in different times are then obtained via autoamtic posture estimating algorithm; after the initiation of model and posture, 3D information are extracted separately from two types of 2D threads including one stereo reconstruction type of the image and the other matching and correcting type of model projection contour and image contour; and the 3D face model is finally established.
Description
Technical field
The method for building up that merges the human face three-dimensional model of various visual angles, multi thread two-dimensional signal belongs to computer face image reconstruction technique field.
Background technology
Existing method according to image establishment three-dimensional face model lacks the strategy that extracts three-dimensional information and merge from multiple two-dimentional clue.Kang, people such as Sing Bing are at " image method and apparatus for semi-automatically mappinga face on to a wireframe topology " (United States Patent6, a kind of semi-automatic method according to the positive visual founder's face model of single width has been proposed in the patented technology 031,539).The principle of this method is: automatically detect a plurality of unique points on people's face image, and then according to the landmark points (landmark points) of positioning feature point face, obtain the displacement of all net points in people's face wire-frame model then in view of the above, create out and the corresponding faceform of this image.The major defect of this method is the information that does not make full use of each visual angle, and this has limited the levels of precision of model.
At " utilizing desktop video camera human cloning face " (" the 8th computer vision international conference collection of thesis ", 2001, second volume, pp.745-745) in the literary composition, people such as author Zhang Zhengyou, Z.Liu have proposed a kind of method of utilizing the corresponding faceform of establishment of two dimension in the monocular video.The principle of this method is: the manual image of selecting two width of cloth near front view, by people's face shape model of camera self-calibration and simplification, based on two dimension to using the model that the non-linear way of finding the solution obtains specific people.Though this method has been utilized various visual angles information, only be to use several images to go up corresponding two-dimensional points information and do not made full use of other shape constraining, the model that obtains is not accurate enough, especially abundant regional all the more so of texture.
These methods have some common shortcomings: (1) model is meticulous inadequately; (2) can not do the compatible animation of MPEG4 (moving picture experts group organizational standard 4).
Summary of the invention
The object of the present invention is to provide a kind of method for building up that merges the human face three-dimensional model of various visual angles, multi thread two-dimensional signal.
But the present invention relates to a kind of modeling method of animation three-dimensional face, at first take people's face of modeling object progressively forwards the side to from the front video sequence (shooting process people face is answered amimia variation) with demarcating good stereo camera; Be that a full automatic attitude algorithm for estimating is to obtain each accurate attitude parameter constantly in the sequence then; Then after model and attitude initialization, according to two class two-dimensional clues by different processing scheme extraction three-dimensional informations: a class is based between image corresponding point and carries out stereo reconstruction, and another kind of is coupling, the correction of model projection profile and image contour; At last, merge in various visual angles, extract, set up the faceform corresponding to the three-dimensional information of the two-dimentional clue of difference.This method and existing method ratio, more careful, more accurate shape that the faceform who obtains has, and can do the human face animation of MPEG4 compatibility.
The invention is characterized in:
It at first use with the people on the face, corresponding two the identical and outer polar curve direction video cameras substantially vertically of upper/lower positions people's face of taking modeling object under the situation of the amimia variation of people's face progressively changes about 90 video sequences of spending to the side from the front, gather up and down the two-way synchronization video with two identical capture cards again and import in the computing machine; Then, from the universal model Face of people's face
gBeginning is progressively made amendment according to the two-dimensional signal of the modeling object that obtains, and finally obtains specific people's faceform Face
s, the three-dimensional point coordinate among the just people who wherein the revises three-dimensional node tabulation P on the face, and people triangle surface tabulation F on the face remains unchanged promptly only changes mould shapes P and the topological structure F of model is constant; Particularly, it contains successively and has the following steps:
(1). input stereoscopic video sequence I
L, tAnd I
R, t:
L, R be corresponding to two video cameras, t=1 ..., N, the different images constantly of expression N frame: the two-dimensional array that I then is made up of pixel, each pixel comprise R, G, three components of B; The moment of t=1 is selected by people's judgement is manual corresponding to the positive attitude of people's face;
(2). attitude is estimated, is promptly found the solution the attitude parameter (being the rigid body translation coefficient) in the number of people rotary course:
Interframe movement with people's face in the above-mentioned image sequence of following rigid body translation formulate:
M
t+1=R
tM
t+T
t,t=1,...,N-1,
M
t, M
T+1The people's coordinate of any point in the L camera coordinate system on the face when representing t frame, t+1 frame respectively; M
t, M
T+1Being people's same position on the face at difference three-dimensional coordinate constantly, promptly is the three-dimensional corresponding point of interframe; Motion with reliable three-dimensional point just can be estimated attitude parameter;
{ R
t, T
t, t=1 ..., N-1} is rotation and the translational component of t frame to t+1 frame attitude parameter;
Attitude is estimated to contain successively to have the following steps:
(2.1), the initial moment: carry out the detection and the coupling of two dimensional character earlier, obtain the three-dimensional point in the initial moment again by stereo reconstruction:
At first, feature detection frame t0=t is set, and at I
L, tMake two dimensional character on the figure and detect, its step is as follows:
With existing KLT algorithm computation I
L, tUnique point p on the figure (x=i, the image change degree G that y=j) locates (i, j):
(i+1 is the gray scale of this some place image j) to I, and the rest may be inferred by analogy;
With tested point p is that the window W at center is of a size of (2*blockx+1) * (2*blocky+1); And then obtain the minimum singular value sing_min of G with existing matlab kit, selecting the very big some p of sing_min again is detected unique point; Being designated as { p in t detected unique point of the moment
L, t(k) }, L, t presentation image sequence number, K (t) is the unique point sum, k is its label, 1≤k≤K (t);
Secondly, to two video camera L of synchronization, two width of cloth images that R shooting Same Scene obtains are that stereographic map is to making the two dimensional character coupling; The projection of same three-dimensional point in different images is two-dimensional points all, and it is corresponding to be called two dimension between them mutually, stereographic map to two-dimentional correspondence be called coupling:
For I
L, tCertain 1 p among the figure
L, t(k)=(x
L, t(k) y
L, t(k))
T, its match point p
R, t(k)=(x
R, t(k) y
R, t(k))
TShould be positioned at one by p
L, t(k) on Jue Ding the straight line, in this one-dimensional space, select to make the point of total gray difference minimum in the local window W as p
L, t(k) match point p
R, t(k), the expression formula of total gray difference:
(i j) is straight line
On any point; The matching result of carving t that clocks is { p
R, t(k) }, total number is that K (t) is individual, every all with p
L, t(k) corresponding, on the straight line of following formulate, search for, make Diff (i, j) Zui Xiao position:
Once more, obtain the synchronization stereographic map to matching result p
L, tAnd p
R, tAfter, obtain the three-dimensional point M that represents with camera coordinate system by stereo reconstruction
M, t, m is video camera L or R, M
M, tBe the coordinate of t moment three-dimensional point in the m camera coordinate system;
Configuration relation formula according to two video cameras:
M
L=R
cM
R+ T
c, M
L, M
RBe respectively the coordinate of same point in these two camera coordinate systems; And camera perspective projection formula:
Obtain the coordinate M of three-dimensional point in video camera L, R
L, tAnd M
R, t:
At the remaining part that attitude is estimated, three-dimensional point all uses the coordinate in the L camera coordinate system to represent, therefore omits subscript L, is about to M
L, tBrief note is M
t
(2.2), the follow-up moment: t+1 stereographic map is constantly followed the tracks of carry out two dimensional character with the KLT algorithm, obtained the three-dimensional point in each follow-up moment more equally by stereo reconstruction; Tracking is meant that the two dimension between the different time chart pictures that same video camera takes is corresponding:
When two dimensional character was followed the tracks of, the unique point of establishing detection was in image I
M, t(p) on (m is L or R), (x=i is arbitrary unique point y=j) to p, and then it is in image I
M, t+1On tracking results be
Wherein
G as above-mentioned, and
Note is followed the tracks of the unique point that obtains and is { p
M, t+1(k) }, m represents L or R, and t+1 is constantly:
For video camera L be:
For video camera R be:
Calculate three-dimensional point { M according to arbitrary trace point to coupling
T+1(k) | k=1 ..., K (t) } (as previously mentioned, having omitted subscript L here);
Wherein
(2.3), attitude parameter initialization: according to one group of three-dimensional corresponding point { M between two moment t and the t+1
t(k) | k=1 ..., K (t) } and { M
T+1(k) | k=1 ..., K (t) } and the expression reliability sequence { bReliable
k, k=1 ..., K (t) is from formula M
T+1(k)=R
tM
t(k)+T
t, t=1 ..., use among the N-1 based on the Robust Estimation algorithm of minimum intermediate value and find the solution R
t, T
t:
If: definition is also brought in constant renewal in boolean's array { bReliable
kBe reliable tolerance sequence, and Yes represents that reliably No represents unreliable, wherein value is that the label of the item of Yes is { k
s, s=1 ..., Kr (t), Kr (t) they are that value is the sum of the item of Yes, from wherein choosing Num_subset subclass, each subclass contains three couples of three-dimensional corresponding point, i.e. { Set
n={ N
1, n, N
2, n, N
3, n, n=1 ..., Num_subset, 1≤N
1, n, N
2, n, N
3, n≤ Kr (t), three-dimensional corresponding point are in each ternary subclass:
With
Can note by abridging and be { (M
Im,M
Im') | im=0,1,2}, its represents three pairs of not three-dimensional corresponding point of conllinear, R
t, T
tSatisfy M
Im'=R
tM
Im+ T
t| im=0,1,2; From following formula, obtain R and T (omitting subscript t) then:
Wherein
[r]
xBe according to 3*1 vector r=[r
xr
yr
z]
TThe antisymmetric matrix of definition:
And
It is unit matrix;
Translational component T is estimated by following formula:
To each subclass Set
nEstimate attitude parameter by above-mentioned formula, be designated as R
T, n, T
T, nAgain to all reliable three-dimensional corresponding { M
t(k
s), M
T+1(k
s), s=1 ..., Kr (t) is calculated as follows corresponding point M
t(k
s), M
T+1(k
s) and attitude parameter R
T, n, T
T, nMatching degree ε
N, s, this value is more little, and then both meet more:
ε
n,s=‖M
t+1(k
s)-R
t,nM
t(k
s)-T
t,n‖
2;
Again from set { R
T, n, T
T, n, n=1 ..., choose R among the Num_subset
t, T
t:
{R
t,T
t}={R
n_m,T
n_m},
Expression is got set { ε at subscript s
N, s| s=1 ..., Kr (t) intermediate value, it constitutes a new array to different n
And
Then be in this intermediate value array, to select to get that n of minimum value;
(2.4), at t=t+1 constantly, be provided with:
Whether need to call again the Boolean denotation amount bReDetect of feature detection and matching module;
Whether need to carry out the Boolean denotation amount bRefine of attitude refinement
(2.5), when t=N, attitude is estimated to finish;
(3). model initialization and attitude initialization:
The model initialization is the initial step of universal model specialization method, and it is from universal model Face
gObtain an initialization model Face
Init, promptly universal model being carried out change of scale, available following formula carries out:
The attitude initialization is in order to obtain the correspondence between model and image, promptly to calculate the rigid body translation that is tied to the L camera coordinate system from people's face generic coordinate; Promptly find the solution R from following formula
G, 1, T
G, 1:
M
1=R
G, 1M
Init+ T
G, 1, M
1And M
InitPeople's face and Face when being moment t=1 respectively
InitBetween arbitrary group of corresponding point; The present invention gets the corresponding point of above-mentioned two models in LE (left eye), RE (right eye) and these three positions of Mid_M (mouth center), by above-mentioned
, θ, R, [r]
x, the various R that tries to achieve successively of T
G, 1And T
G, 1
Press following formula again from R
t, T
t, t=1 ..., N-1 obtains R
G, t, T
G, t, t=2 ..., N:
R
g,t=R
t-1R
g,t-1,T
g,t=R
t-1*T
g,t-1+T
t-1,t=2,...,N;
(4). utilize three-dimensional information correction faceform:
(4.1), pick up the side unique point by hand:
If: the manual side unique point that marks is
Kk=1 ..., NumSF, NumSF=24 are the number of side unique point, side represents the moment corresponding to lateral plan;
(4.2), shape corrections A: Face
InitBe deformed into Face
1We determine the reposition of a subclass Subset among people's face three-dimensional node tabulation P earlier, determine the coordinate of all nodes among the P again by a known radial basis function algorithm;
The radial basis function algorithm is as described below: establishing will be with Face
StartBe deformed into Face
New, the node among the known Subset is at Face
NewIn coordinate be { New
1..., New
s, remember that it is at Face
StartIn corresponding point be { Sta
1..., Sta
s, S is the number of Subset mid point; For the arbitrary position pt in the model, it is at Face now
NewIn coordinate
Can obtain by following known formula:
Be that the pt point is at Face
StartIn coordinate, and { C
1..., C
SCan obtain by following formula
During " shape corrections A " radial basis function algorithm on utilize, Face is set
Start=Face
Init, Face
New=Face
1And the point among the Subset comprises two classes: a class is people's left eye interior angle LE, right eye interior angle RE, the outer lip LM in a left side, four three-dimensional remarkable characteristics of right outer lip RM on the face, is drawn by above-mentioned stereo reconstruction method by two-dimentional remarkable characteristic; Another kind of is above-mentioned 24 side unique points, and their three-dimensional position obtains from the manual two-dimensional points that marks by following step:
If: the three-dimensional coordinate of side unique point in people's face generic coordinate system is
Then, have for arbitrary kk
Wherein
Be the representation in components of attitude parameter, be known quantity; Can directly obtain with the On Solving System of Linear Equations method
In two unknown numbers
With
(4.3), shape corrections B: press aforesaid radial basis function algorithm Face
1Be deformed into Face
2
If Face
Start=Face
1, Face
New=Face
2Among the Subset except comprising the point of " shape corrections A " lining having determined, the point that also has some new interpolations, i.e. the three-dimensional point that the manual facial contour that marks recovers out under the positive attitude, the basic skills that obtains these three-dimensional point is current faceform Face
1Image plane under positive attitude is done perspective projection and is calculated the two-dimensional silhouette point of three-dimensional model projection result, and the coupling that depends on facial contour on these point and the above-mentioned image at last and be closed polygon is calculated its three-dimensional coordinate:
At first, calculate Face
1In the projected outline of the t=1 moment on the image plane of the manual front view of selecting
Be the label of model three-dimensional node tabulation P mid point,
And nNum1 is Cont
1The number on middle summit, they are to the new point that adds of Subset among the shape corrections B; They are to select by the intersection point of judging projection line and model, and promptly on a certain point, projection line and model can not have other intersection points, and its algorithm is as follows: Face
1Be three-dimensional model, the perspective projection center is
Known quantity was then calculated the projection line that Center orders and the intersection point of model to all summits in the model, judged that in view of the above this summit is whether in projected outline, if M
NowBe Face
1Middle any point, Plane
JpBe Face
1In the plane that constitutes of arbitrary dough sheet, calculated Center point and M
NowStraight line, i.e. projection line is with Plane
JpIntersection point be C
NowIf for a certain dough sheet 1≤jp≤nMesh, some C
NowAt line segment CM
NowInside, M then
NowNot in projected outline; Otherwise be projected outline's point;
Calculate Cont again
1In any point
At Face
2Three-dimensional coordinate: the facial contour of establishing on the above-mentioned image is
It is the polygon that is made of the two-dimensional points tabulation, and nNumCI1 is the number of point; Then
At Face
2Three-dimensional coordinate
Wherein,
Be that the pti point is at Face
1In coordinate; Warp
The projecting direction of point is v
PnBe known quantity; v
nFor the pti point at Face
gIn normal direction;
Parametric t
LineCan be according to two-dimentional straight line
With closed polygon Cont_img
1Intersection point obtain:
If Cont_img
1In arbitrary line segment be seg=(x
0+ sd
xy
0+ sd
y)
T, 0≤s≤1, s is the line segment parameter, other all are known quantities; Calculate by following formula earlier
All seg are obtained this two numerical value si and t
LineDo not have for the formula above a lot of seg and to separate, promptly the matrix in the formula is irreversible; The seg that has the si that separates and obtain to satisfy 0≤si≤1 has one or two, gets and point
That seg that distance is near, the t that it obtains
LineBeing institute asks;
(4.4), shape corrections C: press aforesaid radial basis function algorithm Face
2Be deformed into Face
3
If Face
Start=Face
2, Face
New=Face
3, except comprising the point of " shape corrections A, B " lining having determined, also have the point of some new interpolations among the Subset, promptly from the manual front face feature contour point that marks: the three-dimensional point that the point of eyes, nostril and mouth is recovered out:
If the two-dimensional coordinate of certain feature contour point is
This point is at Face
2In coordinate be
If this Z coordinate is correct as calculated, promptly it is at Face
3In the Z coordinate equal
Then it is at Face
3In coordinate be
Be that these feature contour point projections under positive attitude equal the unique point on the actual image;
(4.5), shape corrections D: press aforesaid radial basis function algorithm Face
3Be deformed into Face
4
If Face
Start=Face
3, Face
New=Face
4, except comprising the point of " shape corrections A, B, C " lining having determined, also have the point of some new interpolations among the Subset, even promptly from manual medial view 1 R that marks
G, tRotation angle recover the three-dimensional point of coming out in the facial contour near the moment t of 25 degree, concrete steps are as follows: establish this view corresponding to moment int1, described identical with shape corrections B, calculate Face earlier
3Projected outline when int1
Be the label of P mid point, nNum_int1 is the summit number in the projected outline; For any point
It is at Face
3In three-dimensional coordinate
And
Be the coordinate of L video camera photocentre in the generic coordinate system as previously mentioned, then this is at Face
4In coordinate should satisfy
Parametric t
Line2Equally according to straight line
With closed polygon Cont_img
Int1Intersection point obtain;
(4.6), shape corrections E: press aforesaid radial basis function algorithm Face
4Be deformed into Face
5(be final shape Face
s);
If Face
Start=Face
4, Face
New=Face
5, except comprising the point of " shape corrections A, B, C, D " lining having determined, also have the point of some new interpolations among the Subset, even promptly from manual medial view 2 R that mark
G, tRotation angle recover the three-dimensional point of coming out in the facial contour near the moment t of 40 degree, concrete recovering step is identical with shape corrections D;
(5). texture:
Present texture is front and the lateral plan (I that takes by the L video camera
L, 1And I
L, side) generate, texture is to be the model Face that creates
sGeneration appears to have the cylinder texture maps Texture of the same sense of reality of photograph
s, promptly will be in image conversion to the unified cylindrical coordinate system of gathering and merge, it contains successively and has the following steps:
(5.1), generate cylinder unwrapping figure:
At first to the net result Face of shape corrections E
sIn arbitrary three-dimensional point
Do the cylinder mapping by following formula:
(5.2), generate positive texture maps:
Have a few
At known front attitude parameter R
G, 1, T
G, 1Press following formula down at front view I
L, 1Last projection,
Be Face
sMiddle three-dimensional point
At I
L, 1On two-dimensional projection:
Then, I
L, 1In each pixel mapping to the Cyn plane, promptly to Face
sIn arbitrary dough sheet facet=(pt
1Pt
2Pt
3)
T, m
Proj=Δ (p
1, p
2, p
3) represent at these 3 at I
L, 1The triangle that constitutes on the plane, m
Cyn=Δ (p
1', p
2', p
3') be the corresponding triangle on the Cyn plane, promptly
IT1=1,2,3; Diabolo m again
CynInterior point [lg lt] is calculated as follows [x y]:
a
1~a
6Obtain by following formula:
Make positive texture maps again
(5.3), generate side grain figure:
Have a few
In known lateral attitude parameters R
G, side, T
G, sidePress following formula down at lateral plan I
L, sideLast projection,
Be Face
sMiddle three-dimensional point
At I
L, sideOn two-dimensional projection:
Again to Face
sIn arbitrary dough sheet facet=(pt
1Pt
2Pt
3)
T, establish ms
Proj=Δ (ps
1, ps
2, ps
3),
IT2=1,2,3; Diabolo m again
CynInterior point [lg lt] is calculated as follows [xs ys]:
As
1~as
6Obtain by following formula:
Make side grain figure again
(5.4), the mirrored sides texture maps is to obtain the texture maps of opposite side, and this is because Face
sTopological structure be complete symmetry;
If: that side arbitrary tri patch m on the face that directly obtains texture
Cyn=Δ (p
1, p
2, p
3), it is m at opposite side symmetrical dough sheet on the face
Cyn'=Δ (p
1', p
2', p
3'), wherein
iT3=1,2,3;
To m
CynIn any point p=(lg lt)
TBe calculated as follows [lg ' lt '],
Rs
1~rs
6Obtain by following formula:
Generate opposite side from a directly veined lateral reflection again:
(5.5), the fusion of positive and the texture maps side obtains final texture maps Texture
s
Setting threshold lg
MinAnd lg
Max, Texture
sAt arbitrary position Cyn=(lglt)
TThe value at place provides by following formula:
2. sequence { bReliable is reliably measured in the renewal in the described attitude estimating step
kMethod, it contains successively and has the following steps:
(1). according to sequence Err
tStatistical study calculate correct corresponding number K_True;
(1.1), establish: the initial value of K_True is Kr (t) * ratio, and ratio is predefined constant; Err
t={ ε
s=‖ M
T+1(k
s)-R
tM
t(k
s)-T
t‖
2| s=1 ..., Kr (t) in be E by the subsequence of preceding K_True element composition of ordering from small to large;
(1.2), calculate the average of E
And standard deviation
(1.3), at Err
tThe middle searching satisfied ‖ ε
s-ε ‖〉point of 3* σ, establishing number is InValid_T;
(1.4), if 0≤Kr (t)-K_True-InValid_T≤1, promptly K_True correspondence is correct, stops; Otherwise, change step (1.5);
(1.5), if K-K_True-InValid_T〉1, make K_True add 1, change step (1.2); Otherwise, change step (1.6);
(1.6), if K-K_True-InValid_T<0 makes K_True subtract 1, change step (1.2);
(2). establish Err
tIn before K_True less ε
sCorresponding label is { s
u, u=1 ..., K_True,
In be Yes corresponding to the position assignment of these labels, other assignment are No;
3. the two method of the Boolean denotation amount bReDetect whether setting up in the described attitude estimating step will call feature detection and matching module again, the Boolean denotation amount bRefine that whether will carry out the attitude refinement is as follows:
(1). if t=N, promptly current image is the last frame in the sequence, bReDetect=No then, bRefine=Yes finishes; Otherwise, change step (2);
(2). if current R
tRotation angle θ
tSurpass preset threshold value θ _ bound, bReDetect=Yes then, bRefine=Yes finishes; Otherwise, change step (3);
(3). (i.e. { the b Reliable if the number Kr (t) of reliable three-dimensional point
k, k=1 ..., value is counting of Yes among the K (t)) less than preset threshold value K_bound, bReDetect=Yes then, bRefine=Yes; Otherwise, change step (4);
(4) .bReDetect=No; If t-t0 is the multiple of the constant Inc_refine that presets, bRefine=Yes then, i.e. refinement is just to do once every Inc_refine; Otherwise bRefine=No;
(5). make t=t+1, enter next processing constantly;
4. in the described attitude estimating step, bRefine is that Yes is illustrated in the current time refinement that gestures, and promptly draws more accurate attitude parameter:
Come all relevant attitude parameter { R of refinement if be chosen in N continuous um=t-tStart+1 correct all the time three-dimensional correspondence of the moment
TStart+ τ, T
TStart+ τ, τ=0 ..., t-tStart-1, tStart are the initial moment of refinement computing, now get t second before constantly the moment that bRefine is Yes is set, if tStart<t0 gets it and is t0; In a refinement computing, calculate { Ra earlier
TStart+ τ, Ta
TStart+ τ, τ=1 ..., t-tStart; M
TStart(k
Pp), pp=1 ..., Kr (t) }, make that the total error f_obj of two-dimentional re-projection reaches minimum value in all images:
Ra wherein
TStart+ τ, Ta
TStart+ τBe to be carved into tStart+ τ rigid body translation coefficient constantly during from tStart; { M
TStart(k
Pp), pp=1 ..., Kr (t) is reliable three-dimensional point array in this Num the moment, and Kr (t) is individual altogether, and pp is the label of three-dimensional point; M is L or R, the expression video camera;
Be in m video camera, tStart+ the τ two-dimensional detection in the image or the result of tracking constantly, promptly
Optimize the exact value that f_obj just can obtain attitude parameter; Earlier find the solution { Ra with the Levenberg-Marquardt algorithm
TStart+ τ, Ta
TStart+ τ, τ=1 ..., t-tStart; M
TStart(k
Pp), pp=1 ..., Kr (t) }; Use following formula { Ra again
TStart+ τ, Ta
TStart+ τ, τ=1 ..., t-tStart is transformed into { R
TStart+ τ, T
TStart+ τ, τ=0 ..., t-tStart-1:
R
tStart=Ra
tStart+1,T
tStart=Ta
tStart+1,
Evidence: it has reached intended purposes.
Description of drawings
Fig. 1. people's face modeling process flow diagram
Fig. 2. attitude is estimated process flow diagram
Fig. 3. the texture process flow diagram
Fig. 4. the relation of two-dimensional coordinate system on camera coordinate system and the picture plane
Fig. 5. the remarkable characteristic of front face and line segment
Fig. 6. typical people's face side projection and some side unique points
Fig. 7. the front human face outline that stack shows
Fig. 8. the front face feature contour point that stack shows
Fig. 9. medial view 1 facial contour that stack shows (25 degree angle)
Figure 10. medial view 2 facial contours that stack shows (40 degree angle)
Figure 11. the stereo camera configuration of placing up and down
Figure 12. the typical stereographic map of system acquisition is to (two row correspond respectively to top up and down, bottom video camera, each row is images of synchronization, the numeral that marks is the frame number in whole sequence)
Figure 13. the neutrality of universal model and animated state (from left to right being neutral line chart, neutral dough sheet figure, animation line chart and animation dough sheet figure successively)
Figure 14. the experimental result of feature detection and coupling under the different attitudes (first row is a testing result, second row be corresponding stereographic map to matching result; The numeral that marks is the frame number in whole sequence)
Figure 15. the stereographic map of positive attitude on the two-dimentional remarkable characteristic (unique point stack be presented on the raw image) that marks
Figure 16. the stack of side unique point shows
Figure 17. the actual Texture that obtains
1
Figure 18. the Texture that the 35 step directly obtained
Side
Figure 19. the Texture that finally obtains
Side
Figure 20. cylinder texture maps Texture
s
Figure 21. shone upon Texture
sModeling result Face
s(checking) from different visual angles
Embodiment
We create the faceform under the framework of a universal model specialization, that is: from universal model Face
gBeginning is progressively made amendment according to the two-dimensional signal of the modeling object that obtains, and finally obtains specific human model Face
sIn this course, the three-dimensional point coordinate among the just P of modification, F then remains unchanged, and promptly constant (P and F have represented the shape of people's face to topological structure jointly, i.e. Face={P, F}.Particularly, P={M
Pti=(X
PtiY
PtiZ
Pti)
T| pti=1 ..., nVertex} is people's three-dimensional node tabulation on the face, and F={facet
Fj=(pt
1, fiPt
2, fiPt
3, fj)
T| fj=1 ..., nMesh} is people's triangle surface tabulation on the face, wherein each facet
Fj=(pt
1, fiPt
2, fiPt
3, fi)
TExpression is by three points among the P
A triangle of forming.Pt
Pti, fi, pti=1 ..., 3, fi=1 ..., nMesh is the sequence number of each point among the P, satisfies 1≤pt
Pti, fj≤ nVertex.NVertex and nMesh are respectively the total numbers of three-dimensional point and tri patch.Generally speaking, P representation model shape, F is the topological structure of representation model then).The topological structure of model can be kept along this technology path, thereby human face animation can be realized easily; Simultaneously, this universal model has also been represented our general knowledge to this closely similar object class of people's face, can provide additional constraint for concrete processing procedure.
Below we will describe modeling procedure in detail, and provide a concrete embodiment.
The overall procedure of people's face modeling as shown in Figure 1, the front and back of each step orders and input and output all clearly mark." processing module " this part, the numeral number in the block diagram is in detail this module to be set forth the little section number of part in this section.In addition, " input " refer to before those modelings by the ready data of manual intervention (video data, Face
g, and need the manual two-dimensional points that marks), " intermediate result " then is the data that system obtains in the modeling flow process automatically.
The stereoscopic video sequence of input is designated as I
L, tAnd I
R, t, t=1 ..., N represents the different moment, and N then is the totalframes of video sequence, and L, R be corresponding to two video cameras, the two-dimensional array that I then is made up of pixel, each pixel comprises R, G, three components of B.The moment of t=1 is selected by people's judgement is manual corresponding to the positive attitude of people's face.
Below we describe the concrete disposal route of each step in detail.
1.1 attitude is estimated
This step will be found the solution the attitude parameter (being also referred to as the rigid body translation coefficient) in the number of people rotary course.Be designated as { R
t, T
t, t=1 ..., N-1}, wherein each R
t, T
tBe the people face rigid body translation coefficient of t frame, constantly and the t+1 moment arbitrary promptly to three-dimensional corresponding point M for t to the t+1 frame
tAnd M
T+1(so-called corresponding, as to be meant that they are that people's same position on the face is at difference three-dimensional coordinate constantly) has following formula
M
t+1=R
tM
t+T
t,t=1,...,N-1
Wherein the coordinate system at three-dimensional point place is the L camera coordinate system.The definition of camera coordinate system is a true origin with the photocentre C of perspective video camera as shown in Figure 4, and the Z axle is along optical axis direction, X, and the x of Y-axis two-dimensional coordinate system respectively and on the picture plane, the y axle is parallel.
The entire process flow process that attitude is estimated as shown in Figure 2, below we do simple declaration to the principle of algorithm earlier, introducing each step in detail.
Our thinking is to utilize the motion of reliable three-dimensional point to come estimated parameter, and three-dimensional point needs (to be called as " two dimensional character " below according to the two-dimentional corresponding point that project on the different images.About " two dimension is corresponding ", be meant that same three-dimensional point all is projected as two-dimensional points in different images, be called two-dimentional corresponding point mutually between them; Wherein stereographic map to two-dimentional correspondence be called " coupling ", the different two-dimentional correspondences between images constantly that same video camera is taken then are called " tracking ".Here " stereographic map to " refers to a pair of image that synchronization L and R video camera are taken) calculate.Particularly, we at first will obtain three-dimensional point, and this is to detect and coupling by two dimensional character in the initial moment, obtains by stereo reconstruction then; For the follow-up moment then is to follow the tracks of respectively in L and two image sequences of R, then obtains by stereo reconstruction equally.We just can estimate attitude after the three-dimensional point in the different moment had been arranged.Consider from the number of unknown quantity, only need the corresponding three-dimensional points in three pairs of two moment just can finish, but the three-dimensional point that requires to reconstruct is reliably, and be difficult in the actual experiment guarantee that all three-dimensional point are all reliable.In order to estimate the reliability of three-dimensional point, we are definition and continual renovation array { b Reliable
k, this is boolean's array, value is Yes (representing that this point is reliable) or No (it is unreliable to represent).This array can be updated along with the carrying out of following the tracks of, and it all is reliable that initialization when just beginning have a few, and some was put and then can be judged as insecurely afterwards, and the method that we make this judgement will describe in detail below.
Directly bigger in the attitude parameter error that adjacent two constantly motion calculation go out according to three-dimensional point, we can constantly carry out the attitude refinement to reduce error according to all the reliable three-dimensional point in one period long period at some.When carry out this refinement computing and controlled by Boolean denotation bRefine, its value Yes represents that current time carries out the attitude refinement, and No is then opposite.
Along with the carrying out of following the tracks of, three-dimensional point can be fewer and feweri reliably.In order to make attitude estimate to proceed and to obtain accurate result, we are necessary to carry out again two dimensional character detection and coupling and stereo reconstruction, find out new reliable three-dimensional point.When carry out this operation by Boolean denotation bReDetect control, its value Yes represents that current time need find out the work of new three-dimensional point, and No is then opposite.
The method to set up of these two Boolean denotations of bRefine and bReDetect also will be described in detail below.
Below we elaborate complete process:
1.1.1 the detection of two dimensional character
We utilize existing KLT algorithm to finish the detection of two dimensional character on gray-scale map.Its principle is: establishing the image that constantly m video camera of t (m is L or R, represents some in two video cameras) obtains is I
M, t(p), p=(x, y) any point on the presentation image wherein; The gradient vector of this some place image is
Calculate the matrix of 2*2
Wherein W is to be a rectangular window at center with p.We obtain the minimum singular value sing_min (utilizing existing matlab kit) of G, and select the very big p point of those sing_min as detected unique point.On physical significance, G is actual to be the expression that local gray level on the image is changed the degree of enriching: if local gray scale is constant, at this moment gradient vector g is 0, and G is 0, and all singular values all are 0; If local grey scale change has fixing direction (for example image is gone up more intense single edge), gradient vector g is substantially perpendicular to this direction, and at this moment the singular value of G minimum approaches 0; It is very big only all to compare the minimum singular value that just satisfies G in significant part (for example intersection point on two limits) in the grey scale change along all directions.Therefore, we are actual to be to select those local gray levels to change abundant point as unique point, and such some degree of reliability when following the tracks of and mate is higher.
We need do the discretize processing to top formula during computer realization, and concrete computing formula is as follows:
Here blockx, blocky has determined the net point number that window W comprises.In fact, (i j) is the center to this window, and net point adds up to (2*blockx+1) * (2*blocky+1) with tested point.
In Fig. 2, t detected unique point constantly is designated as { p
L, t(k) }, L, t presentation image sequence number.The total number of unique point is K (t), and k satisfies 1≤k≤K (t), the label of expression point.
At this moment we make t0=t, and which these features that expression will be tracked be detected constantly at.
1.1.2 two dimensional character coupling
Matching problem is finished according to known method.It is as follows simply to introduce its principle:
If the configuration relation between two video cameras is represented with following formula
M
L=R
cM
R+T
c (1.1.3)
L and R represent two camera coordinate systems, M
LAnd M
RBe the coordinate representation of same three-dimensional point in these two coordinate systems; R
cAnd T
c=[T
C, xT
C, yT
C, z]
TBe respectively rotation and translational component (being collectively referred to as the outer parameter of stereo camera, is known quantity), the former is the matrix of a 3*3, and the latter is a tri-vector.
After (1.1.3) arranged, order
(this matrix is called as basis matrix), wherein f
L, f
R, u
0L, v
0L, u
0R, v
0RBeing the intrinsic parameter of video camera, all is known quantity; Its matrix representation is
Then for L, 1 p of certain among the t figure
L, t(k)=(x
L, t(k) y
L, t(k)) T, it is at R, the match point p among the t figure
R, t(k)=(x
R, t(k) y
R, t(k)) T should satisfy following known formula (1.1.4):
(1.1.4) physical significance of formula is p
R, t(k) be positioned at by p
L, t(k) on Jue Ding the straight line, reduce to this straight line for the search volume of seeking this point from whole image like this, promptly dropped to one dimension from two dimension.In this one-dimensional space, the point of selecting to make total gray difference minimum in the local window Wm promptly will make following (1.1.5) formula minimum as matching result:
Diff=∫[I
R,t(p
R,t(k))-I
L,t(p
L,t(k))]
2dp (1.1.5)
After the discretize, the formula during actual computation is:
Wherein (i j) is straight line
On any point.
T matching result constantly is { p among Fig. 2
R, t(k) }, total number also is K (t), and each is put all and p
L, t(k) corresponding.To each p
L, t(k) matching process is independently, promptly searches for the position that makes (1.1.6) formula minimum on the straight line of formula (1.1.4) expression.
Obtain the method acquisition three-dimensional point sequence { M of coupling back with stereo reconstruction
t(k) | k=1 ..., K (t) }.The process of stereo reconstruction is independently to each to match point, and principle is as follows:
At first, we have the perspective projection principle of video camera, i.e. Xia Mian known formula,
Wherein m is L or R, represents in two video cameras any; M
M, tBe the coordinate of t moment three-dimensional point in the m camera coordinate system; s
mIt is a scale factor to be asked.We have omitted (k) and have represented any point.
Can solve three-dimensional point, M in conjunction with the configuration relation between two video cameras (1.1.3) again
L, tExpression formula be
Wherein
And
And M
R, tThen can obtain by following formula
If do not specify, the three-dimensional point that the stereo reconstruction of mentioning in this article goes out all is meant the M in the L camera coordinate system
L, tFor simplicity, we remove L in statement, are abbreviated as M
tAll unique points are repeated top stereo reconstruction step, and we obtain three-dimensional point sequence { M
t(k) | k=1 ..., K (t) }.
On physical significance, each two-dimensional projection's point is actual to be to have determined three-dimensional straight line, according to a pair of two-dimensional points of same three-dimensional point in two video cameras, by finding the solution the intersection point of two 3 d-lines, just can determine this three-dimensional point like this.
Simultaneously, we also define Boolean variable sequence { bReliable
k, k=1 ..., K (t), it represents whether each three-dimensional point is reliable, and value is Yes or No, and all values in the sequence all is initialized as Yes at present, and it all is reliable that expression is had a few.
1.1.3 the tracking of two dimensional character
Two dimensional character on the gray-scale map is followed the tracks of and is utilized existing KLT algorithm to finish equally.Its principle is: the unique point of establishing detection is at visual I
M, t(p) on (m is similarly L or R), wherein p is arbitrary unique point; Now need be at visual I
M, t+1This point of last tracking.Based on the coupling of local grain information, order
(calculating of G is as previously mentioned), then
Be visual I
M, t+1Go up tracking results to p.
It is right to need equally during actual computation
Find the solution and make discretize and handle:
The unique point that tracking obtains is expressed as { p in Fig. 2
M, t+1(k) }.
1.1.4 attitude parameter initialization
Identical stereo reconstruction method can obtain t+1 three-dimensional point sequence constantly when using with coupling for the result who follows the tracks of, and we have just obtained one group of three-dimensional corresponding point { M in these two moment (t and t+1) like this
p(k) | k=1 ..., K (t) } and { M
T+1(k) | k=1 ..., K (t) }.Below according to three-dimensional correspondence of this group and the sequence { bReliable that represents its reliability
k, k=1 ..., K (t) (this sequence is called as " reliably measuring sequence ") finds the solution the posture changing parameters R of t frame to the t+1 frame
t, T
t, upgrade reliable tolerance sequence simultaneously.
At first, consider that three pairs of three-dimensional corresponding point just are enough to calculate attitude parameter from the number of unknown quantity.The principle of simply introducing this algorithm known is as follows:
If { (M
Im, M
Im') | im=0,1,2} is three couples of not three-dimensional corresponding point (M of conllinear
ImAnd M
Im' be the coordinate of same three-dimensional point in the L camera coordinate system in two frames of front and back), attitude parameter R
t, T
tSatisfy M
Im' R
tM
Im+ T
t| im=0,1,2.For determining rotational component R wherein
t, we need determine the central shaft that rotates
(being called turning axle, the vector of 3*1) and around this anglec of rotation θ.The definition vector of unit length
With
Then
Wherein
[r]
xBe according to 3*1 vector r=[r
xr
yr
z]
TThe antisymmetric matrix of definition, promptly
And
It is unit matrix.
Estimate rotational component R
tAfterwards, translational component T
tCan obtain by following formula:
Though top algorithm can obtain result accurately in theory, numerical stability is very poor, promptly at its input { (M
Im, M
Im') | im=0,1, when 2} had less error, the solving result of algorithm can be very inaccurate.And the input data generally have error, even some are wrong in addition.In order to obtain result accurately in this case, we have adopted the Robust Estimation algorithm based on minimum intermediate value (Least-Median-Square) commonly used.Its ultimate principle is: randomly from current reliable three-dimensional corresponding (i.e. { bReliable
k, k=1 ..., value is the item of Yes among the K (t).If these label is { k
s, s=1 ..., Kr (t), wherein Kr (t) is the total number of reliable point) in choose Num_subset subclass, each subclass Set
n, n=1 ..., Num_subset contains three pairs of three-dimensional corresponding point, estimates attitude parameter R by top algorithm in view of the above
T, n, T
T, n, then to all reliable three-dimensional corresponding { M
t(k
s), M
T+1(k
s), s=1 ..., Kr (t) calculates ε by following formula (1.1.8)
N, s:
ε
n,s=‖M
t+1(k
s)-R
t,nM
t(k
s)-T
t,n||
2 (1.1.8)
This ε
N, sPhysical significance be corresponding point M
t(k
s), M
T+1(k
s) and attitude parameter R
T, n, T
T, nMatching degree, this value is more little, illustrates that both meet more.We come from set { R by formula (1.1.9)
T, n, T
T, n, n=1 ..., select R among the Num_subset
t, T
t, i.e. { R
t, T
t}={ R
N_m, T
N_m, wherein
Wherein
Expression is got set { ε at subscript s
N, s| s=1 ..., Kr (t) intermediate value, it constitutes a new array to different n
And
Then be in this intermediate value array, to select to get that n of minimum value.
Find the solution R above
t, T
tAlgorithm can guarantee exist a lot of wrong three-dimensionals at once still obtaining correct result, this be because: the first, according to { M
t(k) | k=1 ..., K (t) } and { M
T+1(k) | k=1 ..., K (t) } in three accurately correspondingly just can calculate attitude parameter; The second, selecting the foundation of attitude parameter by formula (1.1.9) is the intermediate value of whole sequence, is wrong even therefore a lot of three-dimensional correspondences are arranged, and causes a lot of numerical value in this sequence bigger; As long as correct three-dimensional corresponding number surpasses K (t) half, algorithm still can obtain correct attitude estimated result.
Then we need upgrade reliable tolerance sequence { bReliable
k, judge that promptly which three-dimensional correspondence is correct.Concrete grammar is as follows: make Err
t={ ε
s=‖ M
T+1(k
s)-R
tM
t(k
s)-T
t‖
2| s=1 ..., Kr (t) } represent that current reliable three-dimensional point is to R
t, T
tDegree of support, it is correct establishing K_True (computing method are as described below) in the individual point of this Kr (t) individual, we establish Err
tIn before K_True less ε
sCorresponding label is { s
u, u=1 ..., K_True, then
In corresponding to the position value Yes of these labels, other position is No.
The calculation procedure of K_True is:
(1) initial value that K_True is set is Kr (t) * ratio (ratio is a predefined constant); Note Err
tIn be E by the subsequence of preceding K_True element composition of ordering from small to large;
(2) average of calculating E
And standard deviation
(3) at Err
tThe middle searching satisfied ‖ ε
s-ε ‖〉point of 3* σ, establishing number is In Valid_T;
(4) if 0≤Kr (t)-K_True-InValid_T≤1 illustrates that current judgement to the sum of errors mistake is compatible, stop, current K_True is net result; Otherwise, change step 5;
(5) if K-K_True-InValid_T〉1, K_True adds 1 (being to add a bit among the E), changes step 2; Otherwise, change step 6;
(6) K_True subtracts 1 (being to reduce a bit among the E), changes step 2.
The principle of this method is the boundary position of sum of errors mistake on statistical significance.
1.1.5 sign bReDetect and bRefine are set
This is the sign amount of two Boolean types, and value is Yes or No.BReDetect represents whether will call feature detection and matching module again, and bRefine then represents whether will carry out the attitude refinement.Their set-up mode is as follows:
(1) if t=N, promptly current image is last width of cloth in the sequence, bReDetect=No then, bRefine=Yes; Finish; Otherwise, change step 2;
(2) if current R
tRotation angle θ
tSurpassed preset threshold value θ _ bound, bReDetect=Yes then, bRefine=Yes; Finish; Otherwise, change step 3;
(3) if the number Kr (t) of reliable three-dimensional point is (i.e. { bReliable
k, k=1 ..., value is the number of the point of Yes among the K (t)) less than preset threshold value K_bound, bReDetect=Yes then, bRefine=Yes; Otherwise, change step 4;
(4) bReDetect=No; If t-t0 is the multiple of the constant Inc_refine that presets, bRefine=Yes; Otherwise bRefine=No.
From the principle, we change need reform when excessive feature detection and coupling in reliably count out deficiency or attitude; Otherwise just do signature tracking.Refinement is then just done once every regular time interval (Inc_refine).
Make t=t+1 now, prepare to enter next processing constantly.
1.1.6 attitude refinement
BRefine is that Yes is illustrated in current time and does the attitude refinement, at this moment system also needs to be provided with the initial moment tStart of refinement computing, actual we get its for t constantly before second the moment that bRefine is Yes is set, be t0 if the tStart that obtains so just makes it less than t0.
1.1.4 the attitude parameter initialization of joint has only utilized the three-dimensional corresponding of two moment (t and t+1).For obtaining more accurate attitude parameter, we select those to come all relevant attitude parameter { R of refinement N continuous um=t-tStart+1 correct all the time three-dimensional correspondence of the moment
TStart+ τ, T
TStart+ τ, τ=0 ..., t-tStart-1.Specifically, in a refinement computing, we calculate { Ra earlier
TStart+ τ, Ta
TStart+ τ, τ=1 ..., t-tStart; M
TStart(k
Pp), pp=1 ..., Kr (t) }, make that the f_obj in the formula (1.1.10) reaches its minimum value.
Ra wherein
TStart+ τ, Ta
TStart+ τBe to be carved into tStart+ τ rigid body translation coefficient constantly during from tStart; { M
TStart(k
Pp), pp=1 ..., Kr (t) is reliable three-dimensional point array in this Num the moment, comprises the individual point of Kr (t), pp represents the label of three-dimensional point.M is L or R, represents some video cameras.
Be the result of two-dimensional detection or tracking, in m video camera tStart+ τ image constantly, promptly
And
Then be the pp three-dimensional point constantly, project to the result on the m video camera, promptly at tStart+ τ
On physical significance, what formula (1.1.10) was represented is the total error of two-dimentional re-projection in all images.Optimize the exact value that this target can obtain attitude parameter.Parameter { Ra wherein
TStart+ τ, Ta
TStart+r, τ=1 ..., t-tStart; M
TStart(k
Pp), pp=1 ..., Kr (t) find the solution and can call general Levenberg-Marquardt algorithm (utilizing the matlab kit) and finish.
Obtained { Ra
TStart+ τ, Ta
TStart+ τ, τ=1 ..., calculate { R by following formula behind the t-tStart
TStart+ τ, T
TStart+ τ, τ=0 ..., t-tStart-1, thus reach the refinement purpose.
R
tStart=Ra
tStart+1,T
tStart=Ta
tStart+1,
The physical significance of top formula is the compound of rigid motion, for example: { R
TStart+ τ-1, T
TStart+ τ-1Be to be carved into tStart+ τ rigid body translation constantly at tStart+ τ-1 o'clock, and it equals the compound of two rigid body translations, and promptly doing earlier to be carved into tStart rigid body translation constantly from tStart+ τ-1 o'clock (is { Ra
TStart+ τ-1, Ta
TStart+ τ-1Inverse transformation), be carved into tStart+ τ rigid body translation ({ Ra constantly when remaking from tStart
TStart+ τ, Ta
TStart+ τ).
1.2 the initialization of model and attitude
Press the thinking of universal model specialization, at first need from Face
gObtain an initial model Face
InitIn addition, obtained { R in the superincumbent attitude estimating step
t, T
t, t=1 ..., N-1}, and in order to obtain the correspondence between model and projection image, we also need to calculate the rigid body translation R from generic coordinate system (the hypomere introduction is an object coordinates system that is fixed on the number of people) to each frame L camera coordinate system
G, t, T
G, t, t=1 ..., N.These two tasks are finished by model initialization and attitude initialization respectively.
Explain the generic coordinate system earlier.We get the center of gravity Middle that coordinate origin is left eye interior angle LE, right eye interior angle RE, the outer lip LM in a left side, four points of right outer lip RM; Get X-direction for point to the direction of left eye LE from right eye RE, get the direction (direction straight up) that Y direction Cong Xia Chin points to forehead, get Z-direction for point to the direction of nose from hindbrain.Three direction pairwise orthogonals, and constitute right-handed coordinate system.Each unique point above-mentioned is referring to Fig. 5.
The initialized process of model is actual to be that original universal model is applied a change of scale.Promptly finish by following formula (1.2.1):
M wherein
gAnd M
InitBe respectively Face
gAnd Face
InitOn arbitrary group of corresponding point.Three scale factor s
x, s
y, s
zUtilize the distance configuration of people's four principal character points on the face to determine (comprising left inner eye corner, right inner eye corner, the left corners of the mouth and the right corners of the mouth).Particularly: s
xBe modeling object and Face
gThe ratio of middle Seg_E length (distances of two inner eye corners), s
yThen be the ratio of Seg_EM length (distance at mouth center is arrived at two centers), s
zJust be taken as s
yThese line segments also mark in Fig. 5.
Determined behind these three scale-up factors Face
gIn all the point by formula (1.2.1) make change of scale, the result is Face
Init
The initialized task of attitude is to calculate R
G, 1, T
G, 1, promptly the rigid body translation of L camera coordinate system when the generic coordinate is tied to the 1st frame promptly will be found the solution the parameter in (1.2.2)
M
1=R
g,1M
init+T
g,1 (1.2.2)
M wherein
1And M
InitBe respectively 1 o'clock moment people's face and Face
InitArbitrary group of corresponding point between these two models.
For finding the solution R
G, 1, T
G, 1, we select the corresponding point of these two models in LE (left eye), RE (right eye) and these three positions of Mid_M (mouth center), obtain the result by the initialized method of attitude in the front 1.1.4 joint.
Obtained R
G, 1, T
G, 1After, system is compound according to rigid motion, calculates other each attitude parameter R constantly by following formula
G, t, T
G, t, t=2 ..., N:
R
g,t=R
t-1R
g,t-1,T
g,t=R
t-1*T
g,t-1+T
t-1,t=2,...,N
System can judge the moment side corresponding to lateral plan automatically now, promptly selects side for making R
G, tRotation angle near those t of 90 degree.
1.3 shape corrections A
This step will be Face
InitBe deformed into Face
1As previously mentioned, we keep the topological structure of Face, and just revise the wherein coordinate of three-dimensional point.Particularly, we determine the reposition of a subclass Subset among the three-dimensional node tabulation P earlier, determine the coordinate of all nodes among the P again by a known radial basis function algorithm.
Simply introduce earlier the radial basis function algorithm basic principle.If will be with Face
StartBe deformed into Face
New, the node among the known Subset is at Face
NewIn coordinate be { New
1..., New
s, remember that it is at Face
StartIn corresponding point be { Sta
1..., Sta
S; Here S is the number of Subset mid point.For the arbitrary position pt in the model, it is at Face now
NewIn coordinate
Can obtain by following known formula:
Wherein
Be that the pt point is at Face
StartIn coordinate, and { C
1..., C
sCan obtain by following formula
This method is actual to be according to Face
StartMiddle each point determines that to the distance relation of some calibration points (being the point among the Subset) this point is at Face
NewIn the position.φ is known radial basis function (field of definition and codomain all are nonnegative real number), and it is the decreasing function of independent variable r (expression distance), and promptly the calibration point of distance is less to the influence of this point.
This step, we had Face during radial basis function algorithm on utilize
Start=Face
Init, Face
New=Face
1And the point among the Subset comprises two classes: a class is three-dimensional remarkable characteristic, totally 4, obtain by the stereo reconstruction algorithm of two-dimentional remarkable characteristic (referring to people's left eye interior angle LE, right eye interior angle RE, the outer lip LM in a left side, four projections of point on image plane of right outer lip RM on the face) by the front; Another kind of is that the side unique point (refers to the point that those are easy to recognize, as Xia Chin, mouth, nose, bridge of the nose top, forehead etc. on people's face side projection view.Typical people's face side projection and our some side unique points of definition represent with little circle that in Fig. 6 its example is referring to Figure 16), what reality was used has 24, and their three-dimensional position obtains from the manual two-dimensional points that marks by following step.
If the manual side unique point that marks is
Kk=1 ..., NumSF, wherein NumSF=24 is the number of side unique point, side is the side frame label that the front has obtained.Their three-dimensional coordinates in the generic coordinate system are
Then to arbitrary kk, we have
Wherein
It is the representation in components of attitude parameter.Two linear equations are arranged (1.3.1), thereby can directly obtain with the On Solving System of Linear Equations method
In two unknown numbers
With
This method is actual to be the specific position (promptly the X component of coordinate is 0 in the generic coordinate system) that has utilized the side unique point, thereby can directly obtain three-dimensional point from the single width image.
1.4 shape corrections B
This step is utilized the radial basis function algorithm of front, Face equally
Start=Face
1, Face
New=Face
2Except comprising the point of having determined lining, front " shape corrections A ", also has the point of some new interpolations among the Subset, promptly according to the manual front human face outline (facial contour on the face attitude of the making a comment or criticism hypograph that marks, polygonal profile with closure represents that an example as shown in Figure 7.This figure has been added to polygonal profile on people's face image, sees clearlyer) recover the three-dimensional point of coming out.The basic skills that obtains these three-dimensional point is that current faceform is done perspective projection to image plane, and calculates the two-dimensional silhouette point of three-dimensional model perspective projection, and the coupling that depends on the facial contour on these point and the image is at last calculated its three-dimensional coordinate.Concrete step is as described below:
At first need to calculate Face
1Profile in from the t=1 moment (being the manual front view of selecting) to the image plane projection
Wherein
Be the label of P (being the three-dimensional node tabulation of model) mid point, obviously have
And nNum1 is Cont
1In the summit number.Cont
1In point be exactly the point that newly adds in this step shape corrections among the Subset, they are to select by the intersection point of judging projection line and model, promptly on a certain point, projection line and model can not have other intersection point, detailed computing method are as described below:
The note three-dimensional model is Face
Proj(be exactly Face here
1), the perspective projection center is
Be the coordinate of L video camera photocentre in the generic coordinate system, then we calculated the projection line of this point and the intersection point of model to all summits in the model, judged that in view of the above this summit is whether in projected outline.Particularly, M
NowBe Face
ProjMiddle any point, Plane
JpBe Face
ProjIn the plane that constitutes of arbitrary dough sheet, calculated Center and M
NowStraight line (being projection line) and Plane
JpIntersection point be C
NowIf (as previously mentioned, nMesh is Face to some jp that satisfies 1≤jp≤nMesh
ProjIn the dough sheet sum) C that obtains
NowAt line segment CM
NowInside, M then
NowNot in projected outline; Otherwise just should belong to projected outline.
Then to calculate Cont
1In any point
At Face
2Three-dimensional coordinate.If the facial contour on the image is
It is the polygon that is made of the two-dimensional points tabulation, and nNumCI1 is the number of point.Then we determine by following method
At Face
2Three-dimensional coordinate:
If
Point is at Face
gIn normal direction be v
n, at Face
1In coordinate be
And through the projecting direction of this point be
(this three is a known quantity), then the pti point is at Face
2In coordinate
Satisfy
Wherein
v=(v
x v
y v
z)
T=v
pn×(v
n×v
pn) (1.4.2)
Parametric t
LineCan be according to two-dimentional straight line
With closed polygon Cont_img
1Intersection point obtain.Concrete steps are as follows:
If Cont_img
1In arbitrary line segment can be expressed as seg=(x
0+ sd
xy
0+ sd
y)
T, 0≤s≤1, wherein s is the line segment parameter, other all are known quantities.
We calculate by following formula earlier
All seg are obtained this two numerical value si and t
LineDo not have for the formula above a lot of seg and to separate, promptly the matrix in the formula is irreversible.The seg that has the si that separates and obtain to satisfy 0≤si≤1 has one or two, gets and point
That seg that distance is near, the t that it obtains
LineBeing institute asks.
Here v is
The deformation direction of point has utilized the local shape of general people's face to its calculating.In fact, this direction is Face
1Go up the component of this some place normal direction, can keep local curvature so on the one hand as far as possible perpendicular to projection line; Can reduce deformation distance a little on the other hand as far as possible.
1.5 shape corrections C
This step is utilized the radial basis function algorithm of front, Face equally
Start=Face
2, Face
New=Face
3Among the Subset except comprising the point of front " shape corrections A, B " lining having determined, the point that also has some new interpolations, promptly according to the manual front face feature contour point that the marks (point of manual eyes, nostril and the mouth that marks under the face attitude of making a comment or criticism, be some discrete points, as shown in Figure 8, be that stack shows equally) recover the three-dimensional point of coming out.Concrete recovering step is as described below:
If the two-dimensional coordinate of certain feature contour point is
This point is at Face
2In coordinate be
We suppose that the Z coordinate of this point is correct as calculated, and promptly it is at Face
3In the Z coordinate equal
Then it is at Face
3In coordinate be
Be that these feature contour point projections under positive attitude equal the unique point on the actual image.
1.6 shape corrections D
This step is utilized the radial basis function algorithm of front, Face equally
Start=Face
3, Face
New=Face
4, except comprising the point of front " shape corrections A, B, C " lining having determined, also have the point of some new interpolations among the Subset, promptly (selection of this view is judged automatically that by system actual is to make R from the manual medial view 1 that marks
G, tRotation angle near the moment t of 25 degree.So-called " medial view " is meant the middle projection attitude between front and side) facial contour (representing with closed polygon that equally typical example the is Fig. 9) three-dimensional point that recovery is come out, concrete recovering step is as described below:
If medial view 1 is similar to the processing of front view (FV) facial contour with front 1.4 joints corresponding to moment int1, also to calculate Face earlier
3Projected outline when int1
Wherein
Be the label of P (being faceform's three-dimensional node tabulation) mid point, and nNum_int1 is the summit number in the projected outline.The computing method of projected outline are (1.4 joint) as previously mentioned.
Cont
Int1In point at Face
4In coordinate determine with the similar step of 1.4 joints by following: for arbitrary
Definition
Wherein
Be that this point is at Face
3In three-dimensional coordinate, and
Be the coordinate of L video camera photocentre in the generic coordinate system as previously mentioned.Then this is at Face
4In coordinate
Satisfy
Parametric t wherein
Line2According to two-dimentional straight line
With closed polygon Cont_img
Int1Intersection point obtain, method as 1.4 the joint as described in.
1.7 shape corrections E
This step is utilized the radial basis function algorithm of front, Face equally
Start=Face
4, Face
New=Face
5, except comprising the point of front " shape corrections A, B, C, D " lining having determined, also have the point of some new interpolations among the Subset, promptly (selection of this view is judged automatically by system that also actual is to make R from the manual medial view 2 that marks
G, tRotation angle near the moment t of 40 degree) facial contour (typical example is Figure 10) recovers the three-dimensional point of coming out, the processing in concrete recovering step and 1.6 joints is just the same.
1.8 texture
Texture is to be the model Face that creates
sGenerate cylinder texture maps Texture
s, make final modeling result appear to have the same sense of reality of photograph.Present texture is to generate by two images, i.e. the front and the lateral plan (I of L video camera shooting
L, 1And I
L, side), the ultimate principle of this part is gathering in image conversion to the unified cylindrical coordinate system, and merges, its flow process as shown in Figure 3, concrete steps are as follows:
1.8.1 cylinder unwrapping map generalization
System is at first to Face
sIn arbitrary three-dimensional point
Do the cylinder mapping shown in the formula (1.8.1):
The result of mapping is a two dimensional surface, and the point on it is { Cyn
Pt}={ (lg
PtLt
Pt).Here lg
PtBe the longitude station on the cylinder, represent with radian value; Lt
PtIt then is the coordinate position of cylinder axial rotary.
1.8.2 positive texture maps
Wherein
It is the representation in components of rotation matrix and translation vector.
At this moment
Be Face
sMiddle three-dimensional point
At front view I
L, 1On two-dimensional projection.We obtain an I now
L, 1Point on the plane
With put Cyn on the Cyn plane
Pt=(lg
PtLt
Pt)
TTwo-dimentional corresponding relation, this relation be since they all and three-dimensional point
Corresponding obtaining (respectively by formula 1.8.2 and 1.8.1).We based on this corresponding relation with I
L, 1In each pixel mapping to the Cyn plane, promptly to Face
sIn arbitrary dough sheet facet=(pt
1Pt
2Pt
3)
T, establish m
Proj=Δ (p
1, p
2, p
3) represent at these 3 at I
L, 1The triangle that constitutes on the plane, m
Cyn=Δ (p
1', p
2', p
3') be the corresponding triangle on the Cyn plane, promptly
iT1=1,2,3。Diabolo m
CynInterior point [lg lt] calculates [xy] by following formula (1.8.4), and order
Being about to positive texture maps is mapped on the cylinder unwrapping figure.
Six unknown quantity a wherein
IA, iA=1 ..., 6 by formula (1.8.5) obtain.
The result of positive texture is designated as Texture
1
1.8.3 side grain figure
Very similar with the processing of front view (FV), will own
In the lateral attitude parameters R
G, side, T
G, sideBy formula (1.8.6) carries out projection down:
Here same important expression
Then to Face
sIn arbitrary dough sheet facet=(pt
1Pt
2Pt
3)
T, establish ms
Proj=Δ (ps
1, ps
2, ps
3), wherein
iT2=1,2,3。Diabolo m
CynInterior point [lg lt] calculates [xs ys] by the formula (1.8.8) of back, and order
Being about to side grain figure is mapped on the cylinder unwrapping figure.
Six unknown quantity as wherein
IAS, iAS=1 ..., 6 by formula (1.8.9) obtain.
The result of side grain mapping is designated as Texture
Side
1.8.4 the reflection of side grain figure
Because people's face is only to a sideway swivel, so can only obtain one-sided side grain, the texture of opposite side can pass through Texture
SideDoing the reflection processing obtains.Specifically, because Face
sTopological structure be complete symmetry, be provided with that side arbitrary tri patch m on the face of direct acquisition texture
Cyn=Δ (p
1, p
2, p
3) and it is at opposite side symmetrical dough sheet m on the face
Cyn'=Δ (p
1', p
2', p
3'), wherein
IT3=1,2,3, then to m
CynIn any point p=(lg lt)
TBy following formula (1.8.9) calculates [lg ' lt '], and order
Promptly the lateral reflection from direct acquisition texture generates opposite side.
Six unknown quantity rs wherein
IRS, iRS=1 ..., 6 by formula (1.8.10) obtain.
We have obtained complete Texture now
Side
1.8.5 the fusion of two texture maps
We merge Texture
1And Texture
SideObtain final texture maps Texture
sParticularly, setting threshold lg
MinAnd lg
Max, Texture
sAt arbitrary position Cyn=(lg lt)
TThe value at place is determined by following formula (1.8.11)
Modeling method above we have realized in a common table top PC system.This computer CPU is Intel PIII933, is furnished with the Rambus internal memory of 256M, and Microsoft Windows 2000 Professional operating systems have been installed.For carrying out data acquisition, we have used two identical SANYO VCC-5972P type video cameras, and two identical MatroxMeteorII capture cards gather the two-way synchronization video, and picture resolution is 768*576, true colour imagery, 15 frame/seconds of frame rate.(on the one hand, this configuration makes the public view field of two video cameras bigger to stereo camera by configuration up and down shown in Figure 11 during data acquisition; On the other hand, at this moment the direction of outer polar curve substantially vertically, with the people on the face the actual conditions of the most along continuous straight runs of grey scale change adapt), the typical frame in the stereoscopic video sequence of an actual acquisition is as shown in figure 12.The video of actual acquisition has 50 frames (being N=50), and wherein first frame is the manual positive attitude image of selecting.
The universal model Face that we use
gHuman face animation part from the reference software MOMUSYS of MPEG4 is called ISTFACE, by Portuguese Instituto Superior Tecnico exploitation.It comprises the individual node of 1039 (nVertex), form the individual triangle surface of 1704 (nMesh), form a complete number of people, not only comprise people's face surface (for example zone such as organ such as eyes, nose, mouth, ear and cheek, forehead, hindbrain), also comprise eyeball, tooth, these internals of tongue and neck area.It can be driven by MPEG4 human face animation parameter stream, by the distortion realization animation of three dimensional network lattice point.Figure 13 is the neutral state and the animated state (motion of lip) of this model.
Introduce each step in this embodiment below in detail:
The first step: initialization: use camera arrangement shown in Figure 11 to take three-dimensional video-frequency shown in Figure 12; And be provided with:
● time initial value t=t0=1.Wherein the t express time corresponding to the visual sequence number in the video sequence, is represented the image when pre-treatment; The same express time of t0 is represented the moment that current feature of following the tracks of is detected;
● calculate basis matrix
(wherein the meaning of each known quantity is referring to the explanation in the 1.1.2 joint).Actual numerical value is
F wherein
L=1330.3, f
R=1330.6, u
0L=384, v
0L=288, u
0R=384, v
0R=288;
● width, height: the width of image and height, in specific implementation, width=768, height=576;
● blockx, blocky: the net point number that has comprised in the local window when having determined feature detection, tracking and coupling.In fact, (i j) is the center to this window, and net point adds up to (2*blockx+1) * (2*blocky+1) with tested point.In specific implementation, blockx=3, blocky=3;
● quality, dis_min: the threshold value during feature point detection, the quality of the former controlling features point, the spacing of latter's controlling features point.In specific implementation, quality=0.01, dis_min=10;
● Num_subset, ratio: the constant that uses during attitude parameter initialization (1.1.4 joint), the former is the number of the subclass selected at random, how the latter upgrades { bReliablek} with deciding.Actual value is Num_subset=100, ratio=0.9;
● θ _ bound, K_bound, Inc_refine: be that we are for calculating the threshold value that bReDetect and bRefine use.Their value is: θ _ bound=20, unit degree of being; K_bound=10; Inc_refine=4;
● lg
MinAnd lg
Max: be used for determining Texture
1And Texture
SideBetween separation, be a position between the tail of the eye and the ear upper edge.Particularly, lg
MinCorresponding to right side this point on the face, at Face
sMiddle label is 306; Lg
MaxCorresponding to left side this point on the face, at Face
sMiddle label is 120;
● φ: i.e. known radial basis function during shape corrections, we select
Wherein
Just putting it when the representative function value is negative is 0.
Second step: at visual I
L, tLast compute gradient vector
And
1≤i≤width wherein, any point on 1≤j≤height presentation image, blockx, blocky are the constants relevant with window size.If used the point that exceeds image border when calculating, directly order
Then utilize existing mathematical tool software matlab to G
L, t(i j) makes svd, and gets the combination of its minimum singular value and obtain { min_val (i, j) }, 1≤i≤width, and 1≤j≤height, this is the floating number matrix identical with former dimension of picture.
Lift a concrete instance.I at stereoscopic video sequence shown in Figure 12
L, 1On, (i=346, j=297) gray-scale value for every bit on the 7*7 at center (blockx=blocky=3) the local image is with certain point
And { the gx} and { the gy} matrix is of 7*7
The matrix of the 2*2 that calculates by aforementioned formula
Two singular value is 31561.2 and 22449.8, so min_val (346,297)=22449.8 (min_val is less in two singular values here, and argument of function is the two-dimensional coordinate that image is gone up point).
The 3rd step: the position of seeking several conditions below satisfying in matrix { min_val (i, j) } is as detected unique point.(1) the min_val value at this some place is bigger; Particularly, the maximal value of establishing whole matrix is max_min_val, and then the value at unique point place should be greater than quality*max_min_val, and wherein quality is a pre-set threshold, actually is taken as 0.01; (2) min_val at this some place is the maximal value in its 3*3 neighborhood; (3) distance of any two unique points should be greater than dis_min: if the distance of two unique points is less than or equal to this value, then a less position of min_val value is not considered to unique point; Dis_min is actual to be taken as 10.We detect I by these three rules
L, tOn characteristic point sequence, and be designated as { p
L, t(k) }, 1≤k≤K (t).The result who detects is referring to Figure 14.The 4th step: to arbitrary p
L, t(k), calculate the outer polar curve of its correspondence
And calculate
Wherein (i is that (its vector representation is l=[l to straight line l j)
1l
2l
3], promptly the equation of straight line is l
1I+l
2J+l
3=0) any point on promptly satisfies
P then
L, t(k) match point is p
R, t(k)=(i
Minj
Min)
TAll detected unique points are carried out identical processing, and we obtain I
R, tOn the match point sequence, and be designated as { p
R, t(k) }, 1≤k≤K (t).
The 5th the step: according to arbitrary to the coupling p
L, t(k) and p
R, t(k), calculate
And finally obtain three-dimensional point
All paired couplings are carried out same treatment, and we obtain t three-dimensional feature point sequence { M constantly
t(k) | k=1 ..., K (t) }.The Boolean variable of initialization simultaneously sequence { bReliable
k, k=1 ..., K (t) is Yes entirely, it all is reliable that expression is had a few.
The 6th step: at arbitrary p
L, t(k)=(i j) locates to calculate
And
Then this some t+1 corresponding point constantly in the L video sequence are
All unique points are carried out identical processing, and we obtain I
L, t+1On tracking results, and be designated as { p
L, t+1(k) }, 1≤k≤K (t).
The 7th step: handle the R video sequence with similar method.Particularly, press aforementioned formula earlier, calculate on the t figure at R
{ G
R, t(iR, jR) }, then to arbitrary p
R, t(k)=(iR jR) calculates
Finally have
Then this some t+1 corresponding point constantly in the R video sequence are
All unique points are carried out identical processing, and we obtain I
R, t+1On tracking results, and be designated as { p
R, t+1(k) }, 1≤k≤K (t).
The 8th step: use with similar method of the 5th step and obtain t+1 three-dimensional point sequence constantly.Particularly, according to arbitrary to the coupling p
L, t+1(k) and p
R, t+1(k), calculate
And finally obtain three-dimensional point
All paired tracking results are carried out same treatment, and we obtain t+1 three-dimensional feature point sequence { M constantly
T+1(k) | k=1 ..., K (t) }.
The 9th step: establish sequence { bReliable
k, k=1 ..., value is that the label of the item of Yes is (k among the K (t)
s, s=1 ..., Kr (t), wherein Kr (t) is that value is the total number of Yes.The individual ternary subclass { Set of picked at random Num_subset (actual be taken as 100)
n={ N
1, n, N
2, n, N
3, n, n=1 ..., Num_subset, wherein 1≤N
1, n, N
2, n, N
3, n≤ Kr (t).According to
With
Corresponding relation, estimate the rigid motion parameters R by the method for 1.1.4 joint
T, n, T
T, n, then to all reliable three-dimensional corresponding { M
t(k
s), M
T+1(k
s), s=1 ..., Kr (t) calculates ε
N, s=‖ M
T+1(k
s)-R
T, nM
t(k
s)-T
T, n‖
2, n=1 ..., Num_subset; S=1 ..., Kr (t), and obtain { R by following formula
t, T
t}:
{ R
t, T
t}={ R
N_m, T
N_m, wherein
The tenth step: the sequence of calculation
And renewal { b Reliablek
s, s=1 ..., Kr (t).Particularly, we obtain Err
tIn before K_True less ε
sCorresponding label { s
u, u=1 ..., K_True, then
In be changed to Yes corresponding to the position of these labels, other be changed to No.The concrete calculation procedure of K_True as previously mentioned.
The 11 step: four step method by the 1.1.5 joint are provided with the sign amount bReDetect and the bRefine of two Boolean types, and make t=t+1.
The 12 step: if bRefine=No changeed for the 13 step; Otherwise, make tStart be t constantly before second the frame that bRefine is Yes is set, be t0 if the tStart that obtains so just makes it less than t0.Call general Levenberg-Marquardt algorithm (utilizing the matlab kit) then and find the solution formula (1.1.10), obtain { Ra
TStart+ τ, Ta
TStart+ τ, τ=1 ..., t-tStart; M
TStart(k
Pp), pp=1 ..., Kr (t) }, calculate { R by following formula again
TStart+ τ, T
TStart+ τ, τ=0 ..., t-tStart-1:
R
tStart=Ra
tStart+1,T
tStart=Ta
tStart+1,
The 13 step:, changeed for second step if bReDetect=Yes then makes t0=t; Otherwise, make t=t+1, if t=N at this moment then changeed for the 14 step, otherwise changeed for the 6th step;
Above each step finished attitude estimation (contents of 1.1 joints).When the input of Figure 12 is handled, feature detection and the coupling the 1st, 15,29,38 frame places (being the different values of t0), actual result as shown in figure 14.Each whole number of features that detect and mate are respectively 89,97,107 and 61 (being each K (t0)), (are sequence { bReliable in the reliable always unique point in all moment (i.e. signature tracking between twice detection) linings that t0 remains unchanged
kIn value be the number that Yes is ordered) have 12,18,36,14 respectively.
The 14 step: at I
L, 1And I
R, 1Last manual left inner eye corner, right inner eye corner, the left corners of the mouth and these four two-dimentional remarkable characteristics of the right corners of the mouth selected, actual result as shown in figure 15.According to the coupling between two width of cloth figure of these four positions, can obtain four three-dimensional remarkable characteristics by aforesaid stereo reconstruction algorithm, promptly left inner eye corner
Right inner eye corner
The left side corners of the mouth
The right corners of the mouth
Then by formula (1.2.1) is with original universal model Face
gBe deformed into Face
Init
Three scale factor s wherein
x, s
y, s
zComputing method as follows: the order
Be modeling object and Face
gThe ratio of middle Seg_E length (distances of two inner eye corners); Order
Be the ratio of Seg_EM length (distance at mouth center is arrived at two centers); Make s
z=s
yWherein
Be at Face
gGo up the three-dimensional coordinate of these four principal character points.
The 15 step: then we find the solution the rigid body translation R that the generic coordinate is tied to the L camera coordinate system
G, 1, T
G, 1We select
With
These three three-dimensional corresponding point are found the solution by the method for aforesaid attitude parameter initialization one joint.
Obtained R
G, 1, T
G, 1After, R calculates by following formula in system
G, t, T
G, t, t=2 ..., N:
R
g,t=R
t-1R
g,t-1,T
g,t=R
t-1*T
g,t-1+T
t-1,t=2,...,N
Then system can judge the frame side corresponding to lateral plan automatically, promptly selects side for making R
G, tRotation angle near the moment t of 90 degree.Our actual side=50 that obtains, the corresponding anglec of rotation is 86.9 degree.
The 16 step: at visual I
L, sideThe side unique point is picked up in last craft, and experimental result is designated as shown in figure 16
Kk=1 ..., NumSF, wherein NumSF is the number of side unique point, actual is 24.
The 17 step: find the solution formula (1.3.1) by preceding method, obtain three-dimensional side unique point
The 18 goes on foot: the radial basis function algorithm in saving by aforesaid 1.3 is Face
InitBe deformed into Face
1Point among the Subset comprises two classes: a class is three-dimensional remarkable characteristic, totally 4, promptly
Another kind of is the side unique point, and promptly 17 steps were obtained
Kk=1 ..., NumSF.
The 19 step: obtain Face by the method for calculating projected outline in aforementioned 1.4 joints
1In the point that constantly projected to image plane at 1 o'clock
Face
1Be exactly Face there
Proj
The 20 step: the manual I that marks
L, 1On facial contour, as shown in Figure 7.This is the polygon of a closure, is designated as
Be continuous two-dimentional vertex list, nNumCI1 is the number on two-dimentional summit.
The 21 step: (1.4.1) determines Cont according to aforesaid formula
1Mid point is at Face
2In three-dimensional coordinate
The 22 goes on foot: the radial basis function algorithm in saving by aforementioned 1.3 is Face
1Be deformed into Face
2The new point that adds is to obtain in the 21 step among the Subset
The 23 step: the manual I that marks
L, 1Front face feature contour point on the figure (experimental result is Fig. 8), the result is
Wherein nNumCF is the total number of point.
The 24 step: find the solution the 23 three-dimensional coordinate that goes on foot the feature contour point mark by aforesaid formula (1.5.1)
The 25 goes on foot: the radial basis function algorithm in saving by aforementioned 1.3 is Face
2Be deformed into Face
3The new point that adds is to obtain in the 24 step among the Subset
The 26 step: system judges the frame int1 corresponding to medial view 1 automatically, promptly selects int1 for making R
G, tRotation angle near the moment t of 25 degree.Then at I
L, int1Craft on the figure marks facial contour, as shown in Figure 9.Represent with closed polygon equally, be designated as
NNumCI int1 is a number of vertex.
The 27 step: obtain Face by the method for calculating projected outline in aforementioned 1.4 joints
3When moment int1, project to the point of image plane
Face
3Be exactly Face there
Proj
The 28 step: according to Cont_img
Int1Determine Cont
Int1Mid point is at Face
4In three-dimensional coordinate.Particularly, for appointing
Definition
Wherein
Be that this point is at Face
3In three-dimensional coordinate.Then this is at Face
4In coordinate
Satisfy
Parametric t wherein
Line2According to two-dimentional straight line
With closed polygon Cont_img
Int1Intersection point obtain, method as 1.4 the joint as described in.To Cont
Int1In all are pressed top step and calculate, obtain
The 29 goes on foot: the radial basis function algorithm in saving by aforementioned 1.3 is Face
3Be deformed into Face
4The new point that adds is to obtain in the 28 step among the Subset
The 30 step: system judges the frame int2 corresponding to medial view 2 automatically, promptly selects int2 for making R
G, tRotation angle near the moment t of 40 degree.Then at I
L, int2Craft on the figure marks facial contour, as shown in figure 10.Represent with closed polygon equally, be designated as
NNumCIint2 is a number of vertex.
The 31 step: obtain Face by the method for calculating projected outline in aforementioned 1.4 joints
4When moment int2, project to the point of image plane
Face
4Be exactly Face there
Proj
The 32 step: according to Cont_img
Int 2Determine Cont
Int 2Mid point is at Face
5In three-dimensional coordinate.Particularly, for arbitrary
Definition
Wherein
Be that this point is at Face
4In three-dimensional coordinate.Then this is at Face
5In coordinate
Satisfy
Parametric t wherein
Line3According to two-dimentional straight line
With closed polygon Cont_img
Int2Intersection point obtain, method as 1.4 the joint as described in.To Cont
Int2In all are pressed top step and calculate, obtain
The 33 goes on foot: the radial basis function algorithm in saving by aforementioned 1.3 is Face
4Be deformed into Face
5The new point that adds is to obtain in the 32 step among the Subset
Face
5Be final shape Face
s
We have carried out shape corrections altogether five times like this, and corresponding to the shape corrections A-E of 1.3 to 1.7 joints, in fact each goes on foot counting out shown in following table 1 among the Subset that uses.
Different models | Initiate Subset point and number thereof | Subset point sum |
Face 1 | 4 positive notable features (two inner eye corners, two corners of the mouths)+24 side features | 28 |
Face 2 | Projected outline's point (41) of front view | 69 |
Face 3 | The feature contour point (44) of two, two nostrils and mouth in the front view | 113 |
Face 4 | Projected outline's point (37) during medial view 1 | 150 |
Face s | Projected outline's point (44) during medial view 2 | 194 |
Each goes on foot position and number that initiate Subset is ordered in table 1 shape corrections
The 34 step: to Face
sIn arbitrary three-dimensional point
By formula
Do the cylinder mapping, the result is { Cyn
Pt}={ (lg
PtLt
Pt).
The 35 step: will own
At front attitude parameter R
G, 1, T
G, 1Down by formula
Carry out projection, obtain
Then by formula (1.8.3) obtains positive texture maps Texture
1, concrete instance as shown in figure 17.
The 35 step: similar with the 34 step, obtain Face
sSide grain figure Texture
SideParticularly, will own
In the lateral attitude parameters R
G, side, T
G, sideDown by formula
Carry out projection, obtain
Then by formula (1.8.7) obtains side grain figure Texture
Side, concrete instance as shown in figure 18.
The 36 step: on cylinder figure { (lg, lt) }, carry out the texture reflection.That side face that directly obtains texture in the actual experiment is the left side, thus this step according to the texture information in left side by formula (1.8.8) generate the right side face.At this moment complete Texture
SideAs shown in figure 19.
The 37 step: we by formula
Merge Texture
1And Texture
SideObtain final texture maps Texture
s, actual result as shown in figure 20.
We are final, shone upon Texture
sModeling result Face
sAs shown in figure 21.
Claims (4)
1. merge the method for building up of the human face three-dimensional model of various visual angles, multi thread two-dimensional signal, the method that contains the corresponding faceform of establishment of two dimension, it is characterized in that: it at first use with the people on the face, corresponding two the identical and outer polar curve direction video cameras vertically of upper/lower positions people's face of taking modeling object under the situation of the amimia variation of people's face progressively changes about 90 video sequences of spending to the side from the front, gather up and down the two-way synchronization video with two identical capture cards again and import in the computing machine; Then, from the universal model Face of people's face
gBeginning is progressively made amendment according to the two-dimensional signal of the modeling object that obtains, and finally obtains specific people's faceform Face
s, the three-dimensional point coordinate among the just people who wherein the revises three-dimensional node tabulation P on the face, and people triangle surface tabulation F on the face remains unchanged promptly only changes mould shapes P and the topological structure F of model is constant; Particularly, it contains successively and has the following steps:
(1). input stereoscopic video sequence I
L, tAnd I
R, t:
L, R be corresponding to two video cameras, t=1 ..., N, the different images constantly of expression N frame: the two-dimensional array that I then is made up of pixel, each pixel comprise R, G, three components of B; The moment of t=1 is selected by people's judgement is manual corresponding to the positive attitude of people's face;
(2). attitude is estimated, is promptly found the solution the attitude parameter in the number of people rotary course, i.e. the rigid body translation coefficient:
Interframe movement with people's face in the above-mentioned image sequence of following rigid body translation formulate:
M
t+1=R
tM
t+T
t,t=1,..,N-1,
M
t, M
T+1The people's coordinate of any point in the L camera coordinate system on the face when representing t frame, t+1 frame respectively; M
t, M
T+1Being people's same position on the face at difference three-dimensional coordinate constantly, promptly is the three-dimensional corresponding point of interframe; Motion with reliable three-dimensional point just can be estimated attitude parameter;
{ R
t, T
t, t=1 ..., N-1} is rotation and the translational component of t frame to t+1 frame attitude parameter;
Attitude is estimated to contain successively to have the following steps:
(2.1), the initial moment: carry out the detection and the coupling of two dimensional character earlier, obtain the three-dimensional point in the initial moment again by stereo reconstruction:
At first, feature detection frame t0=t is set, and at I
L, tMake two dimensional character on the figure and detect, its step is as follows:
With existing KLT algorithm computation I
L, tUnique point p on the figure (x=i, the image change degree G that y=j) locates (i, j):
(i+1 is the gray scale of this some place image j) to I, and the rest may be inferred by analogy;
With tested point p is that the window W at center is of a size of (2*blockx+1) * (2*blocky+1); And then obtain the minimum singular value sing_min of G with existing matlab kit, selecting the very big some p of sing_min again is detected unique point; Being designated as { p in t detected unique point of the moment
L, t(k) }, L, t presentation image sequence number, K (t) is the unique point sum, k is its label, 1≤k≤K (t);
Secondly, to two video camera L of synchronization, two width of cloth images that R shooting Same Scene obtains are that stereographic map is to making the two dimensional character coupling; The projection of same three-dimensional point in different images is two-dimensional points all, and it is corresponding to be called two dimension between them mutually, stereographic map to two-dimentional correspondence be called coupling:
For I
L, tCertain 1 p among the figure
L, t(k)=(x
L, t(k) y
L, t(k))
T, its match point p
R, t(k)=(x
R, t(k) y
R, t(k))
TShould be positioned at one by p
L, t(k) on Jue Ding the straight line, in this one-dimensional space, select to make the point of total gray difference minimum in the local window W as p
L, t(k) match point p
R, t(k), the expression formula of total gray difference:
(i j) is straight line
On any point; The matching result of carving t that clocks is { p
R, t(k) }, total number is that K (t) is individual, every all with p
L, t(k) corresponding, on the straight line of following formulate, search for, make Diff (i, j) Zui Xiao position:
Once more, obtain the synchronization stereographic map to matching result p
L, tAnd p
R, tAfter, obtain the three-dimensional point M that represents with camera coordinate system by stereo reconstruction
M, t, m is video camera L or R, M
M, tBe the coordinate of t moment three-dimensional point in the m camera coordinate system;
Configuration relation formula according to two video cameras:
M
L=R
cM
R+ T
c, M
L, M
RBe respectively the coordinate of same point in these two camera coordinate systems; And camera perspective projection formula:
Obtain the coordinate M of three-dimensional point in video camera L, R
L, tAnd M
R, t:
At the remaining part that attitude is estimated, three-dimensional point all uses the coordinate in the L camera coordinate system to represent, therefore omits subscript L, is about to M
L, tBrief note is M
t
(2.2), the follow-up moment: t+1 stereographic map is constantly followed the tracks of carry out two dimensional character with the KLT algorithm, obtained the three-dimensional point in each follow-up moment more equally by stereo reconstruction; Tracking is meant that the two dimension between the different time chart pictures that same video camera takes is corresponding:
When two dimensional character was followed the tracks of, the unique point of establishing detection was in image I
M, t(p), m is on L or the R, and (x=i is arbitrary unique point y=j) to p, and then it is in image I
M, t+1On tracking results be
Wherein
G as above-mentioned, and
Note is followed the tracks of the unique point that obtains and is { p
M, t+1(k) }, m represents L or R, and t+1 is constantly:
For video camera L be:
For video camera R be:
Calculate three-dimensional point { M according to arbitrary trace point to coupling
T+1(k) | k=1 .., K (t) }, as previously mentioned, omitted subscript L here;
Wherein
(2.3), attitude parameter initialization: according to one group of three-dimensional corresponding point { M between two moment t and the t+1
t(k) | k=1 ..., K (t) } and { M
T+1(k) | k=1 ..., K (t) } and the expression reliability sequence { bReliable
k, k=1 ..., K (t) is from formula M
T+1(k)=R
tM
t(k)+T
t, t=1 ..., use among the N-1 based on the Robust Estimation algorithm of minimum intermediate value and find the solution R
t, T
t:
If: definition is also brought in constant renewal in boolean's array { bReliableZ
kBe reliable tolerance sequence, and Yes represents that reliably No represents unreliable, wherein value is that the label of the item of Yes is { k
s, s=1 ..., Kr (t), Kr (t) they are that value is the sum of the item of Yes, from wherein choosing Num_subset subclass, each subclass contains three couples of three-dimensional corresponding point, i.e. { Set
n={ N
1, n, N
2, n, N
3, n, n=1 ..., Num_subset, 1≤N
1, n, N
2, n, N
3, n≤ Kr (t), three-dimensional corresponding point are in each ternary subclass:
With
Can note by abridging and be { (M
Im, M
Im') | im=0,1,2}, its represents three pairs of not three-dimensional corresponding point of conllinear, R
t, T
tSatisfy M
Im'=R
tM
Im+ T
t| im=0,1,2; From following formula, obtain R and T then, omit subscript t:
Wherein
[r]
xBe according to 3*1 vector r=[r
xr
yr
z]
TThe antisymmetric matrix of definition:
And
It is unit matrix;
Translational component T is estimated by following formula:
To each subclass Set
nEstimate attitude parameter by above-mentioned formula, be designated as R
T, n, T
T, nAgain to all reliable three-dimensional corresponding { M
t(k
s), M
T+1(k
s), s=1 ..., Kr (t) is calculated as follows corresponding point M
t(k
s), M
T+1(k
s) and attitude parameter R
T, n, T
T, nMatching degree ε
N, s, this value is more little, and then both meet more:
ε
n,s=‖M
t+1(k
s)-R
t,nM
t(k
s)-T
t,n‖
2;
Again from set { R
T, n, T
T, n, n=1 ..., choose R among the Num_subset
t, T
t:
Expression is got set { ε at subscript s
N, s| s=1 ..., Kr (t) intermediate value, it constitutes a new array to different n
And
Then be in this intermediate value array, to select to get that n of minimum value;
(2.4), at t=t+1 constantly, be provided with:
Whether need to call again the Boolean denotation amount bReDetect of feature detection and matching module;
Whether need to carry out the Boolean denotation amount bRefine of attitude refinement;
(2.5), when t=N, attitude is estimated to finish;
(3). model initialization and attitude initialization:
The model initialization is the initial step of universal model specialization method, and it is from universal model Face
gObtain an initialization model Face
Init, promptly universal model being carried out change of scale, available following formula carries out:
The attitude initialization is in order to obtain the correspondence between model and image, promptly to calculate the rigid body translation that is tied to the L camera coordinate system from people's face generic coordinate; Promptly find the solution R from following formula
G, 1, T
G, 1:
M
1=R
G, 1M
Init+ T
G, 1, M
1And M
InitPeople's face and Face when being moment t=1 respectively
InitBetween arbitrary group of corresponding point; It is that left eye, RE are that right eye and Mid_M are the corresponding point of these three positions, mouth center at LE that the present invention gets above-mentioned two models, by above-mentioned
, θ, R, [r]
x, the various R that tries to achieve successively of T
G, 1And T
G, 1
Press following formula again from R
t, T
t, t=1 ..., N-1 obtains R
G, t, T
G, t, t=2 ..., N:
R
g,t=R
t-1R
g,t-1,T
g,t=R
t-1*T
g,t-1+T
t-1,t=2,...,N;
(4). utilize three-dimensional information correction faceform:
(4.1), pick up the side unique point by hand:
If: the manual side unique point that marks is
Kk=1 ..., NumSF, NumSF=24 are the number of side unique point, side represents the moment corresponding to lateral plan;
(4.2), shape corrections A: Face
InitBe deformed into Face
1We determine the reposition of a subclass Subset among people's face three-dimensional node tabulation P earlier, determine the coordinate of all nodes among the P again by a known radial basis function algorithm;
The radial basis function algorithm is as described below: establishing will be with Face
StartBe deformed into Face
New, the node among the known Subset is at Face
NewIn coordinate be { New
1..., New
s, remember that it is at Face
StartIn corresponding point be { Sta
1..., Sta
s, S is the number of Subset mid point; For the arbitrary position pt in the model, it is at Face now
NewIn coordinate
Can obtain by following known formula:
Be that the pt point is at Face
StartIn coordinate, and { C
1..., C
sCan obtain by following formula
During " shape corrections A " radial basis function algorithm on utilize, Face is set
Start=Face
Init, Face
New=Face
1, and the point among the Subset comprises two classes: a class is people's left eye interior angle LE, right eye interior angle RE, the outer lip LM in a left side, four three-dimensional remarkable characteristics of right outer lip RM on the face, is drawn by above-mentioned stereo reconstruction method by two-dimentional remarkable characteristic; Another kind of is above-mentioned 24 side unique points, and their three-dimensional position obtains from the manual two-dimensional points that marks by following step:
If: the three-dimensional coordinate of side unique point in people's face generic coordinate system is
Then, have for arbitrary kk
Wherein
Be the representation in components of attitude parameter, be known quantity; Can directly obtain with the On Solving System of Linear Equations method
In two unknown numbers
With
(4.3), shape corrections B: press aforesaid radial basis function algorithm Face
1Be deformed into Face
2
If Face
Start=Face
1, Face
New=Face
2Among the Subset except comprising the point of " shape corrections A " lining having determined, the point that also has some new interpolations, i.e. the three-dimensional point that the manual facial contour that marks recovers out under the positive attitude, the basic skills that obtains these three-dimensional point is current faceform Face
1Image plane under positive attitude is done perspective projection and is calculated the two-dimensional silhouette point of three-dimensional model projection result, and the coupling that depends on facial contour on these point and the above-mentioned image at last and be closed polygon is calculated its three-dimensional coordinate:
At first, calculate Face
1In the projected outline of the t=1 moment on the image plane of the manual front view of selecting
Be the label of model three-dimensional node tabulation P mid point,
And nNum
1Be Cont
1The number on middle summit, they are to the new point that adds of Subset among the shape corrections B; They are to select by the intersection point of judging projection line and model, and promptly on a certain point, projection line and model can not have other intersection points, and its algorithm is as follows: Face
1Be three-dimensional model, the perspective projection center is
Known quantity was then calculated the projection line that Center orders and the intersection point of model to all summits in the model, judged that in view of the above this summit is whether in projected outline, if M
NowBe Face
1Middle any point, Plane
JpBe Face
1In the plane that constitutes of arbitrary dough sheet, calculated Center point and M
NowStraight line, i.e. projection line is with Plane
JpIntersection point be C
NowIf for a certain dough sheet 1≤jp≤nMesh, some C
NowAt line segment CM
NowInside, M then
NowNot in projected outline; Otherwise be projected outline's point;
Calculate Cont again
1In any point
At Face
2Three-dimensional coordinate: the facial contour of establishing on the above-mentioned image is
It is the polygon that is made of the two-dimensional points tabulation, and nNumCI1 is the number of point; Then
At Face
2Three-dimensional coordinate
Wherein,
Be that the pti point is at Face
1In coordinate; Warp
The projecting direction of point is v
PnBe known quantity; v
nFor the pti point at Face
gIn normal direction;
Parametric t
LineCan be according to two-dimentional straight line
With closed polygon Cont_img
1Intersection point obtain:
If Cont_img
1In arbitrary line segment be seg=(x
0+ sd
xy
0+ sd
y)
T, 0≤s≤1, s is the line segment parameter, other all are known quantities; Calculate by following formula earlier
All seg are obtained this two numerical value si and t
LineDo not have for the formula above a lot of seg and to separate, promptly the matrix in the formula is irreversible; The seg that has the si that separates and obtain to satisfy 0≤si≤1 has one or two, gets and point
That seg that distance is near, the t that it obtains
LineBeing institute asks;
(4.4), shape corrections C: press aforesaid radial basis function algorithm Face
2Be deformed into Face
3
If Face
Star,=Face
2, Face
New=Face
3, except comprising the point of " shape corrections A, B " lining having determined, also have the point of some new interpolations among the Subset, promptly from the manual front face feature contour point that marks: the three-dimensional point that the point of eyes, nostril and mouth is recovered out:
If the two-dimensional coordinate of certain feature contour point is
This point is at Face
2In coordinate be
If this Z coordinate is correct as calculated, promptly it is at Face
3In the Z coordinate equal
Then it is at Face
3In coordinate be
Be that these feature contour point projections under positive attitude equal the unique point on the actual image;
(4.5), shape corrections D: press aforesaid radial basis function algorithm Face
3Be deformed into Face
4
If Face
Start=Face
3, Face
New=Face
4, except comprising the point of " shape corrections A, B, C " lining having determined, also have the point of some new interpolations among the Subset, even promptly from manual medial view 1 R that marks
G, tRotation angle recover the three-dimensional point of coming out in the facial contour near the moment t of 25 degree, concrete steps are as follows: establish this view corresponding to moment int1, described identical with shape corrections B, calculate Face earlier
3Projected outline when int1
Be the label of P mid point, nNum_int1 is the summit number in the projected outline; For any point
It is at Face
3In three-dimensional coordinate
And
Be the coordinate of L video camera photocentre in the generic coordinate system as previously mentioned, then this is at Face
4In coordinate should satisfy
Parametric t
Line2Equally according to straight line
With closed polygon Cont_img
Int1Intersection point obtain;
(4.6), shape corrections E: press aforesaid radial basis function algorithm Face
4Be deformed into Face
5, promptly final shape Face
s
If Face
Start=Face
4, Face
New=Face
5, except comprising the point of " shape corrections A, B, C, D " lining having determined, also have the point of some new interpolations among the Subset, even promptly from manual medial view 2 R that mark
G, tRotation angle recover the three-dimensional point of coming out in the facial contour near the moment t of 40 degree, concrete recovering step is identical with shape corrections D;
(5). texture:
Present texture is front and the lateral plan of taking by the L video camera, i.e. I
L, 1And I
L, side, generating, texture is to be the model Face that creates
sGeneration appears to have the cylinder texture maps Texture of the same sense of reality of photograph
s, promptly will be in image conversion to the unified cylindrical coordinate system of gathering and merge, it contains successively and has the following steps:
(5.1), generate cylinder unwrapping figure:
At first to the net result Face of shape corrections E
sIn arbitrary three-dimensional point
Do the cylinder mapping by following formula:
(5.2), generate positive texture maps:
Have a few
At known front attitude parameter R
G, 1, T
G, 1Press following formula down at front view I
L, 1Last projection,
Be Face
sMiddle three-dimensional point
At I
L, 1On two-dimensional projection:
Then, I
L, 1In each pixel mapping to the Cyn plane, promptly to Face
sIn arbitrary dough sheet facet=(pt
1Pt
2Pt
3)
T, m
Proj=Δ (p
1, p
2, p
3) represent at these 3 at I
L, 1The triangle that constitutes on the plane, m
Cyn=Δ (p
1', p
2', p
3') be the corresponding triangle on the Cyn plane, promptly
IT1=1,2,3; Diabolo m again
CynInterior point [lg lt] is calculated as follows [x y]:
a
1~a
6Obtain by following formula:
Make positive texture maps again
(5.3), generate side grain figure:
Have a few
In known lateral attitude parameters R
G, side, T
G, sidePress following formula down at lateral plan I
L, sideLast projection,
Be Face
sMiddle three-dimensional point
At I
L, sideOn two-dimensional projection:
Again to Face
sIn arbitrary dough sheet facet=(pt
1Pt
2Pt
3)
T, establish ms
Proj=Δ (ps
1, ps
2, ps
3),
IT2=1,2,3; Diabolo m again
CynInterior point [lg lt] is calculated as follows [xs ys]:
As
1~as
6Obtain by following formula:
Make side grain figure again
(5.4), the mirrored sides texture maps is to obtain the texture maps of opposite side, and this is because Face
sTopological structure be complete symmetry;
If: that side arbitrary tri patch m on the face that directly obtains texture
Cyn=Δ (p
1, p
2, p
3), it is m at opposite side symmetrical dough sheet on the face
Cyn'=Δ (p
1', p
2', p
3'), wherein
iT3=1,2,3;
To m
CynIn any point p=(lg lt)
TBe calculated as follows [lg ' lt '],
Rs
1~rs
6Obtain by following formula:
Generate opposite side from a directly veined lateral reflection again:
(5.5), the fusion of positive and the texture maps side obtains final texture maps Texture
s
Setting threshold lg
MinAnd lg
Max, Texture
sAt arbitrary position Cyn=(lg lt)
TThe value at place provides by following formula:
2. the method for building up that merges the human face three-dimensional model of various visual angles, multi thread two-dimensional signal according to claim 1, it is characterized in that: sequence { bReliable is reliably measured in the renewal in the described attitude estimating step
kMethod, it contains successively and has the following steps:
(1). according to correct corresponding number K_True is calculated in the statistical study of sequence Errt;
(1.1), establish: the initial value of K_True is Kr (t) * ratio, and ratio is predefined constant; Err
t={ ε
s=‖ M
T+1(k
s)-R
tM
t(k
s)-T
t‖
2| s=1 ..., Kr (t) in be E by the subsequence of preceding K_True element composition of ordering from small to large;
(1.2), calculate the average of E
And standard deviation
(1.3), at Err
tThe middle searching satisfied ‖ ε
s-ε ‖〉point of 3* σ, establishing number is InValid_T;
(1.4), if 0≤Kr (t)-K_True-InValid_T≤1, promptly K_True correspondence is correct, stops; Otherwise, change step (1.5);
(1.5), if K-K_True-InValid_T〉1, make K_True add 1, change step (1.2); Otherwise, change step (1.6);
(1.6), if K-K_True-InValid_T<0 makes K_True subtract 1, change step (1.2);
(2). establish Err
tIn before K_True less ε
sCorresponding label is { s
u, u=1 ..., K_True is { b Reliablek
sIn be Yes corresponding to the position assignment of these labels, other assignment are No.
3. the method for building up that merges the human face three-dimensional model of various visual angles, multi thread two-dimensional signal according to claim 1 is characterized in that: the two method of the Boolean denotation amount bReDetect whether setting up in the described attitude estimating step will call feature detection and matching module again, the Boolean denotation amount bRefine that whether will carry out the attitude refinement is as follows:
(1). if t=N, promptly current image is the last frame in the sequence, bReDetect=No then, bRefine=Yes finishes; Otherwise, change step (2);
(2). if current R
tRotation angle θ
tSurpass preset threshold value θ _ bound, bReDetect=Yes then, bRefine=Yes finishes; Otherwise, change step (3);
(3). (i.e. { bReliable if the number Kr (t) of reliable three-dimensional point
k, k=1 ..., value is counting of Yes among the K (t)) less than preset threshold value K_bound, bReDetect=Yes then, bRefine=Yes; Otherwise, change step (4);
(4) .bReDetect=No; If t-t0 is the multiple of the constant Inc_refine that presets, bRefine=Yes then, i.e. refinement is just to do once every Inc_refine; Otherwise bRefine=No;
(5). make t=t+1, enter next processing constantly.
4. the method for building up that merges the human face three-dimensional model of various visual angles, multi thread two-dimensional signal according to claim 3, it is characterized in that: in the described attitude estimating step, bRefine is that Yes is illustrated in the current time refinement that gestures, and promptly draws more accurate attitude parameter:
Come all relevant attitude parameter { R of refinement if be chosen in N continuous um=t-tStart+1 correct all the time three-dimensional correspondence of the moment
TStart+ τ, T
TStart+ τ, τ=0 ..., t-tStart-1, tStart are the initial moment of refinement computing, now get t second before constantly the moment that bRefine is Yes is set, if tStart<t0 gets it and is t0; In a refinement computing, calculate { Ra earlier
TStart+ τ, Ta
TStart+ τ, τ=1 ..., t-tStart; M
TStart(k
Pp), pp=1 ..., Kr (t) }, make that the total error f_obj of two-dimentional re-projection reaches minimum value in all images:
Ra wherein
TStart+ τ, Ta
TStart+ τBe to be carved into tStart+ τ rigid body translation coefficient constantly during from tStart; { M
TStart(k
Pp), pp=1 .., Kr (t) is reliable three-dimensional point array in this Num the moment, and Kr (t) is individual altogether, and pp is the label of three-dimensional point; M is L or R, the expression video camera;
Be in m video camera, tStart+ the τ two-dimensional detection in the image or the result of tracking constantly, promptly
Optimize the exact value that f_obj just can obtain attitude parameter; Earlier find the solution { Ra with the Levenberg-Marquardt algorithm
TStart+ τ, Ta
TStart+ τ, τ=1 ..., t-tStart; M
TStart(kpp), pp=1 ..., Kr (t) }; Use following formula { Ra again
TStart+ τ, Ta
TStart+ τ, τ=1 ..., t-tStart is transformed into { R
TStart+ τ, T
TStart+ τ, τ=0 ..., t-tStart-1:
R
tStart=Ra
tStart+1,T
tStart=Ta
tStart+1,
Priority Applications (1)
Application Number | Priority Date | Filing Date | Title |
---|---|---|---|
CNB02146278XA CN100483462C (en) | 2002-10-18 | 2002-10-18 | Establishing method of human face 3D model by fusing multiple-visual angle and multiple-thread 2D information |
Applications Claiming Priority (1)
Application Number | Priority Date | Filing Date | Title |
---|---|---|---|
CNB02146278XA CN100483462C (en) | 2002-10-18 | 2002-10-18 | Establishing method of human face 3D model by fusing multiple-visual angle and multiple-thread 2D information |
Publications (2)
Publication Number | Publication Date |
---|---|
CN1404016A CN1404016A (en) | 2003-03-19 |
CN100483462C true CN100483462C (en) | 2009-04-29 |
Family
ID=4751041
Family Applications (1)
Application Number | Title | Priority Date | Filing Date |
---|---|---|---|
CNB02146278XA Expired - Fee Related CN100483462C (en) | 2002-10-18 | 2002-10-18 | Establishing method of human face 3D model by fusing multiple-visual angle and multiple-thread 2D information |
Country Status (1)
Country | Link |
---|---|
CN (1) | CN100483462C (en) |
Families Citing this family (47)
Publication number | Priority date | Publication date | Assignee | Title |
---|---|---|---|---|
CN100375124C (en) * | 2003-12-17 | 2008-03-12 | 中国科学院自动化研究所 | A skeletonized object rebuild method |
US7598956B2 (en) * | 2004-04-15 | 2009-10-06 | Microsoft Corporation | Blended object attribute keyframing model |
JP4449723B2 (en) * | 2004-12-08 | 2010-04-14 | ソニー株式会社 | Image processing apparatus, image processing method, and program |
CN100407798C (en) * | 2005-07-29 | 2008-07-30 | 北京大学 | Three-dimensional geometric mode building system and method |
US8014565B2 (en) * | 2005-08-26 | 2011-09-06 | Sony Corporation | Labeling used in motion capture |
DE102005044652B4 (en) * | 2005-09-19 | 2009-12-10 | Siemens Ag | Method for generating 2D reconstruction images from a 3D image data set of an examination object, in particular taken by means of a magnetic resonance device, in the context of image post-processing |
CN100346745C (en) * | 2005-09-27 | 2007-11-07 | 四川大学 | Human cranlofacial 3-D measurer and measuring method |
JP2007281680A (en) * | 2006-04-04 | 2007-10-25 | Sony Corp | Image processor and image display method |
CN100386778C (en) * | 2006-06-15 | 2008-05-07 | 西安交通大学 | Human face image age changing method based on average face and senile proportional image |
CN100416611C (en) * | 2006-09-14 | 2008-09-03 | 浙江大学 | Network topology model construction based three-dimensional human face cartoon making method |
CN100416612C (en) * | 2006-09-14 | 2008-09-03 | 浙江大学 | Video flow based three-dimensional dynamic human face expression model construction method |
CN100468465C (en) * | 2007-07-13 | 2009-03-11 | 中国科学技术大学 | Stereo vision three-dimensional human face modelling approach based on dummy image |
CN101488224B (en) * | 2008-01-16 | 2011-01-19 | 中国科学院自动化研究所 | Characteristic point matching method based on relativity measurement |
CN101271578B (en) * | 2008-04-10 | 2010-06-02 | 清华大学 | Depth sequence generation method of technology for converting plane video into stereo video |
CN101394479B (en) * | 2008-09-25 | 2010-06-16 | 上海交通大学 | Teacher movement tracing method based on movement detection combining multi-channel fusion |
CN101383046B (en) * | 2008-10-17 | 2011-03-16 | 北京大学 | Three-dimensional reconstruction method on basis of image |
CN101464132B (en) * | 2008-12-31 | 2014-09-10 | 北京中星微电子有限公司 | Position confirming method and apparatus |
CN101540045B (en) * | 2009-03-25 | 2011-07-27 | 湖南大学 | Multi-source image fusion method based on synchronous orthogonal matching pursuit |
US8334872B2 (en) * | 2009-05-29 | 2012-12-18 | Two Pic Mc Llc | Inverse kinematics for motion-capture characters |
CN101625768B (en) * | 2009-07-23 | 2011-11-09 | 东南大学 | Three-dimensional human face reconstruction method based on stereoscopic vision |
CN101930626B (en) * | 2010-08-04 | 2012-07-04 | 北京大学 | Method and system for computing three-dimensional space layout based on scattered perspective image |
CN102262727A (en) * | 2011-06-24 | 2011-11-30 | 常州锐驰电子科技有限公司 | Method for monitoring face image quality at client acquisition terminal in real time |
EP2754130A4 (en) * | 2011-08-09 | 2016-01-06 | Intel Corp | Image-based multi-view 3d face generation |
CN102436668A (en) * | 2011-09-05 | 2012-05-02 | 上海大学 | Automatic Beijing Opera facial mask making-up method |
CN103186233B (en) * | 2011-12-31 | 2016-03-09 | 上海飞来飞去新媒体展示设计有限公司 | Panoramic interaction control method for eye location |
CN102629382B (en) * | 2012-03-05 | 2014-07-16 | 河南理工大学 | Feature point matching method based on geometric compatibility |
US9189884B2 (en) | 2012-11-13 | 2015-11-17 | Google Inc. | Using video to encode assets for swivel/360-degree spinners |
CN103839223B (en) * | 2012-11-21 | 2017-11-24 | 华为技术有限公司 | Image processing method and device |
CN102999942B (en) * | 2012-12-13 | 2015-07-15 | 清华大学 | Three-dimensional face reconstruction method |
CN104008366A (en) * | 2014-04-17 | 2014-08-27 | 深圳市唯特视科技有限公司 | 3D intelligent recognition method and system for biology |
CN104361362A (en) * | 2014-11-21 | 2015-02-18 | 江苏刻维科技信息有限公司 | Method for obtaining locating model of human face outline |
CN105809664B (en) * | 2014-12-31 | 2020-03-17 | 北京三星通信技术研究有限公司 | Method and device for generating three-dimensional image |
CN105138956B (en) * | 2015-07-22 | 2019-10-15 | 小米科技有限责任公司 | Method for detecting human face and device |
CN105678321B (en) * | 2015-12-31 | 2019-06-21 | 北京工业大学 | A kind of estimation method of human posture based on Fusion Model |
CN106372629B (en) * | 2016-11-08 | 2020-02-07 | 汉王科技股份有限公司 | Living body detection method and device |
CN108509855B (en) * | 2018-03-06 | 2021-11-23 | 成都睿码科技有限责任公司 | System and method for generating machine learning sample picture through augmented reality |
CN109191507B (en) | 2018-08-24 | 2019-11-05 | 北京字节跳动网络技术有限公司 | Three-dimensional face images method for reconstructing, device and computer readable storage medium |
CN109448090B (en) * | 2018-11-01 | 2023-06-16 | 北京旷视科技有限公司 | Image processing method, device, electronic equipment and storage medium |
DE102019100822A1 (en) * | 2019-01-14 | 2020-07-16 | Lufthansa Technik Aktiengesellschaft | Boroscopic inspection procedure |
CN109872343B (en) * | 2019-02-01 | 2020-03-17 | 视辰信息科技(上海)有限公司 | Weak texture object posture tracking method, system and device |
CN110070611B (en) * | 2019-04-22 | 2020-12-01 | 清华大学 | Face three-dimensional reconstruction method and device based on depth image fusion |
GB2584896B (en) * | 2019-06-20 | 2022-10-05 | De Beers Uk Ltd | Gemstone planning |
CN110837797A (en) * | 2019-11-05 | 2020-02-25 | 中国医学科学院北京协和医院 | Nose adjusting method based on face surface three-dimensional grid and application thereof |
DE112020006748T5 (en) * | 2020-02-17 | 2022-12-22 | Mitsubishi Electric Corporation | Head pose estimation apparatus and head pose estimation method |
CN114748083A (en) * | 2022-03-18 | 2022-07-15 | 上海涛影医疗科技有限公司 | Multi-view exposure X-ray image positioning method and system |
CN115278080A (en) * | 2022-07-28 | 2022-11-01 | 北京五八信息技术有限公司 | Mask generation method, mask generation equipment and storage medium |
CN116778043B (en) * | 2023-06-19 | 2024-02-09 | 广州怪力视效网络科技有限公司 | Expression capturing and animation automatic generation system and method |
-
2002
- 2002-10-18 CN CNB02146278XA patent/CN100483462C/en not_active Expired - Fee Related
Also Published As
Publication number | Publication date |
---|---|
CN1404016A (en) | 2003-03-19 |
Similar Documents
Publication | Publication Date | Title |
---|---|---|
CN100483462C (en) | Establishing method of human face 3D model by fusing multiple-visual angle and multiple-thread 2D information | |
CN109840940B (en) | Dynamic three-dimensional reconstruction method, device, equipment, medium and system | |
CN105631861B (en) | Restore the method for 3 D human body posture from unmarked monocular image in conjunction with height map | |
JP2023086791A (en) | Information processing unit, information processing method, and program | |
Tanskanen et al. | Live metric 3D reconstruction on mobile phones | |
EP2538389B1 (en) | Method and arrangement for 3-Dimensional image model adaptation | |
CN102801994B (en) | Physical image information fusion device and method | |
CN104599317B (en) | A kind of mobile terminal and method for realizing 3D scanning modeling functions | |
CN104809638A (en) | Virtual glasses trying method and system based on mobile terminal | |
CN114450719A (en) | Human body model reconstruction method, reconstruction system and storage medium | |
CN106485207A (en) | A kind of Fingertip Detection based on binocular vision image and system | |
CN105844584A (en) | Method for correcting image distortion of fisheye lens | |
CN106981078A (en) | Sight line correction method and device, intelligent conference terminal and storage medium | |
CN111798505B (en) | Dense point cloud reconstruction method and system for triangularized measurement depth based on monocular vision | |
CN113129451B (en) | Holographic three-dimensional image space quantitative projection method based on binocular vision positioning | |
CN103927787A (en) | Method and device for improving three-dimensional reconstruction precision based on matrix recovery | |
KR101454780B1 (en) | Apparatus and method for generating texture for three dimensional model | |
Su et al. | Cross-validated locally polynomial modeling for 2-D/3-D gaze tracking with head-worn devices | |
Wang et al. | Pose determination of human faces by using vanishing points | |
CN116702367A (en) | Customized spectacle frame automatic design method based on three-dimensional face fusion deformation model | |
CN106909904A (en) | It is a kind of based on the face front method that can learn Deformation Field | |
Remondino et al. | 3D reconstruction of human skeleton from single images or monocular video sequences | |
CN112686865A (en) | 3D view auxiliary detection method, system, device and storage medium | |
CN112329723A (en) | Binocular camera-based multi-person human body 3D skeleton key point positioning method | |
Wu et al. | Photogrammetric reconstruction of free-form objects with curvilinear structures |
Legal Events
Date | Code | Title | Description |
---|---|---|---|
C06 | Publication | ||
PB01 | Publication | ||
C10 | Entry into substantive examination | ||
SE01 | Entry into force of request for substantive examination | ||
C14 | Grant of patent or utility model | ||
GR01 | Patent grant | ||
C17 | Cessation of patent right | ||
CF01 | Termination of patent right due to non-payment of annual fee |
Granted publication date: 20090429 Termination date: 20101018 |