CN103530900B - Modeling method, face tracking method and the equipment of three-dimensional face model - Google Patents
Modeling method, face tracking method and the equipment of three-dimensional face model Download PDFInfo
- Publication number
- CN103530900B CN103530900B CN201210231897.XA CN201210231897A CN103530900B CN 103530900 B CN103530900 B CN 103530900B CN 201210231897 A CN201210231897 A CN 201210231897A CN 103530900 B CN103530900 B CN 103530900B
- Authority
- CN
- China
- Prior art keywords
- frame
- expression
- face
- model
- tracking
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Expired - Fee Related
Links
Classifications
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
- G06T13/00—Animation
- G06T13/20—3D [Three Dimensional] animation
- G06T13/40—3D [Three Dimensional] animation of characters, e.g. humans, animals or virtual beings
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
- G06T17/00—Three dimensional [3D] modelling, e.g. data description of 3D objects
- G06T17/20—Finite element generation, e.g. wire-frame surface description, tesselation
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06V—IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING
- G06V20/00—Scenes; Scene-specific elements
- G06V20/60—Type of objects
- G06V20/64—Three-dimensional objects
- G06V20/653—Three-dimensional objects by matching three-dimensional models, e.g. conformal mapping of Riemann surfaces
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06V—IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING
- G06V40/00—Recognition of biometric, human-related or animal-related patterns in image or video data
- G06V40/10—Human or animal bodies, e.g. vehicle occupants or pedestrians; Body parts, e.g. hands
- G06V40/16—Human faces, e.g. facial parts, sketches or expressions
- G06V40/161—Detection; Localisation; Normalisation
- G06V40/167—Detection; Localisation; Normalisation using comparisons between temporally consecutive images
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06V—IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING
- G06V40/00—Recognition of biometric, human-related or animal-related patterns in image or video data
- G06V40/10—Human or animal bodies, e.g. vehicle occupants or pedestrians; Body parts, e.g. hands
- G06V40/16—Human faces, e.g. facial parts, sketches or expressions
- G06V40/174—Facial expression recognition
- G06V40/176—Dynamic expression
Landscapes
- Engineering & Computer Science (AREA)
- Physics & Mathematics (AREA)
- General Physics & Mathematics (AREA)
- Theoretical Computer Science (AREA)
- Multimedia (AREA)
- Oral & Maxillofacial Surgery (AREA)
- Software Systems (AREA)
- Health & Medical Sciences (AREA)
- General Health & Medical Sciences (AREA)
- Human Computer Interaction (AREA)
- Geometry (AREA)
- Computer Graphics (AREA)
- Image Analysis (AREA)
- Processing Or Creating Images (AREA)
Abstract
Modeling method, face tracking method and the equipment of a kind of three-dimensional face model are provided.A kind of modeling method of three-dimensional face model is provided, it include: using preset standard three-dimensional faceform as working model, and specified start frame is set as first frame, face tracking is carried out since the specified start frame of the multiple video frames continuously inputted based on working model, tracking result corresponding with the video frame of predetermined number is generated from the video frame extraction human face characteristic point of tracking, expression parameter and head pose parameter, and according to predetermined condition；Tracking result based on generation updates working model, wherein, if when updating working model, the difference of appearance parameter before determining the appearance parameter of the working model updated and updating is not less than scheduled limit value, continue to continue to execute face tracking to subsequent video frame, the tracking until completing all videos frame；Output services model.
Description
Technical field
This application involves a kind of modeling method of three-dimensional face model, face tracking method and for executing the modeling
The equipment of method and face tracking method more particularly to a kind of face tracking is concurrently carried out from the face video frame continuously inputted
With the modeling of three-dimensional face model, to provide the three-dimensional face model closer to user's face, and/or high-precision is exported
Human facial expression information.
Background technique
With continuous videos be input, existing face tracking/modeling technique can export differing complexity as a result, example
Such as: the classification of expression parameter and its intensity, face two-dimensional shapes, the face 3D shape of low-res or the people of high-res
Face 3D shape.
Face tracking/modeling basic technology is broadly divided into three classes: the technology based on identification, the technology based on fitting and base
In the technology of reconstruction.
But there are the following problems for existing face tracking/modeling technique: firstly, some technologies need to rely on volume
Outer equipment, such as binocular camera, depth camera；Secondly, most of technology needs the cooperation of user, such as manual markings to close
Key point, using preceding progress user's registration, amimia or fixed expression etc. is kept in modeling；In addition, some technologies cannot be defeated
The tracking result of high-res out can not avoid the influence of appearance, posture to expression parameters precision.
Summary of the invention
The purpose of the present invention is to provide a kind of modeling methods of the three-dimensional face model of personalization, from the people continuously inputted
Face video frame is made iteratively face tracking, and updates three-dimensional face model on the basis of face tracking result, to mention
For the three-dimensional face model closer to user's face.
Another object of the present invention is to provide a kind of face tracking methods, iteratively from the face video frame continuously inputted
Face tracking is carried out, three-dimensional face model is updated based on tracking result, to export high-precision human facial expression information.
According to an aspect of the present invention, a kind of modeling method of three-dimensional face model, the three-dimensional face model packet are provided
The information of 3D shape, appearance parameter, expression parameter and head pose is included, the modeling method is the following steps are included: by pre-
If standard three-dimensional faceform be set as first frame as working model, and by specified start frame, perform the following operations: a) base
Face tracking is carried out in specified start frame of the working model since the multiple video frames continuously inputted, is mentioned from the video frame of tracking
Human face characteristic point, expression parameter and head pose parameter are taken, and the video frame with predetermined number is generated according to predetermined condition
Corresponding tracking result, the tracking result include each video frame of tracking and the face spy of each video frame extraction from tracking
Levy point, expression parameter and head pose parameter；B) tracking result based on generation updates working model, wherein if more
When new working model, the difference of the appearance parameter before determining the appearance parameter of the working model updated and updating is not less than scheduled limit
Value, and the video frame after the video frame of the predetermined number is not the last frame in the multiple video frames continuously inputted, then
Using first video frame after the video frame of the predetermined number as specified start frame, step a) is executed；And c) output services
Model.
The three-dimensional face model may be expressed as
Wherein, S is 3D shape, and a is appearance component, and e is expression component, and q is head pose, and T (S, q) expression will be three-dimensional
Shape S carries out function rotationally and/or translationally according to head pose q.
The standard three-dimensional faceform can include: average shape S0, appearance componentExpression componentAnd standard header
Portion posture q0.Wherein, i=1:N,A kind of variation of expression face appearance, j=1:M,Indicate a kind of change of human face expression
Change.
Lower online in advance the standard three-dimensional faceform can be trained using a series of three-dimensional face datas.
Step a) and step b) can be performed in parallel.
Tracking result described in step b) based on generation updates the processing of working model can include: in the tracking knot of generation
In fruit, neutral expression's frame will be selected as closest to the video frame of neutral expression；According to the human face characteristic point in neutral expression's frame
Face outline is extracted from neutral expression's frame；Face outline based on characteristic point and extraction in neutral expression's frame updates Working mould
Type.
In selecting in the processing of property expression frame, can be in the corresponding tracking result of the video frame of T from predetermined number,
It selects neutral expression's frame as follows: calculating expression parameter for the video frame of each trackingWherein, K
For the number of species of expression parameter；By the highest expression parameter value of the frequency of occurrences in every kind of expression parameterAs neutral expression
Value；The deviation of whole K expression parameters and corresponding neutral expression's value is selected to be respectively less than the video frame of scheduled each threshold value
As neutral expression's frame.
Active contour model algorithm can be used to extract face outline from neutral expression's frame.
The face outline of the characteristic point and extraction based in neutral expression's frame updates the processing of working model can include:
The head pose q of working model is updated to the head pose of neutral expression's frame；0 is set by the expression component e of working model；
By the way that working model S (a, e, q) is matched with the characteristic point position of neutral expression's frame, and by working model S (a, e, q)
The face outline of calculating matches the appearance component a for carrying out correction work model with the face outline extracted from neutral expression's frame.
Multiple modeling units can be used to be alternately carried out the processing of the step b) of different iteration, and by the place of different iteration
Reason is merged.
In step a), also the tracking result generated can be exported by input/output interface.
In step b), when completing the update of working model, also the Working mould updated can be exported by input/output interface
Type.
In step a), can be wanted according to the input rate of the multiple video frames continuously inputted, noise quality or the precision of tracking
It asks and determines the predetermined number, and the predetermined number can be constant or variable.
In step a), one of following methods can be used to obtain human face characteristic point, expression parameter and head based on working model
Attitude parameter: active appearance models method (AAM), Active Shape Model Method (ASM) and compound constant active appearance models
(Composite Constraint AAM)。
If in step a), the face tracking processing returns to failure as a result, if can will reset the outer of working model
Looks component a is 0, and returns to the beginning of step a).
According to another aspect of the present invention, a kind of face tracking method based on three-dimensional face model, the three-dimensional are provided
Faceform includes the information of 3D shape, appearance parameter, expression parameter and head pose, and the face tracking method includes
Following steps: using preset standard three-dimensional faceform as working model, 1 is set by modeling instruction, and will specify
Beginning frame is set as first frame, performs the following operations: a) based on working model from the specified start frame of the multiple video frames continuously inputted
Start to carry out face tracking, from the video frame extraction human face characteristic point of tracking, expression parameter and head pose parameter, according to pre-
Fixed condition generates corresponding with the video frame of predetermined number tracking result, the tracking result include each video frame tracked and
From the human face characteristic point of each video frame extraction of tracking, expression parameter and head pose parameter, and export the tracking knot
Fruit；If b) modeling is designated as 1, the tracking result based on generation updates working model, wherein when updating working model,
If it is determined that the difference of the appearance parameter before new appearance parameter and update is less than scheduled limit value, then 0 is set by modeling instruction；
It, will if c) video frame after the video frame of the predetermined number is not the last frame in the multiple video frames continuously inputted
First video frame after the video frame of the predetermined number executes step a) as specified start frame.
The three-dimensional face model may be expressed asWherein, S is
3D shape, a are appearance component, and e is expression component, and q is head pose, and T (S, q) is indicated 3D shape S according to head appearance
State q carries out function rotationally and/or translationally.
The standard three-dimensional faceform can include: average shape S0, appearance componentExpression componentAnd standard header
Portion posture q0, wherein i=1:N,A kind of variation of expression face appearance, j=1:M,Indicate a kind of change of human face expression
Change.
Lower online in advance the standard three-dimensional faceform can be trained using a series of three-dimensional face datas.
Step a) and step b) can be performed in parallel.
Tracking result described in step b) based on generation updates the processing of working model can include: in the tracking knot of generation
In fruit, neutral expression's frame will be selected as closest to the video frame of neutral expression；According to the human face characteristic point in neutral expression's frame
Face outline is extracted from neutral expression's frame；Face outline based on characteristic point and extraction in neutral expression's frame updates Working mould
Type.
In selecting in the processing of property expression frame, can be in the corresponding tracking result of the video frame of T from predetermined number,
It selects neutral expression's frame as follows: calculating expression parameter for the video frame of each trackingWherein, K
For the number of species of expression parameter；By the highest expression parameter value of the frequency of occurrences in every kind of expression parameterAs neutral expression
Value；The deviation of whole K expression parameters and corresponding neutral expression's value is selected to be respectively less than the video frame of scheduled each threshold value
As neutral expression's frame.
Active contour model algorithm can be used to extract face outline from neutral expression's frame.
The face outline of the characteristic point and extraction based in neutral expression's frame updates the processing of working model can include:
The head pose q of working model is updated to the head pose of neutral expression's frame；0 is set by the expression component e of working model；
By the way that working model S (a, e, q) is matched with the characteristic point position of neutral expression's frame, and by working model S (a, e, q)
The face outline of calculating matches the appearance component a for carrying out correction work model with the face outline extracted from neutral expression's frame.
Multiple modeling units can be used to be alternately carried out the processing of the step b) of different iteration, and by the place of different iteration
Reason is merged.
After the execution for completing step c), output services model can be gone back.
In step a), can be wanted according to the input rate of the multiple video frames continuously inputted, noise quality or the precision of tracking
It asks and determines the predetermined number, and the predetermined number is constant or variable.
In step a), one of following methods can be used to obtain human face characteristic point, expression parameter and head based on working model
Attitude parameter: active appearance models method (AAM), Active Shape Model Method (ASM) and compound constant active appearance models
(Composite Constraint AAM)。
If in step a), the face tracking processing returns to failure as a result, if can will reset the outer of working model
Looks component a is 0, and returns to the beginning of step a).
According to another aspect of the present invention, a kind of modelling apparatus of three-dimensional face model, the three-dimensional face model are provided
Information including 3D shape, appearance parameter, expression parameter and head pose, the modelling apparatus include: the first module, are used
It is set as first frame in using preset standard three-dimensional faceform as working model, and by specified start frame, controls the second mould
Block handles the multiple video frames continuously inputted；Second module, for based on working model from the multiple views continuously inputted
The specified start frame of frequency frame starts to carry out face tracking, from the video frame extraction human face characteristic point of tracking, expression parameter and head
Portion's attitude parameter, and tracking result corresponding with the video frame of predetermined number, the tracking result are generated according to predetermined condition
Human face characteristic point, expression parameter and the head pose ginseng of each video frame including tracking and each video frame extraction from tracking
Number；Third module updates working model for the tracking result based on generation, wherein if determined when updating working model
The difference of appearance parameter before new appearance parameter and update is not less than scheduled limit value, and after the video frame of the predetermined number
Video frame be not last frame in the multiple video frames continuously inputted, then by first after the video frame of the predetermined number
A video frame continues to handle the multiple video frames continuously inputted as specified start frame, the second module of control；4th mould
Block is used for output services model.
The three-dimensional face model may be expressed asWherein, S is
3D shape, a are appearance component, and e is expression component, and q is head pose, and T (S, q) is indicated 3D shape S according to head appearance
State q carries out function rotationally and/or translationally.
The standard three-dimensional faceform can include: average shape S0, appearance componentExpression componentAnd standard header
Portion posture q0, wherein i=1:N,A kind of variation of expression face appearance, j=1:M,Indicate a kind of change of human face expression
Change.
The modelling apparatus may also include that training module, be instructed for lower online in advance using a series of three-dimensional face datas
Practise the standard three-dimensional faceform.
Second module and third module can be performed in parallel operation.
Third module can include: the 5th module will be closest to the view of neutral expression in the tracking result of generation
Frequency frame is selected as neutral expression's frame；6th module, for being mentioned according to the human face characteristic point in neutral expression's frame from neutral expression's frame
Take face outline；7th module updates working model for the face outline based on characteristic point and extraction in neutral expression's frame.
5th module can select as follows neutral expression's frame: calculate expression parameter for the video frame of each trackingWherein, K is the number of species of expression parameter；By frequency of occurrences highest in every kind of expression parameter
Expression parameter valueAs neutral expression's value；Select the deviation of whole K expression parameters and corresponding neutral expression's value small
In scheduled each threshold value video frame as neutral expression's frame.
Active contour model algorithm can be used to extract face outline from neutral expression's frame for 6th module.
7th module can based on the processing that the face outline of characteristic point and extraction in neutral expression's frame updates working model
It include: the head pose that the head pose q of working model is updated to neutral expression's frame；The expression component e of working model is set
It is set to 0；By the way that working model S (a, e, q) is matched with the characteristic point position of neutral expression's frame, and by working model S
The face outline that (a, e, q) is calculated matches the appearance component for carrying out correction work model with the face outline extracted from neutral expression's frame
a。
Third module may include multiple modeling units, and the multiple modeling unit is alternately carried out at the modeling of different iteration
Reason, and third module merges the modeling processing of different iteration.
The tracking result of the also exportable generation of second module.
When third module completes the update of working model, third module can also be exported by input/output interface to be updated
Working model.
Second module can be according to the input rate of the multiple video frames continuously inputted, noise quality or the required precision of tracking
Determine the predetermined number, and the predetermined number can be constant or variable.
One of following methods can be used to obtain human face characteristic point, expression parameter and head appearance based on working model for second module
State parameter: active appearance models method (AAM), Active Shape Model Method (ASM) and compound constant active appearance models
(Composite Constraint AAM)。
If the second module carry out the face tracking processing returns to failure as a result, if can will reset working model
Appearance component a is 0, and restarts to carry out face tracking from the specified start frame of the multiple video frames continuously inputted.
According to another aspect of the present invention, a kind of face tracking equipment based on three-dimensional face model, the three-dimensional are provided
Faceform includes the information of 3D shape, appearance parameter, expression parameter and head pose, and the face tracking equipment includes
Following steps: the first module, for setting 1 for modeling instruction using preset standard three-dimensional faceform as working model,
And specified start frame is set as first frame, is performed the following operations: second module, for based on working model from continuously inputting
The specified start frame of multiple video frames starts to carry out face tracking, video frame extraction human face characteristic point, expression parameter from tracking
And head pose parameter, tracking result corresponding with the video frame of predetermined number, the tracking knot are generated according to predetermined condition
Fruit includes each video frame of tracking and human face characteristic point, expression parameter and the head pose of each video frame extraction from tracking
Parameter, and export the tracking result；Third module, if being configured as modeling is designated as 1, the tracking knot based on generation
Fruit updates working model, wherein when updating working model, if it is determined that new appearance parameter and the appearance parameter before update it
Difference is less than scheduled limit value, then sets 0 for modeling instruction；4th module, if being configured as the video of the predetermined number
Video frame after frame is not the last frame in the multiple video frames continuously inputted, then will be after the video frame of the predetermined number
First video frame then controls the second module and continues to handle the multiple video frames continuously inputted as specified start frame.
The three-dimensional face model may be expressed asWherein, S is
3D shape, a are appearance component, and e is expression component, and q is head pose, and T (S, q) is indicated 3D shape S according to head appearance
State q carries out function rotationally and/or translationally.
The standard three-dimensional faceform can include: average shape S0, appearance componentExpression componentAnd standard header
Portion posture q0, wherein i=1:N,A kind of variation of expression face appearance, j=1:M,Indicate a kind of change of human face expression
Change.
The face tracking equipment can further include: training module uses a series of three-dimensional face numbers for lower online in advance
According to training the standard three-dimensional faceform.
Second module and third module can be performed in parallel operation.
Third module can include: the 5th module will be closest to the view of neutral expression in the tracking result of generation
Frequency frame is selected as neutral expression's frame；6th module, for being mentioned according to the human face characteristic point in neutral expression's frame from neutral expression's frame
Take face outline；7th module updates working model for the face outline based on characteristic point and extraction in neutral expression's frame.
5th module can select as follows neutral expression's frame: calculate expression parameter for the video frame of each trackingWherein, K is the number of species of expression parameter；By frequency of occurrences highest in every kind of expression parameter
Expression parameter valueAs neutral expression's value；Select the deviation of whole K expression parameters and corresponding neutral expression's value small
In scheduled each threshold value video frame as neutral expression's frame.
Active contour model algorithm can be used to extract face outline from neutral expression's frame for 6th module.
7th module can based on the processing that the face outline of characteristic point and extraction in neutral expression's frame updates working model
It include: the head pose that the head pose q of working model is updated to neutral expression's frame；The expression component e of working model is set
It is set to 0；By the way that working model S (a, e, q) is matched with the characteristic point position of neutral expression's frame, and by working model S
The face outline that (a, e, q) is calculated matches the appearance component for carrying out correction work model with the face outline extracted from neutral expression's frame
a。
Third module may include multiple modeling units, and the multiple modeling unit is alternately carried out at the modeling of different iteration
Reason, and third module merges the modeling processing of different iteration.
When third module completes the update of working model, third module, which can be exported also by input/output interface, to be updated
Working model.
Second module can be according to the input rate of the multiple video frames continuously inputted, noise quality or the required precision of tracking
Determine the predetermined number, and the predetermined number is constant or variable.
One of following methods can be used to obtain human face characteristic point, expression parameter and head appearance based on working model for second module
State parameter: active appearance models method (AAM), Active Shape Model Method (ASM) and compound constant active appearance models
(Composite Constraint AAM)。
If the second module carry out the face tracking processing returns to failure as a result, if face tracking equipment can will weight
The appearance component a for setting working model is 0, and can restart to carry out from the specified start frame of the multiple video frames continuously inputted
Face tracking.
Detailed description of the invention
By the description carried out with reference to the accompanying drawing, above and other purpose of the invention and feature will become more clear
Chu, in which:
Figure 1A is the flow chart for showing the modeling method of three-dimensional face model of an exemplary embodiment of the present invention；
Figure 1B is the processing for showing the update working model in the modeling method of an exemplary embodiment of the present invention
Flow chart；
Fig. 1 C is the flow chart shown in the face tracking method of an exemplary embodiment of the present invention；
Fig. 2 schematically shows the three-dimensional face model based on the building of general face；
Fig. 3 is the schematic diagram being exemplarily illustrated using characteristic point from video frame extraction face outline；
Fig. 4 is to be exemplarily illustrated progress Feature Points Matching and outline to match between three-dimensional face model and face outline
Schematic diagram；
Fig. 5 A and Fig. 5 B are the modelling apparatus/face tracking equipment signals for showing an exemplary embodiment of the present invention
Figure.
Specific embodiment
Detailed description of the present invention exemplary embodiment that hereinafter reference will be made to the drawings.
The modeling method and face tracking method of three-dimensional face model of the invention can be in general purpose computers or dedicated
It is realized in processor, the method is to include the video frame (a period of time, within a few minutes) of face continuously inputted as defeated
Enter, using preset high-precision standard three-dimensional faceform as working model, based on the working model to the video of input
Frame carries out face tracking, and the tracking result tracked to the face of predetermined quantity is hereafter used to carry out more the working model
Newly/amendment.After update/amendment working model, it is further continued for carrying out face tracking to video frame later, until obtaining
Reach the three-dimensional face model of preset limit value or the face tracking and working model are completed to all videos frame of input
Until update/amendment.According to demand, in the process, output has the face tracking knot of fine expression and head pose information
Fruit, and/or after this process, the personalized three-dimensional face model exported.
The video frame continuously inputted can be to be mentioned in the digital video frequency flow by single general digital camera shooting
The video frame of many integral photographs for taking and handling, is also possible to the video for many photos being continuously shot using digital camera
Frame.The video frame continuously inputted can input to three-dimensional face model for carrying out the present invention by input/output interface
Modeling method and face tracking method general purpose computer or application specific processor.
Fig. 4, which is shown, schematically shows the three-dimensional face model based on the building of any face.The three-dimensional face model
Include, but are not limited to the information of the 3D shape of face, appearance parameter, expression parameter and head pose.In the present invention,
The three-dimensional face model is expressed as Wherein, S is 3D shape, and a is
Appearance component, e are expression component, and q is head pose, and T (S, q) expression rotates 3D shape S according to head pose q
And/or the function of translation.
An exemplary embodiment of the present invention, the lower high density face surface different using expression, posture online in advance
Data train the three-dimensional face model of standard.According to other embodiments of the invention, it is possible to use existing other methods obtain
To the three-dimensional face model of the standard, or as needed, the three-dimensional face model with standard faces feature is determined as
The three-dimensional face model of the standard.
The standard three-dimensional faceform includes:
Average shape S0: the average value of all training samples.
Appearance componentA kind of variation of each component statement face in terms of appearance.
Expression componentA kind of variation of each component statement face in terms of expression.
Head pose q0: standard three-dimensional position and the rotation angle of face are described.
Figure 1A is the flow chart for showing the modeling method of three-dimensional face model of an exemplary embodiment of the present invention.
It referring to Fig.1,, will in behaviour step S110 when starting the operation of modeling method of three-dimensional face model of the invention
Preset standard three-dimensional faceform is as working model, and specified start frame is set as in the video frame continuously inputted
One frame.The standard three-dimensional faceform is a series of three-dimensional trained in advance using various expressions, the human face data of posture
Faceform.
In step S120, people is carried out since the specified start frame in the multiple video frames continuously inputted based on working model
Face tracking, from the video frame extraction human face characteristic point of tracking, expression parameter and head pose parameter, and according to predetermined condition
Generate corresponding with the video frame of predetermined number tracking result, the tracking result includes each video frame tracked and from tracking
Each video frame extraction human face characteristic point, expression parameter and head pose parameter.An exemplary embodiment of the present invention, root
The predetermined number is determined according to the input rate of the multiple video frames continuously inputted, noise quality or the required precision of tracking.This
Outside, the predetermined number be can be with constant or variable.
An exemplary embodiment of the present invention also passes through the tracking of input/output interface output generation in step S120
As a result.
An exemplary embodiment of the present invention is obtained using one of following methods based on working model in step S120
Human face characteristic point, expression parameter and head pose parameter: active appearance models method (AAM), Active Shape Model Method (ASM)
With compound constant active appearance models (Composite Constraint AAM).
Hereafter, in step S130, working model is updated based on the tracking result generated in step S120.Later with reference to figure
1B detailed description updates the specific processing of working model.
An exemplary embodiment of the present invention, in step S130, when completing the update of working model, also by defeated
Enter/output interface output update working model.
In addition, in step S140, if when updating working model, determine the appearance parameter of the working model updated with more
The difference of the appearance parameter of working model before new is not less than scheduled limit value, and the video after the video frame of the predetermined number
Frame is not the last frame in the multiple video frames continuously inputted, then in step S150, after the video frame of the predetermined number
First video frame as specified start frame, return step S120, the working model based on update, from the specified start frame
Continue face tracking.
If the appearance parameter for determining the working model updated when updating working model and the working model before update
The difference of appearance parameter be less than the video frame after the video frame of scheduled limit value or the predetermined number be continuously input it is multiple
Last frame in video frame, thens follow the steps S150.That is, when determination has constructed the best three-dimensional to conform to a predetermined condition
When faceform, or when the processing to all videos frame is completed, the update to working model can be stopped.
In step S160, exported the working model of update as personalized three-dimensional face model.
Figure 1B is exemplarily illustrated the processing carried out in the step S130 of Figure 1A.
B referring to Fig.1, in step S132, by the view in the tracking result that step S120 is generated closest to neutral expression
Frequency frame is selected as neutral expression's frame.Preferred embodiment in accordance with the present invention, in the processing of step S132, it is assumed that tracking result with
The video frame that reservation number is T is corresponding, calculates expression parameter for each video frame of tracking
Wherein, K is the number of species of expression parameter；By the highest expression parameter value of the frequency of occurrences in every kind of expression parameterAs neutrality
Expression value；Then, the deviation of whole K expression parameters and corresponding neutral expression's value is selected to be respectively less than scheduled each threshold value
Video frame as neutral expression's frame.
In step S135, face outline is extracted from neutral expression's frame according to the human face characteristic point in neutral expression's frame.From step
Rapid S120 includes the letter such as human face characteristic point, expression parameter and head pose parameter for each video frame extraction of tracking
Breath.An exemplary embodiment of the present invention, the neutral expression's frame selected using active contour model algorithm from step S132
Extract face outline.
Fig. 3 A~Fig. 3 C is exemplarily illustrated the processing using human face characteristic point from video frame extraction face outline.According to this
The exemplary embodiment of invention is referring to (figure with the characteristic point of the video frame when from the video frame extraction outline of Fig. 3 A
3B), face outline as shown in Figure 3 C is extracted from video frame extraction face outline using active contour model algorithm.In this hair
In bright its face outline similarly can be extracted from neutral expression's frame.
Hereafter, in step S138, the face outline based on characteristic point and extraction in neutral expression's frame updates working model.
Specifically, the head pose q of working model to be updated to the head pose of neutral expression's frame；By the expression component e of working model
It is set as 0；By the way that working model S (a, e, q) is matched with the characteristic point position of neutral expression's frame, and by working model
The face outline that S (a, e, q) is calculated matches the appearance point for carrying out correction work model with the face outline extracted from neutral expression's frame
Measure a.Fig. 4 B shows the work for making its characteristic point be overlapped locking with the characteristic point of neutral expression's frame in Fig. 4 A adjustment working model
Make model；Fig. 4 D, which is shown, is overlapped working model as far as possible to adjust working model, correction with the face outline (Fig. 4 D) of extraction
The processing of its appearance parameter.
During appearance parameter of the correction with appearance representation in components, the value and correction of the appearance parameter before record correction
The value of appearance parameter afterwards, thus the basis compared as step S140.
Step S120~S150 in Figure 1A iteratively carries out face tracking and model modification to the video frame continuously inputted.
An exemplary embodiment of the present invention is performed in parallel the processing of step S120 and S130.By using current Working mould
Type carries out face tracking, extracts human face characteristic point, expression parameter and head pose parameter, and by using including extraction
Human face characteristic point and head pose parameter and corresponding video frame further update working model, can get with user more
For close personalized human face model.At the same time, the face tracking of also exportable each input video frame is as a result, include expression
Parameter, appearance parameter and head pose etc..
Fig. 1 C shows the processing in the face tracking method of an exemplary embodiment of the present invention.
Since the face tracking method is to export face tracking result as main purpose, in fig. 1 c, once
The best model to conform to a predetermined condition is obtained, then no longer carries out the update of working model, but continues subsequent video frame
Face tracking processing.
C referring to Fig.1 is grasping step S110 ' when starting the operation of modeling method of three-dimensional face model of the invention,
Using preset standard three-dimensional faceform as working model, specified start frame is set as first in the video frame continuously inputted
Frame, and it is 1 (or "Yes") that the variable (for example, modeling instruction) for indicating whether to continue working model and updating, which is arranged,.It is described
Standard three-dimensional faceform is a series of three-dimensional face model trained in advance using various expressions, the human face data of posture.
Processing in the processing and Figure 1A of step S120 in Fig. 1 C is essentially identical, but in fig. 1 c, it completes to make a reservation for
After the face tracking of the video frame of number, step S125 and S128 are executed.In step S125, the video frame of each tracking is exported
Tracking result, including expression parameter, appearance parameter and head pose etc..
In step S128, it is determined whether continue the update of working model (modeling indicates whether to be 1).If modeling refers to
It is shown as 1, thens follow the steps S130.The processing of S130 described in the processing of the step S130 and Figure 1A is identical.
In step S140 ', if when updating working model, before determining the appearance parameter of the working model updated and updating
Working model appearance parameter difference be not less than scheduled limit value, then in step S150, by the video frame of the predetermined number
First video frame afterwards is as specified start frame, and then return step S120, the working model based on update are specified from described
Start frame continues face tracking.
On the other hand, if it is determined that the appearance parameter of the working model of update and the appearance parameter of the working model before update
Difference be less than or equal to scheduled limit value, then in step S145, will indicate whether that the modeling for continuing working model update refers to
Show and is set as 0 (or "No").That is, at this time according to the present invention, determining the three-dimensional face mould for having constructed close enough user
Type.In order to reduce operand, indicate that equipment of the invention no longer updates the working model.
Hereafter, in step S148, it is more whether the video frame after determining the video frame of the predetermined number continuously inputs
Last frame in a video frame.If it is determined that the video frame after the video frame of the predetermined number be not continuously input it is multiple
Last frame in video frame, thens follow the steps S150, using first video frame after the video frame of the predetermined number as
Start frame is specified, then return step S120.
If it is determined that the video frame after the video frame of the predetermined number is last in the multiple video frames continuously inputted
One frame then terminates the processing of face tracking method of the invention.
An exemplary embodiment of the present invention, before the processing for terminating face tracking method of the invention, output is last
The working model of update.
It can be seen that carry out face tracking by using current working model, extract human face characteristic point, expression parameter with
And head pose parameter, and by using the human face characteristic point and head pose parameter that include extraction and corresponding video frame
Further update working model, can be made iteratively based on the personalized human face model more close with user face with
Track, and exportable each more accurate face tracking is as a result, include expression parameter, appearance parameter and head pose etc..
Fig. 5 A show the realization three-dimensional face model of an exemplary embodiment of the present invention modeling method and/or
The schematic structure of the equipment of face tracking method.
The equipment includes tracking cell and modeling unit, the tracking cell execute the step S110 in Figure 1A~
S110 '~S125 in S120 or Fig. 1 C, the modeling unit execute step S130~S150 in Figure 1A or Fig. 1 C.
Referring to Fig. 5 A, tracking cell (is initially the standard three-dimensional faceform M using working model0) to video frame 0
To video frame t2- 1 carries out face tracking, and will be including video frame 0 to video frame t2- 1 and from the people of each video frame extraction
Tracking result (i.e. 0~result of result t in Fig. 5 A of face characteristic point, expression parameter and head pose parameter2- 1) it exports.Institute
It states tracking result and is provided to modeling unit, and as needed, can be exported by input/output interface to user.
Tracking result (0~result of result t that the modeling unit is exported based on tracking cell2- 1) working model is updated
(hereinbefore the update of working model is described in detail with reference to Figure 1A and Figure 1B).It for convenience of explanation, will herein more
New working model is known as M1。
Hereafter, tracking cell is according to scheduled rule, the working model M based on update1To video frame t2- 1 to t3 carries out people
Face tracks (Figure 1A description as described previously with reference to), and output tracking result (result t2- 1 to result t3).Modeling unit is based on result
t2- 1~result t3Update working model M1.It is so made iteratively face tracking and updates the processing of working model, until really
Surely it obtains qualified best model or completes to fully enter the processing of video frame.The tracking cell and modeling unit can
Concurrently operated.
Preferred embodiment in accordance with the present invention, the equipment may include training unit, use a system for lower online in advance
Column three-dimensional face data trains standard three-dimensional faceform as working model M0。
Fig. 5 B shows the modeling method of the realization three-dimensional face model in accordance with an alternative illustrative embodiment of the present invention
And/or the schematic structure of the equipment of face tracking method.
Unlike the equipment in Fig. 5 A, the equipment in Fig. 5 B includes multiple modeling units (such as modeling unit A and modeling
Unit B), multiple modeling units are alternately carried out the processing of the step S130 of different iteration, and the processing of different iteration is melted
It closes.
Although show and describing the present invention with reference to preferred embodiment, it will be understood by those skilled in the art that not
In the case where being detached from the spirit and scope of the present invention that are defined by the claims, these embodiments can be carry out various modifications and
Transformation.
Claims (15)
1. a kind of modeling method of three-dimensional face model, the three-dimensional face model includes 3D shape, appearance parameter, expression ginseng
Several and head pose information, the modeling method the following steps are included:
Using a preset standard three-dimensional faceform as working model, and specified start frame is set as first frame, executed
Following operation:
A) face tracking is carried out since the specified start frame of the multiple video frames continuously inputted based on working model, from tracking
Video frame extraction human face characteristic point, expression parameter and head pose parameter, and according to predetermined condition generation and predetermined number
The corresponding tracking result of video frame, each video frame and each video frame extraction from tracking that the tracking result includes tracking
Human face characteristic point, expression parameter and head pose parameter；
B) tracking result based on generation updates working model, wherein if determining the work updated when updating working model
The appearance parameter of model and the difference of the appearance parameter before update are not less than scheduled limit value, and the video frame of the predetermined number
Video frame afterwards is not the last frame in the multiple video frames continuously inputted, then by after the video frame of the predetermined number
One video frame executes step a) as specified start frame；
C) output services model.
2. modeling method as described in claim 1, wherein the three-dimensional face model is represented as
Wherein, S is 3D shape, and a is appearance component, and e is expression component, and q is head pose, and T (S, q) is indicated 3D shape
S carries out function rotationally and/or translationally according to head pose q.
3. modeling method as claimed in claim 2, wherein the standard three-dimensional faceform includes: average shape S0, appearance
ComponentExpression componentAnd standard head posture q0,
Wherein, i=1:N,A kind of variation of expression face appearance, j=1:M,Indicate a kind of variation of human face expression.
4. modeling method as claimed in claim 3, further includes:
The standard three-dimensional faceform is trained using a series of three-dimensional face datas under online in advance.
5. modeling method as claimed in claim 2, wherein be performed in parallel step a) and step b).
6. modeling method as claimed in claim 3, wherein based on the tracking result of generation described in step b), update work
The processing of model includes:
In the tracking result of generation, neutral expression's frame will be selected as closest to the video frame of neutral expression；
Face outline is extracted from neutral expression's frame according to the human face characteristic point in neutral expression's frame；
Face outline based on characteristic point and extraction in neutral expression's frame updates working model.
7. modeling method as claimed in claim 6, wherein in selecting in the processing of property expression frame, from predetermined number be T
The corresponding tracking result of video frame in, it is following to select neutral expression's frame:
Expression parameter is calculated for the video frame of each trackingWherein, K is the type of expression parameter
Quantity；
By the highest expression parameter value of the frequency of occurrences in every kind of expression parameterAs neutral expression's value；
The deviation of whole K expression parameters and corresponding neutral expression's value is selected to be respectively less than the video frame of scheduled each threshold value
As neutral expression's frame.
8. modeling method as claimed in claim 7, wherein extract face from neutral expression's frame using active contour model algorithm
Outline.
9. modeling method as claimed in claim 8, wherein the face of the characteristic point and extraction based in neutral expression's frame
Outline update working model processing include:
The head pose q of working model is updated to the head pose of neutral expression's frame；
0 is set by the expression component e of working model；
By the way that working model S (a, e, q) is matched with the characteristic point position of neutral expression's frame, and by working model S (a,
E, q) calculate face outline the appearance component a for carrying out correction work model is matched with the face outline extracted from neutral expression's frame.
10. modeling method as claimed in claim 2, wherein be alternately carried out the step of different iteration using multiple modeling units
Rapid processing b), and the processing of different iteration is merged.
11. modeling method as described in claim 1, wherein in step a), also exported and generated by input/output interface
Tracking result.
12. modeling method as described in claim 1, wherein also pass through in step b) when completing the update of working model
The working model that input/output interface output updates.
13. modeling method as claimed in claim 2, wherein in step a), according to the input of the multiple video frames continuously inputted
Rate, noise quality or the required precision of tracking determine the predetermined number, and the predetermined number is constant or variable.
14. modeling method as claimed in claim 2, wherein in step a), obtained using one of following methods based on working model
Take human face characteristic point, expression parameter and head pose parameter: active appearance models method (AAM), Active Shape Model Method
(ASM) and compound constant active appearance models (Composite Constraint AAM).
15. modeling method as claimed in claim 13, wherein if in step a), the face tracking processing returns to mistakes
It is losing as a result, then will reset working model appearance component a be 0, and return to step a) beginning.
Priority Applications (3)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201210231897.XA CN103530900B (en) | 2012-07-05 | 2012-07-05 | Modeling method, face tracking method and the equipment of three-dimensional face model |
| KR1020130043463A KR102024872B1 (en) | 2012-07-05 | 2013-04-19 | Method and apparatus for modeling 3d face, method and apparatus for tracking face |
| US13/936,001 US20140009465A1 (en) | 2012-07-05 | 2013-07-05 | Method and apparatus for modeling three-dimensional (3d) face, and method and apparatus for tracking face |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201210231897.XA CN103530900B (en) | 2012-07-05 | 2012-07-05 | Modeling method, face tracking method and the equipment of three-dimensional face model |
Publications (2)
| Publication Number | Publication Date |
|---|---|
| CN103530900A CN103530900A (en) | 2014-01-22 |
| CN103530900B true CN103530900B (en) | 2019-03-19 |
Family
ID=49932878
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN201210231897.XA Expired - Fee Related CN103530900B (en) | 2012-07-05 | 2012-07-05 | Modeling method, face tracking method and the equipment of three-dimensional face model |
Country Status (2)
| Country | Link |
|---|---|
| KR (1) | KR102024872B1 (en) |
| CN (1) | CN103530900B (en) |
Families Citing this family (11)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN105608710B (en) * | 2015-12-14 | 2018-10-19 | 四川长虹电器股份有限公司 | A kind of non-rigid Face datection and tracking positioning method |
| CN108229246A (en) * | 2016-12-14 | 2018-06-29 | 上海交通大学 | Real-time three-dimensional human face posture method for tracing based on vehicle computing machine platform |
| CN108345821B (en) * | 2017-01-24 | 2022-03-08 | 成都理想境界科技有限公司 | Face tracking method and device |
| CN108734735B (en) * | 2017-04-17 | 2022-05-31 | 佳能株式会社 | Object shape tracking device and method, and image processing system |
| US10586368B2 (en) * | 2017-10-26 | 2020-03-10 | Snap Inc. | Joint audio-video facial animation system |
| CN107958479A (en) * | 2017-12-26 | 2018-04-24 | 南京开为网络科技有限公司 | A kind of mobile terminal 3D faces augmented reality implementation method |
| CN108765464A (en) * | 2018-05-31 | 2018-11-06 | 山东工商学院 | Low-rank re-detection context long time-tracking method and system based on residual compensation |
| CN110796083B (en) * | 2019-10-29 | 2023-07-04 | 腾讯科技（深圳）有限公司 | Image display method, device, terminal and storage medium |
| CN112017230A (en) * | 2020-09-07 | 2020-12-01 | 浙江光珀智能科技有限公司 | Three-dimensional face model modeling method and system |
| CN113112596B (en) * | 2021-05-12 | 2023-10-24 | 北京深尚科技有限公司 | Face geometric model extraction and 3D face reconstruction method, equipment and storage medium |
| CN116152900B (en) * | 2023-04-17 | 2023-07-18 | 腾讯科技（深圳）有限公司 | Expression information acquisition method and device, computer equipment and storage medium |
Citations (3)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN101499128A (en) * | 2008-01-30 | 2009-08-05 | 中国科学院自动化研究所 | Three-dimensional human face action detecting and tracing method based on video stream |
| CN101777116A (en) * | 2009-12-23 | 2010-07-14 | 中国科学院自动化研究所 | Method for analyzing facial expressions on basis of motion tracking |
| CN102479388A (en) * | 2010-11-22 | 2012-05-30 | 北京盛开互动科技有限公司 | Expression interaction method based on face tracking and analysis |
Family Cites Families (13)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| KR100292810B1 (en) * | 1999-03-19 | 2001-06-15 | 윤덕용 | A Real time face tracking technique using face's color model and ellipsoid approximation model |
| KR20010089664A (en) * | 1999-10-21 | 2001-10-08 | 요트.게.아. 롤페즈 | System and method for three-dimensional modeling |
| US6845171B2 (en) * | 2001-11-19 | 2005-01-18 | Microsoft Corporation | Automatic sketch generation |
| US7936902B2 (en) * | 2004-11-12 | 2011-05-03 | Omron Corporation | Face feature point detection apparatus and feature point detection apparatus |
| US7755619B2 (en) * | 2005-10-13 | 2010-07-13 | Microsoft Corporation | Automatic 3D face-modeling from video |
| US7536030B2 (en) * | 2005-11-30 | 2009-05-19 | Microsoft Corporation | Real-time Bayesian 3D pose tracking |
| JP4999570B2 (en) * | 2007-06-18 | 2012-08-15 | キヤノン株式会社 | Facial expression recognition apparatus and method, and imaging apparatus |
| KR100896643B1 (en) * | 2007-06-18 | 2009-05-08 | 에스케이 텔레콤주식회사 | Method and system for modeling face in three dimension by means of aam, and apparatus applied to the same |
| CN101159064B (en) * | 2007-11-29 | 2010-09-01 | 腾讯科技（深圳）有限公司 | Image generation system and method for generating image |
| KR100940862B1 (en) * | 2007-12-17 | 2010-02-09 | 한국전자통신연구원 | Head motion tracking method for 3d facial model animation from a video stream |
| KR20110021330A (en) * | 2009-08-26 | 2011-03-04 | 삼성전자주식회사 | Apparatus and method for creating third dimension abatar |
| US8339459B2 (en) * | 2009-09-16 | 2012-12-25 | Microsoft Corporation | Multi-camera head pose tracking |
| KR101615719B1 (en) * | 2009-09-18 | 2016-04-27 | 삼성전자주식회사 | Apparatus and method for extracting user's third dimension facial expression |
-
2012
- 2012-07-05 CN CN201210231897.XA patent/CN103530900B/en not_active Expired - Fee Related
-
2013
- 2013-04-19 KR KR1020130043463A patent/KR102024872B1/en active IP Right Grant
Patent Citations (3)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN101499128A (en) * | 2008-01-30 | 2009-08-05 | 中国科学院自动化研究所 | Three-dimensional human face action detecting and tracing method based on video stream |
| CN101777116A (en) * | 2009-12-23 | 2010-07-14 | 中国科学院自动化研究所 | Method for analyzing facial expressions on basis of motion tracking |
| CN102479388A (en) * | 2010-11-22 | 2012-05-30 | 北京盛开互动科技有限公司 | Expression interaction method based on face tracking and analysis |
Also Published As
| Publication number | Publication date |
|---|---|
| KR102024872B1 (en) | 2019-09-24 |
| CN103530900A (en) | 2014-01-22 |
| KR20140009013A (en) | 2014-01-22 |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| CN103530900B (en) | Modeling method, face tracking method and the equipment of three-dimensional face model | |
| US9939888B2 (en) | Correlating movement information received from different sources | |
| CN108161882B (en) | Robot teaching reproduction method and device based on augmented reality | |
| EP2369553B1 (en) | Three-dimensional template transformation method and apparatus | |
| CN112541963A (en) | Three-dimensional virtual image generation method and device, electronic equipment and storage medium | |
| CN108305283A (en) | Human bodys' response method and device based on depth camera and basic form | |
| CN108664122A (en) | A kind of attitude prediction method and apparatus | |
| US11557391B2 (en) | Systems and methods for human pose and shape recovery | |
| EP3381017A1 (en) | Face reconstruction from a learned embedding | |
| US20090153569A1 (en) | Method for tracking head motion for 3D facial model animation from video stream | |
| CN104978764A (en) | Three-dimensional face mesh model processing method and three-dimensional face mesh model processing equipment | |
| JP7378642B2 (en) | Generative nonlinear human shape model | |
| CN110414546A (en) | Use intermediate loss function training image signal processor | |
| JP2010512581A5 (en) | ||
| US11963741B2 (en) | Systems and methods for human pose and shape recovery | |
| CN111127668B (en) | Character model generation method and device, electronic equipment and storage medium | |
| CN110533639A (en) | A kind of key independent positioning method and device | |
| US20150187114A1 (en) | Method and apparatus for editing 3d character motion | |
| CN110473291A (en) | Container modeling method and device for container acquisition | |
| CN114119912A (en) | Rapid fitting method and device for human body model and storage medium | |
| CN113408443B (en) | Gesture posture prediction method and system based on multi-view images | |
| CN109325994A (en) | A method of enhanced based on three-dimensional face data | |
| CN113821108B (en) | Robot remote control system and control method based on multi-mode interaction technology | |
| CN103839280B (en) | A kind of human body attitude tracking of view-based access control model information | |
| KR102147930B1 (en) | Method and apparatus for recognizing pose |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| C06 | Publication | ||
| PB01 | Publication | ||
| EXSB | Decision made by sipo to initiate substantive examination | ||
| SE01 | Entry into force of request for substantive examination | ||
| GR01 | Patent grant | ||
| GR01 | Patent grant | ||
| CF01 | Termination of patent right due to non-payment of annual fee | ||
| CF01 | Termination of patent right due to non-payment of annual fee |
Granted publication date: 20190319 |