CN102254336B - Method and device for synthesizing face video - Google Patents
Method and device for synthesizing face video Download PDFInfo
- Publication number
- CN102254336B CN102254336B CN 201110197873 CN201110197873A CN102254336B CN 102254336 B CN102254336 B CN 102254336B CN 201110197873 CN201110197873 CN 201110197873 CN 201110197873 A CN201110197873 A CN 201110197873A CN 102254336 B CN102254336 B CN 102254336B
- Authority
- CN
- China
- Prior art keywords
- face
- people
- expression
- image
- facial image
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Active
Links
Images
Abstract
The invention discloses a method and a device for synthesizing a face video. The method for synthesizing the face video comprises the following steps of: establishing a target character expression database comprising a plurality of frames of face images; pre-treating the plurality of frames of face images so as to extract a face posture position and a face expression on each frame of face image; retrieving the joint similarity between the face posture position and the face expression in a user-defined face image sequence and the face posture position and the face expression in the plurality of frames of face images so as to obtain a retrieval image sequence matched with the user-defined face image sequence; and performing winding transform and smoothing treatment on the retrieval image sequence. According to the method and the device for synthesizing the face video disclosed by the embodiment of the invention, for any user-defined expression, a lifelike image sequence relative to a target character can be synthesized conveniently; and the automation degree is high.
Description
Technical field
The present invention relates to field of Computer Graphics, particularly a kind of people's face image synthesizing method and device.
Background technology
In game making, film making and virtual reality, the analogue technique of human face expression has obtained fast development, has proposed the synthetic method of many expressions, yet synthesizing of true personage's expression still can not be satisfied actual demand.On the one hand, some lack the sense of reality based on the expression picture that the picture deformation technology of sample produces, and can not satisfy the requirement of a large amount of expression sequence senses of reality; On the other hand, the facial expression capturing technology can be transferred to another personage with performer's expression, this technology is comparatively ripe, in many film special efficacys (such as 3D film " A Fanda "), application is arranged, but implement comparatively complicated, and the performer need to wear a helmet, and is very inconvenient; Simultaneously, the performer also often need to repeat to perform certain expression until satisfy quality requirements, and this bothers for the performer very much.
Want personage's expression synthetic true to nature, main difficult point is the geometry of people's face and synthesizing of textural characteristics, and when the people did certain expression, his profile changed according to muscle, and the while is affected by ambient lighting also and produces bright alternately dark.Wherein, encoding facial movement system (FACS) is used widely in expression is analyzed and be synthetic, it is divided into a series of moving cells (AU) with people's face, the respective muscle motion of some basic facial expressions is provided, by changing the correlation combiner of these moving cells, just can produce different human face expressions.Although this method can access comparatively real human face expression, can not realize the robotization processing, namely a given personage's expression can not arrive another personage with Expression Mapping under the prerequisite that reduces as far as possible people's intervention.
Summary of the invention
Purpose of the present invention is intended to solve at least one of above-mentioned technological deficiency.For this reason, the present invention need to provide a kind of people's face image synthesizing method and device, and the advantage of this people's face image synthesizing method and device is: for user-defined any expression, can both synthesize easily the corresponding photorealism sequence of target person; Reduce the actor number of times; And automaticity is high.
According to an aspect of the present invention, provide a kind of people's face image synthesizing method, it is characterized in that, comprised step: set up target person expression database, comprise the multiframe facial image in the described target person expression database; Described multiframe facial image is carried out pre-service, with the posture position that extracts people's face on each frame facial image and the expression of people's face; Carry out the associating similarity retrieval of the expression of the posture position of people's face on the expression of the posture position of people's face in the user-defined human face image sequence and people's face and the described multiframe facial image and people's face, to obtain the retrieving images sequence with described user-defined human face image sequence coupling; And to described retrieving images sequence reel conversion and smoothing processing.
According to people's face image synthesizing method of the embodiment of the invention, for user-defined any expression, can both synthesize easily the corresponding photorealism sequence of target person.
According to one embodiment of present invention, the described step of setting up target person expression database comprises: gather a plurality of basic facial expression image sequences of described target person from a plurality of visual angles and a plurality of posture position, each the basic facial expression image sequence in wherein said a plurality of basic facial expression image sequences is expressed one's feelings by the change procedure of neutrality expression and described neutral expression and is consisted of.
According to people's face image synthesizing method of the embodiment of the invention, can comprise great amount of images in the target person expression database.
According to one embodiment of present invention, describedly described multiframe facial image is carried out pretreated step comprise: extract in the described multiframe facial image posture position of people's face in each frame facial image and describe the position of people's face with crab angle, helix angle and side rake angle; People's face in each frame facial image in the described multiframe facial image aimed at positive standard attitude and carry out feature and describe point and detect, describe the expression of people's face with described feature description.
According to people's face image synthesizing method of the embodiment of the invention, posture position and expression that can accurate description people face.
According to one embodiment of present invention, carry out described feature based on active shape model and describe the some detection.
The step of associating similarity retrieval of according to one embodiment of present invention, carrying out the expression of the posture position of people's face on the expression of the posture position of people's face in the described user-defined human face image sequence and people's face and the described multiframe facial image and people's face comprises: according to D
Pose=L (| Y
i-Y
j|)+L (| R
i-R
j|)+L (| P
i-P
j|) similarity of carrying out the posture position of people's face on the posture position of people's face in the described user-defined human face image sequence and the described multiframe facial image calculates, wherein Y represents crab angle, and R represents helix angle, and P represents side rake angle, I
iRepresent the arbitrary frame facial image in the described user-defined human face image sequence, I
jRepresent the arbitrary frame facial image in the described multiframe facial image, L (d) is the sigmoid function, is defined as
γ=ln99 wherein, T and σ are respectively average statistical and the standard deviations of variable d; According to described crab angle, described helix angle and described side rake angle, the posture position of people's face in the described user-defined human face image sequence is registered to positive attitude, then basis
Carry out the similarity of the expression of people's face on the expression of people's face in the described user-defined human face image sequence and the described multiframe facial image and calculate, wherein A
I, kIt is image I
iK feature behind the aligning described some A
J, kIt is image I
jK feature behind the aligning described a little, and J is the number of unique point, w
kBe the weight of k unique point, L (d) is the sigmoid function, is defined as
The unique point range normalization that calculates is arrived [0,1]; According to D (i, j)=D
Pose(i, j)+λ D
Expression(i, j) obtains the associating similarity, and wherein parameter lambda is to regulate the weight proportion of expression similarity and posture position similarity; And obtain image I in the user-defined human face image sequence according to described associating similarity
iA plurality of candidate images in described multiframe facial image.
According to people's face image synthesizing method of the embodiment of the invention, can obtain posture position and the suitable associating similarity between the expression of people's face and obtain qualified a plurality of candidate image.
According to one embodiment of present invention, described a plurality of candidate images for described associating similarity retrieval acquisition, further calculate the associating similarity between described a plurality of candidate images, and use the Dijkstra shortest path first to obtain described retrieving images sequence to guarantee time continuity and Space Consistency.
According to people's face image synthesizing method of the embodiment of the invention, can guarantee accuracy and the flatness of retrieving images sequence.
According to an aspect of the present invention, provide a kind of people's face video synthesizer, it is characterized in that, having comprised: target person expression database, described target person expression database is used for storage multiframe facial image; Pretreatment module, described pretreatment module are used for extracting the posture position of people's face on described each frame facial image of target person expression database and the expression of people's face; Associating similarity retrieval module, the associating similarity retrieval of the expression of the posture position of people's face and people's face on the expression that described associating similarity retrieval module is used for carrying out the posture position of user-defined human face image sequence people face and people's face and the described multiframe facial image is with the retrieving images sequence that obtains mating with described user-defined human face image sequence; And post-processing module, described post-processing module is for described retrieving images sequence is reeled conversion and smoothing processing.
According to people's face video synthesizer of the embodiment of the invention, for user-defined any expression, can both synthesize easily the corresponding photorealism sequence of target person.
According to one embodiment of present invention, described multiframe facial image comprises a plurality of basic facial expression image sequences that gather described target person from a plurality of visual angles and a plurality of posture position, and each the basic facial expression image sequence in wherein said a plurality of basic facial expression image sequences is expressed one's feelings by the change procedure of neutrality expression and described neutral expression and consisted of.
According to people's face video synthesizer of the embodiment of the invention, can comprise great amount of images in the target person expression database.
According to one embodiment of present invention, described pretreatment module is further used for: extract in the described multiframe facial image posture position of people's face in each frame facial image and describe the position of people's face with crab angle, helix angle and side rake angle; An and expression of the people's face in each frame facial image in the described multiframe facial image being aimed at and described with described feature description point people's face with positive standard attitude.
According to people's face video synthesizer of the embodiment of the invention, posture position and expression that can accurate description people face.
According to one embodiment of present invention, described pretreatment module is extracted and is carried out described feature based on active shape model and describe point and detect.
According to one embodiment of present invention, described associating similarity retrieval module is further used for: according to D
Pose=L (| Y
i-Y
j|)+L (| R
i-R
j|)+L (| P
i-P
j|) similarity of carrying out the posture position of people's face on the posture position of people's face in the described user-defined human face image sequence and the described multiframe facial image calculates, wherein Y represents crab angle, and R represents helix angle, and P represents side rake angle, I
iRepresent the arbitrary frame facial image in the described user-defined human face image sequence, I
jRepresent the arbitrary frame facial image in the described multiframe facial image, L (d) is the sigmoid function, is defined as
γ=ln99 wherein, T and σ are respectively average statistical and the standard deviations of variable d; According to described crab angle, described helix angle and described side rake angle, the posture position of people's face in the described user-defined human face image sequence is registered to positive attitude, then basis
Carry out the similarity of the expression of people's face on the expression of people's face in the described user-defined human face image sequence and the described multiframe facial image and calculate, wherein A
I, kIt is image I
iK feature behind the aligning described some A
J, kIt is image I
jK feature behind the aligning described a little, and J is the number of unique point, w
kBe the weight of k unique point, L (d) is the sigmoid function, is defined as
The unique point range normalization that calculates is arrived [0,1]; According to D (i, j)=D
Pose(i, j)+λ D
Expression(i, j) obtains the associating similarity, and wherein parameter lambda is to regulate the weight proportion of expression similarity and posture position similarity; And obtain image I in the user-defined human face image sequence according to described associating similarity
iA plurality of candidate images in described multiframe facial image.
According to people's face video synthesizer of the embodiment of the invention, can obtain posture position and the suitable associating similarity between the expression of people's face and obtain qualified a plurality of candidate image.
According to one embodiment of present invention, for described a plurality of candidate images, described associating similarity retrieval module further uses the Dijkstra shortest path first to obtain described retrieving images sequence to guarantee time continuity and Space Consistency according to the associating similarity between described a plurality of candidate images.
According to people's face video synthesizer of the embodiment of the invention, can guarantee accuracy and the flatness of retrieving images sequence.
Additional aspect of the present invention and advantage in the following description part provide, and part will become obviously from the following description, or recognize by practice of the present invention.
Description of drawings
Above-mentioned and/or the additional aspect of the present invention and advantage are from obviously and easily understanding becoming the description of embodiment below in conjunction with accompanying drawing, wherein:
Fig. 1 is the synoptic diagram of people's face image synthesizing method according to an embodiment of the invention;
Fig. 2 is the process flow diagram of people's face image synthesizing method according to an embodiment of the invention;
Fig. 3 is the process flow diagram of uniting according to an embodiment of the invention the method for similarity retrieval;
Fig. 4 extracts the synoptic diagram that face characteristic is described point based on active shape model (ASM) according to an embodiment of the invention;
Fig. 5 is the synoptic diagram that obtains according to an embodiment of the invention the retrieving images sequence; And
Fig. 6 is the block diagram of people's face video synthesizer according to an embodiment of the invention.
Embodiment
The below describes embodiments of the invention in detail, and the example of described embodiment is shown in the drawings, and wherein identical or similar label represents identical or similar element or the element with identical or similar functions from start to finish.Be exemplary below by the embodiment that is described with reference to the drawings, only be used for explaining the present invention, and can not be interpreted as limitation of the present invention.
Need to prove that in addition, term " first ", " second ", " the 3rd " only are used for describing purpose, and can not be interpreted as indication or hint relative importance or the implicit quantity that indicates indicated technical characterictic.Thus, one or more these features can be expressed or impliedly be comprised to the feature that is limited with " first ", " second ", " the 3rd ".Further, in description of the invention, except as otherwise noted, the implication of " a plurality of " is two or more.
Below with reference to accompanying drawing specific embodiments of the invention are described.
Fig. 1 is the synoptic diagram of people's face image synthesizing method according to an embodiment of the invention.As shown in Figure 1, comprise the multiframe facial image at target person expression database.Expression according to posture position and people's face of people's face in the user-defined human face image sequence, searched targets personage the express one's feelings posture position of people's face of multiframe facial image and the expression of people's face in the database, to retrieve the image sequence of coupling, afterwards to image sequence reel with smoothly.
Fig. 2 is the process flow diagram of people's face image synthesizing method according to an embodiment of the invention.As shown in Figure 2, people's face image synthesizing method may further comprise the steps.
Step S201 sets up target person expression database, comprises the multiframe facial image in the target person expression database.Particularly, target person is made a plurality of basic facial expression image sequences under the various visual angles acquisition system, and target person repeats to do for several times for each basic facial expression in the situation of the posture position that changes head.Gather as far as possible the expression under a plurality of visual angles and a plurality of posture position.Each basic facial expression image sequence is made of the change procedure expression of neutrality expression and neutral expression.Basic facial expression comprises happiness, anger, sadness, detest, fear, surprised etc.
Step S202 carries out pre-service to the multiframe facial image, with the posture position that extracts people's face on each frame facial image and the expression of people's face.Specifically comprise: extract in the multiframe facial image posture position of people's face in each frame facial image and describe the position of people's face with crab angle, helix angle and side rake angle; People's face in each frame facial image in the multiframe facial image aimed at positive standard attitude and carry out feature and describe point and detect, use characteristic is described an expression of describing people's face.
Step S203, carry out the associating similarity retrieval of the expression of the posture position of people's face on the expression of the posture position of people's face in the user-defined human face image sequence and people's face and the multiframe facial image and people's face, to obtain the retrieving images sequence with user-defined human face image sequence coupling.The concrete steps of associating similarity retrieval are described with reference to figure 3.Fig. 3 is the process flow diagram of uniting according to an embodiment of the invention the method for similarity retrieval.As shown in Figure 3, the associating similarity retrieval comprises the steps.
Step S2031 is according to D
Pose=L (| Y
i-Y
j|)+L (| R
i-R
j|)+L (| P
i-P
j|) similarity of carrying out the posture position of people's face on the posture position of people's face in the user-defined human face image sequence and the multiframe facial image calculates, wherein Y represents crab angle, and R represents helix angle, and P represents side rake angle, I
iRepresent the arbitrary frame facial image in the user-defined human face image sequence, I
jArbitrary frame facial image in the expression multiframe facial image, L (d) is the sigmoid function, is defined as
γ=ln99 wherein, T and σ are respectively average statistical and the standard deviations of variable d.
Step S2032 according to described crab angle, described helix angle and described side rake angle, is registered to positive attitude, then basis with the posture position of people's face in the described user-defined human face image sequence
Carry out the similarity of the expression of people's face on the expression of people's face in the user-defined human face image sequence and the multiframe facial image and calculate, wherein A
I, kIt is image I
iK feature behind the aligning described some A
J, kIt is image I
jK feature behind the aligning described a little, and J is the number of unique point, w
kBe the weight of k unique point, L (d) is the sigmoid function, is defined as
The unique point range normalization that calculates is arrived [0,1].The extraction of point is wherein described referring to Fig. 4 Expressive Features.
Fig. 4 extracts the synoptic diagram that face characteristic is described point based on active shape model (ASM) according to an embodiment of the invention.As shown in Figure 4, target person is expressed one's feelings face alignment on each frame facial image in the database to positive standard attitude, then carry out the active shape model feature and describe the some detection, detect a plurality of features to describe a little, these a plurality of features are described the expression that point defines people's face.
Step S2033 is according to D (i, j)=D
Pose(i, j)+λ D
Expression(i, j) obtains the associating similarity of posture position and expression, and wherein parameter lambda is to regulate the weight proportion of expression similarity and posture position similarity.
Step S2034, according to the associating similarity that obtains among the step S2033, for user-defined human face image sequence, retrieve one by one the nearest K frame of target person expression database middle distance image, the K frame of namely uniting the similarity minimum, namely each user images constantly has K candidate image.
Step S2035 for the K that obtains among a step S2034 candidate image, further according to the associating similarity between K the candidate image, uses the Dijkstra shortest path first to obtain the retrieving images sequence to guarantee time continuity and Space Consistency.Wherein, use with above-mentioned steps in associating similarity between the identical method calculating K candidate image.Particularly, utilize the Dijkstra shortest path first to retrieve one and be carved into t=m shortest path constantly during from t=1, wherein m is the length of user-defined human face image sequence.Fig. 5 is the synoptic diagram that obtains according to an embodiment of the invention the retrieving images sequence.Before implementing the Dijkstra shortest path first, need to set up a digraph, as shown in Figure 5, the node of this digraph is by the candidate face image construction of all target persons that retrieve, only allow to have between the candidate face image of adjacent moment directed edge to link to each other, and the length on limit (or being called cost) is defined as:
L(C
t，i，C
t+1，j)＝D(C
t，i，U
t)+D(C
t+1，j，U
t+1)+μD(C
t，i，C
t+1，j)，
Wherein, C
T, iWith C
T+1, jRepresent respectively t i candidate image and j the candidate image in the t+1 moment constantly, U
tWith U
T+1T and t+1 facial image of the definition of difference representative of consumer.The tolerance of this costs between nodes is so that select as far as possible the candidate of rank forward (near as far as possible with the distance of image in the user-defined human face image sequence), again can be so that not significantly sudden change between different constantly candidate image, μ regulates the accuracy of the human face image sequence that retrieves and the parameter of flatness.
Utilize shortest path first, draw node C
1, iWith C
M, jBetween shortest path, can find out that from the permutation and combination of first and last node always total K*K in such path is at this K
2Obtain again that of path minimum in the individual shortest path, just obtained the retrieving images sequence that needs.
Step S204 is to retrieving images sequence reel conversion and smoothing processing.Although the retrieving images sequence that retrieval obtains among the step S203 has kept time continuity and Space Consistency to a certain extent, also have certain difference but compare with the expression of people's face on the target person image, the conversion of therefore need to reeling comes so that the expression of the people's face expression of people's face on the image of the human face image sequence of match user definition more in the retrieving images sequence.Simultaneously, the retrieving images sequence may exist video jitter to jump, and therefore needs further smoothing processing.The concrete steps of smoothing processing are as follows.
For t user images constantly, utilize the feature of extracting to describe the people's face of naming a person for a particular job and be divided into the parts such as eyebrow, eyes, nose, face, calculate respectively U
tAnd C
tThese provincial characteristicss are described the distance of point, if this distance, thinks then that this regional aim personage's action and user's action have certain gap greater than a certain constant δ.For example the action of face not too meets the requirements, and then needs to be used in the target person expression database candidate image that again retrieval and user's face mate most, then the face that mates most that detects is substituted original C
tFace, can use to mend based on people's face of Poisson equation and paint algorithm, thereby when guaranteeing to substitute and the consistance of peripheral region texture, be unlikely to produce obvious feeling of unreality.
On time domain, may there be shake through people's face of mending after painting, needs the distance of adjacent moment between the image after the calculating benefit is painted, if detect shake, take the method for light stream optimization to realize seamlessly transitting, substitute shake constantly some image of front and back occurs.
Fig. 6 is the block diagram of people's face video synthesizer according to an embodiment of the invention.As shown in Figure 6, people's face video synthesizer 10 comprises target person expression database 110, pretreatment module 120, associating similarity retrieval module 130 and post-processing module 140.
Particularly, target person expression database 110 is used for storage multiframe facial image.Pretreatment module 120 is used for extracting the posture position of people's face on target person expression database 110 each frame facial image and the expression of people's face.The associating similarity retrieval of the expression of the posture position of people's face and people's face on the expression that associating similarity retrieval module 130 is used for carrying out the posture position of user-defined human face image sequence people face and people's face and the multiframe facial image is with the retrieving images sequence that obtains mating with user-defined human face image sequence.Post-processing module 140 is for the retrieving images sequence is reeled conversion and smoothing processing.
In one embodiment of the invention, the multiframe facial image comprises that from a plurality of basic facial expression image sequences of a plurality of visual angles and a plurality of posture position collection target person wherein each the basic facial expression image sequence in a plurality of basic facial expression image sequences is expressed one's feelings by the change procedure of neutrality expression and neutral expression and consisted of.Describe a little based on the active shape model detected characteristics.
In one embodiment of the invention, pretreatment module 120 is further used for: extract in the multiframe facial image posture position of people's face in each frame facial image and describe the position of people's face with crab angle, helix angle and side rake angle; And the people's face in each frame facial image in the multiframe facial image is aimed at also use characteristic description put an expression of describing people's face with positive standard attitude.
In one embodiment of the invention, associating similarity retrieval module 130 is further used for: according to D
Pose=L (| Y
i-Y
j|)+L (| R
i-R
j|)+L (| P
i-P
j|) similarity of carrying out the posture position of people's face on the posture position of people's face in the user-defined human face image sequence and the multiframe facial image calculates, wherein Y represents crab angle, and R represents helix angle, and P represents side rake angle, I
iRepresent the arbitrary frame facial image in the user-defined human face image sequence, I
jArbitrary frame facial image in the expression multiframe facial image, L (d) is the sigmoid function, is defined as
γ=ln99 wherein, T and σ are respectively average statistical and the standard deviations of variable d; According to described crab angle, described helix angle and described side rake angle, the posture position of people's face in the described user-defined human face image sequence is registered to positive attitude, then basis
Carry out the similarity of the expression of people's face on the expression of people's face in the user-defined human face image sequence and the multiframe facial image and calculate, wherein A
I, kIt is image I
iK feature behind the aligning described some A
J, kIt is image I
jK feature behind the aligning described a little, and J is the number of unique point, w
kBe the weight of k unique point, L (d) is the sigmoid function, is defined as
The unique point range normalization that calculates is arrived [0,1]; According to D (i, j)=D
Pose(i, j)+λ D
Expression(i, j) obtains the associating similarity, and wherein parameter lambda is to regulate the weight proportion of expression similarity and posture position similarity; And according to the associating similarity obtain image I in the user-defined human face image sequence
iA plurality of candidate images in the multiframe facial image.For a plurality of candidate images, associating similarity retrieval module 130 further uses the Dijkstra shortest path first to obtain the retrieving images sequence to guarantee time continuity and Space Consistency according to the associating similarity between a plurality of candidate images.
People's face image synthesizing method and device according to the embodiment of the invention have following advantage: for user-defined any expression, can both synthesize easily the corresponding photorealism sequence of target person; Reduce the actor number of times; And automaticity is high.
In the description of this instructions, the description of reference term " embodiment ", " some embodiment ", " example ", " concrete example " or " some examples " etc. means to be contained at least one embodiment of the present invention or the example in conjunction with specific features, structure, material or the characteristics of this embodiment or example description.In this manual, the schematic statement of above-mentioned term not necessarily referred to identical embodiment or example.And the specific features of description, structure, material or characteristics can be with suitable mode combinations in any one or more embodiment or example.
Although illustrated and described embodiments of the invention, those having ordinary skill in the art will appreciate that: can carry out multiple variation, modification, replacement and modification to these embodiment in the situation that does not break away from principle of the present invention and aim, scope of the present invention is limited by claim and equivalent thereof.
Although illustrated and described embodiments of the invention, for the ordinary skill in the art, be appreciated that without departing from the principles and spirit of the present invention and can carry out multiple variation, modification, replacement and modification to these embodiment that scope of the present invention is by claims and be equal to and limit.
Claims (8)
1. people's face image synthesizing method is characterized in that, comprises step:
Set up target person expression database, comprise the multiframe facial image in the described target person expression database;
Extract in the described multiframe facial image posture position of people's face in each frame facial image and describe the position of people's face with crab angle, helix angle and side rake angle;
People's face in each frame facial image in the described multiframe facial image aimed at positive standard attitude and carry out feature and describe point and detect, describe the expression of people's face with described feature description;
According to D
Pose=L (| Y
i-Y
j|)+L (| R
i-R
j|)+L (| P
i-P
j|) similarity of carrying out the posture position of people's face on the posture position of people's face in the user-defined human face image sequence and the described multiframe facial image calculates, wherein Y represents crab angle, and R represents helix angle, and P represents side rake angle, I
iRepresent the arbitrary frame facial image in the described user-defined human face image sequence, I
jRepresenting the arbitrary frame facial image in the described multiframe facial image, is the sigmoid function L(d), is defined as
γ=ln99 wherein, T and σ are respectively average statistical and the standard deviations of variable d;
According to described crab angle, described helix angle and described side rake angle, the posture position of people's face in the described user-defined human face image sequence is registered to positive attitude, then basis
Carry out the similarity of the expression of people's face on the expression of people's face in the described user-defined human face image sequence and the described multiframe facial image and calculate, wherein A
I, kIt is image I
iK feature behind the aligning described some A
J, kIt is image I
jK feature behind the aligning described a little, and J is the number of unique point, w
kBe the weight of k unique point, L (d) is the sigmoid function, is defined as
The unique point range normalization that calculates is arrived [0,1];
According to D (i, j)=D
Pose(i, j)+λ D
Expression(i, j) obtains the associating similarity, and wherein parameter lambda is to regulate the weight proportion of expression similarity and posture position similarity;
Obtain image I in the user-defined human face image sequence according to described associating similarity
iA plurality of candidate images in described multiframe facial image, thus retrieving images sequence with described user-defined human face image sequence coupling obtained; And
To described retrieving images sequence reel conversion and smoothing processing.
2. people's face image synthesizing method according to claim 1 is characterized in that, the described step of setting up target person expression database comprises:
Gather a plurality of basic facial expression image sequences of described target person from a plurality of visual angles and a plurality of posture position, each the basic facial expression image sequence in wherein said a plurality of basic facial expression image sequences is expressed one's feelings by the change procedure of neutrality expression and described neutral expression and is consisted of.
3. people's face image synthesizing method according to claim 1 is characterized in that, extracts described feature based on active shape model and describes a little.
4. people's face image synthesizing method according to claim 1, it is characterized in that, described a plurality of candidate images for described associating similarity retrieval acquisition, further calculate the associating similarity between described a plurality of candidate images, and use the Dijkstra shortest path first to obtain described retrieving images sequence to guarantee time continuity and Space Consistency.
5. people's face video synthesizer is characterized in that, comprising:
Target person expression database, described target person expression database is used for storage multiframe facial image;
Pretreatment module, the position that described pretreatment module is used for extracting the posture position of people's face in each frame facial image of described multiframe facial image and describes people's face with crab angle, helix angle and side rake angle, and the people's face in each frame facial image in the described multiframe facial image aimed at positive standard attitude and carry out feature and describe point and detect, the expression of people's face is described with described feature description;
Associating similarity retrieval module, described associating similarity retrieval module is used for carrying out the associating similarity retrieval of the expression of the posture position of people's face on the expression of the posture position of user-defined human face image sequence people face and people's face and the described multiframe facial image and people's face, with the retrieving images sequence that obtains mating with described user-defined human face image sequence, wherein said associating similarity retrieval comprises according to D
Pose=L (| Y
i-Y
j|)+L (| R
i-R
j|)+L (| P
i-P
j|) similarity of carrying out the posture position of people's face on the posture position of people's face in the described user-defined human face image sequence and the described multiframe facial image calculates, wherein Y represents crab angle, and R represents helix angle, and P represents side rake angle, I
iRepresent the arbitrary frame facial image in the described user-defined human face image sequence, I
jRepresenting the arbitrary frame facial image in the described multiframe facial image, is the sigmoid function L(d), is defined as
γ=ln99 wherein, T and σ are respectively average statistical and the standard deviations of variable d; According to described crab angle, described helix angle and described side rake angle, the posture position of people's face in the described user-defined human face image sequence is registered to positive attitude, then basis
Carry out the similarity of the expression of people's face on the expression of people's face in the described user-defined human face image sequence and the described multiframe facial image and calculate, wherein A
I, kIt is image I
iK feature behind the aligning described some A
J, kIt is image I
jK feature behind the aligning described a little, and J is the number of unique point, w
kBe the weight of k unique point, L (d) is the sigmoid function, is defined as
The unique point range normalization that calculates is arrived [0,1]; According to D (i, j)=D
Pose(i, j)+λ D
Expression(i, j) obtains the associating similarity, and wherein parameter lambda is to regulate the weight proportion of expression similarity and posture position similarity; With obtain image I in the user-defined human face image sequence according to described associating similarity
iA plurality of candidate images in described multiframe facial image; And
Post-processing module, described post-processing module is for described retrieving images sequence is reeled conversion and smoothing processing.
6. people's face video synthesizer according to claim 5, it is characterized in that, described multiframe facial image comprises a plurality of basic facial expression image sequences that gather described target person from a plurality of visual angles and a plurality of posture position, and each the basic facial expression image sequence in wherein said a plurality of basic facial expression image sequences is expressed one's feelings by the change procedure of neutrality expression and described neutral expression and consisted of.
7. people's face video synthesizer according to claim 5 is characterized in that, described pretreatment module is extracted based on active shape model and carried out the detection of described feature description point.
8. people's face video synthesizer according to claim 5, it is characterized in that, for described a plurality of candidate images, described associating similarity retrieval module further uses the Dijkstra shortest path first to obtain described retrieving images sequence to guarantee time continuity and Space Consistency according to the associating similarity between described a plurality of candidate images.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN 201110197873 CN102254336B (en) | 2011-07-14 | 2011-07-14 | Method and device for synthesizing face video |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN 201110197873 CN102254336B (en) | 2011-07-14 | 2011-07-14 | Method and device for synthesizing face video |
Publications (2)
| Publication Number | Publication Date |
|---|---|
| CN102254336A CN102254336A (en) | 2011-11-23 |
| CN102254336B true CN102254336B (en) | 2013-01-16 |
Family
ID=44981577
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN 201110197873 Active CN102254336B (en) | 2011-07-14 | 2011-07-14 | Method and device for synthesizing face video |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN102254336B (en) |
Cited By (1)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| US11595617B2 (en) | 2012-04-09 | 2023-02-28 | Intel Corporation | Communication using interactive avatars |
Families Citing this family (19)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| WO2014194439A1 (en) * | 2013-06-04 | 2014-12-11 | Intel Corporation | Avatar-based video encoding |
| CN103647922A (en) * | 2013-12-20 | 2014-03-19 | 百度在线网络技术（北京）有限公司 | Virtual video call method and terminals |
| WO2016101131A1 (en) | 2014-12-23 | 2016-06-30 | Intel Corporation | Augmented facial animation |
| CN104679832A (en) * | 2015-02-05 | 2015-06-03 | 四川长虹电器股份有限公司 | System and method for searching single or multi-body combined picture based on face recognition |
| CN105184249B (en) * | 2015-08-28 | 2017-07-18 | 百度在线网络技术（北京）有限公司 | Method and apparatus for face image processing |
| WO2017101094A1 (en) | 2015-12-18 | 2017-06-22 | Intel Corporation | Avatar animation system |
| CN106303233B (en) * | 2016-08-08 | 2019-03-15 | 西安电子科技大学 | A kind of video method for secret protection based on expression fusion |
| CN109670386A (en) * | 2017-10-16 | 2019-04-23 | 深圳泰首智能技术有限公司 | Face identification method and terminal |
| CN109784123A (en) * | 2017-11-10 | 2019-05-21 | 浙江思考者科技有限公司 | The analysis and judgment method of real's expression shape change |
| CN108510435A (en) * | 2018-03-28 | 2018-09-07 | 北京市商汤科技开发有限公司 | Image processing method and device, electronic equipment and storage medium |
| CN109886091B (en) * | 2019-01-08 | 2021-06-01 | 东南大学 | Three-dimensional facial expression recognition method based on weighted local rotation mode |
| CN109993102B (en) * | 2019-03-28 | 2021-09-17 | 北京达佳互联信息技术有限公司 | Similar face retrieval method, device and storage medium |
| CN110675433A (en) * | 2019-10-31 | 2020-01-10 | 北京达佳互联信息技术有限公司 | Video processing method and device, electronic equipment and storage medium |
| CN111274447A (en) * | 2020-01-13 | 2020-06-12 | 深圳壹账通智能科技有限公司 | Target expression generation method, device, medium and electronic equipment based on video |
| CN111652121B (en) * | 2020-06-01 | 2023-11-03 | 腾讯科技（深圳）有限公司 | Training method of expression migration model, and method and device for expression migration |
| CN111833413B (en) * | 2020-07-22 | 2022-08-26 | 平安科技（深圳）有限公司 | Image processing method, image processing device, electronic equipment and computer readable storage medium |
| CN113269872A (en) * | 2021-06-01 | 2021-08-17 | 广东工业大学 | Synthetic video generation method based on three-dimensional face reconstruction and video key frame optimization |
| CN113436302B (en) * | 2021-06-08 | 2024-02-13 | 合肥综合性国家科学中心人工智能研究院(安徽省人工智能实验室) | Face animation synthesis method and system |
| CN114429611B (en) * | 2022-04-06 | 2022-07-08 | 北京达佳互联信息技术有限公司 | Video synthesis method and device, electronic equipment and storage medium |
Citations (3)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN1920886A (en) * | 2006-09-14 | 2007-02-28 | 浙江大学 | Video flow based three-dimensional dynamic human face expression model construction method |
| CN1920880A (en) * | 2006-09-14 | 2007-02-28 | 浙江大学 | Video flow based people face expression fantasy method |
| CN101179665A (en) * | 2007-11-02 | 2008-05-14 | 腾讯科技（深圳）有限公司 | Method and device for transmitting face synthesized video |
-
2011
- 2011-07-14 CN CN 201110197873 patent/CN102254336B/en active Active
Patent Citations (3)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN1920886A (en) * | 2006-09-14 | 2007-02-28 | 浙江大学 | Video flow based three-dimensional dynamic human face expression model construction method |
| CN1920880A (en) * | 2006-09-14 | 2007-02-28 | 浙江大学 | Video flow based people face expression fantasy method |
| CN101179665A (en) * | 2007-11-02 | 2008-05-14 | 腾讯科技（深圳）有限公司 | Method and device for transmitting face synthesized video |
Non-Patent Citations (3)
| Title |
|---|
| Ira Kemelmacher-Shlizerman, Aditya Sankar, Eli Shechtman, and St.Being John Malkovich.《European Conference on Computer Vision》.2010,1-13. * |
| Qingshan Zhang, et al..geometry-driven photorealistic facial expression synthesis.《IEEE Transactions on Visualization and Computer Graphics》.2006,第12卷(第1期),48-60. * |
| 雷林华.基于视频序列的虚拟人脸合成.《中国优秀硕士学位论文全文数据库 信息科技辑》.2008,(第1期),全文. * |
Cited By (1)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| US11595617B2 (en) | 2012-04-09 | 2023-02-28 | Intel Corporation | Communication using interactive avatars |
Also Published As
| Publication number | Publication date |
|---|---|
| CN102254336A (en) | 2011-11-23 |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| CN102254336B (en) | Method and device for synthesizing face video | |
| CN111652828B (en) | Face image generation method, device, equipment and medium | |
| Ge et al. | 3d convolutional neural networks for efficient and robust hand pose estimation from single depth images | |
| Liu et al. | Fusedream: Training-free text-to-image generation with improved clip+ gan space optimization | |
| Zhou et al. | Photorealistic facial expression synthesis by the conditional difference adversarial autoencoder | |
| Toshev et al. | Deeppose: Human pose estimation via deep neural networks | |
| CN106600626B (en) | Three-dimensional human motion capture method and system | |
| Lei et al. | Whole-body humanoid robot imitation with pose similarity evaluation | |
| KR101711736B1 (en) | Feature extraction method for motion recognition in image and motion recognition method using skeleton information | |
| CN102567716B (en) | Face synthetic system and implementation method | |
| CN104008564A (en) | Human face expression cloning method | |
| KR101563297B1 (en) | Method and apparatus for recognizing action in video | |
| US10147023B1 (en) | Markerless face tracking with synthetic priors | |
| CN102509333A (en) | Action-capture-data-driving-based two-dimensional cartoon expression animation production method | |
| Uddin et al. | Human Activity Recognition via 3-D joint angle features and Hidden Markov models | |
| CN115083015B (en) | 3D human body posture estimation data labeling mode and corresponding model construction method | |
| Lin et al. | Facial expression recognition with data augmentation and compact feature learning | |
| Lu et al. | Multi-task learning for single image depth estimation and segmentation based on unsupervised network | |
| Seddik et al. | Unsupervised facial expressions recognition and avatar reconstruction from Kinect | |
| Camurri et al. | Visual Gesture Recognition: from motion tracking to expressive gesture | |
| Neverova | Deep learning for human motion analysis | |
| CN106778576B (en) | Motion recognition method based on SEHM characteristic diagram sequence | |
| Zhang et al. | Styleavatar3d: Leveraging image-text diffusion models for high-fidelity 3d avatar generation | |
| CN103116901A (en) | Motion characteristic based human motion interpolation calculation method | |
| Tan | Evolution of art form of video animation design under the background of computer graphics system development |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| C06 | Publication | ||
| PB01 | Publication | ||
| C10 | Entry into substantive examination | ||
| SE01 | Entry into force of request for substantive examination | ||
| C14 | Grant of patent or utility model | ||
| GR01 | Patent grant |