CN103279936A - Human face fake photo automatic combining and modifying method based on portrayal - Google Patents

Abstract

The invention discloses a human face fake photo automatic combining and modifying method based on a portrayal. The method comprises the following steps: firstly, automatically generating initial estimation of a fake photo by applying a partial intrinsic transform method; then, extracting hair and facial contour information from an input human face portrayal, and carrying out automatic enhancement on the initial estimation; finally, judging whether partial combining errors exist in the automatically-combined human face fake photo by a user, and if obvious combining errors exist, carrying out local deformation on a corresponding human face of the re-input fake photo by adopting a deformation method based on control points, then carrying out automatic combination again, and therefore modifying the partial combining errors. In a modifying process, an active shape model is used for obtaining the feature points of the human face portrayal, and rigid deformation based on the control points is carried out by using a least square method. The human face fake photo automatic combining and modifying method based on the portrayal can automatically combine a human face fake photo fast and effectively according to the input human face portrayal, and can provides assistance for manual and automatic identification of human face sketch portrayals in criminal investigation, anti-terrorist and other fields.

Description

Automatically synthesize and modification method based on the pseudo-photo of people's face of portrait

Technical field

The invention belongs to technical field of image processing, particularly the pseudo-photo of a kind of people's face based on portrait synthesizes and modification method automatically, and this method can provide auxiliary for the artificial and automatic identification to the human face sketch portrait in the fields such as criminal investigation, anti-terrorism.

Background technology

In the criminal investigation enforcing law, when can not determine that suspect's identity also can't obtain its image data, public security organ can require the professional to draw out suspect's sketch portrait according to eyewitness's description.According to this sketch portrait, related thread and information can be collected to the public by public security organ.

The pseudo-photo of people's face that is converted to by human face portrait has the visual effect similar with the real human face photo, its function and significance mainly contains: 1) traditional face identification method can't be directly used in the identification between human face portrait and the photo, and after being transformed into photo and portrait under the information representation pattern of the same race, can utilize existing face identification method to identify automatically.2) the synthetic pseudo-photo of people's face than the sketch portrait more lively directly perceived, be easy to identification, therefore can play good booster action for manually distinguishing of suspect's identity.3) in the render phase of sketch portrait, the pseudo-photo of synthetic people's face can help eyewitness and artist alternatively to revise drawn portrait.

The difficult point of the pseudo-photo of synthetic people's face is to seek the mapping relations between human face portrait and the human face photo.The most representative method mainly contains following several at present:

People such as Tang have proposed a kind of overall eigentransformation method (Eigentransformation) in document " X.Tang; X.Wang; Face Sketch Recognition; IEEE Transactions on Circuits and Systems for Video Technology14 (1) (2004) 50-57. ", the mapping relations between this method hypothesis photo and the portrait are linear.But studies show that thereafter, these mapping relations can not be summed up as the linear relationship of the overall situation simply.

People such as Liu in the document " Q.Liu; X.Tang; H.Jin; A nonlinear approach for face sketch synthesis and recognition; Proc.IEEE Conference on Computer Vision and Pattern Recognition (2005) 1005-1010. " method of manifold learning introduced human face portrait synthetic in, the nonlinear relationship between utilizing local linear embedding grammar (Locally Linear Embedding) study photo and drawing a portrait.Because this method has only been used several samples of arest neighbors in the training set when finding the solution reconstruction coefficients, therefore can there be bigger reconstructed error in the subregion.

People such as Xiao propose to utilize built-in type hidden Markov model (Embedded Hidden Markov Model) to generate the pseudo-portrait of people's face in document " B.Xiao; X.Gao; D.Tao; X.Li; A new approach for face recognition by sketches in photos; Signal Processing89 (8) (2009) 1576 – 1588. ", because this method has adopted the selectivity integrated technology, need to select K arest neighbors of photo to be transformed to carry out modeling, therefore caused losing of detailed information, the algorithm complex of this method is higher in addition.

For the detailed information to pseudo-photo strengthens, the researchist in order to the pseudo-photo of the synthetic people's face of last method as the initial estimation image, adopt then based on the method for rarefaction representation or based on the detailed information of the synthetic pseudo-photo of method of support vector regression, initial estimation image and detailed information are merged obtain final synthetic result at last.These class methods can be used for the enhancing of the pseudo-photo of people's face and pseudo-portrait simultaneously, but its computation complexity is bigger.

There is tangible resultant fault in regional area at the pseudo-photo of some faces, in order to revise these mistakes, people such as Purkait have adopted the deformation method based on the reference mark in document " P.Purkait; B.Chanda; S.Kulkarni; A Novel Technique for Sketch to Photo Synthesis, Seventh Indian Conference on Computer Vision, Graphics and Image Processing (2010) 219-226. ".Though this method can reduce the resultant fault of subregion, but owing to relate to the bulk deformation of people's face All Ranges in the building-up process, therefore, the pseudo-photo of people's face of output is compared with real human face photo, can occur decline to a certain degree on overall similarity inevitably.And this method must rely on complicated neural network to predict the target location at a large amount of reference mark among the output result.

Summary of the invention

The objective of the invention is to propose the pseudo-photo of a kind of people's face based on portrait synthetic and modification method automatically.This method can be quickly and efficiently according to the human face portrait of the input pseudo-photo of synthetic people's face automatically, can in the fields such as criminal investigation, anti-terrorism to the artificial of human face sketch portrait and automatically identification provide auxiliary.

For realizing this purpose, technical scheme of the present invention is: at first, use the initial estimation image that local eigentransformation method generates pseudo-photo automatically; Then, extraction hair and face mask information strengthen automatically to the initial estimation image from the human face portrait of input; At last, judge by the user whether the pseudo-photo of synthetic automatically people's face exists local resultant fault, if there is tangible resultant fault, then adopt based on the deformation method at reference mark the human face portrait of the pseudo-photo correspondence re-entered is carried out local deformation, and then synthesize automatically again, thereby revise local resultant fault.Implementation step is as follows:

1, divide training set and test set: the human face portrait database is divided into training set and test set, and wherein the shared ratio of test set is 40%～60%, then portrait and photo corresponding in the training set is had overlapping piecemeal.Classify to portrait piece and photo piece in position according to the piece place, all portrait pieces of same position constitute a portrait piece training set, and same, all photo pieces of same position constitute a photo piece training set.

2, input test portrait: in test set, appoint and get a portrait, as the test portrait of input.

3, generate the initial estimation image of the pseudo-photo of people's face automatically:

3a) the test portrait there is overlapping piecemeal;

3b) for a test portrait piece, utilize the eigentransformation principle in the overall eigentransformation method to try to achieve its reconstruction coefficients in the portrait piece training set of same position, utilize this reconstruction coefficients in the photo piece training set of correspondence, to synthesize pseudo-photo piece automatically;

3c) try to achieve step 3a) in the pseudo-photo piece of each test portrait piece correspondence, their position splicings when getting piece are gone back, obtain the initial estimation image of the pseudo-photo of people's face of the overall situation.When the pseudo-photo piece of automatic splicing, for the overlapping region of adjacent block, gray-scale value equals the mean value of each overlapping block.

4, extract hair and face mask information: the face zone of test portrait selected in the step (2) is blocked with white template, only keep zones such as hair, face mask, then gray scale is carried out in the zone that keeps and promote, extract information such as comparatively outstanding edge, texture.

5, the initial estimation image is strengthened automatically: the initial estimation image of the pseudo-photo of people's face that generates automatically in the image that extracts in the step (4) and the step (3) is multiplied each other, realize the automatic enhancing to the initial estimation image.

6, judgement has or not local resultant fault: judged by the user to have or not local resultant fault, if there is not local resultant fault in the pseudo-photo of the people's face after strengthening, then export the pseudo-photo of this people's face.If there is tangible resultant fault in the regional area of the pseudo-photo of people's face, then enter next step.

7, re-enter the test portrait: if there is local resultant fault in the pseudo-photo of people's face, then re-enter the test portrait of this puppet photo correspondence.

8, regional area is deformed to mean place:

(Active Shape Models ASM) obtains the unique point of testing all portraits in portrait and the training set, calculates the mean place of portrait lip region according to unique point 8a) to utilize active shape model;

(Moving Least Squares MLS) carries out distortion based on the reference mark to test portrait, and the lip region on the test portrait is moved to mean place, keeps the invariant position in other face zones simultaneously 8b) to utilize mobile least square method.

9, the processing of passing through (3)～(5) step again of the portrait of the test after will being out of shape synthesizes the initial estimation image automatically and strengthens, and at this moment the resultant fault of its lip region has obtained correction, but its lip region still is positioned at mean place.

10, regional area is out of shape go back to the original position: adopt the deformation method based on the reference mark, according to test portrait original characteristics point position, the lip region of the pseudo-photo of people's face is out of shape go back to its original position, thereby finishes the local correction of the pseudo-photo of people's face and export the pseudo-photo of this people's face.

Compared with prior art, advantage of the present invention is:

(1) the present invention uses the initial estimation image that local eigentransformation method generates pseudo-photo, and this method belongs to nonlinear synthetic method, and its algorithm complex is lower.Local eigentransformation method is improvement and the popularization to overall eigentransformation method, is about to overlapping piecemeal and combines with overall eigentransformation.The pseudo-photo piece of its synthetic people's face is that all the photo piece reconstruct by same position in the training set obtain, and does not relate to the selection of k nearest neighbor in this process, therefore can keep reconstruct effect preferably.

(2) at pseudo-this problem of photo of synthetic people's face, the characteristics according to the human face portrait of importing have proposed the automatic Enhancement Method of a kind of more simple and effective pseudo-photo.

(3) the present invention obtains the unique point of human face portrait equally according to active shape model, but when the distortion of carrying out based on the reference mark, do not need whole human face portrait all is deformed into average shape, to be deformed to mean place with the regional area that mean place differs bigger but use mobile least square method, keep other important area invariant positions simultaneously.So just can guarantee that when revising local resultant fault overall similarity can not descend.In addition, owing to only relate to the distortion of regional area in the method that the present invention adopts, thus do not need to rely on neural network to predict the target location at reference mark among the output result, thus computation complexity reduced.

The present invention can be quickly and efficiently according to the human face portrait of the input pseudo-photo of synthetic people's face automatically, can in the fields such as criminal investigation, anti-terrorism to the artificial of human face sketch portrait and automatically identification provide auxiliary.

Description of drawings

Fig. 1: the ultimate principle block diagram of the inventive method.

Fig. 2: the initial estimation image comparison synoptic diagram (CUHK student human face portrait storehouse) of the pseudo-photo of people's face that the inventive method and classic method are synthetic.Wherein: (a) test portrait; (b) overall eigentransformation method; (c) local linear embedding grammar; (d) the inventive method; (e) real pictures.

Fig. 3: the contrast synoptic diagram (CUHK student human face portrait storehouse) before and after the pseudo-photo that the inventive method obtains strengthens.Wherein: (a) test portrait; (b) the initial estimation image of pseudo-photo; (c) the pseudo-photo after the enhancing; (d) real pictures.

Fig. 4: the pseudo-photo that the present invention obtains carries out the pseudo-photo contrast synoptic diagram (CUHK student human face portrait storehouse) before and after the local correction.Wherein: (a) test portrait; (b) the initial estimation image of pseudo-photo; (c) enhancing and revised pseudo-photo; (d) real pictures.

Concrete implementation step

The specific embodiment of the present invention is as follows:

(1) divide training set and test set:

1a) with CUHK student human face portrait database (the open face database of using, can find http://mmlab.ie.cuhk.edu.hk/facesketch.html in following network address) in paired portrait and photo be divided into training set and test set, wherein training set comprises 88 pairs of portraits and photo, and test set comprises 100 pairs of portraits and photo;

1b) human face portrait paired in the training set and photo are had overlapping piecemeal, the size of lap is for dividing 1/3 of block size between adjacent human face portrait piece or the photo piece.In the present embodiment, the pixel size of the human face portrait in the training set and photo is 200 * 250.They are expanded 4 pixels in the horizontal direction and assignment is 0, expanding 2 pixels and assignment in vertical direction is 0, and then the pixel size of human face portrait and photo becomes 204 * 252 in the training set.To divide block size to be taken as 12 * 12, overlapping size is taken as 4 * 12, thereby every human face portrait or photo can be divided into 775 rectangular blocks.Classify to portrait piece and photo piece in position according to the piece place, all portrait pieces of same position constitute a portrait piece training set, and same, all photo pieces of same position constitute a photo piece training set.

(2) input test portrait: in test set, appoint and get a portrait, as the test portrait of input.

(3) generate the initial estimation image of the pseudo-photo of people's face automatically:

3a) the test portrait there is overlapping piecemeal;

3b) j test portrait piece is expressed as column vector S ^j, by principal component analysis (PCA) (Principal Component Analysis, PCA) calculate the proper vector of the training set portrait piece of its same position:

$U_s^j=A_s^j{V}_s^j{\mathrm{\Λ}}_s^{j-1/2}$

Wherein $A_s^j=[S_1^j-m_s^j,\·\·\·,S_M^j-m_s^j]=[S_1^{\′j},\·\·\·,S_M^{\′j}],$ $S_i^{\′j},i=\mathrm{1,2},\·\·\·,M$ Expression deducts portrait piece average After training set portrait piece, M is 88 in the present embodiment.

With

Expression respectively

Proper vector and eigenwert, the transposition of subscript T representing matrix.

To go the test portrait piece after the average to project to

In obtain projection coefficient

And try to achieve and test the reconstruction coefficients of portrait piece in portrait piece training set:

$c^j=V_s^j{\mathrm{\Λ}}_s^{j-1/2}{w}_s^j={[c_1^j,\·\·\·,c_M^j]}^T$

Utilize this reconstruction coefficients in photo piece training set, to be reconstructed, obtain according to the synthetic pseudo-photo piece of test portrait piece:

$P_r^j=\underset{i=1}{\overset{M}{\mathrm{\Σ}}}{c}_i^j{P}_i^{\′j}+m_p^j$

Wherein

Expression deducts the training set photo piece after the average, Expression photo piece average;

3c) try to achieve step 3a) in the pseudo-photo piece of each test portrait piece correspondence, their position splicings when getting piece are gone back, obtain the initial estimation image of the pseudo-photo of people's face of the overall situation.When the pseudo-photo piece of automatic splicing, for the overlapping region of adjacent block, gray-scale value equals the mean value of each overlapping block.At this moment, the pixel size of pseudo-photo initial estimation image is 204 * 252, reduces pixel value on the edge and is after 0 the row and column, and pixel size becomes 200 * 250.

(4) extract hair and face mask information:

4a) block the face zone of selected test human face portrait with white template, only keep zones such as hair, face mask.In the present embodiment, with (x, y) the expression size is the pixel of 200 * 250 test human face portrait, and the pixel in the upper left corner is designated as initial point (0,0).For pixel (56,106), (145,106), (56,145), (145,145) definite rectangular area and pixel (75,146), (125,146), (75,195), (125,195) definite rectangular area, all gray values of pixel points that are positioned at its scope are 255 by tax all, can realize blocking the face zone like this;

4b) gray scale is carried out in the zone that keeps and promote, namely all gray values of pixel points multiply by 1.5, extract information such as comparatively outstanding edge, texture.

(5) the initial estimation image is strengthened automatically: the initial estimation image of the pseudo-photo of people's face that generates in the image that extracts in the step (4) and the step (3) is multiplied each other, realize the automatic enhancing to the initial estimation image.

(6) judgement has or not local resultant fault: judged by the user to have or not local resultant fault, if there is not local resultant fault in the pseudo-photo of the people's face after strengthening, then export the pseudo-photo of this people's face.If there is tangible resultant fault in the regional area of the pseudo-photo of people's face, then enter next step.

(7) re-enter the test portrait: if there is local resultant fault in the pseudo-photo of people's face, then re-enter the test portrait of this puppet photo correspondence.

(8) regional area is deformed to mean place:

8a) utilize active shape model to obtain 53 unique points of each portrait in test portrait and the training set, calculate portrait 8 unique points of lip region mean place separately according to these unique points;

8b) utilize mobile least square method that the test human face portrait is carried out distortion based on the reference mark.In the present embodiment, choose 8 unique points of 3 unique points of 6 unique points, nasal area of eye areas and lip region as the Deformation control point.Reference mark coordinate before the distortion is designated as d _i, i=1,2 ..., 17, the reference mark coordinate after the distortion is designated as s _i, i=1,2 ..., 17, back 8 reference mark that point is lip region among the set d at reference mark and the s.For the lip region that will test on the portrait moves to mean place, keep the invariant position in other face zones such as eyes simultaneously, need make preceding 9 points of s equal preceding 9 points of d, back 8 points of s equal 8 unique points of lip region mean place separately.What adopt based on the distortion at reference mark among the present invention is reverse mapping, and namely for the arbitrary pixel v in the portrait of the test after the distortion, the position f that need obtain before its distortion (v), composes this position gray values of pixel points to the corresponding pixel in distortion back then.This warping function f can make following formula get minimum value:

$\underset{i}{\mathrm{\Σ}}{w}_i{|f\left(s_{\text{i}}\right)-d_i|}^2$

W wherein _i=1/|s _i-v| ^{2 α}Be weight, α is for regulating the parameter of deformation effect, and value is 2.The warping function of selecting among the present invention is the rigid transformation warping function, and its expression formula is:

$f_r\left(v\right)=\underset{i}{\mathrm{\Σ}}{\hat{d}}_i\left(\frac{1}{k_r}{A}_i\right)+d_{*}$

Wherein:

$A_i=w_i\left(\begin{array}{c}{\hat{s}}_i\\ -{\hat{s}}_i^{\⊥}\end{array}\right){\left(\begin{array}{c}v-s_{*}\\ {-(v-s_{*})}^{\⊥}\end{array}\right)}^T$

$k_r=\sqrt{\underset{i}{\mathrm{\Σ}}{\left(w_i{\hat{d}}_i{\hat{s}}_i^T\right)}^2+\underset{i}{\mathrm{\Σ}}{\left(w_i{\hat{d}}_i{\hat{s}}_i^{\⊥T}\right)}^2}$

Wherein, ${\hat{s}}_i=s_i-s_{*},$ ${\hat{d}}_i=d_i-d_{*},$ $s_{*}=\underset{i}{\mathrm{\Σ}}{w}_i{s}_i/\underset{i}{\mathrm{\Σ}}{w}_i,$ $d_{*}=\underset{i}{\mathrm{\Σ}}{w}_i{d}_i/\underset{i}{\mathrm{\Σ}}{w}_i,$ ⊥ is a bivector operational character, (x, y)=(y, x) ^⊥

(9) processing of passing through (3)～(5) step again of the portrait of the test after will being out of shape synthesizes the initial estimation image automatically and strengthens, and at this moment the resultant fault of its lip region has obtained correction, but its lip region still is positioned at mean place.

(10) regional area is out of shape go back to the original position: adopt the deformation method based on the reference mark, according to test portrait original characteristics point position, the lip region of the pseudo-photo of people's face is out of shape go back to its original position, thereby finishes the local correction of the pseudo-photo of people's face and export the pseudo-photo of this people's face.

The simulation analysis of computer of the inventive method:

Figure 2 shows that the pseudo-photo initial estimation of the synthetic people's face of overall eigentransformation method, local linear embedding grammar and the inventive method image.Local linear embedding grammar is a kind of comparatively classical non-linear synthetic method, and it is that follow-up improvement is carried out on the basis with it that a lot of methods are arranged in recent years, therefore selects its object as a comparison in experiment.From experimental result as can be seen, overall situation eigentransformation method can produce tangible diplopia phenomenon, local linear embedding grammar also can produce more reconfiguring false, and the inventive method is local eigentransformation method, not only removed the diplopia of facial contour, and guaranteed that there is comparatively clear, level and smooth synthetic effect in the face zone.

Figure 3 shows that the pseudo-photo of people's face before and after strengthening, wherein, the initial estimation image of the pseudo-photo of people's face is to adopt local eigentransformation method synthetic.As can be seen from Figure 3, the hair and the face mask that strengthen the pseudo-photo of descendant's face become more clear and outstanding, thereby make its whole visual effect obtain certain lifting.

Figure 4 shows that the pseudo-photo of people's face of local correction front and back.From experimental result as can be seen, the local resultant fault of lip region can significantly be revised, thereby makes the visual effect of the pseudo-photo of synthetic people's face obtain bigger improvement.

Claims (4)

1. the pseudo-photo of the people's face based on portrait synthesizes and modification method automatically, and its characteristics are said method comprising the steps of:

(1) the human face portrait database is divided into training set and test set, portrait paired in the training set and photo are had overlapping piecemeal, according to the position at piece place portrait piece and photo piece are classified, generate portrait piece and photo piece training set;

(2) in test set, appoint and get a portrait, as the test portrait of input, use the initial estimation image that local eigentransformation method generates pseudo-photo automatically;

(3) from the test portrait of input, extract hair and face mask information, utilize these information that the initial estimation image of pseudo-photo is strengthened automatically;

(4) judge by the user whether the pseudo-photo of people's face after the enhancing exists local resultant fault, and if there is no local resultant fault is then exported the pseudo-photo of this people's face; If there is tangible resultant fault in the regional area of the pseudo-photo of people's face, then adopt based on the deformation method at reference mark the test portrait of the pseudo-photo correspondence re-entered is carried out local deformation, and then synthesize automatically again, thereby revise local resultant fault and export the pseudo-photo of this people's face.

2. the pseudo-photo of the people's face based on portrait according to claim 1 synthetic and modification method automatically, its characteristics be, the method for synthetic pseudo-photo initial estimation image is as follows automatically in local eigentransformation in the described step (2):

2a) the test portrait there is overlapping piecemeal;

2b) for a test portrait piece, utilize the eigentransformation principle in the overall eigentransformation method to try to achieve its reconstruction coefficients in the portrait piece training set of same position, utilize this reconstruction coefficients in the photo piece training set of correspondence, to synthesize pseudo-photo piece automatically;

2c) try to achieve step 2a) in the pseudo-photo piece of each test portrait piece correspondence, their position splicings when getting piece are gone back, obtain the initial estimation image of the pseudo-photo of people's face of the overall situation; When the pseudo-photo piece of automatic splicing, for the overlapping region of adjacent block, gray-scale value is got the mean value of each overlapping block.

3. the pseudo-photo of the people's face based on portrait according to claim 1 synthetic and modification method automatically, its characteristics be, the method that in the described step (3) the initial estimation image of pseudo-photo is strengthened automatically is as follows:

3a) with white template the face zone of test portrait selected in the step (2) is blocked, only keep zones such as hair, face mask, then gray scale being carried out in the zone that keeps promotes, namely all gray values of pixel points multiply by 1.5, thereby extract information such as comparatively outstanding edge, texture;

3b) with step 3a) in the initial estimation image of the pseudo-photo of people's face that generates automatically in the image that extracts and the step (2) multiply each other, realize the automatic enhancing to the initial estimation image.

4. the pseudo-photo of the people's face based on portrait according to claim 1 synthesizes and modification method automatically, and its characteristics are that the method for revising the local resultant fault of pseudo-photo in the described step (4) is as follows:

4a) utilize active shape model to obtain the unique point of testing all portraits in portrait and the training set, calculate the mean place of portrait lip region according to unique point;

4b) utilize mobile least square method that test portrait is carried out distortion based on the reference mark, the lip region on the test portrait is moved to mean place, keep the invariant position in other face zones simultaneously;

4c) processing of passing through (2)～(3) step again of the portrait of the test after will being out of shape synthesizes the initial estimation image automatically and strengthens, and at this moment the resultant fault of its lip region has obtained correction;

4d) employing, is out of shape go back to its original position with the lip region of the pseudo-photo of people's face, thereby is finished the local correction of the pseudo-photo of people's face according to test portrait original characteristics point position based on the deformation method at reference mark.

Images